I think the bottom line is that while taking beta blockers you will not have the same aerobic capacity that you had previously. If you were a marathoner, it would definitely affect your performance and slow you way down. The BB decreases your target heart rate, as you've noticed, and you simply don't have the same blood flow any more.
On the other hand, I don't think this will make your heart less strong over time. Beta blockers are given to those with cardiomyopathy to rest their heart, allowing it to strengthen. You'll have to substitute duration for intensity... but exercise will still help.
Here is some previous discussion on BBs and exercise from the doctorm to patient board. This doesn't adderess the specific question you asked but I think you might find it interesting.
http://www.medhelp.org/forums/cardio/messages/33747.html