Given the average HR of 68, normal or high BP assumed, and the low dose level of Toprol XL I think you should be fine. Check you HR a few times a day until you feel comfortable.
If you have an athletic history, a low HR, even in the 50s may be all you need due to a strong/athletic cardiovascular system.
P.S. If your HR gets too low you will know it because you'll get other symptoms, e.g., dizziness, vision problems, sweating... lots of non-fatal symptoms to warn you of shortage of oxygen to your system.
Do you feel the ectopic beats? My test showed that I was getting 341 of them, I feel them all. I can't stand it. I'm on Toprol XL 50mg and it has cut the ectopic in about half. I still hate that I can feel them.