You are already doing so much to reduce your risks, and I would say that you are probably in very good health. You are seeing a cardiologist regularly (I say that because of your prescription to Toprol and the likelihood that you see a cardio at least once per year). Your cardiologist is aware of your family history, and will no doubt monitor you closely, just to be sure. However, you are one of those unlucky, lucky people who have PVC's/PAC's. I say "unlucky, lucky" because they are a real pain to deal with. Very bothersome and worrisome. However, especially when they are benign, they are nice little instigators for going to see a cardiologist regularly and having a great deal of information about your heart. Every so often, those of us with PVC's etc. go for a battery of tests that normally are only done when you have a full blown heart attack. I know more about my heart muscle than my Dad ever has. He's had a massive coronary that he barely survived, and I STILL know more about my heart health than he does.
Your grandfather's death is more a mystery than anything. During that generation, any sudden death was attributed to heart attack, but this was not necessarily the case. However, your father's early demise is evidence that there may be a hereditary component that puts your family at an increased cardiac risk. Your father's lifestyle would certainly NOT increase his survival chances or his overall cardiac health. So, if he had a genetic risk already, his life style most definitely would trigger serious issues.
You, knowing that you may be at risk, have done wonderful things. You eat right, exercise, don't smoke or drink heavily, your BP is good, cholesterol is fine, and you work to manage stress levels through exercise. 99% of the American population is in MUCH worse shape than you are at this point :-) (regardless of their genetic disposition).
I would say keep up the good work. Continue to visit with your cardiologist annually and try not to worry. If you do have some genetic risk, your lifestyle has most definitely decreased its effect at this point.