It is possible your heart size is normal for you; or it could be the result of physical activities (an athlete's heart) that is non-pathological, but the size should reduce when vigorous activities cease. It is unlikely you have a heart problem at your age and no symptoms, however, with a family history it is generally recommended to periodically evaluate.
What caused the heart attacks of your father and grandfather? It's important to know that if it was due to plaque buildup, congestive heart failure, or an enlarged or too small of a heart.
My grandmother had an enlarged heart all her life; she ended up dying from something else. Doctors said that since she had an enlarged heart all her life, that it wasn't a concern to them.
What kenkeith said makes perfect sense as well.
Thank you for the great info and for relieving some of my worries! I guess I'll just go to the family doctor and just get a check-up to make sure all is well and mention my concerns. I'm not really sure what caused my grandfathers but my Dad's was caused by a combination of high blood pressure, over work, and high stress. Since I haven't been to the doctor in 5 years I don't know my blood pressure, but I too have a pretty high pressure job so I guess that is something I should be aware of.
It is important that you see a doctor about this. An enlarged heart can be Athletes Heart or it could be one of the forms of Cardiomyopathy. You should not be doing any real exercising before finding out the cause of the enlargement. Go see a doctor.