It depends on what the what the heart rhythm is. What you describe sounds like you have PVCs and PACs, but I am not sure. Just because you have PACs, does not mean you are going to have atrial fibrillation. If you have atrial fibrillation, this is not the end of the world. Many people live with atrial fibrillation. Beta blockers and calcium channel blockers do not prevent atrial fibrillation. If your heart rate is controlled and you have PVCs/PACs and you have a structurally normal heart, medications aren't always necessary. It is unfortunate that you have problems with beta blockers and diltiazem because they can be helpful medications in the right situations. If the medications are making things worse and there is no definite indication for them, it might be worth trying to live with the extra beats.
As always, you need to discuss any medication changes with your doctor. It is tough for me to say anything concrete with knowing all the details.
I hope this helps.