Although arrythmia's surely can't be good for your heart, the harm may not even manifest itself over a normal life span. That depends on your current age, and heart health, or course.
PVC's and SVT's can cause much more harm to your body by way of panic disorder. All one has to do is read these forums a few days, and they can't deny that arrhythmias and panic disorder are closely related. Either the arrhythmias cause panic attacks, or the arrhythmia and the panic attacks have the same origin. I think a lot more work needs to be done on this, because panic attacks are not trivial. They are not likely mentally triggered, but most likely hard wired into the body. Events occurring in your body trigger them, and a physical feedback loop that's not well understood causes them to spike out of control. They can adversely affect nearly system in the body. They can affect your quality of life far more than the arrhythmia itself, so get them treated.