PVCs in the recovery phase of exercise correlate to an increased risk of cardiovascular events. Skipped beats can be either premature atrial beats (pac's) or premature ventricular contractions (pvc's). The only way to know is see them on a monitor. PVCs during the recovery phase are not a modifiable risk factor for heart disease -- there is nothing you can do to control it, unlike smoking, cholesterol, hypertension and diabetes.
I would talk to your doctor about it. Let them decide if it needs to be worked up further. It sounds like you are otherwise pretty healthy from your description.