There are several different types of atrial septal defects (ASDs). The most common form is a secundum type, where the hole is located in the middle of the atrial septum. After repair of this type of ASD, there is no reason to limit physical activity. Another type is a sinus venosus ASD, where the hole is located in the wall near the sinus node (the heart's natural pacemaker). Repair of these defects is also usually uncomplicated, but some patients can develop late issues with their cardiac rhythm. The last type to consider is called a primum ASD. This type of defect is associated with an abnormal mitral valve that has the potential for leakage through a "cleft" in the valve. Such patients are recommended to refrain from high degrees of isometric activity that can make their valve leak even more. Rugby would certainly be a sport with a high isometric component. So you need to clarify what type of ASD that you had repaired. If is was not a primum ASD, and your cardiac rhythm is normal, then it would be unlikely that any restriction on physical activity is needed.