Genital warts can be in the groin area but his may be just common warts. Warts are a low risk strain on HPV, there are about 160 of these. There are amost 40 high risk strains that cause cell changes, like your abnormal pap shows. Low risk can not cause high risk symptoms. There is no way to know when and who got a strain of HPV, you both could have more than one strain of each type. HPV is very very common and over 80% of people will have HPV in their lifetime. I agree with Soscared, focus on health and not blame.
HPV may lie dormant for years and become activated when the body is weak, like when fighting the flu or something. His warts may be from sex partner he had before you met him. Seems that he has a different strain, as warts are usually from low risk HPV and abnormal PAP is from high risk HPV.
You each have two different strains. It is useless to blame each other. Could be yor entire fault. So you may as well focus on being good to each other and supportive. When you are both over these issues, you won't be able infect each other again. Cannot pass HPV back and forth as long as you are faithful to each other. I recommend stay together and monogamous.