Are you male or female? You mentioned vaginal sex and swallowing semen but your icon shows male. How were you diagnosed as having genital herpes, and has your partner been tested as well?
Swallowing semen is not a risk for herpes. It's transmitted by skin-to-skin contact, along with heat/friction (the virus is actually "massaged" into the tissue).
Genital to oral HSV2 transmissions are actually very uncommon. And plenty of heterosexual couples are having oral sex; as a matter of fact, about 40% of all newly acquired genital herpes transmissions are due to HSV1 - closer to 60% in the college-aged crowd - these coming from oral sex.
I have read somewhere that HSV-2 infections are common in gay men, since unprotected oral sex is popular among the homosexual population. So you can get HSV-1 or HSV-2 in your throat or anywhere in your mouth indeed, likewise for your genitals.