Was it with a condom? Are you latex sensitive or have allergy to latex?
The vagina swelling up after sex could be that you aren't lubricated enough before penetration. Maybe you and your boyfriend should try a store bought lubricant for the occassion. They are actually fun to use and can enheighten even the best sex.
When you went to the doctor did they just do swab tests? There are blood tests that can determine whether you are positive for HSV 2 (herpes simplex 2). Your boyfriend getting a blister on his mouth after giving you oral sex could be pure coincedence. Having a cold sore doesn't necessarily mean that he got it right after giving you pleasure. And, he could have HSV 1 which isn't considered the type that is sexually related. Many people get cold sores, even those not sexually active. My suggestion would be to go back to the doc, tell them about your boyfriend's blister and ask for a blood test specifically for herpes. The test will only tell the doc if antibodies are present--meaning if you've had it before (unbeknownst to you) or if you have been exposed to it.
Best of luck
PS Also, in case you don't know-cold sores are contagious. So refrain from kissing, sharing utensils and such until his blister is completely gone. From what I've heard, studies suggest that you you can't get/give the virus from mouth to genitals, but why chance it?