I don't get cold sores. Never have.
In that case you need to understand your HSV status. Obtain a blood test for IgG antibodies, type specific for HSV1 and HSV2.
The issue will be that a positive HSV1 outcome will not prove a genital infection. It would more likely indicate an oral infection. Do you get cold sores? Do either of your parents?
I had sex with a boyfriend. He was the only person I was sleeping with.
Yes, a visual diagnosis is only 75% accurate at best. It must be confirmed with a swab. A blood test can be of some assistance.
What was your sexual history in the two weeks leading into that first blister?