Welcome to the STD forum. I'll try to help. The main take-home message is good news: most likely you won't have any further trouble with your genital herpes and probably are not at high risk for transmission to your sex partner(s).
Herpes is the most common cause of genital ulcers, so combining that history with your positive blood test is strong evidence that you indeed had an initial genital HSV-1 infection 4 years ago. However, half the population has positive HSV-1 blood tests, usually going back to HSV-1 infections caught during childhood; and not all genital ulcers are due to herpes. If you ever have a new genital outbreak, the lesions could be tested directly for HSV-1 -- but the odds are good that will never happen.
Genital herpes due to HSV-1 has important difference compared to HSV-2. Whereas almost all persons with initial symptoms due to HSV-2 go on to have fairly frequent recurrent outbreaks (average 4-6 times per year), among those with HSV-1, 40% have no additional outbreaks at all and most of the rest have 1-2 outbreaks in the next 2-3 years -- that's all. In addition, asymptomatic shedding of the virus is much less common, so there is little risk of genital-to-genital transmission to sex partners. (In my 30+ years in the STD business, to my knowledge I have never seen a case of genital HSV-1 that was not acquired by oral sex.)
Most likely your occasional oral symptoms are not herpes. Most likely your infection is genital only. However, this also is somewhat uncertain.
Since you have hasd no recurrent genital outbreaks in the past 4 years, most likely it will never happen. If it does, outbreaks probably will be infrequent. And most likely there is little risk of transmission to partners. However, nobody can say the risk is zero, and you should plan to tell your future partners about your infection. But you can do that in a way that reassures them that the transmission risk is low. (In addition, you can expect that at least half your potential partners will already have positive blood tests for HSV-1. Those persons are immune to catching it again, so no transmission risk at all.)
For more detailed discussions of genital HSV-1, use the search link; here is a recent thread you can start with:
http://www.medhelp.org/posts/STDs/Recently-diagnosed-with-Genital-Herpes-HSV1/show/969931
Finally, for detailed, ongoing discussion about living with herpes and what (and how) to say to partners, see the herpes community forum. The moderators there are experienced herpes counselors.
I hope this helps. Best wishes-- HHH, MD