You are not unreasonable at all. The reasonable thing to do is test for hsv in this situation. Is he sure his ex had hsv2? Genital hsv1 is very common as well
I know this post is from 5 years ago, but MY resolution would have been to say "You know what, how about we BOTH go ahead and get tested for EVERYthing under the sun together, that way neither of us have to live in fear." All the best!
Non medical answer here: ANYBODY that actually gets angrybwith you for giving a sh*t about your own health CAN NOT POSSIBLY EVER BE "The one" I would have spit in his face and been out the door!!!