Also, I only smoked from his pen twice for a short period of time.
I'm sorry in advance, because I hate rumors like this. Most of the time, they aren't true, and they are vicious.
I have no idea if they meant oral herpes or genital herpes, but I'll address both.
Genital herpes - no risk at all. None. The only way you'd be at risk from him if he had genital herpes is if you had some kind of sex with him.
Oral herpes - no risk. He had no visible sores, or you'd have mentioned that, and herpes dies quickly outside of it's human host. Sharing utensils and the like isn't really a risk either. The only way you'd get oral herpes from him, in reality, is if you kissed him.
However, if you want to vape, get your own. This is a great way to spread colds, the flu, etc.