You aren't a walking biohazzard who is contagious 24/7/365 for starters!!
Normally the skin on the hands/fingers is too thick for the virus to penetrate even if you were actively shedding the virus at the time of this encounter. If your partner bites their cuticles, had dry, cracked skin or had open cuts on his hands/fingers then there is some risk but again, you'd have to be actively shedding the virus which odds are wasn't happening.
In general, manual stimulation is too low of a risk to be worried about.
grace
no, he can't get it from fingering. if you have genital herpes, then he can't get it from oral sex, but if you have oral herpes and were broke out around your mouth/lips, he could get oral herpes on his genitals.