There is no risk here.
Herpes isn't spread by hands, and it doesn't spread by touching saliva and then touching another place. Herpes isn't spread by saliva, and a scenario with multiple touches and time isn't an efficient way of spreading anything. It allows time for the pathogen to degrade enough or die and not be able to transmit.
Really, your fine. :)