Put HPV entirely out of your mind. You have no worries about HPV transmission, in either direction, by the kinds of sexual contact you describe. And of course neither HPV nor any other STD can be transmitted through shorts or other clothing.
I am unaware of any scientifically valid reports that the HPV types that cause genital infection can be transmitted by hand contact with another person's genitals. There are a few virgin women who have been found to have HPV, but the likelihood is that some weren't truthful about not having had sex; and others had had genital apposition, without penetration. If any of those infections resulted from contact with the hands of partners who had genital infection, it is very rare.
Good luck-- HHH, MD
Also, detecting HPV DNA on hands isn't the same thing as saying the virus can be transmitted by hand contact.
HHH, MD
HHH, MD
I cannot give you a precise numerical estimate of the risk. It's low enough not to worry about, but there are no guarantees.
HHH, MD