You describe a partner at low risk of having HIV. The past "hard drug" use raises the stakes, but only a little in someone who hasn't shared injection equipment with other drug users. His sexual lifestyle is low risk; 5 heterosexual partners is a trivial number with respect to HIV risk in the US and other industrialized countries. Also, if his age is similar to yours, that's also a safety factor; not very many people in their 20s have HIV.
Beyond all that, the sexual exposure you describe did not risk transmission even if he has HIV. Nobody every caught HIV by hand-genital contact, regardless of cuts on the hands, nicks in cuticles, and so on. You didn't have unprotected vaginal or anal sex -- that's all that really matters.
So the bottom lines are that 1) you can expect your partner's HIV test to be negative and 2) you don't need to worry about catching HIV from that exposure, even if he unexpectedly turns out to be infected.
I hope this helps. Best wishes-- HHH, MD