You are correct. Even without a condom the risk of getting HIV or, for that matter, other STDs is quite low. With a condom it is, as you suggest, zero. You have nothing to worry about. EWH
I am more worrried about my stupid behavior and the unloyalty guiltyness to my wife and I have to deal it. Anyway, my recommendation is: "Before you leave your home, put a condom in any shirt or pants pockets because in the night all the cats are blacks".
Thank you,
Hector1967