I totally agree with you about Men's Hygiene , i think it's what you are used to doing , Myself i wash hand's after every visit but Shower Daily Please don't worry about it , Take care
If urine is what you are worried about, it is not a physical necessity to wash it off every time you pee (and in any case, be sure not to get soap in your urethra because it could sting). Urine alone is sterile.
But washing your hands every time makes sense. The skin in the nether regions gets sweaty and likely has some bacterial buildup that you'd probably prefer not have on your hands all day.
(If you are going to worry about skin bacteria, though, your hands themselves are probably the most bacteria-laden part of this equation.)
Don't worry, and don't go to extremes. Just figure out a way that you feel clean and go with it.