Warm soap and water will take care of this. Sperm does not live long outside of the body anyway so you actually would be leaving dead sperm. Do thoroughly wash your hands though.
So does washing your hand wash ALL the semen off?
Use normal soap and water (warm or cold) to wash your hands thoroughly.
You will know when your hands are thoroughly cleaned when you do not feel any "slime". Apparently semen contains Vitamin E which is good for the skin.
Normal washing is fine, so don't stress yourself or get paranoid over this. You do not need to keep washing your hands all day. It is good hygiene to wash your hands after using the toilet and before eating food. You won't get any infections from your semen.