You had absolutely zero risk for HIV. The ONLY sexual risk for HIV is having unprotected, penetrative anal or vaginal sex with someone of unknown status.
How scared should I be? - not at all.
When should I test? - never. You had zero risk for HIV. There's no reason to test for a disease that you cannot possibly have.
How long must I refrain from unprotected sexual activity with my partner. - you can resume sexual activity at any time since you do NOT have HIV.