Your symtoms don't sound like herpes. You don't describe in enough detail what it means to "not have sex" but you had to "kick him off me". If there was genital aposition, even without transmission, then some small risk exists. But probably to small to be a real risk.
But your doctors are wrong: 80% of herpes occurs without symptoms. If you're particularly concerned about that disease, then after 6-8 weeks have passed, have a herpes blood test to be sure you aren't infected with herpes simplex virus type 2. See numerous other strings on this forum about the proper herpes blood tests; some are reliable, others are not.
But I'm quite sure you didn't catch herpes from the episode you describe.
Good luck-- HHH, MD
GO TO A DOCTOR...If its scaring you so much why would you wait 2 weeks?