Has your husband ever been tested to see if he has HSV 1 infection? If not, he should be because clearly this is a worry for you. Because the reality is that you can infect your husband at any time, whether you have an outbreak or not. You shed virus from your mouth intermittently, even when you have no symptoms.
If your husband tests negative, you both need to ask yourselves: is it worth giving up oral sex for the small risk of getting HSV 1 genitally? HSV 1 genitally recurs about .7 times per year on average, and is easily treated with antiviral medicine.
You need to get the test results first. You want a type specific IgG HSV 1 antibody test, no IgM testing, please. You can order this for your husband online, you don't need to see a health care provider for this, but be certain you order exactly what I suggested.
And no, if you did give it to him genitally through oral sex, you won't give it to yourself again because once you have the virus in one place, you will be extremely unlikely to get this in a new place on your body.