I wouldn't be at all concerned about passing hsv2 to your little one. Not only is it rare orally it sheds very little and isn't very potent when it does.
Hsv2 does not like the oral cavity. Hsv2 oral infections are rare. I wouldn't worry about having the infection orally.
Have you ever been tested for hsv1? It is possible to have it and not know it. Many folks get it at a very young age and then don't get cold sores past the primary outbreak. Or have a few outbreaks as a child then it goes dormant.
We also had oral sex before we found out we had it. I developed a sore and after a few days it went away. 6 month later I developed a sore same spot only it took longer to heal so I went to dr and swab confirmed hsv2. I've been symptom free 3 years until last month when I had a pretty bad outbreak.