The hsv1 IgG misses about 30% of hsv1 infections. If you're sure what you've had are cold sores, you can assume you have hsv1, and it missed your infection.
You can have oral hsv2, but it's rare, and when it happens, it typically doesn't recur.
The only way to know for sure is to get an oral sore cultured and typed to see what it is. A blood test can't tell you where you have hsv1 or hsv2.
Typically, if you have hsv1 already, it can make hsv2 milder, so you don't have symptoms, or have really mild symptoms you wouldn't likely attribute to herpes. A 4.08 isn't low enough to need a confirmation test, but if you want to confirm if you have hsv1, you can get a Western Blot test, and at the same time, do a WB hsv2 test.
Your doctor can order this test, or you can go through Terri Warren at the Westover Heights Clinic, but you'll have to self-pay. https://westoverheights.com/getting-a-herpes-western-blot/
There's no right or wrong decision. You have to do what you feel comfortable with. If you don't get oral sores frequently enough that you can get an answer soon, a WB might be a good idea for you.