No that doesn't. The virus isn't just contained in that one spot. The only way it would have been safe is if he used a condom.
Also while performing oral on him I made sure his genitals had no direct contact with the scab. Does that matter?
There was a risk though it is a little more then a low risk I would say if he has symptoms to be seen within 24-48hrs for a lesion culture.
You should avoid oral sexual contact and kissing when you have a coldsore present. Wait for the skin fully heals to avoid the risk of transmission.
Also if your boyfriend hasn't been tested to know his own status he can have a type specific igg hsv test. He may already have HSV-1 himself and not even know it.