It is uncommon. Hsv2 orally is rare - hsv2 doesn't like the mouth, and when it does happen, it sheds a lot less, and recurs far less frequently than either oral hsv1 or genital hsv2.
Have you asked them specifically about their symptoms? Not knowing whether or not you were giving oral sex on a penis or vagina limits things a little, but there are all kinds of things that can cause "rough patches" - shaving bumps, dermatitis, eczema, allergic reactions, dry skin, etc.
You'd be a good partner if you let them know you noticed this, and in a non-accusatory way, suggest that they might need to get it checked out. Your health matters, but theirs does, too. You might get some info from it, too, that could ease your mind.