N.B.: Don''t have sex if you yourself are having symptoms, even while on the medication. You probably already knew this, though.
His symptoms are nonspecific and don't mean they are related to your herpes. Valtrex and condoms together are highly protective for him.
Has he been tested yet for herpes? He has a 25% chance of already having herpes himself. Until he gets tested, I wouldn't worry about it since you are using the medication and condoms. Until he has obvious genital symptoms anyway, there is no way to know if he contracted it. The risk to him is extremely low, given the precautions you are taking. Not every physical symptom somebody complains about is due to herpes.