Were you tested for HIV, Hepatitis, hormone levels, diabetes?
I am reluctant to suggest this, but have you had an HIV test?
Change physicians. I have very little tolerance for idiots in my life and anyone who diagnoses your condition in the manner you describe (and I am not sure he did so, because all we have is hearsay) should not be treating human beings.
There are a dozen possibilities besides HIV, but you appear to immunocompromised. Mold is unlikely, but not an unreasonable possibility.
You have to find some who can give you a diagnosis.
Flee your current health care provider and find someone who takes you seriously.
Hey there how is the condition of your home and what i mean by this is , is there a wet basement anywhere mold can grow I knew a family with smiler problems there was mold in there floor, hope that helps