Exercise and cold symptoms

This may be kind of strange, but it seems like exercise causes cold and flu-like symptoms.  I'm not sure, and was wondering if this sounded familiar to anyone.

I'm a 36 year old woman, about 100 pounds overweight.  8 years ago I was diagnosed with high blood pressure and I take lisinopril.  Last year I was diagnosed with PCOS and endometriosis, and 2 months ago I had a blood test confirm I am now anemic.  I have not yet gotten treatment for the endometriosis.  I have had irregular and sometimes chronically heavy periods for several years, but I don't believe I was anemic until recently.   Also, in the last 2 years I've had frequent sinus infections which respond to antibiotics, and unexplained woozy/dizzy spells and fatigue, but a series of tests showed nothing seriously wrong.  Since I'm anemic, the doctor has assumed it's the cause of the fatigue, wooziness, and susceptibility to sinus infections.

As you can see, I don't have any especially serious illnesses.  But for the last 3 years I can't exercise more than a few days in a row without suddenly getting cold or flu symptoms like sore throat, ear aches, fatigue, muscle aches and occasionally a little bit of a temperature.  This is a change from 3+ years ago, when I could use my exercycle daily, or do low-impact boxing workouts daily, and had no problem.  

While I exercise, I feel fine, not even a headache.  I do about 20-30 minutes of walking and can do this a few days in a row without problem, but then eventually it's like I hit a wall and get sick.  Sometimes the cold doesn't hit me until several hours after exercising, or even the next day.  The colds never last long, either, usually 1 to 2 days.  For a long time I didn't exercise at all, and during those several months, I had no colds at all.  When I started back up with the exercise, sure enough, I got those colds again.

I'm wondering, is this just coincidence?  I wasn't anemic 3 years ago when I noticed this trend, so I don't think it's the anemia, but I just don't know what to think.

Any thoughts would be appreciated.