Unnoticed Sinus Infection?

Almost every morning when I blow my nose after waking up, I blow out a few bits that are yellow or green in color. These bits usually seem to originate deep in my left sinus (below my eye area & behind my cheekbone). Otherwise, everything is always clear. I've always heard that the yellow & green colors were a sign of a sinus infection, yet I had a CT Scan & saw an ENT about a month ago and he could see nothing wrong.

I'm 45 & I've had really bad allergy & sinus problems my entire life (except when I was in my 30s when they were much better). During the last 5 years or so, however, I've been treated for about 2 sinus infections/year. My allergies have really flared up these last 3 years. My nose is constantly stuffy or runny. For my nose I use Veramyst (once daily) & Ipatropium Bromide (every 6 hrs). For my asthma/allergies I take Singulair & Loratadine daily. (I've tried many nasal sprays and allergy meds, and these are the ones that work best for me.)