i had a deviated septem as well, and it was only after the x-rays that the docs realized just how bad it was. maybe you could get just your adnoids out if you don't feel like such invasive surgery.
In the past, my nose was always clear. When I get colds, it really bothers me that I cannot breathe through my nose, as I am not a mouth breather. Especially at night, I hate to breathe through my mouth. For the last two years, my nose gets stuffy on one side when I sleep. To alleviate that, I have to switch sides, which is really annoying and has compromised my quality of sleep. When I get a cold, they usually end up as wicked sinus infections after a few weeks. It is my opinion, that we may or may not have 'allergies' but our sinuses are reacting to an irritant of some sort. My ENT once gave me an injection of some sort of steroid, along with antibiotics (for an infection), and I have to say that those two things, combined, especially the steriod injection, really cleared up my nasal stuffiness for about 6 months. Try finding a good ENT physician, who can evaluate you. Good luck...