Xanax is only a quick fix. That's if want an honest answer. Most doctor's answer to all anxiety problems. Because it works instantly. I was on it for a decade or more. For most of that time I knew it was doing nothing for me at all. Then it just stopped doing anything at all. In other words I relasped back into having bad anxiety and the xanax was doing anything to calm me down. Tolerance levels had long since been reached. But that didn't mean I could just stop taken it. My body still needed it. In my own opinion just come down slowly. Your body will let you know if you are doing it too fast or not. Trust me on that one. But maybe you do need something for your anxiety. Thus when you lowered the dose down in the morning you were feeling normal anxiety that the xanax had brought under control. If you know what I mean. If you anxiety is that bad I would talk with my doctor about a longer acting benzo. One that would not lose tolerance as quickly and won't wear off after a few hours. But by the sound of your post you don't want to be on medication at all? You seem to keep on lowering you doses down. We can imagine we are better because the medication is making us feel good and normal. That does not mean we can go without it. You ever stop to think it might be the medication making you feel good. Then you think that your anxiety is gone and you want off your medication. Just food for thought.
Xanax is literally a tranquilizer for anxiety. So as any tranquilizer does, it wares off after a few hours and then the lion (anxiety) is back on the prowl looking for its next kill (usually your sanity). Xanax is a benzodiazepine, and the body reacts to benzodiazepines almost like alcohol in terms of tolerance of a length of use. You should probably be on a long-term medication and then use Xanax sparingly when needed. I take 300mg of Luvox, and have .5mg Xanax for times when I get too anxious. The Luvox puts me at a much higher level of consistency and then the Xanax is there to pick me back up to that level if I fall into a slump during the day (hard to breath, racing thoughts, derealization, palpitations, etc.). However, I am not a doctor... But I definitely suggest you seeing a psychiatrist and telling them about the constant highs and lows. Had I done that for myself a few months ago, I would be dealing with a lot less anxiety trauma on top of all of the college and organizational responsibilities I have right now.
Also, routine and organization are key for you right now. Healthy dieting, exercise, a relatively consistent sleep schedule, and relaxation time will help you pull through this time in your life much quicker. But as I said, a constant medicine can save you.