Do you have nausea and lack of appetite as well?
That is exceedingly small dose of Xanax, but Xanax is sort of notorious for being the worst of the benzo bunch on addiciton and withdrawal. If it is withdrawal for you, things will likely peek around day 3-4 from when you stopped the Xanax, and start calming down. That doesn't mean it'll be gone, but you'll finally start leaving the haze of the withdrawal and it won't quite be so bad. I'm on day seven of decreasing my xanax and I've only had very minimal reactions that I can mark to the withdrawal today. Day five, I actually started feeling happier again than I have in a while. But yes, Xanax withdrawal is known for giving dysphoria (depression), spiked anxiety, shivers, and a bit of a muddled mind among other reactions. Do keep in mind that anything that your body does in response to your panic attacks, it will appear in your withdrawal symptoms because the xanax withdrawal does increase anxiety. At the dosage you are at, the withdrawal should be less worrisome, but any concerns should be addressed to your doctor.
With benzos like Xanax, you can defintely feel withdrawal symptoms like these and 'rebound anxiety.' Why are you coming off it and are you doing this under the direction of your doctor? Quite often, I see so many people try to adjust dosage or come off of these medications on their own which can have horrible consequences and puts everyone through an experience they need not need to go through.