Yes he should actively consider terminating your use of the Zoloft and continuing your periodic use of Xanax.
He should do this under the supervision of your physician who will recommend that you taper off over a 3 week period. You do that by taking half of your dose each day for week then half a dose every other day for a week and then a quarter of the dose every other day for the third week.
I was on it for Panic Attacks and anxiety for about 6 months - I stopped cold turkey. I was a little shakey and kept getting "zaps" in my head - It only lasted about a week though.
However, everyone is different when it comes to meds - But that was a really crappy week.
-MAZ