Your panic/anxiety is quite normal. Cognitive behavioral therapy could help you understand the reason why you feel what you do and how to change your thought process when you feel panic/anxiety when your routine changes. I recommend purchasing (or borrowing from local library) a work book on Generalized Anxiety/Panic disorder to work through your history and past experiences, and to give you something to work on for the future. there are a lot of people in the world who experience the same feelings you do. The best things you can do for yourself is learn about the disorder and, since you already know the triggers, learn how to control the anxiety/panic attacks. There are a lot of naturopathic supplements you can take that have no after effects.
Good Luck
I would guess the question is why are you anxious when you go on vacation? Is it just a sense of anxiety, a fear of travel, or a fear of people? This is a good topic to bring up to your psychologist or therapist. You may also have an anxiety disorder along with bipolar. I have generalized anxiety disorder, which means I just get anxiety for no apparent reason about everything. But, there are lots of types of anxiety and if it is negatively effecting you, then you should talk to a professional.
Also, try the relaxation tapes and other techniques like breathing exercises or meditation. Sudden changes in the daily routine can be a bother. For me, once something is changed it is hard for me to stick to it. For example, my medication was changed so I have to take my pills 2x a day instead of 1x a day. Now I am having trouble forgetting to take them. I even got that special pill box for 2x a day 7 days a week. I still forget. I even forget to put them into it. It is easier to remember my morning dose because I already have a routine for morning, but my evening dose it the trouble since every day is different.
So, don't feel bad, you're not alone.
try using a planner like a little book that you can use to stay organized as for the panic/anxiety try bringing relaxation tapes or cd's to help keep yourself calm :)