It could be that while working your get 'keyed up' and overly stimulated in a positive way.
Being off, your body got the chance to 'wind down' and relax and your anxiety becomes more noticeable and impacts you harder. I'm not sure this makes any sense to you but I always found 3 day weekends really tough. It was just enough for me to 'wind down' and reach a 'normal' level. Getting back into it after that was always a LOT harder.
Do you get these feelings on any other set of days off? On those four days did you do anything different? Examples: caffeine, alcohol, excitement, anything at all that is different from when working.