Long term reactions to the flu mist?

I'm not sure if this is all related, but for the past 2 months, approximately the same time that he received the flumist vaccine, he has had random fevers.  He will feel  fine all day and out of the blue will feel warm, usually with a fever around 101, this is accompanied by body aches, headache (sinus pain/pressure). Usually the fever is brought down by motrin and he sleeps off the pain. He wakes up  feeling fine after a nap.

His PCM is saying this is all allergies and sent him for a TB test (negative), CT scan and has run a bunch of blood work (waiting for results on the blood work). He is currently on Amoxicillin (day 5 of 7). After another fever last night the Dr on  call, sent us to the ER. Of course by the time we were seen the fever was gone and after  looking at his CT scan and seeing it was clear, the ER Dr said that it was no way sinus' or allergies, and suggested seeing an infectious disease expert.

Is the PCM n the right track, or the ER Dr? Trying to figure out where to direct my husband so that when he  goes into the military clinic tomorrow and they are trying to get rid of him, he  knows what he needs to get the correct care and not the run around.