Please post the result of most current lab report, along with the reference ranges, provided on your lab report, so we can see your actual thyroid hormone status.
Some people have reaction to fillers/binders/dyes in the medication, so that's a possibility. Since Levothyroxine is dosed in mcg, I'm assuming you mean you're on 50 mcg, rather than 50 mg? 50 mcg tablets should be white, with no dye, so that leaves a possible allergy/intolerance to fillers/binders. Often, the best/easiest thing to do is simply ask your pharmacy to change manufacturers, since all manufacturers use different fillers/binders/dyes in their pills, unless you want to go to the trouble of finding out the inactive ingredients in the pill you're taking. Another thing you can do is ask your doctor to prescribe a brand name medication, such as Synthroid, Levoxyl or Tirosint.
Most people seem to have the best luck with Levoxyl or Tirosint... Tirosint is a gelcap that's considered to be hypoallergenic, as it only has 4 ingredients, which are the levothyroxine, water, glycerine and gelatin. The one drawback with Tirosint is that many insurances do not cover it and, from my personal experience, it's very expensive.
It takes 4-6 weeks for the levothyroxine to reach full potential in your blood when you begin taking it, so it takes a while for it get out of your system, as well; being off it for just 2 days won't have much effect.