Eating too little can be as bad as eating too much. You might not be getting all the nutrients you need to produce the energy you need to burn off fat. It sounds like your metabolism of food is off. You might not be getting enough beneficial fats in your system. You're also not exercising enough -- if you're healthy and fit, why not exercise 5 days a week? Obviously, something is off. What kind of exercise do you do, how long do you do it for, how old are you, how fit are you, and what's your diet like?
Has your thyroid function been tested? Often thyroid function can be off, just enough to cause problems, but doctors don't seem to think of this. Too much is blamed on "age" or "menopause"... Your thyroid controls a good share of your metabolic function so it must be in good working order. Ask your doctor to test Free T4, Free T3, and TSH, along with thyroid antibodies, TPOab and TgAb. Hashimoto's is the # 1 cause of hypothyroidism in the developed world.