I agree with your doctor, you do probably have Candida. It does not take a lot to make a person with it feel lousy. Stick with the diet, take a good, pro-biotic supplement, and supplements that do not have yeast or soy in them. Also, if possible, go to the health food store and buy a good intestional de-tox formula, follow the directions on the box, you should start feeling better as the fungi are killed off, but, a word of warning, while it is getting killed, you will feel worse, it is a rebound action and perfectly normal when doing these types of cleansing.
God bless, hope you feel better soon.