You are probably past the worst of the withdrawal, but yes, there often is severe depression when a person stops adderall after years of use, particularly stopping cold turkey. There are a couple different issues; the acute withdrawal, which you are going to make if you hang in there just a bit longer; then the 'staying sober' part, and finally, whether you will be able to function on the job and at home without stimulants. That will depend on whether you truly have ADD, and the nature of your job responsibilities.
You may want to ask your psychiatrist to add bupropion (wellbutrin) to your medications in place of the adderall; it is not addictive, but it does lead to increased dopamine in the brain. We used it to treat ADD in the prisons when we were avoiding stimulants. It might help with your current depression and energy level as well, and can be taken along with your other psych meds.
After a month or two, your body will suddenly start to feel whole again-- it just takes time. I urge you to hang in there, as the hardest part is behind you.
I presently take Adderall XR and Wellbutrin at the same time. I have sever ADHD and have my whole life so I do not relate to the fatigue but Wellbutin has done wonders for me with my Adderall XR. I was very sceptical of taking Adderall because I am an addict but have never abused my ADHD meds as, like you, they calm me and do not give me the high they do others. I opted out of the regular Adderall specifically because of that reason because I work really long days and was afraid I might pop an extra pill here and there. I have found the XR when I take in the morning at a higher dose works through most of the day and then the Wellbutrin brings me down slower and lasts the remained of the day. Mind you I work 10 hour days ending at 11pm and am a single mom and get up no later than 7am. So my day is very long. It has worked wonders for my ADHD but not the massibe drop that I got from my days when I took Ritalin (similar I hear to Adderall.) I have depression and the Wellbutrin has helped with that but it has greatly helped with my compulsive shopping.