I was also on Adderall around the time I got pregnant. I've been on it off and on for about 15 years and was worried about stopping, but both my OB and psych have helped me and it hasn't been a problem. Once you're off, its easy to stay off and there are very little withdrawl symptoms. Good luck!
You need to consult with your OB asap. It can be dangerous coming off of something while pregnant - especially the further along you get adn you need do it all under the guidance of your doctor.