U will be fine. i agree with heather its when its taken for long periods of time. 3 days wont hurt... Dont worry best of luck.
I don't really think just 3 days will have much of an effect. The problem is when people use it to treat breast cancer, so over a longer period of time, and then get pregnant. But I'm no doctor, so definitely bring it up next time you see your RE and see what he/she says. Congrats on your pregnancy!