as someone who also consumed a very large amount of caffeine per day (i work in the ER and swtich days to nights regularly so i need it) i feel your pain.
that said, my RE recommended that I gradually decrease the amount of caffeine prior to starting IVF.
i got down to 1-2 cups a day, then to 1-2 cups per week in the first trimester.
now that i am full term, i am back to a cup a day.
it's not going to cause any permanent damage.
the thought is that excessive caffeine may cause your uterus to be hypersensitive, which MAY cause implantation issues.
there is also thought that very large amounts of caffeine may affect fetal nerve system development, but there is NO research out there showing this at all.
since it is also important to stay hydrated in pregnancy, drinking that much of a diuretic makes it tough to consume enough water in a day as well, so another good reason to cut back.
Just dont smoke, drink, do drugs, be careful about medications you take thats it. Dont worry too much about foods you eat or drink. One or two cups of coffee/tea is fine. Just reduce the intake.
I think you will be fine if you stop or reduce now. I have never heard that caffeine causes long-term damage, just that the caffeine can cause problems if you drink it during the time you are ttc or during pregnancy. Once you stop drinking coffee the caffeine leaves your body pretty quickly.
It was not easy for me to quit drinking coffee. I cut back by first switching to half-caf, then reducing down to only 1 cup of half-caf, and then to decaf. After a month I did not even crave regular coffee anymore. Good luck!