Stress is a big factor in fertility,if you are stressed,you will struggle to release the egg on time,if at all.making your cycles possibly irregular.any medication your partner is taking can have an affect on you concieveing.sounds to me like a timing issue,do you chart or anything,this maybe a good start.
Anxiety med can cause the egg and the sperm not to fertilize correctly and stress is #1 cause for infertility! If your doc thinks that you didn't need surgery then I would go with the doctor. Usually if you hit a year with no baby and actively trying then it is time to see a fertility specialist. I am going through the same thing and we have been trying for 19 months. Best of luck!