I'm happy to try to help. Your question reflects the single most important STD risk for sexually active women. Despite all the discussion in recent years about herpes, HPV, and HIV, and the dominance of questions on this forum on those topics, from a real risk standpoint and the perspective of public health impact, chlamydia remains the Big One. It is so precisely because it is a major cause of infertility due to tubal damage. (Almost none of the many popular media stories about in vitro fertilization mention the fact that past STD, especially chlamydia, probably is the single most important reason for IVF.)
So your concerns are valid. However, your risk still is pretty low. With consistently negative tests, it is unlikely you were infected. But you are correct, it certainly would be possible to acquire chlamydia and spontaneously resolve it over several months, between annual exams. There simply are no data to calculate the numerical odds. However, the majority of chlamydial infections do not result in signficant tubal damage or impaired fertility. Taken together, these facts suggest you are at low risk for that outcome.
The standard advice is to not medically evaluate a couple for infertility until they fail to conceive after a year of attempting to do so. However, in some circumstances, an ObG specializing in infertility might recommend diagnostic tests without that waiting period. But the work-up is expensive (and likely not covered by insurance until infertility is documented), and some are painful or might carry their own risk of adverse effects. This isn't my area of expertise, though.
Bottom line: Most likely you don't have anything to worry about. But if you remain concerned, seek out an infertility specialist and follow his/her advice.
Best wishes-- HHH, MD
I will add that the "questionable" man and I had sex for about eight months. If rumors are correct, he was not faithful.