First, you need a new doctor, if only because your concerns are being brushed off as insignificant. You deserve a complete answer to your questions.
You don't have low T, technically, but that's quite a drop - it usually drops 1% a year starting around age 30. Low T is under 300 ng/L, so you're close.
Other than your testicle symptoms, do you have any symptoms of low T? https://www.healthline.com/health/low-testosterone/warning-signs
Have you had a recent scan on your testicles? Are either still feeling hard? If they are, get that followed up on. I don't know that it has to do with the dropping T, but it seems worthwhile in examining. Can you get in with an endocrinologist? You deserve at least a consult.