Swollen optic nerve one eye

Approximately 3 years ago I was experiencing some issues with my sight. I went to an optomotrist for a routine eye exam. While my eyes were dilated and while she was looking in my left eye she asked if I was sick. I asked what kind of sickness and she mentioned MS or diabetes or anything like that. My reply was no. She informed me that my optic nerve appeared swollen. She then referred me to a opthamologist in my area. He was baffled but said he was thinking it was drusen but couldn't tell for sure. He ordered an MRI on my brain and also had me tested for lyme.
No lesions came back on my brain and the Lyme test came back fine. He then decided to refer me to an optoneurologist in Boston. I went to that appointment and he as well thought drusen was the issue. He sent me for an eye ultrasound and it was determined that I had absolutely no drusen in that eye and that I had swelling in my optic nerve. I had brought a copy of my MRI. He brought me in his office to review it. He acknowledged that he didn't see anything on my brain MRI. He dismissed me and had me come back 6 months later. He never diagnosed anything and basically left it that it must just be a defect in my eye. He told me to get in touch if I ever have any more problems. It's been about 1.5 years since my visit and I notice my vision in that eye is getting slightly worse. I have absolutely no pain and never have. I do have issues with my back and neck especially with the muscles at the top of my back. I've always attributed this to my desk job.

Does anybody have any thoughts or suggestions on how to proceed with getting checked out if you even think it's necessary. I've heard that you can have lesions on your spine even though a brain scan is clean. I'd appreciate any insight. By the way I am a 37 year old male.