A difference of 3+ diopters (called "aniseikonia") will produce symptoms (e.g., headaches, dizziness, photophobia, spatial disorientation, double vision, nausea, impaired depth perception) in almost all people. Available research suggests that people cannot adapt over time. Possible treatments are contact lens wear or specially-designed isokonic glasses.
I have aneiskonia due to a damaged retina. I'd definitely recommend that you pursue one of the three remedies suggested by your doctor. (Laser correction might be safest.) In your place, there's no way I'd leave things as they currently are for you. You will face a lifetime of unpleasant symptoms.