This is a tough spot for both the doctor and the patient. If your discomfort interferes with the majority of your life, then by all means, consult a surgeon about removing the gallbladder. You can get so many different opinions on this subject from so many different docs, and it becomes very confusing for the patient to make an intelligent decision.
You know your body. You know what activities make you who you are. When those areas of your life are jeopardized, it's time for more drastic measures.
I had acalcuous gb disease (no stones/sludge) and it didn't show up on any tests. Even the ejection fraction on the HIDA scan wasn't low enough to really raise any eyebrows. But, I'm the one who must live in this body, and I knew this body was sick. I was a hospital inpatient while all these tests were being performed, and about the time the gastro doc told me he had to discharge me because all my tests were negative, I sank down in the bed just like a child, and refused to leave until he dragged in a surgeon for me to talk with.
So, surgeon comes in. We talk about my symptoms, test results, etc., and he agrees to a diagnostic lap, but cautions me that he will probably find nothing wrong. Fine. I'll take the chance.
Guess what? My gb was so nasty and infected it was ready to rupture, and so many adhesions (scar tissue) had formed in the area that my small intestines were blocked in several places. Tell me I didn't feel vindicated?!
This same surgeon got blindsided by me again a year later with similar symptoms. Again, he agreed to have a look inside but didn't expect to find anything wrong. Any wild guesses? Endometriosis all over my lower pelvis, and an appendix ready to blow. After that, my surgeon knows that when I say it's time to go in my belly, it's the God's honest truth.
Bjut I digress. My point is this: too many doctors rely solely on the solid facts they can glean from tests. The tests are not perfect, and can only tell a part of the story. If you've been through everything they can offer testwise, and you continue to worsen, consult a general surgeon. It's certainly no guarantee of having surgery - the worst they can do is say NO, but maybe you get a reasonable answer about Why.
I would get a second opinion see what they say. If you feel sick get it removed . You can live without a gallbladder. If it doesn't bother you wait and see. I think you are going to be fine. Get a second opion if your not comfortable what the doctor said.