If you're not comfortable, change. When I got pregnant, I switched my doctor because of location and long wait times. I felt I was being treated like cattle, not a person. Pregnancy is a big deal, you should be comfortable with and confident in the care you receive. Also, if you're definitely going to change, there is no harm in telling the truth about why. You might influence the way current and future patients are treated (then again, maybe not, but it wouldn't have an effect on you). Just say it in a neutral way, not an accusing way.
You don't have to give any reason to your doctor about switching. If you feel more comfortable somewhere else then what can he/she do.