I think you can have them removed. What is so odd.....I have one also, but I almost never get in the sun and have always worn sun glasses.
Chronic sun and wind exosure can cause damage to the conjuctiva. These damaged, yellow areas are called pinguecula and to care for them you should use wraparound sunglasses, avoid lots of wind and sun, use artificial tears like Genteal or Thera-tears frequently. They will not go away, unfortunately.
MJK MD