Are you scrapping it off with your fingernail? It most likely it tartar (or what we call calculus). You should go to a dentist and get a thorough cleaning, most likely what it terms root planing or a "deep cleaning." The buildup of tartar is not healthy and needs to be professionally removed because this tartar is most likely building up below the gums as well. You won't be able to reach all of it and it'll be rather painful to do it without some anesthetic. If you allow that tartar to remain below the gum, it will cause bone loss of the jaw and eventually your teeth will loosen, not to mention it may get infected and cause an abscess.
Best to get it checked out and taken care of.
No, I haven't been to a dentist in a few years. I have greyish buildup between the two bottom teeth in the front. If I look at the back of my teeth, it looks like the gums have been decreasing. If it is just plaque, I'll peel it off, I just don't want to lose my teeth.
It may be the tooth or it may be the tartar/plaque that is buidling up in that area. When was the last time you saw a dentist and had a cleaning?
Any other symptoms beside being able to peel away solid pieces of your "tooth"?