Have you had any problems with anxiety besides this? Obviously it is still bothering you five years later. Sometimes, we just feel 'different' from time to time. I have definitely had things happen to me that have caused much worry over the years. Sometimes we cannot explain it, but in my opinion, every single person experiences this from time to time. Anxiety can really take over ones life, if it is not confronted in my opinion. Have you ever talked about your concerns with anybody before?
Try checking out derealization and see if that comes to close to matching what you are feeling. It is the first thing that comes to my mind. Not saying it is it. It just comes with anxiety. One of the symptoms. But it can be frightening. When you read all about it you will know if it matches what you feel or not. Just somewhere to start. As for how to stop it. I found it just stops by itself, if it is derealization. No magic cure. You could use distraction. MP3 player and the likes when out. Just to keep the mind focused. Check it out and let us know. Then maybe we can talk more and others will be able to share the exact same experiences with you.