I have this quite often as well. I have anxiety and have had for a couple of decades now, and have had the same fears as you mentioned. I have also had quite a few attacks of BPPV over the years and it ***** as in addition to the movement induced spins, I have a general off balance feeling.
During other parts of the year, I get the pressure you are referring to and mine is usually due to allergies, sinus congestion, etc. This does make me feel off balanced quite a bit as well also. A lot of times, our perceived imbalance seems worse than the off balance actually is.
Have you visited an ENT at all to diagnose perhaps some cronic sinus problems?