Positional Vertigo perhaps? A doctor familiar with it can check you for it quickly with a simple test.
Iron deficiencies can be a culprit.
Inner ear issues from allergies can be as well.
I have what I call the wabbles, when I get out of a car and start walking. Yours is delayed so I think it is a bit different. I don't know what causes mine yet but I have lived with it all my life. People used to say I was clumsy. It has gotten progressively worse, though still not keeping me from driving, over the years. In fact, when I am sitting down driving, or working at home on my computer or sitting to eat a meal, I am fine. I have recently had my ears checked and had an MRI and MRA though one of those showed sinus issues. I think now, though my doctor doesn't agree, it is sinus related snf you mentioned allergies. I hope something I have shared helps.
Maybe grasping at straws here for ya - did you have any sort of emotional stuff going on when this first started? I don't know, just wondering if maybe this can be some anxiety...I have certainly had my share of anxiety and I know all too well how it can cause many physical symptoms - even though we don't know it. I don't really have anything else that I can think of that could cause that. I ask this b/c my mom was in an accident a long time ago and she has awful physical symptoms when driving more than a few miles from home...it has been hard on her having such anxiety. Hope you find some help. Good Luck.