No, herpes wouldn't cause this.
Herpes infects nerve groups. Genital herpes infects the sacral ganglia, and only affects the area that boxer shorts cover.
Oral herpes infects the trigeminal nerve. This affects the mouth, nerve and eyes. Vestibular neuritis affects the nerve of the inner ear called the vestibulocochlear nerve. They are separate things.
This says that different viruses can cause it, including the herpes family of viruses, including oral herpes, though I'm not sure how that would work (I'm no expert) - https://my.clevelandclinic.org/health/diseases/15227-vestibular-neuritis
Interestingly, there is some research into whether covid causes it - did you have covid?
https://pubmed.ncbi.nlm.nih.gov/33664181/
https://www.nationaldizzyandbalancecenter.com/post-covid-vestibular-and-neurological-symptoms/
It might be from the vaccine - https://pubmed.ncbi.nlm.nih.gov/34898387/
There are a lot of possibilities, obviously. Do you have oral herpes? Have you had covid? Any other recent viruses?