In general, maybe what your doctor really meant was not flu but a virus. The reason I said that is that influenza A and influenza B are usually associated with fever as a hallmark sign. And flu in and of itself is not dangerous. What becomes the issue is a secondary infection in compromised individuals like children and elderly. So, say, they get the flu and are sick. But then develop pneumonia and are that can then have grave consequences. Because in general, everyone gets viruses and the flu and the vast majority are just fine after.
I would stop look at your throat. You do indeed sound highly anxious. No symptoms means you are over worrying. Next visit to the doctor for a well check or if this is still there in say a month or two, then make an appointment and show it to your doctor. But would not be overly worried.
Viruses do get better on their own and you do comfort care like use Tylenol or ibuprofen if you ache or have a sore throat, drink tea, get plenty of fluids and rest, etc. If you begin to have other symptoms such as pain or fever, revisit your doctor.
Do you do anything about your anxiety or is it just at a higher than normal level for you right now?