A hemotologist is a good idea for this.
Are all the white cells low, or just one type, such as neutrophils? That's a big clue.
Overall, a low count means cells are being destroyed too much, or produced too little (in the bone marrow).
Probably any doc would be untroubled, if you've had low-normal-ish neutrophils for many years without symptoms. That's especially true if most or all possibly dangerous causes have been ruled out. They would likely say that you are just that way and it's not a problem -- especially without frequent bacterial and/or fungal infections. Since it's only isolated neutropenia (i.e., not across all white cell lines), a marrow problem seems unlikely.
But the sudden drop seems concerning -- unless it's just from a virus. 'Reactive Lymphocytosis' can come from a virus, and sometimes neutrophils get killed as part of the scenario.
Hopefully you are having frequent CBCs to make sure your neut count doesn't keep plummeting. If your neuts rise back to 400 in a week or so then that's probably the end of their worry.
But you might justifiably want to purse further.
What did your blood smear say? Do you know if your lymphocyte expansion is in the B-cells or the T-cells?
Btw, if you get a fever now, call your doc's office right away so as to fend off any possible runaway infection, to get an immediate CBC and possibly antibiotics. Your doc probably told you that, I'd suppose.
Don't get alarmed by this but apparently there are some mutations (e.g., in the HAX1 gene) which can cause both congenital neutropenia and neurological problems.
You are not severe like that, but you might have something similar. I'd bring that up to any doc you might see, so that they can rule it out -- or say why it doesn't apply. The main point is that such an association can and does exist... sometimes.
The paragraph which begins with: "HAX1 mutations Homozygous mutations in HAX1, which encodes..."
You might be heterozygous, which could possibly be less severe.
"I do have a very hard time fighting any bug."
Then you certainly would want to pursue this until it gets hammered out. You shouldn't have to live that way. Ironically, things getting worse now opens the door for a possible diagnosis.
If it is the HAX1 mutation, there is probably no cure, per se. But you might get a drug such as G-CSF (granulocyte colony stimulating factor aka Neupogen), which is typically used for cancer patients whose neutrophils get dangerously low from chemo drugs.
Did you ever have a blood smear? (They look at blood cells with a microscope.)
I would print out that entire cited paragraph, along with the link, title and date, and show that on any visit with a hema. That is an 'authoritative source which any doc should take seriously, and then they can decide if it fit you. You might also email that to your internist to get any feedback.
Without a cure, they would look to 'manage' your condition.