What are the next steps for a low lymphocyte count?

[HeyDen]

Over the past year my lymphocyte count has been falling and is now well below the normal range. Here are the results from the last few tests: 09/2015: 1,260; 09/2016: 820; 03/2017: 620; and 09/2017: 500.  Aside from my platelets being slightly low (09/2015: 144K; 09/2016: 142K; 03/2017: 142K; and 09/2017: 133K) and my mean platelet volume being high (09/2015: 14.2; 09/2016: 14.5; 03/2017: n/a; and 09/2017: 14.2), the rest of the levels are within normal ranges. I saw a doctor for this in 06/2017, and he said it was basically wait and see since I didn't have any major symptoms. I scheduled another appointment with him, but it's a month away. Should I bypass him and see a specialist instead? Background info: 38 year old male; 5' 7", 140 lbs; exercise regularly; healthy diet; and generally in good shape. Thanks.