Small head size can be due to a variety of factors, including: familial (as you mentioned above), growth restriction during the pregnancy from maternal or placental conditions (for example, maternal hypertension or abnormally placed placenta), prenatal infections, and brain structural abnormalities. The risk for long term neurodevelopmental problems depends to some extent upon the underlying cause. Your pediatrician may be able to give you a better assessment of possible causes based on your pregnancy history and any evaluation that was performed on your son after he was born.
It is reassuring that your son's head is continuing to grow along his curve, and that his development has been within normal limits so far. Your pediatrician will continue to follow his development closely with you.