With your height and dad's the midparental height for your son and all of your sons is around the 50th percentile or about 5'10 inches, now 2 full standard deviations either way is 4 inches so this means that at the 10th percentile is 5'6 and that is the shortest genetic variation that is acceptable. Your child is below the 3rd and you are describing some developmental problems. I would stick with your peds endo, eventually once they establish a growth velocity, by seeing him in their office and plotting measurements they will make a decision about gh stim testing or other testing, like imaging or metabolic testing that can relate to develomental delays.
Yes, if you have history of pubertal delay this is still taken into account but at this time not something I am considering, I am more suspicious of his growth given he was on the chart as an infant and decelerated, I am sure your peds endo agrees but also knows that we have time to make up our minds about treatment path. Hang in there,stay with them and go at least every 4-6 months for remeasurements and assessments.
Glad to hear that thyroid and celiac were normal but if they were a little off if it were me I would be repeating it at another visit-but that's me