I don't know where you live but I would suggest a complete speech evaluation. There are different parts of speech---- expressive (how he lets you know what he wants or tells a story in words), receptive (how he understands you) and articulation. I think the standard of 4 is that strangers understand 95% of what they say. While he may just be slow to speak, if you give him directions and he can't follow them---- it may be more than that. Sensory integration disorder is something for you to look into as well. Some kids have the symptom of sensory in that they need lots of "input" into their system so they will be rough with things, hit things, run into walls, etc. Good luck and I'd say that it is time to force someone to give you some answers.
Did you have his hearing checked? Hope you find out soon whats wrong.