Wow your son sounds like me at his age! From the very start i was a quiet kid that had alot of friends but would also feel more comfortable around girls. Additionally i'd prefer to also spend time in my own world too. I think your son is over sensitive causing him to feel a high level of anxiety. I had huge problems adjusting to every school year, then by the time i'd feel comfortable, it'd be destroyed by the smallest thing such as change in routine. to put it simple, i never felt consistency with how i was feeling one minute i'd feel great and on top of the world, the next someone would say boo to me and i'd go to ground, and i feel this may be what your son is dealing with - some anxiety and lacking a little bit of confidence.
Build him up where possible and continue to build and develop his confidence like your doing. I notice you said he isn't very sporty, i'd suggest to enrol him in any extra curricular activity of his interest, this is a great way to build confidence and for him to feel that he is good at something. I was never sporty due to a childhood hip problem but wish i had've been involved in sport in some way looking back now (in my mid 20's)
Sorry i never normally go about myself, i just see so much of me in your son, all the best and hope everything works out! :)
Your son is not anxious - he is suffering from anxiety. The difference is in "function" and your son is unable to totally "function" in perceived, unsafe environments (i.e. school) - inability to eat is one example. As it appears that your son is able to socialize and learn at school, it does seem as though his anxiety is not severe. By the way, not putting any pressure on him is a wise method to handling anxiety.
He will not know "why" he feels this way because anxiety is usually an inherited trait, not caused by life experience. Your family's "balance" or your "reassurance" will make your son feel safer but it will not fix or cure the problem. I might suggest you google the phrase "anxiety and children" or "anxiety disorders in children" or similar words/phrases to better help understand this issue. There are many downloads on the internet as well as many excellent books to purchase on-line, in your bookstore or borrow from your public library. There should be some advice in some of these items which should be of help.
If you still feel at a loss, then contact your family doctor for advice. If he/she is unable to help you, then ask for a referral to a specialist with experience in anxiety disorders. Frankly, I doubt if you will need to go this route unless his anxiety esculates (which could happen in the later years of middle/high school). Three things I might suggest you ask his teacher - is your son able to use the washroom at school, is he able to interact with both adults and peers at school, and is he able to "ask" for help (whether academically or socially or safety-wise). Please write if you have any additional questions. I wish you the best ....