Is it possible he has a learning disability? Can he recite his ABC's and count to 20 or so, but not recognize them (eyesight/hearing problems) or is he not able to memorize? You need to find out the exact issue first - perhaps the teacher could help you here. Please do not not berate yourself for not "teaching" your son (after all, one mother can only do so much) - but could not one of his siblings help him more at home with his reading and writing? They could even read to him in the car on the way to the various events or as dinner is being prepared.
But, I agree with the above poster - if your child is going to repeat a grade, this is the time; not part-way through school when he has established a core group of friends. Since he is so young, it is quite possible that he will have more "friends" in the younger classes. Also, another consideration - if he is behind in his studies, it is possible that the children in his present class might exclude him from their group and "make fun" of him, especially as he gets older and further behind. Hope this helps ...
I think you should take the teacher's advice and hold him back. There are several boys in my son's kindergarten class who are not turning six until late spring or summer, and their mother's plan to have them repeat kindergarten just because they will be more mature and to let them get the information twice so they know it well. It's not a terrible thing... and better to repeat now and get a good grip on the material than let him flounder through a few grades and have to repeat later, when he and his friends are more socially aware. I think you would be doing him a favor allowing him to enter 1st grade with a good foundation & confidence. By this time in the kindergarten year he should not only recognize the letters & numbers but be writing them and know the sounds. In this generation, almost all kindergarteners are reading easy primers (1 syllable words) by spring.
Don't beat yourself up about not working with him at home before you sent him to school. But from here on in you need to take the few minutes a day to be sure he understands and does his homework and/or schoolwork papers sent home. If one of your older children is mature enough to help him (rather than doing it for him) maybe you can enlist their help on days when you are too busy.