PS and when he has a Tantrum ask him why, maybe hes just expressing something .
Perhaps when he was younger he was always with his immediate Family and Gran and Granpa, he feels safest with the Familiar,and perhaps as he is okay once he is alone ,has he sensed some former anxiety from you ,just keep doing what you are, introduce him to new people and outings and remember as he is fine when alone he'll be okay. He will get into the Habit of you being away sometimes, dont make a big deal of leaving just say Bye see you soon and Go.Hes pretty normal,