Comment on your title - did you ever forget him at a place or were slow to return? This happened to our grandchild - her mother was very, very slow at picking her up one day from nursery school. To this day (10 years later), she is still worried about being abandoned. It was the "little" trauma which exacerbated her anxiety disorder. And, I am not saying this "caused" her disorder; just put it in "full bloom".
I guess you need to constantly reassure him and always be there "ahead" of time when you or someone else go to pick him up. But, your son is displaying "signs" of an anxiety disorder. Is there a family history of this disorder? Perhaps googling the term "childhood anxiety" or "anxiety behaviors" or similar phrases might shed more light on this topic. Please write if you have any questions. I wish you the best ..
What recent happenings? has he been seeing new Family or Baby sitters or friends, or could it be possible he has overheard anything to cause him to feel anxious,as he is worried about you being there could he have overheard any adult conversation, or any Scary TV or Video,on Familys splitting up.Give him a lot of positive attention talk to him try to find out what he has heard and correct any misconceptions he has.