Frightened child

Ihave a Three and a half yr old boy, his mother and I split just after he was born.I have had weekend access from then on,school hols& holidays,and any other times I can.Within the last 3 weeks he has decided he doesn't want to leave his mother tocomewithme when Ipick him up fromher house. He actually hides and cowers in the cornor.
I find this hugley distressing, as he has never been like this before. We do not use any physical punnishment,though as he is cowering, it looks to me like he is frightened.
What is going on? What can I do about it?  

[margypops]

Did you ask his Mom if she had any ideas what has started this behavior, has he any new siblings at your house that he doiesnt get along with or other family members?Have you asked him why is he afriad, when he was cowering what did his Mom say tto him, did she encourage him to come with you, Many unanswered questions here,