This sounds like a concern to me.
It's normal for kids his age to have trouble at the moment of transition, but that's not what he's doing here. He's dreading going home throughout the visit.
What perks my ears up in your post is it sounds like he spends a GREAT deal of time at other relatives homes, instead of being in his own bed every night.
Are his parents both his biological parents, or is this a stepdad?
How do they behave when they are together how does he react to his , mom is he afraid , I think that you should find out why he is saying this, most young children dont ,quite the opposite , who else has he in his life any older siblings cousins ? I agree with mammo dont let it go find out why ...good luck
His words and actions would concern me as well. I have a 3 year old grandson and cannot imagine him using words like this, and he is always happy to see mommy and daddy after being with us for a few days. I do spoil him, but he is still happy to go home. I saw this with my friend's grandson at 2-3 years of age, and it was due to a bad home life, and my friend got custody of him at 3 and is raising him. I'm not saying this is what is happening in your case, but it warrants looking into. Ask him why he doesn't want to go to mommy's house, why is mommy not your friend, and who is your friend? Through little bits and pieces you may be able to figure out what is happening. If he just didn't want to leave you once you got him home, it wouldn't be a big deal. But for him to speak about his mom in the way he does is concerning. Do you spend much time in the home to see how he and mommy interact? Maybe do more of this and see if you can see any reason for his feelings. I have spoiled my 3 grandkids a lot, but they never complained about going home or talked badly about their moms. I wish you all the best and hope you can get to the bottom of this. Take care.