I think you are correct when you say the children at the other house have an influence could be some teasing some upsetting behavior,it would be a good idea to look into it further, he is upset for his sake its good to check it out.Older children do sometimes like to aggrevate a smaller child, especially if any jealousy could be involved .Good Luck
He might not like the transition from you to his mother. It can be very confusing for a little kid. Have you asked him what is wrong or what he wants, when he is crying? If so, what does he say? Even a one-year-old can give you a hint.