I have very similar situation - all due to granny being very insensitive to my son's fear dogs. Her dogs are pretty wild and do not obey...They jump on everybody. She gets her feelings hurt when my son does not want to be around dogs (he loves cats though)...Last time we left my son with her, she let the dogs out and kept holding him down, letting him "face his fear". Of course he does not want to go there anymore.
When granny comes over, everything has to be on "her terms" - I do not recall that she ever was interest in my son showing her his toys, or playing in his room with HIM and HIS stuff. She is only interested in doing what SHE wants to do and gets out of sorts if he is not that excited. She actually got her feelings hurt when she saw my son playing with our cat, but not her dogs....She is also pretty impatient with him. He is pretty sweet to her when she comes over with his uncle though.....And he is better with her when we are not around......
How is your relationship with Granny? My hubby (it is his mother) has it pretty strained.....
Good luck - I know how frustrating it could be....
Is his Granny nice to him? does she play games with him or is she tough on him or he gets bored, there is a reason he say this, check it out and if you see it could be the way she looks after him perhaps talking to her is called for .to see if he needs more positive attention .