The description of events as told by this boy demand some intervention.
One option is to request review by the court of the custody arrangement. In many jurusdictions, one mechanism for this is to have a guardian ad litem appointed by the court. The obligation of the guardian ad litem is to issue a recommendation about how to proceed.
Another option, which of course will promote upset with the father, is to notify the local child protection agency about the neglectful behavior. It is obviously very risky for children so young to be left to their own devices while the father sleeps or, even worse, while he goes out and leaves them alone.
If there is a friend or relative who has the confidence of the father, perhaps that person can be enlisted to broach the concerns with the father.
wow, I have an almost duplicate situation EXCEPT. my daughter is the only child, and I have sole custody, but every time she cries about going to her fathers etc.. I called child protective and stated, while I did not want to keep my child from her father, I felt there was more going on up there, perhaps emotional abuse as to some of the things my daughter told me, similar to yours, increased anger, always yelling at her, spending all his time with his friends etc.. they told me, Unless there is some sort of proof, such as bruises, or the child comes directly out and states something that happened that could potentionally put them in a dangerous situation, such as leaving them alone in the house, there was not a damn thing I could do, This is new york state by the way, I say thats BS.. wait till something happens then you will do something, MY ADVISE to you, take him back to court,. whats the worst that could happen, your grandkids, kids etc. are your sole priorty, or perhaps have your daughter stand up to him, tell him this is what the kids are saying and things need to change or I am taking you back to court.. keep everything thru the courts,. do not do anything on your own, at least that way you have proof.. keep those kids best interest in mind, it only takes one person to screw up their little minds,. I told my ex bottom line, you love your daughter and she loves you but if things dont change on your end, I will make them change on mine. if this man truly loves his kids he will take someones advise, and get a grip before he loses. good luck and keep me posted.