this sounds so painful for your little boy. At his age, it can be very difficult to explain why adults do not do what they should. I applaud your attempts to keep the relationship going with his father. Many divorced parents are in a lot of pain, and do not really understand how important it is for a child that a divorce not cost him a parent. Unfortunately, your situation of having a parent disengage from a child after a divorce is also all too common.
Many parents who lose custody have a hard time tolerating both the end of the marriage and the loss of the child. It can be hard for a person who is not particularly emotionally strong to maintain a relationship with their child once it becomes hard to do. Some people are so eager to avoid all of the disappointment and bad memories of the divorce that they avoid their own child. Sometimes parents are so angry with each other that they use the child as a pawn to enact power struggles (like taking advantage of the other parent or provoking you by being late). Given your description, it is likely that your son's father is thinking much more about himself than he is about his little boy.
The best case scenario would be that you could somehow convince this father to engage with his son on a regular basis. You might want to ask if he would be willing to meet with a neutral party such as a psychologist, counselor or mediator who helps divorced parents work together (often called parent coordinators). If you offer to pay for it, he probably will be willing to give it a try. If you can keep the focus on your son's needs, it can be a productive problem solving meeting.
The basic fact is that the father's behavior is potentially very damaging to this little boy. Children your son's age tend to believe that if an adult lets them down, it is because they did something wrong. They are so afraid of loosing the love of their parents, and can develop all sorts of terrible explanations as to why a parent no longer sees them. Kids who feel rejected by parents are at risk for problems such as anxiety and depression. In this situation, your son would benefit from seeing a psychologist who specializes in young children. The psychologist should work both with your son and with you to assist you in making the best decisions for your son's mental health. The psychologist can help your son understand that just because his father is not making good choices, that he is still a lovable, wonderful kid. Your pediatrician should be able to refer you to someone.
If trying to convince your ex-husband to come back on a regular basis does not work, you may want to speak with your son's guardian ad litem and your attorney to determine what you can do legally. It is quite damaging for a child to be 'jerked around' by a parent. Your son does not deserve that. However, you can not just stop following your custody agreement because of your ex-husband's undesirable behavior, to do so would be contempt of court. You need to proceed carefully to protect your son in ways that are not going to get you in trouble with a family court judge. Your attorney may have advice about documenting the problems and getting your son's emotional state evaluated to see if you can petition the court to make some changes.
Here are some really helpful books that can be found on Amazon. The first focuses on problem solving and picking battles. The second will help you explain things to your son--it is a terrific book, and the third is an invaluable parenting guide that will strengthen your relationship with your son
1. Joint Custody with a Jerk, by Julie Ross
2. Difficult questions kids ask, and are afraid to ask, about divorce, by Meg Schnieder
3. How to talk so kids will listen and listen so kids will talk, by Adele Faber and Elaine Mazlish
Disclaimer: This post was written for informational purposes only. It is never intended to replace face to face psychological or medical care. This post is not intended to create a patient clinic relationship, nor to give or rule-out a diagnosis.