Depends - are you living in your own home? If so, then don't answer the telephone, or plan lunch or theatre dates with your friends, or go on a vacation. If you are "playing house" with this man, then the "son" is part and parcel of the arrangement and that includes childcare. This is what being married and having children is like - there is very little, if any, "my time". You don't get to have it both ways.
Be firm speak to him and make him realise you mean it, he should spend more time with his son what do they do in the evenings , ? How come he only sees him for one hour?