It is hard to bond with someone you don't get quality time with. If all he sees of the little guy is after work when he is tired....its tough. You need to find ways for them to enjoy each other. Game playing, going to the park, etc.
I am also wondering that if his dad had similar issues, if the little guy might have something like ADHD. Your daycare teachers could give you some perspective on this. This is important because there is lot that you can do to make his life and yours a lot better if he does have something like ADHD. It is something that I deal with all the time over on the ADHD forum where I am also the CL.
His dad had add as a child but my son definitely does not have adhd he was having a hard time at school but is now improving massively in leaps and bonds as well as at home but there is still those occasions when he really acts out the school was keeping notes and letting me know what sets him off and there working with him to give him a easier time at school his dad does work amd in the afternoons my son os more restless heading up to dinner time as all kids