Guys have this fear that they will somehow hurt the baby, which they can't. That could be what bothers him? You should ask and see.
Have you tried to get him to open up about why? Just be sure to seem calm and level headed when asking so he won't be afraid of saying "the wrong thing"
It'll get better soon.... Hang in there. Remember your body has never done something as beautiful as a child. And men are scared sometimes or 'donkeys.'