This may not be what you want to here but it may be time to leave him. You can not let your baby be around that. Your baby may grow up and think that those things are okay. It's his job as a father to support you and mainly protect his baby. I know alot of fathers may say the mother is uptight and over exaggerating, but it is up to you to prove that he isn't taking this serious and if not leave him and explain to your family how you baby isn't being treated right.
I agree with ^^^^^ You can leave the situation without leaving him completely. Just live with your family because things are unsafe for your baby. I swear that would really tick me off! Him not saying anything and them not giving the baby back when it's time for feeding!
Your baby has to come 1st!! You already said the environment is unsafe for your baby, I would leave. Not sure of your situation, but I would live with my family, if that's an option. If it's meant to be with your boyfriend, it can wait until after he changes and demands changes for your baby.