In my opinion.. He doesn't respect you or your unborn child. I would request a drastic change in his behavior and if he did not deliver or show consistency, I would no longer consider him my significant other.
Do you think he would agree to have counselling? Do you think it would benefit your relationship? You don't need extra stress, especially during pregnancy. I hope you and he can find a solution that will work for not only you two but your children too.
Try and relax. Think positive thoughts. Get a good book to read and motivate you. It will all work out.