You definitely need to focus on you and baby.. if the father is not willing to sacrifice a little bit on both of your behalfs, then what can you expect for the rest of that child's life.. what is the better situation for that baby?? Sounds like you have a good place where you are, with a job, support, a place for the baby.. those things are most important. I think a father should do anything for their child, as a mother would, and making changes for the BETTER, shows initiative..
It doesn't seem logical for you to move there. Ask if he would feel safe with your baby in the house with his dad. Also point out that there are bound to be disagreements in a family, including his. Is it possible for the two of you to get a place of your own?