Why were you both together with your parents and pastor? Was this a crisis counseling session? Can you schedule a meeting with your pastor (if you are in counseling with him) to get his/her impressions?
What does he mean "you don't bring anything to the table"? Is that his way of saying you aren't making any money?
There are some typos in your post that make it a little hard to understand what you are saying - are your parents wanting you to divorce him? Are there other things he's done besides this very odd thing to say in front of your pastor and parents?
Hi, so what is happening is that you were in a counseling session for issues in the marriage? And he coughed up that he didn't love you or your daughter, and you felt it as a home truth for him and it hurt you that he said he didn't feel that way about his child. The fact that the part that bothers you is that he would say he doesn't love his child indicates you already had an idea he wasn't enamored of you any more -- and when he said this, your parents were saying, OK, let's talk about you two getting a divorce? (It doesn't sound like they were saying not to get one.) If so, this suggests that the problems were pretty serious, if the parents and pastor were there and nobody exclaimed in surprise and insisted the two of you just needed more time, or anything like that.
Here is the thing. Both of you will need to see lawyers, to get a divorce. You can call the nearest law school and find out if they have a legal clinic that handles family law and helps people with no money get a divorce. You need to be ready to ask him for child support, and you need a lawyer to do that.
You also need to make some plans, such as, where will you live, and who will help with the kids, and what money will you live on? Your husband and you can be civil about this if you make it a point to avoid screaming fights, it kind of sounds like you are pretty calm about the probable end of the relationship so that might not be so hard.
I don't know why he came back and claimed he didn't mean it, you know he meant it and if he were to stop blowing hot and cold he would probably agree. If you're pregnant and taking care of a toddler, he can see that you are not at fault for not earning money. (While everyone wishes they had or made more money, and spouses often think it would be nice if their spouse earned more, nobody is blind to the need for someone to take care of the children.)
In your shoes I would talk to my parents about all of this and work to find a legal-aid lawyer who can help you with the divorce. Your daughter is so young that she will not feel the loss of her dad right now, and if the two of you can keep a civil tone going, he might still be a presence in her life as time goes by. Not that it sounds like he much wants to. But in any case, he is obliged to pay support for both children. The law is on your side. Talk to your lawyer, it's important.