He's been with you 4 years? I'd get rid of him, after 4 years he's still not over his ex that's ridiculous, and just cuz his ex doesn't want him doesn't mean don't get rid of him, why keep him? He loves someone else and sure isn't afraid to keep it from you which is disturbing. When your in a relationship it's supposed to be mutual, not one person love the other while the other loves another, find someone who will love you and get over their ex, your basically in a relationship with yourself at this point
It doesn't sound like his ex wants him at all, so why send him away? It's not like he can go to her. Let him moon a while, people say a whole lot of half-baked stuff when drunk. He's just got "first love" up on a pedestal; it's not about who she really is now; he's wistful or mourning for being a boy, someone with no responsibilities, in love the first time. I was at a 20th high school reunion, and we all went out for pizza afterward, and Ronnie told about loving Laurie in high school, and he smiled and said ironically, "and, you know, there was never a love like that love..." I don't know if you can stand to live the mooning-around stage, but if you two have children it is worth trying.
ps -- On the off chance that you won't be able to stand it, please stop having kids. Two will be hard enough to take care of if you become a single mother.
I know this is hard for you to deal with. I'm sorry you are going thru this. Try talking to him and express to him how you feel when he is sober. It's hard to let go but if that's what you need to do you will have to for the sake of your kids. Don't stay in a relationship just for the kids. You can't be in a one sided relationship. so remember to talk to him and express everything to him and make sure he give you clear and complete answers and responses