Yea I can't have dairy, but have been able to tolerate some cheese while pregnant. There's a lot of conflicting studies, but the most recent I read actually said eating certain foods while baby is in the womb can actually prevent allergies. So eating nuts while pregnant lessons the chance of baby being allergic to nuts.
My cousin was allergic to peanuts before she got pregnant. Then when she got pregnant, her allergy went away and she was able to eat them. After giving birth, her allergy came back. Her son isn't allergic to them.
I would talk to your doc. Most likely you're just craving a few vitamins that come in milk. Maybe calcium or vitamin d... I suggest googling what a milk/dairy craving means while pregnant.
Also, babies don't get allergies to foods just because we have them.
I'm in the same boat... for me in not lactose intolerant til the baby comes out then I am again... good luck and enjoy some ice cream