I did the same thing. I concluded it was loss of muscle from changing my workouts.
You don't have to put on any weight during pregnancy.
Most of the weight you gain includes amniotic fluid, baby, growing breasts, etc.
All of that at 40wks generally weighs 15-20lbs
I'm the same way. This is my second pregnancy
I'm 15wks. Started off pregnancy at 128lbs, currently down to 121lbs because of nausea and loss of appetite.
Just take your prenatals and you'll be fine.