I've lost 19 pounds and I'm 18 weeks my baby is fine, i wouldn't worried about it. If your doctor is not concerned then don't worried or stress about it.
As long as your doctor says baby is growing fine then don't worry. Lots of women only gain like 10 lbs and have very healthy babies. Every pregnancy is different :)