As long as your doctor says baby is growing fine then don't worry. Lots of women only gain like 10 lbs and have very healthy babies. Every pregnancy is different :)
I've lost 19 pounds and I'm 18 weeks my baby is fine, i wouldn't worried about it. If your doctor is not concerned then don't worried or stress about it.