I was the same way at 20 weeks. I would only feel him at night when i was lying on my back. I am 27 weeks now and feel him a lot more. I think he dances on my bladder, but I can actually see my belly moving now as well. I have an anterior placenta, but I am still feeling lots of movement. Give it time, the movements will get stronger and you will feel them more regularly.
Totally normal! I worried the same, but now I'm 27 weeks and I think baby boy is having daily dance parties in my belly!