At 20 weeks, it is too early for baby to move consistently and doctors do not recommend starting kick counts until much, much, later for this very reason.
I wouldn't be worried at all. I was only 5'4 and 98 lbs my first pregnancy when I started feeling my daughter. It started at about 15 weeks every day a couple times a day and then at 21 weeks for a couple days I felt nothing. I got worried so I went to L&D and they found her heartbeat, but because I was high risk my doctor had them give me an ultrasound to calm me down. Imagine my shock when I saw my daughter bouncing around and kicking all over the place. It turns out that she had moved against the placenta or something like that and so in that position I was unable to feel her movements. They then advised me that not feeling movements at this point in my pregnancy for a couple of days is ok and that you shouldnt even start doing a kick count until about 26 weeks when the kicks become more regular.
i started to feel my baby at 16 weeks and then 17-20 was just kinda weak and not too often. all of the sudden at 21 they are much strong and much more often. its not that your baby isnt moving, you just cant feel it. my doctor says that the baby will be about 1lb before you feel movements often. im sure it will increase within the next 2 weeks.
i have a doppler so i can tell there are a lot of kicks that i do not feel yet. she makes me feel so guilty when i use it i hear tons of kicks and dont feel many of them.
This is my second pregancy but i miscarried at 8 weeks last time which was over 2 years ago, everyone just just keeps saying its fine and not to worry because they picked up a heartbeat but when you have felt the movements for weeks and then it just stops its really scary trying not to worry but its hard, i am going back tomorrow to listen to the heartbeat again.
It could have something to do with the position of the baby? Is this your first pregnancy? I know that I didn't feel really strong movements (ie. ones that I'd notice even if I was occupied doing something else and not thinking about it) until I was over 20 weeks. Actually, I don't think I was sure that I was even feeling the baby until I was 21 weeks or so - the second time, it happened wayyyy earlier (by 14 weeks!!!)
If you're still concerned, then I'd say you should go back to the hospital just to be on the safe side.