You need to call your Doctor any time there is something like this concerning you. They told me to sit in a quiet place and count the kicks.
Counting your baby’s movements
There are numerous ways to count your baby’s movements and numerous opinions on how many movements you are looking for within a certain amount of time. The American College of Obstetricians and Gynecologists (ACOG) recommends that you time how long it takes you to feel 10 kicks, flutters, swishes, or rolls. Ideally, you want to feel at least 10 movements within 2 hours. You will likely feel 10 movements in much less time.
You might want to start a notebook of your own or use the various charts below. In a notebook, record the time you feel the first movement, place a check mark for each movement you feel until you reach 10, then record the time of the tenth movement. This will help you observe patterns and discover how long it normally takes for your baby to move 10 times. Keep in mind that you are looking for significant deviations from the pattern. It can become easy to expect an exact amount of time every time you do your kick counts; however, there can be a wide range of time differences. So remember to look for significant deviations from the pattern over the course of a few days.
This is where i found the info if you are interested in reading more about it!
http://www.*****************.org/duringpregnancy/kickcounts.htm
Doctors suggest to start monitoring your babies movements at week 28. It could be that your baby is still small enough right now that you can't feel him every time. It could also depend on where he is positioned right now. If you're really concerned I would definitely call your doctor or midwife.
They do slow down, if you are worried I would call the midwife to put your mind at rest that's what they are there for xx