Talk to your doctor and see if the doc will prohibit it, if your husband is not listening to you protest. It sounds like he is bullying you for some reason.
No, baby will be fine. LO is safely snuggled up in your uterus. I shook my belly all the time through my entire first pregnancy and my daughter would kick and wiggle but it didn't bother her otherwise.
If it stresses you out, you can always tell your husband it's making you nervous and you don't want your belly shaken. Even if he thinks you're being irrational, he'll probably understand you want to be as careful as possible.