or let him have his shirt on so the habit lessens a bit...
A. Be glad it's not his penis. A lot of little boys find their little manhood early and like to touch and fondle it.
B. Be glad it's not *your* nipples. I'm sure my son would still be breastfeeding if he were allowed to -- he likes to rub and fondle a slightly protruding mole on my arm because I've told him breasts are private. I think it's a longstanding effect from breastfeeding as a comfort thing ... he gets this glazed look on his face and looks very comforted, just from rubbing the darn mole. I sure would like him to grow out of it soon.
C. If he's doing it around the house only, simply tell him that you would prefer he do that only in private. My son has a clear understanding of things he can do at home around only the family, and things it is OK to do in public. (Come to think of it, teaching him the distinction between private and public a while back was a very useful thing for a lot of reasons.)
My guess is that he is self-soothing, and if he had some other feature he focused on (like a sticking-out mole or a rough spot on his skin) he might stroke it absent-mindedly instead. Some of the women on the site have written in about other habits they had, like stroking their upper lip with a pencil, that persisted into sixth or seventh grade. Chewing the nails, twirling the hair, sucking the thumb -- all of these things defuse stress. If you don't think he will know to control himself in public, talk to him about that, but otherwise I would give it some time and see if the habit drops away. Most of them do.