Have you seen a urologist?
It can shrink as you age, or because of scar tissue, which could happen with injury.
Smoking and some medications can cause it, too.
3 inches is quite a bit of shrinkage, though. I'd definitely get checked by a urologist. At 51, it's time for prostate testing anyway, so it's a decent time to establish yourself with a good urologist. Keep us posted on what they say. And if needed, push your doctor for a referral. You know your body better than your doctor does.