There can be several reasons for this occurrence.
The first is that this could be a sign of an infection underneath the nipple area, which may eventually lead to a condition called mastitis. Additional symptoms pointing to this would include breast pain, swelling, discharge and fever.
This can also be due to lump formation under the breast with can be due to fibrocystic disease, papillomas, or other benign breast conditions which can be likewise seen in men.
More common conditions would include musculoskeletal problems such as inflammation of muscles and connective tissues in that area of the nipple, which is commonly caused by stress or trauma.
As there are several considerations, it is best that you have this seen by your doctor.