There are allergies, and then there are sensitivities.
If you were truly allergic to latex, you wouldn't be able to touch a latex condom, or have a doc touch you with latex gloves. A sensitivity can result in simple redness and irritation, or be more severe.
The thing is, any time you have unusal symptoms, you should get it checked.
AJ
Also it could be red bumps from irritation from shaving!