Dear shubunkin: This sounds like an abcess (pocket of infection). You should keep it clean. If it becomes red or swells, you should see your doctor. He/she may need to express the drainage and/or you could need an antibiotic.
it sounds like an infected sebaceous cyst (oil gland). It will need surgical removal at some point. Not a big deal.