This sounds like a true cyst, common on the penis and scrotum, that is essentially now part of you. You can have it surgically removed, if you want, but I think you're probably better off doing what most people do, which is to accept it and ignore it. See a skin doctor for definitive diagnosis and treatment, if needed.
Best.
Dr. Rockoff