You might talk to a urologist, sometimes a regular MD is not too swift about foreskins. My son's pediatrician tried to pull his back when he was just a baby!
Regarding sex, I am not certain, but if it doesn't hurt when you get an erection, it doesn't seem like it would be problematic for you to have sex with it not retracted. Especially if you are wearing a lubricated condom, it won't tear or anything. The fact that it pulls back slightly without pain seems to rule out phimosis, which (again I think) seems to be characterized by pain.
My son has been reading books like "What's Going On Down There?" and he tells me the range for the membrane that holds the foreskin forward to dissolve is often given as age 9 to 18, but says that "18 is the average higher-end age given, but that high age can be higher." If you don't prefer to take a 9-year-old's advice on this (no matter how well read) you might google on a good health site the average age and top age of the foreskin being able to retract on a health site. I do think I remember (from my own reading when we decided not to circumcise our son) that age 19 is not considered freakishly old to still have the membrane.
If you haven't had a check-up with a doctor in a year or more, you could make an appointment and then bring up your foreskin issue at that time.