Anal and rectal warts occur in people who have never had anal sex (including heterosexual men)--not as commonly as in people who have, but your lesion might well have nothing at all to do with your distant past sexual practices.
You are right that removing rectal warts is difficult, simply because typically there are more than one lesion and it is difficult to identify them all--so recurrence is a common problem. Since most warts eventually resolve spontaneously, some argue there is no point in trying to treat them. All this also is partly influenced by the removal method, however. Surgery is a pretty big deal, but laser cautery (for example) may not be a big deal in an expert provider's hands. And of course not treating is appropriate only if it is certain there is no cancer or pre-cancerous changes in the lesion.
Personally, I have no experience in managing internal rectal warts; like almost all STD clinics, ours always referred such persons to proctologists. That's my only other advice: be sure your provider is very experienced in this area. Some general surgeons are, but not all; if in doubt, getting a second opinion from a proctologist (rectum specialist) would be a good idea.
Good luck-- HHH, MD