HPV (the virus that causes genital warts) is notorious for lying dormant for large amounts of time. It varies within every individual. There is much discussion about this in the expert forum, I would definitely check it out. Warts on your fingers wouldn't cause sexually transmitted warts on your genitals- warts on your fingers are caused by a different strain of HPV (there are nearly 100 altogether- 30 of them which affect the genital region). For example, I acquired genital warts from protected sex with a partner who has NEVER, EVER had a visible genital wart and is totally and completely asymptomatic. As far as I know, his partners (aside from me) are currently asymptomatic as well. It just varies within every individual- he will probably never have a genital wart in his life, and I started showing signs 2 months after exposure. So it completely depends on the person. I'd still see a doctor to be safe and find out what it is, and perhaps to get whatever may be afflicting you removed if you deem it unattractive. Good luck.
So you're saying that it is possible that genital warts can lie dormant for a few years? Also, I had warts on my fingers a couple years ago, but they went away after I wrapped them in duct tape for a week or two. My girlfriend gets a pap smear every 6 months, and she doesn't have genital warts. I'm not concerned, though. They don't itch or cause pain but are pretty unattractive.
Before I worried about tracing my infection (keep in mind this is virtually impossible to do as well, considering HPV lies dormant in a lot of people and many are asymptomatic while their partner[s] aren't), I'd go to a doctor and make sure these are indeed genital warts and not something else.