All that can be said with certainty is that somewhere along the line, you had sex with someone who has an HPV infection with one of the types that causes genital warts, esp. type 6 or 11. The negative pap smear after your first two partners means nothing; you still could have acquired HPV from one of them.
And there is no way to know whether your new partner got warts from you or you were infected by him. Warts can show up anywhere from weeks to months (or years) after catching HPV, so the fact that he or you noticed warts first says nothing about who gave it to whom.
It is a useless game to try to figure out where HPV comes from. Don't try, and don't worry about it. Just follow through with whatever treatment your health care provider advises.
Good luck-- HHH, MD