How about your gf say to her, "Thank you, but I'm not interested in you sexually in any way, and please stop asking. If you can't respect my wishes, I'm afraid I won't be able to include you in my life." It's your gf's job to say this, not yours. This is your gf's problem to deal with, as it's her family member. It's probably also better if your gf confront her, and not you, as a male.
Why is this affecting your relationship? Set the boundary, and stick to it. Even if it's family - maybe even especially because it's family - your gf needs to be polite but firm, and stick to the boundary set.