I had the name Esmè picked out for a girl too and my Husband said no as it was too much like a Twilight name. We're having a boy though so it is now irrelevant
I think the person will outlive the popularity of the book. I've always thought Esme was a pretty name, too bad it got too popular from Twilight. Have you considered Eliza Alise?
Whatever you name the baby, make sure you're happy with it and not be pressured by anyone to name her something you don't like.