So, this is you boyfriend's sister, and you have a child with your boyfriend but you've only met this sister once and she was sweet and welcoming then. YES, you should go with your boyfriend. It's time to forge relationships with his family. He knows them. If he says you're invited too, you're invited. He should RSVP saying Ava and I would love to be there, and then go and have a good time.
What does the invitation say? Is it addressed to you both?
I'd imagine since you have been together long enough to have a baby, you'd both be invited, especially if it's a couples or co-ed shower, but your partner can simply ask his sister if his invitation includes a plus one if your name isn't on the invitation.
Remind him to not make it a huge deal, or to tell his sister you were unsure, just to ask because it wasn't clear.