What changed with this new lover compared to past experiences? Did you change your lube, or the brand of condom? It sounds like irritation or like an allergic reaction, not just the reaction to size per se. (Unless size is wiping out your natural juices and you are getting abraded on by dry skin. That can be very irritating to the tender tissues of the vagina.) If you think it's the latter, get some good quality lube, at the very least KY Jelly, or other lubes made specifically for sexual purposes. Don't use Vaseline, it is tough on the mucus membranes in the vagina.
Everything in that area should swell up, but if there is any issue with you feeling uncomfortable a OBGYN visit should take place. Blood flow speeds up and secretions flow. Sometimes it takes a bit of time for those symptoms to relieve. Some woman get irritated from friction, are easy access to yeast infections, may be allergic to latex condoms or the jells that can be used during sex, and so many other things. And/or just need some education on their woman parts. There are many reasons though that you may have swelling and they should always be addressed properly. A woman's body is amazing and mind boggling! We are all different, with individual issues. I would advise reading all you can about your female parts. Still, it is important to ask questions when needed.
Do you have enough of your own natural lubrication, or, are you using some other form of lubrication? If there's not enough things could be to dry and you're getting irritated.