Does this swelling come and go? Have you ever noticed it disappear entirely, then come back again some other time?
If that's the case, this is no problem, it is quite natural, and is called the Bulbus Glandis, a swelling at the base of the penis, on both sides, which happens when the dog becomes "excited".
If the swelling does NOT come and go, and is there all the time, then that's different.
I shouldn't really hazard any kinds of guesses about what it is, as I might give you information which isn't correct. As these swellings are soft to the touch, my feeling is it could be possibly a fatty tumor, or Lipoma. They are benign and harmless so long as they don't cause any irritation problems. My dog has had a couple of these most of her life. BUT, like I said, I don't know. I can't see it, I can't feel it, and so please dont take what I say as gospel truth!
There really is only one way to deal with this. Make an appointment to see the vet this week. (as quick as possible, or you will only worry) The vet will (or should) do a needle biopsy on these lumps, and find out exactly what they are, then take it from there.
A needle biopsy is done quite quickly, takes only a couple of minutes to do, you will most likely be able to stay with him while it's done, to hold him still and steady while they take the sample. It does not need any anesthetic, and is only mildly uncomfortable for him, for a very short time.
So don't hesitate. Get the lumps checked.
Many many dog-lumps are benign. In fact -most of them are.
is your dog fixed? if not you need to take him to the vet he may have tumors or something.