Sperm doesn't really smell and this sounds like the situation my ex-fiance was in - she would let every guy *** inside of her if he wanted and it was up to him whether he wanted to wear a rubber or not. (I know - not very healthy, right? Let's say I found out this info WAY down the line!) However, her vagina had a very fishy odor as well and it was very difficult to perform oral sex on her. (I got around it by performing it right after a shower.) So, yes, there may not be discharge or anything (which is good) but just clean yourself out by douching one or two times. That's what she did and it was just fine after that. :-)
Bacterial vaginosis is something that comes to mind. This is something you CAN get from a partner and give it back and forth and you can also just 'have it'. The bad news, it is hard to get rid of. You'll need medication and if it persists, so will your husband. Go visit your gyn about it. They'll swab you and treat as necessary. good luck
a bad odor usually means infection or std. If you have with multiple people without protection get an std test done.