As a Remeron user (30 mgs.), I don't know if it has the same effects on men as women - I'm assuming you're a guy?). It's a bit sedating, causes weight gain, increase in appetite, and does effect my sexual drive (it's zero).
However, I also take a low dose of xanax - 0.5mg./day, so who knows if it's also caused by the combo of the two.
I wish I could tolerate the SSRI's - they make me too aggitated.
And, talk therapy, with the right therapist, is very effective, too. I felt my best when I combined it with my Remeron.
Good luck.....
I prefer talk therapy, but if you need medications in addition, then you might try wellbutrin because it does not have sexual side effects, and in most cases is quite effective.