are you sure its not because it was with your ex and you just didnt have any feelings for her? so your penis didnt have any feelings for her either?
I struggled with erectile problems thru most of my life. Unfortunately Viagara, Cialis, etc... were not available when I was younger and needed them. Take advantage of these wonder drugs. Get your courage up, go to your doctor, and get a sample. It works... trust me.
It would be helpful if you back-off on the masturbation too. You will increase sensitivity and the "build-up" will definately give you an extra boost.
Finally, take things slower if you feel that you are anxious. If you like and care for the person you are with then you wont have as much anxiety and nervousness around sex.
Good luck,
J