If the sex was few and far between originally, things have pretty much followed the same course, so it doesn't sound like he is cheating with someone else. He might have ED, he might not be that interested in sex with a woman, he might not be interested in sex for other reasons, maybe he masturbates a lot and doesn't want actual sex much, who knows? If he won't tell you, nobody else can. You certainly can't "figure out what's going on" without asking him. Just ask at a calm time, in a sympathetic way. If you are thinking of leaving, you need to give him the chance to discuss it.
Thank you for the advice. I think it time for a chat and a visit to the doctors. I feel that becauce of the ED hes afraid of a disapointment of not being able to perform. That would be a good reason for the problem we are having in the sex department.