My boyfriend of 10 years has a similar problem. He is never interested in sex, though he does have some nice images of big boobied ladies on his computer! His problem is because he is a vegetarian and eats far too much soy product. This has an oestrogenising effect dulling the sexual drive.
The other issue for me and he is that I am very much like his mother to look at, and of course, that is going to an issue.
For you it might be a good idea to sit him down and ask in a non-confronting way (as I did) something like, "I find you very sexually attractive and I am sad to see you aren't feeling the same way to me. Is there something I am doing wrong?" That's how I found out I am too much like his mum, after he dug out photos and I spoke to people about his mum (she died the year before I met him) plus the soy problem. I learned to live with it, I don't like it,but he is still affectionate.
Good luck
Sounds like he's keeping himself satisfied thru self-pleasure. He love his hand more than he loves you.