Something has obviously scared him. Talk to him. At the age of 9 he should be able to express himself to explain what is bothering him that he is having trouble sleeping. Consider putting a sleep machine in his room at night. Soothing sounds may help him calm down. Reassure him that you are not far away and that you will be there if he has a problem. When he does come running, gently take him back to his room, comfort him, yet allow him to express his fears. Although you may not find them serious or legitimate, they are very real to him. This may alleviate the problems, which could all stem from separation anxiety. If you still find no relief, consult his pediatrition again and consider a child psychologist. He may need outside intervention. When my own daughters went through this, I even switched bedrooms with them, which was a temporary "fix" to the problem, but a welcome reprieve, as like you, I needed the sleep.
This could be related to your divorce, even though it's been several years. He may feel that if he's left alone in his room, that you or your ex will "disappear" or "go away". I'm especially thinking this is a possibility since you say he tells you he is "scared". Have you tried asking him what he's frightened of? Bad dreams? Not being able to find you or your ex if he needs you?
You may want to try putting a night light in his room (or in the hallway just outside his room if his bedroom door is left open) - just this little bit of light may give him enough security if he's able to see where he is (especially if he's going back and forth from your home to his fathers).
One other point - benadryl can actually have the opposite effect on some children (and even some adults) - rather than making them drowsy, it can make them hyper. If you've noticed his waking has gotten worse since being on the benadryl, this could possibly be a partial culprit.
The first thing to do, I feel, would be to sit your son down and ask him if he can tell you what he's afraid of and try to reassure him that both you and his father love him very much and just because you no longer live all together, that you and your ex will NOT leave HIM and you both will always love him. Has there been any other changes in living situations recently - i.e., either one of you getting remarried, having a significant other move in, other children?
Best of luck and please let me know how he's doing.