Welcome to the STD forum. I'll try to help. The bottom line is that you are not infected with HSV-1 or HSV-2. Whatever the cause of your oral and genital symptoms, neither is due to herpes.
Oral sex is safe sex. Not completely risk free, but much safer than vaginal or anal intercourse. It is true that oral HSV-1 can be transmitted to the genitals by oral sex. But that probably happens less than once for ever 1,000 episodes of oral sex. In addition, oral herpes is transmitted primarily when the oral partner has an overt, visible cold sore. If your partner didn't have such symptoms, your risk is even lower. Further, neither your oral nor your genital symptoms are typical for herpes; and a dermatologist's judgment that herpes isn't present (as well as a diagnosis of contact dermatitis) is pretty reliable evidence.
As for the blood tests, I don't know what to make of an IgM antibody screening test result of "Detected A". That makes no sense to me. In any case, IgM antibody testing for HSV is completely and totally useless and should be disregarded. That result does not mean you have ever been infected with HSV at any time. For a comprehensive explantion of why I feel that way, see this thread from a year ago:
http://www.medhelp.org/posts/STDs/hsv-2--recent-vs-not-recent-infection/show/541451
If the irritated area on your penis develops blisters or open sores, return to a health care provider (preferably the dermatologist) right away. Otherwise, I wouldn't worry about herpes. It's too late for a new herpes infection to first show up; it would have given symptoms within a few days of the oral sex event.
I hope this helps. Best wishes-- HHH, MD