Allergic to alcohol (not rubbing alcohol)

When I was younger, I never drank. But, when I started to, every time I would drink, my whole face would break out into a big rash or hives. My choice of drink is rum and coke. Never really liked beer. So now after I started to drink on a regular basis, it stopped and I never broke out anymore. Now recently, I quit drinkin again, but say once a month I will have maybe a drink or 2 when I go out. I am now breaking out again. The other day it spread all down my arms.. It was a bad break out. I asked my doctor the other day and he said he never heard of such a thing but to go home, get rubbing alcohol and put it on my forearm. If a rash appeared, then I am allergic to alcohol. So I did, and nothing happened.. It seems to be only when I ingest alcohol internally. So I'm guessing something in liquor? I have no clue. So, I was wondering if anyone knows what it may be? Oh, also, if I take 2 benadryl it does seem to help. I don't know!!!