4mg of hydromorphone

I have been using hydromophone 4 for about 4 years for pain caused by severe chrons (with fistulas ), it worked well I was taking 4mg every 4-6 hrs, but now my chrons is under control but I continue using 4mg per day, i would like to stop but keep thinking its helping me in some way. I have tried to cut down to 2mg than nothing but as soon as i do that i feel like im having flare up so i go back to the 4mg per day, i have noticed nood swing when i try to stop taking it.
any suggestions on how to stop would be good, also i worry about the damage that the hydromorphone has done?
thank you

[LeaAnn807]

If I am thinking right, hydromorphone is dilaudid, right?  It's very strong, and I'm sure it's hard to get off of.  The best advice on getting off of it, since you get it from a doc is to talk to your doctor about tapering properly, maybe using some other meds to help with the withdrawal???