Definitely follow your doctor's advice, but don't be surprised if you find you need to taper off at a slower pace. Some people must taper off .5 or even .25 mg at a time. It might take a bit of trial and error to find the rate that works best for you.
I absolutely agree about taking your doctor's advice. Whenever I've had to do a slow taper such as you're talking about, my doctor had me stay on the lowered dose for a week before lowering it again. So I would stay at 4 for a week, then go to 3 for a week, so on.....
If you're finding that you're having more symptoms after lowering the dosage, definitely contact your doctor - it's quite possible you will have to increase your dose again, at least for a while before trying to taper down again.
Best of luck - I know it's not fun being on prednisone (I've been on it a LOT and many times at extremely high doses) - and the tapering can be some of the worst of it. Your body needs to readjust to start having your adrenals work on their own again and sometimes it takes a long time for them to start doing that, especially if you've either been on the prednisone for an extended period or even if you've only been on it a short time but at a high dose.
It is best to take your doctor's advice. It is normal that if symptoms increase with the reduction that you go back up for a while until the flare is under control.