I agree, unless you have been taking daily I doubt you will have any real problem with physical withdrawal. It might be mentally challenging to not have them as a cushion, but I think you will be fine. You should speak to your doctor as anything is possible and everyone is different. I am weaning down myself, but I have been taking daily for a year.
That is a very low dose, I would see your doctor for weaning advice but I dont think you will have a problem with withdrawals.