The problem is this. There is a "window of opportunity" for bones to knit, and if this window is lost they may never heal. It is hard yo say what effect this will have on your (a) mobility and (b) pain. I would say avoiding future chronic pain should be the main consideration. . I fractured a metatarsal bone forty years ago and it still gives me pain. You need an MRI for a definitive answer, and a second opinion.