Phyllodes or Fibroadenoma

I discovered a lump in my right breast in Feb. 2008.  At that time it felt about the size of a pea, and being that my Father's family tree is saturated with multiple forms of cancer; particularly breast CA, I made an appointment to see my doctor, which wasn't until May.  She sent me for an ultrasound, at which time the mass was 1.7cm X 2.2cm.  It was diagnosed as a lobular mass with well defined margins; and I was scheduled for a vacuum assisted biopsy.  At the time of the biopsy the beginning of June, the mass was 2.1 cm X 2.8 cm.  The surgeon stated he removed as much of the growth as he dared and his report read that at the end of the procedure "the mass was drastically reduced in size".  The pathology diagnosis was Fibroadenoma.  Four months later I had a follow-up ultrasound which stated the mass was again almost 3 cm in diameter; but the radiologist's report said the mass was "stable and unchanging".  My doctor referred me to a different surgeon, and she said that the first surgeon could not have reduced the size of the tumor with that form of biopsy and where the mass was the same size, she recommended just monitoring it and not removing it.  I don't have another appointment scheduled until the middle of July, but it seems larger to me.  I don't want to keep running to my doctor if I'm only paranoid...so I'm looking for advice.  Anyone??