There is no correct answer to this question. If you are having irregular bleeding in conjunction with the polyp, it should definitely be removed.
If your periods are regular, then the answer is less clear. There was a study out of the University of Iowa several years ago which demonstrated that > 50% of polyps disappear on their own (although the percentage is lower for larger polyps). This suggests that waiting and seeing what happens is reasonable.
On the other hand, I am aware of one case report where a person with regular periods had cancer detected in her polyp when it was removed.
Since removing the polyp is a relatively easy procedure (although no procedure has a 0% complication rate), and you want to get pregnant, I would lean towards having it removed in an outpatient procedure/ home same day.
I cannot comment on the need for IVF as I do not have enough information regarding why this recommendation was made.
Hope this helps.
I had a polyp taken out that had grown to the size of a large orange. Uterine polyps are almost never cancerous (something like 1 in 10,000 are cancerous). But you should get it out before it grows any larger (and it will continue to grow), as the bigger it grows, the bigger the surgery you will need. And if it grows too large and you do get pregnant, the pregnancy could be in danger.