Permanent bottom retainer

I was wondering if anyone has had problems with their teeth moving after they removed their bottom permanent retainer.  I have one that is just the wire behind my bottom front teeth.  It is really aggrivating.  Its hard to clean and floss around and under it.  Heres the thing, I lost my top retainer (the kind you are supposed to wear at night) two months after I had my braces off and my teeth haven't moved yet, and its been three years.  So do you think I would have problems with the bottom teeth moving if I had the retainer taken off?