You need to discuss this with your doctor, he/she will be able to help you wean safely since the baby has been exposed to it. Dont take it into you own hands.
They would not have harmed your baby.
1) Placenta has not yet developed so any medications you have taken up to this point will not be transferred to baby
2) Tramadol is a Class C drug - no evidence has been found to show it causes fetal harm in humans. It can be taken when the benefits outweigh the risks.