I am not a doctor thus what I may say or think is of no importance, never the less I can search in Google and I am a good searcher. I have found two links that are to medical sites were the information you are asking for in well explained. Please forgive me but I don't want my lack of specific knowledge to hurt you.
Please go to these links and read well, if you find anything that suggests there maybe something that will damage your liver, print it and take it to your doctor. A suggestion: be well informed and insist with your doctor they tend to ignore information other than the one they have.
Dear Cinders11: Enclosed in quotations is the section of the femara package insert produced by Novartis. "2.5 Use in Hepatic Impairment
No dosage adjustment is recommended for patients with mild to moderate hepatic impairment, although Femara blood concentrations were modestly increased in subjects with moderate hepatic impairment due to cirrhosis. The dose of Femara in patients with cirrhosis and severe hepatic dysfunction should be reduced by 50% [see Warnings and Precautions (5.3)]. The recommended dose of Femara for such patients is 2.5 mg administered every other day. The effect of hepatic impairment on Femara exposure in noncirrhotic cancer patients with elevated bilirubin levels has not been determined."
We cannot evaluate your hepatic function and make clinical decisions about risk versus benefit via the internet. But hopefully this information may help you to have a good discussion with your doctors.