I have heard the same thing about grapefruit but it doesnt appear to be common knowledge. I believe moderation is always key. .
I was told it could interfere with absorbtion, although 1 or 2 every so often will probably not have much of an effect. I still eat some every once in a while just because I like them. You probably shouldn't eat them all the time, but on occassion should be ok.