Not only would it be really hard for the sperm to reach the egg... but my husband is half hispanic and I am white. He gets quite dark but our little girl is blond with light skin and blue eyes. It all depends on the dominant genes.
It's hard enough for sperm to reach the egg, but to travel from the toilet paper to her egg would be very difficult. Chances are, any sperm on the toilet paper died before she touched it.
After so many seconds or minutes sperm is no good. I just think the baby took after your family not his. The baby does not have to come look liking him.