We, as parents, are not perfect. We make mistakes and it's absolutely impossible to cover EVERY SINGLE potential danger to protect our children. I understand how you feel, but I think you are beating yourself up over nothing. Move him out of the room for the time being, see what the inspector says. If he hasn't experienced any symptoms by now then I don't think it's gonna cause him any problems. As Mamaof4410 said, take him for a check up to help ease your mind.
Mold can be dangerous. We had a leak in our old home and it was in the bathroom. Well the bathroom wall was also part of the wall in my sons room. By the time I actually noticed it the back of his dresser was rotted from the water and so was the floor. but there was mold. He never got sick or anything.Since your baby is so little you can take him for a checkup just to be safe.