Yea wearing gloves and or at least washing your hands you should be fine. this is my second pregnancy and both times I was the one cleaning the litter . I have heard your not supposed to but the main thing is if cats are outdoor catching animals an such but if they are indoor and not catching rodents. everything is ok.
Change it everyday! Its the lingering litter thats the most dangerous
Wearing gloves and washing your hands afterwards is just fine