I used it as well and never had a problem. I am now 35 weeks! =)
I had really dry eyes in my first pregnancy so I used it constantly. Nothing happened to me. I think you are fine.
Visine is a Category C drug.....Meaning that it it has not been sufficiently tested by the FDA to deem it either safe or not safe in pregnancy....Simply not enough testing done on it.....I wouldn't worry about a one time use, but also would avoid using it again until you ask your Dr. about it....just to be on the safe side....