While it is (probably more theoretically than in reality) possible for sperm to be transferred on the fingers, it has to happen quickly. Sperm dies when exposed to the air, and so it would mean it would need to be fully encapsulated by semen to keep it from exposure to oxygen. Pre-ejaculatory fluid might not be thick enough to protect sperm for long at all.
um also I'd like to add that he waited for a good 10-15 seconds before fingering me