How often are you taking Plan B? Are you sure the side effects you think are from the pill aren't from the Plan B?
You don't say what day of your cycle you had the sex on, so it's hard to say what could happen, but if he didn't ejaculate inside you, the chances are really, really low. You also used Plan B, lowering the chances even more.
Since you have such a fear of pregnancy - which is normal if you aren't ready to be a parent - you should explore your birth control options with your doctor. There may be another pill that works for you, or another method.
Between PlanB and no ejaculation the chance of pregnancy is vanishingly small. Condoms can get lost like this because the reservoir does not accommodate the semen which backs up the shaft of the penis and lubricates it enough to slide off. In your case he must have lost his erection and stayed in too long. It’s good practice for someone to reach down and make sure penis and condom come out as one. An IUD might be cheaper in the long run.