Is there a number 1 in this story? If number 2 is the only choice, yes, he's the father. But it sounds like you are asking about two people.
OK, I got it, the sex on May 31 was not with number 2.
Because sperm can live in your body for 5-6 days, sex on May 31 (if unprotected) would give you live sperm in your body on May 31, June 1, June 2, June 3, June 4 and maybe even as long as June 5, from Mr. May 31. Then sex with Mr. June 1 and 2 would have also put his viable sperm into your reproductive tract, and it would have lasted as late as (possibly) June 7.
You've pretty much gotta assume that both guys have an almost equal chance to be the dad, unless you got pregnant later than about June 7. Have you had an ultrasound? Counting back to the first day of the last period is not a reliable way to tell.
OK, let's unpack this, but in short I think you can't be entirely certain. The main thing you have going for the baby being the child of Mr. number 2 and not Mr. May 31 is that Mr. May 31 smokes a lot of weed and has evidently not been successful with other women at creating babies. Unfortunately, though, even heavy use of weed is not guaranteed to make the sperm not work, meaning it's not really a method of certain birth control.
Your cycles being consistent every month, and a bit shorter than average (the average woman's cycle is about 28 days long) would suggest you ovulate on day 12 of your cycle. In this case, that would be June 2, which is consistent with what your ovulation predictor kit suggested. It also pops you back into range for Mr. May 31. But it's not that consistent with your doctor saying your ultrasound said the baby is "behind" what she expected to see in your eighth week, if by "behind," she meant the baby's measurements weren't as big as she expected.
Unfortunately also, by the eighth week, an ultrasound has already started to be a bit less precise at being able to guess when conception was. If your ultrasound had been at 6 weeks 1 day, it would usually only have a margin for error of at most one day. By the eighth week, it could be 1-3 days off either direction. (By the 12th week, a doc will say "plus or minus 7 days" if asked to say when conception was by looking at the size of the baby in the ultrasound, and by your 40th week it can be off by more than 20 days.) This is because some embryos grow faster and some grow slower than average. So, your doc moving your due date to February 26 from February 23 doesn't necessarily suggest that conception was later, it just might suggest a slightly slower-growing embryo. Even if her moving the due date really meant that the baby was conceived on June 5, that's still just barely in range for Mr. May 31. And if you really conceived as late as June 5, that's nine days before your next period would have been due, and it just doesn't make sense for someone with consistent 26-day cycles.
In your original post, you said "I ovulated the 3rd." What gave you that information, did you have an ultrasound that saw that your egg had left your ovary? You can't really tell from a home ovulation test kit exactly when you ovulated, I don't think.
You also said, "Found out June 22 I was 4 weeks according to my last period date of May 21." Where did you get that? If you count from May 21, June 22 is 4 weeks and 4 days from your last period, right?
When you said "the dates line up to me conceiving on the 3-4," what specific dates say that to you? What I get from your dates is that your period's consistency told you that you would ovulate on the 2nd, and your ovulation test kit thought it would be the 1st or the 2nd (unless that was just the measurement of the LH rise, in which case your ovulation test kit was telling you it was your "fertile time" to have sex, but not actually saying it was when the egg pops out). Then your doctor changed your measurement to a date that suggested you ovulated on June 5. Alas, even June 5 is not quite out of range of Mr. May 31, though again, it seems like for other reasons he's not the strong candidate.
This is why I am saying that pretty much almost any of those dates, unless it was past June 7 when you can be sure Mr. May 31's (possibly not very mighty) sperm was all dead, would be at a time when you had sperm in your system from both guys. Your Mr. number 2 is a more likely candidate because his sperm might be more healthy, but that's about all I've got.
Have you considered doing a pre-natal DNA test?