OK, well, as I said before, it sounds like you conceived around December 5. This does not mean you were pregnant three weeks before the sex with the other guy, it means you were pregnant one week before the sex with the other guy. As you can see, this is not what would be called a compelling margin for safety, but pregnant is pregnant, and the other guy's sperm would not have altered or affected your baby. I am sorry it is not months and months different, you are just going to have to live with the nervousness. But your early ultrasound would not indicate a date that is so far wrong as even a week, meaning, it says you were pregnant (by about a week) when you and the other guy had sex.
The reason you have been thinking "three weeks" and I am saying one week is that doctors measure the pregnancy time period as beginning on the first day of your last period, not on the day you conceived. The day you would have conceived (according to your ultrasound evidence, thank heavens) is December 5. That would be a couple of weeks after your first day of your last period as the doctor calculates it. When a doctor says "you're 7 weeks along" or "you're 7 weeks gestational age," he means you're about 5 weeks since conception and 7 weeks since the first day of your last period.
To answer the question of whether a pregnancy can be altered by later sex with a different man, no it can't. The baby was formed when sperm met egg, and that was it. No later sperm can change that.
You have a tighter timeline than you think, though. You were told on January 12 that you were 7 weeks 3 days gestational age, and your due date is August 28, correct? Just to be sure I understand you, did the doctor tell you the due date from looking at the ultrasound on January 12?