It is not at all likely that you'd get herpes from this. The skin on your scrotum is thicker than your anus, and it's not a place that the virus infects if your skin is intact.
It is possible that you could get genital hsv1 from from a rim job, but do you know if you already have hsv1 orally? If you do, like 2/3 of adults globally, then you can't get it again genitally.
Chances of getting it from a quick, one-time encounter when your partner doesn't have any visible sores are low. It's not impossible, but not at all likely.
If you were to get it, you'd likely see sores between days 2-12.