No, everyone gets rubbed with bloody hands when you shake with someone else, knife cuts, etc every month and no one gets hiv.
Your situation involves personal contact with an object in air (broom, maybe blood). No hiv worries, because you can't get hiv from personal contact except unprotected penetrating vaginal or anal, neither of which you did and you didn't share hollow needles to inject with which is the only other way to acquire hiv. Analysis of large numbers of infected people over the 40 years of hiv history has proven that people don't get hiv in the way you are worried is a risk.
HIV is a fragile virus in air or saliva and is effectively instantly dead in either air or saliva so the worst that could happen is dead virus rubbed you, and obviously anything which is dead cannot live again so you are good. Blood and cuts would not be relevant in your situation since the hiv has become effectively dead, so you don't have to worry about them to be sure that you are safe.
There is no reason for a person to test when they are safe. The advice took into consideration that the other person might be positive, so move on and enjoy life instead of thinking about this non-event. hiv prevention is straightforward since there are only 3 ways you can become infected, so next time you wonder if you had a risk, ask yourself this QUESTION. "Did I do any of the 3?" Then after you say "No, I didn't" you will know that it's time to move on back to your happy life. For example in your current situation the answer would be NO, I DIDN'T because there was no unprotected penetrating anal or vaginal sex nor was there any use of a shared hollow needle used for injection, so obviously it is a NO risk.
No, you can't get HIV from a broom stick. HIV is generally sexually transmitted and transmission happens inside the body. This is why there are three risks, having unprotected vaginal sex, having unprotected anal sex or sharing IV drug needles. Just touching a surface doesn't transmit the virus because the surface is exposed to air which inactivates it. There is absolutely zero risk from a broom and touching the broom even if you or he bled. No risk, no worry of HIV here.