Your situation involves personal contact with an object in air (finger, maybe blood). No hiv worries, because you can't get hiv from personal contact except unprotected penetrating vaginal or anal, neither of which you did and you didn't share hollow needles to inject with which is the only other way to acquire hiv. Analysis of large numbers of infected people over the 40 years of hiv history has proven that people don't get hiv in the way you are worried is a risk.
HIV is a fragile virus in air or saliva and is effectively instantly dead in either air or saliva so the worst that could happen is dead virus rubbed you, and obviously anything which is dead cannot live again so you are good. Blood and cuts would not be relevant in your situation since the hiv has become effectively dead, so you don't have to worry about them to be sure that you are safe.
There is no reason for a person to test when they are safe. The advice took into consideration that the other person might be positive, so move on and enjoy life instead of thinking about this non-event. hiv prevention is straightforward since there are only 3 ways you can become infected, so next time you wonder if you had a risk, ask yourself this QUESTION. "Did I do any of the 3?" Then after you say "No, I didn't" you will know that it's time to move on back to your happy life. For example in your current situation the answer would be NO, I DIDN'T because there was no unprotected penetrating anal or vaginal sex nor was there any use of a shared hollow needle used for injection, so obviously it is a NO risk.
You don't have to give up sex, however you put yourself at risk for Covid, so that is the only thing you have to consider as a health risk.
Ya, it does sound a little like a phobia. That anything and everything is making you feel vulnerable to HIV. But if you remember what the risks are, they are simple. Unprotected vaginal sex, unprotected anal sex or sharing IV drug needles. That's it. So, other scenarios like fingering, are never going to be risks. Fingering will not result in HIV. Even the real risks are less than 1 percent transmission. This is why HIV is not all that common. It's not that easy to transmit and is a fragile virus. Air and saliva inactivate it. So, those are the facts. If your rational side of your brain is being overshadowed by the anxiety, it's time to work on the anxiety. A therapist? Might help you.