There are some unlucky men who get anal warts without having received anal sex. It is not as uncommon as you think. Some people think it may have to do sharing towels after sex or from spooning with an infected person. Genital HPV can be found on the thighs, buttocks, scrotom, pubic area and on the penis. Possibly the legs too. Which means warts can also appear on these areas.
The relationship of HPV and symptoms like genital warts to a weakened immune system (not just due to HIV) is speculative but many people believe it. Pregnant women have weak immune systems and can show warts if they are infected with it. Smokers and alcoholics have weak immune systems and are often diagnose with warts if infected with it. Some of us think a strong immune system is the key to clearance (not infectious) and we lead healthy lives to keep the warts away. There are no medical studies proving this but many think it does.