You should see your GP and get a referral to see a urologist, your testicle is probably getting stuck in the inguinal canal and you may need surgery to tighten the opening to prevent the testicle from ascending.
It probably isn't necessary to go to the ER, if you were experiencing testicular torsion you'd be in a very significant amount of pain and likely have substantial swelling of the affected testicle along with other obvious symptoms. However if you do at any point experience those symptoms you do need to get emergency medical assistance.