Are you wearing the right shoes sumtyms shoes can affect the way your ankels and feet feel myou might wanna go to the doctors you could even have a bone problem! ?? :)
In your past if you hit it by mistake it could just be swollen on the inside. : )
You might get checked to see if you have peripheral vascular disease. This disease of blood vessels where plaque formation happens and blocks off blood flow, causing a lack of oxygen to tissues can cause pain when walking. They can check on this with seeing if you have strong pulses in your legs and feet (pedal pulses), normal leg blood pressure and using a doppler to check on blood flow in your lower extremities. Peripheral vascular disease can strike as early as teenage years and can be hereditary.
Another thought would be a musculoskeletal issue, because you mention the irregular surface and ankle disability- would you consider yourself to be in excellent physical condition or more in a deconditioned state?