High heart rate when walking

About a year and a half ago, I used to exercise regularly and each workout included 20-minute treadmill runs. During these runs, I would get up to 6 m/h and my heart rate would be around 130. It would rise gradually, then slow down the same way. Aside from the exercise, I could walk for a couple of hours non-stop and my heart rate would stay under 100 and I would feel no discomfort. Unfortunately, at that time I started having panic and anxiety attacks, which felt more like serious heart problems. This prompted me to stop exercising and go for a thorough test of my heart health. I've had holter monitor test, stress test, ecg, echo ecg, 2 week event monitor test, all blood tests and a chest X-Ray. The cardiologist told me that I am in perfect health and that I can and I should go back to exercising, which should help with my anxiety. The problem is that now I am afraid of starting to exercise because my heart rate jumps to 120-140 even from a short walk (1 mile or even less). A simple climb of a couple of flights of stairs gets my heart rate up to 140 or maybe even higher in a matter of seconds, I also get strong palpitations, arrhythmia and feel shortness of breath. I am not sure if this is due to my anxiety disorder, my lack of exercise in the last year and a half, or whether it is a manifestation of some weird heart problem that the doctors failed to catch. Any comments would be very welcome. Thanks