I am not exactly an expert, but I do have some advice that might help. The first thing I thought of when I read this was that you may be having mild anxiety or stress. The fact that you don't have breathing problems when you exercise points to this, I think. There are three areas where you breathe from: the throat, the upper chest and your stomach basically. When we are stressed (actually most people) breathe shorter, more rapid breaths rather than slowly breathing in filling the throat, then the upper chest, finally the belly and then exhaling (kids breathe more like this). In meditation they focus on deep breathing exercises doing this type of progressive breathing to promote awareness of how you're breathing and of course for overall relaxation.
Here's a great article about deep breathing that everybody could derive some benefit from doing:
The Content on this Site is presented in a summary fashion, and is intended to be used for educational and entertainment purposes only. It is not intended to be and should not be interpreted as medical advice or a diagnosis of any health or fitness problem, condition or disease; or a recommendation for a specific test, doctor, care provider, procedure, treatment plan, product, or course of action. Med Help International, Inc. is not a medical or healthcare provider and your use of this Site does not create a doctor / patient relationship. We disclaim all responsibility for the professional qualifications and licensing of, and services provided by, any physician or other health providers posting on or otherwise referred to on this Site and/or any Third Party Site. Never disregard the medical advice of your physician or health professional, or delay in seeking such advice, because of something you read on this Site. We offer this Site AS IS and without any warranties. By using this Site you agree to the following Terms and Conditions. If you think you may have a medical emergency, call your physician or 911 immediately.