That you have experienced difficulty breathing and shortness of breath is not surprising. The poison ivy could irritate your airways, causing either asthma or acute bronchitis. In addition, the "hairy" component may be an overgrowth of mold or fungus on the poison ivy that could also irritate your airways.
In one report ingestion of poison ivy (Rhus Toxicodendron) resulted in skin lesions, changes in white blood cells, and liver function abnormalities; described this as a toxic reaction. Fever was also described but no mention of the pain you have experienced. Nevertheless, what has been described suggests that your symptoms may be related to the ingestion of the poison ivy.
If your breathing or any other of your symptoms worsen, you may require high dose corticosteroids to quell this reaction. The trouble breathing is especially likely to be due to this inhalation as there have been many instances of what is called hypersensitivity pneumonitis (HP) due to the inhalation of organic materials.