Nobody can make you give up anything, the decision is yours, just use the knowledge you were given wisely. My son tested allergic to dairy, soy, wheat, peanuts and pork. He was eating peanut butter crackers that morning, we didnt see anything alarming, but hes only 2 so we cut everything they said out of his diet, a week later he somehow found a reeses peanut butter cup and ate it through the wrapper, and we were very fortunate he didnt ingest it, because the little bit he did he threw up and his airways started to close from that little bit. Reactions and severity can be unpredictable, and im making decisions for a two year old, so I dont take chances. You know when you dont feel right, so taking a chance isnt as scary for you, if it were me, I honestly wouldnt cut it out if it didnt bother me.
Allergists always say that reaction trumps all blood testing as it is not completely accurate. My dd is allergic to peanut but non reactive so she can eat it but my mom tests negative for soy but it causes her major issues so she is allergic. That is why allergists use both skin and blood testing along with history of past reactions to diagnose allergies.