3 year old has occassional agressive and defiant behaviors

My 3 year old son..to lay the ground work... is extremely self sufficient--likes to try to do everything for himself..dress, brush his teeth, put his baby sister to bed, and would even drive the car if we let him.  My husband is a little stricter than me and doesn't let anything by him.  I am a little more passive and will worn and give time outs all day long over and over...However, most of this is just back and forth normal development with parenting styles and 3 year olds. My Concern is that more frequently there is this odd behavior that is so distinct on certain days I can't help but feel it is related to some chemical or something in our environment.  My husband an
I can both look at each other and go..ok here we go...He gets very aggressive punching, easily frustrated, whiney, and seems to continue bad behavior on purpose all day long.  He usually goes to be and wakes up find or it lasts for another day, and then subsides.  I guess I would explain that the frequency and duration of the continuous bad behavior on these certain days is to the point where something is definitely going on.  I feel if he is exhibiting these behaviors then he is feeling like this inside as well.  I started trying to read into his diet to see if it could be food intolerance, but I'm having a hard time and I'm not that patient.  We recently cut dairy out of his food.  We were going on the second week and this is the 9th day and the behavior returned..even with no dairy in the diet.

I am just wondering if I am on the right track with looking into food being the culprit?  He has no obvious food allergies like rashes or hives or swelling and does not complain of stomach aches.  Although my family has a long history of allergies.  He seems to be extremely hyper and aggressive and it usually starts in the morning or after nap time.  

Any insight or knowledge into these symptoms or guidance on if I should continue doing food exclusion techniques or allergy testing would be greatly appreciated.  

Thanks,
Mother of 3 year old boy