It is likely that your children have some type of reactive airway problem that gets expressed as you have described. This may or may not be asthma. Of course, exposure to any irritant, such as smoke or lots of dust, could aggravate such a condition. While these social situations often are in the middle of medical problems, it is probably best to have your doctor check the children for asthma and determine if exposure to smoke is in fact increasing the problem or not. If the children do not have reactive airways, this may simply be one child passing on viral infections to the other and back again. This is also not uncommon in children, particularly young children and those in day care.
Your child may have Reactive Airway Disease (RAD). It is very common for many things to affect a child with RAD (just like second-hand smoke). Even if your ex-husband smokes outside he carries articulates into the child on his clothing and the child then breathes them in. It is very common and possible to have RAD and not have asthma. Also a disease such as bronchitis at such a young age could be the cause of RAD. My daughter had RAD. She is nine now and much less sensitive to environmental stimulants but any time she gets a respritory infection (of any kind - even a cold) she begins to crackle and wheeze.
I can't explain the link between your partner and you becoming ill at the same time.
Hope this helps. It would be worth asking your pediatrician about it.