Why do you want to get pregnant and have a baby in a household where someone smokes?
Smoking the seeds and stems is very bad for you! Smoking the seeds can lower the sperm count but I wouldn't say he is sterile! If you are both wanting to get pregnant then he needs to stop and also have his sperm count checked. If you don't want to get pregnant and he is saying that so he doesn't have to use protection, then you need to wear protection anyway!
Smoking the seeds and stems gives you headaches. Have you noticed him having headaches? I highly Dout he is smoking the stems and seeds if he is a true pot smoker! Not that I am a expert but I smoked back in the day and stems and seeds are just something u don't smoke. Google it for more info!
You should talk to your husband about this