Make sure your blood pressure isn't high that's what started my terrible headaches with my first had to have him at 38 weeks turned out to be preeclampisa
Agree that you should let your doctor know. I'm sure they check you and you go to the doctor weekly now. I had high blood pressure when pregnant with my second. They monitor you. Do you have swelling or pitting of skin? See floaters? Otherwise, Tylenol daily is not bad. Better than a headache! Try to get rest. And you are at the end of pregnancy, try having something with caffeine in it . Sounds weird but caffeine can help. So rule out high bp and go from there. Almost done!!
When I was pregnant with my first I started having terrible headaches never felt anything like it before then about a week after they started I went into labour. I read somewhere that the headaches were an indication that labour was going to start soon (not sure how true that is) but I would mention it to your midwife just incase it is your blood pressure