One-Year Old with fever bot no other Symptoms

[BooX_BooX]

I have a one year old son, yesterday(Wednesday) around 3 I noticed my son's face felt hot to touch, I took him out of his clothes figuring he was just to bundled up,( he'd been outside in a long sleeve shirt and sweatpants) for a few hours.
I gave him about an hour in nothing but a diaper inside the house, I felt him again noticed his chest, cheeks and forehead was hot. I took his temperature, it was 102.7, I gave him some infant Tylenol and he took a nap.
When he woke up, about 2 hours later, his temp had fallen, to about 100.1.
Then later on in the night he wouldn't go to sleep, he stayed up all night, taking little 30 minute naps before getting back up, crying, he was stumbling over, how tired he was, but wouldn't go to bed until about 4 this morning after he had thrown up.
I checked his temp and it had gone back up to about 101, so I gave him another dose of Tylenol.
He finally went to bed for about 4 hours but when he woke up, he was still hot, I checked his temp and it was at 99.5 So I gave him one more dose of Tylenol.
Hes awake, a little fussy though, he doesn't want to sit on the floor or walk, just be held.

Should I take him to the ER even though all he has is a temp and he's a little fussy and wont sleep for long or wait 'till the morning and time him to see his pediatrician?

And, no one has been sick in the house for months, nor does he come in contact with any child that's been sick.
How ever, myself, his father and aunt go to school, so we do come in contact with others.