Mabye you should try to set time limits for when he can or can't watch TV. Like mab an hour before lunch, after lunch, and after dinner, for a total of 3 hours. Then minus min untill you think it's undercontrol. If he doesn't like the new arrangement then if he watches TV when he's not suppose to then sit him in time-out everytime he disobeys you. You could also try making schedules for everyday when he's not in school or for when he gets home from school. Set activities everynight so he is kept entertained. Hope this helps!
Turn off the TV, this forces him to find other things to do. Play games with him, get him outside doing fun things, have him select some books and read more, schedule play dates for him with friends from school. Since he is an only child it is up to you and his father to interact a lot with him and find other activities for him. There are summer day camps where he can be with other children his age, look into these when he's out of school for the summer. He watches TV becauses he can and he's bored.