Yes, you are, but you'd have to let the hospital know so they know you don't want him there and could deny him access.
Yes you have the right to deny him access just let the hospital staff know, but do you want him to sign the birth certificate? He'll have to be there in the room with you for that part.
I get that you would be uncomfortable but have you spoken to him about him missing his child's birth. Could you not have a rule where he stays above your waist or something or like a meter away and have like your mum or Dad with you?