Why don't you stay awake when he is in there and just walk in the room in the middle of the night. You need to trust what you feel inside and act.
I think you need to have some straight talking here and tell him you are not comfortable with his behavior , havent you asked your daughter about it , you should sit and ask her , when you know he is in her bedroom you should call him out and really not allow it as it seems like enablement .,the fact you know and he is still doing it, if he wants to talk to your daughter tell him you will join in and talk in the living area.
What's your gut telling you?
I think this is one of those things where you need to trust it.
If it's true, you need to get your husband out of the house PRONTO.