You would probably have to the first time, as there wouldn't be any other plan.
But only you know his behavior towards his child in this kind of hungover condition. Not all men will rise to occasion and yes, it could leave the child in a dangerous unsupervised way.
Let me tell you story of a dear friend of mine....many years ago, he worked nights, his wife worked days...they made the classic switch off, he turned the TV on for the little guy while he went back to sleep on the sofa next to him, and his little boy climbed up on the kitchen counter, into a cupboard where he found a bottle of liquid drain opener and was able to open it and drink it.
The poisonous drain opener basically disintegrated this child's esophagus and he died in his father's arms on the way to the hospital in the ambulance, choking on his own blood.
It's funny to say in other's post that they hope the child whines and throws tantrums. But honestly, your child is not being left with a coherent and responsible adult at that particular moment.
I would recommend you make alternative plans or some other back up solution if this ever happens again. But again, you know him best..how likely is it to happen again?