Well, you are making bedtime a battle and he is taking you up on the offer. Turn the lights down, help him go to bed, he needs the one on one time to transition and fall asleep. He cannot be expected to fall asleep on command.
Go in, sit on the bed, read a couple of books, have a routine, let him brush his teeth, get a drink, etc...then sit with him until he is out. Then you can have your time. Help him go to sleep and stop making it a battle.
When my daughter was younger I would have battles like this. I found that if I turned off the lights, put on some soft music, read to her, just acted as if I was going to sleep too, then she would be out quickly and I could do whatever I wanted.
When you say you "have" to close the door and he is scared that tells me it might be a battle but it is one that will backfire on you. If he is even thinking about being scared and is telling you he is then he is internalizing that and it is torture for him to be in there with the door closed. Why do that to him just so you can have an hour alone. Take the time to help him transition and go to sleep and you'll have a much happier child.