S.V.
Hi. It sounds like you are asking all of the right questions and are in tune with what is going on with your son. While there still may be something else on his mind, he may not be ready or know how to address it.
In thinking about what I would do when my son was a kind, confident 9 year old, I realized what my approach would be (and still would be even at his 16 years). These are my thoughts solely based on experience with sleep problems.
A question for you...why do you think it would be bad to soothe him to sleep every night? I know you don't want to make it a habit, and at 9 he should be able to go to bed by himself. But, something is getting to him and it may help him if you are there for him -- at least for a few nights to see what becomes of the issue. Sometimes, when kids are in bed and feeling safe in the moment, they tell you things they may not bring up at other times. Or they ask questions they may feel silly or embarrassed asking. Or maybe with a few nights of good sleep it will help him open up to what is wrong, or to realize he's chasing imaginary monsters because he's so tired. Either way, he needs to feel safe until he gets past whatever is going through his mind.
Maybe you could read something to him that is light/ soothing, or have him read to you. Or, if you are so inclined, you could teach him a bit of meditation or say prayers. Have a fountain trickling in the background while he is trying to go to sleep, or a CD with nature sounds -- whatever he finds relaxing and will let his mind rest.
The other issue could be that he is going through emotional growing pains. Maybe kids his age are changing personalities and he's unsure where he fits in. Or he likes a girl for the first time and not sure what that means. Or school is harder (4th grade -- assuming he is in 4th grade -- can be a tough year as more is expected of the kids)
Don't discount that there might be something else wrong, but look for the "easy" stuff first. It might be a better approach for your son.
Good luck.