I haven't read the other responses, but I might have a different perspective. I think what's going on with your son is totally within the "normal" range. Despite what society says, being a child isn't carefree, and it isn't easy. There's a lot going on in their lives, and a lot going through their little brains. My two older kids, age 4 and 6 1/2, share a bed. They didn't want separate rooms, and share a bed even though they have bunkbeds! My son, who is 6 1/2 never slept well until he started sharing with his sister--he would wake up yelling and screaming, calling for us, etc. I'm sure that will happen again as we're about to separate them at night because my daughter keeps him up late with all her chatting! Before they shared rooms, like you, I felt like we had tried everything--calming music, a nightlight, stuffed animals. The truth is, I think there's truly no replacement for a human being, lol.
So what worked BEST for us? A consistent bedtime routine every night--bed about the same time, and the same order of events...bath (which definitely helps to calm), pjs, story, prayer, bed, etc. Giving him Calms Forte as needed helps. I bought Badger sleep balm, and rub some on his temples before sleep--it's "sweet dreams balm." Does he still ocassionally wake up at night? Yes--especially during times of change or stress.
But I disagree with the idea that he must stay in his own bed every night. If that's what you want, then that's different. But it's completely normal for kids to want and/or need a parent during the night from time to time. My kids know they are welcome in our bed--they need to go to sleep in their own beds, but are allowed to come in during the night. They will go through periods of coming in during the night for a few nights in a row. But other than that, they don't. I think just knowing they can may help them to sleep better.
If you don't want him in your bed, you could also compromise by maybe allowing him to sleep in a sleeping bag on the floor of your room?
Good luck. Hope you all get some rest.
P.S. One other thing that seems to help is, last thing before saying goodnight to him, is asking him what his favorite thing was that day, or the thing he's most looking forward to tomorrow. The idea is to get his brain in a positive mode.