You need a routine and you need to stick to it like glue.
You and he are interacting far too much, in too many different ways, for too long each night at bedtime. You mention that you and he listen to music, read, talk, watch the fish -- it sounds as if you have no real routine but are just hanging out in his room until he's so tired he can't help but go to sleep or until you just have to leave, and then he's upset. This is why he's up so late -- there is no "bedtime" but a lot of mom time, and that mom time gives him all your very focused attention.
He needs to learn that bedtime happens every single night exactly the same way, so he knows what to expect; knowing what to expect makes kids feel more confident and less insecure. He also needs to understand that begging you to stay up and "refusing" to go back to bed simply Do Not Happen. I know that fear is part of this, and you need to realize that this is going to take longer to handle because he has slept with you for so long but he needs to start sleeping IN his room and being treated more firmly -- which is appropriate for his age -- at bedtime.
I would talk to him in the daytime when things are calm and happy (not when he is upset about anything else, not when others are around, etc.) and talk about how he's a big boy and tonight you're both going to start a new routine at bedtime. Then do what you will start doing every single night at the same time -- bath, teeth, one short book that you read to him, lights out. Do not let the process drag on; do not engage in conversations that are about anything other than "Now it's time to put on the night light and when I put it on, I am going to be next door" etc.
He will call. He will cry . He will get up. When he does, walk him immediately back to his bed but do not talk with him! Do not engage him. The first time he gets up, you can say, "It's after bedtime. Time to be in bed." Then walk him back to bed, cover him and leave immediately. He has to learn that getting up, trying to interact with you gets him nothing he wants -- you must not interact; plead with him to go to bed; explain to him why he must go to bed; scold him if you're tired; or anything. Be boring, boring, boring when he gets up or calls.
It sounds as if he truly does have some fears and you want him to know you are there for him, but you also have an issue here with his getting a lot of attention at bedtime and even more attention when he refuses to go to bed; you have to cut off that attention by returning him to bed over and over. It could be dozens of times in one night, frankly. Can you do that or will you cave in and he'll end up in your bed?
Check out books by Jo Frost, the "supernanny" from TV. She does a lot with the idea of returning kids to bed and not interacting when they get up in the night.
Not interacting is not the same thing as denying his fears or making him feel alone if he's truly scared. You have a tough situation here because it's hard to tell how much is real fear and how much is the fact he's learned to manipulate you at bedtime by saying he's scared, knowing that gets your attention. You already know this, from what you say in your post. Returning him to bed lets him know you are physically there but takes away the interaction and attention he craves.
This will take a long time and you need to be really strong. You also have to take care not to let your own childhood make you give in to him -- you mention your own fears as a child and difficulty being away from your parents, but you are not "away" from him -- only in the next room. Both you and he need to focus on that.