I don't have the perfect answer, but I have been going through the same thing with my 4 and a half year old daughter. She too doesn't sooth very well and never has. I started with a sleep specialist, we found out that she had sleep apnea, after taking her tonsils and adnoids out, she is still waking up in the middle of the night. I would start off with the battle by making him fall asleep in his own room, not in the living room. I know you are probably thinking, "this lady has no idea how much he screems" let me assure you I do, my daughter is and was the same way. I am a single mom so it is more difficult for me, as I have no one to trade off with. Between you and your husband, take turns putting him back in bed, as many times as it takes, it is very hard to do, as you get tired of hearing it, and just want him to fall asleep, but after a few days of knowing you won't give up, he will learn to fall asleep in his own bed. As for staying in bed, I will let you know when I get that one figured out, as I said, my daughter at 4 still does it. I hope this helps! I know how awful it is when your not getting the right amount of sleep either!!!!I have been doing it since the day my daughter was born. Good Luck!!!
K.