My son was the same in that he would not go to sleep without me first rocking him when he was a baby, then when he was almost 2 we got it down to just sitting with him until he fell asleep. This was incredibly draining as I would sometimes sit in his room for an hour and a half before he fell asleep and if I got up too soon, he was wide awake and screaming hysterically until I came back, which is, as you say, really difficult to take. There's crying it out, then there are the kids who scream hysterically. Unless you have one of those, you have no idea what it's like and how it completely frazzles the nerves. Once when I was staying with a friend he was chiding me on how we really needed to let our son cry it out (his three kids all went down no problem). So I said, sure, hey, if you don't think your kids will be bothered by it, I'll let him cry. Well, within 30 seconds, my friend was saying, "he sounds like something's wrong up there, don't you think you should go get him?" When I explained that no, this was just the way our son "cried it out", he agreed that maybe his kids would be bothered by it after all... Hmmm... Anyway, needless to say we tried everything. But then I got pregnant with twins and knew I wouldn't have the time to sit with him for hours on end to get him to sleep. As it was, just before hte twins were born, when he was 2 and a half, he gave up napping altogether so we instituted quiet time, which meant that he didn't have to sleep, but he had to stay in his room and sometimes I would even put on a CD for him to listen to during quiet time. Before that, however, we made a sticker chart and gave him a sticker for each time he went to sleep alone. After 10 stickers, he got a new toy (something small for under $3). Anyway, we only reached the 10 stickers three times before he got the hang of it and started going to bed on his own without needing me. Fortunately he never regressed when the babies were born either. Also, another thing that really helped us was getting him a big-boy bed. I got the feeling he always felt confined in his crib and with this new bed he was totally happy and content to stay in bed at night. Fortunately, too, he was not one to get out of bed over and over again. I think he just liked the idea that he could get out if he needed to.
You also mentioned that you are trying to wean him from needing milk or nursing at bedtime (can't remember which). I had a friend who was still nursing her 2 and a half year old daughter every two hours at night. When she asked for advice I told her to use the same principles you would if your child was asking for chocolate bars. Just say no. You wouldn't give your son chocolate before bed, so.... Of course you don't just say no, but something like "we don't drink mommy's milk before bed anymore..."
Anyway, good luck with it all! It can be very trying, I know!