My opinion....babies are meant to be held. There is a reason babies cry less when they are held - they have been held for 9 months in our wombs - moving around, hearing our heartbeats, hearing our voices. They would naturally feel more comfortable and secure since they have only know that.
Not to mention the fact that your son was born early & not quite ready to come out!!! That adds to his stress!
Maybe it is the type of carrier you are using? I used a pouch type sling made by New Native Baby (www.newnativebaby.com). All three of my children were so happy when they were in the sling with me. The sling type of carriers allow the baby to be curled up in the fetal position and next to you - they aren't sitting straight up w/ their legs dangling making it difficult for you to bend down or sit down without smooshing their legs.
Yes, I do realize that means the baby is still being held - but your hands are free and you can still have 100% mobility and he is still where he wants to be, snuggled up next to his mommy.
Good luck and love that little guy! Children are blessings.
I 100% agree with Alexandria - you should read her advice also about another type of sling and www.askdrsears.com