T.T.
I know it's scary, but he will be fine!! I was scared to take my 1st son out, but I dragged my 2nd everywhere (gym classes, stores, playdates) at about a week and he did just fine. Just don't let people too close, especially small children, but that is easily remedied with a light blanket thrown over the carrier/stroller. The first time out is always the hardest (on mom that is)!!
You deserve to get out, go, and enjoy!!