It sounds like you were not in a relationship with your son's father very long before you became pregnant - so of course you don't know him very well. For you child's sake you may want to try to get to know him better if just to ensure you know what type of man he is and if he will be good influence on your baby. You are not obligated to continue a romantic relationship with the man if that is not what you want. Forging something out of nothing just because you have a child with him, or just because other people say you should is not reason enough.
A couple of ways to make your interactions with him easier:
(1) Instead of trying to come to a child support agreement on your on - go to court. That way child support, visitation and a host of other benefits (like medical care and school clothes) can be firmly written up. That will eliminate one stress in the relationship.
(2) If your child is old enough go ahead and let him visit with the father without you having to be there. If the child is too little is there another adult you trust to be at the visits? His mom, sister, Aunt, etc.
(3) If you feel you must stay for the visit leave your purse and phone in your locked car - if he can't get to them he can't go through them. Bring a book, curl up on a chair in the corner and read while he and the child visit together. If he brings up a topic you are not comfortable talking about, get up, and go outside. If he follows you, take your son and leave and tell him you will be back once he understands that these visits are not about the two of you but about him getting to know his son.
Good luck to you, it is hard but you sound like you have your son's best interests at heart. Just love your baby and live your life well.
==============================
Oh, and to Vicki E. - that comment was uncalled for - that was just cruel and pointless.