G.M.
If he is a good man, a good provider and he isn't a drunk or a druggy, then I'd say he's a keeper and you start wising up instead of panicing. Try planning or scheduling "Daddy and baby" time. Make it a point to one day during the week all three of you do something together, like take him to a playground or the zoo, even though he is little those are meaningful experiences. Then, make time for yourself. One evening out of the week, tell him you need to go see your friend that is having "issues" or something and leave him with the baby. You have got to plan these times. If you start doing these things, then you won't resent him so much for the guy stuff he does alone. And of course, you don't have to tell hubby you've orchestrated the outings.
Don't be afraid to tell him that you need him. He needs to know that he is so much more important and needed than he can imagine. After a few of your outings tell him, "I am so glad you were able to keep him! I don't know how single moms do it without a husband!" Make him feel like he is your partner now.
Also, go to church together and pray together every night. Give him a hint with the prayer...say something like, "Dear God, please help us to be good parents, guide our words and our actions that our son will grow see that he is fruit of our love for each other and for you. Let everything we do be for the good of our family and to serve you."
Sorry about rambling, hope this helps.