You're doing a great job! First off, don't expect anything from your son's father. I have been divorced for a year now, and have 2 wonderful daughters with my ex. He sees them every other weekend, and nothing else and does not make any further effort. He's fine with it, he does talk to them throughout the week as well, but they don't talk much, just hi, how are you and that's about it. I do have to say that when we were married, he was in the garage working on his cars, I was in the house taking care of the girls. I put them to bed, when he came in (well after they were asleep) he would go up stairs and say good night. Secondly, I have noticed that his parenting habits come straight from his parents. They are farmers, the dad was always outside, either doing chores, working in the field, or doing maintenance on something with the farm. His mom was always in the house...so, that's where he got it from, and I have to say so did his brother (they are divorced as well.)
Just make sure and reinforce to your son that 'daddy loves you', no matter what...At first, they didn't even like going there, but after me talking to him about it (they didn't like sitting in the house with him in the garage), it did get better, although they do spend more time at the neighbors playing with their kids or at his parents farm, but at least it's his time with them, and if that's how he chooses to spend it instead of bonding with them, that's his choice.
Good Luck!!