This needs to be a transition. This is not going to happen even after 12 visits with father that includes the grandparents spending time all 4 of them together.
This father may need to visit with a family counselor to get an idea of how a child's mind works.
Father will need to be mature enough to understand this is not about HIS needs or feelings, but what his son needs.
It is what it is. This is not "normal" and so it will totally depend on what son feels comfortable with.
Your job is to support your son, but not hinder him.
Try to remember the good things about dad. You must have liked him at one point and for a reason.
Also remember, any bad things people say about dad, can strike to the heart of the child. There is a lot of guilt children carry because of parents behaviors. We know we are supposed to love them, but when we do not, it makes us feel like bad people.
Also in this case, your son can feel a great obligation to you with his feelings. He will fight to NOT like or love his father as much as you, but if this were to happen, he needs to be reassured it is ok with you. You want him to have 2 healthy parents in his life, if possible. Let him know people really do have enough love for lots of different people.
I adore my mother she is my hero. She stood up for us and took a lot of grief on our part from my father. He honestly thought she was the one that was stepping in.. or making up our reasons for not wanting to spend time with our dad and his girlfriends or wife.
The true problem was that my father was VERY immature and had no clue how frightened we were of him.
Family counseling helped me and my sister lose some of our guilt about how we felt about our father. And over time he realized how he was immature, selfish.... and an Alcoholic.
Encourage your son as much as possible. Try to let dad be the dad he is. Try not to interfere if possible, But also listen without assumption to what your son is expressing.
In our case, my mom could really drag out the negative in my father. When in reality, I look back and wish she had listened, but not continued to speak poorly of him in front of us or to others while we were around.
But I can totally understand her resentments, he was a real piece of work, but he was also just a product of his own childhood. It took me having a child to realize this.