A.L.
Ditto to what the others said. I also agree that when your son is visiting with his father, talking with you every 3-4 days is appropriate. If he's talking to you every day (or more than once a day, as I get the impression you might want), he has no chance to adjust to being with his father (and without you), and that defeats the point of the visits. I really don't think he is 'way out of line' in this case.
There seem to be several things complicating the situation. You seem to feel the father doesn't really WANT to spend time with his son (why else did you include the information about him cancelling weekend visits and him asking for visitation when the boy was 5?). You might do well to assume, however, that the dad really DOES want to spend time with him because he did make the substantial effort to establish visitation after 5 years. As well, you seem to feel like your son doesn't even want to be with his father and you aren't really comfortable with him being there either, it seems, or you wouldn't feel so upset about talking to him every few days. Is it possible that your son's reluctance to go is at least partly him picking up on your reluctance? Just a thought to consider.
I do understand that it is hard not to be with your child when you are used to having them there every night. I have been there--not being able to talk to my kids or see them for weeks at a time for my work. However, I do think those periods of absence are really valuable for everyone. Assuming that your son's dad is not mistreating your son when they are together, it might be helpful if you can shift to a mental gear of focusing on what your son can get from having his father in his life and involved with him to some degree. What steps could you take to maximize that benefit and make your son's adjustment easier? Good luck with it!