Okay, let's be real about this. It's not that he really wants his father, it's that he knows saying "I don't want to be with you, I want to be at my daddy's house." causes your guts to twist and I'm sure it shows on your face. Children use emotional blackmail better than mistresses. So the trick is to know how to avoid reacting to it. This website gives you the motives behind bad behavior: http://lifematters.com/step.asp
So you need to stop it. You can offer choices BEFORE he gets to the stage where he wants to wound you with words.
You can choose to not follow instructions, throw that tempertantrum, continue the bad behavior and end up losing something
OR
you can choose to do something elst like picking out a new video, watch tv, play with a new toy, get something back (we'll get to this later)
and if he refuses to choose, then you choose for him and he's going to end up losing something. Start with his favorite things/activities first and work your way down to stripping (one item at a time) his room then leaving him in an empty room just a bed and other furniture too heavy to remove as the last resort --think, losing his freedom.
AND if he's lost his toys then he can't play with anyone elses in the house that means staying out of sibling's rooms. No tv includes watching with you in the evenings. Staying in his room means he gets none of the treats he would normally have in an evening.
Once he's familiar with choices then you can say: "Your father chose to leave and he chooses to stay away. It's not a reflection on you or me. He just made a bad choice and it cost him us."
I know it's hard. I know that you're probably dealing with a huge amount of emotions with this but it's very important to be stone cold at this point. Poker Face MOM. No emotion. This takes practice, you won't get it right the first time.
Now are you setting 20 minutes aside everynight to read to him? Do that everynight from now on. Check out books from the library, write some yourself on construction paper, or ask his teacher to send a few from the school library home. This time will only be for you two even if younger children or older come along.
Here's how we do it here. They 3 year old starts his bath at 8, then dressing for bed, and books until 8:30 then he's off to bed. Then the 6 year old gets into the bath, 5 minutes, dressing Then books for 20 minutes.
If you need another positive pre bed time activity, send me a message and I'll outline PCIT to rebuild your fracturing relationship.