First of all, there is not, nor will there ever be a state free of side effects in this situation. Children are very sensitive to changes in routines, environments, rules, surroundings, etc. It is just an unfortunate fact of join custody that no matter what you do, the kids are going to spend a few days around every transition testing boundaries, breaking rules, feeling out of sorts, and otherwise demonstrating symptoms of stress. They do the same thing when they visit your ex, too.
It is part of how children cope with this difficult situation. That said, though, there are things you can do to make it less difficult and more successful. Most of those things revolve around making sure that as much as humanly possible in their lives if familiar and routine. Making sure they have their favorite pjs, favorite toy, favorite rituals related to bed times, etc all happening with absolute consistency will make it easier for the kids to predict their lives and feel less out of sorts.
As another poster said, you also need to find a way to not be baited by the ex and express some boundaries that you stick to even when that is uncomfortable, without being willing to fight with him.
If it is your preference that you be present for a lot of his parenting time, then you need to find ways to distract yourself and refuse to take him personally during that time, reminding yourself that even if he is being a jerk in some way, your reason for being there is for your kids, and you don't have to take his bait. Let him play mom too if he wants to. Sit and read a book and ignore him unless he crosses an actual health or safety line of some kind.
Make sure that he has no key to your home, and get the caller ID suggested. If you have trouble negotiating without fighting with him, pick a time when he won't be home to leave him a message with the days and times he's welcome to come by and a statement that your home will not be available to him at other than those times, as you are currently attempting to stabilize the routine for the children.
As another person said, it would be best if you were adhering to a court appointed routine of some kind. If the initial arrangement that went with your divorce wasn't clear, or sufficient, get it rectified either through lawyers or mediators so that you have something concrete to point to as the rules of visitation.
It will make it easier to stand up for your boundaries as well as leaving him no room to say that legally speaking, you are being unfair in any way.
But at the end of the day, you need to find the strength inside yourself to fail to be baited, to busy yourself with other things in the vicinity of your kids, without engaging him, and without taking it personally that he's being a doofus. He's testing your boundaries as much as the kids are. He is trying to find all the places he can walk right through them, either to get a reaction, to get back in your life, or to prove to himself that he's a good father.
Whatever his reasons, positive or negative, you need to firm up your boundaries while not being willing to fight. Don't take his calls. Don't answer the door when it's not his day to visit. If you have to, threaten with calling the police as calmly and respectfully as you can if he's turned down your insistence that it is time he leave for the night. Just say "Bob, I need you to leave. It is their bed time and we are working to establish consistent routines." If he blows it off, calmly say "Bob, I won't ask again. It is time for you to go. I would prefer not to need to do so, but if necessary, I will call the police to enforce this boundary in our lives. I would rather us handle this in a friendly way that respects eachother's boundaries, but I will do whatever I have to either way. Please go now."
Usually it only takes a very small handful of times of proving to his brain that you're serious, to get him to take your requests and boundaries more seriously. But you have to refuse to debate them or back down from them in order to make it work. Don't justify yourself or your actions beyond the single statement of intent to establish routines or whatever it is. He has a right to understand why. He does not have a right to argue with it if the limits you're setting are within the court designated rules between you.
If you argue with him, no matter how extremely irritating he's being, he wins because he's asserting power over you by getting you to fight. This is part of why it is so critical that you not fight, just defend your boundaries calmly. Don't give him the impression that he has power over you by arguing about anything with him, and he will tend to test what other power he might assert, less, which will also make other things easier.
Unless you are really worried about their health or safety, under most circumstances, you're better off not trying to share time with the ex and the kids. All those reasons you got a divorce are still there. Yes, the kids will have some insane transition time, which will be frustrating. But it will exist no matter how you handle this problem. And it would be less stressful for everyone for there not to be tension between the two of you through the whole visit. And it helps to clarify for the kids which rules apply where and when, so they feel a little less helplessly adrift on a sea of change.