You can't force him to take the kids. If you even tried you would send a message to the kids that you wanted your time off whether they wanted to go or not. That makes you look like you resent the ex getting the freedom he has as a noncustodial parent. It's not freedom. I couldn't take my oldest one weekend because I had to work. My ex insisted and when I showed up in my uniform with my firearms in the truck he freaked. I told him I would be just getting off work but he was still upset and had my visitation revoked. He said my possession of firearms were threatening to him.
Things at dad's house may not be that great. But it's typical for children to blackmail you by saying theywant to go live with "dad" when they don't get their way. I did it to my mother, my oldest did it to his dad. I know you might like the ease in responsibility but you signed up for it when you concieved the children. Imagine what it would be like if their dad had died instead of y'all getting a divorce. You would be the only parent then too.
Lastly, take the kids to see their uncle. This could be the last time any of you see him. You can't be that selfish as to say I want to party with him instead of sharing the experience with your kids. Again think about what it would be like if you were a widow instead of being a divorced mom. You would take the kids then wouldn't you. No hesitation.
Lastly child support and visitations don't go hand in hand. If he doesn't pay then by all means take him to court but if he's just not visiting the lap up all the time you have with the boys now cause in a few years they aren't gonna want to hang out with either of you and it will be his loss not yours.