K.,
It sounds like you've been having a hard time right now. You can't enforce visitation like you can child support so even if your ex has visitation priviledges and doesn't use them well that's his loss.
Don't look at it as you're letting your oldest boy down because 2 men have abandoned him. Have lovely weekends with him on your own.
When the youngest get aggressive you have to step in. A clear rule like NO HITTING KICKING OR SPITTING should be posted in the house and if he comes in and does these things then he needs to be punished. Going to his father's is a priviledge you can take away from him and you must do that. Call your ex and tell him that's the rule and this will be the punishment. If your ex is encouraging this in anyway he'll protest and threaten so you simply tell him he has to set a good example and he also has to counsel his child to keep his hands, feet and spit to himself. Otherwise you can simply have his visitation suspended. You don't have to have proof of any inpropriety to have it suspended you just have to petition for it. It's up to him to prove him self innocent of inciting abusive behavior.
Now on the other hand, you also have to make sure that your older boy isn't making comments that would cause him to be thumped on. Jealous comments often provoke physical aggression. Teach him the 4 rules I gave you in the other post.
I think a little family counseling will go a long way as well. See if there's a Family and Children's Services office or clinic near you. They offer sliding scale sessions and Soonercare will often pick up the tab if you have it.
One last thought. Make sure you spend 20 minutes a day alone with each of the boys, no tv, not in the car, just you and he talking or playing--even reading or going over home work. Then when your ex says something remind him that without his financial support you have to work 3 times as hard to make the same money then tell him to kiss off.