First of all, I would suggest having a sit down with the grandmother and your exhusband. I realize this may not work since they may see you as just being a mean mom. However, if you child is not eating any good food, it will really hurt him over time. However, I have had my girls doctors tell me that sending them to bed...or away from the table won't hurt them. They may hold out for a couple of days even! But, if you offer them only the good stuff, then when they are hungry enough, they'll eat it. And, I wouldn't start out with something like brussel sprouts! LOL Start with something that kids normally like....such as Mac and Cheese or grilled cheese sandwich or even all beef hotdog! Don't keep the junk food in the house. Then, let him go through and pick what he wants to eat.
My nephew is a good sized 3 year old, but he eats very little. however, when he does, it's usually bacon or some kind of meat. The doctor has said, as long as he isn't replacing his eating with just pure sugar to let him eat what he wants. The problem is that your son may not be getting ANYthing beneficial. So if he does go to eating less, that's ok...just make sure he's getting some good stuff, too.
Hope this helps. I'm sure it'll always be a struggle as long as he's spending time elsewhere, but be patient. You are doing great!!!