Yep, as others said: ONE weekend a month for each set.
You will soon burn out on them and on the traveling. Better to see them when you are all fresh and happy about it, rather than to do it tired and out of a sense of duty.
Are you an SAHM? Then your husband needs weekend time with his kids and you, as a family unit, by yourselves. This is how you really learn to be together, all of you, not just mom-with-the kids or dad-with-the kids. If you are not an SAHM and work outside the home, the weekends are even more important so the kids can see you and your husband, together, for substantial time. Having other people always around -- even loving grandparents -- alters the family dynamic. When with your parents, you and your husband are always going to be the adult children, and that affects your interactions with your own children at those times; when it's the two of you and your kids without grandparents, your interactions with your kids are not being affected by worrying about what MIL will say, or how grandma is feeling today, or whether grandpa should take one child somewhere and not the other. It is great that you have both sets of grandparents but you also must find your footing as a family independent of them.
Also, a reality check: How old are your kids? I'm figuring young, maybe younger than elementary age? Once the kids really get going in elementary school, if they have any activities at all, they are going to have weekend activities: Homework and school projects; play dates with friends they adore; sports, Scout events, dance classes, swimming lessons, music lessons, whatever. The grandparents will suddenly find "their" weekend is gone because "Sally has a recital Saturday and Jimmy has a lesson Sunday." If the grandparents start coming to those events, great, but don't let them take over and move in every weekend.
I would love to have grandparents close by but my folks are dead and my husband's are 6,000 miles away and very infirm. So treasure the grandparents but stop worrying about "balancing time." Cut back and carve out family time for YOUR nuclear family too. It will be hard to explain at first, but tell the folks you are simply busy -- and then BE busy.