Let them all just sit and think what they're going to think. It sounds like this first couple in question has some problems within their own relationship that radiate outward and everyone else will either figure it out soon enough on their own-- or not. That's their business.Now, if you mean that the neighbors have gossiped about your kids not playing with their own kids, then you can say "Oh, we are busy but we'd love an invitation" and leave it at that. Put the ball in their court and see what happens.
Neighbors are neighbors, if you know what I mean. They aren't your best friends, it's not you telling your family or friends that your kids won't play with theirs. IF they are gossiping about this--well, that's just stupid. If they want the kids to play together badly enough to fuss about it, they can extend an invitation for goodness sake.
For what it's worth, our son doesn't play with all the neighbor kids on our street, just a few. He has plenty of friends and not so much in common with some of the neighbor kids. I'd be surprised if any of our neighbors gave it much thought.