I'm with Cheryl O, let them swim anyway! My kids will swim in any weather, rain or shine. We do not have a pool and don't go swimming very often so it's a big treat for them to go swimming. The same might be true for some of the guests.
If you don't go that route pull out all your toys and art supplies... They will find a way to entertain themselves with very little nudging. I'm always amazed at how creative and imaginative kids are when you get a bunch of them together. I bet this is going to be a great party! Have fun and don't stress on it!!