L.S.
Try Mister Clean Magic Erasers...I LOVE THOSE! You can use them to clean anything except for dark colored painted walls.
We have a Bumbo and since they are sealed foam, you should be able to use the magic eraser on it without damaging it. Try a small section first to see if it works.
I use those all over my house to clean, finger prints off light color painted walls, tile grout, glass shower doors (won't scratch), bathrooms, etc.
Good luck