Looks like you need to have a strict parenting plan. One that is simple and clear. He gets him this many days, and so an so picks him up and drops him off. Also you need to talk to your lawyer about if there is a next court hearing that you should have so many days notice in advance, and also wouldnt hurt to do research on your rights. You shouldnt have to put up with all this, especially when you are being nice, and civil as his mother. You are working hard to keep the peace and it is getting you nowhere. He thinks he can take advantage of you probably to see how far he can push you and how much he can get away with. Sounds like Grandma is doing the same by being late all the time too, or sending someone else to do her job of picking him up. It isnt fair to keep you waiting and it isnt fair to your son either.
Think its time you put your foot down and stopped being so nice. Write down what you expect from this court hearing, what you want, and what you will not put up with. What you will accept as well. If you want him to have a certain amount of days a week, or dont want someone seeing him, or him to be home a certain time each time then state so and dont be so quick to accept anything just because it might be easier. Dont be manipulated by him or his family. Do whats best for you and for your son.
Good luck and hopefully everything works out for you.