I feel your pain!
First, talk to your DH about it. He needs to be on board with the decision (and frankly, he needs to be more understanding of you/the demands on you, and maybe he doesn't realize there is an issue). He may say it's no big deal, but let him know that you can't, in essence, take part of 2 days every week away from work--you have to work... so if he's okay with them coming over, that means that you're going to work, and he's going to entertain them & take care of the baby, since you need to work. (Try not to make it confrontational, though).
And if that helps him realize there's an issue (which it should, hopefully), then sit down with him & come up with some recommendations/suggestions with him on what is appropriate....
If they are retired, they need to be understanding that you are not and you have to meet the basic obligations of work/home/family-meaning you, DH & baby- (not necessarily in that order), before anything else, including visits. Tell them, since their schedule is flexible, because they are retired, it would be greatlly appreciated if they came when it was better for you. You could even spin it this way: You want to spend time visiting, but when they try to visit during the day, you are focused on your work, and so cannot enjoy the visits and it is impacting your work negatively, and that this is a problem.
Good luck!