I have relatives who have various levels of mental retardation and regardless of abilities (be they functioning at 6 yrs old or 10), they can learn basics like no hitting. Is it that he's being a toddler and not able to make the leap to appropriate behavior or is it that they don't rein him in when he does this, using the disability as an excuse? I would talk to the sitter first, since you saw it in her care. I would express your concerns, calmly, and ask her how she plans to resolve it such that your child is safe and not being bullied or harmed while with this other child. Is the child in any early intervention program? It may benefit him if he is not, and benefit the sitter, too (and by association, your child). It also matters if he was trying to "play" or if he was being deliberately mean.
We turned down a sitter sharing situation because the older boy (whose only problem is that he's a brat) was mean toward his sister. I didn't want my baby in the middle of that. Just saying that you have to evaluate the situation and make the right choice for your child. If it can be resolved, great! If not, then at least you tried but you need to look out for your kid.
Edit, because I thought the child was smacking your child and no one was doing anything. If YOUR child is smacking his father, then that's something that also needs to be addressed. My daughter (as many toddlers do) went through a hitting stage and it had nothing to do with other children. Not that your child should be hit or have toys thrown at him, but I think it may ALSO be that your child has hit this developmental stage.