No. You don't let it go. You have a talk with the friend who said this and tell him that when he is at your house, YOU make the rules and YOU decide who plays at your house. Not him.
You tell him that what he should have done is come to you and talked to you about his problem with B. Then you could have decided what to do about it. Perhaps you could have mediated between the two.
Have you talked to your own son about what to do about B? I think maybe you two discuss what happened and tell him that the boy was in the wrong, and it totally caught you off guard. Ask him what the problem between them is. If indeed B is being mean, this is a wakeup call for him. If he's not, and this is a bullying tactic by the other boy, mediating between them will probably bear some of it out.
You missed the opportunity to put this kid in his place today, (I'm sure you were just caught flat-footed here), but that doesn't mean that you don't address it. The next time your son plays with him, have him tell the boy that you want to talk to him, and then you talk to him and tell him that he may not run someone off from your house.
You need to get to the bottom of this so that this kid doesn't bully B every time he sees him with your son.