If your neighbors really didn't want to play with your child, they probably wouldn't play in their front yard, right? They know that playing outside is a signal to your kid that they are available.
As far as leaving your kid, I think I would have thought an invitation to your child to come in would have also been an invitation to you as well. When similar things happen with my child, I always expect that I'm invited too. My son is 4, though. Not sure how old yours is.
Regardless, I have a very similar situation with our neighbors, although mine do seem marginally friendlier than yours. But it's tough when all the invitations and overtures are one-sided. And, like your child, my son really likes this boy, but I feel awkward always being the one to knock on their door. Consequently, I've had to tell my son "no" several times when he says that he wants to play with this child, because I too don't always feel super-welcome.
All that being said, when we do get together with them, they are usually pretty friendly. They just kind of give off the vibe that they can take us or leave us, and it's not the most comfortable feeling. I'd say, be friendly, but encourage your child to find other friends, as well.