L.B.
Continue enjoying the cat. We have a very social cat that the neighbors enjoy and we don't mind that our cat visits them. I would be upset if my cat was bothering another neighbor, but not if he is welcome and he is bringing them joy and play. The cat seems to only go to the neighbors that like him and has not intruded on the neighbors that don't. My neighbors are pretty outspoken so if our cat was bothering anyone, I am sure I would hear about it.
If you don't like the cat hanging out, you can make a loud noise or gently squirt him with a squirt gun when he comes to your yard he will eventually get the message.