R.:
My boys hang out with MAYBE 4 kids from their school. That's it. We go to after-school activities (LEGO Club, Science, etc.) but overall - we DO NOT hang out with the other families....only a select few.
You are FINE and in MY opinion this is NORMAL. They are not missing out on friends they - if they truly connected with friends from school - they would ask if they could come over...
You might find things changing in middle and high school. But then - you really need to know the parents and the kids - in MY OPINION so they stay on the straight and narrow!