Now that summer vacation has begun for many people, this concern may just resolve itself. I would definitely try to (somewhat) alternate homes and I would also encourage your son to make other friends, as well, especially since he has others in the subdivision who are available. Get your son into the "call first" policy too. At that age, my children did call their friends, even though they were just across the street and next door. Now that my kid's are older, they just go from house to house but they need to keep me posted on where to find them, otherwise they are in trouble.
Your son is still very young and I think that many kids at that age do have a tendency to have that "one friend" who is special. Unfortunately, at some point, one of them will branch off and the other will get their feelings hurt because the other is leaving them out or one is feeling a bit smothered or controlled. They are just learning the fundamentals of friendships. At that age, I often found that there were those "friends" who only wanted to play at their homes - either because of comfort or they had toys that were bigger and more luring. Do your best to keep that balance of having fun things to play with, without trying to "keep up with the Jone's". That becomes a bad precedence to set. I see it all the time, where I live - I've always done my best not to play that game. Yet, I still have my children's friends who enjoy coming to our home, even though we don't have the biggest and best toys!
Get your son involved in other programs, as well. Library programs are usually free and it keeps him reading through the summer. Park district programs sometimes are a nominal fee and, perhaps, through your park district, they may offer a "picnic in the park" type program.
Also, don't forget about some old "school pals" that your son had. Those moms may be grateful when you call to invite them over, especially by July, when kids start looking for new faces to play with.
Good luck.