Listen I will be blunt since you are still upset about this. There are plenty of people in my church that I really like BUT I would never want to *hang* out with them after church. Just because you go to church with people, doesn't necessarily mean that they want to be one big happy family, in fact there is NO such thing as one big happy family, even in Gods home.
A senior retreat is not meant for people who are not seniors in the congregation. Don't you have to hold some type of title to go on these retreats? at least in our church you do. Now if you really want to go on theses retreats just ask, "what do I have to do to be able to partake in the senior retreats". Or "I love working in the nursery but I think it's time I branched out so I can interact with the church body, is there another job I can do?".
Also your approach in talking to this woman was confrontational and a bit attacking. Sit down with the pastor and ask questions. Ask for ways to become part of this *family*. The other thing is you said the Pastors kids are in their 40's is that correct? would your child have been the only young kid their?
Otherwise you just have to make a decision whether this Church is for you or not. You haven't been there very long at all, a few months is still very very new. If you haven't felt *good* there then maybe it's time to look for another church family. Make sure you fit in, if you don't, then it's uncomfortable for everyone, not just you.
My husband and I took 2 years to find a church we felt good in. We didn't want to settle. Just because it's a church doesn't mean that you are going to feel welcomed. Humans are not perfect and you will get judged, have to deal with cliques, and just not fit in for whatever reason and you cannot force it, but you can leave. I hope you find peace, blessings.