I have a strained relationship with 2 of my 3 brothers. And at times, it has been different brothers. I'm the one who left for college, moved out of state for the first 5 years of my marriage, moved back home when mom was sick (which caused even more issues), moved out of state for another 6 years, and am now back "home" (although 2 hours from them--by choice).
I don't have the same things in common with my brothers or their wives. I live life and parent differently. I'm the odd ball. I'm good with that.
What I've learned to do is keep things on my terms. I visit the family members that I am closest with and when it feels like it's getting too much, we pull back a little and become too busy to visit. With the other ones, I still send bday gifts for the nieces and nephews, say happy bday over the phone and even on FB to the one I can't stand.
I still show that I care (because I do) but keep things comfortable for us.
My advice about the new baby? Option 1. Send a gift to the baby and your sister. Simply congratulate them. Option 2. Fly out there, stay at a hotel, visit for short bit, give a gift, and leave. Don't discuss ANYTHING about what you feel is going on. THat makes it look like you are trying to make the situation about YOU and not the baby. Not the time or place.
It also sounds like your mom is "in on this." She would be the only person I would confide in and talk about this.....unless she is not a healthy person and it would make it worse. If she is unhealthy, that is probably the root of the issue....really.
Keep an emotional distance, but show support and love. It just may take time. If it doesn't work out over time, you have at least distanced yourself with healthier boundaries.
Best of luck to you!