I would encourage you to keep 2nd, 3rd, 4th birthdays very low key. Maybe invite grandparents over for dinner. You don't mind lots of people, but 2 -4 yr olds don't do well with lots of noise and commotion. You don't have to have a birthday party every year for every kid. Talk about exhausting and teaching them the wrong lessons about their bday. Obviously, your mother thinks that the 25 cousins who came to the 1st bday party was too much. Maybe your mom knows how disruptive all those people in one house are to a child. (I would have to agree. Sorry.)
By age 5 or 6, start having parties with son's friends. Good rule of thumb is same number of years = number of kids or some say, one less than number of years old.
Family comes over for dinner at another time.
Once my kids were invited to a neighbor's party where there were 5 boys from the street, and 4 cousins, aunts and uncles and a grandmother in from out of town. Talk about a weird mix of people and personalities. I can say it was the worst party we've ever been to.
Lastly, remember, the bigger the deal you make about the birthday party, the more gifts and material possessions the children will focus on as they get older. I've seen it happen over and over. My youngest (16) goes to school with a girl who was so distraught that none of her friends brought bday gifts to school on her 15th bday for her, that she had to go home. She was a mess. Sad but true about some kids these days.