LOL... depends on the area.
In my area "3" is considered large (as in whenever I have my son and his 2 cousins I get stopped all the time by people saying things like 'Bless You', or 'You sure have your hands full', or 'You're so brave to have so many', or 'How do you deal with SO many?' ???Huh??? 3 kids isn't rocket science
In other areas I've lived, 3 is considered completely normal/small ("Oh, you only have 3, so you should be okay.", or "The next few REALLY make things easier.", or "What are you waiting for? You don't want too big of a gap between them, you know.", or "Where are your other kids?" In these areas anything under 6 kids is a 'small' family.
ROFL...
4 is the "normal/small" number of children in my extended family, so I personally consider 5 to be the beginning of a larger family.
AKA I consider "big" to be 1 more than my own 'normal' or bigger to be 'big'.
But considering that the 'average' family now has 2.1 kids, I think my own is 'overestimating'. So it looks like "3" IS now considered "large".
AKA... with 4 I'm sure you're fine :)