Hm. I think down time is important, but kids need some up time and well....a life, too!!! I see no reason whatsoever that your younger child couldn't go and just hang out while your older child does something fun! There's 24 hours in a day. If your child sleeps 12 hours a day (including a nap), then there's still LOADS of time to unwind!
When to start activities? My oldest turned 7 this past weekend. He started little activities when he was 3. He did gymnastics for a couple weeks but we decided it was too early, all he wanted to do was jump on the trampolene. We put him in half day mothers day out at a church (home in time for lunch), 2 days/week and he loved it and thrived! Over the years, starting at age 4, he's experimented and explored in mommy & me music play, swim lessons, kung fu, soccer, equestrian lessons, awanas (little club at church), an art camp, a zoo camp, and his current activities: cub scouts and flag football. He asked me to let him do some hip hop dance class after football is over, so he'll know how to do something when they have the sock hops and stuff like that at school. (I only know the jitterbug and taught him that, but that requires a partner that knows it too).
He still maintains great grades, all our meals are together as a family and we have family devotional time, he does his homework, has down time just for him (usually he's playing with the dog, reading, or playing with legos), has play time with his little brother and a couple neighbors or some kids at the park when we walk the dog to the park for a little playtime and then walk back home, he has an activity a couple days a week plus a game on Saturday and church on Sunday, and STILL manages to have plenty of time to read, draw, make a poster for Cub Scouts or whatever. Some weeks are really busy, other weeks are slower. We adjust our Sundays accordingly (lots of rest or go out as a family).
As for the younger sibling question: 4 is old enough for it to be a perfectly reasonable expectation that he/she could hang out while big brother/sister has an enrichment activity. My youngest child is 3. He's always been present at everything but the scout's den meetings (though he does go to the larger pack meetings because it's a whole family event). We cheer for big brother. We watch, clap, and then go home and copy. It actually enables big bro to practice at home, helping little bro learn it too. We cheer for our family and absolutely expect big brother to do the same when little brother starts having events. You probably won't be the only mom with 2+ kids at an event, and kids can play together on sidelines. (Soccer: we made great friends with a family whose children were the same ages as ours, we ended up being really good friends for 3 years and counting! Parties, beach trips, picnics, all kinds of fun...Kung Fu, my younger son was well cared for by some preteens that played with him while waiting for their sibling...football, my son plays on the playground or runs around with another ball with another 3 year old during games).You can pack a little go bag for your younger child: a snack, a couple small toys, a book, a notepad and crayons, whatever would rock his/her world and help entertain for a little bit. But it's also an exercise in behaving and understanding that sometimes, the world doesn't just revolve around him/her. However, at my son's football practices, it's very near a playground and we do let little guy play while big guy practices.
As for what my NEARLY 4 year old does for enrichment: we attend church and he loves it. He is in Awanas (a Bible club at church) on Wednesday evenings and loves winning awards and earning patches on his vest or stickers in his book. In summer he went to mommy & me swim lessons. He goes weekly to small fry club with me where parents are invited to stay involved but a "teacher" reads a story, has a game or song, a project for the kids to do, sometimes a guest (firemen with their truck, a dentist, etc),a different theme each week. And we go to storytime at the library and do arts and crafts. He loves to cook with me. We take little field trips during the week when big brother is in school: zoo, museums, stockyards, a farm, etc. Next spring he'll have a choice of Tball, flag football, soccer, or dance as his first team sport (exception being dance). In summer, they'll all take swim lessons again. In the fall, he can choose an activity (martial arts, theater, flag football, whatever he wants). Let him figure it out on his own. We'll make it work.
As for balancing needs of children: the way we did it was to write down whatever needs we could come up with (spiritual, emotional, mental, social, etc, etc---whatever they are to your family) and then I make sure that we cover those needs for all of us in our weekly routines. We may not hit every one of them every day of the week, but they WILL all be covered every week. It's important to not just decide you don't like doing things, and therefore your child who is raring to go must be held back. Explore. Let them develop new interests, acquire new skills, if they like something that's great and if they don't, you can move on to something else. That's my thoughts on it at least.