Sounds like your son is an introvert like his Daddy - someone who recharges by gathering their own energy from inside themselves rather than being outside running around. This is not a bad thing - it really isn't. Here's where your problem lies, however.
My guess is that, like me and most other SAHMs you see your weekend as your social time with your family. I mean, come on - you've been cooped up in the house all week or running the errands and it's your weekend, too, right? And I also guess that you are probably MOSTLY an introvert as well, right? However - 40 hours alone, or alone with a child under the age of 5 is HARD WORK. And doing some of the stuff you would like to do - go for a hike, take your children to the park, go to a museum - may not be within your grasp during the week while your husband is at work because of your work at home.
If what I have written above here hits home at all, let me tell you this - it gets better when the kids go to school, but only slightly. Like your husband - my husband works hard during the week and does not like to have to do anything that resembles hard outdoor activity on the weekend. (Getting leaves raked is a bit of a bear around here - thank goodness we live in a forest so no lawn mowing!) Until then getting outside is on you, unfortunately.
What my husband and I have done is develop a bit of a routine for our weekends. Saturdays we go together to a farmer's market (outside) before going to Barnes & Noble's train table (inside) and running any errands we need on top of that and going home. Then he and my son are free to watch football (and there is a lot of football in this house) and play legos unless there is something specific and important going on like my college homecoming or an exhibit or a hike that we planned out the week before. Just springing the "Oh, come on - I am cooped up in here all week and now it's time to go out" argument SO doesn't work. Not unless you want the "Then you go without us" response.
So, go ahead and make plans with your husband - but make sure you don't make every weekend a BIG weekend or it will become a chore for you and your husband. Besides - sometimes it's nice just to rattle around the house, both inside and out, and sometimes it's nice to be out on your own or with friends.