J.Y.
I am going through the same issue with my 3yr old daughter. Some days she naps, and other days she doesn't seem to need it, but she started getting disruptive and waking my 14 month daughter's as well. I put my baby down for her nap first, then let my 3 yr old have some "special" mommy time downstairs with me. I say something like "we can play a game and read 2 books, and then it's your naptime". She seems to respond well to feeling like she gets something extra special before her nap. When I bring her up for nap, I tell her that if she goes to sleep or plays quietly in her room, she will get a "treat" when she wakes up (usually 5 m&m's). I then remind her (before EVERY nap) that IF she comes out of her room, or is loud and wakes the baby, she will LOSE a priviledge, such as her afternoon tv show. I tell her that she doesn't have to sleep, but she DOES have to stay in her room. That has been working for us for the past several weeks. I'd say my daughter naps about 4 days/week, and has "quiet time" in her room the rest. I do put a few books in her bed, and she has stuffed animals too. On the days when she doesn't sleep, I can hear her "reading" her books and talking to the animals.
Good Luck! He may be ready to give up his nap, but that doesn't mean you have to give up your time, just yet!