Okay, the way I have always understood it is that you dress your children like you would dress for the most part. That being said, I believe an undershirt underneath would depend on what they are wearing on top and where you are going to be. If you are going to the park on a brisk day, an undershirt underneath with a sweatshirt on top may be more convenient and appropriate that say a bulky jacket or a jacket that your child won't leave zipped. Also, if you are going to be taking the kids out to play in the snow, you want to layer and an undershirt is a good place to start. However, if they really aren't going to wear them often, you can substitute a regular t-shirt for an undershirt on those occasions that you think you may need them. Personally, I would probably buy them a pack of them each. Not enough to wear everyday but a few to have on hand in case they are needed. My kids also wear them to bed on occasion.
Basically, it is up to you whether or not to buy them. They are not necessary (since you have other options) but they can be helpful.
I hope this helps.