N.M.
My pedi had told me that itf he's not crying it's ok. That sometimes it's good for babies to have some time to just "space out". But I would suggest that when you are reading to your 3 year old you have the baby in a bouncy seat close by to listen. My girls were learning to read when my son was born, so I had them read to him as an infant, it was great for all of them!