My son is 5 weeks, he is our fourth... he doesn't seem to connect with me yet either. A few times I've felt like he's really seeing me and we've even gotten a few small smiles. If your doctor isn't worried sounds perfectly normal. :)
Funny thing about being a Mom, you spend so much time looking for the *next* big developmental milestone... then they get older and we long for when they were babies again. Just enjoy these early days and try not to worry. Since you have another child you already know how quickly it goes by.