P.W.
Take your baby in. Who cares what the doctor thinks? You are paying the doctor for a service....... besides most doctors would prefer you err on the side of being careful.
Go in. Otherwise you have a 3 day weekend to worry. Baby may have an ear infection without fever or something else..... or nothing, but it is worth a trip to the doctor in my mind if for no other reason to help you relax.