One of my children, who is now 9, also poops infrequently. He often went 14 days without a BM, but when he did go, it was enormous blowout.
Of course, if your breastfed baby was still a newborn, you would expect several BMs a day, but now that your baby is a bit older it is not uncommon to have less BMs.
I am concerned when you say that his stools are not large, and would be curious about his weight gain. If his weight gain is good and he is not uncomfortable I don't think there is a reason to worry. Your decision to wait to give anything other than breastmilk due to a history of allergies is very wise.
Here is what La Leche League says about 'constipation' in the breastfed baby:
"It is normal for the bowel movements of a breastfed to decrease in frequency when the colostrum, which has laxative properties, is completely gone from the mother's milk after about six weeks of age. A baby this age may continue to have bowel movements as frequently as five times a day, sometimes even after every nursing. It is also normal for a breastfed baby older than six weeks to have only one bowel movement every few days. Some healthy babies will have only one bowel movement a week. When bowel movements are less frequent, they should be more profuse in volume. As long as the baby is gaining well, wetting sufficiently, and is happy and content there is no cause to be alarmed by infrequent bowel movements, and it is not necessary to give the baby a laxative, fruit juice, or any other "helpers." In fact, attempting to force bowel movements can have harmful consequences to your baby."
Happy Nursings!
B.