J.D.
If he is good with your kids and shows no sign of issues, then it's you. We have a 106 lb American Bulldog and a 60 lb basset hound. I am more concerned about the basset around my daughter than my bulldog. He is the gentlest, sweetest animal with my 20 mo. He genuinely LOVES her. He has no food aggression like the hound does and I can actually leave the room when he is eating because if my daughter touches his food, he sits and waits for her to hand feed him.
If you have never had a big dog before and have these concerns, maybe you should find him a new home. It isn't fair to you or the dog if you are constantly nervous around him. Dogs are very good at sensing feelings and he is probably trying his best to be accepted by you based on what you are saying.
If you want to keep him I suggest you call my trainer, Aaron: www.theeducateddog.com