J.N.
Depends. If you want a casual "family" dinner, take him with and be ready for all sorts of things to happen. Sometimes kids are great, sometime not. This can continue for some time (even until they're 7 or 8).
If what you want is a nice, relaxing, date-dinner, or if you're going to a nicer place, get a baby sitter. Also, if you're going to be out past his bed-time. You'll enjoy each other's company a lot more without an unhappy kid. Besides, you and hubby need some time to be sweethearts and lovers, besides just parents.