I travelled extensively with my husband on business. Because many of the events were definitely not kid friendly, the company had specific family trips that were meant for the whole dang kit and kaboodle and they paid for all expenses.
However, it would have been extremely inappropriate to take children of any age to the black tie events.
If you're fine leaving the older two children, take your baby and enlist your friend to watch your baby for the adult portions of the trip such as the dinner. I would also have her watch the baby for the spa. I don't know how you could possibly enjoy a spa with a wriggly 6 month old. That kind of defeats the relaxation aspect of it.
I was a nursing mother too, in fact, I nursed my son for 15 months. We travelled, but I didn't take him to swanky events.
It takes a little planning, but it's not impossible to figure out how to have some grown up time with your husband, even if you take all your kids.
San Diego is pretty. You'll really enjoy it.
Best wishes!