I think my oldest son has our families' lineage of anxiety as well, which runs strong on both sides. I see it in how he picks his lips, how he shuts down at certain situation, how he must complete play tasks in a certain order or he goes nuts. It takes one to know one. My second boy seems like he will be spared.
Your doctor will probably say they can't say anything definitively until your son is 3. That's how it was for us. Now that my oldest boy is four, his anxiety is even more clear to me. The good news for all of us is this: there are much better treatment options today than there every were before.
Best advice? Treat YOURSELF first. Manage your own anxiety via Cognitive Behaviorial Therapy, if you have the time and money. Get on Lexapro or some other SSRI. Work at it. Because you are going to be a model for how to deal with life. I try to get my oldest a lot of exercise, as that is a natural method of coping. I prep him for new situations as much as possible, explaining what things will look like, who will be there, etc. I've even looked up You Tube videos of kids doing karate before we went to our first class, so he could have a visual. This helps decrease anxiety.
Another stray thought: asthma. If there is any chance of allergies/asthma in your family, I'd keep an eye on that as well in regards to the breathing issues. You can have had "pregnancy-induced asthma" which would be obvious when having a panic attack. Just a thought.
Good luck. Know that there are worse things to genetically inherit, and that anxious people have traits that are often advantageous to successful lives, if channeled properly.