I'd take him to the doctor again and perhaps get another opinion from a different physician in the practice to see if there's something they're just not catching.
At 6 months, they have weak immune systems compared to those of us that have been exposed to things for years and built up the antibodies.
That being said, I've been so much sicker than normal this year, and my Oncologist's office said that it's just been a rough year. Perhaps that has something to do with it.
Also, you mentioned that the whole family has been sick. If you have older children, it's possible they're bringing home germs that they're able to fight off but he isn't. When I went through chemo, I had no immune system, had 2 kids in day care and worked in and out of doctor's offices all day. When I asked my Oncologist if I should stop working, she said to be diligent in washing my hands and using sanitizers. Perhaps being more conscious of that throughout the whole family may help from exposing him.
But, it sounds like he's getting exposed to some serious things. 103.5 is dangerously high for spiking a febrile seizure. I hope you're able to find the root cause soon and get a healthy happy baby.