They make a brand of non latex condoms or at least they used to last time I checked 10+ years ago. (I was a teaching assistant for a human sexuality class). Latex comes from the rubber plant, the non latex ones are some kind of plastic, polyurithane I think, or you can get some that are made of animal products, labeled "lambskin" . You can probably get them at a drug store (read the package if you are too embarrassed to ask or look online for the brand name). I had a friend with this allergy several years ago in college.
I just reread your post, do you know for sure if it is the latex in the condom or the spermicide? Maybe you can try putting a rubber glove on your hand and see if there is a reaction, then test the spermicide on another patch of skin. Nonoxinol 9, on of the most common spermicides available, can also be an irritant to many people. You can get latex condoms without spermicide (either dry or with a lubricant) if that is what you need. Just be sure if you add any lubricant yourself it is water based. Oil causes latex to disintegrate.