E.A.
Saying "I don't care what you look like" is not what you need to hear. You need to hear that he thinks you are beautiful and sexy. I bet if he changed that one comment, it would help you feel better, I would feel horrible if my husband told me he didn't care what I looked like. Even when I started my diet, he supported me without ever mentioning how fat I had become. You need compliments, not for your concerns to be brushed off like that.
And you two really need to take some time away to talk about this if it is bothering you that much. He needs to show you through words AND actions that he finds you attractive.