T.O.
My son knows his colors, but doesn't recognize letters or numbers. I often get the "I don't know" response...and often I don't know if he's being silly or really doesn't know.
1 thing I did to help learn colors was "color of the week/day/ etc." I would have him wear a red shirt, use a red spoon, drink out of his red cup, have red strawberries for a snack, etc. I would do that everyday with the same color until he started to know red. It took 2 or 3 days per color. That seemed to work for us. Good Luck!