It doesn't sound like he's enjoying the letter and color quizzes, so I would recommend dropping it for a while and then re-introducing it in a fun, pressure-free way. For example, instead of quizzing you could just start talking about colors and letters in books you read together, play I-spy type games (find the red ball, find the green ball, where's the letter "A") so it's fun for him instead of testing.
With letters, it is common for very young children to mix up similar letters and it's something they out-grow. For colors, you could always talk to your pediatrician about colorblindness testing, but again I would guess it's just being tired of being quizzed. Some researchers think that putting the pressure on for academic performance in early childhood leads to poor attitudes toward learning later in life. He'll get his letters and colors in good time, and it's better for him to enjoy the process than for him to be the "first" to know them.