I prefer (the sound of) Caleb Mathew. That's not to say I don't like the name Clement - I actually think it's a really cool name, and I love family name significance (we did that wherever we could).
The thing is - we have found that our kid's middle names never come up in conversation. So if you like it, go ahead and use it.
I use one of my children's middle names quite frequently, only because I love the sound of it with the first name - it just rolls off the tongue and it's kind of a nick name at this point.
Otherwise, middle names are just names you see now and then on forms don't you find? You must have found that with your older child's name. Perhaps your kids are still really young.
Mine are teens. Their middle names never come up. The last time was a document at the high school level where kids made grades (academic) and they listed their full names. There was a lot of 'older' traditional names there, so I think a lot of family names get used.
If this helps - my teen boys actually like being named after grandfathers etc. Other older cousins have been as well, and they like the connection.