Minorities tend to vote less in midterms, and more Republican in midterms (Hispanics for instance tend to vote more Republican in midterms, and very heavy democratic in presidential election years).
Regardless, presidential elections are what matters. And the GOP does terrible with minorities...