It's because claims of true merit have for a long time been used as the justification for the exclusion of people systems have been rigged to exclude.
Just to pick the most obvious example, the US's slave states systematically kept black people uneducated and beaten down, and then used their condition to justify their ongoing subjugation. For example, from the Texas declaration of Secession talks about the how the Northerners had "an unnatural feeling of hostility to these Southern States and their beneficent and patriarchal system of African slavery, proclaiming the debasing doctrine of equality of all men, irrespective of race or color-- a doctrine at war with nature, in opposition to the experience of mankind, and in violation of the plainest revelations of Divine Law." In their view, the southern aristocracy were the obvious and true holders of merit, and the system that put them on top was just proof of their merit.
Or you could look at how women were systematically excluded for generations from education and economic resources, and then their condition was used at the justification for not letting them vote.
Just to pick the most obvious example, the US's slave states systematically kept black people uneducated and beaten down, and then used their condition to justify their ongoing subjugation. For example, from the Texas declaration of Secession talks about the how the Northerners had "an unnatural feeling of hostility to these Southern States and their beneficent and patriarchal system of African slavery, proclaiming the debasing doctrine of equality of all men, irrespective of race or color-- a doctrine at war with nature, in opposition to the experience of mankind, and in violation of the plainest revelations of Divine Law." In their view, the southern aristocracy were the obvious and true holders of merit, and the system that put them on top was just proof of their merit.
Or you could look at how women were systematically excluded for generations from education and economic resources, and then their condition was used at the justification for not letting them vote.