I suspect it's one of those things you kind of have to learn as a child due to brain plasticity. As a native English speaker I've been trying to learn some Serbo-Croatian — which is linguistically much closer to English than any Chinese dialect — and even though my hearing is normal I can't perceive the difference between the "č" and "ć" sounds. They both seem like the English "ch" to me.