The issue is that what keeps bitcoin together is that the computation is used as a proof which transaction is accurate. Smaller number of miners would mean that a party with a very powerful hardware could then decide what's real or not. If someone can get over 50% computation power of bitcoins, that entity would decide what transaction is real or not.

Some alternative crypto currencies use algorithms that supposed to be really hard to do in ASCIC, I suppose those would be more feasible for rel people to use them, but as I understand we are stuck this way with bitcoin.