I'm saying that Mastercard/Visa being able to do this on their own is one half of a free speech issue. It's not a free speech issue of its own, but it creates a free speech issue when combined with them being vulnerable to government pressure.

Imagine that the government instituted a private organization that had the power to cripple any business, with no recourse, and had control over it. The first amendment does not literally prohibit this, yet this would have clearly been a violation of the first amendment in spirit.