The fundamental problem is that Effective Altruism is a political movement that spun out of a philosophical one. If you want to talk about the relative strengths and weaknesses of consequentialism, go right ahead. If you want to assume consequentialism is true and discuss specific ethical questions via that framing, power to you.
If you want to form a movement, you now have a movement, with all that entails: leaders, policies, politics, contradictions, internecine struggles, money, money, more money, goals, success at your goals, failure at your goals, etc.
This is historically inaccurate. EA’s origins are in charity evalutions to quantify the marginal impact of a donation. This motivated philosophical debate about how to operationalize “good,” and later became influential enough to have political impact. Obviously EAs were inspired by philosophical ideas or even were philosophers. But that is not the same as it being the downstream practice of a uniform set of pre-existing philosophical commitments.
If you want to form a movement, you now have a movement, with all that entails: leaders, policies, politics, contradictions, internecine struggles, money, money, more money, goals, success at your goals, failure at your goals, etc.