unless you train directly against solving those problems... in which case how could you theoretically design a test that could stand against training directly against the answer sheet?

That's why they keep the evaluation set private: "Submit a solution which scores 85% on the ARC-AGI private evaluation set and win $600K."

[0] https://arcprize.org/guide