I expect it to get better some day, but not really good in the next 10 years. I've been calling BS on fully autonomous driving for a decade and I still believe it takes AGI to do that and I think that's still a long way out.

LLMs literally grew out of autocomplete, and aside from having a huge amount of information they still don't understand, and certainly don't learn during inference.

The results are already public aren't they. Couldn't have just read out the solution?

Exam was administered on December 7th 2024, so it is possible this is in the training data but unlikely.

The open source model as worked through the problem step by step (it was R1 so I could read its chain of thought). That could be an elaborate ruse, but also unlikely.