"The ultimate test" will be when AI is capable of completely running a startup business. Making itself a website, writing, testing, and iterating its own code, filling out legal forms, marketing its value to consumers, closing sales, and managing its budget. That is the ultimate test that we all fear. If it can do all that, I'll then trust it with making those massive life-death decisions within the government.