That is premature abstraction 101 though. SRE folks will bring parochial philosophy and first principles constraints too early, not understanding the differences between ML system needs and other systems they’ve worked with.
Assuming you could have correctly anticipated those design needs earlier, rather than going back to iterate only after you get concrete evidence of the specific changes you need, is a big trap and time waster.
I don’t agree that you lose because the data scientist failed to consider production SRE patterns early enough. You lose when those SREs fail to take a YAGNI philosophy, let the unusual ML system’s chips fall where they may, and then intentionally go back and iterate on improving it.
Assuming you could have correctly anticipated those design needs earlier, rather than going back to iterate only after you get concrete evidence of the specific changes you need, is a big trap and time waster.
I don’t agree that you lose because the data scientist failed to consider production SRE patterns early enough. You lose when those SREs fail to take a YAGNI philosophy, let the unusual ML system’s chips fall where they may, and then intentionally go back and iterate on improving it.