Agreed with @roenxi and I’d like to propose a variant of your comment:
All evidence is that “more is better”. Everyone involved professionally is of the mind that scaling up is the key.
However, like you said, just a single invention could cause the AI winds to blow the other way and instantly crash NVIDIA’s stock price.
Something I’ve been thinking about is that the current systems rely on global communications which requires expensive networking and high bandwidth memory. What if someone invents an algorithm that can be trained on a “Beowulf cluster” of nodes with low communication requirements?
For example the human brain uses local connectivity between neurons. There is no global update during “training”. If someone could emulate that in code, NVIDIA would be in trouble.
All evidence is that “more is better”. Everyone involved professionally is of the mind that scaling up is the key.
However, like you said, just a single invention could cause the AI winds to blow the other way and instantly crash NVIDIA’s stock price.
Something I’ve been thinking about is that the current systems rely on global communications which requires expensive networking and high bandwidth memory. What if someone invents an algorithm that can be trained on a “Beowulf cluster” of nodes with low communication requirements?
For example the human brain uses local connectivity between neurons. There is no global update during “training”. If someone could emulate that in code, NVIDIA would be in trouble.