In the United States, the working poor are considered deserving of their burdens in an immutable, moralizing, Calvinist way. “They make bad choices.” “They have bad culture.” “They have bad genes.”

Today’s public benchmarks are yesterday’s training data.

I cannot understand this prideful resentment of theory common among self-described practitioners.

Even if existing theory is inadequate, would an operating theory not be beneficial?

Or is the mystique combined with guess&check drudgery job security?

If there were theory that led to directly useful results (like, telling you the right hyperparameters to use for your data in a simple way, or giving you a new kind of regularization that you can drop in to dramatically improve learning) then deep learning practitioners would love it. As it currently stands, such theories don't really exist.

This is way too rigorous. You can absolutely have theories that lead to useful results even if they aren't as predictive as you describe. Theory of evolution for an obvious counterpoint.

Useful theories only come to exist because someone started by saying they must exist and then spent years or lifetimes discovering them.

There are strong incentives to leave theory as technical debt and keep charging forward. I don't think it's resentment of theory, everyone would love a theory if one were available but very few are willing to forgoe the near term rewards to pursue theory. Also it's really hard.

There are many reasons to believe a theory may not be forthcoming, or that if it is available may not be useful.

For instance, we do not have consensus on what a theory should accomplish - should it provide convergence bounds/capability bounds? Should it predict optimal parameter counts/shapes? Should it allow more efficient calculation of optimal weights? Does it need to do these tasks in linear time?

Even materials science in metals is still cycling through theoretical models after thousands of years of making steel and other alloys.

Many mathematicians are (rightly, IMO) allergic to assertions that certain branches are not useful (explicit in OP) and especially so if they are dismissive of attempts to understand complicated real world phenomema (implicit in OP, if you ask me).

Who is proud? What you are seeing in some cases is eye rolling. And it's fair eye rolling.

There is an enormous amount of theory used in the various parts of building models, there just isn't an overarching theory at the very most convenient level of abstraction.

It almost has to be this way. If there was some neat theory, people would use it and build even more complex things on top of it in an experimental way and then so on.

> If you like it or it makes you feel something, that's all that matters.

You’re right!

> Even jazz uses 6-2-5-1's over and over.

You’re not even wrong! I wonder if jazz does anything else besides that?

A major floodplain in Michigan has a similar problem with dioxin contamination.

https://www.michigan.gov/-/media/Project/Websites/mdhhs/Fold...

In another world, macros could have filled the role that programmable yaml fills today.

|-

  help  

  me

You’re narrowly addressing LLM use cases & omitting the most problematic one - LLMs as search engine replacements.

That's the opposite of problematic, that's where an LLM shines. And before you say hallucination, when was the last time you didn't click the link in a Google search result? It's user error if you don't follow up with additional validation, exactly as you would with Google. With GenAI it's simply easier to craft specific queries.

I tire of this disingenuous comparison. The failure modes of (experienced, professional) humans are vastly different than the failure modes of LLMs. How many coworkers do you have that frequently, wildly hallucinate while still performing effectively? Furthermore, (even experienced, professional) humans are known to be fallible & are treated as such. No matter how many gentle reminders the informed give the enraptured, LLMs will continue to be treated as oracles by a great many people, to the detriment of their application.

Wildly hallucinating agents being treated as oracles is a human tradition.

Why are they attempting to model LLM update rollouts at all? They repeatedly concede their setup bears little resemblance to IRL deployments experiencing updates. Feels like unnecessary grandeur in what is otherwise an interesting paper.

The negative tone of some comments here betrays that techno optimist suggestion that tech will solve environmental issues. Clearly, the incentives just aren’t there!

Tech has caused more and worse environmental issues than it has ever solved.