- text classification, not text generation
- operating on existing unstructured input
- existing solution was extremely limited (string matching)
- comparing LLM to similar but older methods of using neural networks to match
- seemingly no negative consequences to warranty customers themselves of mis-classification (the data is used to improve process, not to make decisions)
Which is good because a lot of such matching and ML use cases for products I’ve worked on at several companies fit into this. The problem I’ve seen is when decision making capabilities are inferred from/conflated with text classification and sentiment analysis.
In my current role this seems like a very interesting approach to keep up with pop culture references and internet speak that can change as quickly as it takes the small ML team I work with to train or re-train a model. The limit is not a tech limitation, it’s a person-hours and data labeling problem like this one.
Given I have some people on my team that like to explore this area I’m going to see if I can run a similar case study to this one to see if it’s actually a fit.
Edit: At the risk of being self deprecating and reductive: I’d say a lot of products I’ve worked on are profitable/meaningful versions of Silicon Valley’s Hot Dog/Not Hot Dog.
I agree with you that the headline really needs to be qualified with these details. So there's an aspect of being unsurprising here, because that particular set of details is exactly where LLMs perform very well.
But I think it's still an interesting result, because related and similar tasks are everywhere in our modern world, and they tend to have high importance in both business and the public sector, and the older generation of machine learning techniques for handling these tasks we're both sophisticated and to the point where very capable and experienced practitioners might need an R&D cycle just to conclude if the problem was solvable with the available data up to the desired standard.
LLM's represent a tremendous advancement in our ability as a society to deal with these kinds of tasks. So yes, it's a limited range of specific tasks, and success is found within a limited set of criteria, but it's a very important tasks and enough of those criteria are met in practice that I think this result is interesting and generalizable.
That doesn't mean we should fire all of our data scientists and let junior programmers just have at it with the LLM, because you still need to put together a good day to say, makes sense of the results, and iterate intelligently, especially given that these models tend to be expensive to run. It does however mean that existing data teams must be open to adopting LLMs instead of traditional model fitting.
