What kind of proof are you looking for here, exactly? Lots of businesses are successfully using AI... There are many anecdotes of this, which you can read here, or even in the article you commented on.
What do you mean by “successfully using AI”, do you just mean some employee used it and found it helpful at some stage of their dev process, e.g. in lieu of search engines or existing codegen tooling?
Are there any examples of businesses deploying production-ready, nontrivial code changes without a human spending a comparable (or much greater) amount of time as they’d have needed to with the existing SOTA dev tooling outside of LLMs?
That’s my interpretation of the question at hand. In my experience, LLMs have been very useful for developers who don’t know where to start on a particular task, or need to generate some trivial boilerplate code. But on nearly every occasion of the former, the code/scripts need to be heavily audited and revised by an experienced engineer before it’s ready to deploy for real.
Yeah, I should have posted the first version of my post, pointing out that the problem with this demand for proof (as is often the case) devolves into boring definitional questions.
I don't understand why you think "the code needs to be audited and revised" is a failure.
Nothing in the OP relies on it being possible for LLMs to build and deploy software unsupervised. It really seems like a non sequitur to me, to ask for proof of this.
That’s fair regarding the OP, and if otherwise agree with your sentiments here.
Some other threads of conversation get intertwined here with concerns about delusional management making decisions to cut staff and reduce hiring for junior positions, on the strength of the promises by AI vendors and their paid/voluntary shills
For many like me who have encouraged sharp young people learn computers, we are watching their spirits crushed by this narrative and have a strong urge to push back — we still need new humans to learn how computer systems actually work, and if nobody is willing to pay them for work because an LLM outperforms them on those menial “rite-of-passage” types of software construction, we will find ourselves in a bad place
Software engineering is always light on case studies though, for instance test driven development, or static vs. dynamic typing, people have been debating these for quite a long time.
i mean i can state that i built a company wihtin the last year where id say 95% of my code involved using an LLM. I am an experienced dev so yes it makes mistakes and it requires my expertise to be sure the code works and to fix subtle bugs; however, i built this company me and 2 others in about 7 months for what wouldve easily taken me 3 years without the aid of LLMs. Is that an indictment of my ability? maybe, but we are doing quite well for ourselves at 3M arr already on only 200k expense.
That’s genuinely far more interesting and exciting to me (and I’m sure others too) than this sort of breathless provocation, esp if code and prompts etc are shared. Have you written about it?
i have not written about it due to me being a silent partner in the company and dont want my name publicly attatched to it, but the code and prompts is all more like talking to a buddy is how i use it, i ask it to build specific things then i look through and make changes. For instance a few examples i can give is there is a lot of graph traversal in my data i built, I'm not an expert on graph traversal, so I researched what would be a good algo for my type of data, and then utilized claude to implement the papers algorithm into my code and data structures. I dont have the llm in any steps that the customer interact with (there is some fuzzy stuff but nothing consistently run) but i would say an llm has touched over 90% of the code i wrote. its just an upgraded rubber ducky to me.
If i wasn't experienced in computer science this would all fall apart however i do have to fix almost all the code, but spending 10 mins fixing something is better than 3 days figuring it out in the first place (again this might be more unique to my coding and learning style)
Do I think those rise to "case studies"? No. But to another commenters point, detailed and rigorous case studies have always been hard to come by for any productivity process or technology.
I also think that article is hype, but it's not true that it's vague.
What else are you looking for?