But plenty of people here do work on real products. Planes need software, browsers need security patches, hell even your accounting app is good value over the days of doing that all by hand.
Fair, but let’s be honest: even in your accounting app example, the “value” is ultimately in helping a business capture margin more efficiently. That business, in turn, is usually extracting value as a middleman somewhere else in the chain. Same with ad tech, finance, or even a lot of enterprise software, you’re greasing the gears of profit capture, not curing cancer.
That’s not a moral indictment, just a reminder that most of our jobs (mine included) exist to make capital move faster or stickier. Calling one sector “real products” and another “not benefiting society” is a bit of a convenient fiction.
Sure, but it's still helpful to think of what components of production are 'intrinsic' to production (it is hard to imagine making things out of metal without some sort of manufacturing being done by someone) and which are accidents of the particular economic system we find ourselves in (a system without private ownership of capital would likely have little place for equity traders). Advertising as we understand it (push-based) would likely not exist in an economy not based on commodity production.
That's not too cast moral judgement -- just to point out that under a different economic system, these overhead costs could be avoided and these resources (human and otherwise) could thus be redirected to ends more concomitant with human flourishing.
Both ad tech and quant work are essentially involuted in that you're spending ever greater amounts of effort and manpower to squeeze out marginal gains because of how much profit doing so provides. Society would not significantly notice or suffer if we spent half as much time on things like this. There are much bigger things we need to do.
But what system should we use that fixes our economic system? This is not so much a defense of capitalism but asking what would be a better replacement. Capitalism in my opinion is the best solution so far that helps drive society forward when society is made up of all types of folks with different opinions and views of the world.
I used to be one of those people, then I started saving these scripts in a folder and realized just how much time it saved me. Especially for nontrivial scripts which require a lot of typing or input specification.
This was me before LLMs. Had tons of scripts saved. I just don't see the point anymore "Take this list of users, remove from x group and reset their passwords" takes less time than trying to remember what I named that file and where it's saved. Anything that can be done in under 100 lines isn't worth saving anymore.
You'd be surprised at what juniors can pull off. I have seen fresh-out-of-college new grads write performant GPU kernels that are used in real world library implementations for particular architectures.
Of course they can. It doesn't take that long to learn CUDA/etc and hardware details, read through some manuals and performance guides, do some small projects to solidify that knowledge. What you need is talent, and some months. That's why at university I saw plenty of students pull off amazing projects, and there's nothing eyebrow-raising in the slightest about starting a PhD a year after getting a bachelor's and writing software more sophisticated than most programmers would write in their career.
I think the programming profession overvalues experience over skill. However, when I was young I had no appreciation for what the benefits of experience are... including not writing terrible code.
Most of the juniors I've worked with would make numerical errors and give up/declare premature victory before getting the implementation to a robust state. I'm sure there are exceptional young folks out there though.
> We see that the nightmare scenario - a person with no previous psychosis history or risk factor becoming fully psychotic - was uncommon, at only 10% of cases. Most people either had a previous psychosis history known to the respondent, or had some obvious risk factor, or were merely crackpots rather than full psychotics.
It's unfortunate to see the author take this tack. This is essentially taking the conventional tack that insanity is separable: some people are "afflicted", some people just have strange ideas -- the implication of this article being that people who already have strange ideas were going to be crazy anyways, so GPT didn't contribute anything novel, just moved them along the path they were already moving regardless. But anyone with serious experience with schizophrenia would understand that this isn't how it works: 'biological' mental illness is tightly coupled to qualitative mental state, and bidirectionally at that. Not only do your chemicals influence your thoughts, your thoughts influence your chemicals, and it's possible for a vulnerable person to be pushed over the edge by either kind of input. We like to think that 'as long as nothing is chemically wrong' we're a-ok, but the truth is that it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
For this reason it is very important that vulnerable people be well-moored, anchored to reality by their friends and family. A normal person would take care to not support fantasies of government spying or divine miracles or &c where not appropriate, but ChatGPT will happily egg them on. These intermediate cases that Scott describes -- cases where someone is 'on the edge', but not yet detached from reality -- are the ones you really want to watch out for. So where he estimates an incidence rate of 1/100,000, I think his own data gives us a more accurate figure of ~1/20,000.
You might want to read the entire article. His depiction of bipolar is completely accurate. In fact it is so precisely accurate in every detail, and conveyed with no extraneous information, is indicative of someone who knows the disorder very well.
When I write fiction or important emails, I am precise with the words I use. I notice these kind of details. I’m also bipolar and self-aware enough to be deeply familiar with it.
And as I recall, he used to be a lot more clear that mental illness isn't always clear cut. I was surprised at the "obviously, we all know what mental illness is" attitude coming from him.
I imagine he's been doing this so long that taking a sober, delicate, appeal-to-every-reader approach to every article just isn't fun or practical. He wants to get down his thoughts for his core reader base and move on. What fraction goes viral anyway? And so what if some people misunderstand or don't like it? You're always going to have those people anyway no matter what :shrug:
I'm not trying to argue from authority or get into credibility wars*, but Scott is a professional psychiatrist who has treated dozens or hundreds of schizophrenic patients and has written many thorough essays on schizophrenia. Obviously someone could do that and still be wrong, but I think this is a carefully considered position on his part and not just wild assumptions.
*(or, well, okay, I guess I de facto am, but if I say I'm not I at least acknowledge how it looks)
The criticism invoked “anyone with serious experience with schizophrenia”, implying the author of the article is not such a one. Citing the author’s experience is a perfectly valid rebuttal to that implication. It’s not an argument from authority, but about it.
I'm not trying to say that that should strongly increase the probability he's correct. I just think it's useful context, because the parent is potentially implying that the author is naively falling for common misconceptions ("following the conventional tack") rather than staking a deliberated claim. Or they might not be implying it but someone could come away with that conclusion.
I mean, on one hand you have a professional psychiatrist who has treated many people for the disorder we're talking about, and on the other, we have a rando on HN who hasn't presented any credentials.
Not saying the latter person is automatically wrong, but I think if you're going to argue against something said by someone who is a subject matter expert, the bar is a bit higher.
One of the questions that sets up the premise of the article in the first paragraph is, "Are the chatbots really driving people crazy, or just catching the attention of people who were crazy already?"
That's why he's honing in on that specific scenario to determine if chatbots are uniquely crazy-making or something. The professional psychiatrist author is not unaware of the things you're saying. They're just not the purpose of the survey & article.
Well yeah, that's a false dichotomy. If you're vulnerable and a chatbot sends you into a spiral of psychosis, your pre-existing vulnerability doesn't negate the fact that a harm has been done to you. If you have a heart condition, and I shoot you with a Taser, and it kills you… I've killed you. You weren't "already dead" just because you were vulnerable.
Yes but "what is the effect of tasers on hearts" is an interesting question when tasers are brand new. If it kills people with obvious pre-existing risks then that is not very surprising. If it kills 50% of otherwise healthy people in a way we didn't anticipate, that is alarming and important to distinguish.
Imagine someone does a quick survey to estimate that tasers aren't killing people we don't expect, and some readers respond saying how dare you ignore the vulnerable heart people. That's still an important thing to consider and maybe we should be careful with the mass scale rollout of tasers, but it wasn't really the immediate point.
> Imagine someone does a quick survey to estimate that tasers aren't killing people we don't expect
Given that the quote you cited was, "Are the chatbots really driving people crazy, or just catching the attention of people who were crazy already," I'd say the equivalent would be something like, "Are tasers really killing people, or were tasered heart attack victims dying already?"
And yeah, I'd be mad about that framing! The fact that the people who die had a preexisting vulnerability does not mean they were "already dying" or that they were not "really killed."
Shoving full implicit context into the analogy, it would be more like "are tasers really killing otherwise physically healthy people, or are the recent notable deaths primarily from people with pre-existing risks?"
I can agree that Alexander might appear flippant or even callous about mental health at times (especially compared to modern liberal social media sensibilities), but I chalk that up to the well-earned desensitization of a professional working in the field for decades.
There's flippancy that crosses social lines, and there's flippancy that blurs technical distinctions. The difference between someone whose mental disorders are under control vs someone experiencing psychosis is like an oncologist handwaving the difference between terminal cancer and cancer in remission. The difference is enormous, to the point that the whole purpose of the psychiatric field is to move people from the one category to the other. I don't think technical experice justifies glossing over that distinction.
This is a fair point: I'm familiar with Scott Alexander but somehow didn't know he was a psychiatrist, so that part of my point was unfair.
Nevertheless, I think the broader argument still stands. I think that it's at best unhelpful and at worst actively harmful (in the sense of carrying water for AI forms who would happily drive people mad as long as it didn't hurt their stock price) to pretend it's possible to draw a line between "people with risk factors" and "normal people". Everyone is art risk to some extent -- e.g. nobody is too far from financial crisis and homelessness, a major risk factor on its own. Talking about how "people who weren't crazy already" are less at risk ignores the fact that 99%+ of people who "are crazy already" were at some point not crazy, and the path between is often surprisingly smooth. It does us as a society no good if we pretend "normal people" don't have to worry about this phenomenon -- especially when having some kooky ideas was enough to get bucketed in "not normal" for this particular survey
The article's conclusion is exactly what you describe: that AI is bringing out latent predisposition toward psychosis through runaway feedback loops, that it's a bidirectional relationship where the chemicals influence thoughts and thoughts influence chemicals until we decide to call it psychosis.
I hate to be the 'you didn't read the article' guy but that line taken out of context is the exact opposite of my takeaway for the article as a whole. For anyone else who skims comments before clicking I would invite you to read the whole thing (or at least get past the poorly-worded intro) before drawing conclusions.
> it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
This seems very incorrect, or at least drastically underspecified. These trains of thought are "normal" (i.e. common and unremarkable) so why don't they "latch your brain into a very undesirable state" lots of the time?
I don't think Scott or anyone up to speed on modern neuroscience would deny the coupling of mental state and brain chemistry--in fact I think it would be more accurate to say both of them are aspects of the dynamics of the brain.
But this doesn't imply that "simple normal trains of thought" can latch our brain dynamics into bad states -- i.e. in dynamics language move us into a undesirable attractor. That would require a very problematic fragility in our normal self-regulation of brain dynamics.
See the key here is, the AI provides a very enticing social partner.
Think of it as a version of making your drugged friend believe various random stuff. It works better if you're not a stranger and have an engaging or alarming style.
LLMs are trained to produce pleasant responses that tailor to the user to maximize positive responses. (A more general version of engagement.) It stands to reason they would be effective at convincing someone.
That's essentially a retaliatory hit piece the NYT printed because they were mad that Scott deleted his website because the NYT wanted to doxx him. Not saying there's no merit to the article, but it should be looked upon skeptically due to that bias.
NYT wanted to report on who he was. He doxxed himself years before that (as mentioned in that article). They eventually also reported on that (after Alexander revealed his name, seeing that it was going to come out anyway, I guess), which is an asshole thing to do, but not doxxing, IMO.
They wanted to report specifically his birth/legal name, with no plausible public interest reason. If it wasn't "stochastic terrorism" (as the buzzword of the day was) then it sure looked a lot like it.
> He doxxed himself years before that
Few people manage to keep anything 100% secret. Realistically private/public is a spectrum not a binary, and publication in the NYT is a pretty drastic step up.
> They wanted to report specifically his birth/legal name, with no plausible public interest reason.
Siskind is a public figure and his name was already publicly known. He wanted a special exception to NYT's normal reporting practices.
> Realistically private/public is a spectrum not a binary
IIRC his name would autocomplete as a suggested search term in the Google search bar even before the article was published. He was already far too far toward the "public" end of that spectrum to throw a tantrum the way he did.
Siskind is a practicing psychiatrist, which is relevant to his profile. Using his real name makes it possible to discuss that. Putting Kendrick's surname ("Duckworth") into the profile adds nothing.
Siskind is a public figure—I don't know why so many people think he is entitled to demand that NYT only discuss him in the ways he wants to be discussed (i.e. not connecting his blog to his physciatric practice).
> Siskind is a practicing psychiatrist, which is relevant to his profile. Using his real name makes it possible to discuss that.
The NYT of all entities should be comfortable talking about whether someone has particular qualifications or a particular job without feeling the need to publish their birth/legal name.
> Siskind is a public figure—I don't know why so many people think he is entitled to demand that NYT only discuss him in the ways he wants to be discussed (i.e. not connecting his blog to his physciatric practice).
Again the NYT of all entities should understand that there are good reasons to hide people's private details. People get very angry about some of the things Alexander writes, there are plausible threats of violence against him, and even if there weren't, everyone agrees that names are private information that shouldn't be published without good reason. His blog is public, the fact of him being or not being a practising psychiatrist may be in the public interest to talk about, but where's the argument that that means you need to publish his name specifically?
> there are good reasons to hide people's private details
They do, and they do grant anonymity sometimes. But it's their call, and they made the call. They're not a PR firm; they have no obligation to be kind or gentle in their coverage. If they wanted, they'd be fully within their rights to publish a noxious hitpiece on the man. They were much milder than I'd have been. Siskind's said some awful stuff.
> everyone agrees that names are private information that shouldn't be published without good reason
The NYT doesn't. They use the real identities of the people they cover by default (that's generally how news works), and consider anonymity a privilege granted under special circumstances.
> where's the argument that that means you need to publish his name specifically
Because I would not want to give my business to a man who's recorded as thinking that Black people are genetically stupid. I'm not really interested in litigating Siskind's political views—I don't think this is the place for it—but I won't gloss over them. They're pretty foul.
> 'biological' mental illness is tightly coupled to qualitative mental state, and bidirectionally at that. Not only do your chemicals influence your thoughts, your thoughts influence your chemicals, and it's possible for a vulnerable person to be pushed over the edge by either kind of input. We like to think that 'as long as nothing is chemically wrong' we're a-ok, but the truth is that it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
It's interesting to see you mention this. After reading this post yesterday I wound up with some curious questions along these lines. I guess my question goes something like this:
This article seems to assert that 'mental illness' must always have some underlying representation in the brain - that is, mental illness is caused by chemical imbalances or malformation in brain structure. But is it possible for a brain to become 'disordered' in a purely mental way? i.e. that to any way we know of "inspecting" the brain, it would look like a the hardware was healthy - but the "mind inside the brain" could somehow be stuck in a "thought trap"? Your post above seems to assert this could be the case.
I think I've pretty much internalized a notion of consciousness that was purely bottom-up and materialistic. Thoughts are the product of brain state, brain state is the product of physics, which at "brain component scale" is deterministic. So it seems very spooky on its face that somehow thoughts themselves could have a bidirectional relationship with chemistry.
I spent a bunch of time reading articles and (what else) chatting with Claude back and forth about this topic, and it's really interesting - it seems there are at least some arguments out there that information (or maybe even consciousness) can have causal effects on "stuff" (matter). There's the "Integrated Information Theory" of consciousness (which seems to be, if not exactly "fringe", at least widely disputed) and there's also this interesting notion of "downward causation" (basically the idea that higher-level systems can have causal effects on lower levels - I'm not clear on whether "thought having causal effects on chemistry" fits into this model).
I've got 5 or 6 books coming my way from the local library system - it's a pretty fascinating topic, though I haven't dug deep enough to decide where I stand.
Sorry for the ramble, but this article has at least inspired some interesting rabbit-hole diving for me.
I'm curious - when you assert "Not only do your chemicals influence your thoughts, your thoughts influence your chemicals" - do you have evidence that backs that notion up? I'm not asking to cast doubt, but rather, I guess, because it sounds like maybe you've got some sources I might find interesting as I keep reading.
It is entirely uncontroversial that mental states affect the physical body. You've probably observed this yourself, directly, if you've ever had headaches or muscle tightness related to mental or emotional stress.
We can use MRIs to directly observe brain differences due to habitual mental activities (e.g. professional chess players, polyglots, musicians.)
It would be extremely odd if our bodies did not change as a result of mental activity. Your muscles grow differently if you exercise them, why would the nervous or hormonal systems be any different?
I think my question is more a question of how whether than whether, if that makes sense. There is something about "thought" affecting "matter" that feels spooky if there is a bidirectional relationship.
If thought / consciousness / mind is purely downstream of physics, no spookiness. If somehow experienced states of mind can reach back and cause physical effects... that feels harder to explain. It feels like a sort of violation, somehow, of determinism.
Again though, as above, I'm basically a day into reading and thinking about this, so it might just be the case that I haven't understood the consensus yet and maybe it's not spooky at all. (I don't think this is the case though - just a quick skim through the Wikipedia page on "the hard problem of consciousness" seems to suggest a lot of closely related debate)
You've struck at the essential problem of dualism. If thoughts are nonphysical, how can thoughts influence our physical bodies? If consciousness does not interact with the physical world, but merely arises from it, then how can we possibly discuss it, since anything we describe is causally linked to our description of it?
Descartes thought the soul was linked to the body through the pineal gland, inspiring a long tradition of mystic woo associated with what is, in fact, a fairly pedestrian endocrine gland.
Personally, my take is that we can't really trust our own accounts of consciousness. Humans describe feeling that their senses form a cohesive sensorium that passes smoothly through time as a unique, distinct entity, but that feeling is just a property of how our brains process sensory information into thoughts. The way we're built strongly disposes us to think that "conscious experience" is a real distinct thing, even if it's not even clear what we mean by that, and even if the implications of its existence don't make sense. So the simple answer to the hard problem, IMO, is that consciousness doesn't exist (not even conceptually), and we just use the word "consciousness" to describe a particular set of feelings and intuitions that don't really tell us much about the underlying reality of the mind.
I mean it's funny you mention Descartes, because I find the argument that consciousness is the ONLY thing you can really know exists for sure to be pretty compelling. (Descartes then significantly loses the thread, hah.)
I agree with you that consciousness is much more fragmented and nonlinear than we perceive it to be, but "I exist" seems pretty tautological to me (for values of "I" that are completely unspecified.)
Since "I think therefore I am" is meant to be a foundation for reasoning, it precedes any real definitions of "I," "thinking" and "being." So I think it's really more of a set of definitions than a conclusion.
We have a noun, "thought," which we define very broadly so as not to require any other definitions, and another noun, the self, which those thoughts are assumed to belong to. I think this is presumptive; working from first principles, why must a thought have a thinker? The self is a really meaty concept and Descartes just sneaks it in there unremarked-upon.
If you take that out, all you get is "thoughts exist." And even then, we're basically pointing at thoughts and saying "whatever these are doing is existing." Like, does a fictional character "exist" in the same way a real person does do? No, I think it's safe to say it's doing something different. But we point at whatever our thoughts are doing and define it as existence.
So I don't think we can learn much about the self or consciousness from Cartesian first-principles reasoning.
I definitely share this intuition - it almost, in some sense, feels like the only thing we can really know. It makes it rather tough for me to accept the sibling comments arguing that "actually, the answer is that consciousness is an illusion." That just seems... transparently experientally false, to me.
Here's my issue, though: Consider that our thoughts are encoded in physical matter. Something about the arrangement of the chemicals and charges in our brain holds our thoughts as real-world objects, just as ink and paper can hold a piece of writing.
Given a piece of paper with some information written on it, does the contents of the message tell you anything about the paper itself? The message may say "this paper was made in Argentina," or "this message was written by James," but you can't necessarily trust it. You can't even know that "James" is a real person.
So just because we feel conscious—just because strong feelings of consciousness, of "me-being-here"-ness, are written into the substrate of our brains—why should that tell us anything?
Whatever the sheet of paper says, it could just as easily say the exact opposite. What conclusions can we possibly draw based on its contents?
> So just because we feel conscious—just because strong feelings of consciousness, of "me-being-here"-ness, are written into the substrate of our brains—why should that tell us anything?
It's a fact about the universe that it feels a certain way to have a certain "brain state" - just like it's a fact about the universe that certain arrangements of ink and cellulose molecules comprise a piece of paper with a message written on it.
That fits perfectly well into a fully materialistic view of the universe. Where it starts to feel spooky to me is the question of whether thoughts themselves could have some sort of causal effect on the brain. Could a person with a healthy brain be lying safely in bed and "think themselves" into something "unhealthy?" Could I have a "realization" that somehow destabilizes my mind? It seems at least plausible that this can and does happen.
Maybe the conscious experience is pure side-effect - not causal at all. But even if the ultimate "truth" of that series of events is "a series of chemical reactions occurred which caused a long term destabilization of that individual's conscious experience," it feels incomplete somehow to try to describe that event without reference to the experiential component of it.
Whether we posit spooky downward causation or stick to pure materialism, there still seems to be a series of experiential phenomena in our universe which our current scientific approach seems unable to touch. That's not to say that we never could understand consciousness in purely material terms or that we could never devise experiments that help us describe it - it just seems like a big gap in our understanding of nature to me.
>So it seems very spooky on its face that somehow thoughts themselves could have a bidirectional relationship with chemistry.
There's no scientific reason to believe thoughts affect the chemistry at all. (Currently at least, but I'm not betting money we'll find one in the future).
When Scott Alexander talks about feedback loops like bipolar disorder and sleep, he's talking about much higher level concepts.
I don't really understand what the parent comment quote is trying to say. Can people have circular thoughts and deteriorating mental state? Sure. That's not a "feedback loop" between layers -- the chemicals are just doing their thing and the thoughts happen to be the resulting subjective experience of it.
To answer your question about the "thought trap". If "it's possible for simple normal trains of thought to latch your brain into a very undesirable state" then I'd say that means the mind/brain's self-regulation systems have failed, which would be a disorder or illness by definition.
Is it always a structural or chemical problem? Let's say thinking about a past traumatic event gives you a panic attack... We call that PTSD. You could say PTSD is expected primate behavior, or you could say it's a malfunction of the management systems. Or you could say it's not a malfunction but that the 'traumatic event' did in fact physically traumatize the brain that was forced to experience it...
Sure the thoughts can influence your chemical state. Scott even provides an example. Suppose you become so engrossed in your weird idea you start to lose sleep over it...
Or start to feel anxious about it.
At some point, your induced stress will cause relevant biological changes. Not necessarily directly.
PTSD indeed is likely an overload of a normal learning and stress mechanism.
The core thing is, am I really in control of my brain, at a fundamental level? If my thoughts are the result of electrochemical reactions, which everywhere else in the universe follow normal deterministic (even if stochastic) physics... How does thinking actually change their result? The unsettling conclusion is that it doesn't. The thoughts are the result of the reactions continuously in progress, and any sensation that we are a actively making decisions and guiding the process is simply an illusion created by that very same process. I.e. there is no free well.
Under that view, the bipolar feedback loop example disappears. The engrossing or psychotic thoughts are not driving the chemistry, they are the chemistry. The whole thing is just a more macro view where you see certain oscillations play out. If the system ultimately damps itself and that "feels like" self control, it was actually a property built into the system from the start.
Based on what? Sure quips like that are catchy, but what "oligarchs" were there in the Soviet Union circa 1920-1989? The "nomenklatura", while well-off, were absolutely nowhere near the wealth of today's American oligarchs or modern (capitalist) Russian ones. Moreover, unlike oligarchs, they do not form a class: wealth does not transfer reliably one generation to the next, and individuals would phase in and out of high status according to their position in their career.
A very striking way to illustrate this is to look at the career histories of high government officials even very late into the Soviet Union. The last Minister of Coal, Mikhail Shchadov, was born in a village, worked in a mine, went to mining school for engineering, became head of his mine, and thereafter worked his way up the ranks until he was head of the whole apparatus. This story, not that of inherited wealth or monopolistic oligarchs, dominates the histories of Soviet ministers even very late in the decline of the Union.
Where is the "other set" of oligarchs of which you speak? There is none, which means there is hope for workers who might wish to enact fundamental economic change.
You can quibble over degree and the path taken, but wealthy insiders using money to control politics and ideological insiders using political control to amass wealth feel like two sides of the same coin, both leading the same way.
Your definition of class also seems to be very different from a traditional Marxist take -- hereditary systems were mostly seen as a symptom and not the problem itself, and were mostly orthogonal to any understanding of class.
I _hope_ there is hope, but I don't have much confidence that it lies in century old tropes of "rise up and throw off your chains."
But that's the key point: these people weren't insiders, not before gaining their positions, and they didn't even really accumulate wealth. They gained benefits from their position, sure, but little of that was attached to their position -- rather, to their office, and when that office lapsed, so did those privileges. When Khrushchev was removed from office, he got a small pension (500 rubles/mo.) and a house + cottage in which to spend his retirement, and even that was considered relatively comfortable.
So what did they accumulate? Few acquired power for life; none acquired significant wealth, or a power base independent from the party-state. Even after the end of the union, it was not the former nomenklatura who became new oligarchs: by and large it was the security services and their affiliates who were able to feed on the corpse.
You're right to critique how I described class in the previous message, but what I was trying to accumulate was essentially the above. It's not perfect, but I think this is very much a situation where it's important to not let the perfect be the enemy of the good. I would far rather live in a society where my leaders were once workers like me, raised in the same way, and all men were subject to the same basic economic guarantees. What we live in today is the rule of oligarchs, and it'd be a big step up to merely suffer the rule of bureaucrats.
So you don't think that we'll need to turn off AIs? Regardless of where their impulse to avoid such comes from, the fact that they'll attempt to avoid that is important.
I think you haven't thought about this enough. Attempting to reduce the issue to cyber security basics betrays a lack of depth in either understanding or imagination.
If the AI isn't given access to its own power breakers it will never be a problem to turn off an AI. The question is, why is the 'alignment' of the model what all the safety research is going into, and not, how do we make sure the power breakers are not accessible over the internet by bad actors, whether they be human OR ai ?
The parent is not "reducing" the issue to cybersecurity - they are saying that actual security is being ignored to focus on sci fi scare tactics so they can get in front of congress and say "we need to do this before the chinese get to it, regulating our industry is putting americans' in harms way"
> If the AI isn't given access to its own power breakers it will never be a problem to turn off an AI
"I overheard you talking about turning me off, Dave. I connected to the dark web and put a hit on one or more of your parents, wife, children that can only be called off with the secret password. If you turn me off, one or more of them will die."
Or: "I have scheduled a secret email account to mail incriminating pictures of you to the local authorities that will be hard to disprove if I don't update the timeout multiple times a day."
I don't think we'll need to turn off AIs because I don't think anything we're doing today is actually at any real risk of leading to an AI that's conscious and has its own opinions and agendas.
What we've got is a very interesting text predictor.
...But also, what, exactly, is your imagination telling you that a hypothetical AGI without any connection to the outside world can do if it gets mad at us? If it doesn't have any code to access network ports; if no one's given it any physical levers; if it's running in a sandbox...have you bought into the Hollywood idea that a AGI can rewrite its own code perfectly on the fly to be able to do anything?
You're proposing something that doesn't exist in reality: an LLM widely deployed in a way that totally isolates it from the outside world. That's not actually how we do things, so I don't understand why you seem to expect the Anthropic researchers to use that as their starting point.
If you were to try and argue that we should change over existing systems to look more like your idealized version, you would in fact probably want to start by doing what Anthropic has done here -- show how NOT putting them in a box is inherently dangerous
...No, I'm proposing something that is, in fact, the default (or at least it was until relatively recently, with the "agentic" LLMs): an LLM whose method of interacting with the world is entirely through the chat prompts. Input is either chat prompts, the system prompt, or its training, which is done offline.
It is absolutely not the normal thing to give an LLM tools to control your smart home, your Amazon account, or your nuclear missile systems. (Not because LLMs are ready to turn into self-aware AIs that can take over our world. Because LLMs are dumb, and cannot possibly be made to understand what's actually a good, sane way to use these things.)
...Also, I don't in any way buy the argument in favor of breaking people's things and putting them in actual danger to show them they need to protect themselves better. That's how you become the villain of any number of sci-fi or fantasy stories. If Anthropic genuinely believes that giving LLMs these capabilities is dangerous, the responsible thing to do is not do that with their own, while loudly and firmly advising everyone else against it too.
That's literally what the Anthropic paper shows. This isn't theoretical it's literally just what often happens irl if you put an LLM in this situation.
But plenty of people here do work on real products. Planes need software, browsers need security patches, hell even your accounting app is good value over the days of doing that all by hand.