I just explained explicitly how and why an LLM is not, and never will be, an "extremely powerful tool".
A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.
Can you give an example of a concrete task that you believe that neither GPT-4, nor its successors, will be able to accomplish within the next 2 years? Keep in mind that ChatGPT Plugins[1] exist.
If you can't think of any tasks that a tool won't be able to accomplish within a 2 year period, I question your definition of "not extremely powerful".
Sure, it can exhibit these behaviors, but each behavior itself never actually leaves the text that humans wrote it into.
That means we are reliant on how well the behaviors we want can be modeled into text patterns.
Unfortunately, natural language is ambiguous. There are many identical patterns that contain different behavior.
GPT's authors introduce some of these patterns as both "feature" and "limitation":
Features:
> The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.
Limitations:
> ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.
> given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
These are not separate. They are two sides of the same coin.
GPT will never be able to look objectively at its own model. It will never be able to subjectively categorize patterns or tokens. If it could, it would not be an LLM: it would be an entirely new technology.
I'm not entirely sure I understand your argument but when you say:
> but each behavior itself never actually leaves the text that humans wrote it into...
I assume you mean that all the LLM can do is produce text so it's not inherently dangerous, but it's rather trivial to hook an LLM up to controls to the outside world by describing an API to it and then executing whatever "commands" it produces on its behalf and feeding the responses back to it. Wouldn't a setup like that let its behavior "leave the text," as you put it? I'm sure curious people are already doing experiments like this.
The LLM doesn't have to have agency to interact with the world in a way that mimics an entity that does have agency, that's what they are already doing with the text they generate, and why people are so quick to personify LLMs to say things like "they reason", "they understand", etc. If the text they produce, which mimics a sentient actor, is embedded in contexts that allow it to affect the world outside of the text interaction, there you have it, the LLM affects the outside world. Wouldn't you agree?
> I assume you mean that all the LLM can do is produce text so it's not inherently dangerous, but it's rather trivial to hook an LLM up to controls to the outside world by describing an API to it and then executing whatever "commands"
Yes, you can do that, but the result is guaranteed to be silly.
The LLM isn't conceptualizing what it reads. That was already done when the human writing it used language patterns to encode their own conceptualization as data.
Instead, the LLM takes an implicit approach to modeling that data. It finds patterns that are present in the data itself, and manipulates that text alrong those patterns.
Some of the LLM's inferred patterns align to the language structure that was intentionally used by the human writing to encode a concept into that data.
Humans look objectively at the concepts they have in mind. From that perspective, we use logic or emotion to create new concepts. If a human could attach their mind to API endpoints, there would be no need to use language in the first place. Instead of encoding concepts into intermediary data (language in text) to send to a machine, they could simply feel and do the API calls.
LLMs don't look objectively at their model. They don't have a place to store concepts. They don't feel or do any arbitrary thing.
Instead, an LLM is its model. Its only behavior is to add new text and inferred patterns to that model. By modeling a new prompt, any familiar text patterns that exist in that prompt's text will be used to organize it into the existing model. A "continuation" essentially prints that change.
When you attach that to API endpoints, the decision making process isn't real. There is no logically derived new concept to determine which API call to call. Instead, there is a collection of old concepts that were each derived logically in separate unrelated contexts, then encoded into language, and language into text. Those are just being recycled, as if their original meaning and purpose is guaranteed to apply, simply because they fit together like puzzle pieces. Even if you get the shape of them right (by following the patterns they are encoded with) there is no place in this process to introduce why, or to decide the result is nonsense and avoid it.
In short, the LLM can be made to affect the world around it, and the world can affect it back; but there is nothing in between it being affected, and it affecting the world. No logic. No intent. Only data.
I am not sure what you mean by "each behavior itself never actually leaves the text that humans wrote it into". Do you mean that it cannot generalize outside of its training distribution, or are you gesturing at something else?
I notice that I am particularly confused by the example of "categorize" -- I am pretty sure you're not saying that ChatGPT can't categorize things, but then I have no good hypothesis for what you do mean.
> I notice that I am particularly confused by the example of "categorize" -- I am pretty sure you're not saying that ChatGPT can't categorize things, but then I have no good hypothesis for what you do mean.
That is exactly what I mean. GPT infers patterns. Some language patterns themselves are categorization, like words surrounding "or". It's important to recognize that humans are creating the categorization in these cases.
GPT does not differentiate at all between one pattern and another. It doesn't categorize the pattern that represents words, or the pattern that represents sentences, or the one that represents an answer in response to a question, etc. These are all symbolic categorizations that humans do to text; and we call that set of categories "language". GPT, ironically, has no concept of language or grammar. It only sees patterns as patterns.
Can you give a concrete example of a way that a human is capable of categorizing, where GPT is not? Some examples of concrete examples might be
- Humans can take a list of things like ["goldfish", "cat", "shark", "tree", "whale", "rock", "human"] and a list of categories like ["animal", "plant", "non-living"], and say which category each thing belongs to. GPT cannot do this.
- Humans can take a list of things like the above, say which categorization scheme is most appropriate for the thing they are trying to do, and then, according to the categorization scheme they said, say what category each of the things belongs in. GPT cannot do this.
- Humans can take a list of things like the above, and describe a novel categorization scheme for them. Humans can then say out which category things that were not in the original list belong to. GPT cannot do this.
- Some other thing where humans can describe how to categorize something but GPT cannot do so.
- Humans do something internally that GPT cannot do, but that thing has no observable effects from outside the system
Portobello mushroom is certainly wrong, I also would have included color for salmon but that one is more up for debate I think. I think some humans would probably write the same categorizations though.
[[[Me]]]
> I will provide two lists of items inside brackets, separated by commas. I want you to categorize the entities in the first list into the categories in the second, keeping in mind that an entity in the first list may fit into multiple categories:
[goldfish, shark, salmon, blue, tiger, cake, cyan, portobello mushroom, tree]
[animal, fish, food, color, mammal, plant]
[[[GPT4]]]
> Here's the categorization of the entities in the first list into the categories in the second list:
Humans can look at a list of words separated by commas, and call it a "list". We can also call it, "not a CSV file".
Humans can look at a 5, and call it five. We can hold up a hand, and say, "this many".
This behavior is named "semiotics". It's the act of using one thing to represent another.
LLMs are designed to intentionally avoid this approach. Instead of constructing an explicit grammar (like a parser), an LLM takes the opposite approach: inference.
Inference makes it possible to model the ambiguous patterns that natural language is made of. Inference also makes it impossible to define symbols.
Humans use both semiotics and inference. So far, it looks like no one has quite cracked how to do that artificially.
It's (to some subset of the audience), an extremely persuasive tool, which is powerful in its own right.
It's actual capabilities mean very little, if some people can be convinced into doing decision making without actually doing any decision making by it (or literally any tool), that's cause for concern.
have you tried gpt-4? how can you possibly say it's not an extremely powerful tool?
what happens when someone determined links it - or thousands of its - or thousands of next year's 3x as powerful its -- up to the internet with a code interpreter and a hard-coded poor intention?
What could it do that isn't already being done? I ask this sincerely. We already have bots constantly attacking every server they find 24/7. We already have botnets
Is the concern that GPT will be able to find and exploit e.g. a novel RCE exploit in SSH?
For example, it could scan social media and use that information to send a very large number of people very personalized messages that manipulate them into doing X. Here, X may include meeting other people to tempt/coerce them into doing Y.
The reason this isn't possible now is that nobody has the time to look at millions of social media profiles and then write millions of personalized messages. To do it, you would need a lot of people who you would have to convince, including paying them. And even so, things like doomsday cults and terrorist organizations already exist.
A program that is actually a person in a black box would require the level of concern you are showing. Thankfully, that is not even remotely what we have here in reality.