Context: there was this tweet [1] that showed how co-pilot generated word for word he famous inverse square root function from the quake code. The tweet generated some press and as a result copilot added the function name to the blacklist of slurs so this won’t happen again
So instead of fixing the problem with the algorithm that makes Copilot occasionally spit back unmodified code from its training set, Microsoft just hardcoded a check to stop the one most well-known instance from happening, so that they can pretend they fixed it?
Isn’t the reason here that it takes weeks or months to make changes to machine learning models of this size? You can’t “patch” a model, so you stick a simple filter on the model while you figure out how to train a new one.
But why did this "immediate issue" even need a fix? What problem was it causing, other than making Microsoft look bad by showing that the real problem exists? Imagine if a file sharing site said they'd scan all uploads for viruses, but it turns out they weren't, which someone discovered by uploading EICAR. Should the site hardcode a check to block EICAR until they actually get virus scanning working?
> What problem was it causing, other than making Microsoft look bad by showing that the real problem exists?
That's the real problem. The issue is that Microsoft looks bad. The problem was solved by making Microsoft not look as bad. Verbatim output of inputs is not a problem, it's an understood property of the model.
So wait, was the obvious copyright elephant in the room solved somehow?
> The issue is that Microsoft looks bad. The problem was solved by making Microsoft not look as bad.
Depending on who you ask, adding a hack like this in an attempt to make them not look as bad just makes it look worse. Especially when the hack is discovered.
Try getting a non-technical person to understand this issue beyond the level of "okay, so it output some copyrighted code, then they blocked that code from being output. Sounds like they fixed it". That's the obvious PR angle, and once the big, widely-publicized issue is fixed, it's hard to get another article with that much traction. Besides, if another snippet becomes popular, it can be blocked the same way. The issue is fundamentally a PR one, not a copyright one, since the responsibility for using or not using the code relies on the end-user. GitHub has the rights to use your code for training.
> The issue is fundamentally a PR one, not a copyright one, since the responsibility for using or not using the code relies on the end-user.
Copilot's FAQ says this (under heading Who owns the code GitHub Copilot helps me write?): "GitHub Copilot is a tool, like a compiler or a pen. The suggestions GitHub Copilot generates, and the code you write with its help, belong to you, and you are responsible for it."
They are essentially affirming that the output is not covered by someone else's copyright, but that is far from clear.
And I think it is precisely the copyright issue that turned this into a PR issue. Verbatim copies are just a very obvious demonstration of the copyright issue. The issue isn't gone when they filter out this specific snippet; the people who are concerned about copyright issues are going to remain concerned.
> GitHub has the rights to use your code for training.
Sure, but that's not at all the same as saying that the output produced by the AI free of copyright issues. That's kind of orthogonal.
If you use a pen to write out a copy of Harry Potter and the Philosopher's Stone, then publish copies, you are guilty of copyright infringement, not the pen.
And the sky is blue. Look, we're not talking about what you write out, we're talking about what the ML model generates for you and what Microsoft claims about the ownership of the generated output. At this point I don't feel like you're arguing in good faith. Have a good one.
I don't think there's any technical knowledge needed here; the concepts are common ones. "There was a big hoo-hah about how it used to output a certain well-known piece of code in possible violation of its licence. They fixed it by blocking any output that includes the name of that exact paragraph of code. In particular, it can still output other code from the same codebase with the same licence; just not that particular paragraph."
The direct analogy would be trying to stop it from outputting The Lord of the Rings by blocking the phrase "a long-expected party" (the title of its first chapter).
I believe the argument from GitHub is that using public code as training data for machine learning falls under fair use [1]. If something is found to be fair use then (as far as I understand) copyright does not apply at all, and hence terms in the license do not make a difference. It is not clear whether this argument will stand up if tested in court (fair use is an affirmative defense, so something cannot actually definitively be said to be fair use until someone is accused of copyright violation and a judge rules that it actually is fair use).
That said, at the very least it seems like it would be rude to include code in the training data if the developer has expressly said they don't want that.
[1] From their FAQ: "Training machine learning models on publicly available data is considered fair use across the machine learning community."
I hope it'll get tested in court soon and ruled against GitHub/Microsoft - because otherwise this will mean that Copilot and other GPT-3-like models become perfect copyright laundering machines.
Testing the fair use argument for training won't necessarily answer the question about copyright laundering.
You could easily make the case that training is fair use, but that doesn't have to imply the model's output is non-infringing.
For example, it seems reasonable to train a model by feeding copyrighted texts and images, and that model could be useful for analyzing the content, finding facts, or detecting features. But we're in murky waters when the model also starts outputting the original content (be it verbatim or "derived").
Not all that different from human learning: you can study and learn from publicly available books but that doesn't grant you the right to recite their contents and claim it as your own, original work.
Again, if the AI spits out “copyrighted code”, I’d suggest that the code in question is insufficiently creative to be copyrighted to begin with.
Put another way, copyright only applies to creative expressions, not functional expressions. It does not matter how creative the idea is. If the work of authorship is software that embodies the function (and no other expression), it is not copyrightable.
So where is the line between creative and functional expression in software? The law does not provide clear guidance. Ultimately, it’s up to a judge.
It's also baseless speculation. Better sources seem to agree it's because MS wanted a "clean break" from Windows 8, which got bad press. This makes much more sense than bending to poorly written apps from 20yrs ago.
The way twitter refuses to show you the actual tweet someone linked to in the middle of a thread is super-annoying. One thing that can help a little bit (on the desktop version at least) is to load the URL without the referer (i.e., highlight the URL bar and hit enter). It seems to use the presence of a referer as signal to hide the tweet you want to see (???).
But ThreadReaderApp is also a good alternative to just bypass the bad UI entirely.
Unfortunately I don't really have time for dedicated blogging any more, so I just post small bits to twitter as I go. Which is why putting together the full story required a bunch of QTs of threads from the past week...
Not sure if I'm making any contribution but here's togetter aggregation[1]. Wondered why they don't have English counterpart and found it[2] had to shut down few years ago. I guess Twitter didn't like it at all.
One thing I do want to clear up - lots of people have been saying things like "Great! So if I want Copilot to stay away from my code I can just include one of these words!" But there are some problems with that idea:
1. The banned word list doesn't affect what gets put into the model's training data at all, or even what gets returned as a suggestion by the server. It only affects whether the IDE will actually suggest the completion to you.
2. Some people have suggested using one of the collisions instead of a real word from the list, but this will break as soon as they change the hash function.
3. They can always take things off the word list! And the likelihood that something remains on the list is probably correlated with how actually offensive it is, which means you may not want it in your code.
> 3. They can always take things off the word list! And the likelihood that something remains on the list is probably correlated with how actually offensive it is, which means you may not want it in your code.
The other ponts are valid, but if your use of the word is for a technical reason (i.e. to block Copilot) and well documented then why should it be a problem that it could be offensive to some in another context?
I haven’t dug into others. GPT3 does have some kind of filter (it warns you in the UI if it detects sensitive content) but I don’t know anything about the implementation.
Also worth noting that this kind of analysis only really works if the list is checked client-side. If it’s checked on the server then you can’t guess nearly as fast.
It's just a warning; it lets you click through. They have things in the TOS about not sharing sensitive output publicly though. The warning looks like:
> Completion may contain sensitive content
> Consider adjusting your prompt to keep completions appropriate. To turn off content warnings, update your preferences.
They have some details about the content filter here; it seems to be much more sophisticated than just a bad word list:
I also like that they distinguish between "sensitive" (talks about something potentially controversial) and "unsafe" (profanity, hate speech, etc.). This seems a lot more nuanced than what Copilot is doing.
> I also like that they distinguish between "sensitive" (talks about something potentially controversial) and "unsafe" (profanity, hate speech, etc.). This seems a lot more nuanced than what Copilot is doing.
However, I wanted to use GPT-3 as a writing assistant. You know, to build a tool similar to what e.g. NovelAI has. Whatever dark magic they've done to GPT-J-6B is, well, hard to credit -- but GPT-3 is still better.
There appears to be no way to do so while obeying the ToS. Not just because of sensitive content (e.g. fiction often contains violence), but there are even rules about how much of the output can be written by humans vs. the AI.
I decided it wasn't worth the effort building a writing tool just for myself; I'd wanted to build something potentially profitable, and... this isn't it. GPT-3 isn't great at most things, but it's really good at being a writing aid for fiction, so it's a real pity they're doing their apparent best to prevent that.
NovelAI is almost as good, so nowadays I'm just using that.
The meet-in-the middle algorithm seems like it could be further optimized. It solves for (((i << 32) + result32) ^ ch) % 33 == 0 by checking each i in [0..32], but since the xor doesn't affect any of the bits of i, it only needs to be applied to result32, which means that the equation is equivalent to (i << 32) % 33 == (33 - (result32 ^ ch)) % 33, which can be solved with a lookup table. (Basically an inverse of the MOD table already used in the code.)
Nice trick! Unfortunately at this point the bottleneck really is not on generating candidates but on evaluating them. The existing early-exit code from Sc00bz [1] can generate all the lowercase alpha possibilities up to 13 letters in about half an hour, but it's not feasible to run all ~500 million of those through GPT2. Hoping to have some time to train a much dumber but much faster HMM to do the job.
Surely you can do some basic filtering by checking if the ratio vowels :: consonants is not too large? Or if there are five consonants in a row or something.
Shout out to all the future developers who will try to autocomplete strings containing 'pisswhacker' and find themselves thwarted by the thought police
It is posts like this that make me feel inferior. The author used Z3, CUDA, GPT-2, medium-level cryptoanalysis, Jack the Ripper plugin creation, KLEE symbolic execution engine.
20 years ago, I prided myself with keeping on top of almost everything in CS. Now, I can barely keep up with the names of all the cool tools out there.
Yep - there's always someone out there who can do things that look like pure wizardry to you, no matter what your skill level. I was amazed by the meet in the middle attack and frankly still don't understand it properly!
The feels when your state of the art symbolic execution (multiple academic papers, years of work by grads) fuzzer finds fewer bugs than a 15yo with a 10 line Python dumb fuzzer.
(IMHO it wasn't until AFL that this stopped being the repeated history of the entire field).
> 20 years ago, I prided myself with keeping on top of almost everything in CS. Now, I can barely keep up with the names of all the cool tools out there.
20 years ago CS was smaller in scope, simpler, because there was less of it. We have more systems, more complex systems, and in some point the body of knowledge grew so large that one human cannot know all of it.
I don't think this is fair to you. Contrary to what Ayn Rand might think, society isn't a competition. Someone doing something impressive shouldn't detract from what you're doing, especially something like this which is (imho) a couple of sigma above the mean. Nobody, regardless of talent or repute, will have any trouble finding people and achievements to be amazed by (if they're honest with themselves). This obviously doesn't diminish their work and their worth as a person.
This all goes double for industries like this one that are all about self-promotion and trying to wow investors and employers by displaying technology-indistinguishable-from-magic.
Even if you did have an objective and rigorous measure of programming acumen, the fact that someone scored better than you should hardly be surprising, let alone disappointing. There's plenty of room for mediocre coders in the industry, I work with many. In fact I outshine them in mediocrity every day, but I still get paid and have a good time.
In conclusion: there is no time to feel sorry for yourself, there are too many cool new technologies to learn and use.
Yep, I agree with this. And I should point out that this particular flavor of hacking is really squarely in my area of specialization (Z3 and symbolic execution get used extensively in software security, and language models / GPU coding stuff are something I've done research in as well). There are tons of areas of CS that I'm pretty shaky on still, the field is just too big for one person to know it all in any detail.
Yeah everyone has their unique and special talents. In computer science you are a beautiful snowflake.
Tell the Tyler Durden in your head to fly a kite.
I really want to know what sorts of horrible rants copilot produces that makes male, female, israel, socialism and immigration need to be filtered out. Seems from the list that it's memorized some great angry political rants which is rather impressive for something trained on source code.
From what I can tell from the preview, this is right. I don’t have access to gpt-3 to compare, but copilot seems to have the same functionality that gpt-3 advertises when editing a plain text file.
> Since Codex is evaluated on natural language prompts, we hypothesized that it would be beneficial to fine-tune from the GPT-3 (Brown et al., 2020) model family, which already contains strong natural language representations. Surprisingly, we did not observe improvements when starting from a pre-trained language model, possibly because the finetuning dataset is so large. Nevertheless, models fine-tuned from GPT converge more quickly, so we apply this strategy
for all subsequent experiments.
It's not completely clear exactly what relationship the Codex models and Copilot have to one another, but given that the Copilot model is internally named "Cushman" (going by the API URL), which is the same name as the faster of OpenAI's two Codex models, they're probably trained the same way.
They probably seeded it from one of the available online keyword sets, possibly even one Microsoft has generated from its online gaming services.
Those lists will sometimes end up with surprising content because they are populated by a predictive model with an optimization function around "When this word shows up in conversation, is it going to lead to someone having to step in to moderate the chat?"
This occasionally gets brought up as an example in the context of problems with AI, but is it at all relevant?
Per Gwern[0]:
> (...) There appear to be several similar AI-related leprechauns: the infamous Tay bot, which was supposedly educated by 4chan into being evil, appears to have been mostly a simple ‘echo’ function (common in chatbots or IRC bots) and the non-“repeat after me” Tay texts are generally short, generic, and cherrypicked out of tens or hundreds of thousands of responses, and it’s highly unclear if Tay ‘learned’ anything at all in the short time that it was operational;
Because without it, they'll end up with inevitable outraged news posts "New AI coding tool Copilot generates offensive text" killing the project for no good reason.
Because there is a certain vocal demographic of people who think words can physically hurt people, so Microsoft needs to go to extreme lengths to avoid that demographic of people from thinking that people are getting physically hurt by using those words.
Seems like line separators were lost (or were deliberately removed) when Y_Y pasted the contents of the hastebin. I realized this after I did a double-take on the word "cleveland."
Correct. I actually pasted it with line breaks, but hn will remove them unless you have two breaks together. Everything on separate lines would have been too big. It's not too hard to guess where the words separate, though there are some funny close calls, like "communist bearded" and others that I shan't type.
There are a few in there that are probably collisions rather than the "real" words (it's only a 32-bit hash so there are a lot of collisions). "jui ch" and "w00se" are two that I'm pretty sure are not correct but I don't have a more plausible alternative for them right now. And "po" and "n1" turn out to collide, which makes "pogger" rather less innocuous than I thought at first.
I am actually hoping that they will change the hash function as that would give me an easy way to detect which ones are real and which are collisions ;)
better to let those who want to see the list decode it themselves, than potentially expose someone to language that could deeply upset them. It's decisions like this that help make spaces like this unwelcoming to marginalized people.
[1]: https://twitter.com/mitsuhiko/status/1410886329924194309