The feeling I'm getting is management took such a huge stance (the capex, the name change, etc) on it it'd be humiliating to turn 180 and ditch it now (they've spent 50 billion on Reality Labs so far). So they're quietly pushing it aside and one day when people aren't looking, they'll take it to the back of the house and shoot it in the head. Probably call it an "internal re-org" or something like that.
The only reason I can think that this atrocity got deployed was from an engineering perspective, creating AI profiles to engage theoretically gives FB a powerful lever to increase engagement, because you'd have this entire new field to collect data, experiment and test people's responses. In other words, it would turn engagement into a form tractable to optimisation algorithms.
I'd argue that it hasn't come to pass, but it is not wrong that its a better way to teach. Virtually all education has been set up in an industrial line style of pre-u education -> university -> companies -> profits -> some value to society. Every part has been attempted to be structured to optimise for the subsequent stage - companies optimisie for profits, universities optimise for employment rate and salary, high schools optimise for college entrance rates, etc. Yet because of Goodhart's and organizational incompetence, each part optimises badly, so education gets exponentially disconnected from each. His proposals are valuable to society, but the system is structurally against it.
The only way this works if it skips the assembly line right down to being of value to society, ie you set up a system which explicitly transforms that teaching into things like startups, research, nonprofits etc.
From the amount of data each successive generation used (which grew many orders of magnitude each time) to the decreasing, logarithmic performance, it's quite clear the steam is running out on shoving more data into it. If one plots the data to performance graph, its horribly logarithmic. In another perspective, the ability of LLMs to transfer learning actually decreases exponentially the larger they and the data sets get. This fits into the how humans have to specialise in topics because the mental models of one field is very difficult to transfer to another.
We are likely going to get there. Similar to the steam/combustion engines (and other core technologies like computers, wireless transmission etc) there's first a massive rush to increase the power of it, at the cost of efficiency and effectiveness for more niche use cases. Then it is specialised to various use cases with large improvements in efficiency and effectiveness. My own prediction for where most gains will now come is
1) Creating new "harnesses" for models that connect to various systems, APIs, frameworks, etc. While this sounds "trivial", a lot of gains can come from this. Similar to how the voice version of ChatGPT was (apparently) amazing, all you really had to do was create an additional voice to text layer and another text to voice layer.
2) Increasing specialisation of models. I predict over time that end user AI companies (e.g those that just use models and not develop them), will use more and more specialised models. The current, almost monolithic, system where every service from text summary to homework help is plugged into the same model will slowly change.
We kind of have, that's what fine tuning is trying to achieve.
We haven't seen wholesale specialised models yet because creating foundation models is expensive and difficult and the current highest ROI is to make a general model.
In what measure, loss? Loss can't go below 0 plus the inherent entropy in the text (other than that with overfitting it could reach nearer to 0, but not fully if it is next token and there are multiple same prefixes).
With respect to hallucinations 4 got incredibly better over 3
In intelligence/performance. It's admittedly a fuzzy notion. Most benchmarks will probably show decreasing gains between generations. Similar to time/space complexity, trying to debate about what performance/intelligence is will get into a million definitions, caveats and technicalities. But a relative comparison between inputs and outputs is gives us useful information.
The inputs - data, compute and parameters - going into training these models have grown by many orders of magnitude between each gen. There's a lot of fuzziness about how much better each gen has gotten, but clearly 4 is not many orders of magnitude better than 3 by any reasonable definition. This mental model isn't useful to say how good each gen is, but it is quite useful to see the trend and make long term predictions.
You'd think after a decade of writing forms and form validation on the internets I'd get better at it, but nope, have to reinvent the wheel every time for every framework/form library because nothing is ever finished or good enough. I had hopes HTML5 would fix it but it didn't.
You sound much more experienced as a web developer than me, so you probably know this exists, but just trying to be helpful I want to be sure you know about client side form validation in html5 https://developer.mozilla.org/en-US/docs/Learn/Forms/Form_va...
I appreciate this doesn't cover anything more than the basics, so something like the normal behaviour of comparing two password fields for the same content doesn't work, but I find these controls are useful for getting something simple up and running.
Something interesting I heard from a chip company that's huge in the space and work intimately with many automotive companies - why Chinese companies grow so fast is because their development cycle for a car is ~2-3 years, compared to traditional manufacturers who take 5-7 years. This is a massive edge in pushing out new features and exploiting the very rapid new tech - batteries, self driving, etc.
Stellantis is the perfect illustration: they kept putting all efforts into pushing their flawed ICE engine (PureTech) because they wanted to make profits from it for at least 10 years. And now the efforts are still not on making a good affordable car but on lobbying to revoke the EU ban on the sale of petrol and diesel cars.
There exists a third possibility where the US and China sign an under the table deal for China to invade, the US to saber rattle and China to allow the flow of chips to continue. The present direction seems to be the US is "de-risking" from Taiwan by moving chip production to the US, so if China does invade they aren't caught in a bind.
> The present direction seems to be the US is "de-risking" from Taiwan by moving chip production to the US, so if China does invade they aren't caught in a bind.
This idea that the US is protecting Taiwan for its semiconductor prowess (aka the "silicon shield") is a very confusing idea to me as it ignores the period from the 40's to the 90's when Taiwan had no semiconductor manufacturing that wasn't being done as well or better elsewhere, yet the US was a ardent supporter, to the point of almost entirely shunning the People's Republic of China over it.
It's a smart sounding idea (especially if you don't know your 20th century Chinese history) but the facts just don't back it up.
China didn't have ICBM capability until like 1980. There was much less risk declaring Taiwan support until then, and after that China feigned liberalization just enough to not be seen as a threat.
I am reasonably familiar with it, you're missing mentioning why the US defended Taiwan initially. The US stopped any formal defense of Taiwan in 1980, not to the 1990s. During the period it did have a defense treaty it was within the context of creating an anti communist bulwark, and such defence treaties (even with highly suspect partners) was the bread and butter of geopolitics in the Cold War. It was not because of some deep love of Taiwan or democracy, in fact Taiwan would only become a democracy after the defense treaty was terminated. And "ardent supporter" is a very tricky term, the US signed a declaration for the One China policy, which is completely against the idea of Taiwan as a separate nation(although at the time the KMT and people largely did not view Taiwan as an separate nation).
The silicon shield is undoubtedly a significant part of the calculus around Taiwan, especially wrt direct military intervention. Successive administrations have clearly shown their emphasis on maintaining access to key strategic resources (the Middle East and oil).
What One China policy did the US sign? China has a One China Principal which some countries uphold, but the US is not one of them. Our _policy_ is to acknowledge China’s position but does not endorse or challenge it. Chinese diplomats are eternally trying to conflate the two, saying even the US admits there is but one China, and thus supports China’s position on the Taiwan question, but that is not true.
As for the silicon shield, yes it’s probably a major factor, but not close to the only factor. If PRC realizes its claims over Taiwan and the South China Sea, the entire region, including many US allies, will be under Chinese hegemony. It would spell the end of the current economic order. Japan, Philippines, basically all of East and Southeast Asia would then be trading within China’s new backyard.
If one gets into the weeds of the whole messy thing, the Shanghai Communique acknowledges the position of both sides (Taiwan and China) that there is one China, the 1982 Joint Communique acknowledged the Chinese position and said the US had no intention of pursuing a "two Chinas" or "one China and Taiwan" policy. And of course there's a non stop stream of such political speak, which is a red herring for the whole issue. Beyond minor changes to wordings, the main things the US did that screwed over Taiwan:
1) Revoking the mutual defense treaty which legally bound the US to defend Taiwan, replacing it with a more vague law where military intervention was not clear.
2) Recognizing the PRC as the legitimate representative of China.
To align your claims slightly, Taiwan also claims the same section of the SCS (actually it claims a slightly larger part), so strictly speaking it is what happens if the "Chinese claims" are realised (Taiwan and China work jointly to support their claims, ironically enough). In real terms the economic effects of China with respect to the SCS are greatly exaggerated for a number of reasons. First being that shipping can be routed through Indonesia. It is not a chokepoint, it just happens to be the shortest route. Second, blockades have little to do with recognising swaths of ocean as territory. These are enforced by navies, and most blockades in history have not happened within the blockaders' own waters (for obvious reasons). It is no easier for China to blockade trade in the region if it claims the SCS. And third, of all the major economic powers, China has historically been the least likely to enact economic warfare like blockades or sanctions. There is also a fourth aspect where in the current political environment, the globalised, trade based economic order is the least popular in the US and assorted European states, not China (who in fact desperately needs trade).
The politics of the US are very different from the 40's to 90's. Good luck explaining to the American public why a bunch of their kids need to go die defending the Taiwanese in 2024 unless you can base it in some cold hard economic reality.
The reason we supported them back in the 20th century is because the Red Scare was the big boogeyman of the time and we needed military bases and friendlies in that part of the world.
China was a commie state like the USSR countries back then. I don't see how the west could have treated one as the idiologocal archenemy and nemesis and not extent the same animosity to the other.
Back when? China, despite being a "commie state" (still is btw) split up with USSR by 1969 then struck a deal with Nixon to partner up with the US to become a manufacturing hub for the States.
There is believed to be a software remote disable in the ASML provided machinery but it doesn't have to be effective - an invasion would cut off the machines from the maintenance they require https://www.theregister.com/2024/05/21/asml_kill_switch/
It's completely speculative for us, but I'd point out only the US has incentive to do that, not Taiwan. The only reason Taiwan would do it is if they adopted a scorched earth policy like Saddam and the burning Kuwaiti oil fields. But even with a psychopathic dictator like Saddam, he never burned his own oil fields.
Not even necessary. The equipment will cease to function if it is cut off from ASML servers in the Netherlands. But yes, a destruction protocol exists for most strategic resources located near adversaries that are considered critical to national security.
Weapons systems and strategic production capabilities are the one area where I think that DRM actually makes sense. Not printer cartridges and coffee. FFS.
But taiwan has nukes now in all but name? As the us becomes a non-reliable ally everyone with money and a shopping list bordering a totalitarian country heads for Pakistan ?
Since when has Taiwan had nukes? There was a nuclear program in the 80s and the US pressured them to abandon it. Even civilian nuclear engineering is not very popular in Taiwan. I’d be kind of surprised if Taiwan had any kind of nuclear weapons program and it wasn’t an international crisis, given how thoroughly each side of the straight spy on the other.
That is false equivalence, there's a class of things that should rightfully be discouraged from military use (chemical/biological weapons, land mines, phosphorous weapons) because of the significant harm/side effects they cause beyond some definition of acceptable.
Relatedly, when I ask Llama how to make red phosphorus it tells me off. What's the military going to do with an AI that refuses to help write plans on how to invade a country or kill an enemy. Then again, maybe its military masters are better at threatening it than I and can jailbreak it more easily.
Thanks for pointing that out. I think this clearly shows the difference between 'allowing use' and 'supporting a use case' when it comes to bio-tech. Vaccinating a soldier is allowing use, creating a biological weapon is supporting a use case. I wouldn't want to work for a bio weapons manufacturer but working for a vaccine maker would be just fine with me even if my efforts were used by the military.
That seems like a good idea. I am puzzled by what benefit the RL has in OP. It seems like a well defined constraint optimisation problem that could be done without RL, for example in the way you mentioned.
