The problem is the same problem with crypto dao projects - cryptographic certainties only apply to mathematical structures; you can't validate that someone actually holds a quality until you can embed that digitally. That turns out to be very hard to do for most things.
Yes, what Zero Knowledge proofs give you however is composability.
Eg suppose you have one system that lets you verify 'this person has X dollars in their bank account' and another system that lets you verify 'this person has a passport of Honduras' and another system that lets you verify 'this person has a passport of Germany', then whether the authors of these three systems ever intended to or not, you can prove a statement like 'this person has a prime number amount of dollars and has a passport from either Honduras or Germany'.
Where I see the big application is in compliance, especially implementing know-your-customer rules, while preserving privacy. So with a system outlined as above, a bank can store a proof that the customer comes from one of the approved countries (ie not North Korea or Russia etc) without having to store an actual copy of the customer's passport or ever even learning where the customer is from.
As you mentioned, for this to work you need to have an 'anchor' to the real world. What ZKP gives you is a way to weave a net between these anchors.
With things like tlsnotary you should be able to prove to a third party anything that you can request over https. I.e. <domain> says that <fact about me>. Or uk-identity.com says that I'm a human and I'm >18 years old. Bank says I can pay for this etc.
As I understand it, you can do arbitrary computations on https responses and prove that you didn't tamper with the response or the computation.
In a mathematical sense - absolutely. You can dual halting problem against many very tangible qualities - like whether a (proved) statement is true or false. A (large-n) halting program is closer to an instantly halting program not just because n is always closer to 0 than inf, but because 'large n halting' and 'instantly halting' are ontologically similar in a way they just aren't with unhalting programs.
An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong?
I feel like everyone is saying 'we're still discovering what LLMs are good at' but it also feels like we really need to get in our collective conscious what they're really, really, bad at.
> An LLM - which has functionally infinite unverifiable attack surface - directly wired into a payment system with high authentication. How could anyone anticipate this going wrong?
If you didn’t catch it, this scenario was fabricated for this blog post. The company writing the post sells vulnerability testing tools.
This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I guess someone who doesn’t know better could connect Stripe and iMessage to Claude Desktop and then give the Stripe integration full permissions. It’s possible. But this post wasn’t an exploit of a real world system they found. They created it and then exploited it as an example. They sell services to supposedly scan for vulnerabilities like this.
> This isn’t what a real production system even looks like. They’re using Claude Desktop. I mean I guess someone who doesn’t know better could connect Stripe and iMessage to Claude Desktop and then give the Stripe integration full permissions.
The core issue here is not whether or not people will connect stripe and iMessage at the same time or not. The issue is that as long as you connect iMessage, attackers can call any arbitrary tools and do what they want. It could be your Gmail, Calendar, or anything else. This is just showcasing that Claude can not distinguish between fabricated messages and real ones.
Even if this is a fabricated system, there are all sorts of sensitive things that might be made accessible to an LLM that is fed user-generated data.
For instance, say you have an internal read-only system that knows some details about your proprietary vendor relationships. You wire up an LLM with an internal MCP server to "return the ID and title of the most appropriate product for a customer inquiry." All is well until the customer/attacker submits a form containing text that looks like the JSON for MCP back-and-forth traffic, and aims to exfiltrate your data. Sure, all that JSON was escaped, but you're still trusting that the LLM doesn't get confused, and that the attention heads know what's real JSON and what's fake JSON.
We know not to send sensitive data to the browser, no matter how obfuscated or obscure. What I think is an important mental model is that once your data is being accessed by an LLM, and there's any kind of user data involved, that's an almost equally untrusted environment. You can mitigate, pre-screen for prompt injection-y things, but at the end of the day it may not be enough.
Companies are rushing or skipping a lot of required underlying security controls in a quest to be first or quick to market with what they think is transformative applications of AI. And so far, probably very few have gotten it right and generally only with serious spend.
For instance, how many companies do you think have played with dedicated identities for each instance of their agents? Let alone hard-restricting those identities (not via system prompts but with good old fashioned access controls) to only the data and functions they're supposed to be entitled to for just that session?
It's a pretty slim number. Only reason I'm not guessing zero is because it wouldn't surprise me if maybe one company got it right. But if there was a way to prove that nobody's doing this right, I'd bet money on it for laughs. These are things that in theory we should've been doing before AI happened, and yet it's all technical debt alongside every "low" or "medium" risk for most companies because up until now, no one could rationalize the spend.
The sad thing is it's not even difficult to get right. I've got something launching soon with a couple different chatbots that I'll share with you later, and it would never even have occurred to me to rely on system prompts for security. A chatbot in my mind is just a CLI with extra steps; if the bot is given access to something, the user is presumed to have equal access.
Honestly, I cannot even believe that Stripe MCP exists, outside of maybe being a useful tool for setting up a Stripe test environment and nothing more. I'm terrified of giving an LLM access to anything that is not a text document that I can commit to git and revert if it does something wrong.
> Somehow this site keeps making these posts and making it up front page and people keep sharing the same opinions
You sure? In their 5 month submit history, they’ve got one post with nearly 900 votes, this post, one post with 17, and a handful of others that didn’t break 10. Perhaps you’re confusing it with another site.
This event was predicted by the Oracle of Delphi. Seriously, everyone knew this was just waiting to happen. The pawning will continue until everyone stops water-hosing the kool aid.
Good catch! A minor correction: Magma - M(ultimodal) Ag(entic) M(odel) at M(icrosoft) (Rese)A(rch), the last part is similar to how the name Llama came out, :)
I'm honestly not sure that their analysis passes muster. It seems that the main consideration is that Harris underperformed compared to down-ballot races and that the underperformance was ahistoric. However, the campaign was also ahistoric: she ran as a pseudo-incumbent under an unpopular presidency without as much of the name recognition incumbency usually offers. It seems extremely likely to me that this drop off in early voting numbers is indicative of an exceptionally weak campaign as opposed to widespread (consistent across all swing states) manipulation.
Their specific claim is odd, it's that the record of every machine in the county showed an expected random pattern of votes for the first 300 or so votes ..
( "random" here means more chaotic and unpredictable )
after which there was a more correlated bias toward one candidate that had a stong early trend toward a particular outcome (consistent clumping with little bounce).
The assertion is that this rarely seen in "real free voting data".
I wouldn't know where I was supposed to draw that dotted line if it weren't already there. And I'd expect there to be less variance in vote percentages among machines that processed many votes than those that only processed a couple. But okay, that picture shows that Trump overperformed in the early vote among machines in Clark County that processed many votes (and that Harris overperformed among those that processed few.) Couldn't this effect emerge from the geographic distribution of voting locations? The points at the right of the scatterplot would tend to represent red rural precincts serving many early voters, while those on the left would represent urban areas denser with machines than they are with early voters. (And there are other considerations, such as that Trump voters may have been more likely to show up in person to early vote than to mail in votes. The vote totals by voting method would seem to show this—but, fine, they're under dispute here.)
These analysts acknowledge the "deep red areas" explanation in their pdf, but they handwave it away in an unconvincing way: they say that the same effect doesn't occur for election day voting, only the early vote. But most voting in Nevada doesn't happen on Election Day. According to the data they present, every single voting machine in Clark County processed less than 150 election-day votes, with most well under 100. That is, they'd all be well to the left of the dotted line. So even in the vote-manipulation scenario, these analysts should expect to be seeing no separation effect for the election-day vote. Its absence tells us nothing.
The main consideration is at the beginning: the stats largely resemble the patterns of verified instances of voter fraud, as in Russia and Georgia.
It seems that you're suggesting some fairly obvious factors working against Harris weren't considered by an organization whose entire purpose is to sniff out voter fraud. Are you suggesting that they overlooked such an obvious detail, or that they're willfully ignoring it?
Another commenter has posted a colab notebook. You've been dismissive in every post, but based on your own value judgments and not based on statistical analysis. At least copy and paste their colab notebook (and read the original analysis) before dismissing.
You are missing most of their analysis. The surprising anomaly (the so called "Russian Tail") appears in early votes but not the Election Day votes or mail-in ballots. There analysis is worth reading again to catch what you missed. Another commenter has posted their colab notebook, so you can dig in if you want to see the details
"Recorded history" in the title refers to the period of history where the agency has been recording the numbers. It might not be the best phrasing, but it's not strictly untrue; the underwritten thesis (TB is on the rise) is still supported by the evidence.
"recorded history" sounds like it's how you divide pre-colonial Americas from modern (15th Century CE onward) Americas. For example, many weather features have been recorded in the Americas since 17th century CE. Does "recorded history" refer to only "[this particular metric's] recorded history"?
I agree that it's not the best term, but I don't think its so disqualifying that it makes the claim untrue: it's misleading at worst, and that imprecision only kinda interacts with the underlying claim.
I guess the better phrasing would be "Kansas tuberculosis outbreak is largest since (org) has been collecting data", which honestly doesn't change the implications for me.
The actual work underlying the essay - the one published in Cambridge Core - is pretty strong and has a lot of pretty compelling analysis. It's just long.
I disagree that it's strong, I got up to section five and it's written in the same "I am very smart and this is very important" style while saying very little of substance and bouncing around like a rabbit on speed.
The central thesis that demand creates supply is also just very obviously false, no one was searching for "100 identical twins fight for $250k" before Mr beast made that video. People watch Mr beast because they want 20 minutes of whimsical predictably mind numbing colorful emotional fast paced distraction and Mr beast fills that need perfectly with videos of all kinds. He transports viewers to a fantasy land far beyond their real world, where boredom doesn't exist and crazy things are possible. No one knew they wanted an iPod until Steve jobs showed it to them; they just wanted a better CD player. Same thing with YouTube; people just show up and click on something the algorithm puts in their way, there isn't a demand for anything really. No one knew they wanted a Mr beast. The consumer demand is some combination of distraction, entertainment, or education, it's not much deeper than that.
This seems an insufficient analysis. The meaning expressed by contemporary music, film media, or streaming television isn't very profound, but they at least still make a passing effort to "signify" something. The highest grossing movie of 2024 - Inside Out 2 - is not a deep text, but it does have a thesis.
The "Pixar apparatus" is definitely increasingly consumed by audience demand, but they're at a minimum in a transitional phase: something like Seeing Red would never get workshopped out of committees.
Youtube and other social media (emphasis on media) is ground zero for the decay of meaning into intensity; the ultimate incestuous product of auto-simulacra.
Your conclusion is the exact same one the above essay comes to in the next paragraphs. It concludes that the "alignment" between MrBeast and his work is a result of the larger thesis: creators are ultimately created by the audience conditions of the platform. Or, authenticity doesn't mean much when the root of the creator isn't a ground truth, but a synthesis of demand.
We do not come to the same conclusions. The last paragraph is this:
“Luckily, I’d say I’m a pretty predictable guy.” Luck has nothing to do with it, Mr. MrBeast. Your predictability is the result of years of an information diet consisting of audience feedback metrics. You are the proudest creation of the YouTube Apparatus."
Audience feedback metrics are only part of what a creator does. They are people with complex motivations and being sensitive to audience feedback metrics doesn't eliminate that. I could see saying the ecosystem being a synthesis of demand (I mean that's just trivially true, we don't need an essay on Mr Beast to say that) but from interviews Mr Beast is very much a product of the pressure of YouTube mixed with the very specific nature of who Jimmy is as a person (from interviews he seems like a pretty weird guy and his videos definitely reflect his particular quirks).
Edit: I'm trying to see how the author and I agree but unless the author is saying that people who succeed on a platform are the people that do things that are successful on the platform (i.e. align with audience metrics) then I don't think we agree. And I don't think that's all they are saying because that's just trivially true everywhere and it would make all the talk about philosophy and authenticity useless cruft. I think that Mr Beast is the sorta platonic ideal of a "content creator" driven by metrics and even he cannot escape his own Jimmy Donaldson-ness in his videos.
The way I read it, the author says not that the creator as persona is synthesized by metrics, but rather that Jimmy himself is subsumed by MrBeast. He is being authentic, but that's true only because the platform inculcated those desires over a years long "information diet".
It's a view I sympathize with, even though I'm reticent to apply it to specific cases like this. Rather, when we look at these systems, we should treat their demands as prescriptive, rather than descriptive. Spotify is an excellent example. Say the recommendation algorithm starts recommending a genre, but not for any aesthetic reason. Over time other musicians that internalize the aesthetics associated with that genre and will succeed because they also get picked up by the algorithm. These artists are succeeding by making algorithm bait, but they're also being authentic because the algorithm they're courting shaped their sensibilities as artists and as people.
The article isn't making a novel observation that people are shaped by the systems around them but applying that idea to the creator economy.
-16ff714.png){kind=link}