I previously submitted this story a few days ago[1], where it garnered some discussion and a bit over 100 upvotes, but never made the front-page due to some down-weights in HN's algorithm. I'm reposting this story at Daniel's suggestion. He agreed it deserved a second chance to break out.
--
I noticed that it's commonly accepted that Bloomberg's 2018 stories on the Supermicro hack were bogus, costing them a huge amount of reputational damage. However, Bloomberg stood by the stories. I'm very curious to see how this one will be received. I was never quite convinced by the naysayers or the denials by the government, Apple, or Amazon. I think it's quite likely that Bloomberg will be proven right in the end, this story having been accurate but suppressed for "secrecy" reasons all along.
Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. This time, it seems they have been able to do so.
Either way, I'm thrilled to see an outlet do original reporting and stand by their work in the face of universal condemnation. We are better off for having strong dissenting views informing us and expanding the narrative. I'm finding myself increasingly distrustful of the work of journalism outlets, but this (and those gone independent on Substack) gives me hope.
It's not just the lack of sources, it's the fact that Bloomberg made up complete nonsense to fill in the bits of the picture that they didn't have.
Off the top of my head from that time, I remember they asked someone about the plausibility of some part of the story about the component types that might have been altered, he pointed them at some random example from Digi-Key, and that example became their photoshoot prop.
All of their photos were provably fake and staged, and made no technical sense.
There was that whole sub-story about how supposedly Ethernet jacks having metal shields was one of the indicators of compromise (what? almost every piece of datacenter kit uses Ethernet jacks with metal shields).
The more you looked into the details, the less they added up.
The only thing we could confirm was that, at some point, Supermicro shipped compromised drivers. That part I can absolutely believe, and like the last bit of this story update claims, that may well have been an APT hack from a state-actor. That side of the story is entirely believable and self-consistent.
But the hardware modifications? Sorry, but the way that story has been reported is complete garbage. I'm not saying it isn't true, I'm saying Bloomberg provided no credible evidence that it is true, and plenty of evidence that they don't have the slightest clue what they're doing, and that their reporting is dishonest.
Originally I gave them the benefit of the doubt, but when they published the Ethernet jack story with less than 5 days to check anything from that guy, they lost all semblance of credibility.
The reason that the information about hardware was vauge is probably because it was (is) most likely classified.
IMHO Bloomberg is trying to break this story without giving malicious actors useful information.
As a proof of concept experiment, I hacked together an Mcu, a few caps, resistors, diodes, and a small power transistor to see what could be done on a "single component" attack. I was able to receive data and inject malicious messaging into the data from a serial eeprom using a "filter capacitor" made up of my device (only 2 wires, 1 to signal, 1 to ground)
Obviously, not having access to a fab or able to integrate microscale circuits, my device would have been trivially detected (about 1 cubic centimeter) but with access to a fab and a bespoke lab, I have no doubt it could be manufactured down to the size of a standard filter capacitor (~3mm3).
I do not know if this is what malicious actors would do or not, but if I can build a 2 wire device to compromise the data loaded from serial flash, then any state actor can. If you compromise this data on the control firmware for a PC, you own the board.
All of the claims that the hardware attack is fanciful are clearly naieve, based on my experiment.
It is worth mentioning that the attack that I experimentally replicated is the hard version. The easy one is to substitute a modified device for the serial eeprom itself... Also invisible, and orders of magnitude more potent. A compromised eeprom could include advanced features including large amounts of storage and built in rf communication capabilities.
The naysayers in this case saying that the attack would be technically infeasible or excessively difficult to implement undetected are simply wrong from a technical point of view.
Compromised devices could be mass manufactured for < 10usd each, and if you can get a reel of them into a manufacturers supply chain, you can permenantly compromise thousands of devices with a single attack. The required tech is within reach of any state actor with access to a fab.
>The reason that the information about hardware was vauge is probably because it was (is) most likely classified.
This is irelevant to the article though. The article was clearly written by someone who didn't even know what an Ethernet cable looks like and someone who provided fake photos for the article. It makes no difference if it could have happened to the clearly fakenews article. There's a discussion to be had about the implecications of something like this happening but baseing it on a fluff piece is not the way to get anything useful.
There are entire MCUs that fit into the volume of an Ethernet connector's shielding. So there would be no visual way to tell if one were present on a board, unless the shielding or connector design was different.
Industrial X-Ray capability is trivially accessible and is frequently used for relatively simple tasks like ensuring an assembly is put together properly or that the correct part was ordered etc.
If your premise is that the article might tip off attackers to the idea that they have been caught, these points are moot. Either they have been caught, or the article is actually bullshit.
But that's okay, I'm going to hold your hand for a bit. Pretend the article said "three hidden microphones were found to be added to this board." That tells the attacker which board is being tested, that they are testing for microphones, and how many of their microphones have been found.
If your job is to hide microphones every day, do you see how this information could help you moving forward? Do you see why this is more than a binary caught/not caught?
There was no reason not to answer straightforwardly at the beginning. Weird that you didn’t.
Yes, it’s possible there is more than one attack and they want to avoid revealing which ones have been detected.
This may be the class of information they are protecting.
If so, the journalism leaves us in the same position as the attackers - the number of attacks we know have been detected is in the range 0 - n.
I.e. it’s possible the article is complete bullshit. Another possibility is that the article is itself disinformation. If attackers can’t tell then neither can we.
>it’s possible the article is complete bullshit. Another possibility is that the article is itself disinformation.
Okay, that's fair.
Your question was about what kinds of information an attacker can use. I didn't think of your question in the context of the entire article being disinformation.
I need to think more about that.
>Yes, it’s possible there is more than one attack and they want to avoid revealing which ones have been detected.
>This may be the class of information they are protecting.
>There was no reason not to answer straightforwardly at the beginning. Weird that you didn’t.
I'm glad I didn't, and I'm glad you took the time to puzzle that through.
>Nice try
You want to give your attacker as little information as possible. Even information about the kinds of information you're restricting can be used against you.
> I'm glad you took the time to puzzle that through.
I didn’t puzzle anything through. I just kept asking until you responded straightforwardly.
I just wanted to know what classes of information people thought the journalists might be protecting.
>Nice try
> You want to give your attacker as little information as possible. Even information about the kinds of information you're restricting can be used against you.
At some level yes, but not at the level of this discussion. The ideas we’re talking about here are widely present even in pop-culture.
> I just wanted to know what classes of information people thought the journalists might be protecting.
I just made a joke. You didn't get it (?), so I encouraged you to think through some possible answers to your own question. If you can think of more than one class of information that you wouldn't want your attacker to know you have, then you can arrive at the punchline yourself.
It's not too late to try it, by the way. Thinking like the attacker is a good exercise. The attacker doesn't even have to be real.
>At some level yes, but not at the level of this discussion. The ideas we’re talking about here are widely present even in pop-culture.
I'm not sure what you mean by this, but operational security would still apply even when you have to make press releases. Companies juggle this all the time, e.g. sharing just enough about their proprietary technology to attract new employees/customers without giving everything away.
> I just made a joke. You didn't get it (?), so I encouraged you to think through some possible answers to your own question.
If you really were making a joke, you could have just said so when I ‘didn’t get it’. I’d probably have said something like ‘fair play’, or ‘nice’.
Because you didn’t we are left with other possibilities to consider:
1. You were just being condescending from the beginning, and are now trying to claim it was a joke to save face.
2. It was a bad joke, and you doubled down on condescension to save face.
The evidence supports either of these because ‘encouraging someone to think something through’ when you haven’t engaged in good faith conversation is an ignorant and condescending move.
Have you considered the possibility that I am asking what other people think because I want to know what other people think, and that ‘being encouraged to think it through’ will simply not answer that question?
There is nothing to suggest that you are an expert on this topic. Your own knowledge of it is limited despite having ‘thought it through’, proven by the fact that my response to your first real answer immediately showed you had missed something.
There is nothing wrong with making jokes, even bad ones, but following up with condescension makes it look like you weren’t actually joking.
We’ll never know now.
> I'm not sure what you mean by this, but operational security would still apply even when you have to make press releases.
Are you trying to create the impression that you are in personal possession of secrets about this attack?
>If you really were making a joke, you could have just said so when I ‘didn’t get it’. I’d probably have said something like ‘fair play’, or ‘nice’.
Lol ok
>Have you considered the possibility that I am asking what other people think because I want to know what other people think, and that ‘being encouraged to think it through’ will simply not answer that question?
Had you considered the possibility that you asked a low-effort question? "What kind of useful information [might malicious actors look for]?" is about as deep as "hey how long should my password be". You are on Hacker News. I honestly thought you were joking when you asked it.
Do you really just want to talk about how the article makes China look bad? That's legit, but not at all related to the question you asked.
>There is nothing to suggest that you are an expert on this topic. Your own knowledge of it is limited despite having ‘thought it through’, proven by the fact that my response to your first real answer immediately showed you had missed something.
Nobody said I'm an expert? You asked about very basic threat modeling and I tried to walk you through a very basic exercise.
The thing I missed (the article itself is disinformation) wouldn't actually change much about the threat model. The writers still have to decide what/how much information about their business practices to share, and the attackers still gather every bit of info they can. The attack in the article could be a lie. The article could still accidentally revealed something new (or confirm something old) to any potential attackers. If the article mentions a specific factory, for example, another data point an adversary can use to model their supply chain. If it mentions a name and that name leads to a LinkedIn profile, the attackers get a little bit of the company's org chart. Those little leaks add up and must be actively managed.
>There is nothing wrong with making jokes, even bad ones, but following up with condescension makes it look like you weren’t actually joking. We’ll never know now.
I invite you to puzzle that one through yourself :-)
>Are you trying to create the impression that you are in personal possession of secrets about this attack?
> Had you considered the possibility that you asked a low-effort question?
As I said - condescending - and now we can see that your answer was in bad faith. That’s not encouraged on hacker news.
> "What kind of useful information [might malicious actors look for]?" is about as deep as "hey how long should my password be". You are on Hacker News. I honestly thought you were joking when you asked it.
Obviously not. You can’t claim to know much about the subject.
> is about as deep as "hey how long should my password be"
You don’t actually know what range of answers a good faith commenter might have.
You are simply confirming that your response was in bad faith, as it appeared.
> Do you really just want to talk about how the article makes China look bad?
That's not how this works. Journalists frequently don't provide public proof, because that is frequently impossible. They convince themselves an article is true (often by using sources they cannot reveal) and then publish whether or not they are able to find proof that they can reveal.
Yeah, I'm confused by all the posts acting like the objections were just that Bloomberg's first story didn't provide any evidence. It was quite a bit worse than that; they included some very obviously faked evidence and a lot of utter nonsense. I wouldn't be surprised if it turned out to be "based on a true story" but even if every single one of their claims turns out to have been a distorted retelling of something real then I'll still stand by my opinion that it was terrible reporting.
Oftentimes, media will use imagery or illustration that are there for illustrative purposes. I don't think Bloomberg claimed to be showing actual photographic evidence. Not to mention, oftentimes the story and the presentation come from separate sources — the editors taking the written story and deciding the headline, and how to present and illustrate it.
That is a huge pet peeve of mine, using images for "illustrative purposes" does NOT mean "to illustrate, i.e. show in images, what we're talking about".
Instead, it means "to give you an image to rest your eyes on, so reading all of these annoying letters doesn't make them catch on fire" or something.
I agree, it annoys me too - especially in astronomy posts. ...but the point in this case is that it should not be used to discredit the Bloomberg article.
> Oftentimes, media will use imagery or illustration that are there for illustrative purposes
"Hitler was a really bad man. Here is a picture of Hackernews user `kenneth` for illustrative purposes."
Something like that happened in France a few years ago. A TV channel illustrated a news segment about cybercrime with the first B-roll of computers they could put their hands on. It turned out to be shots from a segment on a not-for-profit org teaching computer skills to the poor (including former inmates). The org was pissed, the TV channel just said it was "for illustrative purposes", as if that was justification enough.
I worked at a scientific institute. Every time we got some airtime, they asked the director to dress in a cleanroom suit. This man has never worked in a cleanroom, ever. But hey, it looks interesting for the layman. I've gotten extremely sceptical at whatever images the news is showing. They don't date nor give source for their material.
How is that different from any other article on any other topic? Everytime I read an article on a topic I am familiar with, it is full of approximations, misunderstanding of why things work in a particular way, exaggerations, etc. The standard you apply would remove any credential to every newspaper.
I don't think it's very common for newspapers to post comparable allegations of fraud or murder. Do you?
When reporting that someone has been arrested, indicted or is under investigation for such crimes newspapers aren't actually alleging that they did the thing. That seems quite different from what happened here.
>Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity.
It's not just that - it's that no other reputable outlet was able to confirm the story, and many tried. This just doesn't happen with controversial but true stories with many sources - some of those sources will be willing to talk to other outlets.
Bruce Schneier said last time that he didn’t believe it. This time he said he was wrong. When pressed by a skeptic in the comments, he said, “Independent from this article, I know that the DoD believes this threat is real and is giving away R&D funding for security solutions. So I have more than the Bloomberg reporting to go on.”
All you people are in denial. The first bloomberg story was credible but was protecting sources. They go further and get people on the record and double down....literally stretching their necks out, and you people keep propping up strawmen from the original article. Admit it. You were skeptical, determined that it couldn't be true, ran around shooting your mouth off about how it was fake news and now you can't help but continue smearing the egg all over your face.
As others have pointed out, the problem was not (just) with lack of sources, but with the fact that the few technical details they gave didn't make sense. As a result, the article read like it was written by someone who was completely ignorant and was just not credible on the face of it.
I think this twitter thread is a very reasonable analysis of this article: https://twitter.com/pwnallthethings/status/13602327536236298.... I think critically there are a lot of claims here, and none of them provide _concrete_ proof of any form. There's a lot of 'senior official said X' and X number of people in the know declined to comment, but these are just appeals to authority/credentials. People are flawed and second hand rumours of rumours can easily bend the truth. I don't think it's necessarily fair to put out a story claiming this is 'the truth' when alot of the sourcing is indirect and 'declined to comment'. It's just not there with the facts! True or not, we can't know, but this report need more facts.
> Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings.
Only if you believe that the warnings went out because of the "alarm over the devices' sophistication" and not "alarm over the idea this could happen" or even "alarm over a misidentification of a legitimate component and now we're too embarrassed to issue a retraction to those 10 CEOs"
The article certainly quotes a wealth of sources - but all of them seem to be vague, third-hand stuff.
An unnamed "adviser" to security firms that analyzed Supermicro equipment. An executive for some unnamed company, who received a briefing. A venture capitalist who received a briefing. A retired FBI agent who was told there was an "additional little component" by someone he can't name. Some former FBI officials that refuse to name supermicro, and say only that supply chain attacks are possible. Fifty interviews with officials, all of whom asked not to be named.
So apparently this information is so public that everyone and their dog is happy confirm it - yet at the same time, so classified that the victims and the 50 sources can't be named?
That's pretty vague compared reports of hardware implants like credit card skimmers [1] which put pictures of the hardware front and centre and make it clear that the author has personally seen it and knows (basically) how it works.
first, I don't understand how this quote squares with your overall description -- it's specific, first-hand, from a named source, who should (on its face) know what he is talking about:
> “This was espionage on the board itself,” said Mukul Kumar, who said he received one such warning during an unclassified briefing in 2015 when he was the chief security officer for Altera Corp., a chip designer in San Jose. “There was a chip on the board that was not supposed to be there that was calling home—not to Supermicro but to China.”
second, the article presents a plausible scenario for why it would be difficult to get additional details: the U.S. govt's strategy for dealing with this was to let it play out, so they could learn more about the nature of the threat. any public disclosures about this are at odds with the strategy of the U.S. government in combating it.
it seems like a very realistic scenario to me that some of the details of what happened became "lost in translation" - but that there is a real underlying truth. the first article wasn't convincing to me, but this one is very difficult to dismiss.
> first, I don't understand how this quote squares with your overall description
A person who "received a warning in an unclassified briefing" isn't first-hand. At best it's second-hand, if whoever analysed the hardware implant was going out giving briefings in person. More likely it's third-hand.
> it seems like a very realistic scenario to me that some of the details of what happened became "lost in translation"
If the claim was an evil driver update, or a backdoored BIOS, that would be completely believable.
Indeed, most of the details of this attack could just be a miscommunication about a BIOS backdoor - supply chain attack, malicious, code that shouldn't have been there, stored in an eeprom on the motherboard, undetectable by visual inspection.
But the much more astonishing claim of a malicious hardware implant between layers of a PCB? Something that surprising needs the testimony of an electronics expert, not a cop or a C-suite officer. Especially after the purported victims from the first article denied knowing anything about it.
"first hand" as in, 'the govt told me that china bugged supermicro motherboards,' as opposed to, 'I was told the govt briefed tech companies about how china bugged supermicro motherboards.' honestly, describing the testimony I quoted as "third hand", simply because the person doing the briefing for the govt was not the same person to have discovered the breach, is completely ridiculous.
do you agree the article establishes that the U.S. govt repeatedly warned private sector companies that supermicro servers had been bugged by china? maybe the us govt was wrong about that! could be! but it's still a huge story and much more than what was established by the first bloomberg article.
> do you agree the article establishes that the U.S. govt repeatedly warned private sector companies that supermicro servers had been bugged by china?
I agree the article establishes that unnamed sources in government are giving out briefings.
But I don't think that was ever in doubt.
Even when the first article was written, I could easily believe the journalist's claim they were told what they wrote by some sketchy anonymous government source. I just think they didn't demonstrate the truth of their source's claim.
There's always some anonymous "senior government official" ready to tell a journalist that Iraq has weapons of mass destruction (but they can't show the evidence as it's classified) or whatever other narrative they're pushing at the time. It's a journalist's responsibility to apply due skepticism.
for whatever reason, you are refusing to grapple with the fact that the second, just-released article, does not rely on the testimony of anonymous government sources, "sketchy" or otherwise, but on multiple, named, private sector sources. the people who are quoted have reputations that are important to them. they are claiming to have participated in concrete events. it's a very different category of testimony than "government sources say."
the difference is, the motivation for the two kinds of communication are completely different. anonymous govt sources talking to the media want the info to be public. govt officials telling a limited circle of people involved in core infrastructure generally do not want the info to be public. if you are trying to broadcast info to the public, your message is much more suspect, because the likelihood of an intent to deceive is higher. if you wanted the message to get out, there are far easier ways that telling a small circle of private sector people. generally govt isn't going around telling dams, nuclear power plants and drinking water plants about bogus chinese hacking threats on purpose. they can easily be wrong. it's a lot less likely they are lying about something like this.
This is `usually how national security problems are reported on, due to the classified nature of the work. You don't have to believe it, but it's naive to think that you're going to see actual chips. Currents and formers can talk about the what, but not the how. I mean even China knows the "what". Methods and means are a big deal, PRB kills entire chapters because of it.
Why do you completely cut tie with a big supplier over security concerns and at the same time don't disclose why? I'd say this is exceptional in the business of IT-security.
For a story claiming physical alteration of products, they should have had photographs of altered products. Otherwise it's just "anonymous loudmouths blow smoke".
Everything that would be done with the extra component could more easily be done with BIOS code, attracting no attention. So it doesn't pass the sniff test.
I don't think you deserved the downvotes as I felt somewhat the same way. Part of that comes from the reporting, which appears to have been done by someone who doesn't understand the things he's reporting on (or, at least, some of the people quoted are saying things that are nonsense that someone outside of our industry wouldn't necessary recognize)
i.e. "had a chip encrypted on the motherboard that would record all the data"
... one does not simply ... encrypt a chip ...
With these kinds of explanations or statements used as evidence, it's frustrating trying to figure out what "really happened". Personally, I don't doubt there's something to all of this. It's a "real threat" as has been reported; whether or not it's "theoretically real" or "actually used" isn't that relevant: if it's a "real threat" someone has figured out precisely how to do it or has seen it (or close-enough variations of the "it") being done that it's worth taking action.
My hunch is that it's a firmware-related hack; no hardware added to the boards. The article talks around BIOS and such, indicating that some of the code involved was inserted by employees directly connected to the company. It explains the lack of photographic evidence--what's the evidence[0]? And wouldn't the attacker prefer it that way? It also helps explain the "encrypted chip" -- perhaps code, encrypted on the chip (decrypted/executed at boot) was what was intended by that statement[1].
[0] Disassembled code on Github would have been nice, but the US government isn't likely to drop that.
[1] And maybe it was said in that bizarre manner intentionally, who knows?
>Everything that would be done with the extra component could more easily be done with BIOS code
This is tricky because on newer platforms the BIOS is signed and verified by the CPU. The signing keys are likely going to be located in the engineering offices (in the US) rather than just lying around in the factories, so getting them would require much more access than just bribing/threatening a few technicians at the factory.
> attracting no attention. So it doesn't pass the sniff test.
Compromised BIOS is a known threat, so I'd presume paranoid customers would verify/reflash the BIOS upon receiving them. This can be done without booting the motherboard at all (ie. attaching SPI clips to the flash chip and reading/writing directly). A tampered bios would get detected this way so it's not exactly "no attention".
> We are better off for having strong dissenting views informing us and expanding the narrative.
Strong dissenting views is a means to an end, not an end.
The truth is some point in the space of plausible ideas. If the region covered by existing ideas doesn't overlap the truth, then yes, you need to expand it. If it does, then looking elsewhere is guaranteed to bring you farther away from the truth.
If the story is false (or, more precisely, Bloomberg's reporters are talking to people who aren't technically qualified to analyze the subject and are misinterpreting it and Bloomberg is putting its own spin of misinterpretation on it), then Bloomberg's contribution to the discourse - to the pursuit of truth - is of no more value than the National Enquirer saying that Batboy hacked all our computers. Possibly of less value, in that people know not to take the Enquirer seriously.
The ability to have a wide range of views expressed is good for the pursuit of truth, yes. That doesn't mean that a dissenting view is of merit because it is dissenting. And in the case of this article, when it was based on a previous one that was attacked on the merits and not because of how dissenting it was, and seems to be more of the same, a commitment to the search for truth requires us to dismiss it.
This is interesting how and why reasonably credible stories instead of being investigated get ignored by mainstream media all the time, recent example being NY Post.
In this case Bloomberg comes back with the vengeance, but we need to see if it will picked up or will be buried again.
NYP is garbage, quite famous for splashing completely untrue stories on the front (or top article on WWW) and then retracting it a few days later in the most remote areas of their media.
> Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity.
Wait is this all of a sudden a prerequisite for something being believable?!?!?! What the hell did I live through since 2016?
As a bicoastal American, there is a strong East-West schism in that. West Coast techies, largely disconnected from D.C., being (rightfully albeit overconfidently) sceptical. East Coasters being (almost always overconfidently) confident in their friend at such and such (almost always unrelated) agency vouching for the story.
Well actually kenneth is behind everything and we can prove is because we have the sources to back it up.
No of course we can't give out the sources, our journalistic integrity requires us to protect our sources. But we fact checked everything and it's 100% true.
Its weird to see a follow-up to this that doesn't even begin to acknowledge the broad skepticism of claims except for saying that the previous push-back from the alleged targets was a drop in the bucket.
I don't doubt there are highly-targeted supply chain attacks that happen at the whims of state level actors but the specifics presented by this and its previous reporting still are difficult at best to believe with the reporting as it is presented.
It's weird to see how the skepticism is so one-sided on this issue. Any time I've commented that I believe the hack is probably largely true even if the story details aren't 100% accurate I just get downvoted to oblivion and/or called names. My kind of skepticism is open to all possibilities.
> 3), there are probably other stories that happened and didn't get reported in media.
Agreed 100%, that's why it feels so weird that conversation around this story always seems to drift toward reasons it must not have happened instead of looking for the islands of truth in a story that's probably way way way worse than any of us realize.
Because looking for 'islands of truth' is basically the behaviour of conspiracy theorists: wade through mountains of evidence, ignoring everything to the contrary to find the few scraps which support their theory. You can't just ignore the huge numbers of things in this story which straight up don't add up. It would be great to have more credible examples of such compromises, but this ain't it.
You don't know how Bloomberg obtained their evidence, and unlike conspiracy theorists who lose nothing to spread their misinformation, Bloomberg stakes their reputation, and business, on the credibility of their stories. Incorrect details is one thing. Falsified story in such sensitive matter is another thing. Especially now that they've doubled down on it.
> You can't just ignore the huge numbers of things in this story which straight up don't add up
The rebukes can boil down to: a) companies involved denied it; b) nobody else confirmed it; c) the picture in the story didn't seem to be legit.
Perhaps I miss other rebukes, as I didn't follow it very closely, but none of the above is convincing enough to dismiss the original story.
The strongest hypothetical case for this story is that some NSA/CIA people told them on deep background that it's definitely happening, Bloomberg went out to try for parallel construction that proves it, and epically owned themselves with poor execution.
Falling back on "you can't prove it isn't happening" is a really weak defense. If they have stronger evidence, they should either present it or stop talking about it.
The whole thing is strange. Despite what various tech commentators may think, Bloomberg is a legit media organization which doesn't make major accusations casually. Doesn't mean they don't screw up from time to time. But it's at least "interesting" that they not only didn't back down but they doubled down on this story. This story would not have run if editors weren't convinced it was substantially true for whatever reason.
More reputable organizations than Bloomberg made bigger claims that turned out to be false. Remember WMDs and the Nasrallah testimony?
What is asserted without evidence can be dismissed without evidence. There is no evidence here. And it's not even a case of not being able to produce it, you could literally just get your hands on a compromised board and x-ray it. People do that all the time.
In trying to make rational decisions, it is foolish to pretend that one's own decisions (past, present, or future) are always (or even mostly) rational, let alone other people's.
Rather, most purportedly rational or logical decisions are instead rationalized ones, and it is rational to acknowledge this.
I'm not suggesting throwing the baby out with the bathwater, just awareness that as decision makers our ability to apply logic is limited (in economic terms 'bounded').
In fact, perhaps our limited ability to reason is best imagined as a baby: full of endless potential, yet weak and fragile, easily perturbed, and requiring near constant care, attention, even vigilance.
If you’re running sensitive servers, you might put the burden of proof on the vendor to convince you that the equipment they’re selling you isn’t backdoored. (You should also do your own checks)
We're talking about data centre servers here, that's what the Bloomberg story is about. For a backdoor like this to work it has to be able to phone home, and that means getting transmissions out of the DC network. This would require knowledge about the target network infrastructure (which isn't impossible if there are specific targets) but there's no way to do that without the transmissions being visible. I think a hack like this is much more likely to be viable against consumer and mobile devices.
Given that this is all a genuine risk, the Bloomberg story is particularly harmful. It creates the real risk that reports of genuine hacks or vulnerabilities will be sidelined due to reporting on issues like this being tarnished by association. Crying wolf is not good preparation for dealing with real wolves.
For a backdoor like this to work it has to be able to phone home, and that means getting transmissions out of the DC network. This would require knowledge about the target network infrastructure
I don't think knowledge about the target network is required, just the ability to detect what default route is used by the OS. It also wouldn't need to "phone home", just phone into a controlled network. It wouldn't be a complete fantasy to assume that some software telemetry/update endpoints are compromised.
And steganography using TCP sequence numbers is a real possibility, so once a link to a friendly endpoint is established, the backdoor wouldn't even need to generate its own packets.
China has a motive and they have the means. That much is undeniably true. What I expect to see is actual proof and not a case built on hearsay. My brand of skepticism requires it.
Before Snowden revealing NSA's shadowy campaigns, any claims of such effort was met with strong skepticism, just like yours. Which may have been justified at the time. But that was eight years ago. This is 2021. You are absolutely right to raise your skepticism, but dismissing it outright because proofs haven't been presented is naive.
Many many classified information didn't see the light decades after they happened. I don't see why this isn't such a case.
If Bloomberg had written about "this is happening according to sources" people would not be reacting as they do to an "this happened, look at these pictures that are obviously fake" article.
This is nothing like a pre-Snowden talk about NSA where someone theorised what was happening. This would be like if Snowden was quoted anonymously in an article and proof that was clearly fake was shown as actual leaks. Sure hardware modifications like this is happening (we know NSA does so) but this particular article (well, two now) is pure fluff and FUD pretending to have real proof. There's an interesting discussion to be had about this topic but those articles would make anyone a laughingstock if used as facts in the discussion.
> This is nothing like a pre-Snowden talk about NSA where someone theorised what was happening.
There was something visceral about seeing a picture[1] of the NSA giving an intercepted Cisco router a hardware "upgrade" (beacon) in the Snowden files. Bloomberg tried to something similar for their story, but with fake pictures - that was a terrible idea.
And after claims of Iraqi involvement in 9/11 and WMDs that never materialized, I'm pretty skeptical about unproven, geopolitically convenient news reporting.
The thing is that NSA was engaging in a shadowy campaign. It wasn't possible to conclusively prove or disprove it. This story isn't. It's making factual claims, that there are physical backdoors in hardware that is publicly available. From then on its possible to settle the matter by buying such a motherboard, and using this to produce hard evidence of the backdoor. It wasn't possible to go to the nearest NSA field office to get yourself a copy of XKeyScore.
I was of the same mind ant the beginning. Then time passed and Bloomberg failed to provide any evidence. And independent security researches expressed their doubts, too. Of course Apple and AWS issued strong statements without any room for discussion. So at this point, if Bloomberg really had anything to back up their claims, they should have told us so, but they didn't. So I'm more of the opinion their story is inaccurate but they just don't want to admit it.
The consultant straight up admitted that he presented his hacking scenario as merely a hypothetical example. This all got embroidered by an unethical journalist looking to make a splash by "moving markets".
I do, and based on that it feels perfectly reasonable to believe that every other superpower probably has an equivalent. It also feels perfectly reasonable that server BMCs would be a common target for all such units considering the ludicrous access they provide, e.g. remote BIOS flashing, continuous screen recording, etc.
That's a great reason for believing some BMCs have been compromised, but it's not a good reason for believing these BMCs specifically have been compromised (and in this way).
"No evidence" is maybe a little harsh? They have government sources testifying, they have people working for semiconductor manufacturers testifying on getting briefings from FBI. They have court documents on similar incidents with Lenovo? I get that this story very much is about who you chose to believe, but to say "no evidence" makes me think you didn't bother to read the article.
When it comes to estimating unverifiable classified truths, your personal estimates are a exercise in balancing probabilities.
So based on motive, means, and some circumstantial evidence, we can estimate that this is likely happening. Enough truth, at least, to guide purchasing choices away from SuperMicro.
These low value articles about such-and-such security vulnerability that provide zero technical information allowing me to identify signs of impact or remediation steps really piss me off.
Do enough people have their heads firmly planted in the sand that they believe there is no state sponsored bumps in the night? That is the only reasonable defense for such low value publication.
That’s the kind of faux-skepticism that leads to Qanon, pizzagate, and broad xenophobia. “Oh China must be doing this stuff, I mean we can’t produce a shred of evidence but foreigners == bad so that’s proof enough for me”
I'm sad this story focused so much negative attention on Supermicro, not only because I'm a big fan of their products but because it will surely stop other companies from coming forward or cooperating with investigations for similar attacks now and in the future.
Well, the push back was a huge part of the skepticism. It just wasn’t plausible that tech companies would lie so strongly and in such detail to protect a government investigation, especially not Apple after their famous controversies with the FBI. The follow up doesn’t claim to confirm that part of the original, and they did find tech industry people willing to go on the record about this.
Speaking for myself, I’m still not gonna just take three letter agencies at their word, but the source article shifted me from “probably false” to “probably true”.
Supply-chain vulnerability is no different than a software vuln, if it looks exploitable it should be patched regardless of if Bloomberg can show you a photo of an attack in the wild.
There are a lot of these soft underbellies around, like the fact your browser trusts CAs from potential adversaries, people walking around with phones whose firmware is entirely controlled by potential adversaries. It's painful to think about what would happen should a war ever break out.
They mentioned the denials, and then in the next paragraph went on to say it was bigger than they'd previously reported.
They didn't at any point address the broad degree of skepticism that exists among security researchers. This is a story which was met with a pretty wide swath of knowledgeable and credentialed people (much like the primarily alleged sources in the article) who pushed back on it, let alone the alleged targets.
In fact, one of the few named sources from the original article (Joe Fitzpatrick) made statements after the publication of the article against his contributions to the piece after its publication.
Again, I don't doubt for a second that these types of attacks happen/exist but I have a hard time accepting the `facts` as they are presented in this case and I think that is how a lot of people feel.
>They shared one other trait; U.S. spymasters discovered the manipulations but kept them largely secret as they tried to counter each one and learn more about China’s capabilities.
Putting aside the concerns with the believability of this particular Bloomberg article... how many legitimate security holes are ignored in software purely on the basis of "the NSA figures it's NOBUS (nobody knows but us) and it'd be good to use it on China?" I'm personally convinced that the spymasters are more worried about keeping a good stock of security holes in reserve instead of actually securing infrastructure.
Espionage and counter-espionage are two concerns that need to be balanced against each other. They’re both handled by the same organisations (sort of), so it’s reasonable to ask where those organisations are setting the balance
Can't see what they'd ever gain from going public.
Counter-intelligence is their job and I'd be highly surprised if they'd alert their opposition simply for some free PR. Far better to try work out what has been compromised and try feed misinformation into that data exfil for as long as possible. That's what we pay them to do, not gallivant around in the media tooting their own horn.
> Can't see what they'd ever gain from going public
0. It went public with the initial Bloomberg article, so it was lost as a misinformation channel years ago.
1. To get the security problem fixed, thus protecting US interests
2. To support US foreign policy goals like "Europe should reject Huawei 5G equipment" with concrete evidence
3. To demonstrate that, if a US company thinks they've discovered a different supply chain attack attempt, that they're interested, competent to help, and able to speak to you.
4. To show potential hires that that they're protecting the country from real threats, and that there are opportunities to do work that's neither illegal nor unethical
5. To demonstrate to the taxpayer that they're doing their job
Intelligence operations like the NSA, CIA, etc operate under a dual mandate: 1) gather information on foreign governments and 2) keep domestic information secure. However, with everyone using the same software, both mandates are in conflict. Any attempt to secure domestic information by fixing security holes also diminishes information-gathering capability elsewhere.
My personal opinion is that spymasters have decided as a class to leave things open in the name of keeping a job. Better to let China read your secrets than to lose access to China's.
Worth reviewing the B-Sides PDX 2018 panel on the Supermicro "hack". The reporter was mirroring and testing for plausibility of comments from two or more security researchers attacks on each other. The reporter would take the attack that Joe came up with and ask another security researcher does this seem like a real threat. Then take the attack that the other researcher came up with and test Joe. https://www.youtube.com/watch?v=8omlNyUVllY
Reporters do and should use this technique to understand what their article is about and verify a base understanding. They should be careful when it becomes an awkward game of telephone. [1]
As Joe Fitz and Joe Grand say in the panel it is a theoretical attack. There are plenty of ways to do an attack with similar effectiveness for cheaper and with more precision. See NSA's ANT catalog where individual boards and devices were created to target specific individuals or organizations.
A follow up devoid of any concrete evidence to a flaming article devoid of any concrete evidence. Why are they even doubling down on this?
If this is as wide-spread as they claim, I'm sure someone donated a "bugged" board for them to have prodded by experts, right?
If the assertion is that the customers targeted are a number of very high-profile companies and the US gov't, how often does either group refresh and then eventually sell off or recycle their surplus hardware publicly?
I would expect, if this is true, a lot of the affected hardware to still be running for many years to come, and the majority of the private hardware to be disposed of in a recycling pile rather than showing up on the secondhand market.
The government, as far as I know, does resell their used hardware sometimes in large lots, so perhaps some lucky customer cheaping out on buying hardware will find some exciting packets in their future...
No. As with the previous article I am yet to see a single shred of hard evidence. I would take this series with a healthy dose of skepticism.
Speculation: I read last time that Bloomberg rewards the authors of stories that cause stock swings and the last time this came up there was a sizeable swing which had almost returned to normal when this story was released.
FWIW, of the claims made in the article, there are two relatively independently true facts one can find.
One is the cited testimony of someone regarding a (large?) batch of Lenovo laptops found to be phoning home in secret, from public court testimony.
The other is the incident with Intel reporting that a modified update package was downloaded from Supermicro's update site and infected a system, causing it to phone home, and both Intel and Supermicro admitted (assuming you believe the journalists aren't making things up out of whole cloth) that something happened resulting in Intel contacting Supermicro to report the incident.
> batch of Lenovo laptops found to be phoning home in secret
Every software or hardware phone their place of origin naturally. Only authoritarian countries like China would require vendors to host services inside sovereign territory.
> As with the previous article I am yet to see a single shred of hard evidence. I would take this series with a healthy dose of skepticism.
"As long as knowledge has not been attained, the skeptics aim not to affirm anything."
I would ask HN readers if they believe the Snowden leaks about NSA's TAO unit? If so, why so much doubt that other superpowers practice similar operations?
People seem to be reading way too far into looking for a "custom chip" when the actual hack is probably just backdoor-implanted BMC firmware on a generic SPI flash chip.
They do explicitly distinguish between compromised firmware and distinct hardware implants in these systems, in this article, and claim examples of both.
//They showed an actual chip last time[1] but it seemed to me that it couldn't possibly house the logic necessary for data exfiltration. It looked like some sort of voltage conditioner. I am not an electrical engineer though.
> Can anyone show us any code? Or an actual chip? Cool infographics don't cut it for me after the first time around with this.
Amazing how the level of evidence required for this is completely different and significantly higher from almost everything else that dominated the news since 2016.
The first three aren't believed. They were proven. The Intel ME (and AMD PSP) are literally physically indistinguishable from a backdoor. NSA spying was proven by Snowden et al. Marketers fingerprinting us to the level that personal targeting is possible was proven.
Is it possible that China is bugging motherboards? Yes. But if they are, it should not be difficult at all to prove. Just buy some motherboards and analyze the chips to find where the backdoor is.
The fact that in over two years that still hasn't been done is just very suspicious. If someone is making a large claim but not providing information that can be found using public, verifiable sources, then it's very reasonable to doubt it.
Evidence isn't based purely on what seems right. If hard evidence can be provided but isn't, why should I believe you?
> Just buy some motherboards and analyze the chips to find where the backdoor is
If you were bugging motherboards, you would surely anticipate this. Some possible mitigations include only bugging a subset of the motherboards, and bugging them in ways that make them physically indistinguishable from a normal unit. Modified silicon or firmware would be very hard to detect.
> The fact that in over two years that still hasn't been done is just very suspicious. If someone is making a large claim but not providing information that can be found using public, verifiable sources, then it's very reasonable to doubt it.
If the bugged hardware is rare (it would be if the attackers were smart), then the public would not have access to it and the government agencies looking into this may still well be investigating this or keeping it quiet for whatever reason.
I'm not convinced either way on this situation, but it's far from impossible that the allegations are true and it is seems worth discussing.
Of course. But it's been two years now. There are many civilians that would have such bugged motherboards, and they've had two years now to find out where the flaw is. Including civilian organizations that Bloomberg explicitly said were targeted.
That said, about this : Modified silicon or firmware would be very hard to detect.
This is not compatible with the mechanism Bloomberg is proposing. They are suggesting that some silicon in the motherboard is intercepting memory in real-time, processing it, and modifying it on-the-fly to make the CPU do what is wanted.
That is not something that can be done by merely modifying existing silicon. RAM is directly, physically connected to the CPU. You would need to add an extremely high performance chip that shouldn't be there between the CPU and the RAM, and there is simply no way to hide that.
Firmware attacks are possible, yes, but you can readout the content of firmware chips too.
It's not a mechanism of attack that could be done invisibly. At the easiest, you could detect it by a timing attack, at the worst you can just x-ray the board. But it cannot be invisible.
>That is not something that can be done by merely modifying existing silicon. RAM is directly, physically connected to the CPU. You would need to add an extremely high performance chip that shouldn't be there between the CPU and the RAM, and there is simply no way to hide that.
Why? Look at the die shot[1] of a zen 2 IO die (manufactured using 14 nm process), and see how small the DDR4 PHY blocks are in comparison to the rest of the chip. Then consider the whole picture only covers 9.3mm x 13.2mm. If you only cared about manipulating a few bits of the DDR4 data bus, I don't see why the package has to so large that you can't hide it. You could also go for the opposite approach (ie. rather than decoding the signal and injecting the correct signal, you introduce a glitch at the analog level, similar to how rowhammer works).
Look at a modern motherboard. See how thick the RAM bus is? On my X370 motherboard it's over 10 cm thick and goes down to around 3 cm thick as it goes through vias.
For your chip to interface, you'd have to reroute all those traces to it (because RAM is random access, you wouldn't have the context necessary to know when to inject or modify if you weren't accessing most of the traces).
On an X-ray of the motherboard, you'd see all those traces converge to a point, then diverge back onto the socket. On a normal motherboard x-ray[1], you see the traces make a trapezoidal pattern.
Such tampering would be immediately visible to a motherboard x-ray. And by the way, they would likely be detectable by software too, even just the delay introduced by the length of the traces is detectable.
In addition, you'd actually need to have the die also process the data from memory, and it would need to be quite a bit faster than the relatively slow Ryzen memory subsystem.
I always thought an interesting version of this might be to have a marginal clock circuit that would fail randomly about 95% of the time if manufactured to spec but the 5% (because of design variation) that worked could send out backdoor information. Prime candidates would anything that's in the network pathway. I'm sure anyone doing it would be more sophisticated than me though :)
Probably should add in Russia. They are a far more sophisticated threat than China is (at this point in time) when it comes to network infiltration and security breaches.
I don't think anyone would be surprised if china was bugging motherboards - probably the extensiveness and efficacy would be the big shocks.
I'd say Russia is bolder and will take more risks than China, but China - technologically - is quite likely much further ahead than Russia. The resources ($$$ and PhD) at China's disposal are magnitudes larger than Russia - and perhaps greater than the US (in effective spent<->result).
> “In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm.
is that a "bland quote"? do you think that meeting did not happen? do you think the fbi was wrong?
No, I would describe this quote as a giving the story the illusion of truth. Adding a 3 letter agency does nothing but add sizzle and sensation instead of substance and truth. I have no way of knowing if that meeting happened. I do not trust the FBI or any other 3 letter agency to tell the truth at all times.
Haha, this is what I'd say if I wanted to ascribe more importance to myself than is worthy.
Two security companies I advise = companies I have a brief relationship with, as a consultant
were briefed by = were questioned as a matter of routine
the FBI's counterintelligence division = some FBI agent's 400th cold lead on this
investigating the discovery of = looking to see if some loony internal theory is real
Of course you read it and conclude "Oh, the FBI thinks this is credible enough that they told security companies it's happening and they should watch out for it" but that's because that's intentional.
> Today’s follow-up from Bloomberg offers no evidence either.
The follow-up article actually has quite a bit of evidence.
e.g.
> “In early 2018, two security companies that I advise were briefed by the FBI’s counterintelligence division investigating this discovery of added malicious chips on Supermicro’s motherboards,” said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm.
maybe there are reasons not to believe the testimony of Janke. but saying the article "offers no evidence" is just bunk.
The word of someone who's not an expert in the subject, who wasn't in the meetings and only heard about them via an unspecified number of intermediates, and doesn't even work there but "advises" the companies? Forgive me, but for the only sourcing of their main claim, that's a bit too weak to clear my standard of evidence.
it's not the "only" sourcing of the main claim, merely an example. (did you read the article?)
> Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings. While most executives asked not to be named to discuss sensitive cybersecurity matters, some agreed to go on the record.
> “This was espionage on the board itself,” said Mukul Kumar, who said he received one such warning during an unclassified briefing in 2015 when he was the chief security officer for Altera Corp., a chip designer in San Jose. “There was a chip on the board that was not supposed to be there that was calling home—not to Supermicro but to China.”
> As military experts investigated the Pentagon breach, they determined that the malicious instructions guiding the Pentagon’s servers were hidden in the machines’ basic input-output system, or BIOS, part of any computer that tells it what to do at startup.
Not really, neither this nor the original article claimed it was done on a mass sale (ie. every machine coming off the line). It was likely only done to juicy targets only.
So, what, some time after the line, after a sale was made, during fulfillment, a Chinese super-spy managed to open up specific server that the juicy target was getting and solder some special chip onto the board somehow in a way that worked instead of bricking the board?
I mean, honestly, hats off if that's the case. That would be pretty cool. But you need actual evidence for such a tale.
>So, what, some time after the line, after a sale was made, during fulfillment, a Chinese super-spy managed to open up specific server that the juicy target was getting and solder some special chip onto the board somehow in a way that worked instead of bricking the board?
I'm not sure why you're implying that you need a super-spy to pull this off. If this was done at the factory level they likely had access to the same manufacturing equipment used to make the motherboards, so there's no need to manually solder anything. The implant was alleged to be a surface-mount component so it could be as simple as reprogramming the pick-and-place machine or swaping out the reels. Given that this is china enlisting a couple of technicians to your cause wouldn't be too hard[1]. From there once they figure out a particular order is going to a juicy target they can ship the bugged boards in place of the untampered boards.
> But you need actual evidence for such a tale.
Agreed. My main takeway from the article is that this hack could happen, not necessarily that it has happened.
[1] it's not unlike the concerns that there are NSA backdoors in intel cpus (eg. AMT/vpro, or RDRAND), or windows (NSAKEY).
> I'm not sure why you're implying that you need a super-spy to pull this off.
The factory doesn't have assembly lines marked like 'for apple', 'for the nsa'. It's just an assembly line making a bunch of identical boards. You'd have to identify a specific board way after it's left the assembly line, and probably after being integrated into a chassis to put your spy chip on it if it's targeted at a specific customer.
The way we in the West do it is intercept the shipment en route and make modifications in a special facility. TAO have significant hardware capabilities and do this stuff routinely.
So what you are saying is that they (China?) can and do intercept shipments from Supermicro inside the US? Or how do you get the chip onto the correct piece of hardware when you don't know where each motherboard ends up when it leaves the factory?
If the warehouses and manufacturing lines are all in China (as is the case with just in time manufacturing) I don't see why this is difficult to pull off.
I'd expect shipments across the Pacific to be aggregated into the smallest possible number of containers and only subdivided into batches for individual customers at the last possible moment, in order to minimize both transportation cost and the risk that any single customer loses their entire order when a container falls off the ship.
Is that not how it works? If so, I'd like to know why.
That'd depend on how the shipping is done. What you described would apply for something like retail, where the factory only deals/sells to a distributor (stateside), which then sells/ship them to retailers. Under that setup it would be hard to accurately predict where a particular motherboard would end up at the factory. However, I suspect for large orders, they won't bother with that and would send an entire shipping container or pallet to the customer straight from the factory.
Surely a "juicy target" then would have sufficient resources to confirm or deny this? Last time this came out though, both Apple and Amazon -- who use SuperMicro servers -- explicitly denied that the servers were compromised.
The admitted it when they discovered the NSA was tapping their fiber links. They not only admitted it, they were furious, and encrypted all their internal traffic.
What's more, they didn't just keep quiet, they made very strong public denials, the kind that would result in SEC and shareholder lawsuits if they were proven to be false.
From the outside Apple (the company) looks to be a lot more subservient when it comes to its relations with China compared with its relations with the US government. Parts of the second one can be bought/lobbied almost all the way to the top, China is a little bit more challenging for a Western company, there's Xi and a handful of his underlings who can decide your future as a company on the mainland.
China reacts differently than the US. If you publicly bad mouth the Chinese government there are consequences.
Contrast this to a public showing of anger against the NSA which is beneficial to the company and to the NSA. The NSA wants users to feel secure and not take other measures.
Shareholders can sue for anything (everything is securities fraud), the act of exposing client details to the NSA without telling shareholders is no less problematic than continuing to lie to shareholders about dealings with the NSA.
I would go through zero-day bug instead of this. If one can build a single chip which read sensitive data from hard disk, steal private key from main memory, and send data via the ethernet port, that chip must have a lot of pins connecting to pcie bus, qpi etc., and super complex. This super spy chip must use the latest technology to build such as TSMC which is not possible in mainland China.
>If one can build a single chip which read sensitive data from hard disk, steal private key from main memory, and send data via the ethernet port, that chip must have a lot of pins connecting to pcie bus, qpi etc., and super complex.
Nope, if you read the original article the chip is much simpler than what you described.
>the primary role of implants such as these is to open doors that other attackers can go through. [...] In simplified terms, the implants on Supermicro hardware manipulated the core operating instructions that tell the server what to do as data move across a motherboard, two people familiar with the chips’ operation say. This happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects.
These buses are operating at gigahertz speeds with nanosecond timings. You would need cutting edge chips to do that. The note is correct. They could of course be manufactured, but it would very hard and very noticeable.
It's so simple: You need to be credible to make valid claims. If you get caught for faking photo evidence on a story accusing China bugging the motherboard, your story is done. And even if they did, they are getting away with it because you lost credibility in the first place.
> Two people with direct knowledge said the manipulation combined two pieces of code: The first was embedded in instructions that manage the order of the startup and can’t be easily erased or updated. That code fetched additional instructions that were tucked into the BIOS chip’s unused memory, where they were unlikely to be found even by security-conscious customers. When the server was turned on, the implant would load into the machine’s main memory, where it kept sending out data periodically.
Asking the obvious...
If this description is accurate, would the second piece of code probably be pretty easy to identify, as a suspicious extraneous block of bits, by a curious ordinary techie who rigged up a RasPi to read the Supermicro's BIOS flash? (Using a tutorial intended for Coreboot installation, for example.)
And does anyone know whether the first piece of code would be in that same flash (perhaps harder to find, as part of a blob of BIOS/init code), or stored in the BMC package, or somewhere else?
If it turned out that (alleged) backdoored units were widespread, or some rare (alleged) backdoored units possibly were mixed into the supply of cheap used Supermicro servers on eBay, or among units taken home when retired from IT people's workplaces... then might some random techie fiddling around at home find a backdoor?
And if someone did find a backdoor that way, would their first call be to their national authorities (to alert, and ask whether it would be bad if they talked about it), or to a journalist, or to post on social media, or to write a haxor paper/post?
I'm not going to look myself; just wondering whether this story might suddenly get more credible and problematic that way.
The Occam's Razor for this is that you have to work harder to disbelieve it than to believe it.
You actually have to actively avoid knowing or recognizing things, and have to actively construct rationalizations to explain not taking things seriously.
Additionally, completely seperately and standing on itcs own: There is no benefit to you or me to disregard it rather than regard it.
That is a whole sufficient argument all by itself.
Additionally, completely seperately and standing on it's own: There is no benefit to Bloomberg to fabricate a story like this that they know can't at least eventually be backed up, especially not twice. It has only cost them so far. Did they not know it would cost them?
That's a whole sufficient argument all by itself.
What is the scam if we are tricked into being careful about hardware supply chains really (1)? China and a dozen others are NOT trying to hack in any way they possibly can? It's NOT perfectly possible to do everything claimed?
(1) We're being manipulated! In what direction and for what purpose and how is it against our interests?
> There is no benefit to Bloomberg to fabricate a story like this that they know can't at least eventually be backed up, especially not twice.
There is a real benefit for the author of the story - he gets paid. There is a small benefit for Bloomberg - they get clicks. What I assume is happening here is a mix of incompetence and confidence that they can just get away with this even if the story is false.
If they were just saying that there were BIOS alterations, code injection, something soft, I have no trouble accepting that as a credible threat.
It's feasible, and probably quite hard to detect.
But they made fairly specific claims that named companies had discovered extra/altered chips on their boards. So they are saying someone has the mobos, Supermicro has the BOM.
Is this part on the BOM?
Where did we buy this part from?
This is a resistor package, why is there an IC inside?
Altered firmware/software would be an admitted beast to wrestle.
But the easiest work of all these should be to find the funky part in boards.
It must be true because Bloomberg wouldn't lie twice?
You recall the "Bloomberg for President" campaign? And their multiple incidents of "vandalism" across the country on the same night? Which was all oddly respectful of property...
my question is: who benefits? Follow the money. About if this hack is true or not: It should have been simple to prove true.
Just show a single compromised article with an extra chip, at least one is bound to show up on ebay? Can three letter agencies round up all compromised hardware in all of America in secrecy?? To this day none has shown up, it's all theoretically true. A compromised sample is worth at least 1M vues on youtube, it's worth real money, yet it remains elusive.
A compromised BIOS update is more likely, a BMC IMPI infected by a trojan thing; the new piece is hinting to that in the FUD. Then again, that should be possible to find in a compromised board on ebay (prolly worth less yt vues since nothing physical to look at), and therefore prove this article is not a hit piece for someone with a short position.
> The machines turned out to be loaded with unauthorized instructions directing each one to secretly copy data about itself and its network and send that information to China, according to six former senior officials who described a confidential probe of the incident. The Pentagon found the implant in thousands of servers, one official said; another described it as “ubiquitous.”
assuming this is true, there is a universe of "thousands" of supermicro servers purchased by the pentagon that were targeted.
my expectation is that most supermicro servers would not be targeted, just those sold to certain buyers. does the pentagon sell used supermicro servers on ebay? is it easy to obtain a used supermicro server from the pentagon? (I don't know the answer to those - genuine questions).
even if you had one in your possession, it wouldn't be easy to find the exploit, which was (again, assuming it exists) installed by nation state with the intention of concealing it from another nation state (the world's most powerful). for example, it might only turn on under certain conditions. I wouldn't know where to start.
I'm not saying it's impossible, but I am saying it seems much more daunting then you make it out to be ("should have been simple to prove").
The article said something about messing with traces on the circuit board to hide a component.
I think it would be far more likely to start with a well known component like a network or bus drive and produce a modified chip with identical packaging and markings. Only one person in the board vendor's supply chain needs to swap spools of tampered chips into the manufacturing stream.
It could sit dormant in most situations unless it saw, say, Pentagon LAN traffic. This means the EBay case is covered; the machine would be normal for everyone else, including the board vendor's QA.
You'd have to simulate the target's traffic to see the board doing something wrong. Or decap the chip and read it out.
Reading these articles and the history of tradecraft has pushed me in Bloomberg's corner. Even without access to any privileged information, I am highly certain that history will come down on Bloomberg's side. It is simply more likely - from first principles reasoning alone - that Bloomberg's story is true than it is false.
A famous example is Operation GUNMAN. At the height of the Cold War, the KGB bugged typewriters used by the US Consulate in Leningrad. The breach was so significant that it took some time for them to figure out what had happened, but eventually the NSA got involved (as did the NYT). They of course denied that anything like this ever happened (until a few decades had passed), and they tried to reverse it/control what the soviets saw etc.
> “He wanted the Soviets to hear 95% of what he had to say--when he briefed a congressman, for example,” the diplomat said. “This was one way he had of getting his ideas across to the Soviets. For the other 5%, you had the secure rooms.”
> The wife of a former security officer at the American Embassy, for instance, recalls complaining long and distinctly to her kitchen walls several years ago about the disappearance of her favorite butcher knife, seemingly purloined by the Soviet maid. She came home from work a few days later to find a lump under the living room rug. It was the knife.
It is difficult to find citations for this, but the NSA knew about the typewriters since at least 1982. However, the State Department "refused comment" on the stories published in '87, and the NSA (which wasn't well known at the time) IIRC denied it,
> Embassy officials have refused comment on a CBS News report last March that the Soviets spied on the U.S. embassy from 1982-84 by slipping eavesdropping devices into typewriters.
For a more modern example, there's STUXNET. Sure we know a lot about it, and they've more or less gloated on background, but they still technically deny any involvement. The official line is that it didn't happen. And there is no real confirmation that it was the US or Israel. Just assumptions.
As the decades pass, and the utility and political impact fades, STUXNET will be written about and redacted reports about it will be published. This timeline exists for a success. The timeline for a failure is likely to be longer.
None of the participants involved have the incentive to confirm this story. Apple and other companies will lose face. The NSA and FBI would inadvertently disclose sources and methods. SuperMicro would lose customers.
It is more believable to me that a state that has billions of dollars at its disposal has deployed such methods against major US companies than it is to believe that such an event hasn't happened. It becomes far more believable given the push to bring back foundries etc to the States. Is this a shot across the bow? A way to let most of congress know that this is a real concern? Who knows?
The balance of probabilities falls on the side of this being true rather than false. If you and I could think of this as a method to capture incidental usage data to map out US facilities etc and catch OPSEC fails, then the Chinese security apparatus can do so as well. However, unlike us, they have the ability to print money and throw both capital and labor at the problem until it resolves in their favour.
Are you saying it would be totally cool if the newspapers would get an AI to write totally random claims from a pool of somewhat likely scenarios and sell it as fact, because the fact that they don't provide sources and do provide fake pictures is completely irrelevant?
I don't think anyone says this cannot happen. What every claim against Bloomberg I have seen says are that these articles are not claiming it could and likely is happening but that they have proof and "look here it is" <shows fabricated pictures and writes text that shows the author doesn't even know how Ethernet cables work>.
Realistic scenario? Yes. Fake news? Most definitely.
I don't think there's a huge amount of disagreement that such attacks are possible and are carried out. The skepticism is that the specific claims of Supermicro's servers being hacked by adding another chip to the board and that Apple and Amazon were affected lack credibility (and in fact Bloomberg has many sources for the parts of the article relating to the idea that China carries supply-chain attacks on IT hardware but only one source on the chip story and plenty of very specific denials).
If you combine the tradition of "ghost" run and drop-shipping, you do not need that many control of points in the production and delivery chain to inject your payload.
The payload can be at the Bios or hardware level.
Haha yes, these guys have returned. Looking forward to buying some discount SMCI. Last time I doubled my money. Can't wait for the suckers who read Bloomberg News to give me more money.
Haha, looks like a "Fool me once, shame on you. Fool me twi- you can't fool me again" situation. No one listened to Bloomberg. Sad day. I was hoping to rob infants of their candy.
I have been thinking since the first time this report went around... what if it's remarkably clever disinformation? If you search for "China backdoor computers" this report comes up as the first zillion hits on google, there's no physical evidence attached, and if some fraction of the target population believes it anyway, it's pointing at a Taiwanese company, and not a mainland one.
we must have read different articles. I read one where multiple credible people went on the record to say the FBI had warned them that supermicro servers were compromised.
Not sure if it is an urban legend, but some people say "journalists" at Bloomberg get a bonus depending on how much their articles move the stock price of their target^^
> In an unusual disclosure for any public company ... We experienced unauthorized intrusions into our network between 2011 and 2018
Seems like the author is trying to create drama where there isn't any, here.
Is that an unusual disclosure given that it was made in 2019? I know very little about this subject having never worked (at least directly) in financials/compliance. I've participated in various parts of these things (mostly centered around IT SOX compliance about a decade ago). Something I recall about those days, though, was to "err on the side of disclosure". The game was always "The Department doesn't want to say (this thing) but the auditors want us to[0]". Even if it was based on very little, it's better to explain it than have it come up in a lawsuit.
By 2019, this Supermicro story was considered "fact" despite there being enough questions as to call the whole thing into question. The disclosure, from my perspective, served to help the company in two ways:
(1) Whatever Supermicro's official position about "being breached" up to that point were, they weren't clear. Whether or not upper management's official position was "we were breached/we were not breached" is irrelevant. In a lawsuit, discovery would likely turn up piles of e-mail from people far down the chain speculating on a breach in a manner that implies "the organization knew all along!" So they've put it to bed "we've been breached" is the official position. That's the starting point of "moving on"...
(2) Assuming it's true, they've now answered the question -- for customer's who took their business elsewhere -- is it safe to do business with Supermicro? Apparently a lot of people still think so because "None of these intrusions, individually or in the aggregate, has had a material adverse effect on our business, operations, or products."
Perhaps it's an unusual disclosure, but you could say the same about a good chunk of the disclosures we ended up putting together during compliance.
[0] "The thing" was always some reporting artifact that appeared to indicate a serious financial issue but was easily normalized -- i.e. an unexpected domain controller was added, an artifact of our reporting caused devices that logged in via that DC to report two of every license required, causing an $40 million gap in the budget. Tweak a SQL query and it's fixed, add the DC to the knowns and it's fixed, but spending an hour arguing why the original, wrong, report should not be disclosed was often the next required step.
--
I noticed that it's commonly accepted that Bloomberg's 2018 stories on the Supermicro hack were bogus, costing them a huge amount of reputational damage. However, Bloomberg stood by the stories. I'm very curious to see how this one will be received. I was never quite convinced by the naysayers or the denials by the government, Apple, or Amazon. I think it's quite likely that Bloomberg will be proven right in the end, this story having been accurate but suppressed for "secrecy" reasons all along.
Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. This time, it seems they have been able to do so.
Either way, I'm thrilled to see an outlet do original reporting and stand by their work in the face of universal condemnation. We are better off for having strong dissenting views informing us and expanding the narrative. I'm finding myself increasingly distrustful of the work of journalism outlets, but this (and those gone independent on Substack) gives me hope.
[1]: Previous discussion here: https://news.ycombinator.com/item?id=26112436