I previously submitted this story a few days ago[1], where it garnered some discussion and a bit over 100 upvotes, but never made the front-page due to some down-weights in HN's algorithm. I'm reposting this story at Daniel's suggestion. He agreed it deserved a second chance to break out.
--
I noticed that it's commonly accepted that Bloomberg's 2018 stories on the Supermicro hack were bogus, costing them a huge amount of reputational damage. However, Bloomberg stood by the stories. I'm very curious to see how this one will be received. I was never quite convinced by the naysayers or the denials by the government, Apple, or Amazon. I think it's quite likely that Bloomberg will be proven right in the end, this story having been accurate but suppressed for "secrecy" reasons all along.
Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. This time, it seems they have been able to do so.
Either way, I'm thrilled to see an outlet do original reporting and stand by their work in the face of universal condemnation. We are better off for having strong dissenting views informing us and expanding the narrative. I'm finding myself increasingly distrustful of the work of journalism outlets, but this (and those gone independent on Substack) gives me hope.
It's not just the lack of sources, it's the fact that Bloomberg made up complete nonsense to fill in the bits of the picture that they didn't have.
Off the top of my head from that time, I remember they asked someone about the plausibility of some part of the story about the component types that might have been altered, he pointed them at some random example from Digi-Key, and that example became their photoshoot prop.
All of their photos were provably fake and staged, and made no technical sense.
There was that whole sub-story about how supposedly Ethernet jacks having metal shields was one of the indicators of compromise (what? almost every piece of datacenter kit uses Ethernet jacks with metal shields).
The more you looked into the details, the less they added up.
The only thing we could confirm was that, at some point, Supermicro shipped compromised drivers. That part I can absolutely believe, and like the last bit of this story update claims, that may well have been an APT hack from a state-actor. That side of the story is entirely believable and self-consistent.
But the hardware modifications? Sorry, but the way that story has been reported is complete garbage. I'm not saying it isn't true, I'm saying Bloomberg provided no credible evidence that it is true, and plenty of evidence that they don't have the slightest clue what they're doing, and that their reporting is dishonest.
Originally I gave them the benefit of the doubt, but when they published the Ethernet jack story with less than 5 days to check anything from that guy, they lost all semblance of credibility.
The reason that the information about hardware was vauge is probably because it was (is) most likely classified.
IMHO Bloomberg is trying to break this story without giving malicious actors useful information.
As a proof of concept experiment, I hacked together an Mcu, a few caps, resistors, diodes, and a small power transistor to see what could be done on a "single component" attack. I was able to receive data and inject malicious messaging into the data from a serial eeprom using a "filter capacitor" made up of my device (only 2 wires, 1 to signal, 1 to ground)
Obviously, not having access to a fab or able to integrate microscale circuits, my device would have been trivially detected (about 1 cubic centimeter) but with access to a fab and a bespoke lab, I have no doubt it could be manufactured down to the size of a standard filter capacitor (~3mm3).
I do not know if this is what malicious actors would do or not, but if I can build a 2 wire device to compromise the data loaded from serial flash, then any state actor can. If you compromise this data on the control firmware for a PC, you own the board.
All of the claims that the hardware attack is fanciful are clearly naieve, based on my experiment.
It is worth mentioning that the attack that I experimentally replicated is the hard version. The easy one is to substitute a modified device for the serial eeprom itself... Also invisible, and orders of magnitude more potent. A compromised eeprom could include advanced features including large amounts of storage and built in rf communication capabilities.
The naysayers in this case saying that the attack would be technically infeasible or excessively difficult to implement undetected are simply wrong from a technical point of view.
Compromised devices could be mass manufactured for < 10usd each, and if you can get a reel of them into a manufacturers supply chain, you can permenantly compromise thousands of devices with a single attack. The required tech is within reach of any state actor with access to a fab.
>The reason that the information about hardware was vauge is probably because it was (is) most likely classified.
This is irelevant to the article though. The article was clearly written by someone who didn't even know what an Ethernet cable looks like and someone who provided fake photos for the article. It makes no difference if it could have happened to the clearly fakenews article. There's a discussion to be had about the implecications of something like this happening but baseing it on a fluff piece is not the way to get anything useful.
There are entire MCUs that fit into the volume of an Ethernet connector's shielding. So there would be no visual way to tell if one were present on a board, unless the shielding or connector design was different.
Industrial X-Ray capability is trivially accessible and is frequently used for relatively simple tasks like ensuring an assembly is put together properly or that the correct part was ordered etc.
If your premise is that the article might tip off attackers to the idea that they have been caught, these points are moot. Either they have been caught, or the article is actually bullshit.
But that's okay, I'm going to hold your hand for a bit. Pretend the article said "three hidden microphones were found to be added to this board." That tells the attacker which board is being tested, that they are testing for microphones, and how many of their microphones have been found.
If your job is to hide microphones every day, do you see how this information could help you moving forward? Do you see why this is more than a binary caught/not caught?
There was no reason not to answer straightforwardly at the beginning. Weird that you didn’t.
Yes, it’s possible there is more than one attack and they want to avoid revealing which ones have been detected.
This may be the class of information they are protecting.
If so, the journalism leaves us in the same position as the attackers - the number of attacks we know have been detected is in the range 0 - n.
I.e. it’s possible the article is complete bullshit. Another possibility is that the article is itself disinformation. If attackers can’t tell then neither can we.
>it’s possible the article is complete bullshit. Another possibility is that the article is itself disinformation.
Okay, that's fair.
Your question was about what kinds of information an attacker can use. I didn't think of your question in the context of the entire article being disinformation.
I need to think more about that.
>Yes, it’s possible there is more than one attack and they want to avoid revealing which ones have been detected.
>This may be the class of information they are protecting.
>There was no reason not to answer straightforwardly at the beginning. Weird that you didn’t.
I'm glad I didn't, and I'm glad you took the time to puzzle that through.
>Nice try
You want to give your attacker as little information as possible. Even information about the kinds of information you're restricting can be used against you.
> I'm glad you took the time to puzzle that through.
I didn’t puzzle anything through. I just kept asking until you responded straightforwardly.
I just wanted to know what classes of information people thought the journalists might be protecting.
>Nice try
> You want to give your attacker as little information as possible. Even information about the kinds of information you're restricting can be used against you.
At some level yes, but not at the level of this discussion. The ideas we’re talking about here are widely present even in pop-culture.
> I just wanted to know what classes of information people thought the journalists might be protecting.
I just made a joke. You didn't get it (?), so I encouraged you to think through some possible answers to your own question. If you can think of more than one class of information that you wouldn't want your attacker to know you have, then you can arrive at the punchline yourself.
It's not too late to try it, by the way. Thinking like the attacker is a good exercise. The attacker doesn't even have to be real.
>At some level yes, but not at the level of this discussion. The ideas we’re talking about here are widely present even in pop-culture.
I'm not sure what you mean by this, but operational security would still apply even when you have to make press releases. Companies juggle this all the time, e.g. sharing just enough about their proprietary technology to attract new employees/customers without giving everything away.
> I just made a joke. You didn't get it (?), so I encouraged you to think through some possible answers to your own question.
If you really were making a joke, you could have just said so when I ‘didn’t get it’. I’d probably have said something like ‘fair play’, or ‘nice’.
Because you didn’t we are left with other possibilities to consider:
1. You were just being condescending from the beginning, and are now trying to claim it was a joke to save face.
2. It was a bad joke, and you doubled down on condescension to save face.
The evidence supports either of these because ‘encouraging someone to think something through’ when you haven’t engaged in good faith conversation is an ignorant and condescending move.
Have you considered the possibility that I am asking what other people think because I want to know what other people think, and that ‘being encouraged to think it through’ will simply not answer that question?
There is nothing to suggest that you are an expert on this topic. Your own knowledge of it is limited despite having ‘thought it through’, proven by the fact that my response to your first real answer immediately showed you had missed something.
There is nothing wrong with making jokes, even bad ones, but following up with condescension makes it look like you weren’t actually joking.
We’ll never know now.
> I'm not sure what you mean by this, but operational security would still apply even when you have to make press releases.
Are you trying to create the impression that you are in personal possession of secrets about this attack?
>If you really were making a joke, you could have just said so when I ‘didn’t get it’. I’d probably have said something like ‘fair play’, or ‘nice’.
Lol ok
>Have you considered the possibility that I am asking what other people think because I want to know what other people think, and that ‘being encouraged to think it through’ will simply not answer that question?
Had you considered the possibility that you asked a low-effort question? "What kind of useful information [might malicious actors look for]?" is about as deep as "hey how long should my password be". You are on Hacker News. I honestly thought you were joking when you asked it.
Do you really just want to talk about how the article makes China look bad? That's legit, but not at all related to the question you asked.
>There is nothing to suggest that you are an expert on this topic. Your own knowledge of it is limited despite having ‘thought it through’, proven by the fact that my response to your first real answer immediately showed you had missed something.
Nobody said I'm an expert? You asked about very basic threat modeling and I tried to walk you through a very basic exercise.
The thing I missed (the article itself is disinformation) wouldn't actually change much about the threat model. The writers still have to decide what/how much information about their business practices to share, and the attackers still gather every bit of info they can. The attack in the article could be a lie. The article could still accidentally revealed something new (or confirm something old) to any potential attackers. If the article mentions a specific factory, for example, another data point an adversary can use to model their supply chain. If it mentions a name and that name leads to a LinkedIn profile, the attackers get a little bit of the company's org chart. Those little leaks add up and must be actively managed.
>There is nothing wrong with making jokes, even bad ones, but following up with condescension makes it look like you weren’t actually joking. We’ll never know now.
I invite you to puzzle that one through yourself :-)
>Are you trying to create the impression that you are in personal possession of secrets about this attack?
> Had you considered the possibility that you asked a low-effort question?
As I said - condescending - and now we can see that your answer was in bad faith. That’s not encouraged on hacker news.
> "What kind of useful information [might malicious actors look for]?" is about as deep as "hey how long should my password be". You are on Hacker News. I honestly thought you were joking when you asked it.
Obviously not. You can’t claim to know much about the subject.
> is about as deep as "hey how long should my password be"
You don’t actually know what range of answers a good faith commenter might have.
You are simply confirming that your response was in bad faith, as it appeared.
> Do you really just want to talk about how the article makes China look bad?
That's not how this works. Journalists frequently don't provide public proof, because that is frequently impossible. They convince themselves an article is true (often by using sources they cannot reveal) and then publish whether or not they are able to find proof that they can reveal.
Yeah, I'm confused by all the posts acting like the objections were just that Bloomberg's first story didn't provide any evidence. It was quite a bit worse than that; they included some very obviously faked evidence and a lot of utter nonsense. I wouldn't be surprised if it turned out to be "based on a true story" but even if every single one of their claims turns out to have been a distorted retelling of something real then I'll still stand by my opinion that it was terrible reporting.
Oftentimes, media will use imagery or illustration that are there for illustrative purposes. I don't think Bloomberg claimed to be showing actual photographic evidence. Not to mention, oftentimes the story and the presentation come from separate sources — the editors taking the written story and deciding the headline, and how to present and illustrate it.
That is a huge pet peeve of mine, using images for "illustrative purposes" does NOT mean "to illustrate, i.e. show in images, what we're talking about".
Instead, it means "to give you an image to rest your eyes on, so reading all of these annoying letters doesn't make them catch on fire" or something.
I agree, it annoys me too - especially in astronomy posts. ...but the point in this case is that it should not be used to discredit the Bloomberg article.
> Oftentimes, media will use imagery or illustration that are there for illustrative purposes
"Hitler was a really bad man. Here is a picture of Hackernews user `kenneth` for illustrative purposes."
Something like that happened in France a few years ago. A TV channel illustrated a news segment about cybercrime with the first B-roll of computers they could put their hands on. It turned out to be shots from a segment on a not-for-profit org teaching computer skills to the poor (including former inmates). The org was pissed, the TV channel just said it was "for illustrative purposes", as if that was justification enough.
I worked at a scientific institute. Every time we got some airtime, they asked the director to dress in a cleanroom suit. This man has never worked in a cleanroom, ever. But hey, it looks interesting for the layman. I've gotten extremely sceptical at whatever images the news is showing. They don't date nor give source for their material.
How is that different from any other article on any other topic? Everytime I read an article on a topic I am familiar with, it is full of approximations, misunderstanding of why things work in a particular way, exaggerations, etc. The standard you apply would remove any credential to every newspaper.
I don't think it's very common for newspapers to post comparable allegations of fraud or murder. Do you?
When reporting that someone has been arrested, indicted or is under investigation for such crimes newspapers aren't actually alleging that they did the thing. That seems quite different from what happened here.
>Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity.
It's not just that - it's that no other reputable outlet was able to confirm the story, and many tried. This just doesn't happen with controversial but true stories with many sources - some of those sources will be willing to talk to other outlets.
Bruce Schneier said last time that he didn’t believe it. This time he said he was wrong. When pressed by a skeptic in the comments, he said, “Independent from this article, I know that the DoD believes this threat is real and is giving away R&D funding for security solutions. So I have more than the Bloomberg reporting to go on.”
All you people are in denial. The first bloomberg story was credible but was protecting sources. They go further and get people on the record and double down....literally stretching their necks out, and you people keep propping up strawmen from the original article. Admit it. You were skeptical, determined that it couldn't be true, ran around shooting your mouth off about how it was fake news and now you can't help but continue smearing the egg all over your face.
As others have pointed out, the problem was not (just) with lack of sources, but with the fact that the few technical details they gave didn't make sense. As a result, the article read like it was written by someone who was completely ignorant and was just not credible on the face of it.
I think this twitter thread is a very reasonable analysis of this article: https://twitter.com/pwnallthethings/status/13602327536236298.... I think critically there are a lot of claims here, and none of them provide _concrete_ proof of any form. There's a lot of 'senior official said X' and X number of people in the know declined to comment, but these are just appeals to authority/credentials. People are flawed and second hand rumours of rumours can easily bend the truth. I don't think it's necessarily fair to put out a story claiming this is 'the truth' when alot of the sourcing is indirect and 'declined to comment'. It's just not there with the facts! True or not, we can't know, but this report need more facts.
> Alarmed by the devices’ sophistication, officials opted to warn a small number of potential targets in briefings that identified Supermicro by name. Executives from 10 companies and one large municipal utility told Bloomberg News that they’d received such warnings.
Only if you believe that the warnings went out because of the "alarm over the devices' sophistication" and not "alarm over the idea this could happen" or even "alarm over a misidentification of a legitimate component and now we're too embarrassed to issue a retraction to those 10 CEOs"
The article certainly quotes a wealth of sources - but all of them seem to be vague, third-hand stuff.
An unnamed "adviser" to security firms that analyzed Supermicro equipment. An executive for some unnamed company, who received a briefing. A venture capitalist who received a briefing. A retired FBI agent who was told there was an "additional little component" by someone he can't name. Some former FBI officials that refuse to name supermicro, and say only that supply chain attacks are possible. Fifty interviews with officials, all of whom asked not to be named.
So apparently this information is so public that everyone and their dog is happy confirm it - yet at the same time, so classified that the victims and the 50 sources can't be named?
That's pretty vague compared reports of hardware implants like credit card skimmers [1] which put pictures of the hardware front and centre and make it clear that the author has personally seen it and knows (basically) how it works.
first, I don't understand how this quote squares with your overall description -- it's specific, first-hand, from a named source, who should (on its face) know what he is talking about:
> “This was espionage on the board itself,” said Mukul Kumar, who said he received one such warning during an unclassified briefing in 2015 when he was the chief security officer for Altera Corp., a chip designer in San Jose. “There was a chip on the board that was not supposed to be there that was calling home—not to Supermicro but to China.”
second, the article presents a plausible scenario for why it would be difficult to get additional details: the U.S. govt's strategy for dealing with this was to let it play out, so they could learn more about the nature of the threat. any public disclosures about this are at odds with the strategy of the U.S. government in combating it.
it seems like a very realistic scenario to me that some of the details of what happened became "lost in translation" - but that there is a real underlying truth. the first article wasn't convincing to me, but this one is very difficult to dismiss.
> first, I don't understand how this quote squares with your overall description
A person who "received a warning in an unclassified briefing" isn't first-hand. At best it's second-hand, if whoever analysed the hardware implant was going out giving briefings in person. More likely it's third-hand.
> it seems like a very realistic scenario to me that some of the details of what happened became "lost in translation"
If the claim was an evil driver update, or a backdoored BIOS, that would be completely believable.
Indeed, most of the details of this attack could just be a miscommunication about a BIOS backdoor - supply chain attack, malicious, code that shouldn't have been there, stored in an eeprom on the motherboard, undetectable by visual inspection.
But the much more astonishing claim of a malicious hardware implant between layers of a PCB? Something that surprising needs the testimony of an electronics expert, not a cop or a C-suite officer. Especially after the purported victims from the first article denied knowing anything about it.
"first hand" as in, 'the govt told me that china bugged supermicro motherboards,' as opposed to, 'I was told the govt briefed tech companies about how china bugged supermicro motherboards.' honestly, describing the testimony I quoted as "third hand", simply because the person doing the briefing for the govt was not the same person to have discovered the breach, is completely ridiculous.
do you agree the article establishes that the U.S. govt repeatedly warned private sector companies that supermicro servers had been bugged by china? maybe the us govt was wrong about that! could be! but it's still a huge story and much more than what was established by the first bloomberg article.
> do you agree the article establishes that the U.S. govt repeatedly warned private sector companies that supermicro servers had been bugged by china?
I agree the article establishes that unnamed sources in government are giving out briefings.
But I don't think that was ever in doubt.
Even when the first article was written, I could easily believe the journalist's claim they were told what they wrote by some sketchy anonymous government source. I just think they didn't demonstrate the truth of their source's claim.
There's always some anonymous "senior government official" ready to tell a journalist that Iraq has weapons of mass destruction (but they can't show the evidence as it's classified) or whatever other narrative they're pushing at the time. It's a journalist's responsibility to apply due skepticism.
for whatever reason, you are refusing to grapple with the fact that the second, just-released article, does not rely on the testimony of anonymous government sources, "sketchy" or otherwise, but on multiple, named, private sector sources. the people who are quoted have reputations that are important to them. they are claiming to have participated in concrete events. it's a very different category of testimony than "government sources say."
the difference is, the motivation for the two kinds of communication are completely different. anonymous govt sources talking to the media want the info to be public. govt officials telling a limited circle of people involved in core infrastructure generally do not want the info to be public. if you are trying to broadcast info to the public, your message is much more suspect, because the likelihood of an intent to deceive is higher. if you wanted the message to get out, there are far easier ways that telling a small circle of private sector people. generally govt isn't going around telling dams, nuclear power plants and drinking water plants about bogus chinese hacking threats on purpose. they can easily be wrong. it's a lot less likely they are lying about something like this.
This is `usually how national security problems are reported on, due to the classified nature of the work. You don't have to believe it, but it's naive to think that you're going to see actual chips. Currents and formers can talk about the what, but not the how. I mean even China knows the "what". Methods and means are a big deal, PRB kills entire chapters because of it.
Why do you completely cut tie with a big supplier over security concerns and at the same time don't disclose why? I'd say this is exceptional in the business of IT-security.
For a story claiming physical alteration of products, they should have had photographs of altered products. Otherwise it's just "anonymous loudmouths blow smoke".
Everything that would be done with the extra component could more easily be done with BIOS code, attracting no attention. So it doesn't pass the sniff test.
I don't think you deserved the downvotes as I felt somewhat the same way. Part of that comes from the reporting, which appears to have been done by someone who doesn't understand the things he's reporting on (or, at least, some of the people quoted are saying things that are nonsense that someone outside of our industry wouldn't necessary recognize)
i.e. "had a chip encrypted on the motherboard that would record all the data"
... one does not simply ... encrypt a chip ...
With these kinds of explanations or statements used as evidence, it's frustrating trying to figure out what "really happened". Personally, I don't doubt there's something to all of this. It's a "real threat" as has been reported; whether or not it's "theoretically real" or "actually used" isn't that relevant: if it's a "real threat" someone has figured out precisely how to do it or has seen it (or close-enough variations of the "it") being done that it's worth taking action.
My hunch is that it's a firmware-related hack; no hardware added to the boards. The article talks around BIOS and such, indicating that some of the code involved was inserted by employees directly connected to the company. It explains the lack of photographic evidence--what's the evidence[0]? And wouldn't the attacker prefer it that way? It also helps explain the "encrypted chip" -- perhaps code, encrypted on the chip (decrypted/executed at boot) was what was intended by that statement[1].
[0] Disassembled code on Github would have been nice, but the US government isn't likely to drop that.
[1] And maybe it was said in that bizarre manner intentionally, who knows?
>Everything that would be done with the extra component could more easily be done with BIOS code
This is tricky because on newer platforms the BIOS is signed and verified by the CPU. The signing keys are likely going to be located in the engineering offices (in the US) rather than just lying around in the factories, so getting them would require much more access than just bribing/threatening a few technicians at the factory.
> attracting no attention. So it doesn't pass the sniff test.
Compromised BIOS is a known threat, so I'd presume paranoid customers would verify/reflash the BIOS upon receiving them. This can be done without booting the motherboard at all (ie. attaching SPI clips to the flash chip and reading/writing directly). A tampered bios would get detected this way so it's not exactly "no attention".
> We are better off for having strong dissenting views informing us and expanding the narrative.
Strong dissenting views is a means to an end, not an end.
The truth is some point in the space of plausible ideas. If the region covered by existing ideas doesn't overlap the truth, then yes, you need to expand it. If it does, then looking elsewhere is guaranteed to bring you farther away from the truth.
If the story is false (or, more precisely, Bloomberg's reporters are talking to people who aren't technically qualified to analyze the subject and are misinterpreting it and Bloomberg is putting its own spin of misinterpretation on it), then Bloomberg's contribution to the discourse - to the pursuit of truth - is of no more value than the National Enquirer saying that Batboy hacked all our computers. Possibly of less value, in that people know not to take the Enquirer seriously.
The ability to have a wide range of views expressed is good for the pursuit of truth, yes. That doesn't mean that a dissenting view is of merit because it is dissenting. And in the case of this article, when it was based on a previous one that was attacked on the merits and not because of how dissenting it was, and seems to be more of the same, a commitment to the search for truth requires us to dismiss it.
This is interesting how and why reasonably credible stories instead of being investigated get ignored by mainstream media all the time, recent example being NY Post.
In this case Bloomberg comes back with the vengeance, but we need to see if it will picked up or will be buried again.
NYP is garbage, quite famous for splashing completely untrue stories on the front (or top article on WWW) and then retracting it a few days later in the most remote areas of their media.
> Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity.
Wait is this all of a sudden a prerequisite for something being believable?!?!?! What the hell did I live through since 2016?
As a bicoastal American, there is a strong East-West schism in that. West Coast techies, largely disconnected from D.C., being (rightfully albeit overconfidently) sceptical. East Coasters being (almost always overconfidently) confident in their friend at such and such (almost always unrelated) agency vouching for the story.
Well actually kenneth is behind everything and we can prove is because we have the sources to back it up.
No of course we can't give out the sources, our journalistic integrity requires us to protect our sources. But we fact checked everything and it's 100% true.
--
I noticed that it's commonly accepted that Bloomberg's 2018 stories on the Supermicro hack were bogus, costing them a huge amount of reputational damage. However, Bloomberg stood by the stories. I'm very curious to see how this one will be received. I was never quite convinced by the naysayers or the denials by the government, Apple, or Amazon. I think it's quite likely that Bloomberg will be proven right in the end, this story having been accurate but suppressed for "secrecy" reasons all along.
Notably, one of the criticisms at the time was that Bloomberg didn't name credible sources with first-hand accounts of the events — understandably given the sensitivity. This time, it seems they have been able to do so.
Either way, I'm thrilled to see an outlet do original reporting and stand by their work in the face of universal condemnation. We are better off for having strong dissenting views informing us and expanding the narrative. I'm finding myself increasingly distrustful of the work of journalism outlets, but this (and those gone independent on Substack) gives me hope.
[1]: Previous discussion here: https://news.ycombinator.com/item?id=26112436