That’s a cop-out. It should be the case that even administrator access should not be abusable to implant permanent backdoors.
Anything that makes privileges escalation exploits more damaging is a real problem. I’m getting tired of how these are being dismissed as if admin access should mean that you can ignore any security issues. There are things that even admin accounts should not be able to change at the hardware level, or if they can they must be reversible in the future by another user with admin access.
> The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.
This is good practice but it shouldn’t excuse poor security at the hardware level.
Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
> It should be the case that even administrator access should not be abusable to implant permanent backdoors.
It's really the "permanently" which is the design flaw. Boards should have a mechanism to recover from bad firmware, and the same mechanism is useful to recover from a bad flash.
Make the flash chip removable, or leave a JTAG. Or have a bit of actual ROM with the write lines not even connected and just enough of a firmware to be able to reflash the main one.
The EFI firmware flasher or some other method needs to be able to erase and flash the entirety of BMC/SOC/BIOS ROMs from a trusted environment not dependent on a potentially-infected APT BIOS. Perhaps the SD card slot on many Supermicro boards and/or certain USB port should be able to flash a bricked BIOS & BMC just like some of the Asrock boards can flash themselves without booting using an additional button-holding sequence.
It is removable, by desoldering. This is not uncommon and Ars's sensationalized reporting does not help
This is exactly the kind of barrier you want for something with so much power over the system, otherwise you're not much better off than where you started as physical access allows for quick swaps of chips.
Desoldering is ridiculous. It's much more likely to damage the board, requires a much less common level of skill and doesn't allow you to check the existing data or do the reset prophylactically without performing the dangerous and expensive operation.
Meanwhile it provides no meaningful resistance against physical access because someone with physical access can swap the entire board or a dozen other things.
Many Supermicro server motherboards I've seen place both the BIOS flash chip and the BMC firmware flash chip in a SOIC socket, so that the flash chip can absolutely be removed without desoldering.
I have replaced thousands of flash chips on a running server farm, the guy who did the soldering had a 100% success rate in the end. My part was not perfect, so I agree it was hard but perfectly doable.
In theory, desoldering works. But so would scrapping & replacing all your servers after any "attacker might have gained BMC access" security incident.
You might see that as a facetious comparison. But the number of orgs which actually would desolder the chips in that circumstance is very close to the number which actually would scrap and replace. And if 99% of orgs won't actually do it when needed, then a "works in theory" method of re-securing servers is real-world useless.
The switch alone does not provide security if the supply chain is compromised. I believe a malicious actor could act along this chain by setting the switch to ON and rewriting the firmware, just like they would replace a removable chip. A step in this direction has been taken by "Server Configuration Lock" (e.g. HPE) while servers are in transit
I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Some of the Supermicro boards don't even have a separate BMC NIC, the only choice is to bond it to one of the main NICs or sacrifice one of them to be BMC only. I try to pay attention to that now after being surprised by that once on some servers we bought.
> I don't disagree, but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
Yes, all of which can be reversed by another admin in the future. That is expected.
It should not be the case that getting admin access one time can result in modifying the hardware in a way that can’t be reversed by future admin, short of physically reflashing the chip on the board.
On some boards you can access can reconfigure GPIO pins of the chipset and bitbang SPI from the application processor (aka your normal x86_64 CPU) without firmware support.
Depending on the implementation, kinda, but maybe not in the way you are thinking.
More generally, when you get down to the bottom of the pile of elephants, you are requesting some software currently running on your computer to write some bits to some kind of storage medium.
But there is no law of physics that says the software must to do as you ask! If the software is malicious, it can refuse. It could even pretend that it updated the bits but not actually do so.
"Oh, but I booted into $OTHER_PROGRAM and it writes the bits!"
Maybe. But how do you know that the boot loader faithfully loaded it? You don't. Maybe the boot loader is malicious and patches your firmware updater so that it won't actually write new firmware.
If you squint and tilt your head, it kinda looks like Ken Thompson's "Reflections on Trusting Trust".
> but if you have admin access to the BMC you can access the console, reboot the machine into single-user mode or even into another OS entirely, and then implant any malware you want, wipe the storage, etc.
True in the common case, but this can/should be guarded against by disk encryption and secured boot chains.
> Supermicro motherboards also commonly default to having a feature that bonds the BMC network interface to one of the main NICs if you don’t plug a cable into the BMC interface. It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Even if you changed the default from "failover" in the firmware you're just one CMOS reset (for whatever reason) away from it reverting back to putting the BMC on whatever network interface. This should really be something you can jumper off.
Well they'd still have the right, just not the ability (this is actually a distinction US courts have made regarding arbitration clauses and legal recourse).
This logic doesn't hold. If I choose to dban or degauss something, I don't expect I should be able to recover it later. Admins absolutely have the option to make irreversible changes, and do this quite often.
If you are magnetically destroying hard drives as part of decommissioning, that's not really the same thing. You're not using admin access to do it, and you're not making a change that permanently applies to all future use of the device (because there is no future use).
Yes but we were talking about hardware level changes and the person you responded to specifically said flashing was okay. So I thought you had something relevant to that in mind.
Anyone can delete a file. Nobody wants to ban deleting files.
Just because some administrative decisions are permanent and destructive doesn't mean that every operation should be made permanent or destructive.
Should every software config require buying new hardware because the initial config gets permanently flashed with an e-fuse to only allow a single write? You could even make a security argument for such a setup, but good luck getting approval for your 15th motherboard this quarter because you typo'd the config.
Also, dban and degaussing is not entirely equivalent -- from a practical perspective the equivalent is hard drive shredding (because the hardware cannot be used again in the old/non-malware config -- dban and degaussing are more like factory default resets). Do some organisations need to do this? Sure. Should we design systems with the assumption that any mistake means that the hardware is destined for the shredder? I would hope not...
Degaussing (for orgs who do it) is just as operational a task as updating. There will be an SOP that covers data storage decommissioning. It's not as frequent, but it's not any less 'normal', certainly not ad-hoc or one-off. You don't invest in a degausser "just in case".
> It’s common for people to be surprised that their BMC is exposed on their main network because they didn’t plug in a cable on the BMC NIC port at all.
Is this documented?
If it's documented, it's not an excuse for not reading documentation. You're not supposed to throw stuff (hardware or software) in production without reading the documentation.
I've made an effort but I can't find where this is truly irreversible. It just talks about usual things like reinstalling the OS not working because the malware is active before EFI. I didn't see reference in either this article or the linked one from Binarly.
I don't work with physical servers, so this is a gap in my knowledge. Isn't it the entire purpose of BMCs to allow for remote management?
So you'd definitely have to have it connected to the internet somehow, even if very indirectly, and in an independent manner (different network with no direct routes).
Of course a network can be offline. I believe that is what you describe, a network with no routes is not connected to anything else, and certainly not to the Internet?
It is common to keep admin and backup functions on separate network interfaces, on a disconnected network. You have to physically connect to the network in a secure location to use it.
No, that is not common. Management networks are almost never air gapped, they're just segregated from publicly-accessible or higher-exposure networks (DMZ, and hopefully prod). Requiring a (role-restricted) VPN connection is the most common way to control access to management networks.
Yes, you need some way to connect to the BMC. That is never going to be something like "eh just throw it on the public internet" or similar. It's going on a separate, isolated network with strict access controls.
There are others here more qualified to comment, but I do have some old rack mount enterprise servers making a racket in my basement, so I can provide some sort of half-informed opinion here--
Besides the security issue with the BMC here, it sounds like SuperMicro _also_ has absolutely insane defaults.
Every system I have has a separate, dedicated NIC for the BMC. There is an _option_ which is disabled by default to instead have it share one of the other interfaces. So the only ways you're going to "accidentally" put the BMC on an insecure network are:
1. Rebooting the server, going into the BIOS configuration, going into the options for the BMC, and explicitly telling it which other network device to use.
2. Physically accessing the server, attaching an ethernet cable to a clearly labelled BMC port, and attaching the other end to an insecure network.
From what I'm reading here, SuperMicro's default approach is apparently more "eh, just use whatever happens to be plugged in at the time". So even if you do have it running on a separate, secure interface... if someone unplugs it, it's going to switch to using whatever other connection is available!
To connect to the BMC on my servers you first need to be on the internal network already. Then you connect via wireguard to the router on the rack. Then you can connect to it and given a username/password you can log in. I would be pretty pissed if a cable got unplugged and that meant that the server decided to instead throw the BMC _on the interface connected directly to the public friggin' internet_. And this is just equipment I use for play and hosting my own random junk!
Yep. Defense in depth. Port-based VLAN at a minimum. Only total morons place their web-based and ssh remote/KVM management directly on the internet. Note that many Supermicro boards permit (as an option) to use the regular on-board NICs reducing the need to install a second or third cable for the BMC only adapter on mixed networks. Generally, one shouldn't be placing any box of any sort directly on the 'net unless it's hardened and behind sufficient network/application filtering.
Then you've already lost.
The BMC needs to be ideally on a physically isolated network, or at least a separate one that has no route from the outside nor on the machine itself.