> This is the most important infrastructure project that we’ve had in this country for decades. Everyone agrees — this is non-partisan. Everyone knows we have to do it.
Considering the current political climate and rampant government cuts to important services, I very much doubt “everyone agrees” and that this is the best time to be planning such an important transition.
Yeah, couldn't this easily split in a group supporting the FAA to implement a better system, versus a group trying to contract it out to the private sector? Before you know it, IBM* is printing money again. (* substitute with Evil Corp of your choosing)
And then what? Those words mean nothing to the people with the most power and motivation (or lack of care) to derail the whole thing.
It’s about as effective as placing a monkey in a porcelain shop then walking away while commenting loudly “Now now, it is very important none of the porcelain breaks, everyone knows it must remain intact”. The monkey doesn’t give a shit.
Butchering a proverb: “The best time to reorganise your porcelain store was before you bought a monkey. The second best time is after you sell the monkey.”
I sincerely don’t even know what you’re talking about right now. But it definitely isn’t related to my original argument. My point was about the task and the right time to tackle it, while you seem to be hung up on the words of the Secretary. The words are immaterial to this practical matter, as is any vague general concept of “changing the world”. I’m talking about this specific case, not building a philosophical thesis on the subject of improving humanity.
I get that FAA hardware/software is a time-tested, safety-critical system that has resisted many prior modernization efforts but...how do other countries run their systems? Surely they're not all using floppies. I doubt there are many (any?) countries with a flight volume like the US but overall, flight safety is pretty good world-wide (again, with exceptions).
Their governments fund the upgrades instead of running around claiming their flight agencies are full of corruption and inefficiency with no basis in reality.
The upgrades have been funded for decades. It is an execution issue, not a money issue. Many other parts of the Federal government are in the same condition: software upgrades that are infinite money sinks that never produce much after decades of effort.
I've worked around some of these programs. I've had visibility into some of them for 15 years over which there has been zero forward progress despite unreasonably large amounts of money being spent. It is no secret why those programs are permanently broken but no one wants to have that conversation.
I think most takes on this are overly reductive. The whole situation is sad really.
The root cause, to the extent that one exists, is that no one is accountable for successful execution in a very literal and systemic way. Some parts of the government I've worked in are worse than others, but it is endemic. This leads to a textbook case of Pournelle's Iron Law. There are no negative consequences for a handful of people aggressively maximizing their personal benefit and acquisition of power as their primary objective. This is how you get the fiefdom-building, feather-bedding, and the usual revolving-door corruption that these programs are notorious for.
Most people involved in these programs aren't like that but enough people are that it is impossible for people trying to do their jobs competently to get anything done. The people that defect are the people that end up controlling the programs because that is how the incentives work.
Inefficiency and corruption are a symptom, not the disease. The incentives virtually guarantee that these programs become playgrounds for sociopaths. Average workers on these programs are put in the demoralizing position of either having their good effort constantly undermined by leaders that don't care about the mission and are openly making decisions for personal benefit or to defect to the side of the sociopaths so they at least get some personal benefit out of it. Most of the best and most competent people I know eventually leave Federal service entirely.
A second-order consequence of this is that over time, no one competent wants to work on the programs that are run this way. Through churn these programs slowly fill up with mostly useless seat warmers who don't mind a job where no one expects productive outcomes. It is a kind of stealth UBI for government employees. Some people request assignment to these programs.
You never hear about the programs where the leadership is actually competent and cares about the objective because these actually function pretty well. But the incentives are such that this is the exception rather than the rule.
I'm not even sure how you would fix it, I suspect it is politically impossible. When companies become overtly like this they tend to slowly self-immolate into irrelevancy. Governments lack these negative feedback loops in any meaningful sense.
I remember reading a drone startup saying they had an easier time operating in Kenya than in the US because Kenya's ATC system was fully modernized, with every aircraft tracked at all times.
For retrofit purposes, it's probably attainable to use solid state (no moving parts) floppy disk emulators that use USB thumb drives or CF/SD cards instead of error-prone, real floppy disks. Every time a floppy drive moves over a sector to read or write, it wears that area mechanically. Magnetically, bits just seem to rot from floppy disks randomly with time more likely failure mode for previously good floppies.
Let me complain you about how error-prone and unreliable are real floppy disks. ):
> For retrofit purposes, it's probably attainable to use solid state (no moving parts) floppy disk emulators that use USB thumb drives or CF/SD cards instead of error-prone, real floppy disks.
Yes, but if it is just a PC running Windows 95, likely simpler to get the software working under newer Windows, or if worst comes to worst, keep Windows 95 and stick it in a VM. I doubt there is any specialised hardware on the Windows 95 machines, the specialised hardware is likely connected to something else.
The use case where physical floppy emulators really shine is with much more exotic legacy systems. Some years ago there was a furore that the US nuclear arsenal was still being managed using 8-inch floppy disks (used in IBM Series/1s, 16-bit minicomputers from the 1970s). USAF was proud to publicly announce they’d successfully transitioned the US nuclear arsenal to be floppy-free. I don’t know if they said publicly exactly how they did it, but I suspect they kept the Series/1 minicomputers and just replaced the 8-inch floppy drives with hardware emulators (which probably each cost an utter fortune when you add up the premiums anyone will charge for it being the military, being highly classified, and above all being related to glowing things that go boom)
As of the early 2000s, ATC was still using vacuum tubes. In fact, the FAA was the single biggest buyer of vacuum tubes in the world at the time, almost all of them sourced from former Soviet bloc countries. I think they've all been replaced by now, but I can't say that with 100% certainty.
Which is what baffles me about the current situation and gives me a lot of hope for this effort. We should've been updating this stuff in the 90s, but successive administrations of both parties have just passed the ball on this one.
I see this the opposite way: kudos to the FAA for sticking it out so long on legacy hardware and software as long as they have!
ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now. Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
If they're saying that they need the upgrade now, I'll trust them on that, but it was the right call to make it last.
> ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now.
The problem is that Eurocontrol (for example) has modernized their systems without much fuss, and UK NATS even has remote tower ATC now (https://www.youtube.com/video/Ii_Gz1WbBGA). It seems that FAA is stuck in the past, not just using old systems because it's reliable.
> Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
I would agree if the system is still fit and proper, but even in 2005 the ATC systems in the US is not really fit and proper that there has been multiple plans to rehaul the system. It is really miraculous that the only system failure happened in 2023 (NOTAM offline), but that's due to tireless dedication that's certainly burning unneded manpower.
Unlike in Europe where civil servants have the sway to just do it, it seems that the US is an expert in political bickering on things that aren't really political.
My understanding is that remote tower ATC is something that had to happen at that airport due to geographic constraints rather than some kind of next step for ATC in general. Given a choice between being able to physically look out the window and not, from what I understand being able to see out is always preferable.
The rest I don't know enough to comment on, so I'll assume you're correct.
> ATC is a safety-critical function that has what amounts to a 100% uptime requirement. Whatever system they're running currently either works or has known flaws that they know how to work around, and air traffic controllers have been trained on these systems for more than a generation now. Upgrading merely for the sake of being up to date would have been foolish no matter how much funding Congress would have given them.
I do not have enough knowledge to disagree on this. But I will say the FAA is still on floppy disks when the US Nuclear Arsenal moved off floppies back in 2019.
Yes, they have different requirements and yes, SACCS was using 8 inch IBM mainframe floppies from the 70s, but they are both 24/7 critical systems.
> If they're saying that they need the upgrade now, I'll trust them on that, but it was the right call to make it last.
The real answer is likely embarrassing incidents that came up during the start of this presidency. There is now political will to address it; instead of 'before' it becomes a problem. They are on Windows 95-it was budget issues.
> I do not have enough knowledge to disagree on this. But I will say the FAA is still on floppy disks when the US Nuclear Arsenal moved off floppies back in 2019.
Well this isn't very long in terms of overhauling safety-critical systems that have many decades worth of processes and infrastructure built up around them, is it?
The problem is all of the big software consultancy services are optimized to maximize revenues / minimize their own risk when working with big / dumb government agencies.
A first step to mitigate some of the risk would be to move the
system to a virtualised system. This could be in each location
or more centralised which would make the maintenance of the
fleet of old computers easier.
Floppy can be copied to hard disks and will not have to worry
about failures of mechanical parts involved in reading floppy drives.
Developing a brand new system would take quit a lot of time.
As all systems du if they need extreme uptime.
Starting that effort now is ok but I would guess it would be take
at leas a couple of years. Significant work would have to understand
in detail what the current system does and does not do, and then
map out what a system should do.
I wonder if anyone makes a virtual floppy drive that replicates the performance characteristics. I.e. to avoid a faster virtual drive uncovering dormant race conditions. Something like a developer assuming "I have enough time to do this processing before the disc makes another rotation" etc.
I think any of the "off the shelf" gotek emulators should suffice for this. They're made for people to keep playing games on old hardware. I would assume copy protection and other shenanigans would be the creme de la creme of abusing the hardware.
This is to get rid of the media only. You'll still be using the original compute hardware. But it would be an interesting step.
I feel that most of the desire to upgrade is cultural and not technical. People love to talk about the floppies being used while its just a small part of the equation. Cost and risk of creating a new system with the same reliability expectations is hard when the incumbent has decades of iteration. For systems that do not require more performance or energy efficiency the accounting on upgrading looks very different.
We IT folks tend to quickly propose solutions to systems whose complexities we do not completely understand. That's fine when it is about serving ads or managing book orders. It's not ok when the stakes are high.
Virtualization just adds another layer of complexity to an already fragile system which literally thousands of human lives depend on every day. Adding more complexity is not a neutral act here, but neglectful manslaughter waiting to happen. Aviation is a low-tech, never-touch-a-running-system, risk-averse environment for a reason.
Floppies were useful because you could easily take them and take them to another, secondary, sometimes air gapped backup system. Replacing this functionality means replicating not just the data transfer, but also the safety architecture - which includes physical isolation and manual fallback paths. To recreate, the best chance would probably be something like storing the relevant info on thumb drives - but then you have whole new family of attack vectors by hostile forces (anyone still remember Stuxnet), which floppies did not have in that form?
And then there's the pesky aspect of international interoperability. One country alone cannot just storm forward. We are looking at decades of upgrades and alignments here. And that process already is underway. But proposing a radical change without acknowledging the full scope of what that entails - from certification cycles to human factors to geopolitical coordination - is not progress, it’s hubris.
Eeeeexcept that floppies are horrifically unreliable. I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install. I'm still salty about that one.
"It's not broken" is the cry of the bad manager that hasn't done the proper analysis, hasn't actually looked at the pros and cons, but has simply become complacent and comfortable with the devil they know.
If they're still using physical floppies, then their process is broken now, so virtualising it will almost certainly un-break it.
A simple "clarifier" for this kind of thought process that I like to use is: If you were already using the new option (virtualised legacy hardware), would you think it a good idea to convert it to using open drives with convenient dust ingress, non-existent support and supply chain, glacially slow mechanical moving parts, and hilariously antiquated crunching noises for all data access? Would you? Really? Or would you recoil in horror at the very idea?
I use the same kind of logic on people who think staying on Windows Server 2012 in <current year> is a good idea. Would you downgrade Windows Server 2025 to 2012? Why not? You think it's a great platform, apparently!
PS: I worked on a large scale DOS-era software virtualisation project where we moved ~20K users onto a Windows + Citrix platform. We eliminated about 6000 floppy drives and about a million(!) tapes, and the resulting system was so much faster and reliable than the original that people were trying to bribe the project manager to be put at the front of the migration queue.
> I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install.
I love this fixation on floppy disks. The article likely brought it up because it is a recognizably obsolete technology, but didn't cite why (or even if) it was a problem. I'm sorry, but a nightmarish software installation scenario doesn't cut it. It is highly unlikely that they are doing in situ software installations from floppy diskettes.
The danger in such armchair quarterbacking is that it undermines the authority of the agencies that are in charge of making decisions. If there are legitimate reasons to question their authority, by all means do so. Yet, when doing so, understand their requirements and provide evidence as to why their authority should be questioned. Also be prepared to be unsatisfied by some of their answers due to differences in perspectives.
You make good points, but now Citrix and Microsoft have them over a barrel. Curious how such a migration looks in 2025 with Microsoft pushing everything to Azure, though and Citrix's acquisition by Vista Equity (2022).
> I remember feeding disk number 27 out of 33 only to get a "bad sector" error an hour into a software install. I'm still salty about that one.
That's why mission-critical systems have several sets of floppy disks, and disk-multiplication stations.
> Would you? Really? Or would you recoil in horror at the very idea?
Depends. If the old system is certified and has all error modes defined, while the other new system is a black box with exciting new ways to screw up, I'd go old system ten out of ten times. Which incidentally is why NASA uses ancient chips when they build new robotic drones.
> I worked on a large scale DOS-era software virtualisation project where we moved ~20K users onto a Windows + Citrix platform.
Respectfully: How many lives would you have extinguished had your new system failed? How many failure modes did you encounter during your virtualisation project? How many external systems - which also relied on a very specific way of doing things and would have murdered people if talked to wrongly did you interface with?
No need to answer. We have all had such projects. We know things break before, during, and after the switchover. Only in some environments, systems absolutely cannot break, ever. Aviation is not your average 'let's get us a new mail server' migration project.
He also pointed at the root cause and possible solution there. Which is re-categorizing spending on this as essential instead of as something that's nice to have and becomes the victim of cuts almost immediately after anytime some budget is actually allocated.
An interesting point here is maybe that there's a whole world outside the US where planes fly and communicate. For example the EU has its own issues on this front but is modernizing what it does. Airspaces here are pretty dense and busy. It's not necessary to reinvent a lot of wheels here. The US could just look across its borders and learn from what is being done there.
As soon as there's a reasonable budget for this, there are all sorts of perfectly reasonable things that can be done. The core issue isn't technical.
> For example the EU has its own issues on this front but is modernizing what it does.
The EU has been almost single-handedly designing the international interoperability standards for the kinds of information that the US uses floppies to move around.
I'm not sure any country still remembers the lessons that would be useful for the US on this one situation.
Would you like to trust your life win95 and floppies definitely no but paper strips is something really robust and in light of crowd-strike or the outage in Newark I think a truly independent backup ‚system‘ is a good idea. Particularly as the next system will come with some early bugs.
Setting a protocol to handle air traffic control and collision prevention in airspace around airports is a 100% automatable problem. You don't even need a centralized control system. This can be handled entirely with software running on each plane. Same way a flock of birds can fly and never collide with each other.
Unfortunately that's not how things work in practice https://en.wikipedia.org/wiki/Split-brain_(computing) If Jepsen fails every database coming from single source, imagine the chaos of synchronising a P2P of various clients of various versions over a very noisy link. We can't even achieve that with home automation meshes that send maybe 3 types of messages!
Also you need to handle planes without computers - you can land a personal plane at almost any airport. (With lots of caveats but still) Also you need to handle planes with failing automation. Also you really want to know the situation on the runways, so there's really no need to remove the single source of truth here.
We operate cars on the road with not only no centralized system, but also minimally defined and enforced protocol, and yet Waymo has achieved a near zero collision rate inside a swarm of cars that are not running equivalent software. And this is in a situation where cars are only a few feet from each other while operating at top speed. So you can come up with a million objections but they are all solvable. As for automation failure, the rate of that can be easily made lower than the rate of human failure, which currently is fatal to a plane.
What happens when an airplane's pilots have to radio ATC to request an emergency landing, and the planes' sensors have failed so it can't safety land itself?
If the plane is able to communicate with ATC it is able to broadcast to the other planes that it is on an emergency landing, so this is not an issue. Even if it has lost all communication this is still not an issue because all the sensors of the other planes can see it. So if the disabled plane immediately goes for an emergency landing, all other planes in the area are able to see its position and that it is not responding to pings, and therefore set safe courses that avoid it. This really isn't a very difficult problem.
The problem, once Congress gets wind of the amount of real money that will need to be spent, plus the time it will really take to develop and fully test, it is cancelled.
Of course I fully expect this to be TIP (Test in Production), thus for maybe 10 years, flying in the US could be very dangerous. Lets hope the pilots will be able to manually avoid other planes.
Does that matter? Manufacturing them ain't hard and current supplies are big enough to sustain the small demand. What matters is that I can buy floppies and will be able to for a very long time if not indefinitely.
Considering the current political climate and rampant government cuts to important services, I very much doubt “everyone agrees” and that this is the best time to be planning such an important transition.
reply