MIMO and nonlinear control theories are probably some of the hardest topics in all of engineering. SpaceX control system also has to compensate for the fuel moving inside their rockets so the control algorithms probably involve some kind of fast numerical fluid simulation.
Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips.
> control system also has to compensate for the fuel moving inside their rockets
My stepfather worked as a programmer on the Apollo program, and the thing he always talked about as his biggest accomplishment was working on the "slosh problem" -- so yeah, props to the SpaceX team for managing that landing. And props to my stepdad for managing it on hardware that was... a billion times less capable? :-)
I might be misremembering but I think slosh was the failure cause for one of the three failed Falcon 1 flights. It was number 11 out of a pre-flight list of top 10 most likely failure scenarios. Definitely a difficult problem.
> Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips.
This has been known in the high availability and safety systems industry for a while and a good book to learn these reliability engineering techniques is "Reliability Evaluation of Engineering Systems".
One downside of using non-rad hard parts is degradation from TID (gamma) and latch up effects. You can have chips monitoring other chips to reset whenever they latch up but TID is mostly permanent. The good thing is that TID in LEO, where SpaceX mostly operates, is relatively lower than GEO so they can get by with mostly commercial parts. It's not like the big defense contractors haven't figured out the same thing, they do fly stuff using commercial parts as well, they are just slower to adopt the same culture. SpaceX and the companies that built components using commercial parts are building the new-space industry.
But would using redundant systems separated in space connected with each other not offset the chance that they all would be affected at the same time? This is actually not rocket science .. just hard engineering and hardware/software design for redundant systems which is also usable on the ground.
High energy gammas have a relatively low cross section, most are going to pass right through the chip. If you add a too little shielding, or don’t layer shielding appropriately you are going to stop more gammas but produce lower energy x-rays from the shielding, which have a higher cross section, potentially increasing your chip dose.
Would it be possible to create a "skip" EM shield that does the opposite - increasing the energy of the gamma rays thereby reducing the likelihood of stopping them?
No idea how. Energies of most chemical bonds / electrons around atoms are not very high, not sufficient to emit proper gamma rays AFAICT. High-energy gamma rays are produced in nuclear reactions. While "clean" nuclear reactions that emit only gamma rays and not neutrons do exist, they are very high-energy and thus hard to initiate, and I don't think it would be easy to capture the energy of incoming gamma efficiently enough.
No, that's correct. Of course there's still some level of reduction beyond which the gamma rays don't matter, but where you want to place it is somewhat arbitrary.
A box with 1.3’ walls seems doable, actually, depending on how small the chips are. Might still be cheaper and more effective than specialized chips. But I know nothing, so am probably wrong.
Hah, beat me to the nerd snipe. Moreover, that sphere would cost $10k to make and, at a launch price of $1500/kg, cost $4.5 million to launch into orbit.
The aim for the launch price of the entire rocket is to be around 5 million (once it's fully re-usable and in production). Basically the price of fuel and maintenance.
So something might be off with your assumption of 1500 usd / kg.
Yes, it's based in the real world. This was the Falcon 9 launch price that I could come up with in the amount of time I was willing to spend on a shitpost. I agree that launch prices will continue to come down, but launchers will always be mass-constrained and launching lead spheres into orbit will never be a practical solution.
IIRC the CPUs are much less susceptible to damage when powered-off ? So have a bunch of them in cold standby or even as additional pluggable modules on missions with humans on board & swap to good ones when needed? :)
If the only thing that effectively shields these processors from radiation is lead, concrete etc (per earlier comments), what design changes / quality improvements can compensate?
You don't need to block gamma radiation completely to increase the electronics reliability :)
Maybe you could improve the system availability considerably by a bit of gamma radiation protection combined with some more parallelism of the components ..
The point is that shielding turns a single high energy particle that would otherwise strike and probably destroy a single transistor, into a veritable spray of lower energy particles causing bit flips or worse all over the circuit. This spray of particles can be stopped... with 1.3 feet of lead shielding.
"Hollman also found that creativity got him a long way. He discovered, for example, that changing the seals on some readily available car wash valves
made them good enough to be used with rocket fuel."
I have seen some people who decide to keep moving forward with whatever they have at the time. Sure what they produce is way less than perfect, but what they produce is way ahead of what everybody else is doing.
Perhaps the key is to be relentless, and resourceful.
Considering the political views of Elon Musk, it might be worth noting that his biographer is not the same Vance who is currently running for election as vice-president of the USA!
From an article for this I remember one more interesting side effects of this approach - the flight computer ends up as a generic x86/ARM board that the engineers can just have on their desk during development. Previously the dev boards would use the same rad hard chips and would be as expensive and scarse as flight hardware, resulting in much harder development & engineers having much less experience with the real hardware.
But it turns out, it doesn't matter how many redundant backup diesel generators you've got if a 45-foot wave comes along and they're all left underwater.
I always thought the liquid sloshing would be one of the hardest to simulate (considering how chaotic fluid mechanics is). Interestingly, I think this caused the 2nd Falcon launch to fail (the LOX sloshing).
It is difficult, but there are modeling approaches that work, such as VoF (https://en.wikipedia.org/wiki/Volume_of_fluid_method). Basically, in addition to velocity, pressure, temperature, etc., you store an additional scalar in each cell of your computational mesh representing the liquid's volume fraction. Then, you solve an additional equation to transport that scalar.
Solving the Navier-Stokes equations numerically in 3D is very time-consuming, even on HPC clusters, not to mention the additional modeling required for multiphase flows. Your answer implies that the solutions are obtained almost instantaneously, which is not the case.
I think the reason these kind of simulations are fast enough is because they are very coarse and approximate. Don't think of asking how exactly the foam swirls around the individual longerons, more like a very rough estimation of which side of the tank the liquid is slumped to. Remember it doesn't have to be "exact" just close enough to be useful.
By their very nature model predictive controllers operate in a world where not everything is perfectly modelled. Engineers do their best and whatever is left is the "error" the controller is trying to deal with.
Maybe they don't need to model the fluid dynamics, they just need to detect the mass movement / acceleration forces caused by it, and use those sensor inputs to inform a picture that's fed into their correction thursting.
Sort of like how you can balance a few pitchers of beer on a tray in your hand by remaining aware of the weight, even when people remove one! hahaha :)
Still if there indeed is "free" mass moving about, you need to make sure your control inputs don't make it slosh harder, so you compensate for that, so it sloshes even harder, etc - basically avoiding oscilation. :)
Oh no, apologies if that was the impression I gave!! I actually perform CFD simulations in HPC clusters, and in fact I'm an admin of the small cluster at my research institute =)
These are indeed heavy computations. What I meant is that VoF is one additional equation to be solved besides the N-S equations (either filtered as in LES or Reynolds-averaged as in RANS), the energy equation, your turbulence model equations, and so on. Certainly, not instantaneous at all, but simply an additional "simple" model that we can hook into our current way of doing CFD.
So, my point was, sloshing is a problem that we know how to simulate, although certainly you need HPC resources. Though, looking at those 100k NVIDIA H100 Elon has, I guess they have them! :P
It really depends on the problem to be solved (domain size, complexity of physical phenomena such as turbulence model, heat transfer, acoustics, multiphase flows, combustion, etc., number of time steps required...). In our case we perform for instance simulations of turbomachinery acoustics that can take 3-4 weeks running in a few hundreds of CPU cores, combustion acoustics simulations that can take a week or two running in 1k-2k cores...
They don't need to solve the Navier-Stokes equations, they don't care how the fluid is actually behaving, they just need to approximate how the mass is moving within a margin of error that the control system can handle.
Maybe the tank is just not a large hollow structure but contains fins/compartments/whatever to restrict the sloshing motion and it's not that big a contribution to the overall motion.
If it's no stronger than a sudden wind gust, it's just something the controller has to be able to take care of without a heads-up.
I remember a very similar anecdote about Von Braun & the early Juno/Jupiter rockets - with someone pointing out issues with sloshing on a press conference & Von Braun brushing it off as insignificant.
Then the next launch crashed due to slosh induced oscillation - and the one rocket after that had anti-slosh baffles. ;-)
At least for the retro-propulsive landing burn, I think the modeling problem is probably aided by the high G-forces that must keep the fuel very close to the bottom of the tank. Even before re-light the booster is falling near terminal velocity (I think?), so the fuel is likely sitting at the bottom.
I think it's a huge problem when re-lighting the engines in orbit, though.
Also IIRC the massive main tanks in Super Heavy should be basically empty at landing & the landings propelants come from a set of small header tabks that are near the central axis of the vehicle & arr completely full. This should reduce or even fully remove sloshing issues at landing time.
I think some Kerbel Space Program players have attempted to approximate the liquid sloshing as an inverted single or double pendulum problem inside the rocket that the control algorithm has to take into consideration in addition to the primary control of the rocket.
Has it been considered to spin the fuel via some centrifuge mechanism as a way to remove sloshing from the equation, or is that more complex/expensive/error-prone than just predicting it via simulation?
I'm thinking we will eventually end up with "active fuel management" techniques like this for in space vehicles.
Bug tanks make sense there & they might not be always full. So I can imagine all kind of interesting ways you can work with the fuel in zero go to avoid not only slosh but also the need for ullage thrusters. Eq. some programmable nozzles using in-tank gas to nudge liquid fuel blobs to move in the right direction. Or even some nets or bags that herd in the fuel in the middle of the tank + prevent it from directly touching the side, reducing boil-off or refrigeration requirements. :)
Interestingly cheap redundancy is also how life does things for the most part. Most biological organisms just replicate a lot to guarantee success, so it's clearly a good strategy and an efficient use of energy.
People don't realize how powerful applied math (especially in the areas you've mentioned) has become. Same tools can be applied to people in the ad tech/social media.
Just as a note, Space Engineers has a mod that accounts for fuel in the tanks and also various orbital mods. If one feels inclined to try it for themselves ;)
as someone who absolutely loves SE -- please don't.
the orbital and planetary mechanics kind of suck. They're meant to provide a decent 'arcade realism' for the sake of player/player interactions and pvp/pve.
if you want to experience fuel slosh/weight during a vertical ascent/descent go with kerbal. It models a lot of that stuff without mods -- and mods can make the model even more accurate.
I know MPC takes a LOT of compute power. It's not like a finely tuned PID loop or even a cascade of PID loops, computationally.
Does anyone know (or have educated speculation on) what kind of hardware is running these algorithms? Like, do they have a linux machine that's running the control loops? Are we talking megabytes, gigabytes of SRAM?
I would think no -- you would definitely need hard real time for something like this. But my only experience with real time systems is in tiny MCUs with kb of SRAM. That's definitely too small for a controller like this.
MPC doesn't need to take a ton of compute power. It all comes down to how sophisticated the underlying model is. You can have a MPC with 20 variables and run it at multiple kilohertz on a tiny microcontroller.
When you build something like this, you're torn between having a big model that represents everything and a smaller model that is easier to validate and reason about. Based on simulation, you might go for a smaller model that "knows" to stay away from operating areas where hidden variables (like really complicated tank slosh) invalidate the small model.
I doubt the actual control loop is too much processing, but it's certainly possible to build controllers with SDRAM, millions of variables of state, and hard realtime processing, though I wouldn't build it on top of preempt-rt. ;)
"instead of using $100,000+ radiation-hardened aerospace/defense grade chips"
Well, that makes perfect sense considering that both the spaceport infrastructure, and the booster need to do their calculation on the ground level instead of the highly radiated environment that is space. However, for the rockets themselves, which happen to reach that harsh environment, they may use more resilient and expensive hardware in the future, after passing over the current "let it splash in the Indian Ocean" development and testing phases.
Less than 5% of a full load. Any extra fuel you brought to the edge of space and back is lost performance, so substantial efforts are made to minimize this lost mass fraction.
>probably involve some kind of fast numerical fluid simulation
Sometimes even a simple approach can work. On Apollo they developed (at the time cutting-edge) passive RC filter networks, to avoid the control system "exciting" the rocket at frequencies of the slosh/bend/torsion modes.
I never thought of using fluid dynamics in the rocket stabilization algorithm—maybe it's something that could be useful to prevent many of the accidents involving liquid-transport trucks
I have been told by people who worked on them that you get radiation hardened aerospace/defence grade chips by backing off the clock speed about 20% to give signal stabilization slightly longer time. I can understand the population being confused about this but industry being confused seems to have more to do with regulatory capture and beaurocratic moats which SpaceX does seem to be bypassing.
You also have to add massive amounts (relatively) of static sink by approaches like ‘silicon on insulator’ to prevent energetic electrons from hopping into the transistor layer.
> SpaceX control system also has to compensate for the fuel moving inside their rockets so the control algorithms probably involve some kind of fast numerical fluid simulation.
Surely this isn't necessary with a small enough sensor granularity or whatever the terminology is. You can have very dumb software if it reacts quickly enough to changes in perception.
(Not an engineer either) My understanding is that it's been done before on smaller scales but having a giant piston in the tank requires a good seal, railings to keep it straight, and overall way too much mass and rigidity. Consider that the tank walls are only a few millimeters thick.
Instead, it's more common to use gasses injected at the top of the tank to push the liquid to the bottom. Falcon 9 uses helium. Starship uses https://en.wikipedia.org/wiki/Autogenous_pressurization as well as small header tanks for the landing propellants.
Another interesting thing SpaceX is doing is to use consumer-grade chips in triple redundancy configurations instead of using $100,000+ radiation-hardened aerospace/defense grade chips.