1. I didn't see it stated explicitly, but I presume the neural net is on the far end of a radio link somewhere, not running on hardware physically mounted on the drone?
2. After viewing the FPV video on the linked page: how the hell do human pilots even come close to this pace? Insane (even assuming that the video they're seeing is higher quality than what's shown on YouTube – is it?)
3. The control software has access to an IMU. This seems to represent some degree of unfair advantage? I presume the human pilots don't have that – unless the IMU data is somehow overlaid onto their FPV view (but even then, I can't imagine how much practice would be needed to learn to make use of that in realtime).
1) No, this is interesting specifically because it was all onboard, the drone has Jetson Orin NX on it.
2) No, the video the pilot sees is usually quite bad. Racing pilots usually use either HDZero (mid resolution video with weird pixel artifacts sometimes) or analog video (looks like a broken 1980s VCR). It’s amazing what they can fly through. These DCP spec drones are also slow by racing standards. Look up MultiGP racing, it’s even faster.
3) It can be overlaid but it’s useless. The human pilot is using the control sticks as the input to an outer rate regulation loop which contains the gyro as input to an inner stabilization loop though, so the IMU is still in the mix for human control.
2. The video they're seeing is worse. Spectators typically see the frames saved directly from the camera, but the pilot will be seeing them compressed and beamed over the air to their headset. See vid.
3. The human pilots do actually have access to it. Not directly, but the flight controller translates their inputs and makes use of the IMU to do so.
I’m reminded of when the US military figured out it should just replace all its proprietary field drone controllers with Xbox controllers because every single grunt that enlisted already had 10,000 hours on the things. If the future of warfare is drones, Christ, that video is terrifying.
Funny you should say that. Gamepads are not quite what you want for drone piloting for three main reasons:
1. Less precise. Gimble size matters.
2. All inputs sprung. This is exactly what you want for your three rotational axis, but you absolutely do not want your throttle resetting to 50% when you lay off. You can fix this using 3D mode where the zero setting is in the middle, but then you lose even more precision.
3. Circular inputs. This means at low or high throttle you have less roll available.
The main reason you'd want a gamepad is the size and shape. They do make gamepad-style radios, like the Radiomaster Pocket, which combine the best of both worlds.
You can pick up a simulator for $10-20 if anyone wants to give it a whirl, and many are even on Steam, but the general recommendation is to pick up a dedicated radio as soon as possible.
Note that this mainly applies to FPV quadcopters, due to how sensitive and twitchy they can be. When it comes to controlling pretty much anything else (I'd argue even most planes) these advantages are no longer relevant.
The US military is not limited to using stock COTS hardware. They have imitated the form factor and general feel of those controls, but custom built and ruggedized.
1. I didn't see it stated explicitly, but I presume the neural net is on the far end of a radio link somewhere, not running on hardware physically mounted on the drone?
2. After viewing the FPV video on the linked page: how the hell do human pilots even come close to this pace? Insane (even assuming that the video they're seeing is higher quality than what's shown on YouTube – is it?)
3. The control software has access to an IMU. This seems to represent some degree of unfair advantage? I presume the human pilots don't have that – unless the IMU data is somehow overlaid onto their FPV view (but even then, I can't imagine how much practice would be needed to learn to make use of that in realtime).