Really excited to try this out. I have a fleet of containers on ubuntu + incus. Not only does this do ZFS optimization, I look forward having easy container optimized backup, live cluster migration (to a different machine without downtime) and so much more.
I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot
I was hoping for easy backup via zfs send as well, but turns out it’s not so easy atm.
IncusOS does not give you shell access, you have to figure out IncusOS ways to do things via their CLI/API. I haven’t found an easy way to do incremental backup of the whole system yet. You can backup individual instances/volumes via incus export (which seems to use zfs send under the hood), but not the whole thing.
I have mixed feelings about their decision not to give you shell access. Guess those who want flexibility can always just install Incus on top of any Linux they like, but it would be nice to have an escape hatch for when IncusOS gives you almost everything you want…
I occasionally contemplate that, if I were designing an OS meant to be sort-of-immutable (like Incus OS or Fedora Silverblue etc or MacOS), I would probably build it like this:
The main filesystem is verified and immutable. Everything that isn't configuration or the user-controlled payload is genuinely read-only, and the system will even cryptographically verify it on boot or first use. You cannot modify /bin/bash, etc.
If you want to test a modification, you can configure an overlay, and you can boot with that overlay live. You can configure the overlay to also be immutable or you can make the overlay mutable. But the choice of booting into the overlay is controlled by code that cannot by overlaid, so you can always turn the overlay off no matter how much you screw it up.
The user may get root access, but if your system is remotely attested or uses a TPM or such for security, then that policy will find out if you do so before you can do anything as root. So you can shell in and attach a debugger to a system service, but you cannot do that and also pretend to your orchestration tools that you have not done so.
The default configuration is mostly empty. When you change a default, you are not modifying the middle of a giant plist where no one will ever understand what happened. You only create new configuration, and deleting it is just fine.
The result would, I think, give system owners plenty of ability to hack on their own systems, but they could also unhack their systems easily. There are very few systems out there with both of these properties right now...
Check out SmartOS, it's illumos/solaris based but I think you'll find it is a nice middle ground. Not as abstracted, nice tooling that makes common tasks simple but not so opinionated you have to de-abstract things to get under the hood. Not painless but what is?
I use Proxmox on fat servers, but for homelab-like setup Incus OS seems more like a sweet spot