> It seems like it would be cool for an SBC, but the compilation (or setting up cross-compilation) was always too much;

Back in the day I had a few relatively slow machines and used to compile my kernels using distcc to offload the tasks to them. I never used cross compilation but I see it is supported so it may be a possibility for small SBCs.

https://www.distcc.org/