> When I used to build docker images they would sometimes take hours, for things that were less than a minute on x86.

That is almost certainly because it was pulling a majority of prebuilt binaries, IIUC.