If you want an example of where these things go badly:
the standard compression level for rpms on redhat distros is zstd level 19.
This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower.
This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller.
The only reason to use zstd is because you want fairly good compression, but fast.
So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly.
Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice).
I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode)
Using zopfli would be like choosing level 19 zstd for npm.
While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing.
You will make certain cases just insanely slow.
You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU.
zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9.
Doesn't npm support insanely relaxed/etc scripting hooks anyway?
If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm.
Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more.
[1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.
> the standard compression level for rpms on redhat distros is zstd level 19
> The only reason to use zstd is because you want fairly good compression, but fast
I would think having fast decompression is desirable, too, especially for rpms on redhat distros, which get decompressed a lot more often than they get compressed, and where the CPUs doing decompression may be a lot slower than the CPUs doing the compression.
Just two tables for comparison, first one shows only the decompression for Firefox RPM, second shows compression time, compressed size, and decompression for a large RPM.
Let me start by reiterating - zstd is a great option. I think zstd level 5-10 would have been an awesome choice. I love zstd - it is a great algorithm that really hits the sweet spot for most users between really fast and good compression, and very very fast decompression. I use it all the time.
In this case, yes, zstd has faster decompression , but xz decompression speed is quite fast, even before you start using threads. I have no idea why their data found it so slow.
Even on large compressed archives, xz decompression times does not go into minutes - even on a mid range 16 year old intel CPU like being used in this test. Assuming the redhat data on this single rpm is correct, i would bet it's more related to some weird buffering or chunking issue in the rpm's compressor library usage than actual xz decompression times. But nobody seems to have bothered look at why the times seemed ridiculous, at all, they just sort of accepted them as is.
They also based this particular change on the idea that they would get similar compression ratio to xz - they don't, as i showed.
Anyway, my point really wasn't "use xz", but that choosing zstd level 19 is probably the wrong choice no matter what.
There own table, which gives you data on 1 whole rpm, shows that zstd level 15 gave them compression comparable to xz (on that RPM, it's wrong in general), at compression speed similar to xz (also wrong in general, it's much slower than that).
It also showed that level 19 was 3x slower than that for no benefit.
Result: Let's use level 19.
Further the claim "Users that build their packages will experience slightly longer build times." is total nonsense - their own table shows this. If you had an RPM that was 1.6gb, but took 5 minutes to build (not uncommon, even for that size, since it's usually assets of some sort), you are now taking 30 minutes, and spending 24 of it on compression.
Before it took ... 3 minutes to do compression.
Calling this "slightly longer build times" is hilarious at best.
I'll make it concrete: their claim is based on building Firefox and compressing the result, and amusingly, even there it's still wrong.
Firefox RPM build times on my machine are about 10-15 minutes. Before it took 3 minutes to compress the RPM. Now it takes 24.
This is not "slightly longer build times". Before it took 30% of the build time to compress the RPM.
Now it takes 24 minutes, or 2.5x the entire build time.
That is many things, but it is not a "slightly longer build time".
I'll just twist the knife a little more:
RPM supports using threading for the compressors, which is quite nice. It even supports basing it on the number of cpus you have set to use for builds. They give examples of how to do it, including for level 19:
/usr/lib/rpm/macros:
# "w19T8.zstdio" zstd level 19 using 8 threads
# "w7T0.zstdio" zstd level 7 using %{getncpus} threads
The table with this single rpm even tested it with threads!
Despite this - they did not turn on threads in the result...
First, here is no data or evidence to suggest this is the case, so not sure why you are trying to make up excuses for them?
Second, zstd is fully deterministic in multithreaded cases.
It does not matter what threading you select, it will output byte for byte identical results.
I believe all of their compressors are similarly deterministic regardless of number of threads, but i admit i have not checked every one of them under all conditions.
If they had questions, they could have, you know, asked, and would have gotten the same answer.
But that just goes back to what i said - it does not appear this change was particularly well thought out.
If you want an example of where these things go badly:
the standard compression level for rpms on redhat distros is zstd level 19.
This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower.
This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast.
So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly.
Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice).
I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode)
Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU.
zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9.
Doesn't npm support insanely relaxed/etc scripting hooks anyway?
If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm.
Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more.
[1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.