Except building the same crates multiple times across the depedency graph.
Additionally, from my point of view, not caring about initial build times is a crappy development experience, versus what is offered by apt/yum/NuGet/vcpkg/conan.
All of the packages I'm using for my most recent project (https://github.com/zig-for/snfm/blob/main/vcpkg.json) required building on machine. Takes about an hour and doesn't run in parallel (now there's a good PR for someone...). Vcpkg packages _can_ be distributed in binary form, it just doesn't happen.