Wow, it's already 20 years. Was setting up distcc in my first job, with ccache for armcc. Good times, great experience. This was a life-safer in terms of compilation speed (just needed wait 20 mins for consolidation and like the same or more for loading symbols to the debugger, which was bearable comparing to spending another 2 or 3 hours taken by compilation). Having the opportunity, would like to thank you!
https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.
https://github.com/mozilla/sccache is another option which addresses the use cases of both icecream and ccache (and also supports Rust, and cloud storage of artifacts, if those are useful for you)
ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that kind of work as useful so no point fighting those battles.
My codebase is significantly larger than yours (mine's a mix of mostly-C++ & some C) — perhaps 10–12 million lines. Clean builds are ~10m; clean-with-ccache are ~2m; incremental are millisecond.
I know this probably won't help with your current project, but you should think of your compiler as an exotic virtual machine: your code is the input program, and output executable is the output. Just like with a "real" CPU, there are ways to write a program that are fast, and ways to write a program that are slow.
To continue the analogy: if you have to sort a list, use `qsort()`, not `bubble sort()`.
So, for C/++ we can order the "cost" of various language features, from most-expensive-to-least-expensive:
1. Deeply nested header-only (templated/inline) "libraries";
2. Function overloading (especially with templates);
3. Classes;
4. Functions & type definitions; and,
5. Macros & data.
That means, if you were to look at my code-base, you'll see lots-and-lots of "table driven" code, where I've encoded huge swathes of business logic as structured arrays of integers, and even more as macros-that-make-such-tables. This code compiles at ~100kloc/s.
We don't use function-overloading: one place we removed this reduced compile times from 70 hours to 20 seconds. Function-overloading requires the compiler to walk a list of functions, perform ADL, and then decide which is best. Functions that are "just C like" require a hash-lookup. The difference is about a factor of 10000 in speed. You can do "pretend" function-overloading by using a template + a switch statement, and letting template instantiation sort things out for you.
The last thing is we pretty much never allow "project" header files to include each other. More importantly, templated types must be instantiated once in one C++, and then `extern`ed. This is all the benefit of a template (write-once, reuse), with none of the holy-crap-we're-parsing-this-again issues.
I love your comment and it is 100% spot-on. extern C is the magic sauce for making anything fast.
The only downside is that it adds a ton of boilerplate and a lot of maintenance overhead. You need separate compilation units for everything and then you need a sub-struct to use the pimpl approach. Fast pimpl (in-place new in reserved space in the parent struct itself) gets rid of the heap allocations but you still have a pointer indirection and prevent the compiler from properly stripping out unused code across translation units normally (that’s where LTO comes in these days).
Really, the problem is just that it’s a PITA to write compared to sticking everything in the header file.
(It’s ironic that rust meets the first two rules by design but is still much slower than C++ to compile, though it does imply what’s already known, specifically that there’s a lot of room for improvement.)
You are probably aware, but for. others with ccache this is called "cache sloppiness", which is my favourite term.
You can set this via config, as by default ccache is paranoid about being correct. But you can tweak it with things like setting a build directory home (this is great for me, as I'm the only user but compile things in say `/home/josh/dev/foo` and `/home/josh/dev/bar` and have my build directory as my dev directory and it's shared. (see https://ccache.dev/manual/latest.html for all the wonderful knobs you can turn and tweak).
Fantastic tool, the compression with zstd is fantastic as well.
I played with distcc (as I have a "homelab" with a couple of higher end consumer desktops), but found it not worth my time as compiling locally was faster. I'm sure with much bigger code bases (like yours) this would be great. Reason I used it it that archlinux makes it super easy to use with makepkg (their build tool script that helps to build packages).
The best use of distcc "at home" is when you have one or more "big iron" (desktop, server, whatever) and a few tiny machines that work just fine but don't have much processing power.
For example, with some work, you can setup distcc to cross-compile on your amd64 massive box for your raspberry pi.
For builds that large, I (personally) start evaluating Bazel. Bazel has distributed build + shared cache features built-in. But I’ve always just dug into reducing build times in any large C or C++ code base I’ve worked on—damn what management says is important. And the switch to Bazel can be costly (effort) and it may be difficult to get team buy-in.
The maintainers of icecream and distcc have an understanding that we want to bring the two back together. It will probably never happen as nobody wants to do the work (and there may be some differences in how they work that are not resolvable)
Back at Uni (two-and-a-half decades ago now), I setup a hacky version of this sort of thing to distribute compilations over several workstations:
* use make -j to run multiple task at once
* replace¹ gcc with a script that picked a host and ran the task on that²
This had many limitations that distcc lists it doesn't have (the machines had to have the same everything installed, the input and output locations for gcc had to be on a shared filesystem mounted in the same place on each host, it only parallelised the gcc part not linking or anything else, and it broke if any make steps had side-effects like making environment changes, etc.) and was a little fragile (it was just a quick hack…), but it worked surprisingly well overall. For small processes like our Uni work it didn't make much difference over make -j on its own³ but for building some larger projects like libraries we were calling that were not part of the department's standard Linux build, it could significantly reduce build times. On the machines we had back then a 25% improvement⁴ could mean quite a saving in wall-clock time.
One issue we ran into using it was that some makefiles were not well constructed in terms of dependency ordering, so would break⁵ with make -j even without my hack layered on top. They worked reliably for sequential processing which is presumably the only way the original author used them.
----
[1] well, not actually replace, but arrange for my script to be ahead of the real thing in the search path
[2] via rlogin IIRC, this predates SSH (or at least my knowledge of it)
[3] despite the single-core CPUs we had at the time, task concurrency still helped on a single host as IO bottlenecks meant that single core was less aggressively used without -j2 or -j3
[4] we sometimes measured noticeably more than that, but it depended on how patballisable the build actually was, and the shared IO resource was also shared by many other tasks over the network so that was a key bottleneck
[5] sometimes intermittently, so it could be difficult to diagnose
Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well!
But after a few weeks every Gentoo box in the house started crashing regularly. It took me a while to figure out what was going on: one of the slower machines had developed a single-bit memory error and was sharing corrupted .so files with all other machines.
Yeah. Everyone I’ve talked to who has run a build cluster has recommended ECC for the build cluster, even if they’ve decided not to use ECC for other systems. Some people would run Hackintosh-like setups for macOS build clusters, just for the ECC.
I had to break into a laptop for my dad last week. Last used 7 years ago. Running win2k on a pentium 3.
Had to jump through various hoops and find creative ways around problems (laptop too old to boot from USB, all my modern machines lack optical drives. And so on) but I did eventually break in.
I still need to get around to setting up distcc; I only have two Gentoo servers, but one is so much more powerful than the other, and their CPUs are close enough I might be able to use ccache, too ...
25+ years ago, our company used Clearcase for version control and it's clearmake had distributed build capability. Clearcase used a multi version file system (MVFS) and had build auditing so clearmake knew exactly what versions of source files were used in each build step. It could distribute build requests to any machine that could render the same "view" of the FS.
Even without distributed builds, clearmake could re-use .o files built by other people if the input dependencies were identical. On a large multi-person project this meant that you would typically only need to build a very small percentage of the code base and the rest would be "winked in".
If you wanted to force a full build, you could farm it out across a dozen machines and get circa 10x speedup.
Clearcase lost the edge with the arrival of cheaper disk and multi-core CPUs. I'd say set the gold standard for version control and dependency tracking and nothing today comes close to it.
Clearcase was utter crap. 6 hour code checkouts and 2 weeks to setup a new developer is a freaking joke. I literally did a conversion from Clearcase to git and reduced the setup time to 15 minutes and this is for a code base older than Clearcase is.
Not to mention the absolutely bad design for handling merge conflicts (punt to human if more than 1 person touched a file seriously???)
If you're talking about Clearcase snapshot views, I agree they were garbage. And IIRC merging in a Clearcase snapshot view was also a hot mess. Snapshot views was a bold-on that we were forced to use in later years. TBH the migration to other VCSs was already underway by then in our company but snapshot views was the last straw for us.
On the other hand Clearcase dynamic views were pretty awesome. You just needed to edit your view config spec and the view FS would render the right file versions immediately. No checkout required. There was even view extended naming to let you open multiple versions of the same file directly from any editor.
As for merging Clearcase set the gold standard for three-way automatic merges and conflict resolution that wasn't matched until git came along. It's still superior in one important way - Clearcase had versioned directories as well as files so you could commit a file move and someone else's changes to the same "element" in the original file location would be merged correctly after the move. No heuristics, just versioned elements with versioned directory tree entries. Backporting fixes was a breeze, even with a significant refactor in between.
Git more or less wins hands down today because it is fast, distributed and works on change sets. But something with git's storage model, a multi version file system and Clearcase's directory versioning would be an awesome VCS.
Clearcase's (clearmake's) cloud building capability sounds nice, but I have to pile on to this:
> I'd say set the gold standard for version control and dependency tracking and nothing today comes close to it.
In my 2005-2011 experience with Clearcase, it was slow and required dedicated developers just to manage the versions, and I'm so happy its version control model has been an evolutionary dead-end in the greater developer community. The MVFS is an attractive trap. Giving people the ability to trivially access older versions means you've just outsourced the job of keeping everything working together to some poor SOB. It was very much a "enough rope to hang yourself" kind of design.
As I said, it was slow, because MVFS. The recommended solution from Clearcase/IBM was to break up the source tree into different volumes (or whatever Clearcase's "repo"-analogue was named), which just increased the pain of keeping things in sync.
Additionally, it was an "ask-permission" design for modifying files, where you had to checkout a file before being able to modify it, and you couldn't checkout if someone else had, which added a ton of painful overhead.
I'll grant that my company/group didn't know what they were doing, but following IBM/Clearcase's guidance was not a good idea.
These days, I use Clearcase as a warning to the younger generation.
Around that time I went down the rabbit hole of "what if I wanted to write a filesystem for Windows NT?" It turns out that, aside from network redirectors, Microsoft didn't think this was something people ought to do very often. The people at Atria (which got bought by Rational) who were working on ClearCase were pioneers, and they bought a source license from Microsoft to be able to do it. The result was that there was a lot of information from them on a mailing list about how do implement a filesystem. However, it was mostly incomprehensible to me, because kernel programming on NT is quite different from what one would expect coming from other backgrounds, including Windows 95, OS/2, and Linux.
Brings back memories. Fresh out of college, I was given the additional job of being the Clearcase and Unix admin for my team. Not that I had any special skills but others didn't know a few Unix commands (System V) that I did. But Clearcase was such a good product and was used in the Telecom companies that I worked for (Motorola, Lucent etc.) It was owned by Rational at that time and if memory serves me right, were acquired by IBM.
To this day, I find Clearcase's way of doing things is the better way to do version control. Git, in comparison, kind of feels alien and I could never really get the same type of comfort on it.
That was also the standard development workflow at Nokia Networks, back when NetAct was being developed for HP-UX.
Nowadays most of it has been ported into Java, if the ongoing efforts when I left, finally managed to migrate everything away from C++, Perl and CORBA.
Fastbuild https://www.fastbuild.org/docs/home.html is the free distributed compilation system many game companies use. The combination of automatic unity builds (simply appending many .cpp source files together into very large combined files), caching and distributed compilation together gives you extremely fast C++ builds.
Also supports creating project files that are compatible with XCode and Visual Studio so you can just build from those IDE's and a pretty flexible dependency based build file format that can accomodate any kind of dependency.
Someone just linked Fastbuild few days ago in one community i follow and i did take a look as i havent heard about that project before. I didnt really like the idea that it would need to have its own build rules but its understandable - where as distcc would just integrate directly to what ever build tool you would be using. Also the syntax didnt look so "pleasing" but i guess the behefits can out weight the learning curve of yet an another build language (compared to meson/bazel/cmake et al) ..
Having distributed builds on msvc thought with free software sounded very promising thought.
Isn't it a misnomer to call it a "compiler"? Even it's github README says otherwise,
> distcc is not itself a compiler, but rather a front-end to the GNU C/C++ compiler (gcc), or another compiler of your choice. All the regular gcc options and features work as normal.
I guess it depends. You use this software to convert your source code into the binary blobs efficiently. From high level, it behaves like a compiler, but internals are just using other compilers.
A lot of other compilers are the same way. gcc and g++ are higher level wrappers around their low level compiling and linking executables. It depends how you define the terms, including what you mean by "program".
20+ years ago I used distcc + jpegtran to rotate a few thousand jpegs. From what I remember I just had to write the Makefile as you would normally.
I think the end result was that the transfer time at 10mbps speeds made it take longer than it would have just transforming locally, but it was neat to see it work!
Also had much success with it in the past. In my previous job we wanted to use it when building Xcode projects with a large C++ library. However, it's not compatible with Apple's modified Clang. I spent some time maintaining a fork where I hacked in support for passing along the Apple specific switches but eventually gave up. Seems like a cool feature would be the ability to define non-standard switches in a config rather than have them hard coded.
edit: just remembered I wrote up a blog post about it when I worked there
We used to use this in a previous company. It reduced the build time from ~20 minutes to ~2 minutes; until a bug in the ld linker at the time (linking is not distributed - only compilation is) pushed it back up to 20 minutes. We had a hell of a time finding the issue, but luckily there was a newer versions of binutils available where it was fixed; and upgrading to it got the build time back under 2 minutes.
Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines, with multiple gpu(s) and cores.
I'm curious about the security implications with using distcc. Doesn't this mean that if one computer gets compromised, the attacker can run code on all other computers using distcc, or secretly inject malicious code in the build result.
So using distcc means that all computers using it must be trusted. And that means that using it on "all developers computers to share the load" is good for performance but bad for security.
Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.
I'm not sure whether distcc affects reproducible builds?
You could, in any case, have tighter controls on the release builds, which would be done on a CI machine before signing.
(Back when I used distcc we didn't distribute across the dev machines, we had an entire build farm of two racks of 2U servers!)
> Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.
I would not generalise so quickly.
Is every computer compromised in the internet if one is compromised?
No. It highly depends on the trust between those machines and whether they share similar services with critical vulnerabilities. Only then, they might be compromised together.
But the world has evolved and not everyone anymore bases their total trust and security thinking for "no outside internet connection, we are fine".
> Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.
That might've been true 10 years ago and still is in some case but I wouldn't assume that now, far more things run over encryption for example, or have firewall
Yes, that's correct: reusing developer's computers means you're trusting all of them. At small scale, that may be acceptable because you probably trust them all a lot anyhow, and perhaps are unlikely to have a strictly hermetic and isolated build system.
When the team is bigger or when security is more important, it's important to have a build system where you're confident that no one can subvert the output, and that includes ensuring that very few people can control machines running distccd.
Another way in which it shows its age is that, by default, there are only netblock based restrictions on clients, and connections are over unencrypted TCP by default (last time I looked), although there is an option to use SSH (or I guess Tailscale or similar.)
I don't know about other platforms... but on Gentoo, distcc is restricted to only running a small list of programs (ie gcc). That means you can't just send random commands to the distcc host from your "slow machines".
Granted, if your distcc host gets compromised, the compiled output shouldn't be trusted until the server can be reprovisioned.
IIRC, when using distcc you must make sure that the Toolchain is ABI-perfect identified by its path.
So:
/opt/ourcompany/dev/bin/cc
absolutely *needs* to be the same on all machines involved or you risk very hard to spot issues.
For that reason, either use the absolute same distribution for everyone (and then run /use/bin/cc but watch out for alternatives!) or roll-out your own toolstack but make sure to put its version in the path.
Since I was just going down this rabbit hole recently, I kind of wonder if it's possible to set the filesystem on something more like the BitTorrent protocol so things like the libraries/compilers/headers that are used during compilation dont all need to come from the main pc. It probably wouldn't be useful until you reached a stupid number of computers and you started reaching the limits of the Ethernet wire, but for something stupid that can run on a pi cluster it would be a fun project.
How do these distributed build tools work across the Internet, i.e. outside of a local LAN office setting? Does the increased latency and slower bandwidth become a bottleneck?
They will work over the internet. You should use SSH or a VPN like Tailscale because the native protocol is not encrypted.
Bandwidth and latency effects will have some effect on how well it will scale, but on a reasonably high latency and low RTT connection it's probably still faster than purely local compilation. Sending work over the network takes time but not much CPU, so the local CPU is freed up to do other work.
If you distribute preprocessed code, you need LAN speeds/latency for it to be faster than a local compilation. Otherwise, you need to have the exact same headers in addition to the exact same compiler installed everywhere. It MAY work otherwise, until it breaks in a subtle / obscure way one day and after tracking it down you will think maybe compilation doesn't take THAT long on a single machine :)
So all in all, I wouldn't say works outside of a well-controlled cluster on a LAN.
Assuming you don't use NFS to continuously read and write, I think it would be fine. Compiling is generally speaking CPU bound.
Even the website mentions that it just copies all preprocessed code to the workers, so I expect this to be a one-time overhead per job.
Good memories! I had fun with distcc by compiling kernels across a few local machines back when desktops for mere mortals were dog slow, and it helped a lot. I never used it for cross compiling though, which is something that could help today when starting compilations from small embedded boards. Did anyone have success in a mixed environment, such as a small ARM board with native GCC plus one or more faster x86 machines with cross compiling tools installed?
Just tried this. I have a slow ARM laptop and a fast x86 desktop. The ARM laptop seems to detect the desktop as specified in the environment variable, but it doesn't accelerate compiling at all. The x86 box sits idle while the ARM laptop compiles. I have the aarch64 cross compiler installed on the desktop as well.
Maybe I'm just bad at RTFM, but I can't seem to get it to work.
I don't have enough details to debug, but something I discovered just the other day might be helpful if you're building the kernel. For kernel builds, you have to specify CC= after `make`, not before. E.g. `make -jX CC="distcc alpha-unknown-linux-gnu-gcc"`.
It works transparently with any project using autotools/cmake/meson. (which is 95% of open source projects, and most likely most enterprise software) No need to convert your project to starlark BUILD files.
Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your build result accurate so that it could be cached remotely (like using ccache).
On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.
Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.
Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.
But it has protobufs interfaces (IIRC), so a distributed build farm would generate the grpc endpoints for their implementation and then you tell bazel on the command line (or via .bazelrc) the address of the build farm it can use.
I doubt that it is the main feature. RBE is difficult to deploy and consequently rarely used. On the other hand of you assume that 99% of bazel invocations are done by google, then weighted that way RBE is nearly universal.
Ooh this brings back memories. This project made my embedded cross-compilation iteration speed go from like 2-30 minutes to 5. This was 20 years ago. I don't understand why this isn't more of a thing these days. I think Bazel can do this, but I've felt nothing but pain from using Bazel.
Can anyone explain the architecture/how it works at a high level? I get that it is distributed. Does it basically copy the complete source tree to every worker and have them compile some independent subset of the object files? Does performance scale linearly with the number of worker nodes?
“The client is invoked as a wrapper around the compiler by Make. Because distcc is invoked in place of gcc, it needs to understand every pattern of command line invocations. If the arguments are such that the compilation can be run remotely, distcc forms two new sets of arguments, one to run the preprocessor locally, and one to run the. compiler remotely. If the arguments are not understood by distcc, it takes the safe default of running the command locally. Options that read or write additional local files such assembly listings or profiler tables are run locally”
Scalability:
“Reports from users indicate, distcc is nearly linearly scalable for small numbers of CPUs. Compiling across three identical machines is typically 2.5 to 2.8 times faster than local compilation. Builds across sixteen machines have been reported at over ten times faster than a local builds. These numbers include the overhead of distcc and Make, and the time for non-parallel or non-distributed tasks”
Last time I looked, it basically ran per-file "cc -E" on the source machine to get a compilation unit (optionally checking for a ccache cache hit at this point), then piped the result to "cc" running on the target machine, and copied the resulting object file back.
> Does performance scale linearly with the number of worker nodes
Yes, for small N.
Overall scaling was limited by how much "make -j" the source machine could cope with.
> distcc sends the complete preprocessed source code across the network for each job, so all it requires of the volunteer machines is that they be running the distccd daemon, and that they have an appropriate compiler installed.
So all the "environment" is on the source machine and just a bare compiler is required on the remote machines for compilation.
distcc distributes work from a client machine to any number of volunteer machines. (Since this is free software, we prefer the term volunteer to the slaves that are used by other systems.)
distcc consists of a client program and a server. The client analyzes the command run, and for jobs that can be distributed it chooses a host, runs the preprocessor, sends the request across the network and reports the results. The server accepts and handles requests containing command lines and source code and responds with object code or error messages.
I used this in a uni lab (good lord 15 years ago) when we needed to compile for a robot. It took maybe an hour or two to compile run directly on the robot but I was able to setup distcc to run on all the lab machines in that room and get things done fast.
Used distcc back in mid-2000s to crosscompile NetBSD kernel/userland for a TS-7200 SBC ARM device. Back then those devices didn't have much processing power, would have taken days otherwise.
storage becomes much faster than network today, so is it still worth it ? (naive question sorry) or maybe the cpu demand is much bigger than IO demand ?
I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants, a mutation testing tool for Rust.