distcc for Distributed Compilation – gotchas!

After getting our builds run with gnu make in parallel, we spent some time evaluating “distcc” for distributed compilation. I had some hiccups and learnings. I’m hopeful that the following shall be useful and handy if you are just starting to use distcc.

  • In our build environment, the directory where compiler is stored keeps changing as we have very frequent SDK updates. The compiler is accessed via soft link, and only this soft link is moved to the new path when we have SDK updates. So, we do not have a fixed absolute location to compiler that can be used by the distcc wrapper, and I’d to make sure that the distcc wrappers are written on-fly with absolute path to compiler before the build starts. So, what I did was, while the CC/C++ Compiler path was initialized, I ensured that the wrappers are written into the user home directory/a directory(DISTCC_WRAP_STORE) that is available both on build machine, and the distcc servers.


CC := $(PUMP) $(DISTCC) $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc -m32

$(shell echo $$(readlink -f $(LINUX_I686_GCC_BIN_DIR)/x86_64-unknown-linux-gnu-gcc) \""\$$@"\" >> $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc )
$(shell chmod a+x $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc)

  • disctcc wrappers must be executable files, and they seem to work only when they have the absolute path to the compiler in them! Otherwise, the errors you see are quite weird, and they give no clue about the missing execute permissions or the relative path.
  • If you are using .d files generated by GCC compilers, then the .d files may end up created in current directory from where compilation starts, while you might be expecting them in the object directory This problem can only occur if you’re using gcc 3.0 or later, have the source and object files in different directories or under different names, and you are using -MD or -MMD but not -MF. Even with -MF in place, the first line in the .d file, which is the path to the object file, may not really contain the object.o, instead it will be source.o if source, and object files have different name! ie. you are compiling errorHandler.c to errorHandlerM.o and in this case, you gotta make sure you correct the .d file after it is generated.

I will add more findings, if I come across any. We have some troubles with PUMP mode right now, hopefully, we will get through them.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s