The New Environment That Performs Like “Wow!”

We are again stuck with hellish build time problems. Our ccache solution is suffering and I think it is because of the underlying infrastructure limitations. I hate to say “I think” because it kills me not to know what exactly is wrong with the infrastructure. I’m not an expert in network, storage, linux administration, and it made me handicapped. We have to resort the IT support for help all the time, and they are at my organization not up to our expectations.

We thought we are in trouble. Luckily, and strangely, the new cloud environment that we recently got, where we are run our build farm and evaluating our builds on RedHat 6.5 is performing very well. The first impression is that this new environment is 6 times faster! Very surprising results.

I verified a few basic differences between the old and new build farm, and the differences/improvements are:

  1. RedHat 5.5 vs RedHat6.5
  2. 16 cores hyperthreaded vs 16 cores hyperthreaded
  3. 16GB RAM vs 80GB RAM
    • This seem to have improved the compile-times. I’m not sure to what extent.
  4. Disk storage type:
    • The storage used by the build comes from epeheral. I don’t know what benefits this gave.
  5. Network Connectivity between machines in the build farm.
    • Our builds run using a comerical make flavor called “ElectricAccelerator” Builds are distributed, and they share the intermediate build results across the build farm.
    • That obviously needs connectivity across the build farm to be fast.
    • My observation is, the old and new infrastructure differ a lot with the way they are connected to each other.
    • ping showed some visible difference.

I’m happy with the preliminary results with the build times, and hopefully when we are in production with this setup, our developers will hopefully spend less time waiting, and we can spend more time on some quality topics than monitoring the environment all the time. Comments are welcome.

distcc for Distributed Compilation – gotchas!

After getting our builds run with gnu make in parallel, we spent some time evaluating “distcc” for distributed compilation. I had some hiccups and learnings. I’m hopeful that the following shall be useful and handy if you are just starting to use distcc.

  • In our build environment, the directory where compiler is stored keeps changing as we have very frequent SDK updates. The compiler is accessed via soft link, and only this soft link is moved to the new path when we have SDK updates. So, we do not have a fixed absolute location to compiler that can be used by the distcc wrapper, and I’d to make sure that the distcc wrappers are written on-fly with absolute path to compiler before the build starts. So, what I did was, while the CC/C++ Compiler path was initialized, I ensured that the wrappers are written into the user home directory/a directory(DISTCC_WRAP_STORE) that is available both on build machine, and the distcc servers.


CC := $(PUMP) $(DISTCC) $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc -m32

$(shell echo $$(readlink -f $(LINUX_I686_GCC_BIN_DIR)/x86_64-unknown-linux-gnu-gcc) \""\$$@"\" >> $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc )
$(shell chmod a+x $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc)

  • disctcc wrappers must be executable files, and they seem to work only when they have the absolute path to the compiler in them! Otherwise, the errors you see are quite weird, and they give no clue about the missing execute permissions or the relative path.
  • If you are using .d files generated by GCC compilers, then the .d files may end up created in current directory from where compilation starts, while you might be expecting them in the object directory This problem can only occur if you’re using gcc 3.0 or later, have the source and object files in different directories or under different names, and you are using -MD or -MMD but not -MF. Even with -MF in place, the first line in the .d file, which is the path to the object file, may not really contain the object.o, instead it will be source.o if source, and object files have different name! ie. you are compiling errorHandler.c to errorHandlerM.o and in this case, you gotta make sure you correct the .d file after it is generated.

I will add more findings, if I come across any. We have some troubles with PUMP mode right now, hopefully, we will get through them.