Workshops, Collaboration, Many Unknown Topics

There was a workshop at my organization this week. The focus was to eliminate/minimize the quality problems in one of the software component delivered by my team to the business lines. Between business line who consume the end software, and my team, there is a platform services who deliver the LFS(Linux From Scratch) This software component built by my team is having interfaces that are common between LFS, and us, which create problems at times, and the potential solution was to develop, build and test our component along with LFS. I think it is a right solution.

I’m a Build Automation engineer and I’m rewriting our software component’s build system using CMake. Rewriting is necessary as this component cannot be built by just running “make my_component”. Our build system is way too old, and inseparable from the other components that coexist. I like this task as it is challenging. Ensuring the accuracy of build, to match it to the legacy build system is reasonably challenging! I’m enjoying this work so far. There are some organizations where people are asked to do too many things at a time. That is suffocating. It happens with us some times.

As much as the workshop was inspiring, it was discouraging too, that I do not know most part of this software component, the development, and testing aspects in particular. That went in two entirely different directions. I sort of concluded it that I think I get discouraged rather quickly when I don’t know things. There perhaps was no need for discouragement. It just isn’t my area of work perhaps. But it is always good to know a little bit of everything.  I think I will go forward in that direction; to know what this piece of software does, and then, how, and then even try some hands on.

A Recognition

I just won myself a recognition at the office in this month’s “Star of the month” competition. It was for my one month effort to deploy, and stabilize ccache in our considerably large software project that I was awarded this. This was done keeping the build time, and the cost of ElectricAccelerator’s license in mind. While I partially won by decreasing the build time by nearly 40% on an average, and meeting the first expectation, I failed to save the cost, as the pressure on resource crunch was huge, and we had to purchase more licenses.

We are aiming at our next promising thing, Tup, a blazing fast, and incredibly accurate, OpenSource build tool. While its adoption on the OpenSource world looks very modest, it is by far, the best OpenSource build tool available as of now, from what I know. I’m hopeful that this should help us accomplish both the targets, of increasing the developer productivity by not wasting lot of time waiting during building stuff, and saving the organization some cost.

I’m not going to quit until this Tup build system is rolled out!

distcc for Distributed Compilation – gotchas!

After getting our builds run with gnu make in parallel, we spent some time evaluating “distcc” for distributed compilation. I had some hiccups and learnings. I’m hopeful that the following shall be useful and handy if you are just starting to use distcc.

  • In our build environment, the directory where compiler is stored keeps changing as we have very frequent SDK updates. The compiler is accessed via soft link, and only this soft link is moved to the new path when we have SDK updates. So, we do not have a fixed absolute location to compiler that can be used by the distcc wrapper, and I’d to make sure that the distcc wrappers are written on-fly with absolute path to compiler before the build starts. So, what I did was, while the CC/C++ Compiler path was initialized, I ensured that the wrappers are written into the user home directory/a directory(DISTCC_WRAP_STORE) that is available both on build machine, and the distcc servers.

CC := $(PUMP) $(DISTCC) $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc -m32

$(shell echo $$(readlink -f $(LINUX_I686_GCC_BIN_DIR)/x86_64-unknown-linux-gnu-gcc) \""\$$@"\" >> $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc )
$(shell chmod a+x $(DISTCC_WRAP_STORE)/x86_64-unknown-linux-gnu-gcc)

  • disctcc wrappers must be executable files, and they seem to work only when they have the absolute path to the compiler in them! Otherwise, the errors you see are quite weird, and they give no clue about the missing execute permissions or the relative path.
  • If you are using .d files generated by GCC compilers, then the .d files may end up created in current directory from where compilation starts, while you might be expecting them in the object directory This problem can only occur if you’re using gcc 3.0 or later, have the source and object files in different directories or under different names, and you are using -MD or -MMD but not -MF. Even with -MF in place, the first line in the .d file, which is the path to the object file, may not really contain the object.o, instead it will be source.o if source, and object files have different name! ie. you are compiling errorHandler.c to errorHandlerM.o and in this case, you gotta make sure you correct the .d file after it is generated.

I will add more findings, if I come across any. We have some troubles with PUMP mode right now, hopefully, we will get through them.