Last week was very discouraging. On a bad Friday, the running builds queue was too long, and the license constraints kept a lot of builds waiting for resources, and it was chaos. In the midst of all this, we have under-performing Linux boxes, bad NFS performance, and a g++ compiler that has just started compiling code slower than earlier.
This delayed the CI builds. Our ccache setup to speed up builds wasn’t useful because of the various factors.
Now, everything is nearly normal as we dedicated a few Linux machines for CI, and CI builds run on these dedicated machines and builds are comparably faster with ccache.
And, more than what I learnt about debugging these nasty environment issues, I learnt about people and how much they react if something you own is not not doing well. It wasn’t pleasant as everybody was under pressure. Understood.
By the way, in the processing of debugging these problems, I made a very tiny contribution to OpenSource and I’m happy about it. :) I’m a big fan of OpenSource projects. They are free, and everything is “open”. There are no hidden intricacies, because, if you are capable, you can always go through the source code.
Anyways, I’m hoping that we solve the remaining problems, or get them solved. I’m glad I saved a few seconds of compilation time, and gave my cent to OpenSource.