I left Nokia for Cisco last week. It was 4yrs 6months of tenure. I’m a lot confident than I was when I joined, I made some good friends, learnt new things, and above all, I got to know myself more.
I’m missing my team, and new office is a little boring now. I’m still juvenile. I wish to grow up and growing up sucks. I think I will get busy with work next week, and I’m sure it is all going to be fine. I don’t recall any such feeling of going away from people from the last 2 companies I left. I felt nothing. I was happy to leave those places. This one was tough probably because I stayed longer, and I had many people. May also be because I’m probably going through the terrible phase of the quarter life crisis! I cried during the farewell and I can’t explain what I was going through. The farewell was such overwhelming. For some time I feared if I deserved all that good treatment. My team was so nice.
Gone are the days we miss. A lot of people walk in, and out of our lives. It is matter of staying happy with the moment that should be practiced. Tough at times though..
Anyways, I’m waiting eagerly to get my access rights set so that I can start working on things. I took the new job hoping to experiment a new tool in real time as I think it solves the problems here at the new assignment. I should wait and see how it works for me. I’m excited. Cisco is a big organization, and the environment looks very encouraging. Hopefully I will be able to bring something valuable to the table.
We are again stuck with hellish build time problems. Our ccache solution is suffering and I think it is because of the underlying infrastructure limitations. I hate to say “I think” because it kills me not to know what exactly is wrong with the infrastructure. I’m not an expert in network, storage, linux administration, and it made me handicapped. We have to resort the IT support for help all the time, and they are at my organization not up to our expectations.
We thought we are in trouble. Luckily, and strangely, the new cloud environment that we recently got, where we are run our build farm and evaluating our builds on RedHat 6.5 is performing very well. The first impression is that this new environment is 6 times faster! Very surprising results.
I verified a few basic differences between the old and new build farm, and the differences/improvements are:
- RedHat 5.5 vs RedHat6.5
- 16 cores hyperthreaded vs 16 cores hyperthreaded
- 16GB RAM vs 80GB RAM
- This seem to have improved the compile-times. I’m not sure to what extent.
- Disk storage type:
- The storage used by the build comes from epeheral. I don’t know what benefits this gave.
- Network Connectivity between machines in the build farm.
- Our builds run using a comerical make flavor called “ElectricAccelerator” Builds are distributed, and they share the intermediate build results across the build farm.
- That obviously needs connectivity across the build farm to be fast.
- My observation is, the old and new infrastructure differ a lot with the way they are connected to each other.
- ping showed some visible difference.
I’m happy with the preliminary results with the build times, and hopefully when we are in production with this setup, our developers will hopefully spend less time waiting, and we can spend more time on some quality topics than monitoring the environment all the time. Comments are welcome.
I’m writing this down to catch the attention of some folks who are into this warp/warpdrive or those who put this thing on github.
I’ve been trying to use compile and use it. With some tweaks, I could compile it. But, the source tree and files uploaded at git master branch at https://github.com/facebook/warp is somewhat outdated, and it is missing a whole lot of information.
Although I appreciate the contribution to open source from facebook folks, it seems to me that it is not done in proper way.
- There are plenty of hard-coded paths in Makefiles(ok, I can fix them),
- Missing scripts – builtin_defines.sh is missing(its tricky to take a guess, and even if I could arrive at something, I’m not sure if it is optimum)
- Basic documentation going missing, example, the documentation says,
This will produce
warp (the core program) and also the drivers
warpdrive_clangdev, each packaged for the respective compiler and version.
- but, none of the Makefiles build these drivers, which makes the whole understanding unclear.
- It is a pretty incomplete project uploaded at github and it asks for a lot of time being invested to get this tool working, which perhaps could have been easily avoided, making the contribution much more worthwhile at the open source standards, if someone from FB checked it more carefully and uploaded it to github.
I tried contacting nearly all of those who are contributors on github to this project to kind of help me by uploading that missing piece of information, and I haven’t got any reply from anyone yet. So, I’m writing this blog post hoping to grab their attention or someone who knows warp/warpdrive better already so that the missing information could be corrected and the project is more useful.
This weekend has almost been like a weekend without internet. I mostly spend my time on my laptop during weekends. I had plans to read some stuff on and off but without internet this plan didn’t fall in place. It seems to me that I’m addicted to my laptop. While I don’t see nothing wrong with it, I’m trying to figure something that keeps me very busy. Like handful of interesting work. I don’t know when would I find that.
Last week was very discouraging. On a bad Friday, the running builds queue was too long, and the license constraints kept a lot of builds waiting for resources, and it was chaos. In the midst of all this, we have under-performing Linux boxes, bad NFS performance, and a g++ compiler that has just started compiling code slower than earlier.
This delayed the CI builds. Our ccache setup to speed up builds wasn’t useful because of the various factors.
Now, everything is nearly normal as we dedicated a few Linux machines for CI, and CI builds run on these dedicated machines and builds are comparably faster with ccache.
And, more than what I learnt about debugging these nasty environment issues, I learnt about people and how much they react if something you own is not not doing well. It wasn’t pleasant as everybody was under pressure. Understood.
By the way, in the processing of debugging these problems, I made a very tiny contribution to OpenSource and I’m happy about it. 🙂 I’m a big fan of OpenSource projects. They are free, and everything is “open”. There are no hidden intricacies, because, if you are capable, you can always go through the source code.
Anyways, I’m hoping that we solve the remaining problems, or get them solved. I’m glad I saved a few seconds of compilation time, and gave my cent to OpenSource.