Friday, August 24, 2012

Migrating linux32 and linux64 builds to the cloud

In order to free up hardware resources and reduce build wait times, Release Engineering are moving a chunk of the Firefox builds onto Amazon EC2 instances (see Bug 772446).  As you may know, B2G and Fennec builds are already on EC2 instances.

The actual builds will run under 'mock', allowing us to create custom build environments as well as build either 32 or 64 bit builds on the same build machine.  Under CentOS 6, libstdc++, glibc and gtk2 will get a version rev, too.  See Bug 772563 for details.

Note that this applies only to the linux32 and linux64 platforms of mozilla-central, try, and all of the 'project' branches (such as mozilla-inbound).
It does not yet apply to aurora, beta, release, or ESR builds (these will ride the trains), or to Thunderbird builds (though Thunderbird comm-central and try-comm-central will be moved in the near future).

Among other things, this will unblock the reimaging of linux32 and linux64 physical build machines as win64 builders (see Bug 780022), thus reducing wait times on our Windows platform.

The move is planned for next week after the migrations have completed and will not require any tree closure.

8 comments:

  1. Are the Linux builders still running an ancient Python 2.5 or were they upgraded to 2.7 per bug 602908?

    ReplyDelete
    Replies
    1. I'm happy to say that the new Linux builders will have Python 2.7.

      Delete
    2. You have no idea how happy this makes me.

      Delete
  2. How does this affect build times on these machines? Are they faster, slower, or about the same speed as they were before?

    ReplyDelete
    Replies
    1. We are using higher end AWS instances to do the builds so the build times should be around the same, if not faster, than what they are today. As an example, our mobile builds are faster on AWS than they were on the physical build hardware we were using.

      Delete
  3. Any reasons why Windows Build aren't in Cloud as well?

    And i hope Mozilla get some very special discount with AWS, because EC2 are one of the most expensive, slowest cloud out there.

    ReplyDelete
    Replies
    1. Linux was the easiest platform to move first but we are planning to look at virtualizing Windows in the future. As I understand it, the last time we tried a Windows instance we were limited to building on a single core. Since each core on an instance was slower than a hardware core, the build times were too slow to be acceptable. We need to try this again with Pymake and building on multiple cores.

      Delete