Wednesday, September 19, 2012

[updated] Migrating linux32 and linux64 builds to the cloud

In my last post, I explained how Release Engineering are moving the mozilla-central, try, and project branch Linux 32- and 64-bit builds to our new platform.  This change allows us to do builds in the cloud as required and in response to build demand.  Now that testing is complete, we are planning to deploy this change tomorrow (September 20).

Friday, August 24, 2012

Migrating linux32 and linux64 builds to the cloud

In order to free up hardware resources and reduce build wait times, Release Engineering are moving a chunk of the Firefox builds onto Amazon EC2 instances (see Bug 772446).  As you may know, B2G and Fennec builds are already on EC2 instances.

The actual builds will run under 'mock', allowing us to create custom build environments as well as build either 32 or 64 bit builds on the same build machine.  Under CentOS 6, libstdc++, glibc and gtk2 will get a version rev, too.  See Bug 772563 for details.

Note that this applies only to the linux32 and linux64 platforms of mozilla-central, try, and all of the 'project' branches (such as mozilla-inbound).
It does not yet apply to aurora, beta, release, or ESR builds (these will ride the trains), or to Thunderbird builds (though Thunderbird comm-central and try-comm-central will be moved in the near future).

Among other things, this will unblock the reimaging of linux32 and linux64 physical build machines as win64 builders (see Bug 780022), thus reducing wait times on our Windows platform.

The move is planned for next week after the migrations have completed and will not require any tree closure.

Thursday, March 22, 2012

Try repo head limits

Each time someone uploads code to our try repo (, another Mercurial ‘head’ is added.  Once the number of heads reaches around 2500, we start having trouble cloning the repo for builds.

In the past, when this happens we’ve had to reset the repo back to a fresh copy of mozilla-central.  This requires tree downtime and coordination with IT.

Another approach which I’ve run twice now is to periodically clone the try repo, *merge* all the older heads, and push those merges back in.  This requires no tree downtime or coordination and does not trigger any builds.  The only negative side effect I’m aware of is that displays those 1000-1500 merge changes until they scroll off-screen when developers check in new changes.

Bug 734225 proposes that we run this automated head merge at some interval (daily or weekly).  Are there any objections to this proposal?  Please comment here or in the bug.  Thanks.