Zuul, a project gating system from OpenStack

DevOps landscape is growing thick and fast with new tools and methodologies. And, because almost all of them offer some kind of automation or infra benefits, it sometime becomes difficult to identify the tool best suited to your use case.

Recently, OpenStack switched from Jenkins to Zuul for all of their CI requirements. Zuul is a project gating system, developed in house by the OpenStack team and is available under Apache 2.0 license. Now, when a project of such stature makes a bet on a relatively new product (though they have been using this in parts for few years) it makes a compelling case for the product itself. So, I decided to take Zuul for a spin. Read on to know what I learnt!

What is project gating?

Software projects, especially those with several developers face the problem of broken trunk quite frequently. Developers commit code on a daily basis to the main repo and sometimes the new code (or the combination of two or more code changes by different people) renders the trunk unusable. There are CI tools like Jenkins that can quickly identify such situations, but the time lost in this process causes wasted productive time and resources.

Many projects practice an informal method where developers with mainline commit access ensure that a test suite runs before merging a change. With more developers, more changes, and more comprehensive test suites, that process does not scale very well, and is not the best use of a developer’s time.

What if you have a software gatekeeper who kept a check on code committed to the main repo, and merged the code only if the specified tests pass. Sounds good right? This is exactly what a project gating system does. There are a few other things that the gating system does, but let's focus on this use case for now.

What is Zuul?

As per James E. Blair, Principal Infrastructure Software Developer, OpenStack,

Zuul is a system to drive project automation. It directs our testing, running tens of thousands of jobs each day, responding to events from our code review system and stacking potential changes to be tested together.

Once you set up the relevant connections and triggers, Zuul automatically decides which jobs to run, and when and where to run them. It integrates with other tools like Gerrit to manage the intermediate code.

Gerrit and Gearman

Before we go any further, let me introduce you to Gerrit and Gearman as both of these are extensively integrated with Zuul to enable various features.

• Gerrit: Gerrit is a web-based code review tool built on top of the git version control system. Gerrit is deployed in place of the central repository and adds an additional concept, a store of pending changes. Everyone still fetches from the authoritative repository but instead of pushing back to it, they push to this pending changes location.

• Gearman: Gearman is a generic application framework to schedule tasks to other machines or processes that are better suited to do the work. A Gearman powered application consists of three parts: a client, a worker, and a job server. The client is responsible for creating a job to be run and sending it to a job server. The job server will find a suitable worker that can run the job and forwards the job on. The worker performs the work requested by the client and sends a response to the client through the job server.

Zuul components

There are several components that help Zuul do what it does. Triggers, Reporters, Cloners, Mergers, Launchers are all various components that integrate with Zuul. Let’ understand what each one of these does:

• Triggers: The process of merging a change starts with a proposal for the change to be merged. This is handled by a triggering system. Primarily, Zuul supports Gerrit as a triggering system. A simple timer trigger is available as well. It supports triggering jobs in a pipeline based on cron-style time instructions. With modular design, Zuul also allows alternate triggering and reporting systems to be integrated.
• Reporters: Zuul can communicate results and progress back to configurable protocols. Whether it is the start, success or failure, each stage can have multiple reports. Currently reports can be posted back to Gerrit via a connection or sent via mails based on SMTP.
• Cloners: Zuul includes a simple command line client that may be used to clone repositories with Zuul references applied.
• Mergers: Zuul Merger communicates with the main Zuul server via Gearman. Its purpose is to speculatively merge the changes for Zuul in preparation for testing.
• Launchers: Modular design to launch jobs allows any system to run jobs for Zuul simply by interfacing with a Gearman server.

Zuul connections and configurations

The main component of Zuul is a scheduler. This core system talks to multiple different systems via connections. It receives events related to proposed changes, triggers tests based on those events, and reports back — all via connections. For example, Zuul connects with Gerrit, to manage the intermediate code changes and review process. You can also connect Zuul with SMTP, to send reports via mails.

Zuul connections are configured via the zuul.conf file. You can find it in the /etc/zuul/zuul.conf or ~/zuul.conf. You can also specify path to the configuration file using the -c option in command line. Here is how the conf file looks:

You can see the Gerrit connection details under the [connection gerrit] section. Similarly SMTP connection details are below [connection smtp]. There are other two configuration files

• layout.yaml: Project and pipeline configuration — what Zuul does. This is the main configuration file for Zuul, where all of the pipelines and projects are defined, what tests should be run, and what actions Zuul should perform.
• logging.conf: This stores the Python logging config. This is an optional file. If provided, it should be a standard logging.config module configuration file. If not present, Zuul will output all log messages of DEBUG level or higher to the console.