Gerrit: Google-style code review meets git [LWN.net]

October 30, 2009

This article was contributed by Don Marti

Gerrit, a Git-based system for managing code review, is helping to spread the popular distributed revision control system into Android-using companies, many of which have heavy quality assurance, management, and legal processes around software. HTC, Qualcomm, TI, Sony Ericsson, and Android originator Google are all running Gerrit, project leader Shawn Pearce said in a talk at the October 2009 GitTogether event, hosted at Google in Mountain View.

The Gerrit story starts with the progressive escape of an in-house Google process and tool. Google requires code review for any change to company code or configuration files; there are a few exceptions, but those are subject to review after deployment. The code review process started out using lots of email, but for the past several years it has been automated. When Guido van Rossum, creator of the Python language, began working at Google in 2005, he started developing a tool, in Python naturally, to coordinate code reviews. The result, called Mondrian, lets users view the proposed change as a side-by-side comparison, and participate in comment threads attached anywhere in the code under review. An overview page shows a to-do list of incoming changes to review and reviewers' comments. Van Rossum presented Mondrian at a public talk in 2006. (video).

Mondrian has been a huge success inside Google, Pearce said. "Almost every engineer uses this as their daily thing." But Mondrian is heavily dependent on Google's internal infrastructure, including the in-house Bigtable non-relational table store and the proprietary Perforce revision control system. Google is a huge Perforce shop, and has built its own highly-customized IT infrastructure, including Perforce-dependent tools.

The first step in making a Mondrian-style tool available to a wider audience was van Rossum's 2008 release of Rietveld, which uses Subversion instead of Perforce, and the public interfaces of Google App Engine instead of Google internals. It's named for modern architect Gerrit Rietveld. As Google began the Android project, though, developers demanded a Mondrian-like tool for their codebase, tracked with Git. Google App Engine was a deal-breaker, because mobile hardware vendors working on Android-based products maintain internal repositories, and won't rely on an outside service.

Shawn Pearce, who previously reimplemented git in Java as JGit, and is now at Google, took on the project; the result is Gerrit Code Review, now used to track public proposed changes to Android. Android's applications are written in Java, so writing the new tool in that language should make it more accessible to would-be contributors among Android developers.

Gerrit runs a copy of the Mina SSH daemon, along with JGit, which is now maintained as part of the Eclipse EGit project. Although the combination is slower than original git over OpenSSH, it's fast enough for the Android developers. "The entire Android team uses this as their interface to Git," Pearce said. The server-side dependencies are Tomcat and an SQL database, which so far can be either MySQL, PostgreSQL, or H2. Gerrit uses OpenID for authentication by default, but can be configured to use HTTP basic (or digest) authentication, or Siteminder, a single-sign-on system from Computer Associates.

On the UI side, Gerrit uses Google Web Toolkit, an Apache-licensed project that compiles Java to JavaScript with AJAX functionality. The UI has a few tiny Flash widgets for convenience, - to copy Git command lines to the clipboard, for example - but Flash is not required. A user who prefers not to use the web interface can also ssh to the Gerrit server to execute commands. Gerrit doesn't enforce any particular processes to make git look more like the centralized revision control systems that spawned Mondrian and Rietveld. A Gerrit-using developer has a full git install and can still do distributed revision control tricks, such as cherry-picking from a newer upstream release. Gerrit just guards access to its own repository. A developer can set up a git repository with "origin" pointing back to an ssh:// URL on the Gerrit server, and do something like centralized development, or do "drive-by" interactions with a Gerrit server like any other Git repository.

To propose a change for approval through Gerrit, a developer must start a branch in git for that change. Each change, and each iteration of a reworked change, becomes a new branch. In order to preserve information among successive versions of the same work, Gerrit includes a git hook to apply a "Change-Id" line to commit messages. After doing a git push to the Gerrit server, the developer can come back to the web dashboard and see the status of the pending change, then request a code review. Alternatively, a wrapper called Repo lets the developer specify a reviewer on the command line when doing the push.

Once a reviewer is lined up, Gerrit starts sending email, giving both the URL for the Gerrit page and a git command line for the reviewer to pull the change. On the change page, a reviewer can see the change side-by-side with the original or as a diff, and add review comments anywhere in the code along with a "cover sheet" message. Approval has multiple levels, with configurable access to the range that a reviewer can apply. Typically, an individual developer would be able to apply -1 or +1, which are "prefer you don't submit this" and "I like it," and some would have access to the -2 "do not submit" and +2 "Approved" levels. The web interface is not required--a reviewer can ssh to the Gerrit server to approve or reject a change.

A rejected and reworked change with a proper "Change-Id" line preserves Gerrit metadata, and the reviewer can see his or her original comments and the submitter's replies, join an existing comment thread on the previous, rejected version, or start new comment threads anywhere in the new version. If the change is not accepted, the new version has to be a new branch.

Kernel developer David Brown, at the Qualcomm Innovation Center, uses Git and Gerrit with his team. "The biggest complaint people have so far about Gerrit is people have to be constantly rebasing their changes," he said. However, the company has an extensive review process in order to make anything available under a free software license, and Gerrit streamlines the process of approving changes for the people who are authorized to check outgoing code. "The biggest thing that's changed since last year is Gerrit. The second biggest thing that's changed since last year is Gerrit," Brown said. But, he added, doing things the Gerrit way does work. "Most people learn a really small subset of git, I mean a really really small subset of git," he said.

Gerrit can be set up to automatically enforce some policies. "There's a lot of different work models people want," Pearce said. For example, Gerrit can be set up to enforce a check for a signed contributor agreement. The public Gerrit instance for Android enforces the contributor agreement requirement for all modules except the kernel, where only a "Signed-off-by" line is required. Gerrit can be integrated with a bug tracking system (BTS), but the integration is still based on site-specific tricks, since everyone is on a different bug tracker and nobody seems to like theirs very much. Besides better BTS integration, Pearce is looking at ways to store Gerrit metadata in git. "We'd like to do all the things that Gerrit does, offline," he said. "The fact that it doesn't work offline is a bug."

The Android developers are still figuring out how to connect with upstream. Staging maintainer Greg Kroah-Hartman plans to drop Android drivers from drivers/staging as of 2.6.33, as "no one wants to maintain them and help get them merged into the kernel," he said in email. Behind the apparent driver slowness are substantial corporate culture changes, though, with both Qualcomm and TI starting programs to manage outgoing code. Qualcomm is the lead sponsor of Code Aurora Forum, and TI is behind OmapZoom.org. In the potential minefield that is the mobile industry, with considerations such as not offending carrier partners, securely supporting third-party applications, deploying codecs and GUI code without patent troubles, and complying with radio regulations, Gerrit seems to be a needed focus for gatekeeping efforts.

(Log in to post comments)

Gerrit: Google-style code review meets git

Posted Oct 31, 2009 17:18 UTC (Sat) by bradfitz (subscriber, #4378) [Link]

It's good that you're asking the hard questions. I mean, it's important to
choose ones tools based on their implementation language, not what they do!
:-)

Gerrit: Google-style code review meets git

Posted Oct 31, 2009 17:21 UTC (Sat) by zwily (guest, #61712) [Link]

I'm guessing it's Java mostly because of the Shawn's (the author) preference. He's the JGit author
and has a stated preference for GWT.

Luckily my love of Gerrit overcomes by loathing of Java. Anyone who dismisses Gerrit because of its
Java underpinnings is doing themselves a disservice.

Want GWT, get Java

Posted Nov 1, 2009 15:38 UTC (Sun) by dmarti (subscriber, #11625) [Link]

AFAIK there's nothing like GWT for anything except Java -- if you want to develop a desktop-style UI, then deploy the same code unchanged to a browser, the only other alternative is Flash, which is one more thing to keep up to date on the client side.

Java also gets you the world's second-most-complete git implementation, so you don't need to fork and exec to run one of the git tools from your server software.

And, of course, anyone who wants to write Android apps has to learn Java anyway, so it's one less barrier to contribution from the original user base.

Want GWT, get Java

Posted Nov 1, 2009 15:46 UTC (Sun) by jake (editor, #205) [Link]

> AFAIK there's nothing like GWT for anything except Java

Pyjamas seems plausible for Python fans: lwn.net/Articles/348341/

jake

Why Java

Posted Nov 3, 2009 15:48 UTC (Tue) by spearce (guest, #61702) [Link]

> Java also gets you the world's second-most-complete git implementation,
> so you don't need to fork and exec to run one of the git tools from your
> server software.

This is the primary reason why Gerrit is written in Java. We didn't want
to have to fork+exec the C git implementation in response to web requests,
and there really isn't a suitable C library available for access to a Git
repository. I've chosen to invest in JGit, the folks at GitHub have been
investing in Grit (Ruby Git library) and Dulwich (Python Git library) for
the same reasons.

The second reason is, some companies prefer web software written in a
higher level than C, because there is less risk of a buffer overflow
ruining the day. There is still a chance, but the odds are much lower when
a virtual machine (e.g. JVM) is enforcing array bounds checks.

The third reason was mentioned in the article, much of Android's user space
code is developed in Java, so the original audience for Gerrit would be
more comfortable patching a Java tool than a C or Python tool. In practice
I'm not sure how much this has helped Gerrit. We've received quite a few
contributions from non-Google companies who use Gerrit, and I have no idea
whether or not the Java language has helped or hurt those contributions.

I'm really just tired of building websites. GWT makes it a bit easier to
build a web based UI. Pyjamas is an alternative if you are in Python, but
see my point above trying to gain contributors who are primarily Java
developers that are already using the tool. (There's nothing in open
source like using a frobinator that you can freely patch when its broken,
annoys you, or doesn't do what you want it to.)

Finally, Java is open source. There are openjdk builds now available in
Debian that are licensed under GPLv3. Java pretty much runs everywhere,
and so does Gerrit. I know people are running Gerrit Code Review on both
Linux and Windows servers. I've also heard of success running JGit on
OpenVMS, something no other Git implementation has done before.

Why Java

Posted Nov 3, 2009 22:00 UTC (Tue) by salimma (subscriber, #34460) [Link]

I thought it's GPLv2, not GPLv3, but otherwise, a good rationale.

Want GWT, get Java

Posted Nov 4, 2009 21:37 UTC (Wed) by roelofs (guest, #2599) [Link]

Sounds like you missed the SVLUG presentation on XML11 a couple years back. It's not entirely divorced from Java insofar as there's a Java server involved, but the client code is all JS with X11-like XML messaging connecting the two. It includes (among other things) an AWT implementation in JS.

I initially thought GWT might be a descendant of XML11, but I'd guess it's actually an independent implementation of some of the same ideas.

Greg

SVLUG

Posted Nov 8, 2009 22:21 UTC (Sun) by wt (guest, #11793) [Link]

I don't think that people who enjoy their sanity and well-being attend SVLUG anymore. It's core
members are a poisonous group that do nothing helpful.

I am a little biased as my wife ended up walking away from one of their meetings with bruises.

Gerrit: Google-style code review meets git

Posted Nov 9, 2009 15:10 UTC (Mon) by spearce (guest, #61702) [Link]

The centralized focus has occurred precisely because of the corporate
framework that exists around the original user group. Google built Gerrit
Code Review because we needed a way for engineers to get their code peer-
reviewed, and then submit it to a central repository that everyone else on
the team can go to obtain the reviewed trunk of the project from.

The biggest problem with any DVCS is "where is the trunk of this project".
When the project involves only 3 or 4 people, its easy for everyone to
declare that one user has the trunk on some machine, and that user just
pulls from the other contributors as necessary. When a project involves
30+ people working full-time for a company, often the company doesn't want
to pay a developer to sit there and pull from each person every day. Enter
the concept of a central repository that everyone gets read/write access
to. Its been that way since CVS/SVN/ClearCase, and will be that way far
into the future.

In my past I've also written an email-hook based review system around Git,
for a prior employer. It worked well-enough as you stated, but after using
something like Mondrian/Rietveld/Gerrit Code Review, I'm not sure I'd want
to go back. Editing comments and reading code in a syntax colored side-by-
side viewer is much easier than doing it in an email client against a
unified format patch. Especially when the change is large, such as 500+
lines and spanning 10 files.

Its quite unfortunate that Gerrit only works in the centralized approach
right now. I consider it to be a major design mistake. We cut some
corners early on and didn't make Gerrit properly support the DVCS model its
built on top of it. We are trying to fix those now, but its going to take
a while as its not our major focus. The plan is to:

- Make Gerrit easier to install.

If installation within your own user account can be reduced to
`sudo apt-get install openjdk-6-jre && wget gerrit.jar` then it
becomes more likely you might install it for your own use, even
if the upstream project doesn't use it. If the data store is a
local file in your home directory, with no database server needed,
its possible for the average user to run when they need to.

- Make Gerrit run on top of a distributed database.

My intern this past summer started a distributed database project on
top of git, called "gimd". In theory, if the comments are stored in
files in a git repository, you can share them through the same means
that you share the code itself. This should support a distributed
peer-review network.

- Add email interfaces to Gerrit.

Some people are just die-hard email fans. We should be able to
send them a unified patch by email from within Gerrit, but also
receive their reply and merge their patch comments back into the
side-by-side view of the Gerrit user. Just because you have to
work with a die-hard email user doesn't mean you should also have
to lose the features Gerrit offers.

Gerrit: Google-style code review meets git

Posted Nov 19, 2009 23:03 UTC (Thu) by khilman (subscriber, #37671) [Link]

Curious if there are any efforts underway for email interfaces?

I see this as the primary obstacle to switching to Gerrit for existing projects that are using email review. Migration of an existing project would require some transition, and after using Gerrit along side of an email interface for a little while might convince some to switch to Gerrit.

From a project maintainers point of view, I see an email interface as a pre-requisite as I want to be sure that all review comments are incorporated whether they come from the mailing list or from Gerrit.

Gerrit: Google-style code review meets git

Posted Dec 1, 2009 17:24 UTC (Tue) by spearce (guest, #61702) [Link]

An email interface is planned, but nobody is working on it. Issue 228 [1]
in our issue tracker at least documents the feature request. :-)

Issue 97 [2] might be related, as it talks about accepting a GNU diff
file via a form upload. That's probably most of the effort required
to also accept patches off an email list.

[1] code.google.com/p/gerrit/issues/detail?id=228
[2] code.google.com/p/gerrit/issues/detail?id=97