New in 1.2: Build comparison & Gradle upgrade testing

Edit Subject

Hi there,

[Gradle 1.2-rc-1 was released earlier this week](forums.gradle.org/gradle/topic...). One of the new features added by this release is the initial support for *comparing builds*.

Changing a build by migrating from one build tool to another is a mission critical endeavor. To a lesser extent this is also true for upgrading to a new version of your build system. The build affects the daily work of each and every developer, tester, product manager, and operations staff that touches the software. Even more: your ability to release depends on a reliable build. A poorly executed build migration can easily turn into an expensive and frustrating experience for everyone involved by hobbling the productivity of your entire delivery pipeline and adding unnecessary delay to your next release. A key practice to minimise those risks is to be able to compare the old and new builds.

When we talk about comparing builds, we are talking about comparing the observable *outcomes* of two builds. An *outcome* is something as obvious as a generated `zip` or `jar` archive but can also be something a little less obvious such as which tests have been executed. That is, builds are treated like a black box that can be poked to perform some actions. This enables the comparison of builds by different build systems.

There are 3 primary use cases that we had in mind when designing and developing this feature:

1. Helping users upgrade their Gradle version with confidence (by comparing the current build with the new Gradle version)
2. Help users porting a build to Gradle (by comparing the current build to the Gradle build)
3. Help users test Gradle build refactorings (by comparing before and after the changes)

While all three use cases are important and will be eventually addressed, helping users upgrade is what we deemed to be top priority. As such, Gradle 1.2 supports comparing the zip binary archives (including `jars`, `wars` and `ears`), produced by two Gradle builds. Future versions of Gradle will offer more capabilities. At the end of this post we'll discuss such potential capabilities.

If you want to skip this introductory article and jump straight to the nitty gritty, see the [new User Guide chapter](gradle.org/docs/1.2-rc-1/userg...).

## What Gradle versions must I use?

You can use the build comparison support in the 1.2-rc-1 release to test an upgrade from a Gradle 1.0 or later build to 1.2-rc-1.

In any build comparison, there are actually 3 builds involved:

* The source build - the current state
* The target build - the proposed, or different, state
* The host build - the build that runs the comparison process

All 3 builds can be the same Gradle project, but they do not need to be. The *host* build must be Gradle 1.2-rc-1 or newer. The *source* and the *target* builds must be Gradle 1.0 or newer, and one of them must be Gradle 1.2-rc-1 or newer.

## Example upgrade test

Let's assume that we have a project that is currently building with Gradle 1.1. Let's execute it with Gradle 1.2-rc-1 and compare the results.

### Create the host build

Because the *host* build needs to be Gradle 1.2-rc-1, we are going to create a separate host build project. To do this, we create a new Gradle project and use the following for the `build.gradle`.

task wrapper(type: Wrapper) {
gradleVersion "1.2-rc-1"
}

Assuming that I have some version of Gradle installed on my system, I can now initialise the wrapper for this project via:

gradle wrapper

(If you don't know about the Gradle wrapper, checkout [this screencast](gradleware.com/registered/scre...)).

If we were upgrading from Gradle 1.2 to Gradle 1.3, we wouldn't need to create a separate host build. We could just use the actual build that is being compared as the host.

### Configure the host build

Now that we have a Gradle 1.2-rc-1 host build ready, let's configure it to do the comparison. Add the following to the `build.gradle`.

apply plugin: 'compare-gradle-builds'

compareGradleBuilds {
sourceBuild {
projectDir = "/path/to/my/gradle/project"
gradleVersion = "1.1"
}
targetBuild {
projectDir = sourceBuild.projectDir
gradleVersion = "1.2-rc-1"
}
}

We have added a task called `compareGradleBuilds` to the host build and configured it to compare the project @ `/path/to/my/gradle/project` with Gradle 1.1 and Gradle 1.2-rc-1.

### Run the comparison

Next, we run the comparison by running:

./gradlew compareGradleBuilds

Now the following will happen:

* The source and target builds will be executed (tasks: “`clean assemble`”, which can be configured) by the [Gradle Tooling API](gradle.org/docs/current/usergu...) (which will handle downloading the necessary Gradle versions if they aren't already on your system).
* The outcomes will be compared, and a HTML report produced @ `«host build»/build/reports/compareGradleBuilds/index.html`.
* If there are _any_ differences in what is compared, the build will fail with an error.

Keep in mind that there may be legitimate reasons for there to be differences in what is compared. Some outcomes are extremely volatile, such as files that contain timestamps.

### What is compared with Gradle 1.2?

Gradle 1.2 supports comparing zip artifacts. More specifically, it compares artifacts of type `zip`, `jar`, `war` and `ear` that have been added to the `archives` configuration, which is the way that an artifact is specified to be public in Gradle (for more on artifacts, configurations and publishing see [this User Guide](gradle.org/docs/current/usergu...) chapter). Most builds will not have to do any extra configuration to make its artifacts comparable, as all of the standard Gradle plugins automatically add their artifacts to the `archives` configuration (for example, the jar produced by the “`java`” plugin is configured this way).

## Future directions

This is just the start of this feature. Expect to see it evolve and grow with future versions of Gradle.

Here are some of the things we are planning to do:

* Support comparing more types of outcomes (e.g. test execution)
* Support finer grained comparison (e.g. to compensate for volatile data like timestamps)
* Support comparing against Apache Maven builds
* Support comparing against Apache Ant builds
* Increased convenience for comparing against new Gradle versions (i.e run a comparison with the new Gradle version without modifying any files)

The build comparison functionality will also evolve into a more general toolkit to facilitate more complex, thorough and custom comparisons. You’ll be able to plug in support for custom types of build outcomes and executing any type of build.

## Providing feedback

We absolutely need your feedback on this feature. Please try it out and let us know what your think. Your feedback is absolutely critical to helping the Gradle developers deliver top quality, value adding, features.

Feel free to post your feedback as a reply on this forum topic or to start a new one.