Optimise Your Acceptance Tests

by Jason Sankey on Tuesday, 09 March 2010

Introduction

Many developers have a love-hate relationship with automated acceptance tests. Although they are important for end-to-end testing, creating and maintaining an acceptance test suite tends to be difficult. One major sticking point is the time acceptance tests take to execute, which can easily cause a blow out of project build times. Since we believe in the value of both automated testing and fast feedback, we're always looking for ways to maintain a strong yet fast acceptance test suite. In this article I'll review 7 successful test optimisation techniques we've put to work in our own projects. This is not an exhaustive list by any stretch: but we can vouch for each of these techniques from personal experience.

Technique 1: Write More Unit Tests

Speed up your test suite by writing more tests? No, I haven’t gone insane, there is a clear distinction between the two types of tests:

  • Acceptance tests verify the functionality of your software at the users level. In our case these tests run against a real Pulse package which we automatically unpack, setup and exercise via the web and XML-RPC interfaces.
  • Unit tests verify functionality of smaller 'units' of code. The definition of a unit varies, but for our purposes a single unit test exercises a small handful of methods (at most).
From a practical standpoint, acceptance tests are much slower than unit tests. This is due to their high-level nature: each test will typically exercise a lot more code. Add to this the considerable overhead of driving the application via a real interface (e.g. using Selenium), and each acceptance test can take several seconds to execute. All this time stacks up and can considerably slow a full build and test.

The huge difference between the time taken to execute a typical unit and acceptance test leads to a simple goal for speeding up your build as a whole: prefer unit tests where possible. The important thing is to do so without compromising the quality of your test suite. Remember: at the end of the day what happens at the users level is all that matters, so you can’t compromise your acceptance testing. However, there are many examples where you can push tests down from the acceptance to the unit level.

So, next time you’re creating some acceptance tests for your software, remember to ask yourself if you can push some tests down to a lower level. Do all the combinations need to be tested at a high level, or are they all equivalent from the UI perspective? A little forethought can shave considerable time off your build.

Technique 2: Use Fast Dependencies

Acceptance tests usually need to interact with external dependencies. These may include the file system, a database, an external server, and so on. Knowing that not all such dependencies are created equal, it is worthwhile identifying and using the fastest set of dependencies when running your acceptance tests.

For example, Pulse makes heavy use of a database. We support HSQLDB, MySQL and PostgreSQL. Of these, HSQLDB has the least overhead, as we can embed it directly in Pulse. Further, HSQLDB supports a completely in-memory option, which is very fast for tests that do not require data durability. Hence we run our test suite against HSQLDB by default.

A pitfall with this approach is moving your acceptance test suite too far from reality. To avoid this, it's best to regularly run your test suite against each of the dependencies that you intend to support. Just keep in mind that not do this in your most time-critical builds (e.g. your continuous integration builds). Use the fastest dependency combination for most test runs, and test other combinations as you can.

Technique 3: Trade Off Granularity

In general we prefer each of our test cases to test one thing only. However, in the case of acceptance tests, sometimes a lot of setup is required just to get to that one thing we need to test. If this setup takes a long time, and needs to be done before testing several things, that time can add up.

Our first preference in these cases is to apply Technique 1; i.e. push through the setup for one end-to-end acceptance test, and explore the full set of combinations at the unit level. Where this is not possible, though, sometimes we sacrifice the goal of granular tests. This is a classic trade-off: although we prefer smaller test cases, sometimes it just isn't worth the cost of slowing the build down.

Note that in this situation it may be tempting to make your tests dependant on one another: the first case does the setup, and remaining cases reuse it. In our experience the cost of maintaining a suite with interdependent cases is too great to make this worthwhile. Further, these dependencies make it tricky to run subsets of your suite during development. Large single cases may be ugly, but they have the advantage of keeping all the related code in one spot.

Technique 4: Use a Remote API

Server applications can often benefit from exposing a remote API. Such an API enables users to both automate tasks and integrate with other systems. In the case of Pulse, we expose an XML-RPC remote API that allows remote control and monitoring of the Pulse server.

Apart from being a great feature, a remote API has another significant benefit: it enables us to control Pulse through a (relatively) fast interface during acceptance testing. The primary Pulse UI is accessed via a web browser. Although it is possible to automate web UI testing using tools such as JWebUnit and Selenium RC, the resulting tests are slow. Where we are not testing the web UI, or are testing only an isolated part of it, the overhead of driving the UI causes a huge blowout in testing time.

Thus, in many of our acceptance tests, a lot of the peripheral work such as setting up suitable data is done using the remote API. We also use the reporting functionality of the remote API to assert the current state of Pulse where possible. Tests only drive the web UI when they are testing the operation of the UI itself. The resulting tests are a lot quicker, enabling us to run our acceptance test suite more frequently.

Technique 5: Optimise Your Application

As our acceptance test suite for Pulse grew, naturally it began taking longer to execute. More problematically, however, several tests towards the end of the suite were taking longer and longer to run. This clearly highlighted a scalability issue: as more data was piled into the testing installation, tests took longer to execute. The situation became so bad that a single save in the configuration UI could take tens of seconds to complete.

A simple way to solve this would be to blow away the data after each test (or at regular intervals) so that we don’t see the build up. This would improve the performance of the test suite, but would also hide a clear performance issue in the application. Better to keep on eating our own dog food and fix the underlying problem itself. This way not only would the tests execute more quickly, but (more importantly) the product would also be improved.

In our example, basic profiling identified the major bottleneck, which we were able to overcome using a simple cache. With this one change the time taken to run our test suite was better than halved! More importantly, the latency in the Pulse UI was decreased by several times to an acceptable level.

Naturally optimisation is not always this easy. The more tuned your application, the harder it becomes to find such dramatic improvements. However, it pays to keep an eye on your test suite performance, and profile if things don’t feel right. You might just find a simple win-win situation like the one described.

Technique 6: Streamline the Environment

I'll assume, first of all, that your acceptance tests are allowed to run on a dedicated box. At the very least, while the tests are running, the box should not be doing anything else. The simplest way to ensure this is to give the tests their own box, with a clean operating system install, and only the minimal dependencies required to run your suite.

It is often worth a bit of extra effort to ensure your initial setup is as streamlined as practical. For example, even a clean operating system install could include a lot of software that is not required for your tests. Even worse, there could be a number of services running in the background, contending for resources. Take a few minutes during or after installation of the box to turn off these superfluous services.

Less obviously, although perhaps more importantly, ensure you maintain a clean environment. Over time, the cruft accumulated by running thousands or millions of tests can severely degrade the performance of a box. For example, our Pulse acceptance test suite makes heavy use of temporary files. If left unchecked, these files can accumulate on the box, in some cases causing severe performance issues (I'm looking at you, Windows XP!). The first line of defence is to write tests that clean up after themselves. However, getting this perfect is rarely practical, so it is wise to regularly clean up the box independently of the tests.

If you are maintaining a lot of boxes, or indeed find that your tests are rapidly building up cruft, consider automated streamlining. For example, you can take use virtualisation to take and restore snapshots of your streamlined environment.

Technique 7: Buy a Bigger Box

Sometimes the best solutions are the simplest: just run your tests on faster hardware. When our acceptance test suite started reaching the limits of our last set of build boxes, the performance degradation was severe. We purchased a couple of new boxes, and instantly our build times were reduced by 30%. Trust me: you don't regret spending that small chunk of cash!

Sometimes even a tiny upgrade can have a huge impact. Say, for example, that your tests are exhausting physical RAM and beginning to swap. An extra stick of RAM, a trivial cost, can have a huge impact on your build times.

Return to articles index.