Running parallel test jobs in Bamboo
Looking for ways to run 21,000 functional tests in under an hour? If so, parallelization is your friend.
Massive builds are a fact of life. You start with a little bit of testing, fighting to keep your builds trim and terrific. Eventually you need to make your test suites run against different browsers, operating systems, databases servers, and so on.
Eventually, you have to break up your builds into many smaller chunks that can run in parallel. This is when the power of TestNG and JUnit comes in, and works hand-in-hand with Bamboo's jobs and stages to help you cut your build times to a third (or a tenth, or how ever many batches you have) by breaking up your build. Read on to learn more about why you should use these two together and how the combo works with Bamboo.
TestNG groups are our favorite! They are the most flexible way to split your tests into sub-suites. At Atlassian, we use Maven as our build engine, which supports TestNG out of the box. By sorting your test suite into groups, then creating separate jobs in Bamboo to run each group, you can speed up your testing stages by 100% – and likely, better. TestNG is designed for Java tests, but can also be used for Ruby projects. And it’s not the only framework to offer groups.
Let’s look at a Bamboo inspired example:
Each test has been assigned to groups representing functional areas of the application, level of technology stack, maturity level, and whether it should be run as a post-deploy smoke test. This is incredibly powerful because you can use these assignments at run time to pull tests from disparate classes or packages and run them inside a single job. A job that runs all the API-level tests having to do with configuration functionality, for example. Or a job that runs all the tests for features that are still in development and not really expected to pass yet (TDD, anyone??).
And you guessed it: the logical next step is to create jobs that cover the full body of tests you wish to execute, and run those jobs in parallel inside a stage. More on that farther down the page.
DataProvider is a great way to do parametrized tests. It comes to the rescue when we need to run the same test over and over again with different inputs. A good example is when we need to check if our validation algorithm for a certain operation is scanning all the possible use cases out there. Here’s a simple example from TestNG’s site:
But wait: Junit supports parameterized tests, which is essentially what this is. So why bother with TestNG? First, TestNG’s implementation is slicker and easier than JUnit’s. Second, and more importantly, the reporting in TestNG is much clearer. TestNG tells you exactly which inputs were used for each execution, making it much easier to diagnose failures. JUnit just tells you which array the parameters came from, and their positions therein – which is about as clear as mud.
Adherents to the D.R.Y principle (and that’s most of us) rely on @BeforeTest or @BeforeClass set-up methods. But if something goes awry in set-up, there’s no point in trying to run the actual test. At that point, we already know it's gonna fail.
With TestNG, the default behavior is to simply skip a test if its upstream dependency fails. We love this for two reasons. First, executing the actual test is a waste of time and just means you wait that much longer to find out this revision of your code no es bueno. Second, when you look at your test results, your count of skipped tests makes it immediately clear that something fundamental to your application is borked. Much faster than diving into 350 test failures and, somewhere around number 20, noticing that they’ve all failed on exactly the same call.
And it’s not just set-up methods that can trigger a skip. If you’ve declared TestA as being dependent upon TestB, and TestA fails, TestB will be skipped. Here’s what that looks like in your build logs:
Implementing groups in your test code
Like a handful of other testing frameworks, TestNG takes its cues from an @Test annotation. Inside that, you can place a number of attributes, one of which is Groups. Group names are both dynamic and arbitrary –that's a small sentence, with big implications.
First, you can create a new group on the fly just by assigning a test to it. No need to declare and define the group in a separate file. The act of assigning a test to a group will also create that group, if it doesn’t already exist. (Now, that’s a double-edged sword because it also means you can end up with a group called “smoke_test” and another called “smoke_tets” if you’re careless.)
Second, you can devise whatever system (or systems) of meaning you like for your groups. Slice them along functional area of the application, by level of the code stack, by whether you currently expect the test to pass, etc. And tests can belong to multiple groups. So a single test might conceivably belong to the “user_authentication”, “API”, “smoke_test” and “in_development” groups. TestNG even supports groups of groups! Look at their documentation to find out more.
Keep in mind that if your application’s super-structure doesn’t know about TestNG, you’ll get all sorts of errors and your builder won’t know what do with all those groups at runtime. So be sure to declare TestNG as a dependency for your project or module.
Using groups in Bamboo builds
Bamboo allows setting up jobs based on the groups you’ve defined in your test code. If you’re executing tests via Maven, Gradle, or Ant, this is pretty trivial. In the example shown here, we are passing a group name into Maven as an argument. When it’s time to run the tests, TestNG will scour our project, pull out all the tests belonging to that group, and run only those. It doesn’t matter if the tests for that group are located in different classes or packages. As long as they’re in the same project or module, TestNG will find them.
Continuous integration is all about quick feedback, and of course the number of tests you run has a huge impact on the feedback process and speed of your integration pipeline. It does not take the same amount of time to run 10 tests as it takes for 10,000. With Bamboo's parallel jobs feature, and TestNG allowing us to split tests into batches we can save lots of the test execution time compared to if we have to run things in sequence.
Note that because each job has its own working build directory, you’ll need to add a Source Code Checkout task at the beginning of each job to pull down the test code. If your code base is large, you can take advantage of the option to check out from a sub-directory in your repo and only pull down that part of the tree.
Agents of change
It’s probably obvious, but splitting your test suite into 10 batches isn’t going to help you much if you only have one or two build agents available to run them. Of course we’d love if if everyone added more agents to their Bamboo license so they can really take advantage of parallelization. Maybe that’s feasible for you, and maybe not. Either way, be mindful of your agent count when considering how to split up your test suite so you’re not just spinning your wheels.
When thinking about agents, don't forget about our latest agent's feature: dedicated agents. This feature allows assigning an agent to a certain build or deployment so it doesn't have to sit in the build queue.
A parallel universe
As with most things in build engineering, there are several ways to accomplish parallelized testing. Tinkering with custom scripts or your suite.xml are two other common approaches. For big multi-module projects, it might make sense to split the suites/groups into separate repositories and run a job for each repo. Regardless of which method you chose, you’ll get the most bang for your buck by focusing your parallelization efforts on integration-, API- and UI-level tests. Unit-level tests already run quite fast, so they’re unlikely to be the bottleneck in your builds. (Extra credit for the over-achievers: many testing frameworks allow you to run unit tests multi-threaded!)
Let me summarize this article with one strong conclusion: if you are looking for ways to run 21,000 functional tests in under an hour (as the JIRA Software team does), parallelization is your friend. Combine the power of JUnit and TestNG with Bamboo's stages, then grab a coffee (or something stronger – you deserve it!) lay back, relax and let the tests do their thing.
Artifact passing for agile teams
Let's talk about artifact passing. It helps with fast feedback, and the benefits to agile teams in particular go way beyond that, as you’ll see.
Continuous delivery from code to customer
Bamboo is a continuous delivery tool that ties automated builds, tests and releases together in a single workflow. Take the tour to learn more.