In the previous part (Running Android UI Tests — Part 1) we explained how we run Android UI tests and could easily clear data, clear notifications, pass params by adding simple Java annotations to our test methods. In addition, we collect logs, dump local db and preferences xml file, record and extract mp4 video and more. Read part 1 and check our open source project for step by step tutorials and examples.

In this part we will show how to run many tests in parallel on multiple devices. We will also show how to run group of sequencing tests on the same device when running in parallel. We improved execution UI tests time by 40% 💫!

Parallel — The regular approach vs. push approach

Before diving into optimizations and how we improve parallel execution test times, we explain the current sharding option that is provided by the Android testing framework.

The regular approach - Sharding

The Android Instrumentation runner supports test sharding using the numShards and shardIndex arguments (documentation).

Run:

adb -s {device} shell am instrument -w -e numShards 4 -e shardIndex 1

• numShards = number of groups we ask Android to divide all our tests.
• shardIndex = the group index we want to run on specific {device}.

This means, that if we want to run in parallel on 4 devices, we need to execute the same shell command with shardIndex starting from 1 to 4.

The main issues of this approach are:

1. Execution time — Android will split the tests to the groups without taking into consideration the execution time of each group. It might happen that the group of test in shard #1 will run for 1 hour, while the other shard will finish in 20 minutes.
2. Grouping following tests — Android will split the tests to groups randomly. So if you decided to split a very big scenario to sequential tests in the same class, sharding might break your logic by putting the tests in different shards. And you have no control over this.

The push approach

The idea is simple. Instead of splitting all tests into equal groups, we push the next test to a free device. If all devices are running, we wait until one of the devices completes the test execution.

For example: We took 16 UI tests and ran them by using 2 different approaches. You can see an improvement of 40% when pushing tests to free devices, instead of allocating an equal number of tests per device.

Running the script

./scripts/8/raw.sh artifacts/raw-tests.txt
./scripts/8/plan.sh artifacts/raw-tests.txt artifacts/plan-tests.txt
./scripts/8/run-shard.sh artifacts/plan-tests.txt artifacts

Check the source code and detailed results here.

The script will:

1. Dump tests in raw format.
2. Prepare execution plan and split the tests to groups.
3. Run the groups of tests in parallel on all connected devices and emulators.

Grouping following tests

It might be that we have a big scenario and we split it to multiple tests. The tests methods will follow each other in the same Class. This is fine if we run the tests on one device. But if we decide to use sharding, Android will split the tests to different devices. This is bad, because some of the following tests will fail.

This is why we added the @Following annotation and this annotation will enforce grouping and executing all following tests on the same device when running in parallel.

For example: We set tests number 2,3 and 4 to be sequential. This means that test number 3 can successfully run only if test number 2 passes. Same for test number 4 that is following test number 3.

If we run the regular way, the tests will be split randomly between shards. In this example, luckily tests number 2 and 3 are running on the same device while unluckily test 4 will always fail! :(.

If we use the @Following annotation and push the group of following tests to a free device, we will be able to keep the following dependency between tests on the same device. In addition, you can see that the tests are distributed differently than before and we could still improve 40% of max execution time. Amazing 💪 :)

Running the script

# dump tests to raw file
./scripts/9/raw.sh artifacts/raw-tests.txt

# build a plan
./scripts/9/plan.sh artifacts/raw-tests.txt artifacts/plan-tests.txt

# run tests in parallel on all connected devices & emulators
./scripts/9/run-shard.sh artifacts/plan-tests.txt artifacts

Check the source code and detailed results here.

The script will:

1. Dump tests in raw format.
2. Prepare execution plan and split the tests to groups.
3. Run the groups of tests in parallel on all connected devices and emulators.

Summary

In this part, we learned how we can run our UI tests in parallel and improved the max time execution by 40%. In addition, we can safely run following tests on the same device when running in parallel.

This is an addition to what we already learned in part 1. And you can use combination of all annotations as it used in our sample tests. Just run our full examples and check it yourself.

Source code: https://github.com/medisafe/run-android-tests

This post was also posted on Medium