CI: Heuristically Extracting Test Methods Before Runtime

2020, Nov 8

1. Introduction

Reducing pipeline time is a key problem in the pipeline time reduction epic. As mentioned in the previous post, this can be done by dividing tests in the project into smaller sets and run them in parallel CI jobs (sometimes called runners). The assignment of what tests to what jobs usually takes place at the end of the build job, before tests being executed. In order to assign/distribute tests to runners, we must know what tests we have. A test, in this context, can be either:

A test target, for ex. AppUnitTests, AppUITests.
A test class (subclass of XCTestCase) in a test target.
A test method inside a test class. A test method is a non-private method, prefixed with test, and has no arguments, no return value.

Usually, there are only a few test targets in the project. Those targets can be easily derived from the scheme, or from the *.xctestrun file after building the project successfully. However, splitting tests by target is not a good choice because:

A target might finish in a really long time. In our project, it takes approximately 50 mins to run all tests in UI test target.
There could be a significant imbalance in test time between targets. One could take only 5 mins to finish while another takes up to 30 mins.

Therefore, splitting tests by test class or test method is a key to the efficiency of the infrastructure. Unfortunately, it’s not trivial to extract test classes and test methods in the project before runtime. This blog post introduces 2 approaches to heuristically detect test classes and test methods of a test target.

2. Heuristic Approaches

2.1. Using Sourcery

Sourcery is a code generator for Swift. It’s built on top of SourceKit. For those who did not know, SourceKit is the magic behind Xcode indexing which brings Xcode auto-completion to life. When Xcode indexes source files of a project, you can notice a running process called SourceKitService. Basically, what it does is to build the Abstract Syntax Tree (AST) from the source files. From then, you can query information about avalaible classes, structs, protocols, methods and so forth.

For example, the following stencil template would generate a list of test methods of classes inheriting from BaseUITestCase.

{% for type in types.inheriting.BaseUITestCase %}
{% for method in type.methods where method.accessLevel != "private" %}
{% if method.selectorName|hasPrefix:"test" and method.parameters.count == 0 %}
{{type.name}}.{{method.selectorName}}
{% endif %}
{% endfor %}
{% endfor %}

What the above template does is simple:

For each subclass of BaseUITestCase, look for methods prefixed with test that is not private and does not have any argument, then print its class and method.

When running the CLI, we need to specify the source folder/files and the stencil templates we’re using. Then, Sourcery scans code in the given source folder/files, and generates data based on the given templates.

$ sourcery --sources AppUITests --templates all_ui_tests.stencil

The output would look like this:

TestCaseA.testA1
TestCaseA.testA2
TestCaseB.testB1

Now, let’s talk about pros and cons.

One big advantage of this approach is that it does not require any build products. This tests extraction step can be done as early as you wish. Also, it’s easy to adopt this solution as the setup is lightweight.

However, this approach has some drawbacks:

It does NOT work well with pre-processing flags. Following is the example in which we encounter the issue while migrating from KIF tests to XCUITests. Since Sourcery has no idea about pre-processing flags and it just scans code from top to bottom, the type of BaseUITestCase can be wrongly resolved to BaseKIFTestCase while it should be BaseXCUITestCase.

#if XCUITEST
typealias BaseUITestCase = BaseXCUITestCase
#else
typealias BaseUITestCase = BaseKIFTestCase
#end

It might take a fairly long time to finish if there are a lot of source files to be scanned. With our project, it took around 10 secs to run with the template above.
Moreover, your source files should be well structured to give an accurate output.

2.2. Parsing Symbols of Target Binaries

Another approach is to parse the symbols of a test target binary. Assume we have a UI test target AppUITests with a test class UITestCaseX of which the declaration is as follows:

class UITestCaseX: XCTestCase {
  func testX1() { }
  func testX2() { }
  func testX000(_ x: Int) { }  // <-- Not a test method b/c of having arguments
  private func testX999() { }  // <-- Not a test method b/c of being private
}

In this class, there are 2 test methods testX1 and testX2. Meanwhile, testX000 and test999 are not test methods, meaning, they won’t be picked up when running UITestCaseX.

…

After compiling the project, the symbols of this class will be merged into the target executable (or target binary). As AppUITests is a UI test target, its binary locates inside the Plugins folder of the runner app bundle: AppUITests-Runner.app/Plugins/AppUITests.xctest/AppUITests.

We can look up the symbols of this binary using the nm (or, xcrun nm) command. Since the symbols were obfuscated, we’re gonna pipe the output to xcrun swift-demangle to make it more readble:

$ xcrun nm ${DERIVED_DATA_PATH}/AppUITests-Runner.app/PlugIns/AppUITests.xctest/AppUITests \
  | xcrun swift-demangle

Let’s take a quick look at the output.

...
0000000000001740 t @objc SpecExtractorUITests.UITestCaseX.init(invocation: __C.NSInvocation?) -> SpecExtractorUITests.UITestCaseX
00000000000014b0 T SpecExtractorUITests.UITestCaseX.testX1() -> ()
00000000000014d0 t @objc SpecExtractorUITests.UITestCaseX.testX1() -> ()
0000000000001d04 S method descriptor for SpecExtractorUITests.UITestCaseX.testX1() -> ()
0000000000001510 T SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001530 t @objc SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001d0c S method descriptor for SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001780 T SpecExtractorUITests.UITestCaseX.__allocating_init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
00000000000017b0 T SpecExtractorUITests.UITestCaseX.init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
0000000000001840 t @objc SpecExtractorUITests.UITestCaseX.init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
0000000000001570 T SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
0000000000001590 t @objc SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
0000000000001d14 S method descriptor for SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
00000000000015e0 T SpecExtractorUITests.UITestCaseX.(testX999 in _52DD9D513BE9E5B52CB47DED7BE88AA3)() -> ()
0000000000001d1c s method descriptor for SpecExtractorUITests.UITestCaseX.(testX999 in _52DD9D513BE9E5B52CB47DED7BE88AA3)() -> ()
...

Each row in the output represent the nlist entry in the symbol table.
The second column in each entry is a letter representing the segment type in the binary. T and t refer to __TEXT and __text which translates to the segments containing the compiled code to be run. These letters being in uppercase or lowercase means the symbol is external or non-external respectively. Non-external, in this context, means private.

For our case, we only need to extract symbols with type T that end with .test<BlahBlah>() -> (). A simple regex could help with the extraction: \S+ T (\S+\.\S+\.test\S+)\(\) -> \(\).

As compared to the previous approach (using Sourcery), this solution provides a more accurate output since it parses symbols from the target binary. We won’t face the issue with pre-processing flags mentioned earlier. Moreover, the time to extract tests is pretty fast as it depends solely on the binaries (not the source files).

3. Discussion

In this post, we have got to know 2 different techniques to heuristically extract test methods of the project. Each has its own pros and cons. Which one to use really depends on your project. If your project has a lightweight and consistent codebase, using Sourcery probably suffices. Otherwise, you can consider the second approach which requires some additional efforts to search for the binaries, then extract the symbols of interest from them. In our project, we initially used the former approach and then migrated to the latter due to some issues mentioned before. Kudos to my colleague Alex for coming up with this approach.

There is one case we have not yet covered. If you use Quick to write tests, then extracting test methods does not make sense anymore because Quick swizzles to make up tests of a test class in a different way. To be more specific, a test method of a test class will not be picked up to run if that test class is a subclass of QuickSpec. In this case, what you can do is to extract test classes instead.

…

Stay tuned for more tips ahead!

Tech

ci ios