CI: Heuristically Extracting Test Methods Before Runtime

1. Introduction

Reducing pipeline time is a key problem in the pipeline time reduction epic. As mentioned in the previous post, this can be done by dividing tests in the project into smaller sets and run them in parallel CI jobs (sometimes called runners). The assignment of what tests to what jobs usually takes place at the end of the build job, before tests being executed. In order to assign/distribute tests to runners, we must know what tests we have. A test, in this context, can be either:

Usually, there are only a few test targets in the project. Those targets can be easily derived from the scheme, or from the *.xctestrun file after building the project successfully. However, splitting tests by target is not a good choice because:

Therefore, splitting tests by test class or test method is a key to the efficiency of the infrastructure. Unfortunately, it’s not trivial to extract test classes and test methods in the project before runtime. This blog post introduces 2 approaches to heuristically detect test classes and test methods of a test target.

2. Heuristic Approaches

2.1. Using Sourcery

Sourcery is a code generator for Swift. It’s built on top of SourceKit. For those who did not know, SourceKit is the magic behind Xcode indexing which brings Xcode auto-completion to life. When Xcode indexes source files of a project, you can notice a running process called SourceKitService. Basically, what it does is to build the Abstract Syntax Tree (AST) from the source files. From then, you can query information about avalaible classes, structs, protocols, methods and so forth.

For example, the following stencil template would generate a list of test methods of classes inheriting from BaseUITestCase.

{% for type in types.inheriting.BaseUITestCase %}
{% for method in type.methods where method.accessLevel != "private" %}
{% if method.selectorName|hasPrefix:"test" and method.parameters.count == 0 %}
{% endif %}
{% endfor %}
{% endfor %}

What the above template does is simple:

When running the CLI, we need to specify the source folder/files and the stencil templates we’re using. Then, Sourcery scans code in the given source folder/files, and generates data based on the given templates.

$ sourcery --sources AppUITests --templates all_ui_tests.stencil

The output would look like this:


Now, let’s talk about pros and cons.

One big advantage of this approach is that it does not require any build products. This tests extraction step can be done as early as you wish. Also, it’s easy to adopt this solution as the setup is lightweight.

However, this approach has some drawbacks:

typealias BaseUITestCase = BaseXCUITestCase
typealias BaseUITestCase = BaseKIFTestCase

2.2. Parsing Symbols of Target Binaries

Another approach is to parse the symbols of a test target binary. Assume we have a UI test target AppUITests with a test class UITestCaseX of which the declaration is as follows:

class UITestCaseX: XCTestCase {
  func testX1() { }
  func testX2() { }
  func testX000(_ x: Int) { }  // <-- Not a test method b/c of having arguments
  private func testX999() { }  // <-- Not a test method b/c of being private

In this class, there are 2 test methods testX1 and testX2. Meanwhile, testX000 and test999 are not test methods, meaning, they won’t be picked up when running UITestCaseX.

After compiling the project, the symbols of this class will be merged into the target executable (or target binary). As AppUITests is a UI test target, its binary locates inside the Plugins folder of the runner app bundle:

We can look up the symbols of this binary using the nm (or, xcrun nm) command. Since the symbols were obfuscated, we’re gonna pipe the output to xcrun swift-demangle to make it more readble:

$ xcrun nm ${DERIVED_DATA_PATH}/ \
  | xcrun swift-demangle

Let’s take a quick look at the output.

0000000000001740 t @objc SpecExtractorUITests.UITestCaseX.init(invocation: __C.NSInvocation?) -> SpecExtractorUITests.UITestCaseX
00000000000014b0 T SpecExtractorUITests.UITestCaseX.testX1() -> ()
00000000000014d0 t @objc SpecExtractorUITests.UITestCaseX.testX1() -> ()
0000000000001d04 S method descriptor for SpecExtractorUITests.UITestCaseX.testX1() -> ()
0000000000001510 T SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001530 t @objc SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001d0c S method descriptor for SpecExtractorUITests.UITestCaseX.testX2() -> ()
0000000000001780 T SpecExtractorUITests.UITestCaseX.__allocating_init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
00000000000017b0 T SpecExtractorUITests.UITestCaseX.init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
0000000000001840 t @objc SpecExtractorUITests.UITestCaseX.init(selector: ObjectiveC.Selector) -> SpecExtractorUITests.UITestCaseX
0000000000001570 T SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
0000000000001590 t @objc SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
0000000000001d14 S method descriptor for SpecExtractorUITests.UITestCaseX.testX000(Swift.Int) -> ()
00000000000015e0 T SpecExtractorUITests.UITestCaseX.(testX999 in _52DD9D513BE9E5B52CB47DED7BE88AA3)() -> ()
0000000000001d1c s method descriptor for SpecExtractorUITests.UITestCaseX.(testX999 in _52DD9D513BE9E5B52CB47DED7BE88AA3)() -> ()

For our case, we only need to extract symbols with type T that end with .test<BlahBlah>() -> (). A simple regex could help with the extraction: \S+ T (\S+\.\S+\.test\S+)\(\) -> \(\).

As compared to the previous approach (using Sourcery), this solution provides a more accurate output since it parses symbols from the target binary. We won’t face the issue with pre-processing flags mentioned earlier. Moreover, the time to extract tests is pretty fast as it depends solely on the binaries (not the source files).

3. Discussion

In this post, we have got to know 2 different techniques to heuristically extract test methods of the project. Each has its own pros and cons. Which one to use really depends on your project. If your project has a lightweight and consistent codebase, using Sourcery probably suffices. Otherwise, you can consider the second approach which requires some additional efforts to search for the binaries, then extract the symbols of interest from them. In our project, we initially used the former approach and then migrated to the latter due to some issues mentioned before. Kudos to my colleague Alex for coming up with this approach.

There is one case we have not yet covered. If you use Quick to write tests, then extracting test methods does not make sense anymore because Quick swizzles to make up tests of a test class in a different way. To be more specific, a test method of a test class will not be picked up to run if that test class is a subclass of QuickSpec. In this case, what you can do is to extract test classes instead.

Stay tuned for more tips ahead!