Swift Testing and the Compatibility With xcodebuild

Introduction

Swift Testing is a new test framework introduced by Apple at WWDC 2024. It offers a range of macro-based features, making test writing more expressive. One of the key features driving migration from XCTest to Swift Testing is parameterized testing. While this concept has been widely available in other languages (ex. pytest introduced it in 2011), Swift Testing now enables repeated test execution with different inputs and expected outputs.

Despite these enhancements, one important consideration is whether Swift Testing integrates well with xcodebuild, the foundation of iOS testing with CI/CD.

Coexistence of Swift Testing and XCTest

Swift Testing and XCTest can coexist within the same Xcode project. This enables an incremental and hybrid migration strategy. For a typical iOS engineer working within Xcode most of the time, their workflow remains unchanged. They can write, run, and view test results directly in Xcode. However, infra engineers may face additional challenges when integrating Swift Testing with xcodebuild.

Take parallel testing for example, a common approach is discovering tests in the project, dividing them into multiple chunks, and running each of them in concurrent jobs. This blog post explores two key issues in that pipeline:

• Extracting test functions (from the xctest binary)
• Running selective tests with xcodebuild

Extracting Tests From the xctest Binary

Discovering tests and splitting them into chunks is relatively easy in scripting languages like Ruby or Python, where tests can be easily scanned from source files. However, in an iOS project, there are no public APIs to retrieve a list of tests before runtime. A common heuristic approach is parsing symbols in the xctest binary (see this post for more details). This technique relies on XCTest’s conventions:

• Test functions must be prefixed with test
• They must be non-private, and without any arguments

How Swift Testing Changes Test Discovery

Swift Testing does not follow these conventions. Any function, even a private one, can be turned into a test using the @Test macro. Test functions do not need to be inside an XCTestCase. They can be in any class, struct, enum, or even exist as free functions. Additionally, parameterized tests accept arguments. In the following example, check, foo, bar and fib are all test functions.

@Test private func check() { }

struct NewTestCase {
  @Test func foo() { }
  @Test func bar() { }

  @Test(arguments: [(1, 1), (2, 1), (3, 2), (4, 3), (5, 5)])
  func fib(n: Int, expected: Int) { }
}

Extracting Swift Testing Functions from Binary Symbols

Fortunately, it is still possible to extract test functions from the xctest binary by examining symbol names:

• The expansion of the @Test macro includes a special marker, __🟠$test_container__function__func, which helps identify test functions.

  

• The output from nm -gU <path/to/xctest/binary> | swift demangle contains both the original function and its expanded form. Comparing the two allows us to extract test functions programatically.

  

Running Selective Tests With xcodebuild

In Xcode, individual tests can be run by clicking the diamond button in the Test Navigator. For automation, xcodebuild provides the -only-testing argument, which accepts test identifiers in the following formats:

• <Target> (ex. AppTests): Runs all tests in the target
• <Target>/<Class> (ex. AppTests/TestCaseA): Runs all tests in a specific class
• <Target>/<Class>/<function> (ex. AppTests/TestCaseA/testA1): Runs a single test function

This works well with XCTest but not with Swift Testing. For example, running:

xcodebuild test ... -only-testing 'AppTests/NewTestCase/foo'

results in no tests being executed. Let’s explore why.

Debugging the Issue

We know for sure that running selective tests in Xcode works. So, let’s run a given test in Xcode, and inspect xcresult logs (using xcparse) to understand what’s happening under the hood.

The logs show a key difference between XCTest identifiers and Swift Testing identifiers. Swift Testing identifiers include parentheses at the end (ex. AppTests/NewTestCase/foo()).

Given this clue, I attempted to run again with parentheses added to the test identifiers as follows:

$ xcodebuild test ... -only-testing 'EXTests/NewTestCase/foo()'

The result still remained unchanged, ie. no tests getting executed. The logs show that the identifier was just like before (ie. without parentheses).

....
  identifiersToRun:<XCTTestIdentifier leaf, method, swift ["NewTestCase", "foo"]>
....

My hypothesis is that xcodebuild strips the ending parentheses, but only the last pair. Therefore, I tried adding an extra pair of parentheses to the identifier. Surprisingly, it works, confirming my hypothesis.

$ xcodebuild test ... -only-testing 'EXTests/NewTestCase/foo()()'

Note that, for free test functions, we don’t need to add extra parentheses because the strip behavior only applies to the 3rd component (ie. test function) of the identifier.

Patching the Issue With XCTestConfiguration

Although we can now run selective tests with xcodebuild, it occurs to me that this compatibility issue is more of Apple (developer tools) issue. Because of it, we - as engineers, need to bear the additional maintenance cost, including:

• Keeping track of Swift Testing’s tests
• Modifying the test identifiers passed to xcodebuild accordingly, excluding free test functions.

From an engineer’s perspective, EXTests/NewTestCase/foo should work as there’s a matching test with the identifier. This motivated me to come up with a patch in the native code instead.

Now, back to the xcresult logs. The following content suggests that XCTestConfiguration may be a good start.

Running tests with active test configuration: <XCTestConfiguration: 0x101b09260>
	                  testBundleURL:.........
	         testBundleRelativePath:.........
	              productModuleName:EXTests
	                  testSelection:<XCTTestSelection: 0x600000c5f540>
	            identifiersToRun:<XCTTestIdentifier leaf, method, swift ["NewTestCase", "foo()"]>
	            identifiersToSkip:
...

Checking XCTestConfiguration’s header, we find activeTestConfiguration and testsToRun, which are related to the problem.

A possible approach to fix this issue is:

• (1) Swizzle the XCTestConfiguration.activeTestConfiguration function.
• (2) In the swizzling handler, loop identifiers in testsToRun. For each one, check if the last component ends with a closing parenthesis. If not, add a new identifier with parentheses appended to testsToRun.

Check out the demo code: here.

@implementation XCTestConfiguration (Swizzlings)
+ (void)load {
  Method m1 = class_getClassMethod(self, @selector(activeTestConfiguration));
  Method m2 = class_getClassMethod(self, @selector(hooked_activeTestConfiguration));
  method_exchangeImplementations(m1, m2);
}

+ (id)hooked_activeTestConfiguration {
  XCTestConfiguration* res = [self hooked_activeTestConfiguration];
  NSMutableSet* swiftTestingIdentifiers = [NSMutableSet set];
  for (XCTTestIdentifier* identifier in res.testsToRun) {
    // Test identifiers suffixed with parentheses `()`, `(x:)` are recognized as swift-testing identifiers.
    // Therefore, we just need to add a swift-testing identifier to `testsToRun`.
    // If an identifier is added but there is no such a test function, no worries, it's gonna be skipped.
    if (![identifier.lastComponent hasSuffix:@")"]) {
      NSMutableArray* components = [NSMutableArray arrayWithArray:identifier.components];
      components[components.count - 1] = [NSString stringWithFormat:@"%@()", identifier.lastComponent];
      [swiftTestingIdentifiers addObject:[[XCTTestIdentifier alloc] initWithComponents:components argumentIDs:nil options:0]];
    }
  }
  [res setTestsToRun:[res.testsToRun setByAddingTestIdentifiersFromSet:swiftTestingIdentifiers]];
  return res;
}
@end

This patch ensures that xcodebuild treats Swift Testing identifiers correctly without requiring changes to infra-related code.

Conclusion

Swift Testing brings powerful features but also introduces challenges when integrating with xcodebuild, particularly in use cases requiring -only-testing for selective test execution. By analyzing binary symbols and patching XCTestConfiguration, we can ensure compatibility and maintain efficient automation workflows… Ideally, Apple should improve xcodebuild’s support for Swift Testing in future updates, eliminating the need for these workarounds 😅.