In Search of Test Discovery Solutions in iOS

Introduction

Five years ago, I wrote a blog post about extracting test methods before runtime in iOS. This falls under a broader category of test discovery - the process of identifying test cases in a codebase.

While test discovery is relatively straightforward in scripting languages, it is more challenging in iOS and other statically typed languages. For a Swift package project, this can be done with the swift test list command. Meanwhile, projects using .xcworkspace and .xcodeproj do not have such APIs. As a result, most existing approaches rely on heuristics.

In the above post, I discussed extracting tests by reading symbols from compiled xctest binaries, based on certain XCTest conventions. However, with the emergence of the Swift Testing framework, those conventions no longer hold.

Interestingly, Xcode appears to recognize tests very early in the development process. As soon as a project is opened, the Test Navigator displays discovered tests. This suggests that understanding how Xcode discover tests could help achieve similar results. Unfortunately, there is little documentation on this topic. My intuition is that it might involve SourceKit or Xcode’s indexing mechanism.

Exploring SourceKit-LSP

While working on a Swift package project in VSCode, I noticed that the Testing view displayed the project’s test cases. According to the vscode-swift documentation, this extension uses SourceKit-LSP for code completion as well as test discovery.

However, SourceKit-LSP does not support Xcode projects or workspaces. If you open an iOS project with .xcworkspace and .xcodeproj, VSCode won’t display any tests, and code completion is also broken.

Initially, I suspected that this might be a limitation of the vscode-swift extension. So, I attempted to patch the extension, hoping to make it work with Xcode projects. However, after digging into the SourceKit-LSP codebase, I concluded that it lacks built-in support for Xcode-managed projects. Instead, we need to implement a Build Server Protocol (BSP) server that understands the .xcworkspace and .xcodeproj formats. Related discussion: sourcekit-lsp/issues/730.

Integrating SourceKit-LSP with a custom BSP server would require further research, which I plan to explore later.

Leveraging Index Store

A key observation is that after disabling Xcode indexing, tests no longer show up in the Test Navigator. This demonstrates that indexing does affect test discovery.

This makes sense because Xcode begins indexing the project shortly after opening it, for code highlighting and completion, which also leads to early test discovery.

In the project’s derived data, you notice a directory: Index.noindex/DataStore. This is where the indexing results are written to. Here, we can roughly understand the term “Index Store” as the storage of the indexing process, which corresponds to the directory above.

Note that Xcode itself does not directly handle indexing. Instead, it communicates with an indexer service - SourceKit, via cross-process communication (XPC).

 -----------------
| IDE             |
| (Xcode)         |
|  |              |
| Indexer Service | → Index Store
| (SourceKit)     |       ↑
|                 |       |
| swiftc/clang    | ------|
 -----------------

Even when Xcode indexing is disabled (IDEIndexDisable=1), the SourceKitService process still runs. The name IDEIndexDisable clearly states the scope: IDE-driven indexing. This explains the fact that although Xcode indexing is disabled, the jump-to-definition feature still (sometimes) works.

Additional, when a project is built - whether via Xcode or xcodebuild, the index store is still generated. This behavior is controlled by the index-while-building setting, which is enabled by default. In fact, it’s swiftc/clang that writes to the index store. When inspecting swiftc/clang processes, you should see the argument -index-store-path .../Index.noindex/DataStore.

The key takeaway here is: Once a project is built, its index store is available, containing symbols of that project. This is why static analysis tools like Periphery build the project first, before performing their analysis.

Test discovery with index stores can be broken into two sub-problems:

• How to ensure an index store?
• Given an index store, how to extract test methods?

Now, let’s begin with the latter because it requires the most effort.

Extracting Test Methods From the Index Store

I used a demo project containing tests for both XCTest and Swift Testing frameworks.

Apple provides a package called indexstore-db, which allows querying an index store. I used it to parse relevant symbols after building the project.

Interestingly, this package provides a dedicated function to retrieve unit test symbols: IndexStoreDB.swift#L470. Unfortunately, the results only contains XCTest test cases 😞. Swift Testing test cases remain undetected.

To extract those of Swift Testing, we need some extra works. Here’s a rough approach:

• (1) Find all test source files:
  Identify files containing symbols marked with __🟠$test_container__function__
• (2) Load all symbols in these files
• (3) For each symbol:
  • (3.1.) Check if its related occurrences contain a Test macro symbol.
    If so, this symbol represents a test method.

Source code: https://github.com/trinhngocthuyen/itest-scanner

// (1) Find test source files
let sourceFiles = allSymbolNames()
  .filter { $0.contains("__🟠$test_container__function__") }
	.compactMap { definition(of: $0)?.location.path }

// (2) Load all symbols in these files
let symbolsInSourceFiles = sourceFiles.flatMap { symbols(inFilePath: $0) }

// (3) Loop each symbol
return symbolsInSourceFiles.compactMap { symbol in
  // (3.1) Check if its related occurrences contain a Test macro symbol
  if occurrences(relatedToUSR: symbol.usr, roles: .containedBy).hasTestMacro() {
    return definition(of: symbol)
  }
  return nil
}

......
extension Sequence<SymbolOccurrence> {
  func hasTestMacro() -> Bool {
    contains { $0.symbol.kind == .macro && $0.symbol.name.hasPrefix("Test(") }
  }
}

Ensuring Index Store Availability

The approach above works if an index store is available. The question is: how do we ensure it exists?

During local development: Xcode indexing ensures the index store is available. Testing with the Wikipedia project shows that test-related symbols appear in the index store within a minute, even with no cache.

On CI/CD: Relying on Xcode indexing is not the case for automation. Since the index-while-building setting ensures a generated index store, we can extract test symbols after the build process. However, some projects disable this setting to reduce extra build tasks, to optimize build time. In such cases, you may consider enabling it for test targets only.

I wish there’s a way to trigger indexing for a project without requiring a full build, as building is a heavyweight operation. This is an area for improvement, where 3rd-party tools could help bridge the gap.

Conclusion

With the rise of Swift Testing, test discovery becomes more challenging. Neither Apple’s SourceKit-LSP nor IndexStoreDB natively supports Xcode projects which are prevalent in iOS development. Further effort is needed to make it work for both XCTest and Swift Testing frameworks.

In this blog post, I have presented an approach to extract test methods by querying symbols from index stores. While this technique is a practical solution, there are still many areas for improvement.

The lack of built-in support for test discovery in Xcode projects means an opportunity for better tooling, whether from Apple or from community.