google / xctestrunner Goto Github PK

View Code? Open in Web Editor NEW

147.0 19.0 68.0 263 KB

A executable binary to run prebuilt iOS tests on iOS simulator/real device.

License: Apache License 2.0

Python 99.30% Starlark 0.70%

xctestrunner's Introduction

XCTestRunner

A tool for running prebuilt iOS tests on iOS real device and simulator.

Features

It supports XCTest (Xcode Unit Test), XCUITest (Xcode UI Test).
It supports iOS 11+ iOS real device, iOS simulator.
It supports launch options configuration: test methods to run, additional environment variables, additional arguments.
It supports Xcode 10+.

Prerequisites

Install Xcode (Xcode 10+).
Install bazel (optional).

Installation

You can download the ios_test_runner.par binary in release

or build the ios_test_runner.par binary by bazel:

$ git clone https://github.com/google/xctestrunner.git
$ cd xctestrunner
$ bazel build :ios_test_runner.par
$ ls bazel-bin/ios_test_runner.par

Usage

Build your app under test and test bundle. You can use Xcode.app, xcodebuild command line tool or bazel.
Run the ios_test_runner.par binary.

In overview, there are two sub-commands in the runner binary.

test: Run test directly on connecting iOS real device or existing iOS simulator.
simulator_test: Run test on a new created simulator, which will be deleted after test finishes.

See more details by running ios_test_runner.par -h in terminal.

Notes

Disclaimer: This is not an official Google product.

XCTestRunner uses Apple native tool xcodebuild, simctl to control iOS Simulator and launch tests on iOS devices.

xctestrunner's People

Contributors

Stargazers

Watchers

xctestrunner's Issues

This copy of libswiftCore.dylib requires an OS version prior to 12.2.0

Hi :)

When running version 0.2.11 of the xctestrunner I am getting the following error when running unit tests on swift modules:

================================================================================
==================== Test output for //Libraries/SwiftModule3:SwiftModule3Tests:
2020-04-28 10:48:56,633 Will consider the test as test type logic_test to run.
2020-04-28 10:48:56,990 Failed to import biplist module. Will use tool /usr/libexec/PlistBuddy to handle the binary format plist.
2020-04-28 10:48:58,542 Failed to import biplist module. Will use tool /usr/libexec/PlistBuddy to handle the binary format plist.
2020-04-28 10:48:58,550 Creating a new simulator:
Name: New-iPhone SE (2nd generation)-13.4
OS: iOS 13.4
Type: iPhone SE (2nd generation)
2020-04-28 10:48:58,787 Created new simulator EFF9153E-C6BF-4FD6-9DAF-1084F2D4F270.
This copy of libswiftCore.dylib requires an OS version prior to 12.2.0.
Child process terminated with signal 6: Abort trap
2020-04-28 10:49:08,039 Deleting simulator EFF9153E-C6BF-4FD6-9DAF-1084F2D4F270 asynchronously.
2020-04-28 10:49:08,049 Done.

I see this happening consistently in some scenarios:

I saw it happening with bazel 2.2.0 and 3.0.0. I saw it happening with Xcode 11.4 and 11.4.1, but not with 11.3.1. I saw it in my CI (github actions) and also running locally.
I have seen it when running unit tests for swift modules, but not when running unit tests for iOS apps (with a host app).

Find a CI run failing due to this error here: https://github.com/acecilia/BuildSystemGenerator/actions/runs/89648161

Rolling back to version 0.2.10 seems to fix the problem.

XCTestSupport Issues with 14.3?

Hey folks,

Checking if anyone ran into this issue with Xcode 14.3:

Library not loaded: @rpath/XCTestSupport.framework/XCTestSupport

I can send a PR for this if needed.

Unable to run tests on an iOS 9.3 iPad Retina

xctestrunner.shared.ios_errors.IllegalArgumentError: The max OS version of iPad Retina is 9.255. But current OS version is 9.3

The maxRuntimeVersion is 9.255.255, which gets truncated to 9.255 before being turned into a float. Version comparison should probably be done lexicographically, rather than via float, since 9.3 as a version should be considered less than 9.255

Provide a full end to end example

I'm a long-time Bazel user getting into iOS development. Some of the terms are still mysterious to me. I think it would be helpful to this project and to bazelbuild/rules_apple to provide an end to end example of how to use xctestrunner.

What is a app under test?
- What's required? How much needs to be stubbed out?
- Provide a minimal template that can be used directly or extended?
What is the test bundle?
How do they fit together with xctestrunner to run on device.
Is it possible to build a multi-arch binary so that the same test can be run on both simulator and device?

'Tests_to_run' ignores the specified tests

Is the tests_to_run key supported in the current version?
My Xcode version: 11.6
Swift 5
Argument: launch_options_json_path
In my case i use json

{
    "skip_tests": ["AutoplayTests"]
}

For this launching excludes AutoplayTests class and includes other.
But when i do this case

{
    "tests_to_run": ["AutoplayTests"]
}

launching ignores all tests including AutoplayTests class

Xcode 11.3 validation issues

When I run xctestrunner on Xcode 11.3 I get the following error:

  File "../xctestrunner/file/downloaded/__main__.py", line 307, in main
  File "../xctestrunner/file/downloaded/__main__.py", line 215, in _SimulatorTest
  File "../xctestrunner/file/downloaded/__main__.py", line 170, in _RunSimulatorTest
  File "../xctestrunner/file/downloaded/__main__/xctestrunner/simulator_control/simulator_util.py", line 314, in CreateNewSimulator
  File "../xctestrunner/file/downloaded/__main__/xctestrunner/simulator_control/simulator_util.py", line 599, in _ValidateSimulatorTypeWithOsVersion
TypeError: float() argument must be a string or a number

When I add 13.3 to the test runner rule, under os_version I get:

xctestrunner.shared.ios_errors.IllegalArgumentError: The max OS version of iPhone 7 is 13.2. But current OS version is 13.3

When I add 13.2 to the test runner rule, under os_version I get:

xctestrunner.shared.ios_errors.IllegalArgumentError: The simulator os version 13.2 is not supported. Supported simulator os versions are [u'13.3'].

After reading the rules_apple documentation, I suspected the os_version means simulator runtime.

It seems like there is confusion between the SDK version Bazel is using, and the runtime version that rules_apple is using. Perhaps this is out of scope of the test runner, as os_version is determined by rules_apple, Bazel.

Add Support for Xcode 15 and iOS 17

We are trying to use xctestrunner to run testing on iOS 17 devices with xcode 15.0

We are running into the follow errors when passing a command:
First when running the following command:
ios_test_runner --app_under_test_path ios-configurator.ipa --test_bundle_path ios-configurator-ui-tests.zip --work_dir ~/ios_test_runner_work_dir --test_type xcuitest test --id 00008020-001524340281002E

we get the following error regardless of what iOS our device is on:

Command line invocation:
    /Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/xctestrun.plist -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult -collect-test-diagnostics=never

User defaults from command line:
    collect-test-diagnostics = never
    IDEBuildOperationResultBundlePath = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult
    IDEDerivedDataPathOverride = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u
    IDEPackageSupportUseBuiltinSCM = YES

Writing result bundle at path:
	/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult

xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.: File is of an unsupported type; must be either an xctestrun file or xctestproducts bundle.

I am able to fix this issue by manually changing the name of the xctestrun.plist file to test.xctestrun and rerunning the command:
/Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/test.xctestrun -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test1.xcresult -collect-test-diagnostics=never

After doing this we run into another error:

Command line invocation:
   /Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/test.xctestrun -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test1.xcresult -collect-test-diagnostics=never

User defaults from command line:
    collect-test-diagnostics = never
    IDEBuildOperationResultBundlePath = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult
    IDEDerivedDataPathOverride = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg
    IDEPackageSupportUseBuiltinSCM = YES

Writing result bundle at path:
	/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult

Testing started
2023-07-06 16:21:39.428 xcodebuild[67560:380128] [MT] DVTDevice: Error locating DeviceSupport directory using Optional("arm64e") or Optional("arm64e"): nilError
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 11.913 elapsed -- Testing started completed.
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 11.913 sec, +11.913 sec -- end

Test session results, code coverage, and logs:
	/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult

We do not run into this second error with iOS 16, only with iOS 17.

--incompatible_disallow_empty_glob

This project isn't compatible with the --incompatible_disallow_empty_glob flag.

It looks like this commit removed the TestProject/ files referenced in a glob in BUILD.

How to pass -retry-tests-on-failure to the test runner?

There is any way to pass this options to the test runner?

xcodebuild: error: Existing file at -resultBundlePath "/..sandbox/darwin-sandbox./execroot/.../test.outputs/test.xcresul

I've encountered this on a recent PR: bazel-ios/rules_ios#246 . It seems like the simulator infrastructure is flaking, but writing a result bundle. Finally when it tries to re-launch, there's an existing result bundle so it can't retry. Note that this is ran inside of the sandbox ATM.

Please find the github action here:
https://github.com/bazel-ios/rules_ios/runs/2786242526

2021-06-09 17:47:48,662 Failed to launch test on simulator. Will relaunch again.
xcodebuild: error: Existing file at -resultBundlePath "/var/tmp/_bazel_runner/c2342bca25695567c8c3f3be166b56b3/sandbox/darwin-sandbox/1286/execroot/build_bazel_rules_ios/bazel-out/darwin-dbg/testlogs/tests/ios/unit-test/ExplicitHosted/test.outputs/test.xcresult"

2021-06-09 17:47:50,048 Failed to launch test on simulator. Will relaunch again.
xcodebuild: error: Existing file at -resultBundlePath "/var/tmp/_bazel_runner/c2342bca25695567c8c3f3be166b56b3/sandbox/darwin-sandbox/1286/execroot/build_bazel_rules_ios/bazel-out/darwin-dbg/testlogs/tests/ios/unit-test/ExplicitHosted/test.outputs/test.xcresult"

No way to get xcresult from running logic tests

xctestrunner currently assumes that any test without a test host (a logic test) must be run directly via the xctest agent, and not via xcodebuild test-without-building. This means there's no way to get an xctestresult bundle from logic tests, making it much harder to analyze what happens during the run of a log test.

Changing xctest_session.py to not check if test_type != ios_constants.TestType.LOGIC_TEST, and unconditionally run logic tests via xcodebuild seems to work just fine, and is "just" slower:

Direct:
Stats over 10 runs: max = 1.0s, min = 0.9s, avg = 0.9s, dev = 0.0s

Xcodebuild:
Stats over 10 runs: max = 2.9s, min = 2.7s, avg = 2.7s, dev = 0.1s

Xcodebuild, with the sleeps shortened:
Stats over 10 runs: max = 2.5s, min = 2.2s, avg = 2.2s, dev = 0.1s

Would an option to run logic tests via xcodebuild be acceptable?

Can it run xcuitest bundle and get the report

Create a new release

Can someone with push create a new release, potentially updating 5bb42a5 first, so we can push this to the bcr

Latest xctestrunner is not compatible with remote execution

All of our tests failed with this same error after updating xctestrunner to the latest commit (7f8fc81). Running them locally doesn't reproduce the problem.

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //sometest
-----------------------------------------------------------------------------
2021-10-29 23:01:13,665 Creating a new simulator:
Name: New-iPhone 13 Pro Max-15.0
OS: iOS 15.0
Type: iPhone 13 Pro Max
2021-10-29 23:01:14,639 Created new simulator 14753767-9861-450F-BF51-FF192687DF13.
2021-10-29 23:01:14,688 Will consider the test as test type logic_test to run.
An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=404):
Invalid device: The specified device is not available for spawning processes.
2021-10-29 23:01:18,106 Deleting simulator 14753767-9861-450F-BF51-FF192687DF13 asynchronously.
2021-10-29 23:01:18,112 Done.

xcresult bundle doesn't always include `diagnosticsRef` leading to crash

Currently xctestrunner expects the diagnosticsRef key to always exist:

xctestrunner/xctestrunner/test_runner/xcresult_util.py

Line 40 in f52307f

diagnostics_id = action_result['diagnosticsRef']['id']['_value']

This isn't the case, as described in xcrun xcresulttool formatDescription:

  - ActionResult
    * Kind: object
    * Properties:
      + resultName: String
      + status: String
      + metrics: ResultMetrics
      + issues: ResultIssueSummaries
      + coverage: CodeCoverageInfo
      + timelineRef: Reference?
      + logRef: Reference?
      + testsRef: Reference?
      + diagnosticsRef: Reference?

Apparently being optional here means the key may be missing entirely.

In our case we instead have both logRef and testsRef, so those could potentially be used instead.

Example:

{
    "_type": {
        "_name": "ActionResult"
    },
    "coverage": {
        "_type": {
            "_name": "CodeCoverageInfo"
        }
    },
    "issues": {
        "_type": {
            "_name": "ResultIssueSummaries"
        },
        "testFailureSummaries": {
            "_type": {
                "_name": "Array"
            },
            "_values": [
                {
                    "_type": {
                        "_name": "TestFailureIssueSummary",
                        "_supertype": {
                            "_name": "IssueSummary"
                        }
                    },
                    "documentLocationInCreatingWorkspace": {
                        "_type": {
                            "_name": "DocumentLocation"
                        },
                        "concreteTypeName": {
                            "_type": {
                                "_name": "String"
                            },
                            "_value": "DVTTextDocumentLocation"
                        },
                        "url": {
                            "_type": {
                                "_name": "String"
                            },
                            "_value": "url"
                        }
                    },
                    "issueType": {
                        "_type": {
                            "_name": "String"
                        },
                        "_value": "Uncategorized"
                    },
                    "message": {
                        "_type": {
                            "_name": "String"
                        },
                        "_value": "XCTAssertTrue failed"
                    },
                    "producingTarget": {
                        "_type": {
                            "_name": "String"
                        },
                        "_value": "Runner"
                    },
                    "testCaseName": {
                        "_type": {
                            "_name": "String"
                        },
                        "_value": "LoadDemosTests.testLoadAllDemos()"
                    }
                }
            ]
        }
    },
    "logRef": {
        "_type": {
            "_name": "Reference"
        },
        "id": {
            "_type": {
                "_name": "String"
            },
            "_value": "..."
        },
        "targetType": {
            "_type": {
                "_name": "TypeDefinition"
            },
            "name": {
                "_type": {
                    "_name": "String"
                },
                "_value": "ActivityLogSection"
            }
        }
    },
    "metrics": {
        "_type": {
            "_name": "ResultMetrics"
        },
        "testsCount": {
            "_type": {
                "_name": "Int"
            },
            "_value": "1"
        },
        "testsFailedCount": {
            "_type": {
                "_name": "Int"
            },
            "_value": "1"
        }
    },
    "resultName": {
        "_type": {
            "_name": "String"
        },
        "_value": "action"
    },
    "status": {
        "_type": {
            "_name": "String"
        },
        "_value": "failed"
    },
    "testsRef": {
        "_type": {
            "_name": "Reference"
        },
        "id": {
            "_type": {
                "_name": "String"
            },
            "_value": "..."
        },
        "targetType": {
            "_type": {
                "_name": "TypeDefinition"
            },
            "name": {
                "_type": {
                    "_name": "String"
                },
                "_value": "ActionTestPlanRunSummaries"
            }
        }
    }
}

0.2.12 broken on macOS 10.14

The 0.2.12 par file upload to the releases uses this shebang:

#!/usr/bin/env python2

Which doesn't exist on macOS before Catalina. Instead it should be /usr/bin/python2.7 or /usr/bin/env python2.7

Add support for Python 3?

Are there any plans to drop the python 2 requirement and move to python 3 to use this library? If so I could assist in that effort

UIScreen returning wrong values when unit testing without host library

Hi:

I noticed that when unit testing with the ios_unit_testrule:

UIScreen.main.scale returns 1.
UIScreen.main.bounds.size returns 0 for height and width.

Reading the docs for ios_unit_test it says that:

ios_unit_test targets can work in two modes: as app or library tests. If the test_host attribute is set to an ios_application target, the tests will run within that application's context. If no test_host is provided, the tests will run outside the context of an iOS application. Because of this, certain functionalities might not be present (e.g. UI layout, NSUserDefaults). You can find more information about app and library testing for Apple platforms here.

Up until this point makes sense.

But, then, how is it possible that running unit tests from xcode without hostApp, accessing UIScreen properties returns the correct values for the simulator used? How does Apple do it?

My use case is snapshot testing: at the moment I do snapshot testing without hostApp. With bazel I need to use hostApp in order to access the screen properties (scale and size), which sometimes results in slightly different snapshots than the ones taken using Xcode, breaking my tests.

Would it be possible to configure the test runner, so it passes this information correctly through UIScreen?

Thanks :)

Unable to point to git repo as drop-in bazel replacement for release

One of the things that makes contributing to this repo difficult (as @keith alluded to in bazelbuild/rules_apple#495 (comment)) is that the xctestrunner git repo can't be used as a drop-in replacement for the released archive when using bazel.

The release archive has its test runner executable available as @xctestrunner//file:file, which does not exist in this repo at all.

I tried to work around this by adding

patch_cmds = ["""
        mkdir file
        echo 'alias(name = "file", actual = "//xctestrunner:ios_test_runner.par", visibility = ["//visibility:public"])' > file/BUILD.bazel
        """]

to my WORKSPACE's git_repository for xctestrunner, but then I get Python errors:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "../xctestrunner/xctestrunner/ios_test_runner.par/__main__.py", line 33, in <module>
ModuleNotFoundError: No module named 'xctestrunner.shared'

Ideally, we'd be able to point our @xctestrunner to the git repo when necessary, but currently I don't see how that's possible

Support Stacktrace when test crashes

Currently, when tests crashes, seems there is no stack trace. That would help a lot to investigate issue, especially for me when I migrate a project to bazel and some unit test crash.

How to reproduce:
Just put fatalError("") in unit tests or in testing swift code.

A similar issue is opened in Bazel but it seems belong the this xctestrunner: bazelbuild/bazel#8050