Git Product home page Git Product logo

arc-xcode-test-engine's Introduction

xcode-test-engine for arc

xcode-test-engine is a test engine for use with Phabricator's arc command line tool.

Features

xcode-test-engine presently works well with projects that have a single root workspace and a single unit test target.

Supports arc unit:

~ $ arc unit
   PASS   42ms★  MyTests::Test1
   PASS    3ms★  MyTests::Test2
   PASS    2ms★  MyTests::Test3
   PASS    2ms★  MyTests::Test4
   PASS    1ms★  MyTests::Test5

And arc unit --coverage:

~ $ arc unit --coverage
COVERAGE REPORT
      5%     src/Widget.m
     18%     src/SomeDirectory/AnotherWidget.m
     69%     src/SomeDirectory/WidgetBuilder.m
     80%     src/Controller/WidgetViewController.m
     93%     src/Controller/WidgetTableViewController.m

Installation

Project-specific

Add this repository as a git submodule.

git submodule init
git submodule add <url for this repo>

Your .arcconfig should list xcode-test-engine in the load configuration:

{
  "load": [
    "path/to/arc-xcode-test-engine"
  ]
}

Global

Clone this repository to the same directory where arcanist and libphutil are globally located. Your directory structure will look like so:

arcanist/
libphutil/
arc-xcode-test-engine/

Your .arcconfig should list arc-xcode-test-engine in the load configuration (without a path):

{
  "load": [
    "arc-xcode-test-engine"
  ]
}

Usage

Create a .arcunit file in the root of your project and add the following content:

{
  "engines": {
    "xcode": {
      "type": "xcode-test-engine",
      "include": "(\\.(m|h|mm|swift)$)",
      "exclude": "(/Pods/)"
    }
  }
}

Feel free to change the include/exclude regexes to suit your project's needs.

Now modify your .arcconfig file by adding the following configuration:

{
  "unit.xcode": {
    "build": {
      "workspace": "path/to/Workspace.xcworkspace",
      "scheme": "UnitTestsScheme",
      "configuration": "Debug",
      "destination": "platform=iOS Simulator,name=iPhone 6S"
    },
    "coverage": {
      "product": "SomeFramework.framework/SomeFramework"
    }
  }
}

Configuration options

Build configuration options

Any value provided to "build" will be passed along to xcodebuild as a flag.

"unit.xcode": {
  "build": { ... }
}

Picking the coverage product

Provide the path to the product for which coverage should be calculated. If building a library/framework this might be the framework binary product.

"unit.xcode": {
  "coverage": {
    "product": " ... "
  }
}

Pre-build commands

Some projects require a pre-build script that runs before the xcodebuild command.

"unit.xcode": {
  "pre-build": "execute this command before the build step"
}

For example, CocoaPods projects may need to be re-generated.

  "pre-build": "pod install --project-directory=path/ --no-repo-update"

Supporting multiple test engines

We support Arcanist's multi test engine extension. Visit the [extension]((https://github.com/tagview/arcanist-extensions#multi_test_engine) page to learn more about it.

Your .arcconfig file will have something similar to the following configuration:

{
  "unit.engine": "MultiTestEngine",
  "unit.engine.multi-test.engines": [
    {
      "engine": "XcodeUnitTestEngine",
      "unit.engine.xcode.config": {
        "build": {
          "workspace": "YourWorkSpace.xcworkspace",
          "scheme": "Scheme1",
          "configuration": "Debug",
          "destination": "platform=iOS Simulator,name=iPhone SE"
        },
        "coverage": {
          "product": "SomeFramework.framework/SomeFramework"
        }
      }
    },
    {
      "engine": "XcodeUnitTestEngine",
      "unit.engine.xcode.config": {
        "build": {
          "workspace": "YourWorkSpace.xcworkspace",
          "scheme": "AnotherScheme",
          "configuration": "Debug"
        }
      }
    }
  ]
}

Viewing coverage results in Phabricator

Coverage will appear for affected source files in Side-by-Side mode as a colored bar.

License

Licensed under the Apache 2.0 license. See LICENSE for details.

arc-xcode-test-engine's People

Contributors

hossamghareeb avatar jverkoey avatar kevm avatar mazyod avatar shepting avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

arc-xcode-test-engine's Issues

Support multiple products for code coverage

Hello again!

In our project, we actually have a workspace with many different subproject .. Then, we use an aggregate target to run all the tests. It would be great if we could specify "products" instead of just "product" under code coverage to pull the data from all the available targets.

Thanks!

No possible to set a right coverage product path

    "coverage": {
      "product": "MSAppModuleKit-Example/libMSAppModuleKit.a"
    }

then I traced by arc unit --trace
You can see the product path and Coverage.profdata path alway has a MSAppModuleKit-Example(target name) . maybe you are not using Xcode7.3, it works differently.

MSAppModuleKit itself is managed by CocoaPods, I think it wouldn't affect arc unit

xcrun llvm-cov show -use-color=false -instr-profile "/Users/jenkins/Library/Developer/Xcode/DerivedData/MSAppModuleKit-dluocdmaoluajyeejotbfynnkjpa/Build/Intermediates/CodeCoverage/MSAppModuleKit-Example/Coverage.profdata" "/Users/jenkins/Library/Developer/Xcode/DerivedData/MSAppModuleKit-dluocdmaoluajyeejotbfynnkjpa/Build/Intermediates/CodeCoverage/MSAppModuleKit-Example/Products/Debug-iphonesimulator/MSAppModuleKit-Example/libMSAppModuleKit.a"

Code coverage reports aren't being parsed properly

Hey, thanks for making this project!

After successfully integrating unit test runner with arcanist, I started to dabble a bit with code coverage, and it seems that Xcode code coverage format might have changed.

Even though when I run xcrun llvm-cov I can clearly see something like

   35|       |    /* optional */
   36|      0|    public var timeout: TimeInterval? { return nil }
   37|      0|    public var body: [String: Any]? { return nil }
   38|      0|    public var additionalHeaders: [String: String?]? { return nil }
   39|      0|    public var requestModifiers: [RequestModifier] { return [] }
   40|       |    
   41|       |    
   42|     52|    public var fullpath: [String] {
   43|     52|        return Self.api.basePath + path
   44|     52|    }

All files show up as 100% covered. Looking into the library, I think this project is parsing only the first part on the left, and not the second part. I'd send a PR if I had experience with PHP .. Sadly, I don't.

Arc unit catches logs as broken tests

If there is a piece of code that prints a log containing error: anywhere inside it the arc unit output will report that the tests are BROKEN.
error log

I'd be happy to submit a Pull Request if you could point me to the likely culprit.

I originally just tried to change this check if (strpos($line, 'error:') !== false) to if (strpos($line, ': error:') !== false) (adding the extra colon and space) but it didn't seem to catch this issue. https://github.com/google/arc-xcode-test-engine/blob/develop/parser/XcodeTestResultParser.php#L66

arc unit doesn't work as expected from sub-directories

   BROKEN  Xcode test engine
xcodebuild: error: 'example/Catalog.xcworkspace' does not exist.

Likely due to the fact that we're not using absolute paths.

The current solution is to run arc unit from the root of the git repo.

Finding the product for coverage easily breaks

Hey,

In the engine code, there is a find command that can easily break if the framework was already built for another target:

$future = new ExecFuture("find %C | grep %C", $objroot, $this->coverage['product']);

I'm trying to find a good fix for this... perhaps by checking the returned paths for CodeCoverage

Using test engine with Quick testing library gets "Unable to parse line" error

It seems that the test engine is looking for results with test methods to start with the word test. This is not required by Quick specs. I created a quick (pun intended) example repo with specs that pass in Xcode and at the command line but have weird parse errors.

> arc unit
Unable to parse line:
Test Case '-[QuickArcExampleTests.CalculatorSpec calculator__should_add_two_numbers]' passed (0.004 seconds).
Unable to parse line:
Test Case '-[QuickArcExampleTests.CalculatorSpec calculator__should_subtract_two_numbers]' passed (0.000 seconds).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.