webdriverio-community / wdio-teamcity-reporter Goto Github PK

When having a test, that fails in the beforeEach block, it is not reported as failed by the TeamCity reporter. The TeamCity UI shows the test as "run & passed", which is not correct and very misleading.

Here is a test to reproduce and the output I get. The test is reported correctly by the spec reporter.

Test

describe('Before each', function() {
    beforeEach(function() {
        browser.url('https://www.google.com');
        throw new Error("Simulate test failing in beforeEach");
    });

    it('test should fail with error in "beforeEach"', function() {
        expect(browser).toHaveUrlContaining('google');
    });
});

TeamCity reporter output:

[0-0] ##teamcity[testSuiteStarted name='Before each' flowId='3e837eaa83a635949c0abb4a8b45d276/0-0']
[0-0] ##teamcity[testStarted name='test should fail with error in "beforeEach"' captureStandardOutput='false' flowId='3e837eaa83a635949c0abb4a8b45d276/0-0']
[0-0] ##teamcity[testSuiteFinished name='Before each' flowId='3e837eaa83a635949c0abb4a8b45d276/0-0']

Spec reporter output:

[chrome 86.0.4240.183 mac os x #0-0] Spec: ../test-desktop/specs/before-each.spec.ts
[chrome 86.0.4240.183 mac os x #0-0] Running: chrome (v86.0.4240.183) on mac os x
[chrome 86.0.4240.183 mac os x #0-0] Session ID: 3e837eaa83a635949c0abb4a8b45d276
[chrome 86.0.4240.183 mac os x #0-0]
[chrome 86.0.4240.183 mac os x #0-0] Before each
[chrome 86.0.4240.183 mac os x #0-0]    ? test should fail with error in "beforeEach"
[chrome 86.0.4240.183 mac os x #0-0]    ✖ "before each" hook for "test should fail with error in "beforeEach""
[chrome 86.0.4240.183 mac os x #0-0]
[chrome 86.0.4240.183 mac os x #0-0] 1 failing (2.9s)
[chrome 86.0.4240.183 mac os x #0-0]
[chrome 86.0.4240.183 mac os x #0-0] 1) Before each "before each" hook for "test should fail with error in "beforeEach""
[chrome 86.0.4240.183 mac os x #0-0] Simulate test failing in beforeEach
[chrome 86.0.4240.183 mac os x #0-0] Error: Simulate test failing in beforeEach
[chrome 86.0.4240.183 mac os x #0-0]     at Context.<anonymous> (../test-desktop/specs/before-each.spec.ts:4:15)

Hey @sullenor,

first off, thank you for creating this plugin and extending the WebdriverIO ecosystem!

We have been looking into successful WDIO plugins like this one as we want to ensure that its development can be continued even if the original maintainer decides to move on. Therefore on behalf of the WebdriverIO community I would like to ask if you would be interested to move this repository into the WebdriverIO Community organisation. It would allow us to help maintain this package, automate a few things like releases and allow others to contribute to it and make the package even better. You would of course continue to have full access to it.

What do you think? Please let me know if you have any questions.

Hi folks, The tests in CI and locally. What's our strategy to maintain the repo and how can we help?

Skipped tests are logged via ##teamcity[testFailed ...] but this is not correct.

test.skip() should mark test as ignored instead of failed.

As part of the v8 effort we are currently migrating over to ESM as many packages will stop support CJS in the future. This means we have to transition all plugins to ESM as well, ideally with continuous support for CJS.

node version: v18.12.1
wdio version: 8.0.15

I recently upgraded the node and libraries on our CI and getting the following error (when using teamcity reporter):

[0-0] 2022-12-29T10:06:50.764Z ERROR @wdio/local-runner: Failed launching test session: Error: Couldn't initialise "wdio-teamcity-reporter".
[0-0] Error [ERR_REQUIRE_ESM]: require() of ES Module /usr/lib/node_modules/@wdio/reporter/build/index.js from /usr/lib/node_modules/wdio-teamcity-reporter/index.js not supported.
[0-0] Instead change the require of /usr/lib/node_modules/@wdio/reporter/build/index.js in /usr/lib/node_modules/wdio-teamcity-reporter/index.js to a dynamic import() which is available in all CommonJS modules.
[0-0]     at Object.<anonymous> (/usr/lib/node_modules/wdio-teamcity-reporter/index.js:1:22)
[0-0]     at async Promise.all (index 0)
[0-0]     at async safeImport (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/utils/build/utils.js:188:21)
[0-0]     at async initialisePlugin (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/utils/build/initialisePlugin.js:32:20)
[0-0]     at async BaseReporter._loadReporter (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/reporter.js:178:30)
[0-0]     at async Promise.all (index 0)
[0-0]     at async BaseReporter.initReporters (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/reporter.js:23:27)
[0-0]     at async Runner.run (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/index.js:68:9)
[0-0]     at safeImport (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/utils/build/utils.js:210:15)
[0-0]     at async initialisePlugin (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/utils/build/initialisePlugin.js:32:20)
[0-0]     at async BaseReporter._loadReporter (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/reporter.js:178:30)
[0-0]     at async Promise.all (index 0)
[0-0]     at async BaseReporter.initReporters (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/reporter.js:23:27)
[0-0]     at async Runner.run (file:///usr/lib/node_modules/@wdio/local-runner/node_modules/@wdio/runner/build/index.js:68:9)
[0-0] FAILED

I tried replacing require with import statements and that giving not a module error..
Am I missing any other dependency here?

Hola 👋

WebdriverIO released v5 around Christmas, see also this blogpost. To make this reporter usable with the new version there are some small adjustments necessary. There are some minor changes that need to be made. You can find a general overview here. Most important ones to point out are:

• set "@wdio/cli": "^5.0.0" as peerDependency
• let the reporter extend from @wdio/reporter
• change the constructor so it calls super()
  Note: if your reporter is designed to report to stdout have it the following way:

  /**
   * make reporter to write to output stream by default
   */
  options = Object.assign(options, { stdout: true });
  super(options)

  if your reporter should rather report to a file per default just call:

  super(options)

• all file or stdout messages need to be propagated by this.write("my log message")

Feel free to drop by the webdriverio/v5 support channel in case you need support.

Thanks so much for contributing to the WebdriverIO community. We really appreciate it 🙏 !

I have added the following to my wdio config file...

reporters: ['teamcity'], reporterOptions: { captureStandardOutput: true, flowId: true, message: '[browser]/[title]' }

However, the browser name is never added to the test results. I also tried updating some of the other options, such as the flowId and captureStandardOutput, but it looks like the defaults never change. I am using wdio-teamcity-reporter v3.0.0 and WDIO v.7

When installing the wdio-teamcity-reporter using WebdriverIO 7, I'm getting the following error:

npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR! 
npm ERR! While resolving: @ecgdk/[email protected]
npm ERR! Found: @wdio/[email protected]
npm ERR! node_modules/@wdio/cli
npm ERR!   @wdio/cli@"7.4.5" from the root project
npm ERR! 
npm ERR! Could not resolve dependency:
npm ERR! peer @wdio/cli@"^5.0.0" from [email protected]
npm ERR! node_modules/wdio-teamcity-reporter
npm ERR!   wdio-teamcity-reporter@"2.3.0" from the root project
npm ERR! 
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

When installing it with the --legacy-peer-deps option, it works as expected and also the reporter itself works fine with WebdriverIO 7.

The error occures, because the peer of @wdio/cli@"^5.0.0" does not include @wdio/[email protected]. It would be nice to be able to install it without the error and a --legacy-peer-deps/--force tag.

(I'm using Node v15.14.0 and npm v7.9.0)

Since Webdriver.IO v5 and v6 there is not reporter output until all suites are finished.

After some investigation, I have figured out that the reason is this line of code in WDIOReporter:
this.outputStream = this.options.stdout || !this.options.logFile ? options.writeStream : _fs.default.createWriteStream(this.options.logFile);

Since reporter uses write() method and stdout flag is true looks like writeStream is missed.

According to Reporting Tests chapter: https://confluence.jetbrains.com/display/TCD65/Build+Script+Interaction+with+TeamCity

Current version has dependencies fixed for WebdriverIO V7, so it won't install when using V8.
An update for V8 would be appreciated.

As a user, I would like to suggest to use same naming conventions as other webdriver-io packages since v6 to something like @wdio/teamcity-reporter

Description
When you have specFileRetries set in wdio, if the attempt fails but passes in a retry, TC will report the status as failed.

[0-0] RUNNING in chrome - test.spec.js
[0-0] AssertionError in "wdio with retries should fail the first time and pass the second"
expected undefined to equal 'test'
[0-0] RETRYING in chrome - test.spec.js
[0-0] RUNNING in chrome - test.spec.js
[0-0] PASSED in chrome - test.spec.js (1 retries)

 "teamcity" Reporter:
##teamcity[testSuiteStarted flowId='9f87a7ad1732e5139b0e6b2ab519724f' name='wdio with retries']
##teamcity[testStarted flowId='9f87a7ad1732e5139b0e6b2ab519724f' name='should fail the first time and pass the second' captureStandardOutput='false']     
##teamcity[testFailed flowId='9f87a7ad1732e5139b0e6b2ab519724f' name='should fail the first time and pass the second' message='']
##teamcity[testFinished flowId='9f87a7ad1732e5139b0e6b2ab519724f' name='should fail the first time and pass the second']
##teamcity[testSuiteFinished flowId='9f87a7ad1732e5139b0e6b2ab519724f' name='wdio with retries']
##teamcity[testSuiteStarted flowId='dc2318c8593b42e5da1f1a5f032ba1cc' name='wdio with retries']
##teamcity[testStarted flowId='dc2318c8593b42e5da1f1a5f032ba1cc' name='should fail the first time and pass the second' captureStandardOutput='false']     
##teamcity[testFinished flowId='dc2318c8593b42e5da1f1a5f032ba1cc' name='should fail the first time and pass the second']
##teamcity[testSuiteFinished flowId='dc2318c8593b42e5da1f1a5f032ba1cc' name='wdio with retries']

Spec Files:      1 passed, 1 retries, 1 total (100% completed) in 00:00:13

It doesn't look like Teamcity will ever support retry functionality:
https://youtrack.jetbrains.com/issue/TW-48674?_ga=2.41109922.1467143230.1580470363-826807720.1535725114
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/115000640030-TeamCity-Cucumber-3-release-of-retry-only-counts-failed-version-

Proposed Solution
Instead of reporting the results as they happen, bundle them up and output them once in the onRunnerEnd hook. Each time a result is received, overwrite the previously saved value so only the latest result is ever reported on. Could make this behaviour opt in from the options, or do it automatically based on the value set in the config.

If you run your tests in parallel in different browsers, multiple tests of the same name show up. For example, with webdriverIO you can define multiple browsers in your config file to run a suite across multiple browsers. Unless you modify the name of the test in the reporter to explicitly state what browser it is run on, you will get two tests with the same name.
If a test fails in one browser, it is more difficult to determine which browser had an issue. You are also not able to easily compare how tests are performing in different browsers.
My team wrote our own TeamCity reporter that we have been using for a few weeks now and our solution was to append the browserName to the test name in the reporter.
Something along the lines of test.runner[Object.getOwnPropertyNames(test.runner)].browserName should get you the name of the browser.

It also looks like the tests get nested inside each other during parallel execution which may mess up timings.