browserstack / browserstack-runner Goto Github PK

We on the QUnit team have been discussing the possibility of working with other JS test frameworks, especially those that can be run client-side (e.g. Mocha, Jasmine, Intern, Buster, etc.), to agree upon a common Reporter interface so that we could hopefully share Reporter plugins between testing frameworks.

This would most likely come in the form of:

• a common Reporter API/Interface, e.g.
  • an EventEmitter interface (.on(...)/.off(...)) OR an object with standard "hook" properties
  • maybe a standard-ish way to register a Reporter, e.g. MyLib.addReporter(x), MyLib.reporter = x;, etc.
• a minimum viable set of standardly-named events
  • an associated standard set of data/details provided for each event
• a minimum viable set of standard test status types (e.g. pass, fail, skip, todo, pending, etc.)
• updating all participating test frameworks to support this new common Reporter interface

As a likely consumer of such a shared Reporter interface, we suspect you may be interested in participating in the discussions as well. Such a reporter should eliminate the need for any framework-specific shims/plugins/etc. that you may be employing right now to standardize results.

Would you guys be interested in discussing this further with us? If so, please let me know who I should invite to participate.

Centralized Discussions: https://github.com/js-reporters/js-reporters/issues/

Cross-reference issues:

• qunitjs/qunit#531 (original discussion)
• mochajs/mocha#1326
• jasmine/jasmine#659
• theintern/intern#257
• busterjs/buster#419
• caolan/nodeunit#276
• vowsjs/vows#313

I am unable to figure out how to test urls (rather than files) that are accessed through a proxy. BrowserstackLocal is running, and the proxy information is correct, but test_paths fails unless it is a valid path.

What should a browserstack.json file look like in my case?

I've been working on a similar project for a while. I just tried to publish mine on NPM, but ran into a naming conflict, which is how I found out about this project. I've renamed my project to browserstack-workers (https://github.com/bramstein/browserstack-workers) and published it.

After looking through the code it seems we're taking a very similar approach. Would you be interested in merging the projects or some other form of collaboration? I wasn't really planning on creating and maintaining a project like this, but there was nothing like it around at the time I started working on it, and I really needed it to run my Mocha and Jasmine tests.

Currently each run gets blocked for a few seconds with this:

Downloading BrowserStack Local to '/Users/jza/dev/browserstack-runner/lib/BrowserStackLocal'

I'm not sure if that actually downloads the file each time, but since it won't change inbetween each run, checking if the file actually changed before downloading it may save several seconds.

In latest version, i have to go in and "stop session" for each worker launched if I kill mid test.

Hey,

I have a large suite of tests written in QUnit which take around 5 mins to run in Browserstack (EmberJS integration tests) . Can you tell me what causes the timeout counter to reset? Is there some keepalive mechanism which tells browserstack-runner that your tests are still running or is it simply a timer from the time your tests start until the tests are fully completed?

As my tests take around 5 mins to run and my browserstack-runner timeout is 5 mins I get a lot of timeouts if they run slightly over. I've just updated my timeout to be 7 mins so I don't get the timeouts anymore, but should browserstack-runner know that my tests are still in progress?

Thanks

I got this from a run of a the resizable test suite in jQuery UI:

~/dev/jquery-ui [git:drag-resize-relative-position+?] $ browserstack-runner
Launching 12 workers
[Windows 8, Chrome 32.0] Completed in 3890 milliseconds. 232 of 232 passed, 0 failed.
[Windows 8, Chrome 32.0] Screenshot: https://s3.amazonaws.com/testautomation/205cc92c471c3b479cc6788ab39bcc47b7dd9d3f/js-screenshot-1396366499.png
[Windows 8.1, Firefox 27.0] Completed in 2534 milliseconds. 232 of 232 passed, 0 failed.
[Windows 8.1, Firefox 27.0] Screenshot: https://s3.amazonaws.com/testautomation/411709057a8b4b6f73380570fe2f240965b4b91e/js-screenshot-1396366506.png
[Windows 8, Firefox 28.0] Completed in 2939 milliseconds. 232 of 232 passed, 0 failed.
[Windows 8, Firefox 28.0] Screenshot: https://s3.amazonaws.com/testautomation/4b9a29143a5b2c7741b89ec16f3c65f45742a443/js-screenshot-1396366588.png
[Windows 7, Opera 19.0] Completed in 14457 milliseconds. 232 of 232 passed, 0 failed.
[Windows 7, Opera 19.0] Screenshot: https://s3.amazonaws.com/testautomation/18fa0a004f56c3828e93df42ed1db9aa348ad5ac/js-screenshot-1396366597.png
[OS X Mavericks, Safari 7.0] Completed in 1255 milliseconds. 232 of 232 passed, 0 failed.
[Windows 7, Ie 9.0] Completed in 44 milliseconds. 0 of 0 passed, 0 failed.
[OS X Mavericks, Safari 7.0] Screenshot: https://s3.amazonaws.com/testautomation/c90a6de932909ffe308c0ae62502752a84632650/js-screenshot-1396366638.png
[Windows 7, Ie 9.0] Screenshot: https://s3.amazonaws.com/testautomation/2d29d9b8250688dcdf0c1069a8c0879d7404c697/js-screenshot-1396366641.png
[Windows 8.1, Opera 20.0] Completed in 3967 milliseconds. 232 of 232 passed, 0 failed.
[Windows 8.1, Opera 20.0] Screenshot: https://s3.amazonaws.com/testautomation/77c2599aa28752c52ff4c26100a566b5981c8eac/js-screenshot-1396366653.png
[Tue Apr 01 2014 17:39:19 GMT+0200 (CEST)] [Runner alert] Tests timed out on: Windows 8, Chrome 32.0
[Tue Apr 01 2014 17:39:25 GMT+0200 (CEST)] [Runner alert] Tests timed out on: Windows 8.1, Firefox 27.0
[Tue Apr 01 2014 17:39:32 GMT+0200 (CEST)] [Runner alert] Tests timed out on: Windows 8.1, Opera 20.0
[Tue Apr 01 2014 17:39:42 GMT+0200 (CEST)] [Runner alert] Tests timed out on: OS X Mountain Lion, Safari 6.1
[Tue Apr 01 2014 17:39:46 GMT+0200 (CEST)] [Runner alert] Tests timed out on: OS X Mavericks, Safari 7.0
[Tue Apr 01 2014 17:39:54 GMT+0200 (CEST)] [Runner alert] Tests timed out on: Windows 7, Ie 9.0
[Tue Apr 01 2014 17:40:00 GMT+0200 (CEST)] [Runner alert] Tests timed out on: Windows XP, Ie 8.0
[Tue Apr 01 2014 17:40:07 GMT+0200 (CEST)] [Runner alert] Worker inactive for too long: Windows 8, Ie 10.0
[Tue Apr 01 2014 17:40:15 GMT+0200 (CEST)] [Runner alert] Worker inactive for too long: Windows 7, Ie 11.0
[Tue Apr 01 2014 17:40:23 GMT+0200 (CEST)] [Runner alert] Worker inactive for too long: Windows 8, Chrome 33.0
All tests done, failures: 10.
Exiting

For example, Chrome 32 completed, but also timed out. I can't tell why that would happen. Let me know what other information I could provide to help debug this.

For jQuery projects, we'd like to be able to specify browsers like:

"browsers": [
    "chrome_previous",
    "chrome_current",
    "firefox_previous",
    "firefox_current",
    "ie_6",
    "ie_7",
    "ie_8",
    "ie_9",
    "ie_10",
    "ie_11",
    "opera_12_1",
    "opera_previous",
    "opera_current",
    "safari_5_1",
    "safari_previous",
    "safari_current"
]

"current" is already supported for the browser_version field, but we'd also need "previous", to implement "(Current - 1)".

We don't (want to) care about the platform/OS each browser runs on, but it would be good to be able to mix this compact format with the more verbose one, like this:

"browsers": [
    "chrome_previous",
    "chrome_current",
    {
      "browser": "firefox",
      "browser_version": "15.0",
      "device": null,
      "os": "OS X",
      "os_version": "Snow Leopard"
    },
    "ie_6"
]

This currently makes browserstack-runner completely unusable for me. The error is pretty useless:

~/dev/jquery-ui [git:optimize-testsuites?] $ browserstack-runner

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: connect ETIMEDOUT
    at errnoException (net.js:904:11)
    at Object.afterConnect [as oncomplete] (net.js:895:19)

To get some more output, I've installed the long-stack-traces module and required it in cli.js, then installed my local copy globally (npm install -g .). With that in place, I get a longer stacktrace:

~/dev/jquery-ui [git:optimize-testsuites?] $ browserstack-runner
Launching 2 worker(s) for 2 run(s).
Uncaught Error: connect ETIMEDOUT
    at errnoException (net.js:904:11)
    at Object.afterConnect [as oncomplete] (net.js:895:19)
----------------------------------------
    at EventEmitter.on
    at Socket.<anonymous> (_stream_readable.js:688:33)
    at http.js:1753:12
    at process._tickCallback (node.js:419:13)
----------------------------------------
    at process.nextTick
    at ClientRequest.onSocket (http.js:1730:11)
    at Agent.addRequest (http.js:1269:9)
    at new ClientRequest (http.js:1416:16)
    at Object.request (http.js:1843:10)
    at Version.request (/usr/local/lib/node_modules/browserstack-runner/node_modules/browserstack/lib/browserstack.js:163:18)
    at Version.getWorkers (/usr/local/lib/node_modules/browserstack-runner/node_modules/browserstack/lib/browserstack.js:95:8)
    at [object Object].<anonymous> (/usr/local/lib/node_modules/browserstack-runner/bin/cli.js:195:14)
----------------------------------------
    at Object.setInterval
    at Object.start (/usr/local/lib/node_modules/browserstack-runner/bin/cli.js:194:27)
    at /usr/local/lib/node_modules/browserstack-runner/bin/cli.js:278:22
    at [object Object].<anonymous> (/usr/local/lib/node_modules/browserstack-runner/lib/local.js:62:11)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
----------------------------------------
    at Object.setTimeout
    at Socket.<anonymous> (/usr/local/lib/node_modules/browserstack-runner/lib/local.js:61:9)
    at Socket.emit (events.js:117:20)
    at Socket.<anonymous> (_stream_readable.js:745:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:407:10)
    at emitReadable (_stream_readable.js:403:5)
----------------------------------------
    at EventEmitter.on
    at tunnelLauncher (/usr/local/lib/node_modules/browserstack-runner/lib/local.js:51:23)
    at /usr/local/lib/node_modules/browserstack-runner/lib/local.js:87:7
    at Object.cb [as oncomplete] (fs.js:168:19)
Uncaught undefined

/usr/local/lib/node_modules/browserstack-runner/node_modules/long-stack-traces/lib/long-stack-traces.js:80
                    throw ""; // TODO: throw the original error, or undefined?
                    ^

That last part seems like an issue in the long-stack-traces module, which isn't very useful. Thought at least there are multiple references to locations within this tool itself, which should help. I'm not familiar with that part of the codebase though.

There's currently no license information in the repository. This should be addressed. MIT would be good.

On linux, after installing npm install -g browserstack-runner just fine, I ran into this issue.

$ browserstack-runner

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: EACCES, open '/usr/lib/node_modules/browserstack-runner/lib/BrowserStackLocal'

It happens /usr/lib/node_modules/browserstack-runner is installed with nobody:<my_user_group> ownership. The above error is actually about not being able to open file for write (not for read as I initially thought).

I had to change ownership chown -R ... the above directory with my user credentials in order to make it work.

It would be good to state that in the documentation, or ask for user input, or make the directories group writable to avoid such problem.

Thanks

Can you please tag your releases.

It's impossible to put browserstack.json outside of base directory. :(

When running browserstack-runner, currently it seems to dump two files into the working directory, browserstack-run.pid and BrowserStackLocal. I don't think either should be there.

When specifying multiple files in test_path, the current output will just log Completed messages for each file, e.g. [Windows 8.1, Chrome 33.0] Completed... multiple times, for each file. The number of tests ran might vary, but that's not enough information.

This is less of a problem when something fails, where the test name and stack trace indicate which test suite is affected.

I did a run against a single testsuite in jQuery UI, with this configuration:

{
    "test_framework": "qunit",
    "test_path": ["tests/unit/autocomplete/autocomplete.html"],
    "browsers": [
        "chrome_previous",
        "chrome_current",
        "firefox_previous",
        "firefox_current",
        "opera_previous",
        "opera_current",
        "safari_previous",
        "safari_current",
        "ie_8",
        "ie_9",
        "ie_10",
        "ie_11"
    ]
}

On the first try, seven browsers completed, then the runner died with this (useless) stacktrace:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: getaddrinfo ENOTFOUND
    at errnoException (dns.js:37:11)
    at Object.onanswer [as oncomplete] (dns.js:124:16)

On the next run, one browser completed, the rest timed out with a message like this for each:

[Fri May 02 2014 18:00:20 GMT+0200 (CEST)] [Runner alert] Worker inactive for too long: Windows XP, Chrome 33.0

Looking at the screenshots at browserstack.com/automate, the workers failed to load the page, e.g. https://s3.amazonaws.com/testautomation/1fe1c46220e4d579ed7e80664f97d5cc5fc23614/js-screenshot-1399046509.png - all 11 failed workers show a error message like that.

If you've followed the changes and my comments above, continue here: My changes here break the runner of anything that isn't QUnit. Addressing that requires a lot more work and testing with each supported framework. Since there are no unit tests, testing each changes requires a rather slow run of browserstack-runner. Improving the development workflow should therefore have the highest priority, before anything else is addressed, since that will speed everything else up a lot. I have no idea how to do that, though.

Reference: #35

Calling browserstack-runner should output its own version and exit.

The readme currently doesn't mention any mobile browser configuration at all. Would be good to have a sample and the link to the relevant docs.

I've working on integrating browserstack-runner into QUnit builds on Travis, using simple obfuscation to embed credentials to allow runs against pull requests, without making it too easy to find the credentials. The configuration sets up 14 workers to run 4 test pages, which works fine locally. When running on Travis, each worker runs the first page, then fails to change the URL and eventually times out. I pushed another commit to the same branch to verify the result, and got the same failure - all 14 workers time out after running one page.

Since the same setup works fine locally I have no clue what the issue is. There are no errors displayed in the Travis output and the BrowserStack build page doesn't show anything either. Looking for some ideas how to debug or better yet, solve this.

npm WARN package.json [email protected] No repository field.
npm ERR! Error: shasum check failed for /var/folders/gd/xzyzg3891lvgsgynzl72vlyc0000gn/T/npm-1106-xXXQklPd/registry.npmjs.org/browserstack-runner/-/browserstack-runner-0.2.2.tgz
npm ERR! Expected: 4a50ced875371dbd0906ffea60788abea4a2b5cf
npm ERR! Actual:   c84a458caca706d44407076c7ab5a3ef5cab6626
npm ERR! From:     https://registry.npmjs.org/browserstack-runner/-/browserstack-runner-0.2.2.tgz
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/sha/index.js:38:8
npm ERR!     at ReadStream.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/sha/index.js:85:7)
npm ERR!     at ReadStream.emit (events.js:117:20)
npm ERR!     at _stream_readable.js:943:16
npm ERR!     at process._tickCallback (node.js:419:13)
npm ERR! If you need help, you may report this *entire* log,
npm ERR! including the npm and node versions, at:
npm ERR!     <http://github.com/npm/npm/issues>

npm ERR! System Darwin 14.0.0
npm ERR! command "node" "/usr/local/bin/npm" "install"
npm ERR! cwd /Users/yorkie/workspace/pixbi/duplo
npm ERR! node -v v0.10.32
npm ERR! npm -v 1.4.28
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/yorkie/workspace/pixbi/duplo/npm-debug.log
npm ERR! not ok code 0

as i known if the owner of this repo release with --force? suggest bump version to resolve this issue, thank you :-)

As seen at qunitjs/qunit#773, the Android browsers are always returning a timeout from the runner.

Reference: Modernizr/Modernizr#1247 (comment)

In runner.js, if a worker times out, alertBrowserStack is called, causing the process to exit. This prevents the rest of the workers/vms from attempting to run their tests. We should allow them all to run so we can potentially fix/validate tests manually per platform before running the suite again.

Here's an interesting issue I just ran into trying to use browserstack-runner to run all unit tests in jQuery UI. This gist has my configuration as well as the output I got: https://gist.github.com/jzaefferer/d79717435a5a537b2a67

This run failed pretty quickly, after it tried to launch 264 workers (12 browsers x 22 test suites), with "Error: Cannot queue more than 200 workers". I guess this is an area where the approach taken by TestSwarm, with all its overhead, is somewhat superior, where a run like this needs a minimum amount of 12 workers (maybe a few more if a worker gets killed inbetween or a test has to run again).

Maybe there is some way to reuse workers within browserstack-runner? Or a different approach to this problem?

Currently in server.js, any html file gets injected with the framework patch scripts. This approach causes issues when loading other html files, most likely as iframes.

This gist, running against the sizzle test suite, shows what happens in that case: https://gist.github.com/jzaefferer/401dd88e83443143d351 - its kind of obscure, but the lack of the window object led me to my conclusion above.

I've worked around this by only applying the patching to files called index.html. A proper solution should compare the filename to the one given in config.test_path. I'm not sure how to do that properly since test_path can be an array. Probably compare to each?

Dig into https://api.qunitjs.com/

and get method/class + stacktrace of failure.

While working on #66 I ran into this, crashing the server with this stacktrace:

/Users/jza/dev/browserstack-runner/lib/server.js:150
          logger.info(chalk.red("[%s] Error:"), worker.string, formatTraceback
                                                      ^
TypeError: Cannot read property 'string' of undefined
    at /Users/jza/dev/browserstack-runner/lib/server.js:150:55
    at Array.forEach (native)
    at progressHandler (/Users/jza/dev/browserstack-runner/lib/server.js:149:26)
    at IncomingMessage.<anonymous> (/Users/jza/dev/browserstack-runner/lib/server.js:235:46)
    at IncomingMessage.EventEmitter.emit (events.js:92:17)
    at _stream_readable.js:920:16
    at process._tickCallback (node.js:415:13)

If it helps, here's the full output after I applied the workaround I described in #66: https://gist.github.com/jzaefferer/3a8d2ddd88a886c242d8

Use exe for windows.

If possible.

I’m trying to run a mocha test suite with browserstack-runner, and the suite runs and passes (looking at the screenshots or in interactive sessions).

Nevertheless browserstack-runner does not pick up the results, and the session times out (message is “Timeout (Session timed out because the browser was idle for 300 seconds”).

My configuration file:

{
    "username": "…",
    "key": "…",
    "test_framework": "mocha",
    "test_path": "test.html",
    "browsers": [
        "firefox_latest"
    ]
}

An example:

https://travis-ci.org/twbs/bootstrap/jobs/17672316

I did a run with multiple IE versions. In the output, IE 9 is "Launching", "Launched" and "Acknowledged", but never "Terminated". IE 10 is on "Launching" and "Launched", but doesn't get anywhere else.

The full output is here (see current.sh, the second file in the gist): https://gist.github.com/jzaefferer/3f4a9e516a02494140a7

My configuration looks like this:

{
    "username": "jzaefferer",
    "key": "...",
    "test_framework": "qunit",
    "test_path": ["tests/unit/resizable/resizable.html"],
    "browsers": [
        "chrome_previous",
        "chrome_current",
        "firefox_previous",
        "firefox_current",
        "ie_8",
        "ie_9",
        "ie_10",
        "ie_11"
    ]
}

Capitalization is funky. Should either be "IE" or expanded out to "Internet Explorer".

Using browserstack-runner on a project with a dozen test_path entries and a dozen browsers, I find myself editing the config whenever I need to run a subset of those browsers against a subset of paths. I have to remember reverting those changes to avoid accidentally committing them.

An idea to address that: Two extra CLI options to filter both paths and browsers, so the usage could look something like this:

browserstack-runner -p menu -b chrome

That would match all paths that contain "menu" and all browsers that contain "chrome", based on what's in the configuration.

Would be good to get some input from others, maybe there's a better way to achieve the same thing.

Env vars are mentioned twice, here the user var is wrong: https://github.com/browserstack/browserstack-runner#enviroment-variables

Must be BROWSERSTACK_USERNAME, not BROWSERSTACK_USER.

The browserstck-runner local executable runs fine on my local machine but when I push a build to Travis I just get a notification that the command has exited with a value of 1. I've set the credentials as secure environment variables if that makes any difference. Any ideas as to what might be going on?

To reproduce, start browserstack-runner, then abort immediately using Ctrl+C. This is the output I got, trying to abort twice:

~/dev/qunit [git:master?] $ node_modules/.bin/browserstack-runner
Using config: /Users/jza/dev/qunit/browserstack.json
Launching server on port: 8888
^CExiting
Non existent tunnel
Downloading BrowserStack Local to `/Users/jza/dev/browserstack-runner/lib/BrowserStackLocal`
^CServer already closed
Exiting
Non existent tunnel
fs: missing callback Error: ENOENT, unlink '/Users/jza/dev/qunit/browserstack-run.pid'

A leftover from #97. Needs a separate fix. I don't know what input it deals with. Currently running npm test produces this error:

lib/utils.js: line 11, col 22, Bad or unnecessary escaping.

A unit test for this method would help a lot. The fix itself should be trivial, probably removing one of the slashes. But that needs a way to verify the correctness of the method.

The purpose of the method:

json response (stack trace etc) we send back from browser to runner sometimes have newline and single quote chars which confuses the JSON.parse(). So, escapeSpecialChars handles that for us.

Even with just eight browsers, the console output is pretty chaotic, since workers terminate in random(?) order. I've pasted a recent output into a gist, along with a more compact version, which skips pretty much all the extra logging and only outputs relevant information: https://gist.github.com/jzaefferer/3f4a9e516a02494140a7

For any debugging, of the tests or the tool, a --verbose option would help, outputting as much, or more, as right now. But the default should only log relevant information.

Either way, errors should always be logged. I don't need to know about "launching", "launched", "acknowledged", "terminated", as long as I'm informed if one of those steps goes wrong or doesn't happen.

Create a JSON file that can be loaded in a subsequent run to allow the user to re-run failed tests (could implement in CLI similar to how VIM behaves when it detects a swap file).

$ browserstack-runner
env: node\r: No such file or directory

When I manually convert the file to use UNIX line-endings, everything works:

$ browserstack-runner
Using config: /Users/ron/Projects/bmp-kernel.js/browserstack.json
Configuration file `browserstack.json` is missing.

The git:// type URL (package.json) gets blocked by corporate proxy when performing a npm install. It would be useful to have https URLs instead.

TIA,

This is an odd encoding issue that only appears when running tests through the runner. It happens for my jquery-validation project, specifically in this test. That contains a few lines like this:

    ok( method( "£9", "£"), "Symbol no decimal" );
    ok( method( "£9.9", "£"), "£, one decimal" );
    ok( method( "£9.99", "£"), "£, two decimal" );
    ok( method( "£9.90", "£"), "Valid currency" );

When the file is server by the built-in server (instead of my local nginx), it looks like this:

    ok( method( "£9", "£"), "Symbol no decimal" );
    ok( method( "£9.9", "£"), "£, one decimal" );
    ok( method( "£9.99", "£"), "£, two decimal" );
    ok( method( "£9.90", "£"), "Valid currency" );

That even has the correct character, but for some reason puts the  in front of it.

The only difference I noticed is that nginx servers the js file with Content-Type:application/x-javascript, while the server here uses "text/javascript". I tried changing the content-type, but that didn't help, the result was the same.

I also tried to explicitly set the charset as part of the content-type, forcing it to utf-8. That shouldn't be necessary, since the index.html has a <meta charset="utf-8"> tag, and it didn't help either.

Hoping for some ideas what else could be causing this.

Here's my browserstack.json I used in a checkout of jquery-validation:

{
    "test_framework": "qunit",
    "test_path": ["test/index.html"],
    "browsers": [
        "chrome_latest"
    ]
}

i set a timeout of 20 seconds, yet even though a test completes (in miliseconds), it says it times out (i assume its tracking backed on VM queue time, and not true execution). should only begin timer once VM is running

The README documentation only points a link to the supported browsers without proper documenting how to use them.

Also, I can't find a list of the current support for compact browsers list anywhere.