knapsackpro / knapsack-pro-cypress Goto Github PK

Knapsack Pro for Cypress.io test runner in JavaScript. It splits tests in dynamic way across parallel CI nodes to get faster CI build time.

Home Page: https://knapsackpro.com

License: MIT License

JavaScript 50.75% TypeScript 45.53% Shell 3.72%

knapsack-pro cypress javascript testing cypress-io parallelization typescript parallelism

knapsack-pro-cypress's People

Contributors

Stargazers

Watchers

Forkers

lienlotus neswork-z native-finance jagcf goodproblems isabella232 renehernandez

knapsack-pro-cypress's Issues

False positive (zero exit code) returned when versions mismatch

We had an incident on our CI when we cleared cache on buildkite and got a version mismatch between cypress and npm.

The error we saw looks like this:

Log looked like this:

+ npm bin
--
  | + /app/node_modules/.bin/knapsack-pro-cypress --spec cypress/integration/**/*.spec.js --reporter junit --reporter-options mochaFile=artifacts/cypress/results-[hash].xml
  |  
  | 2020-09-24T23:05:26.483Z [@knapsack-pro/core] info: POST https://api.knapsackpro.com/v1/queues/queue
  | DEPRECATION WARNING: `preview_path` will be removed in v3.0.0. Use `preview_paths` instead. (called from <top (required)> at /app/config/environment.rb:5)
  | WARNING: Stimulus Reflex requires caching to be enabled. Caching allows the session to be modified during ActionCable requests.
  | To enable caching in development, run:
  | rails dev:cache
  | DEPRECATION WARNING: Initialization autoloaded the constants API, API::Auth, API::BaseController, API::GraphqlController, Search::Query, Middleware, Middleware::VersionHeader, ActionText::ContentHelper, and ActionText::TagHelper.
  |  
  | Being able to do this is deprecated. Autoloading during initialization is going
  | to be an error condition in future versions of Rails.
  |  
  | Reloading does not reboot the application, and therefore code executed during
  | initialization does not run again. So, if you reload API, for example,
  | the expected changes won't be reflected in that stale Module object.
  |  
  | These autoloaded constants have been unloaded.
  |  
  | Please, check the "Autoloading and Reloading Constants" guide for solutions.
  | (called from <top (required)> at /app/config/environment.rb:5)
  |  
  | 2020-09-24T23:05:27.197Z [@knapsack-pro/core] info: 200 OK
  |  
  | Request ID:
  | 68593e37-feba-46e5-b71d-20ec94d029ed
  |  
  | Response body:
  | {
  | queue_name: '1335:0e2799c3a307ced616ceaf2a6dbceb6e',
  | build_subset_id: null,
  | test_files: [
  | {
  | path: 'cypress/integration/seller/spa_customer/view-customers.spec.js',
  | time_execution: 70.619
  | }
  | ]
  | }
  | It looks like this is your first time using Cypress: 5.2.0
  |  
  | ✖  Verifying Cypress can run /root/.cache/Cypress/5.2.0/Cypress
  |  
  | (node:3551) UnhandledPromiseRejectionWarning: Error: Cypress failed to start.
  |  
  | This is usually caused by a missing library or dependency.
  |  
  | The error below should indicate which dependency is missing.
  |  
  | https://on.cypress.io/required-dependencies
  |  
  | If you are using Docker, we provide containers with all required dependencies installed.
  |  
  | ----------
  |  
  | /root/.cache/Cypress/5.2.0/Cypress/Cypress: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory
  |  
  | ----------
  |  
  | Platform: linux (Debian - 10)
  | Cypress Version: 5.2.0
  | at /app/node_modules/@knapsack-pro/cypress/node_modules/cypress/lib/errors.js:393:15
  | at tryCatcher (/app/node_modules/bluebird/js/release/util.js:16:23)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:547:31)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | at Promise._settlePromise0 (/app/node_modules/bluebird/js/release/promise.js:649:10)
  | at Promise._settlePromises (/app/node_modules/bluebird/js/release/promise.js:729:18)
  | at Promise._fulfill (/app/node_modules/bluebird/js/release/promise.js:673:18)
  | at Promise._resolveCallback (/app/node_modules/bluebird/js/release/promise.js:466:57)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:559:17)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | at Promise._settlePromise0 (/app/node_modules/bluebird/js/release/promise.js:649:10)
  | at Promise._settlePromises (/app/node_modules/bluebird/js/release/promise.js:729:18)
  | at Promise._fulfill (/app/node_modules/bluebird/js/release/promise.js:673:18)
  | at Promise._resolveCallback (/app/node_modules/bluebird/js/release/promise.js:466:57)
  | at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:559:17)
  | at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:604:18)
  | (node:3551) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
  | (node:3551) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  | + TESTS_EXIT_STATUS=0

Our configuration looked like this:

set -x

export RAILS_ENV=test \
       KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
       RAILS_ENABLE_TEST_LOG=true \
       CYPRESS=true \
       UV_THREADPOOL_SIZE=1024; # Option for node, helps with local DNS requests and cypress

# run migrations
bundle exec rake db:create
bundle exec rake db:migrate

# Run puma in background
bundle exec bin/puma -p 3001 -e test &

# Run knapsack-pro-cypress with junit reporter
$(npm bin)/knapsack-pro-cypress --spec "cypress/integration/**/*.spec.js" \
            --reporter "junit" \
            --reporter-options "mochaFile=artifacts/cypress/results-[hash].xml"

# Save knapsack-pro-cypress exit status
TESTS_EXIT_STATUS=$?

# Upload artifacts
buildkite-agent artifact upload "artifacts/**/*.xml"
buildkite-agent artifact upload "cypress/screenshots/**/*"
buildkite-agent artifact upload "cypress/videos/**/*"
buildkite-agent artifact upload "log/*"

# return exit status of cypress
exit $TESTS_EXIT_STATUS

The line where we execute knapsack should actually be using yarn instead of npm. So I think this had a lot to do with it.

We were also using the version of knapsack pro before you introduced the peer dependency of cypress (#32).

The problem to investigate is why did knapsack pro return a 0 exit code where clearly there is a failure? Is it a problem with the knapsack runner? Or is it a problem with cypress not catching uncaught exceptions?

If it is knapsack, my best guess is that that the queue mode simply didn't see any jobs and didn't take into account this particular error scenario. Thus it reported it as a success.

Can't force disable Cypress recording

I previously had my test runner set up to record my tests and send them to Cypress.io, and now I'm attempting to use this library instead. I have it up and running and it seems great so far, except I get this message in between each test.

This project has been configured to record runs on our Dashboard.

It currently has the projectId: xxxxx

You also provided your Record Key, but you did not pass the --record flag.

This run will not be recorded.

If you meant to have this run recorded please additionally pass this flag.

  cypress run --record

If you don't want to record these runs, you can silence this warning:

  cypress run --record false

https://on.cypress.io/recording-project-runs

Following its advice, I instead ran knapsack-pro-cypress run --record false, but now Cypress refuses to run at all, saying:

You passed the --ci-build-id, --group, --tag, or --parallel flag without also passing the --record flag.

The --group flag you passed was: fff90030-b205-42da-aadb-d9f53b8120b0

These flags can only be used when recording to the Cypress Dashboard service.

https://on.cypress.io/record-params-without-recording

This seems to be caused by this line added in #47 which adds a group ID whenever the --record flag is passed.

Granted, I believe it's only doing this because there is a CYPRESS_RECORD_KEY set in my environment, and that removing that would also solve the issue for me personally. However, I can imagine users potentially wanting to be able to disable test recording sometimes without having to change their environment variables.

I propose changing this to only add the group ID if the --record argument isn't false.

Update for Cypress 12

It's possible to update cypress dependency to cypress current version? 12

Missing build id environment variable for github actions

In the record section setting of the readme(https://github.com/KnapsackPro/knapsack-pro-cypress#how-to-record-ci-builds-in-cypress-dashboard) the environment variable to use for the BUILD_ID from github actions is missing.

Does anyone know what the ENV variable on github actions is and if it can be added there?

Rerunning pipelines may not work with --project flag.

Running into a small issue with using this package, specifically in regards to using the --project flag on cypress and using the KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true env variable. Bit of a text dump here, sorry about that 😅 . For context, I'm using Bitbucket Pipelines, and the config file looks something like

      - step:
          name: Cypress 1/N
          script:
          - export KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
          - export KNAPSACK_PRO_TEST_FILE_PATTERN=subdir1/subdir2/cypress/integration/**/*.js
          - <All other knapsack pro env variables set appropriately>
          - yarn install --frozen-lockfile
          - yarn cypress install
          - yarn run knapsack-pro-cypress --project subdir1/subdir2

On first run, this works great. However, when rerunning a failed pipeline, instead of passing in the file paths as subdir1/subdir2/cypress/integration/<sometestfile>, cypress seems to be instead called with cypress/integration/<sometestfile>, which results in an error that looks like:

Can't run because no spec files were found.
We searched for any files matching this glob pattern:
../../cypress/integration/<sometestfile>.

I did a bit of digging about why this might be, and in the knapsack-pro-cypress.ts, line 61 you have:

  const recordedTestFiles: TestFile[] = tests.map((test: any) => ({
    path: test.spec.relative,
    time_execution:
      // test.stats.wallClockDuration - Cypress 3.x and 4.x
      // test.stats.duration - Cypress 5.x
      (test.stats.wallClockDuration || test.stats.duration) / 1000,
  }));

Based on what cypress says in their code here: https://github.com/cypress-io/cypress/blob/4d76049d2c419d77968beafcdda45585744fc542/cli/types/cypress-npm-api.d.ts, test.spec.relative returns paths relative to the project root. So, because I set the project root to be subdir1/subdir2, the path returned by test.spec.relative will be cypress/integration/<sometestfile>. Not going to pretend I know how the internals of the API (or client for that matter) work, but I'm assuming that this is later used to deal with rerunning a pipeline when the KNAPSACK_PRO_FIXED_QUEUE_SPLIT env var is set to true.

So, the file path that cypress is called with on the first and 2nd run will be different, and while the first run works great, any reruns of the pipeline will break.

I think the solution to this could either be as simple as changing test.spec.relative to test.spec.absolute, or by just using the file paths that are originally provided instead of the file paths that cypress.run returns. FWIW, I think the first solution I proposed might not work for all CI runners, for example if the absolute path would change on rerun (haven't seen this for bitbucket but it could be the case for other CI runners), but I think the second solution would probably work?

More than happy to put up a PR for this with either proposed solution, just let me know!

Thanks!!

Pass CLI arguments to cypress.run

If you would like to pass argument like --record to cypress then normally you would do:

npx cypress run --record

At the moment we cannot do the same with @knapsack-pro/cypress as it is not passing the CLI argument to cypress.run (see source code of our lib).

# --record will be ignored
$(npm bin)/knapsack-pro-cypress --record

Cypress docs says that cypress.run accepts the arguments:
https://docs.cypress.io/guides/guides/module-api.html#cypress-run

Note some of them are camelCase like ciBuildId but CLI arguments are in format --ci-build-id:
https://docs.cypress.io/guides/guides/command-line.html#cypress-run

TODO

check if there is a way to pass CLI arguments down to cypress.run function.
alternative solution could be to create ENV variables for each argument and then if KNAPSACK_PRO_CYPRESS_RECORD=true then set record: true in cypress.run function etc. This might be silly idea I guess.

Side problem

If you would use record: true argument in cypress.run function then batch of tests fetched from Knapsack Pro API Queue is executed and shown on Cypress Dashboard as single CI build.
So when you consume Queue then multiple CI builds are shown in Cypress Dashboard for each batch of tests fetched from the Queue.

I tried to group the batches with ciBuildId that requires to use group argument at the same time.

  const { runs: tests, totalFailed } = await cypress.run({
    spec: testFilePaths,
    record: true,
    ciBuildId: 'some-build-id-here',
    group: '@knapsack-pro/cypress',
  });

But this does not work as paid plan is required on Cypress to use grouping. We could add this to FAQ once the root issue with passing CLI arguments is solved.

Bitbucket Guideline - Issues to know exactly how I have to install this by using bitbucket

I'm pretty new to this software and I'm looking to move forward with your tool to install it on my current cypress project and my bitbucket pipeline, but I'm not sure how I have to configure my YAML file. Do you have any guides for bitbucket or any examples of how the YAML file should look?

Feature: Support exclusion pattern with KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN env var

As the title say, leverage KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN in knapsack-pro-cypress

[Security] Upgrade axios dependency in `@knapsack-pro/core` to 0.21.2 or later

There's a known security vulnerability in axios. Upgrade axios to version 0.21.2 or later.

GHSA-cph5-m8f7-6c5x

> yarn why axios -R
   └─ @knapsack-pro/cypress@npm:4.5.0 [dc3fc] (via npm:^4.5.0 [dc3fc])
      └─ @knapsack-pro/core@npm:3.1.1 (via npm:^3.1.1)
         └─ axios@npm:0.21.1 (via npm:0.21.1)

Update cypress dependency

Can we change the dependency of cypress to be

"cypress": ">= 4.9.0"

So that it works for 4.10, 4.11 and so on.

As a workaround I did:

// package.json
{
  "devDependencies": {
    "cypress": "^4.11.0",
   } ,
  "resolutions": {
    "cypress": "^4.11.0"
   }
}

Retries with Cypress 5.x and BuildKite

Running CI tests in BuildKite only end up with the first attempt running without following retries:

Running:  path/to/integrationTest.spec.ts                                                               (1 of 2)
--
  |  
  |  # What shows up
  | Description of spec
  | (Attempt 1 of 2) displays direct report users and links user through to request feedback for them
  | 1) displays direct report users and links user through to request feedback for them
  |  # omitted but shows rest of test (with successful specs) without re-attempting
  |  
  | 9 passing (24s)
  | 1 failing

I've looked through the docs, but currently have not identified if there is something I am missing that is required for these retries to work.

More information:

Running tests on BuildKite in parallel using the approach shown in RSpec examples.