Git Product home page Git Product logo

orchestrator's Introduction

๐Ÿ”ฅ orchestrator ๐Ÿ”ฅ

orchestrator

Orchestrator executes all cypress specs across n parallel docker containers based on a configuration file.

๐Ÿ˜Ž Orchestrator Tutorial:

1- Cypress parallelization with the Orchestrator โ€” part 1

2- Cypress parallelization with the Orchestrator โ€” part 2 โ€” ShowCase

๐Ÿ˜ Usecases:

Check the following repo as a public use case.

โ™Ÿ๏ธ Orchestrator mechanism:

  • Pares a config file.
  • Create (config.parallelizm * config.browsers.length) containers in parallel.
  • Recursively list all the specs files
  • Split all the specs across all those machines based on their execution time.
  • Collect all the execution json reports from those containers.
  • Down all the running containers.
  • Generate one HTML report that has all specs execution results.
  • Analyse the execution time for each spec.
  • Generate the execution time reports per browser under ExecutionTimeReport dir.
  • In the next run, The orchestrator will split the test cases based on this execution time report to reduce the execution time.

๐Ÿน The Splitting mechanism:

The orchestrator can measure and report the execution time for each spec per browser. It will report it as mochawesome-report/specsExecutionTime-chrome.json file. If you provided this path as specsExecutionTimePath in the next run, The orchestrator will split the specs-based on its execution time to minimize the total execution time ๐Ÿš€.

โŒจ๏ธ Operating Systems:

  • Linux: working out of the box.
  • MacOS: please exeucte brew install coreutils command.
  • Windows 10: fully supported via WSL.

๐Ÿ‘Œ Installation:

  • Install from npm
npm -g install @0xislamtaha/orchestrator
  • Install from Github branch
npm -g install 0xislamtaha/orchestrator

๐Ÿ”‘ Requirements to use orchestrator:

1- docker-compose file with a cypress service. here is an example of it.

version: '3.8'
services:
  cypress-container:
    image: 0xislamtaha/cypress-snapshot-image:latest
    network_mode: "bridge"
    volumes:
      - ./cypress/:/cypress_testing/cypress
      - ./mochawesome-report:/cypress_testing/mochawesome-report
      - /dev/shm:/dev/shm

2- use mochawesome as a reporter in cypress.json, just add the following snippet to your cypress.json.

{
  "reporter": "mochawesome",
  "reporterOptions": {
    "reportDir": "cypress/report/mochawesome-report",
    "overwrite": false,
    "html": false,
    "json": true
  }
}

3- Edit the orchestrator configuration file with your configuration. Here is the description of each configuration option.

- parallelizm:
    description: number of container machines per browser
    type: Integer
    example: 2

- browser:
    description: list of browsers
    type: list
    example: ["chrome", "firefox"]

- timeout:
    description: timeout of each process of cypress 
    type: string
    example: "20m"

- environment:
    description: enviroment variable to be exported 
    type: dict
    example: {"DOCKER_TAG": "master_283"}

- preCommands: 
    description: list of commands to be executed befor the deployment of the cypress containers
    type: list
    example: ["ls -al", "mkdir -p test"],

- dockerComposeOptions:
    description: docker-compose options to be passed to the docker-compose commands
    type: dict
    example: {"-p": "project_name"}

- dockerComposePath:
    description: path to the docker compose file.
    type: string
    example: "/opt/code/github/cypress.docker-compose.yml"

- specsHomePath:
    description: path to the specs dir in the host machine.
    type: string
    example: "/opt/code/github/cypress/integration/"

- specsDockerPath:
    description: path to the specs dir in the cypress container.
    type: string
    example: "/cypress/integration"

- cypressContainerName:
    description: the name of cypress service.
    type: sting
    example: "cypress_service"

- mochawesomeJSONPath:
    description: path to the mochawseom dir in the host machine.
    type: string
    example: "mochawesome-report/*.json"

- reportPath:
    description: path to save the generated HTML report dir.
    type: string
    example: "./"

- specs:
    description: array of specific specs to be executed
    type: array
    example: ["test.js", "test2.js"]

- analyseReport:
    description: boolen value to generate an execution time report. 
    type: boolen
    example: true

๐ŸŽฎ Usage:

  • With your configuration file
npx orchestrator --config "/path/to/orchestrator.json"
  • You can overwrite any configuration param on the fly, simply pass the new configuration as a parameter.
npx orchestrator --config ./src/orchestrator.json --parallelizm 2 --environment '{"DOCKER_TAG":"master_283"}' --browsers "[chrome, firefox]" --specs "[alerts.js, avatar.js]"

๐Ÿ“– Reports:

The orchestrator generates two reports by default:

  • The HTML report under the mochawesome-report dir.
  • The execution time reports per browser und ExecutionTimeReport dir.

๐ŸŽฌ To-Do:

  • list configuration rather than multiple files for multiple test suites.
  • Provide --help option.

License:

The orchestrator is licensed under the MIT license.

orchestrator's People

Contributors

0xislamtaha avatar idbartosz avatar madhan-mettle avatar pshergie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

orchestrator's Issues

Questions

  1. Does this support new cypress version 12. or 13.
  2. I have a requirement where i need to group my spec files and run them in parallel. For example under e2e folder I have group1 folder, group2 and group3. Can i specify to run group1 specs in parallel, and group2 and so in parallel ?
  3. Will this work with cypress.config.js ?
  4. My files are names cy.js instead of spec.js will this work ?

Question on set up

Hello,
I have a docker container where my current tests running. Is this set up works like running multiple containers from my current one to parallize my test files? Just wanted to understand how it works if its a recommended way.

Thanks
Dennis

Can't find tests inside multiple folders

@0xIslamTaha

My folder structure looks like the attached screenshot, above navigation is the folder called integration(src/integration/navigation/tests/)

Screen Shot 2021-10-14 at 8 40 58 AM

I have added the paths like this

"specsHomePath": "src/integration/navigation/tests/**/*.spec.js",
"specsDockerPath": "src/integration/navigation/tests/**/*.spec.js",
"specsHomePath": "src/integration/navigation/tests/**",
"specsDockerPath": "src/integration/navigation/tests/**",
  "specsHomePath": "src/integration/navigation",
  "specsDockerPath": "src/integration/navigation",
  "specsHomePath": "src/integration/navigation/",
  "specsDockerPath": "src/integration/navigation/",

but I'm getting the following error

moisessiles@msilesMacbook project.tests.cypress % npx orchestrator --config orchestrator.json
START ORCHESTRATOR
report does not exist, are you sure there is a json report in undefined path?
timeout --preserve-status 30m docker-compose -f docker-compose.yml run --name container_25678__0 cypress-container bash -c 'exit_code=0; npx cypress run -b chrome --headless --spec src/integration/navigationpage-classes || exit_code=$? ; pkill -9 cypress ; exit $exit_code'
timeout --preserve-status 30m docker-compose -f docker-compose.yml run --name container_30793__1 cypress-container bash -c 'exit_code=0; npx cypress run -b chrome --headless --spec src/integration/navigationtests || exit_code=$? ; pkill -9 cypress ; exit $exit_code'
Creating projecttestscypress_cypress-container_run ... 
Creating projecttestscypress_cypress-container_run ... 
Creating projecttestscypress_cypress-container_run ... done
Creating projecttestscypress_cypress-container_run ... done
Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src/integration/navigationtests

Relative to the project root folder:

/orechestrator_usecase
Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src/integration/navigationpage-classes

Relative to the project root folder:

/orechestrator_usecase
1
1
Removing container_30793__1 ... 
Removing container_25678__0 ... 
Removing container_25678__0 ... done
Removing container_30793__1 ... done
generate the report ....
/Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/mochawesome-merge/lib/index.js:14
    throw new Error(`Pattern ${pattern} matched no report files`)
          ^

Error: Pattern cypress/report/mochawesome-report/*.json matched no report files
    at /Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/mochawesome-merge/lib/index.js:14:11
    at Array.map (<anonymous>)
    at /Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/mochawesome-merge/lib/utils.js:3:46
    at merge (/Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/mochawesome-merge/lib/index.js:82:17)
    at generateReport (/Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/@0xislamtaha/orchestrator/src/orchestrator.js:224:10)
    at afterPromises (/Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/@0xislamtaha/orchestrator/src/orchestrator.js:257:3)
    at /Users/moisessiles/Downloads/Documents.GAP/GAPProjects/myproject/project.tests.cypress/node_modules/@0xislamtaha/orchestrator/src/orchestrator.js:279:7

And I tried with the slash at the end

Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src/integration/navigation/page-classes

Relative to the project root folder:

/orechestrator_usecase
Can't run because no spec files were found.

We searched for any files matching this glob pattern:

src/integration/navigation/tests

Relative to the project root folder:

dockerComposeOptions should be passed to the preCommands too

Description
As we introduced the dockerComposeOption starting from version 2.1.0, It produced some errors in case of there are docker-compose commands in the preCommands list.

AC

  • dockerComposeOptions should be passed to the docker-compose commands in the preCommands list too.

bin/ execution on yarn install is case sensitive

If you check your package.json:

{
"bin": {
   "@0xIslamTaha/orchestrator": "bin/orchestrator"
 }
}

0xIslamTaha is camel case but your package name not, which makes yarn fail when installing.

2/4] ๐Ÿšš  Fetching packages...
error An unexpected error occurred: "ENOENT: no such file or directory, symlink '../../bin/orchestrator' -> '/Users/x/Library/Caches/Yarn/v4/npm-@0xislamtaha-orchestrator-1.1.2-3bd6d08f9632e101e6a7f0dda285d12567d9cfe6/node_modules/@0xislamtaha/orchestrator/.bin/@0xIslamTaha/orchestrator'".

If 1 container finishes its job with some test failures then all other containers stop their jobs

Hi,
First of all thank you very much for sharing this useful tool :)
I'm not sure if it's on purpose, but I've noticed that after 1 of the containers finishes its chunk of the test suite with errors, then all other containers are stopping their work without finishing their part. I think it's caused by using Promise.all in orchestrator.js line 286 - what do you think about using Promise.allSettled instead, so we can wait until all containers finish their jobs and then check if any of promises was rejected?
Thanks,
Rafal

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.