autifyhq / autify-cli Goto Github PK

To make sure our CI/CD integrations work with the new code before release, we'd like to run their integration test inside CLI workflow as well.

An idea is:

1. Clone one integration repository
2. Swap installation or override the autify path
3. Run its integration test with @autifyhq/autify-cli-integration-test

Hello! Great job for creating a client to interact with the autify platform, we are currently using it for our e2e tests in my actual company.

I have a question, sometimes the test cases are running for long time and we don't know the actual status of them. Is it possible to add some features like:

• Execute all the tests in a timeout window
• Show the current status and the step
• Cancel a test case

On Windows:

Run autify update
@autifyhq/autify-cli: Updating CLI... not updatable

With brew on GitHub Actions:

Run autify update
@autifyhq/autify-cli: Updating CLI... fetching manifest

@autifyhq/autify-cli: Updating CLI from 0.10.0-beta.0 to 0.10.0-beta.0-2f002ba (beta)... done
node:internal/process/promises:2[7](https://github.com/autifyhq/autify-cli/actions/runs/3031442476/jobs/4878335436#step:8:8)[9](https://github.com/autifyhq/autify-cli/actions/runs/3031442476/jobs/4878335436#step:8:10)
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open '/Users/runner/.local/share/autify/channel'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/Users/runner/.local/share/autify/channel'
}

Note: autify update works on my macOS laptop (autify is installed via brew).

GitHub Actions example: https://github.com/autifyhq/autify-cli/actions/runs/3031442476

We should not implement such fake logic of autifyconnect here. Instead, it should be provided along with autifyconnect.

Reasons:

• The fake command should reside in autifyconnect's repository to reuse most of the real code.
• Windows invokes a script command and a .exe file differently. Thus, the fake command should be also .exe file same as autifyconnect.exe.

Since Windows doesn't have a concept of "signal", it is very tricky to emulate the simlar behavior to Unix. Also, for some reasons, only Jest invocation can't handle signal properly. So, we gave up and used different behavior when Jest on Windows: https://github.com/autifyhq/autify-cli/blob/main/src/autify/connect/spawnClient.ts#L178

Once we settle a way to terminate autifyconnect without signal, we should quickly switch to it and remove all the hacks for signal:

https://github.com/autifyhq/autify-cli/blob/main/patches/oclif%2B3.1.2.patch#L152-L153

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• fix(deps): update all non major npm update (@oclif/core, @oclif/plugin-not-found, @types/tar)
• chore(deps): update dependency eslint to v9
• chore(deps): update dependency eslint-plugin-unicorn to v52
• chore(deps): update dependency ts-morph to v22
• chore(deps): update mcr.microsoft.com/vscode/devcontainers/typescript-node docker tag to v1
• chore(deps): update typescript-eslint monorepo to v7 (major) (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• fix(deps): update dependency archiver to v7
• fix(deps): update dependency http-proxy-middleware to v3
• fix(deps): update dependency node-emoji to v2 (node-emoji, @types/node-emoji)
• fix(deps): update dependency node-fetch to v3
• fix(deps): update dependency strip-ansi to v7
• fix(deps): update dependency tar to v7
• fix(deps): update dependency xstate to v5
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update node.js to >=20.12.2
• fix(deps): update dependency @oclif/plugin-warn-if-update-available to v3
• fix(deps): update dependency get-port to v7
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• fix(deps): update dependency inquirer to v9 (inquirer, @types/inquirer)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Currently, it relies on zip binary installed on the host machine but it's not universal e.g. Jenkins CI servers don't have it: https://github.com/jenkinsci/autify-plugin/pull/16/checks?check_run_id=8119937975

Use JSZip to generate a zip file by pure JavaScript: https://stuk.github.io/jszip/documentation/examples.html

Note: This opens Windows support but be careful to use UNIX style path inside zip file because Autify for Mobile runs on macOS environment for iOS app.

https://github.com/autifyhq/autify-cli/actions/runs/3323103848/jobs/5493079029

We should skip homebrew publish if there is no change.

When running a test that takes a few minutes, hundreds of lines of logs are output.
We want to consider a way to reduce the amount of log output. For example, it might be a good idea to write a log entry when the status of a test execution changes.

In some cases, Autify users want to automatically retry specific tests a limited number of times (once or twice) until it passes. If Autify CLI can support this (with --retry option along with --wait option), that would be great.

To measure where the CLI is used, we should have configuration for user agent via environment variable like AUTIFY_CLI_USER_AGENT. Then, each official integration overrides (or appends identifier) to the user agent and we can measure their usage.

Currently we provide only 32 bit binary for arm architecture. This won't work with 64 bit architecture.

Example with public.ecr.aws/amazonlinux/amazonlinux:2 docker image:

bash-4.2# curl https://autify-cli-assets.s3.amazonaws.com/autify-cli/channels/stable/install-standalone.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2046  100  2046    0     0  10459      0 --:--:-- --:--:-- --:--:-- 10492
Installing CLI from https://autify-cli-assets.s3.amazonaws.com/autify-cli/channels/stable/autify-linux-arm.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 31.5M  100 31.5M    0     0   792k      0  0:00:40  0:00:40 --:--:-- 1257k
qemu-arm: Could not open '/lib/ld-linux-armhf.so.3': No such file or directory
autify installed to /usr/local/bin/autify
Error: node is not installed.

bash-4.2# autify --version
Error: node is not installed.

(In the installer, it removes node when node -v doesn't work and delegates system-wide node but no node is installed, hence the last error).

We should provide linux-arm64 binary because Node.JS officially provide such binary for node as well.

High level commands such as web test run use stdout for informing to the human user. Therefore, it is not easy to pass the re-usable data to other process via pipe.

We could use stderr for logging and stdout for machine-readable information but such convention will break easily. Also, machine-readable information is not needed for human.

Thus, we should put such data on data directory like ~/.data/autify/output/web-test-run-<timestamp>-<pid>.json then print the location as a log. The other process like CI/CD integration can parse the file location from log easily and get the output in json format.