open-rpc / test-coverage Goto Github PK

View Code? Open in Web Editor NEW

3.0 2.0 0.0 1.24 MB

Test your OpenRPC Document against an API.

License: Apache License 2.0

JavaScript 4.45% TypeScript 95.55%

openrpc-document test-coverage

test-coverage's Introduction

OpenRPC Test Coverage

This tool is meant to help you test your OpenRPC Document against an API.

Need help or have a question? Join us on Discord!

The Problem

Even after writing your OpenRPC Document, you want to test that the OpenRPC Document does represent an actual API.

Solution

generate fake data for the OpenRPC Documents method parameters schemas
use example pairings when available
use the servers[] defined in the OpenRPC Document to make the JSON-RPC API call
report back the coverage results
show errors and the result to help fix inconsistencies

Extending with a `Rule`

Rules are a way to extend the test coverage tool to check for specific things when calling your JSON-RPC API. For example, you may want to check that some fields are always present in the result. You can write a rule to check for that.

A rule is a class that extends the Rule class and implements the getCalls function. The getCalls function should return an array of calls to make to the JSON-RPC API. The Rule class will then make the calls and check the results. It checks the results with the validateCall function, it expects that you mutate the call.valid property to true if the call is valid, and false if the call is invalid.

Lastly there are lifecycle events that you can hook into to do things like setup and teardown. The onBegin function is called before any calls are made, and the onEnd function is called after all calls are made. There are also beforeRequest, afterRequest, and afterResponse lifecycle functions that are called before the request is made, after the request is made, and after the response is received respectively. See the Rule interface for more information.

Custom Reporters

You can write custom reporters to output the results of the test coverage tool in a different format. A reporter is a class that extends the Reporter class and implements the lifecycle functions. The Reporter class will call the lifecycle functions at the appropriate time. The onBegin function is called before any calls are made, and the onEnd function is called after all calls are made. There are also onTestBegin and onTestEnd lifecycle functions that are called before and after each test respectively. See the Reporter interface for more information.

Installation:


npm install -g @open-rpc/test-coverage

Usage:

open-rpc-test-coverage -s https://raw.githubusercontent.com/open-rpc/examples/master/service-descriptions/simple-math-openrpc.json --transport=http --reporters=console --skip=addition

Screenshot

console reporter

test-coverage's People

Contributors

Stargazers

Watchers

test-coverage's Issues

add dependabot

this will help with dependency management

Tool should return non-zero exit code if any tests were failed

Currently open-rpc-test-coverage returns zero even if tests were failed, which makes it hard to use it in automated environments

How to reproduce

Grab any example of schema and server with some failing test

open-rpc-test-coverage -s ./openrpc.json --transport=http --reporter=console
if [[ $? -eq 0 ]]; then echo "Should be non-zero"; fi

Flag to skip tests if there's no example defined

Currently test-coverage always tries to run tests even if there's no examples being defined for a method, and in case of stateful apps like signer -- auto-generated random input doesn't really work.

So it would be awesome to have a --examples flag that forces the tool to only test those methods that have example-result pairs defined.

Dependabot couldn't authenticate with registry.npmjs.org

Dependabot couldn't authenticate with registry.npmjs.org.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Validate return values

Describe the issue
Currently it only creates parameters from the openrpc.json and is a Success when it returns results with no errors.

Describe the solution
It should only have success it the result values match the openrpc.json result schema

Run methods with incorrect params, expect correct errors to be returned

As an additional testing type we should generate data that specifically does not match the methods parameters.

Missing `lodash` dep

jade-signer-rpc|feat/openrpc-integration-tests⚡ ⇒ open-rpc-test-coverage -s ./openrpc.json --transport=http --reporter=console
internal/modules/cjs/loader.js:628
    throw err;
    ^

Error: Cannot find module 'lodash'
Require stack:
- /usr/local/lib/node_modules/@open-rpc/test-coverage/node_modules/@open-rpc/schema-utils-js/build/src/generate-method-id.js
- /usr/local/lib/node_modules/@open-rpc/test-coverage/node_modules/@open-rpc/schema-utils-js/build/src/index.js
- /usr/local/lib/node_modules/@open-rpc/test-coverage/bin/cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:625:15)
    at Function.Module._load (internal/modules/cjs/loader.js:527:27)
    at Module.require (internal/modules/cjs/loader.js:683:19)
    at require (internal/modules/cjs/helpers.js:16:16)
    at Object.<anonymous> (/usr/local/lib/node_modules/@open-rpc/test-coverage/node_modules/@open-rpc/schema-utils-js/build/src/generate-method-id.js:3:16)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:643:32)
    at Function.Module._load (internal/modules/cjs/loader.js:556:12)
    at Module.require (internal/modules/cjs/loader.js:683:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/local/lib/node_modules/@open-rpc/test-coverage/node_modules/@open-rpc/schema-utils-js/build/src/generate-method-id.js',
    '/usr/local/lib/node_modules/@open-rpc/test-coverage/node_modules/@open-rpc/schema-utils-js/build/src/index.js',
    '/usr/local/lib/node_modules/@open-rpc/test-coverage/bin/cli.js'
  ]
}

Option to validate schema of method results, and not compare values

Why?

In some cases the output of a method may be randomized, hence it's impossible to match the result value one-to-one with an examples defined in the openrpc.json schema. One way around it would be optionally dropping the value check and performing just schema check instead (to at least ensure the correct object is returned).

Possible solution

We could add the --only-validate-result-schema flag (or a shorter one) that

if passed without a parameter, skips value matching for all tests
if passed with a comma-separated list of methods, skips value matching for the provided methods

missing commander dep

jade-signer-rpc|feat/openrpc-integration-tests:zap: ⇒ open-rpc-test-coverage -s ./openrpc.json --transport=http --reporter=console 2| pbcopy
internal/modules/cjs/loader.js:628
    throw err;
    ^

Error: Cannot find module 'commander'
Require stack:
- /usr/local/lib/node_modules/@open-rpc/test-coverage/bin/cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:625:15)
    at Function.Module._load (internal/modules/cjs/loader.js:527:27)
    at Module.require (internal/modules/cjs/loader.js:683:19)
    at require (internal/modules/cjs/helpers.js:16:16)
    at Object.<anonymous> (/usr/local/lib/node_modules/@open-rpc/test-coverage/bin/cli.js:2:17)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:643:32)
    at Function.Module._load (internal/modules/cjs/loader.js:556:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:839:10) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/usr/local/lib/node_modules/@open-rpc/test-coverage/bin/cli.js' ]
}