Git Product home page Git Product logo

oatts's Introduction

OpenAPI Test Templates (oatts)

Generate basic unit test scaffolding for your OpenAPI specification.

Disclaimer

This is not an officially supported Google product.

oatts is based off of the swagger-test-templates module and the lessons learned during its development.

This is a work in progress.

Goal

The goal of oatts is to provide a standalone module for generating Node.js unit test code scaffolding based on a given OpenAPI specification.

The hope is that by providing such a tool, API developers will be encouraged to test the contract between their spec and backend early, often and continuously as the project grows.

Usage

There are a couple ways to use oatts.

Module

Install via npm

npm install --save oatts

Then use it in code

var oatts = require('oatts');

var options = {
    // see "Options" section below for available options
};

var tests = oatts.generate('/path/to/openapi.yaml', options);

console.log(tests)

Command line interface

Install globally via npm

npm install -g oatts

Then use in your command line

> oatts generate --help

  Usage: generate [options]

  generate unit test scaffolding for a given OpenAPI/Swagger Spec

  Options:

    -h, --help                             output usage information
    --host <host>                          target hostname to use in test generation
    -p, --paths <paths>                    comma separated list of paths to generate tests for
    -e, --samples                          generate sample response bodies rather than schema, if applicable
    -s, --spec <spec>                      path to the target OpenAPI/Swagger spec document to consume
    -w, --writeTo <writeTo>                directory to write the generated tests to file
    -c, --consumes <consumes>              consumes/content-type to use in request when applicable to the API resource
    -o, --produces <produces>              produces/accept to use in request when applicable to the API resource
    -u, --customValues <customValues>      custom request values to be used in generation; takes precedent over a customValuesFile
    --customValuesFile <customValuesFile>  path to JSON file with custom request values to be used in generation
    -m, --scheme <scheme>                  which scheme to use if multiple are present in spec
    -t --templates <templateDir>           path to direcotry of custom templates
    -S, --status-codes <statusCodes>       comma separated list of status codes to explicity generate tests for

> oatts generate -s ./path/to/openapi.yaml -w ./output/dir
> ls ./output/dir
pet-test.js  pet-{petId}-uploadImage-test.js  user-test.js 
. . .

Using the result

The resulting test files are built using the mocha testing framework and chakram API testing framework. Thus, you will need both of these dependencies installed in order to run your newly generated tests.

After installing these, you can run the tests with mocha:

# start your API server to test against!!
> mocha --recursive <test directory>


    tests for /goodbye
        tests for get
            ✓ should respond 200 for "Success" (57ms)

    tests for /hello
        tests for get
            ✓ should respond 200 for "Success"


    2 passing (82ms)

Custom Values

Custom values can be supplied through both the command line and a JSON file. The in-line, command line supplied JSON will take precedent.

An example custom values JSON file can be found here.

Custom Templates

Custom templates can be supplied via the templates option. The directory pointed to by the option must contain 4 Handlebars templates named the same way as those found in ./templates.

  • topLevel.handlebars: the top level template for a single test file
  • pathLevel.handlebars: the path level template, usually the beginning of a test suite for a specific path
  • operationLevel.handlebars: the operation level template, for a single operation test suite
  • transactionLevel.handlebars: the template for a single transaction, or a single response code's unit test

The data available to be used in the templates is specified in the ProcessedSpec type.

There are also a few helpers available to be used in the Handlebars templates, which can be found in the templateHelpers documentation namespace. Use the default templates as examples of how to use them.

Options

The following options can be passed to the generation function, some/all are exposed in the accompanying CLI:

Name CLI Flag Default Required Description
spec --spec -s n/a true Path to a swagger.yaml or openapi.yaml
host --host spec.host false Hostname to put in test requests; defaults to host in given spec
paths --paths -p spec.paths false API paths to generate tests for; defaults to all paths in given spec
samples --samples -e false false Toggle generating sample responses for assertion
writeTo --writeTo -w n/a false Directory to write generated tests to; will create the directory if it doesn't exist
consumes --consumes -c operation.consumes[0] | | spec.conumes[0] false Consumes header to use in a request when applicable
produces --produces -o operation.produces[0] | | spec.produces[0] false Produces header to use in a request when applicable
customValues --customValues -u n/a false Values to be populated in requests where specified; overrides customValuesFile
customValuesFile --customValuesFile n/a false Path to a JSON file with values to populate in requests
scheme --scheme -m spec.schemes[0] false Override for multiple scheme present in a spec
templates --templates -t './templates' false Path to directory containing custom templates
statusCodes --status-codes -S operation.responses false comma separated list of status codes to explicity generate tests for
jsonRefs n/a false (See JsonRefs~JsonRefsOptions)
customFormats n/a false The key/value pair of custom formats (The keys are the format name and the values are async functions. See ZSchema Custom Formats)
customFormatGenerators n/a false The key/value pair of custom format generators (The keys are the format name and the values are functions. See json-schema-mocker Custom Format)
customValidators n/a false The custom validators. See DocumentValidationFunction

Testing

To test this module simply use the npm script

npm test

Discussion

If you have a question or a topic you'd like to discuss, please feel free to open a discussion on our Google Group oatts-users.

Contributing

Contributors are welcome! Please see CONTRIBUTING.md.

Copyright

Copyright 2018, Google Inc.

License

See LICENSE file.

oatts's People

Contributors

codyreichert avatar dependabot[bot] avatar incrediblehannes avatar jochenrousse avatar mteres avatar noahdietz avatar paulway avatar prabhukathiresan avatar radeklat avatar uzlopak avatar yaroslavyaroslavtsev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oatts's Issues

Json body in request when testing POST

This is probably possible, but I'm not able to figure out how. I am trying to have the generated test send a simple json request-body when testing a POST to my api. Like this:

        it('should respond 201 for "Null response"', function() {
            var response = request('post', 'http://localhost:8080/pets', {
                'body': {"name": "Jeffrey", "species": "dude"},
                'headers': {"Content-Type":"application/json","Accept":"application/json"},
                'time': true
            });

I have tried various ways in my customValuesFile, but none is working.

How should I go about to achieve this?

Generate tests for specific set of operations for a particular path

Hi, I would like to open a discussion regarding a more granular control over generated tests. More specifically, the possibility of generating only specific operations for a particular path.

For example, provided /pet path contains GET, POST and PUT operations, I would prefer to generate tests only for GET and PUT.

Maybe such functionality already exists and I just missed it?

support for multiple status code entries in custom values generating multiple tests

When supplying custom values for a specific path > operation > status code, supplying an array of objects for the status rather than just a single one could generate multiple unit tests within the operation level test block.

an example custom values blob would be as follows:

...
"200": [
       {
        "header": {
          "header-test": "a"
        },
        "query": {
          "status": [
            "notanenum3"
          ]
        },
        "response": [
          {
            "name": "sparky",
            "photoUrls": [
              "url1",
              "url2",
              "url3"
            ]
          },
          {
            "name": "champ",
            "photoUrls": [
              "url4",
              "url5",
              "url6"
            ]
          }
        ]
      },
      {
        "header": {
          "header-test": "b"
        },
        "query": {
          "status": [
            "otherstatus"
          ]
        },
        "response": [
          {
            "name": "fido",
            "photoUrls": [
              "url789"
            ]
          }
        ]
      }
],
...

Resulting in two tests (it clauses) being generated under the same operation-level block for the same status code, but with different request/response payloads.

Nested definitions are not well recognized

When I tried to generate the test, I receive the following error:

node_modules/.bin/oatts generate -s src/api/swagger.yaml -w test

TypeError: Cannot read property 'memberships' of undefined at reduce (/home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1691:35) at /home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1752:23 at Array.forEach (<anonymous>) at reduce (/home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1750:24) at /home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1752:23 at Array.forEach (<anonymous>) at reduce (/home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1750:24) at /home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1752:23 at Array.forEach (<anonymous>) at reduce (/home/pato/Workspace/Clients/TLC/apis/global/api/node_modules/json-schema-faker/dist/index.js:1750:24)

When i delete the references, it runs ok!

My yaml file is the following:

definitions: carriers: type: "object" properties: id: type: "integer" format: "int64" name: type: "string" deleted: type: "boolean" locations: type: "object" properties: id: type: "integer" format: "int64" name: type: "string" deleted: type: "boolean" accounts: type: "object" properties: id: type: "integer" format: "int64" name: type: "string" deleted: type: "boolean" memberships: type: "array" items: $ref: "#/definitions/memberships" accountEmails: type: "array" items: $ref: "#/definitions/accountEmails" journeys: type: "object" properties: id: type: "integer" format: "int64" idCarrier: type: "integer" format: "int64" carrier: $ref: "#/definitions/carriers" idLocationStart: type: "integer" format: "int64" locationStart: $ref: "#/definitions/locations" idLocationEnd: type: "integer" format: "int64" locationEnd: $ref: "#/definitions/locations" startDate: type: "string" format: "date" endDate: type: "string" format: "date" deleted: type: "boolean" deliveries: type: "array" items: $ref: "#/definitions/journeyDeliveries" deliveries: type: "object" properties: id: type: "integer" format: "int64" idAccount: type: "integer" format: "int64" account: $ref: "#/definitions/accounts" idLocation: type: "integer" format: "int64" location: $ref: "#/definitions/locations" issuedAt: type: "string" # format: "date" issuedBy: type: "string" reference: type: "string" quantity: type: "integer" format: "int64" weight: type: "number" format: "float" volume: type: "number" format: "float" value: type: "number" format: "float" comment: type: "string" deleted: type: "boolean" handedOvers: type: "array" items: $ref: "#/definitions/handedOvers" journeys: type: "array" items: $ref: "#/definitions/journeyDeliveries" handedOvers: type: "object" properties: id: type: "integer" format: "int64" idDelivery: type: "integer" format: "int64" delivery: $ref: "#/definitions/deliveries" idLocation: type: "integer" format: "int64" location: $ref: "#/definitions/locations" issuedAt: type: "string" format: "date-time" issuedTo: type: "string" quantity: type: "integer" format: "int64" weight: type: "number" format: "float" volume: type: "number" format: "float" value: type: "number" format: "float" comment: type: "string" deleted: type: "boolean" journeyDeliveries: type: "object" properties: id: type: "integer" format: "int64" idJourney: type: "integer" format: "int64" journey: $ref: "#/definitions/journeys" idDelivery: type: "integer" format: "int64" delivery: $ref: "#/definitions/deliveries" quantity: type: "integer" format: "int64" weight: type: "number" format: "float" volume: type: "number" format: "float" value: type: "number" format: "float" deleted: type: "boolean" users: type: "object" properties: id: type: "integer" format: "int64" name: type: "string" identification: type: "string" email: type: "string" emailVerified: type: "boolean" password: type: "string" format: "password" passwordReset: type: "boolean" role: $ref: "#/definitions/usersRole" deleted: type: "boolean" memberships: type: "array" items: $ref: "#/definitions/memberships" usersRole: type: "string" enum: - "COMMON" - "VIEWER" - "EDITOR" - "ADMIN" loginObject: type: "object" properties: user: $ref: "#/definitions/users" token: type: "string" expDate: type: "string" format: "date-time" memberships: type: "object" properties: id: type: "integer" format: "int64" idUser: type: "integer" format: "int64" user: $ref: "#/definitions/users" idAccount: type: "integer" format: "int64" account: $ref: "#/definitions/accounts" receiveEmail: type: "boolean" role: $ref: '#/definitions/membershipsRoles' deleted: type: "boolean" membershipsRoles: type: "string" enum: - "VIEWER" - "EDITOR" - "ADMIN" accountEmails: type: "object" properties: id: type: "integer" format: "int64" idAccount: type: "integer" format: "int64" account: $ref: "#/definitions/accounts" name: type: "string" email: type: "string" deleted: type: "boolean"

Keep package up-to-date in npm

We are having to npm install straight from git... which isn't terrible... but it's also not the documented install approach and I just want to make sure people are installing the latest versions of this useful tool ;)

An updated package hasn't been published to npm in 6 months: https://www.npmjs.com/package/oatts -- also, the version in npm doesn't align with the release versions here on GitHub.

Could we possibly wire up a travis-ci job (or similar) to auto-push package updates?

Values in header will not be overwritten when specified in custom values file

I was trying to generate tests for the 415 unsupported mediatype status code.
In order to specify an mediatype that is not supported I used a custom value file containing the following definition:
"415": { "header": { "Accept": "undefined" }
The Accept field will be overwritten by the test generation. It works other keys that won't be automatically generated.

I already fixed this bug and I will contribute a PR.

Using uuid format in parametesr

In my specification I using parameters:

        - in: header
          name: My-Header
          schema:
            type: string
            format: uuid

And for this format I got an error:

$ sudo oatts generate -s ./openapi.yaml -w ./tests/
Error: unknown generator for "uuid" in /
    at generate (/usr/local/lib/node_modules/oatts/node_modules/json-schema-faker/lib/jsf.js:74:13)
    at Parameter.getSample (/usr/local/lib/node_modules/oatts/node_modules/sway/lib/types/parameter.js:94:39)
    at /usr/local/lib/node_modules/oatts/lib/process.js:275:90
    at Array.forEach (<anonymous>)
    at processParams (/usr/local/lib/node_modules/oatts/lib/process.js:259:24)
    at /usr/local/lib/node_modules/oatts/lib/process.js:202:22
    at Array.forEach (<anonymous>)
    at processTransactions (/usr/local/lib/node_modules/oatts/lib/process.js:196:29)
    at /usr/local/lib/node_modules/oat

OAS v3.x.x Support

This project has a dependency on sway for parsing and traversing a given OAS document. Until sway supports v3, oatts cannot.

When sway releases support for v3, this issue will be used to track the work for supporting it in oatts.

Oatts throws an error while running mocha command

Hi,

As given in the github documentation(https://github.com/google/oatts), following command generates the .js file for mentioned specification.
oatts generate -s ./path/to/openapi.yaml -w ./output/dir

For next steps, while running them using mocha (mocha --recursive (test directory)), this gives me following error for JS file.
"Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves."

Is there a way to resolve this? Which rules(validation) does Oatts uses?

Thanks.

Optional parameters are always included

There is no way to specify that an optional parameter is not supplied. This can change the test results.

For instance, a view `/songs/' might have an optional parameter 'genre'. If songs are found with that genre, then they are listed, otherwise all songs are listed. If no songs have the given genre, a 404 is returned. If oatts picks a random genre name, this most often results in a 404, which gives a false negative.

Dynamically populate path parameters from example property

I am unable to dynamically generate the in path parameters for the tests using the example property. Instead, it is populated with lorem ipsum. (see example below)

I did notice you can get the path parameters to dynamically populate by using an enum property. There also appears to be the option of using a custom values json file, but this defeats the purpose of it being dynamically generated directly from the swagger definition file.

Example input

   /api/v1/avatar/generate/from/{encodedAvatar}/me.png: 
    parameters:
      - in: path
        name: encodedAvatar
        required: true
        schema:
          type: string
          example: ZXllczkvbm9zZTgvbW91dGgxL2Y2YmU1ZC9tb25zdGVy
    get:
      summary: Get the png image from the provided encoded avatar details
      tags:
        - Avatar
      responses:
        '200':
          description: OK
        '401':
          $ref: '#/components/responses/UnauthorizedError'

OUTPUT

'use strict';
var mocha = require('mocha');
var chakram = require('chakram');
var request = chakram.request;
var expect = chakram.expect;

describe('tests for /api/v1/avatar/generate/from/{encodedAvatar}/me.png', function() {
    describe('tests for get', function() {
        it('should respond 200 for "OK"', function() {
            var response = request('get', 'http://localhost:3000/api/v1/avatar/generate/from/ut ad elit/me.png', { 
                'time': true
            });

            expect(response).to.have.status(200)
            return chakram.wait();
        });    
    });
});

Expose handlebars object to add custom helpers

My sugestion is to expose the oatts handlebars objects (inside compile.js) so we could add custom handlebars helpers to process some custom templates with them.

Do you think it's a good idea?

Empty request body for openapi 3.0 specs

Hi,

I've been attempted to use oatts in a project to generate high level tests for an API with an openapi-3.0 specification. From reading the handlebar templates and running the test cases it appears there is support to populate a request body using an openapi spec (see attached)

Though the generated test (see below) does not contain the body parameter (please note, I've used customValues already to set the bearer token successfully)

describe('tests for /example', function() {
    describe('tests for post', function() {
        it('should respond 201 for "Object created"', function() {
            var response = request('post', 'http://localhost:5000/example', { 
                'time': true
            });

            expect(response).to.have.status(201)
            return chakram.wait();
        });    
    });
});

Can you please advise whether this should be possible? Alternatively I'm prepared to dig in and add the functionality.

example-spec.yaml.txt

spoofed request parameters aren't URL safe

I noticed in the output given in #1 that the path param was replaced with a value that wasn't URL safe. A string with a space in it.

When spoofing path params, we need to ensure that they are safe to have in the URL path.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.