Git Product home page Git Product logo

mltap's Introduction

mltap Build Status

mltap is a unit testing framework for MarkLogic. It includes a test harness for running tests in Server-Side JavaScript in MarkLogic, an assertion library, and a Node.js-based command-line wrapper.

mltap (partially) implements the tape API, a popular Node.js test framework. This allows you to run the same tests in Node.js, modern browsers, and MarkLogic with no modifications to the test code.

mltap produces standard TAP output so you can use it in other tools that can consume TAP, for example, for continuous integration, IDE integration, or reporting.

Getting Started

First, install the libraries and the command-line interface,

git clone https://github.com/jmakeig/mltap.git && cd mltap
npm install
npm run setup
npm test
npm install --global . # optional

This installs the the command-line interface on your $PATH and configures MarkLogic with the necessary library modules and security settings. You’ll only need to do this once.

Then from within the project you’d like to test,

mltap test/*.test.sjs

Behind the scenes, mltap will run each of local the *.test.sjs modules against the target MarkLogic JavaScript environemnt. mltap collates the results and produces a summary of the passed and failed tests in TAP format to stdout. You can use any TAP consumer to further process the output. For example, tap-diff pretty prints TAP on the command-line.

npm install -g tap-diff
mltap test/*.test.sjs | tap-diff

Example Test and TAP Output

'use strict';

const test = isMarkLogic() ? 
               require('/mltap/test') : 
               require('tape');

test('Throws an error after some assertions pass', (assert) => {
  assert.true(true, 'true is true');
  throw new Error('Test error');
  assert.true(!false, 'not false is also true');
  assert.end();
});

test('This test has a fulfilled plan', (assert) => {
  assert.plan(2);
  assert.true('asdf'.length === 4,  'asdf is length 4');
  assert.true('asdfs'.length === 5, 'asdfs is length 5');
});

test('This test has an unfulfilled plan', (assert) => {
  assert.plan(25);
  assert.true('asdf'.length === 4,  'asdf is length 4');
  assert.true('asdfs'.length === 5, 'asdfs is length 5');
});


function isMarkLogic() {try {return xdmp && cts;} catch(e) {return false;}}
TAP version 13
# Throws an error after some assertions pass
ok 1 true is true
not ok 2 Error: Test error
  ---
    operator: error
    actual: undefined
    at: Test.impl (/test/test.test.sjs:9:9)
    stack: |-
      Error: Test error
          at Test.impl (/test/test.test.sjs:9:9)
          at Test.run (/mltap/test.js:76:12)
          at Object.run (/mltap/test.js:49:28)
  ...
# This test has a fulfilled plan
ok 3 asdf is length 4
ok 4 asdfs is length 5
# This test has an unfulfilled plan
ok 5 asdf is length 4
ok 6 asdfs is length 5
not ok 7 Planned for 25 assertions, got 2
  ---
    operator: ok
    expected: 25
    actual: 2
    at: Test.run (/mltap/test.js:81:10)
  ...

1..7

Development Set-Up

git clone https://github.com/jmakeig/mltap.git && cd mltap
npm install
# Copies the MarkLogic library modules to $ML_HOME/Modules/mltap
# You’ll need to run this anytime the MarkLogic modules change
# This should theoretically work with a modules database in the future
npm config  # Requires MarkLogic running (defaults to admin:admin@localhost:8000)
npm test

For VSCode

Install Node.js TypeScript types to enable code completion,

npm install -g typings
typings install dt~node --global

Set up a task to run tests via npm. The most important part is that the command return the TAP output unaldulterated, i.e. not reformatted or rewritten, as most console pretty printers will do. Anything that you pipe the TAP output into, such as tap-notify, must be a transparent pass-through. The problemMatcher below relies on the specific error reporting format of mltap, which isn’s wholly standardized in TAP.

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "0.1.0",
  "command": "npm",
  "isShellCommand": true,
  "showOutput": "silent",
  "suppressTaskName": true,
  "tasks": [
    {
      "taskName": "test",
      "args": [
        "run",
        "test"
      ],
      "problemMatcher": {
        "owner": "javascript",
        "fileLocation": [
          "relative",
          "${workspaceRoot}"
        ],
        "pattern": {
          "regexp": "^\\s+at: (.+) \\((.+):(\\d+):(\\d+)\\)",
          "file": 2,
          "line": 3,
          "column": 4,
          "message": 1
        }
      }
    }
  ]
}

mltap's People

Contributors

jmakeig avatar

Watchers

 avatar

mltap's Issues

Relative paths on `at` values

The at tests should just test the end of the string, not the whole path.

For example, at.endsWith('/throws.test.js:21:14)') below,

not ok 33 Failure location
  ---
    operator: equal
    expected: |-
      '/throws.test.sjs:20:24'
    actual: |-
      '/marklogic/test/throws.test.sjs:20:24'
    at: remote.then.then.tap (/Users/jmakeig/Workspaces/mltap/node/test/throws.test.js:21:14)
  ...

Harness runs previous test again

For each module the harness runs, it runs those tests and the tests of any previous modules. The cache of tests to run needs to be cleared between modules.

Eval bootstrap is too permissive

The eval call requires the tester role to have eval privileges. This is way too permissive.

const bootstrap = `require('/mltap/mltap')(tests, root, 0);`;

client.eval(bootstrap,

The runner should instead be an invoke of a module that calls the bootstrap:

client.invoke('/mltap/runner.sjs', 

JavaScript linting

  • Node vs. MarkLogic: Node is ES6, MarkLogic is ES5
  • MarkLogic globals, e.g. xdmp, cts
  • Shared base so each doesn’t have to re-declare the basics
  • Whole lot of clean-up, especially indentation and JSDoc

Make npm install script cross-platform (starting with Linux and OS X)

The npm install script runs after an npm install mltap. It needs to do two things:

  • Set up the MarkLogic security settings (config)
  • Deploy the modules to a modules database or the Modules directory of the MarkLogic installation (deploy)

File system deployment requires sudo on Linux (and probably admin on Windows) because MarkLogic is installed into /opt/MarkLogic. On OS X, MarkLogic is installed in the user’s home directory and thus doesn’t require sudo.

Note that on Travis sudo npm … won’t work because npm isn't in the root user’s PATH, because Travis is using nvm.

TAP 13 serialization

Serialize harness output as TAP version 13.

TAP version 13
# Markdown generation
ok 1 Not unescaped
ok 2 Escaped
ok 3 Not unescaped
ok 4 Escaped
ok 5 Total lines
# escapeForHTML
ok 6 should be equal
ok 7 should be equal
ok 8 should be equal
# canary
ok 9 Indentity transform should be the same
# Workspace
ok 10 Total lines
ok 11 should be equal
# Workspace not found
ok 12 Throws a SelectorError
# Multiple workspaces
ok 13 Total lines across template and two examples
# Many workspaces
ok 14 Sum of 100 iterations
# XPath select query
ok 15 Selects one query
ok 16 Node data is string
# XPath selects nothing
ok 17 Intentionally empty selection

1..17
# tests 17
# pass  17

# ok

Indented writer

The indentation code in the TAP serialization is ugly. Create a class that can push/pop indentations while writing.

Non-zero exit code when tests fail

In jmakeig/iterable:

> $(npm bin)/mltap test/*.sjs
…
ok 57     at Array.map (native)
ok 58     at IterableArray.map (/iterable-array.js:37:39)

1..58
# tests 58
# pass 57
# fail 1

# not ok
> echo $?
0

Harness summary report

Tape adds a summary to the end of its report

# tests X
# passed Y
# failed Z

# [not] ok

Deploy dependencies from npm

I'm currently copying dependencies manually into the lib directory. This mixes concerns and will not scale. (How will I accommodate updates, for example?)

Figure out a way to deploy dependent libs from the local node_modules directory:

  1. Install: npm install --save pretty-format (same for update)
  2. Deploy: npm run deploy cherry-picks stuff from node_modules and puts it in the correct place in $ML_HOME/Modules.

Probably shouldn’t try to solve the multiple versions problem, so store the dependencies under a vendor folder in $ML_HOME/Modules/mltap.

This works for the case of simple libraries with zero dependency. Re-writing dependency paths to convert from npm/Node to MarkLogic module resolution is out of scope.

Parameterize MarkLogic connection settings

mltap runs the tests by eval’ing a call against the Node.js eval endpoint. The eval imports the /mltap/mltap library (which is why deploy writes this to $ML_HOME/Modules). The eval passes in the list of test module paths to run and root and modules paths against which to invoke each test. Thus the connection to MarkLogic is via the default REST API (localhost:8000). The other context is set at runtime (hence, the security settings).

  • User (tester): The user running the tests. config installs a default user with the correct roles, but in the real world this would be an actual user with the mltap-tester role.
  • Password (tester): Figure out how to prompt for this in TTY mode and pass it in securely on non-interactive mode
  • Database (Documents): The content database against which to run the tests. (mltap.js will need to be updated to accommodate this.)
  • Host (localhost): Only needed if you want to run against a custom REST API app server (not typical)
  • Port (8000): See Host above

How would this work with a Modules database, rather than the file system?

Proper security

A regular user must not have blanket invoke or change modules privileges.

  • mltap-internal role that mltap.runner() gets amped to
  • mltap-tester role that only has xdbc-eval in order to bootstrap
  • mltap-tester privilege that gets tested in the eval bootstrap
  • ignoreAmps in the xdmp.invoke that runs the actual test

Better explain the features and benefits

Update the README to better reflect why this is useful. Don’t assume the reader knows anything about tape or TAP. The important take-away is unit testing in MarkLogic and, secondarily, API compatibility with tape/Node.

Figure out how to test for MarkLogic

Since it’s not possible (as far as I can tell) to use actual tape in MarkLogic, testing code needs to decide which library at runtime.

'use strict';
function isMarkLogic() {
  try {
    return xdmp && cts;
  } catch(e) {
    return false;
  }
}
const test = isMarkLogic() ? 
               require('/mltap/test') : 
               require('tape-catch');

Yikes! That’s ugly.

In strict mode it’s not safe to check for the existence of globals, hence the try/catch.

Could we push this up to the harness or runner to only do it once?

Amps shouldn't be inherited by tests

Tests are run by invoking main modules from harness.sjs#runner(), which is amped to mltap-internal. Tests themselves shouldn’t get the elevated privileges, namely changing the modules path.

A unit test should try to change the modules root and call should fail with a security exception.

Fail fast on uncaught errors

Tape adopts the Node philosophy exiting the process on any uncaught errors. It even removed the ability to log an uncaught error as a failure from the API. (tape-catch extends Tape to support this behavior.) mltap should do the same.

  1. Don’t catch errors in Test.prototype.run()
  2. Remove tape-catch in favor of just tape
  3. Verify that you get good stack traces to figure out exactly where the error is

Linux support

Some of the configuration relies on OS X and bash.

Handle no expected value in unary assertions

In assert.true() the expected value should be true. It currently comes up as undefined because it’s not being set. Presumably this happens for assert.false() and any other unary assertions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.