An implementation of SRI (Standard ROA Interface). SRI is a set of standards to make RESTful interfaces. It specifies how resources are accesses, queried, updated, deleted. The specification can be found here.

The implementation supports storage on a postgres database.

Installation is simple using npm :

$ cd [your_project]
$ npm install katholiek-onderwijs-vlaanderen/sri4node

You will also want to install Express.js :

$ npm install express

Express.js is technically not a dependency (as in npm dependencies) of sri4node. But you need to pass it in when configuring. This allows you to keep full control over the order of registering express middleware.

Logging can be enabled in the sri4node configuration by specifying the logdebug property. This property should be defined as an object with following fields:

• channels: mandatory, can be either 'all' to get all possible logging or a list with names of the channels for which logging is desired. The available logchannels are:
  • general: information about plugins loaded and routes registered
  • db: information about database interaction (task/transaction start/commit/rollback/end)
  • sql: the sql being executed
  • requests: logs start and end of a request with timing
  • hooks: informations about hooks being executed (start/stop/failure with timing)
  • server-timing: logs timing information about the request (same as in the ServerTiming header would be returned)
  • batch: information about batch parts being executed
  • trace: detailed information of interal sri4node flow (schema validation, expansion, count, ...)
  • phaseSyncer: information about the phase syncer mechanism
  • overloadProtection: information from the overload protectionmechanism
  • mocha: extra debug logging when running the mocha test suite
• statuses: optional, a list with status codes; if specified, logging is only done for requests returning the specified statuses

An example:

logdebug: {
  channels: [ 'general', 'requests', 'trace' ] },
  statuses: [ 500 ]
}

On a running sri4node instance the logdebug configuration can be altered with a POST call to /setlogdebug with the new logdebug configuration object as body.

An example of such POST call with curl:

$ curl -u <username> --request POST 'https://<servername>/setlogdebug' \
--header 'Content-Type: application/json' \
--data-raw "{ \"channels\": [ \"general\", \"requests\", \"trace\", \"batch\" ] }"

Start by requiring the module in your code. Then we'll create some convenient aliasses for the utility functions bundled with sri4node as well.

const express = require('express');
const app = express();
const sri4node = require('sri4node');
const $u = sri4node.utils;
const $m = sri4node.mapUtils;
const $s = sri4node.schemaUtils;
const $q = sri4node.queryUtils;

Finally we configure handlers for 1 example resource. This example shows a resource for storing content as html with meta-data like authors, themes and editor. The declaration of the editor is a reference to a second resource (/person), which itself is not shown here.

const sriConfig = {
        // Log and time HTTP requests ?
        logrequests : true,
        // Log SQL ?
        logsql: false,
        // Log debugging information ?
        logdebug: false,
        // Log middleware timing ?
        logmiddleware: true,
        // The URL of the postgres database
        defaultdatabaseurl : "postgres://user:pwd@localhost:5432/postgres",
        forceSecureSockets: true,
        description: 'A description about the collection of resources',
        resources : [
            {
                // Base url, maps 1:1 with a table in postgres
                // Can be defined as /x or /x/y/..
                // Table name will be the last part of the path
                type: '/content',
                // Is this resource public ?
                // Can it be read / updated / inserted publicly ?
                public: false,
                // Multiple function that check access control
                // They receive a database object and the security context
                // as determined by the 'identify' function above.

                // Standard JSON Schema definition.
                // It uses utility functions, for compactness.
                schema: {
                    $schema: "http://json-schema.org/schema#",
                    title: "An article on the websites/mailinglists.",
                    type: "object",
                    properties : {
                        authors: $s.string("Comma-separated list of authors."),
                        themes: $s.string("Comma-separated list of themes."),
                        html: $s.string("HTML content of the article.")
                    },
                    required: ["authors","themes","html"]
                },

                // Supported URL parameters are configured
                // for list resources. $q is an alias for
                // sri4node.queryUtils.
                // This is a collection of predefined functions.
                // You can build your own (see below).
                // These functions can execute an arbitrary set
                // of preparatory queries on the database,
                // You can execute stored procedures and create
                // temporary tables to allow you to add things like :
                // ' AND key IN (SELECT key FROM mytemptable) '
                // to the query being executed.
                // Allowig any kind of filtering on
                // the resulting list resource.
                query: {
                    editor: $q.filterReferencedType('/persons','editor'),
                    defaultFilter: $q.defaultFilter
                },
                // The limit for queries. If not provided a default will be used
                defaultlimit: 5,
                // The maximum allowed limit for queries. If not provided a default will be used
                maxlimit: 50,
                // All columns in the table that appear in the
                // resource should be declared in the 'map' object.
                // Optionally mapping functions can be given.
                // Mapping functions can be registered
                // for onread, onwrite and onupdate.
                //
                // For GET operations the key in the 'map' object
                // is the name of the key as it will appear in the JSON output.
                //
                // For PUT operations it is the key that appears
                // on the input resource. The end result after mapping
                // is created / updated in the database table.
                map: {
                    authors: {},
                    themes: {},
                    html: {},
                    // Reference to another resource of type /persons.
                    // (mapping of 'persons' is not shown in this example)
                    // Can also be referenced as /x/y/.. if another resources is defined like this
                    editor: {references : '/persons'}
                },

                // After read, update, insert or delete
                // you can perform extra actions.
                beforeInsert: [ validateAuthorVersusThemes ]
                beforeUpdate: [ validateAuthorVersusThemes ]

                afterRead: [ checkAccessOnResource, checkSomeMoreRules, addAdditionalInfoToOutputJSON ],
                afterUpdate: [ checkAccessOnResource, checkSomeMoreRules ],
                afterInsert: [ checkAccessOnResource, checkSomeMoreRules ],
                afterDelete: [ checkAccessOnResource, checkSomeMoreRules, cleanupFunction ]

                // custom routes can be defined. See #custom-routes
                customroutes: [
                  { routePostfix: '/:key/:attachment'
                  , httpMethods: ['GET']
                  , handler:  getAttachment
                  },
                ],
            }
        ]
    });

Many properties of the sri4node config object have defaults and can be omitted.

There is only 1 way to configure the database which sri4node uses:

• Specify sri4nodeConfig.databaseConnectionParameters in the sri4node configuration.
  • These are any format as supported by connection syntax of the DB library + a 'schema' property
  • You can also specify the pg-promise initialization options by adding a property databaseLibraryInitOptions

These mechanisms have been made obsolete:

• [OBSOLETE > 2.3] ~~Specify an already initiated database object in the sri4node configuration. This way has two variants:
  • db: the global sri4node pg-promise database object used for all database work.
  • dbR and dbW: specify different global sri4node pg-promise database objects for read-only database work (dbR) and other database work (dbW), useful if you work with read-replica's.~~
• [OBSOLETE > 2.3] Specifying the database connection string in the DATABASE_URL environment variable
• [OBSOLETE > 2.3] Specifying the database connection string in the defaultdatabaseurl field in the sri4node configuration

To enable pg-monitor (detailed logging about database interaction), set the boolean enablePgMonitor in the sri4node configuration on true.

Next step is to pass the sri4node config object to the async sri4node configure function. The configure function will return a sriServerInstance object containing a reference to pgp (an initialised version of the pgPromise library, see http://vitaly-t.github.io/pg-promise), db (pgPromise database object, see http://vitaly-t.github.io/pg-promise/Database.html) and app (the Axpress application).

const sriServerInstance = await sri4node.configure(app, sriConfig);

Now we can start Express.js to start serving up our SRI REST interface :

app.set('port', (process.env.PORT || 5000));
app.listen(app.get('port'), function() {
    console.log('Node app is running at localhost:'' + app.get('port'))
});

In the latest version, we decided to rewrite a few things in order to be able to fix long-known problems (including the fact that GETs inside BATCH operations was not properly supported).

So if your new to sri4node, it's best to jump to Usage for the general principles first and then come back to read about the changes in the latest version!

In order to fix the BATCH problem, we had to replace the express request object by our own version (which is very similar). At any point in the chain, you can add properties to this object that you might need later on, like the current user or something else.

The 'sriRequest' object will have the following properties (but the developer can of course add its own extra properties like user or something else to this object)

• path: path of the request
• originalUrl: full url of the request
• query: parameters passed via the query string of the url, as object like express { key: value, key2: value2 }
• params: parameters matched in the request route like UUID in /person/:UUID => /persons/cf2dccb2-c77c-4402-e044-d4856467bfb8, as object like express { key: value, key2: value2 }
• httpMethod: http method of the request (POST GET PUT DELETE (PATCH))
• headers: headers of the original request (same for all requests in batch)
• protocol: protocol of original request (same for all requests in batch)
• body: body of the request
• sriType: type of the matching entry in the sri4node resources config
• isBatchPart: boolean indicating whether the request being executed is part of a batch
• dbT: database task (if the request is read-only) or transaction object (if the request is not read-only), which can be used to access the database.
• context: an object which can be used for storing information shared between requests of the same batch (by default an empty object)
• SriError: constructor to create an error object to throw in a handler in case of error
  new SriError( { status, errors = [], headers = {} } )
• userData: an object which can be used by applications using sri4node to store information associated with a request. It is initialized as an empty object.

We also removed the me concept, allowing the implementer to add stuff to the sriRequest object as it sees fit. If that info is 'the current user', so be it, you can just as well build an API that doesn't need an identity to work.

The possibility of appending '/validate' to a PUT request to see wether it would succeed is replaced by a more general 'dry run' mode. This mode is activated by adding ?dryRun=true to the request parameters. This means that the request is executed and responsed as normal, but in the end the database transaction corresponding to the request is rolled back.

A valid patch is in [RFC6902 format][https://tools.ietf.org/html/rfc6902].

Whenever an uncaught exception happens, sri4node should be smart enough to send a 500 Internal Server Error, but when something else goes wrong, the implementer probably wants to be able to set the http response code himself (and expect the running transaction to be rolled back).

throw new sriRequest.SriError( { status = 500, errors = [], headers = {} } )

When you throw an SriError somewhere in your hook code, the current request handling is terminated (of course the db transaction is also cancelled) and the request is answered according to the SriError object. So statuscode and headers of the response are set according to the values of the fields in the SirError object and the body of the response will be a JSON object contain the status (status code) and errors (errors provided in the SirError object, with 'type' field of each error set to 'ERROR' if not specified). All parameters of the SriError constructor have default values, so each parameter can be omitted.

If your code ever needs to catch SriError object (for example to do some logging), you should rethrow the original SriError at the end of the catch. At some places in sri4node the type of the error object is checked and treated different in case the type is SriError. So in case something else is throw or returned at the end of a catch, sri4node might behave unexpected.

An example of the usage of setting extra http headers is setting the Location headers in case of redirect when not logged.

At the same time we took this opportunity to rename, add and remove a few 'hooks' (functions like onread, on...).

As a general principle for renaming those functions, we think that sri4node should make less assumptions about what these hooks are meant for, but make it clear to the implementer when the hook will be called.

For example: 'validate' will be replaced by 'beforeinsert/beforeupdate' to make it clear when the operation happens, but the function name doesn't suggest anymore what you should do at that time. Of course you could do some validation of the input before an insert, but you might as well dance the lambada if you wish. Sri4node shouldn't have an opinion about that.

In the sri4node configuration, hooks are always configured as an array of hook functions.

All hooks are 'await'ed for (and the result is ignored). This makes that a hook can be a plain synchronous function or an asynchronous function. You could return a promise but this will not have any added value.

Gobal hooks can be defined in the root of the sri4node config and are called for each request (unless an error occurs earlier in the request processing flow).

startUp(db, pgp);

This function is called during sri4node configuration, before anything is registered in express. It is only called once during the lifetime of an sri4node instance. It can be used to check and update the db schema before starting the API, or to do very specific stuff on the pgp library instance.

transformRequest(expressRequest, sriRequest, tx);

This function is called at the very start of each http request (i.e. for batch only once). Based on the expressRequest (maybe some headers?) you could make changes to the sriRequest object (like maybe add the user's identity if it can be deducted from the headers).

New hook which will be called before each phase of a request is executed (phases are parts of requests, they are used to synchronize between executing batch operations in parallel, see Batch execution order).

beforePhase(sriRequestMap, jobMap, pendingJobs);

• sriRequestMap is a Map (phaseSyncer id => sriRequest) containing all sriRequests being processed (one for each parallel batch operation).
• jobMap is a Map (phaseSyncer id => phaseSyncer) containing all phaseSyncer objects (one for each parallel batch operation).
• pendingJobs is a Set containing the ids of phaseSyncer objects which are still pending.

This hook will be called in case an exception is catched during the handling of an SriRequest. After calling this hook, sri4node continues with the built-in error handling (logging and sending error reply to the cient). Warning: in case of an early error, the parameter sriRequest might be undefined!

errorHandler(sriRequest, error);

New hook which will be called after the request is handled. At the moment this handler is called, the database task/transaction is already closed and the response is already sent to the client.

afterRequest(sriRequest);

Resource specific hooks can be defined in an element of the resources list in the sri4node config and are called for each matching request of the resource type for which it is specified (unless an error occurs earlier in the request processing flow).

These functions replace validate and secure. They are called before any changes to a record on the database are performed. Since you get both the incoming version of the resource and the one currently stored in the DB here, you could do some validation here (for example if a certain property can not be altered once the resource has been created).

• beforeRead( tx, sriRequest )
• beforeUpdate( tx, sriRequest, [ { permalink: …, incoming: { … }, stored: { … } } ] ) )
• beforeInsert( tx, sriRequest, [ { permalink: …, incoming: { … }, stored: null } ] ) )
• beforeDelete( tx, sriRequest, [ { permalink: …, incoming: null, stored: { … } } ] ) )

The tx is a task or transaction object (a task in case of read-only context [GET], a transaction in case of possible write context [PUT,POST,DELETE]) from the pg-promise library, so you can do DB queries (a validation check that can only be done by querying other resources too) or updates (maybe some logging) here if needed. tx.query(...)

The last parameter will always be an array (but it will most of the time only contain one element, but in the case of lists it could have many element). This makes it easy to implement this function once with an array.forEach( x => ... ) and allows you to make optimizations (if possible) for list queries.

Below is a code example showing how validation can be done with one beforeInsert/beforeUpdate hook. For validation, it of course makes sense to provide an error message with all validation errors. Therefore one hook function is needed, evaluating all validation function and throwing an SriError in case one or more validation functions has failed:

const validateNoDuplicateEmailsForSamePerson = async (tx, incomingObj) => {
    const res = await readDuplicateEmailsFromDatabase(incomingObj.email)
    if (res.length > 0) {
        throw {code: duplicate.email.address, error: ... }
    }
}

const validateNationalIdentityNumber = (tx, incomingObj) => {
    ...
    if (controlNumber != calculatedControlNumber) {
        throw {code: invalid.national.identity.number, error: ... }
    }
}

const validationHook = (tx, sriRequest, elements) => {  // can be used for beforeInsert and beforeUpdate:
    pMap(elements, ({incoming}) => {
        const validationFuns = [ validateNoDuplicateEmailsForSamePerson, validateNationalIdentityNumber ]
        const validationResults = await pSettle(validationFuns.map( f => f((tx, incoming) ))
        const validationErrors = validationResults.filter(r => r.isRejected).map(r => r.reason)
        if (validationResults.length > 0) {
            throw new sriRequest.SriError({status: 409, errors: [{code: 'validation.errors', msg: 'Validation error(s)', validationErrors }]})
        }
    }
}

The existing afterread, afterupdate/afterinsert and afterdelete functions will get a new signature, more in line with all the other functions (same parameters).

stored should still be the one before the operation was executed.

• afterRead( tx, sriRequest, [ { permalink: …, incoming: null, stored: { … } } ] ) )
• afterUpdate( tx, sriRequest, [ { permalink: …, incoming: { … }, stored: { … } } ] ) )
• afterInsert( tx, sriRequest, [ { permalink: …, incoming: { … }, stored: null } ] ) )
• afterDelete( tx, sriRequest, [ { permalink: …, incoming: null, stored: { … } } ] ) )

This replaces handlelistqueryresult(rows).

This will be called just before sending out the response to the client, and it allows you to transform any response (not only list results) at will.

This can be used if you want to generate custom output that is not necessarily sri-compliant on some route. It should not be necessary for a normal sri-compliant API (but you never know).

transformResponse(tx, sriRequest, sriResult);

The sriResult object has at least following properties:

• status
• body And optionally:
• headers

Resource map specific hooks can be defined in a field/column object in the map object of an element of the resources list in the sri4node config. The mapping hooks are called when mapping database rows to/from sri objects of the resource type for which the are specified.

Individual properties can be transformed when going to or coming from the database.

For example: an sri array of references could be stored as an array of permalinks on the DB, but should be transformed to [ { href: "..." }, { href: "..." }, ... ] in the API. These functions could do that mapping from API-to-DB and back. Also when storing dates as dates, but outputting them as strings in the API, this would be the place to do the transformation.

These hooks replace the now obsolete onread/oninsert/onupdate... and should be configured as part of the 'mapping' component in your sri config object.

• fieldToColumn(propertyName, value)
• columnToField(propertyName, value)

A count query is a heavy query in Postgres in case of a huge table. Therefore the count in the list result is made optional. By adding listResultDefaultIncludeCount: false to the configuration of a resource the count will be omitted by default. This setting can be overriden in a list query by appending ?$$includeCount=[boolean] to the query parameters.

Other performance enhancement is to use key offsets for the next links instead of count offsets (prev links are skipped due to not being usefull in a scenario with key offsets). Count offset is becoming more and more time consuming when the count increases (as postgres needs to construct all the results before the requested set to get the starting point) while for key offset an index can be used to determine the starting point.

The sri4node config object contains a single array of plugins, that must be smart enough to make the necessary changes to the configuration object.

This means that you don't need to set hooks to functions from plugins anymore (no more onread: myplugin.onread etc.), but that a plugin will add the proper hooks itself.

You should check the readme of the plugin itself for information how it should be instantiated. Often they will require some dependency to be handed to them, and so they will export a factory function that gets the current sri4node library, and some configuration object as input.

Example:

import { myPluginFactory } from 'my-plugin'

sri4node.configure( app,
  {
    // add some sri4node plugins
    plugins : [
      myPluginFactory({ sri4node, prop: value, ... })
    ],
    ...

On top of that: if a plugin needs another plugin to work, it should be smart enough to automatically add that other plugin also to the list of plugins, so no more 'plugin A doesn't work because it needs plugin B' (cfr. AccessControl needing OAuth, including AccessControl should suffice). Alternatively, the plugin should define a peer dependency, making sure that npm install will inform you if another plugin that it depends on is not installed or an incompatible version.

The plugins work plug and play. An 'install' function will be called by sri4node during initialization with 2 parameters (sriConfig object and a writable database transaction) and the plugin will manipulate this sriConfig object to insert its hooks and other additions to the config.

(In the future it should return a new version of the config without actually touching the input config, but that is not how it currently works).

A valid sri4node plugin consists of an object, exposing an install(SriConfig, dbW) method that will be called when sri4node starts.

If your plugin instance needs an instance of the current sri4node library (for example because it wants to log certain info with the debug or error methods) make sure you get that instance when creating the plugin.

Our suggestion is that every plugin author exports a single myPluginFactory(...) function that gets everything it needs as an argument and which returns the plugin instance.

Example:

function afterUpdate(sri4node) {
    sri4node.debug('general', 'myPlugin: after update!')
}

function myPluginFactory({ sri4node, prop }) {
    const hook = () => afterUpdate(sri4node);

    return {
        install: function(sriConfig, dbW) {
            // modify sriConfig for example add some hooks for every resource
            sriConfig.resources.forEach(
                (r) => r.afterUpdate = r.afterUpdate
                        ? [...r.afterUpdate, hook)]
                        : [hook]
            );
        }
    }
}

export {
    myPluginFactory,
}

In a batch all operations in an inner list are executed in 'parallel' (in practice with a concurrency limit) but are 'phaseSynced' at several different points like at the start of the 'before'/'after' 'read'/'update'/'insert'/'delete' hooks and at the start of database operations. With 'phaseSynced' is meant that all operations need to be at the sync point before the operations continue (again in 'parallel'), so you can be sure that at the moment a validation rule in an after hook is evaluated all database operations of the inner batch list have been executed.

An overview of what happens during the different phases of all possible batch operations:

At the start of each phase a hook beforePhase() is executed, these are not shown in the table above. Marked in blue are hooks which might be provided by the application using sri4node. Marked in green are the beforePhase hooks used by sri4node itself to implement QBK and multi insert/update/delete.

QKB (QueryByKey) is a mechanism to bundle read requests of different elements in batch (instead of doing potentially a lot individual database operations). In the prepare phase all types and keys are collected, and in the next phase elements are fetched with one query for each resource type.

Multi insert/update/delete is a mechanism to bundle insert, update and delete operations of a batch (instead of doing potentially a lot individual database operations). In the prepare phase all relevant data is collected, and in the next phase a bulk db operation is executed for each kind of operation and each resource type.

If a batch contains multiple lists, these lists are handled in order list by list (with the inner lists executed in 'phaseSynced parallel' as described above).

So how you construct your batch determines which operations go 'phaseSynced' parallel and which go in order.

A batch like the one below will be able to retrieve a newly created resource:

[
  [ {
      "href": "/organisationalunits/e4f09527-a973-4510-a67c-783d388f7265",
      "verb": "PUT",
      "body": {
        "key": "e4f09527-a973-4510-a67c-783d388f7265",
        "type": "SCHOOLENTITY",
        "names": [
    {
      "type": "OFFICIAL",
      "value": "Official 1",
      "startDate": "2017-01-01"
    },
    {
      "type": "SHORT",
      "value": "Short 1",
      "startDate": "2017-01-01"
    }
        ],
        "description": "Some description...",
        "startDate": "2017-01-01"
      }
    } ],
  [ {
      "href": "/organisationalunits/e4f09527-a973-4510-a67c-783d388f7265",
      "verb": "GET"
    } ]
]

At the beginning of all transactions in sri4node the database constraints in Postgres are set DEFERRED. At the end of the transaction before comitting the constraints are set back to IMMEDIATE (which results in evaluation at that moment). This is necessary to be able to multiple operations in batch and only check the constraints at the end of all operations. For example to create in a batch multiple resoures which are linked at with foreign keys at database level (example a batch creation of a person together with a contactdetail for that person).

But this will only work for certain types constraints and only if they are defined DEFERRABLE. From the postgres documentation (https://www.postgresql.org/docs/9.2/sql-set-constraints.html):

Currently, only UNIQUE, PRIMARY KEY, REFERENCES (foreign key), and EXCLUDE constraints are affected by this setting. NOT NULL and CHECK constraints are always checked immediately when a row is inserted or modified (not at the end of the statement). Uniqueness and exclusion constraints that have not been declared DEFERRABLE are also checked immediately.

An example from samenscholing where foreign keys are defined DEFERRABLE:

CREATE TABLE organisationalunits_relations (
    key UUID,
    type text NOT NULL,
    "from" UUID NOT NULL references organisationalunits DEFERRABLE INITIALLY IMMEDIATE,
    "to" UUID NOT NULL references organisationalunits DEFERRABLE INITIALLY IMMEDIATE,
    "startDate" date NOT NULL,
    "endDate" date,
    "$$meta.created" timestamp with time zone DEFAULT now() NOT NULL,
    "$$meta.modified" timestamp with time zone DEFAULT now() NOT NULL,
    "$$meta.deleted" boolean DEFAULT false NOT NULL,
    PRIMARY KEY (key)
 );

To be able to keep track of requests in the server logging and at the client, sri4node generates a short id for each request (reqId - not guaranteed to be unique). This ID is used in all the sri4node logging (and also in the logging of sri4node application when they use debug and error from sri4node common), sri4node error responses and is passed to the client in the vsko-req-id header.

Sometimes one wants to do sri4node operations on its own API, but within the state of the current transaction. Internal requests can be used for this purpose. You provide similar input as a http request in a javascript object with the database transaction to execute it on. The internal calls follow the same code path as http requests (inclusive plugins like for example security checks or version tracking). global.sri4node_internal_interface has following fields:

• href: mandatory field
• verb: mandatory field
• dbT: mandatory field - database transaction of the current request
• parentSriRequest: mandatory field - the sriRequest of the current request
• headers: optional field
• body: optional field

In case of a streaming request, following fields are also required:

• inStream: stream to read from
• outStream: stream to write to
• setHeader: function called to set headers before streaming
• setStatus: function called to set the status before streaming
• streamStarted: function which should return true in case streaming is started

The result will be either an object with fields status and body or an error (most likely an SriError).

Remark: sri4node makes a distinction between a database task (consider this as a connection, no commit/rollback) and a transaction. For GETs a task database object is provided, while requests which potentially may modify the database receive a transaction object. With internal requests this has the consequence that in case your initial sri4node requests is a read-only operation (GET), you get database task for your internal request to operate on. If the internal request then writes via this task, the changes will not be rollbacked in case the request is not succesfully ended.

An example:

    const internalReq = {
        href: '/deploys/f5b002fc-9622-4a16-8021-b71189966e48',
        verb: 'GET',
        dbT: tx,
        parentSriRequest: sriRequest,
    }

    const resp = await (global as any).sri4node_internal_interface(internalReq);

An extra hook is defined to be able to copy data set by transformRequest (like use data) from the original (parent) request to the new internal request.

transformInternalRequest(tx, sriRequest, parentSriRequest);

This function is called at creation of each sriRequest created via the 'internal' interface.

Sri4node contains a protection mechanism for in case of overload. Instead of trying to handle all requests in case of overload and ending with requests receiving a response only after several seconds (or worest case even timing out), some requests are refused with an HTTP 503 (too many concurrent requests) error.

The mechanism works quite trivial by requiring a 'pipeline' for each requests running in parallel and defining a maximum number of 'pipelines'. As these 'pipelines' are just a virtual concept, they are implemented by a counter. If at the start of a request the pipelinesInUse counter has not reached the maximum, processing of the request starts and the counter is incremented. If the pipelinesInUse counter has reached the maximum, the request is refused with a 503 error. At the end of allowed requests, the counter is decremented. A batch is a special case, as a batch can run multiple operations in paralellel. Therefore, the number of pipelinesInUse is incremented/decreented with the number of batch operations running in parallel.

To enable this mechanism, an overloadProtection object needs to be configured in the sri4node configuration with following fields:

• maxPipelines: mandatory field - the maximum number of requests being processed at the same time; addional requests will be refused with a 503 error

For example:

    overloadProtection: {
        maxPipelines: 15
    },

With the isPartOf query one can check if a given raw url A is a subset of the given raw urls in list B.

The syntax of the query request is POST to /[resource]/isPartOf with a JSON body containing an object with 2 fields:

• a: an object with field href containing a raw url (can be a single resource or a list url)
• b: an object with fields hrefs containing a list of raw urls (each element in list B can also be a single resource or a list url) The reply is a JSON array with all raw urls from list B for which url A is a subset (urls B and A are compared by resolving them to a set of single resources and checking if set A is a subset or equal of set B).

An example request:

POST /messages/isPartOf
{ 
  "a": { "href": "/messages?descriptionRegEx=^Ik.*$"  },
        "b": { "hrefs": [ "/messages?type=request"
                        , "/messages?titleRegEx=^Wie.*$"
      , "/messages"] }
}

with an example reply:

[ "/messages?type=request", "/messages" ]

Remark: the raw url A and all raw urls in list B needs to be of the same type [resource].

• [OBSOLETE] dbConnectionInitSql (use databaseConnectionParameters.connectionInitSql instead): optional sql string which will be executed at the start of each new database connection

By default sri4node will initialize the database connection based on the sri4node configuration object. But one can also pass database connection(s) to sri4node (initialized with the pgInit and/or pgConnect functions - see also General Utilities, just below) in case extreme customization is required (you should be able to handle almost all cases by specifying databaseConnectionParameters and databaseLibraryInitOptions). These database connection(s) can be passed with following fields in the sri4node configuration object:

• db: a pgp database connection object,
• dbR and dbW: two pgp database connection objects, one for reading and one for writing (can be used when working with database followers)

There are 4 columns that every resource table must have (it's mandatory).

Those are:

• "$$meta.deleted" boolean not null default false,
• "$$meta.modified" timestamp with time zone not null default current_timestamp,
• "$$meta.created" timestamp with time zone not null default current_timestamp,
• "$$meta.version" number which is increased on each change of the resource

The application will fail to register a resource that lacks these fields (and show a message to the user)

For performance reasons it's highly suggested that an index is created for each column:

• CREATE INDEX tablecreated ON _table ("$$meta.created");
• CREATE INDEX tablemodified ON _table ("$$meta.modified");
• CREATE INDEX tabledeleted ON _table ("$$meta.deleted");
• CREATE INDEX tableverion ON _table ("$$meta.version");

The following index is for the default order by:

• CREATE INDEX tablecreated_key ON _table ("$$meta.created", "key");

It is also highly suggested to have indices on fields which can be used to filter the resources in a list resource request. Both a plain index as a LOWER() index are required as the default equality check is a case insensitive check.

sri4node has a very simple processing pipeline for mapping SRI resources onto a database. We explain the possible HTTP operations below :

• reading regular resources (GET)
• updating/creating regular resources (PUT/PATCH)
• deleting regular resources (DELETE)
• reading list resources (queries) (GET)

In essence we map 1 regular resource to a database row. A list resource corresponds to a query on a database table.

Expansion on list resource can be specified as expand=results, this will include all regular resources in your list resource. A shorthand version of this is expand=full. Expansion on list resource can also be specified as expand=results.x.y,results.u.v.w, where x.y and u.v.w can be any path in the expanded regular resource. This will include related regular resources.

Expansion on regular resource can be specified as expand=u.v,x.y.z, where u.v and x.y.z can be any reference to related regular resources. This will include related regular resources.

When reading a regular resource a database row is transformed into an SRI resource by doing this :

1. Execute transformRequest functions.
2. Execute beforeRead functions.
3. Retrieve the row and convert all columns into a JSON key-value pair (keys map directly to the database column name). All standard postgreSQL datatypes are converted automatically to JSON. Values can be transformed by an columnToField function (if configured). By default references to other resources (GUIDs in the database) are expanded to form a relative URL. As they are mapped with { references: '/type' }.
4. Add a $$meta section to the response document.
5. Execute afterread functions to allow you to manipulate the result JSON.
6. Execute transformResponse functions to allow you to manipulate the request result.

When creating or updating a regular resource, a database row is updated/inserted by doing this :

1. Execute transformRequest functions.
2. Perform schema validation on the incoming resource. If the schema is violated, the client will receive a 409 Conflict.
3. Execute beforeInsert or beforeUpdate functions.
4. Convert the JSON document into a simple key-value object. Keys map 1:1 with database columns. All incoming values are passed through the fieldToColumn function for conversion (if configured). By default references to other resources (relative links in the JSON document) are reduced to foreign keys values (GUIDs) in the database.
5. insert or update the database row.
6. Execute afterUpdate or afterInsert functions.
7. Execute transformResponse functions to allow you to manipulate the request result.

When deleting a regular resource :

1. Execute transformRequest functions.
2. Execute beforeDelete functions.
3. Delete the row from the database.
4. Execute afterDelete functions.
5. Execute transformResponse functions to allow you to manipulate the request result.

When reading a list resource :

1. Execute transformRequest functions.
2. Execute beforeRead functions.
3. If count is requested: Generate a SELECT COUNT statement and execute all registered query functions to annotate the WHERE clause of the query.
4. Execute a SELECT statement and execute all registered query functions to annotate the WHERE clause of the query. The query functions are executed if they appear in the request URL as parameters. The query section can also define a defaultFilter function. It is this default function that will be called if no other query function was registered.
5. Retrieve the results, and expand if necessary (i.e. generate a JSON document for the result row - and add it as $$expanded). See the SRI specification for more details.
6. Build a list resource with a $$meta section + a results section.
7. Execute afterRead functions to allow you to manipulate the result JSON.
8. Execute transformResponse functions to allow you to manipulate the request result.

That's it ! :-).

If logmiddleware is true in the configuration the application will display a log of the timing of each middleware.

Below is a description of the different types of functions that you can use in the configuration of sri4node. It describes the inputs and outputs of the different functions. These functions are assumed to be asynchronuous are will be 'awaited'. Some of the function are called with a database transaction context, allowing you to execute SQL inside your function within the current transaction. Such a database object can be used together with sri4node.utils.prepareSQL() and sri4node.utils.executeSQL(). Transaction demarcation is handled by sri4node, on a per-request-basis. That implies that /batch operations are all handled in a single transaction. For more details on batch operations see the SRI specification.

Database columns are mapped 1:1 to keys in the output JSON object. The columnToField function receives these arguments :

• key is the key the function was registered on.
• element is the result of the query that was executed.

Functions are executed in order of listing in the map section of the configuration. No return value is expected, this function manipulates the element in-place. These functions allow you to do al sorts of things, like remove the key if it is NULL in the database, always remove a certain key, rename a key, etc.. A selection of predefined functions is available in sri4node.mapUtils (usually assigned to $m). See below for details.

JSON properties are mapped 1:1 to columns in the postgres table. The onupdate and oninsert functions recieves these parameters :

• key is the key they were registered on.
• element is the JSON object being PUT.
• isNewResource is a boolean indicating wether a resource is created or updated.

All functions are executed in order of listing in the map section of the configuration. All are allowed to manipulate the element, before it is inserted/updated in the table. No return value is expected, the functions manipulate the element in-place. A selection of predefined functions is available in sri4node.mapUtils (usually assign to $m). See below for details.

All queries are URLs. Any allowed URL parameter is interpreted by these functions. The functions can annotate the WHERE clause of the query executed. The functions receive these parameters :

• value is the value of the request parameter (string).
• select is a query object (as returned by sri4node.prepareSQL()) for adding SQL to the WHERE clause. See below for more details.
• parameter is the name of the URL parameter.
• tx is a database task object that you can use to execute extra SQL statements.
• count is a boolean telling you if you are currently decorating the SELECT COUNT query, or the final SELECT query. Useful for making sure some statements are not executed twice (when using the database object)
• mapping is the mapping in the configuration of sri4node.

All the configured query functions should extend the SQL statement with an AND clause.

The functions are assumed to be asynchronuous and are 'awaited'. When the URL parameter was applied to the query object, then the promise should resolve(). If one query function rejects its promise, the client received 404 Not Found and all error objects by all rejecting query functions in the body. It should reject with one or an array of error objects that correspond to the SRI definition. Mind you that path does not makes sense for errors on URL parameters, so it is ommited.

If a query parameter is supplied that is not supported, the client also receives a 404 Not Found and a listing of supported query parameters.

Hook for post-processing a GET operation (both regular and list resources). It applies to both regular resources, and list resources (with at least expand=results). The function receives these parameters :

• tx is a database task object, allowing you to execute extra SQL statements.
• sriRequest is an object containing information about the request.
• elements is an array of one or more objects:
  • permalink is the permalink of the resource
  • incoming is the received version of the resoured (null in case of afterRead)
  • stored is the stored version of the resource

The functions are assumed to be asynchronuous and are 'awaited'. If one of the afterread methods rejects its promise, all error objects are returned to the client, who receives a 500 Internal Error response by default. It should reject() with an object that correspond to the SRI definition of an error.

Hooks for post-processing a PUT operation can be registered to perform desired things, like clear a cache, do further processing, update other tables, etc.. The function receives these parameters :

• tx is a database transaction object, allowing you to execute extra SQL statements.
• sriRequest is an object containing information about the request.
• elements is an array of one or more objects:
  • permalink is the permalink of the resource
  • incoming is the received version of the resoured
  • stored is the stored version of the resource (null in case of afterInsert)

The functions are assumed to be asynchronuous and are 'awaited'. In case an error is thrown, all executed SQL (including the INSERT/UPDATE of the resource) is rolled back.

Hook for post-processing when a record is deleted. The function receives these parameters :

• tx is a database transaction object, allowing you to execute extra SQL statements.
• sriRequest is an object containing information about the request.
• elements is an array of one or more objects:
  • permalink is the permalink of the resource
  • incoming is the received version of the resoured (null in case of afterDelete)
  • stored is the stored version of the resource

The functions are assumed to be asynchronuous and are 'awaited'.

Each resource needs to have a meta type specified. This meta type will be set in the meta section of each returned resource as $$meta.type. For example, a resource like '/sam/persons' can have a metaType like 'PERSON'.

Can be used to restrict the methods which are allowed on a resource. If not specified the default is [ 'GET','PUT','PATCH','POST','DELETE' ]

Can be used to override the tablename in case it does not match the resource name.

There are three different custom scenario's possible. Two parameters are needed in all scenario's:

• routePostfix: is appended to the route of the resource where the custom route is defined, example '/:key/simple'
• httpMethods: array with http verbs the custom route matches

Optionally alterMapping can be used to create an altered mapping version for the custom route based on the normal resource mapping. For example, in the custom mapping transformResponse can be defined to alter the response specificly for the custom route.

• alterMapping: function (mapping) => {}

Optionally readOnly can be set to true to get a task pg-promise object instead of a transaction object in the custom route handler.

The possbile scenario's:

• A 'like' scenario: this scenario acts similar as an existing resource, only with a different custom mapping created with an alterMapping function. Parameters:

  • like: defines the path of regular resource to used example: "/:key".

• Plain custom handler: a handler generates all the custom output. Parameters:

  • handler: function dealing with the request: (tx, sriRequest, customMapping) => {}. Expected return is an object containing status, body and optionally headers. Optionally a before- and afterHandler can be defined:
  • beforeHandler (tx, sriRequest, customMapping) => {}
  • afterHandler (tx, sriRequest, customMapping, result) => {}

• Streaming scenario. The output stream can be JSON or binary stream

  • streamingHandler (tx, sriRequest, stream) => {}. The streamingHandler should only return after streaming is done. Optionally a beforeStreamingHandler can be defined to set status and headers (as they cannot be changed anymore once streaming is started):
  • beforeStreamingHandler (tx, sriRequest, customMapping) => { }. Returns an object containing status and headers. Headers is a list of [ headerName, headerValue ] lists.

  To enable binary streaming:

  • binaryStream: true When doing binary streaming it makes sense to use the beforeStreamingHandler to set some 'Content-*' headers specifying the type of content.

  In the streaming scenario it is also possible to (streamingly) read multipart form data with busBoy:

  • busBoy: true The busBoy event handlers can be set in the beforeStreamingHandler or the streamingHandler.
  • busBoyConfig: optional config object to be passed to busBoy (headers will be set by sri4node).

Streaming custom requests cannot be used in batch, the others can be used in batch.

For examples of all the custom scenarios, see the code in the sri4node tests:

• https://github.com/katholiek-onderwijs-vlaanderen/sri4node/blob/master/test/testCustomRoutes.js
• https://github.com/katholiek-onderwijs-vlaanderen/sri4node/blob/master/test/context/persons.js

The following attributes dictate how the lists are paginated:

• defaultlimit: the number of resources per page. If empty, a default of 30 is used.
• maxlimit: The maximum limit allowed. If empty, a default of 500 is used.

The limit query parameter can be used to specify the amount of retrieved results. A special case is allowed where the limit value is '*' and the expand parameter is 'NONE', this means unlimited results.

These utilities live independently of the basic processing described above. In other words, they provide no magic for the developer. They are provided for convenience. If you understand the above processing pipeline, reading the source for one of these functions should contain no surprises.

The utilities are found in sri4node.utils.

Used for preparing SQL. Supply a name to keep the query in the database as a prepared statement. It returns a query object with these functions :

• sql() is a method for appending sql.
• param(value) is a method for appending a parameter to the SQL statement.
• array(value) is a method for appending an array of parameters to the SQL statement (comma-separated). Useful for generating things like IN clauses.
• keys(value) adds all keys in an object comma-separated to the SQL statement.
• values(value) is a method for appending all values of an object as parameters to the SQL statement. keys and values have the same iteration order.
• with(query, virtualtablename) is a method for adding a different query object as WITH statement to this query. Allows you to use postgres Common Table Expressions (CTE) in your request parameters. You can refer in the query to the virtual table you named with virtualtablename. Use $u.prepareSQL() to build the SQL statement for your CTE.

All the methods on the query object can be chained. It forms a simple fluent interface.

Example of using a common table expression :

var query = $u.prepareSQL();
query.sql('SELECT * FROM xyz WHERE c1 IN (SELECT * FROM virtualtable)');
var cte = $u.prepareSQL();
cte.sql(...);
query.with(cte,'virtualtable');

Used for executing SQL. Call with the a database object you received, and a query object (as returned by prepareSQL(), or as received for query functions). The function returns a Q promise. It's not a responsible of this function to close the connection on error since it's an argument, hence the caller must properly make sure that the connection is disposed regardless of the result.

Afterread utility function. Adds, for convenience, an array of referencing resource to the currently retrieved resource(s). It will add an array of references to resource of type to the currently retrieved resource. Specify the foreign key column (in the table of those referencing resource) via foreignkey. Specify the desired key (should be $$somekey, as it is not actually a part of the resource, but provided for convenience) to add to the currently retrieved resource(s) via targetkey.

Receives a query object and constructs the SQL for a list query.

Arguments:

• req is the request object.
• mapping is the mapping in the configuration of sri4node.
• count a boolean to indicate if the query wanted is a count query or not.
• database a database obejct, allowing you to perform queries.
• query a query obtain via prepareSQL

This returns a promise that it's fulfilled when the query object contains the constructed SQL.

Provides various utilities for mapping between postgres and JSON. These functions can be found in sri4node.mapUtils.

Remove key from object if value was null/undefined.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        title: { onread: $m.removeifnull }
        ...
    },
    ...
}

Always remove this key.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        title: { onread: $m.remove }
        ...
    },
    ...
}

Override with current server timestamp.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        publicationdate: { onupdate: $m.now, oninsert: $m.now }
        ...
    },
    ...
}

Override with a fixed value.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        status: { oninsert: $m.value('active') }
        ...
    },
    ...
}

Convert string into JSON.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        details: {
            onread: $m.parse,
            oninsert: $m.stringify,
            onupdate: $m.stringify
        }
        ...
    },
    ...
}

Convert JSON into string.

sri4node = require('sri4node');
$m = sri4node.mapUtils;
...
{
    type: '/content',
    ...
    map: {
        ...
        details: {
            onread: $m.parse,
            oninsert: $m.stringify,
            onupdate: $m.stringify
        }
        ...
    },
    ...
}

These functions are found in sri4node.schemaUtils. Provides various utilities for keeping your JSON schema definition compact and readable. description is always used to document your resources. General usage:

sri4node = require('sri4node');
$s = sri4node.schemaUtils;
...
{
    type: '/content',
    ...
    schema: {
        $schema: "http://json-schema.org/schema#",
        title: "An article on the websites/mailinglists.",
        type: "object",
        properties : {
            title: $s.string('Title of the article.',1);
        },
        ...
    },
    ...
}

We describe the generated JSON schema fragment below. You can use these functions, but when they are insufficient you can insert any valid JSON schema manually in the schema property of a resource configuration. They are only provided for convenience.

Used for declaring permalinks. Example : $s.permalink('/persons','The creator of the article.'). Generated schema fragment :

{
    type: "object",
    properties: {
        href: {
            type: "string",
            pattern: "^\/" + name + "\/[-0-9a-f].*$",
            minLength: name.length + 38,
            maxLength: name.length + 38,
            description: description
        }
    },
    required: ["href"]
}

As you should define your postgres columns as type text setting minimum and maximum length is usually omitted. Example: $s.string('Title of the article.',5). Generated schema fragment :

{
    type: "string",
    description: description,
    minLength: min, // if supplied.
    maxLength: max, // if supplied.
}

Defines a property as numeric. Example: $s.numeric('The amount of ...'). Generated schema fragment :

{
    type: "numeric",
    multipleOf: "1.0",
    description: description
}

Defines an email. Example: $s.email('Personal email of the customer.'). Generated schema fragment :

{
    type: "string",
    format: "email",
    minLength: 1,
    maxLength: 254,
    description: description
}

Defines a URL. Example: $s.url('The homepage of the organisational unit.'). Generated schema fragment :

{
    type: "string",
    minLength: 1,
    maxLength: 2000,
    format: "uri",
    description: description
}

Defines a Belgian zipcode. Example: $s.zipcode('The zipcode of the address'). Generated schema fragment :

{
    type: "string",
    pattern: "^[0-9][0-9][0-9][0-9]$",
    description: description
}

Defines a telephone number. Example: $s.phone('The telephone of the customer'). Generated schema fragment :

{
    type: "string",
    pattern: "^[0-9]*$",
    minLength: 9,
    maxLength: 10,
    description: description
}

Defines a JSON timestamp. Example: $s.timestamp('The creation date/time of this resource'). Generated schema fragment :

{
    type: "string",
    format: "date-time",
    description: description
}

Defines a JSON boolean. Example: $s.boolean('Does she love me or does she not ?') Generated schema fragment :

{
    type: "boolean",
    description: description
}

Defines a column as GUID. Example: $s.guid('API-key for a plugin') Generated schema fragment :

{
  type: 'string',
  description: description,
  pattern: '^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$'
}

The functions are found in sri4node.queryUtils. Provides pre-packaged filters for use as query function in a resource configuration. The example assume you have stored sri4node.queryUtils in $q as a shortcut.

var sri4node = require('sri4node');
...
var $q = sri4node.queryUtils;

Can be used to filter on referenced resources. Example: /content resources have a key creator that references /persons. A list resource /content?creator=/persons/{guid} can be created by adding this query function :

{
    type: '/content',
    map: {
        ...
        creator: { references: '/persons' },
        ...
    },
    ...
    query: [
        creator: $q.filterReferencedType('/persons','creator')
    ],
    ...
}

Do a query to retrieve all content, created by person X :

GET /content?creator=/persons/{guid-X}

The value being passed in as a URL parameter can be a single href, or a comma-separated list of hrefs. The filter will match on any of the given permalinks.

An implementation of sri-query. SRI-query defines default, basic filtering on list resources. The function is a default, shared implementation of that.

{
    type: '/schools',
    map: {
        ...
    },
    ...
    query: [
        ...
        defaultFilter: $q.defaultFilter
    ]
}

Read the specification for details. Example queries are :

GET /schools?institutionNumberGreater=100000
GET /schools?nameContains=vbs
GET /schools?nameCaseInsensitive=Minnestraal
GET /schools?seatAddresses.key=a39c809e-a3a4-11e3-ace8-005056872b95

When a resource is detected as a relation (has from and to properties) some query filters are added for the list resources.

{
    type: '/relations',
    map: {
        from: {references: '/messages'},
        to: {references: '/messages'}
    },
    ...
    query: {
        ...
    }
}

Can be used to filter those relations where the 'from' resource is some of the given types.

Can be used to filter those relations where the 'to' resource is some of the given types.

Filter those relations where the 'from' resources is one of the given ones.

Filter those relations where the 'to' resources is one of the given ones.

Example queries are :

GET /relations?fromTypes=request,offer
GET /relations?toTypes=response
GET /relations?froms=/messages/{guid}
GET /relations?tos=/messages/{guid}

Documentation will be generated based on the configuration. On the /docs endpoint you can access general documentation about all the resources that are available. When you want more information about a resource you can access /resource/docs

To document validate functions you need to add validateDocs to the resource configuration. validateDocs has to include a description and possible error codes of the validate function.

validate: [ validateAuthorVersusThemes ], validateDocs: { validateAuthorVersusThemes: { description: "Validate if author or theme exists", errors: [{ code: 'not.a.desert', description: 'This is not a desert.' }] } }

To document a custom query function you need to add queryDocs to the resource configuration. queryDocs has to include the description of the query function.

query: { editor: $q.filterReferencedType('/persons','editor'), defaultFilter: $q.defaultFilter }, queryDocs: { editor: 'Allow to filer on an editor.' }

You can describe your sri interface by using the description variable in the root for your configuration

description: 'A description about the collection of resources'

You can describe a resource by using the to use schema > title

title: 'An article on the websites/mailinglists'

If you want to describe a property of a resource you need to use schema > properties > property > description :

properties : { authors: { type: 'string' description: 'Comma-separated list of authors.' } }

Or use the schemaUtils function:

properties : { authors: $s.string('Comma-separated list of authors.') }

Contributions are welcome. Contact me on dimitry-underscore-dhondt-at-yahoo-dot-com.

The software is licensed under LGPL license.