Assemble a json document via a variety of steps, functions, and filters, using Vert.X and reactive programming.

An input document provides a seed for a completed document. Our motivation for development is preparing search queries and indexing documents for search, but we see broader opportunities for the general system.

A pipeline provides a set of instructions for how to transform an input document into its final result. The Engine class is responsible for executing a pipeline. Pipelines consist of Sections and Steps. The Steps in a section will execute as soon as their prerequisites are ready.

The EnginePool manages a collection of pipelines that might do different things - for use in developing pipelines and/or organizing pipelines.

See the abstract-step-test-pipeline.json for the basics. A pipeline consists of a tacit or declared outer section, which includes a list of steps. The steps are executed in the order they are ready (which may not be their order of appearance in the pipeline). Variables for each step are processed by the step, and they are evaluated through the Jinja template engine before use.

Steps carry out core functionality for construction of the document. Logically lighter weight activities may be carried out as fiilters and functions within the Jinja template engine. For example, a step would call out to a search engine, while a set of filters might manipulate a query string in preparation for it.

When a step executes, AbstractStep assembles its environment with the variables provided in the "vars" block and some standard keys:

Values not in system and doc (vars), are evaluated with Jinja templating.

Jinjava provides a number of filters and functions.

Additional functions built in to this project are re:m(string, pattern) which returns the first match of the string, and re:s(string, pattern, replacement) which replaces all occurrences of pattern with replacement.

Filter reMatch(pattern) returns the first match of the pattern in the string piped in, and reSub(pattern,replacement) replaces all occurrences of pattern with replacement.

You can write your own steps, functions, and filters, too.

Extend AbstractStep (you could use the Step interface, but AbstractStep has a whole lot of basic functionality to it) and annotate your class with this:

import org.kohsuke.MetaInfServices;

@MetaInfServices(Step.class)

Implement JinjaFunctionDefinition and annotate your class with this:

import com.redhat.vertx.pipeline.templates.JinjaFunctionDefinition
import org.kohsuke.MetaInfServices;

@MetaInfServices(JinjaFunctionDefinition.class)

Annotate your public static methods with @ELFunction(value,namespace), where "value" is the name of the function and "namespace" will come before, like re:match("foobar","[fo]*"). Namespace may be left blank.

Implement a Jinja Filter and annotate your class with this:

import com.hubspot.jinjava.lib.filter.Filter;
import org.kohsuke.MetaInfServices;

@MetaInfServices(Filter.class)