Git Product home page Git Product logo

filter-component's Introduction

Filter Component NPM version circle-url

Description

A component to filter the incoming data based on an arbitrary JSONata expression.

How it works

Filter will pass through the incoming data if it matches the JSONata condition specified in the configuration. The expression can be any valid JSONata expression, so you can be creative. Here are some examples that are possible:

  • true
  • false
  • $not(false)
  • $not(true)
  • 20 > 5
  • body.foo - will be true if body.foo is defined and not false

Requirements

Environment variables

By default no environment variable is necessary to operate the component.

Triggers

This component has no trigger functions, so it is not selectable as a trigger (first step in an integration flow).

Actions

Simple JSONata Filter

This action has two parameters and a checkbox:

Parameters:

  • Filter condition - A JSONata expression passed in through the cfg. The expression will be evaluated to a value of true or false. If it is evaluated to false, a message will be logged to the console and the msg will not be sent forward to the next component. If evaluated true, a message will be passed forward along with all data that passed the condition.

Checkbox:

  • If checked, the checkbox adds Assertion functionality. Instead of doing nothing, the component will throw an error when the condition is not met.

  • Metadata To Response Adding passthrough from a previous step to message body as elasticioMeta variable, if enabled.

Additional Notes

  • The JSONata Filter expression can be a valid expression however it can cause an error to be thrown if it is invalid based on the context. For example, $number(hello) > 5 where hello: "world". This JSONata expression is valid but an error will be thrown as hello is NaN.

License

Apache-2.0 © elastic.io GmbH

filter-component's People

Contributors

a3a3e1 avatar denyshld avatar dvd-z avatar jhorbulyk avatar khanzadyan avatar olegosh avatar pnedelko avatar stas-fomenko avatar uaarsen avatar zainnoor99 avatar zubairov avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

flowground

filter-component's Issues

Add Assertion Checkbox to JSONata Filter

This new functionality should be similar to the existing Simple JSONata Filter however, unlike the existing functionality which logs and then discards messages which do not meet the criteria, this new functionality would log the failure and then throw an exception if the checkbox is selected.

Input field data in credentials cannot be evaluated as field type `JSONataView` does not work correctly for credentials

Description:
Input field data in credentials cannot be evaluated as field type JSONataView does not work correctly for credentials.
Can't be fixed from component side, requires platform fix.
STR:

  1. create flow: 1st step (any component) -> step2 - configuration component
  2. Create creds and add some input
  3. Click on Evaluate button

AR:
Value is not evalueated as this button doe not work in credentials section
image

ER:
Value is evalueated.

Bug: Filter brings metadata into body of emitted message

Steps to Reproduce

  1. Create a flow: Simple Trigger -> Rest API -> Filter
  • For the REST API component, make a GET request to https://google.com
  • For the filter component, use the condition 0 != 1
  1. Generate a sample on the Filter component

Expected result

  • Body of produced sample is exactly the same as the sample produced by the REST API component.

Actual Result

  • Body of produced has an additional elasticioMeta which contains the meta information of all proceeding steps.

Pass incoming message when filter condition is true

I strongly believe title of the issue should be the filter behaviour. It is absolutely counter intuitive to see empty message as a valid output of filter component. It's not user friendly nor understandable for me.

Retrieve sample process stuck if JSONata condition not met for a sample

When using Filter component, it was noticed that retrieving a sample process is stuck if JSONata condition not met for a sample used from previous steps:

filter_issue

filter_logs

Reproducing steps:

  1. create a new flow;
  2. add first step with any component for records source (in my case it was Google Spreadsheet);
  3. in the first step retrieve a sample;
  4. add second step using Filter component;
  5. in "Configuration" add a JSONata expression that does not match the sample from step 1 (based on any property);
  6. click "retrieve new sample".

Actual result: the loading is stuck and no sample is retrieved.
Expected result: a sample is retrieved.

Error when the "Send reject message to" is selected

When the (Optional) Send reject message to parameter is selected an error is reported.

To Replicate

  1. Build an integration flow in any workspace where you have another active flow starting with WebHook.
  2. The second step is the Filter component.
  3. Input any expression for the Filter condition, need to evaluate true/false. Use JavaScript.
  4. Select the drop-down menu (Optional) Send reject message to to select any flow.

ER: You get the list of Flow IDs

AR: The following Error is reported:

403 - {"errors":[{"detail":"Workspace is not found","status":403,"code":403,"title":"Unauthorized","meta":{}}]}

The log has more on this error:

Starting your Docker container ...
[INFO  tini (1)] Spawned child process '/run.sh' with pid '7'
Starting application in apprunner...
Starting slug download ...
Successfully downloaded and extracted slug file
node: /usr/lib/libstdc++.so.6: no version information available (required by node)
node: /usr/lib/libstdc++.so.6: no version information available (required by node)
at Request.plumbing.callback (/home/apprunner/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.emit (events.js:194:7)
at emitNone (events.js:91:20)
at endReadableNT (_stream_readable.js:975:12)
at process._tickCallback (internal/process/next_tick.js:104:9)
highWaterMark: 16384,
buffer: [Object],
defaultEncoding: 'utf8',
_newSessionPending: false,
_controlReleased: true,
_tlsOptions: [Object],
_eventsCount: 9,
_sockname: null,
_pendingEncoding: '',
_idlePrev: null,
_idleTimeout: -1 },
_secureEstablished: true,
_securePending: false,
authorized: true,
_handle: null,
readable: false,
destroyed: true,
_pendingEncoding: '',
server: undefined,
_requestCert: true,
_idlePrev: null,
httpVersionMinor: 1,
complete: true,
_maxListeners: undefined,
read: [Function],
_consuming: true,
_dumped: false,
output: [],
outputEncodings: [],
outputCallbacks: [],
writable: true,
upgrading: false,
_removedHeader: {},
finished: true,
socket: [Object],
connection: [Object],
_header: 'GET /v2/flows/ HTTP/1.1\r\nhost: api.elastic.io\r\nauthorization: Basic dGlncmFuQGVsYXN0aWMuaW86NTJjNGFlYWUtOTYyYS00ZjExLWI5NDMtMzU4MWI0MDcwOGY5\r\naccept: application/json\r\nConnection: close\r\n\r\n',
path: '/v2/flows/',
parser: null,
maxHeadersCount: null },
Request {
readable: true,
_qs: [Object],
_oauth: [Object],
_rp_resolve: [Function],
dests: [],
setHost: true,
pool: {},
host: 'api.elastic.io',
agent: [Object],
[INFO  tini (1)] Main child exited normally (with status '0')
at emitOne (events.js:96:13)
at Request.emit (events.js:191:7)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
statusCode: 403,
error: { errors: [ [Object] ] },
transform: undefined,
resolveWithFullResponse: false,
IncomingMessage {
ReadableState {
pipesCount: 0,
flowing: true,
endEmitted: true,
emittedReadable: false,
needReadable: false,
readableListening: false,
ranOut: false,
_events:
{ end: [Object],
_securePending: false,
_SNICallback: null,
_eventsCount: 9,
_sockname: null,
_pendingEncoding: '',
_idlePrev: null,
_idleTimeout: -1 },
_secureEstablished: true,
_securePending: false,
authorized: true,
_handle: null,
readable: false,
destroyed: true,
_pendingEncoding: '',
server: undefined,
_requestCert: true,
_idlePrev: null,
httpVersionMinor: 1,
complete: true,
_maxListeners: undefined,
read: [Function],
_consuming: true,
_dumped: false,
output: [],
outputEncodings: [],
outputCallbacks: [],
writable: true,
upgrading: false,
_removedHeader: {},
finished: true,
socket: [Object],
connection: [Object],
_header: 'GET /v2/flows/ HTTP/1.1\r\nhost: api.elastic.io\r\nauthorization: Basic dGlncmFuQGVsYXN0aWMuaW86NTJjNGFlYWUtOTYyYS00ZjExLWI5NDMtMzU4MWI0MDcwOGY5\r\naccept: application/json\r\nConnection: close\r\n\r\n',
path: '/v2/flows/',
parser: null,
maxHeadersCount: null },
Request {
readable: true,
_qs: [Object],
_oauth: [Object],
_events: [Object],
_parent: null,
_host: 'api.elastic.io',
_readableState: [Object],
_maxListeners: undefined,
_writableState: [Object],
writable: false,
_pendingData: null,
server: undefined,
_server: null,
parser: null,
_httpMessage: [Object],
authorized: true,
npnProtocol: false,
encrypted: true,
_events: [Object],
connecting: false,
_maxListeners: undefined,
_sockname: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
connecting: false,
httpVersion: '1.1',
readable: false,
_readableState: [Object],
_writableState: [Object],
allowHalfOpen: false,
_server: null,
ssl: null,
_requestCert: true,
_idleNext: null,
_idlePrev: null,
domain: null,
_eventsCount: 5,
_last: true,
chunkedEncoding: false,
_hasBody: true,
_trailer: '',
_onPendingData: null,
method: 'GET',
timeout: undefined,

Filter. Update to latest sailor 2.5.4

DoD
Development:

  • use new logger
  • component should be dockerised, add: "buildType":"docker" line into component.json
  • sailor version is changed from -.-.- to -.-.-
  • CHANGELOG.md is updated
    QA:
  • Smoke test is finished
  • Regression testing is finished (optional)
  • Component is deployed to the production stage (app.elastic.io) and/or another required stage

Cannot read property ‘info’ of undefined

With the latest component version ( commit 931c4a6 ), customers are getting error messages if the latest component version is used in a flow.

[2020-01-07 12:17:43.311]: Starting application in apprunner...
[2020-01-07 12:17:43.319]: Will try to start app at /home/apprunner
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:43.969]: node: /usr/lib/libstdc++.so.6: no version information available (required by node)
[2020-01-07 12:17:45.269]: Cannot read property 'info' of undefined
[2020-01-07 12:17:45.282]: [INFO  tini (1)] Main child exited normally (with status '1')```

Filter Does Not Handle Passthrough Correctly

Steps to Reproduce

Create the following Flow:

  • Step1: Code component with code
// Please note only Node.js code is supported here
async function run(msg) {
	const body = { one : 'sample' };
	// You can emit as many data messages as required
	await this.emit('data', { body });
}
  • Step2: Code component with code
// Please note only Node.js code is supported here
async function run(msg) {
	const body = { two : 'sample' };
	// You can emit as many data messages as required
	await this.emit('data', { body });
}
  • Step 3: Filter with condition
elasticio.step_1.body.one = two

Expected Result

Filter component always emits result

Actual result

Filter produces The message was received but did not meet the condition.

Notes

The JSONata transform component has some special code that handles this that was previously overlooked.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.