elasticio / csv-component Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 7.0 691 KB

CSV Reading and Writing component for elastic.io platform

License: Apache License 2.0

JavaScript 100.00%

csv-component's People

Contributors

Stargazers

Watchers

Forkers

agworld flowground t4runsingh andreas-fintricity

csv-component's Issues

Default value cannot be set as 'Include Headers' field is required

Issue valid for actions:

Write CSV attachment from JSON Object
Write CSV attachment from JSON Array

AR: Error thrown on sample retrieve attempt if no option selected
ER:

'Include Headers' is not required
OR
'Include Headers' field is required and placeholder is changed: Include headers? Default Yes -> Include headers?

CSV component when displaying the header which contains number in the text. *(add _ between the text and number)

The client is having trouble when configuring the Headers as a string the component automatically creates (_) space between the text and the number
as an example

The CSV contains sch_h2 which in the sample data the header turns to "sch_h_2". When the user saves the sample with "sch_h_2". This confuses the user and also causes difficulty to map the correct header at any following steps.

here's a step for replication.

Fix helper actions links

Fix needed for actions:

Write CSV attachment from JSON Object
Write CSV attachment from JSON Array

Currently, it leads to a valid CSV component page but does not scroll to a proper action section

CSV should support dynamic metadata

Deprecate the existing "read" trigger

csv-component/component.json

Line 8 in ca19647

"read": {

isn't particularly useful. It should be deprecated.

Images

Migrate from TravisCI to CircleCI

In order to keep the component updated and to follow best practices, it should be migrated from TravisCI to CircleCI.

Action: Read CSV attachment does not support local agents

Description: Action: Read CSV attachment does not support local agents as it uses Steward.
Blocked by this issue

STR:

Create flow sftp-> CSV (Action: Read CSV attachment)
Select agent on second step and retrieve sample

AR: Sample is not retrieved, but error message is shown: Error! Failed to retrieve data sample from the component.

ER: Sample is retrieved on LA.

Notes:
Platform logs:
This process is being executed on your agent "stas_agent". Please see the logs locally. Error: ERROR_TASK_FAILED at new e (https://app.elastic.io/frontend.min.js:59:20458) at e (https://app.elastic.io/frontend.min.js:59:19348) at a (https://app.elastic.io/frontend.min.js:59:22251) at b (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:15:4738) at y (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:15:4836) at v (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:15:4641) at https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:9:33813 at invokeWithOnError (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:1:10663) at r.flush (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:1:11115) at n.flush (https://cdnjs.cloudflare.com/ajax/libs/ember.js/1.13.13/ember.min.js:1:9143)

Local agent logs:
[INFO tini (1)] Spawned child process '/run.sh' with pid '6' Starting application in apprunner... Starting slug download ... Successfully downloaded and extracted slug file node: /usr/lib/libstdc++.so.6: no version information available (required by node) node: /usr/lib/libstdc++.so.6: no version information available (required by node) node: /usr/lib/libstdc++.so.6: no version information available (required by node) node: /usr/lib/libstdc++.so.6: no version information available (required by node) node: /usr/lib/libstdc++.so.6: no version information available (required by node) node: /usr/lib/libstdc++.so.6: no version information available (required by node) invokeModuleFunction – init is not found Incoming message is {"id":"01a1854e-356c-4152-b548-e0b8d7038c4a","attachments":{"CSV_SAILOR_TEST2.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/6f61d8e4-47d5-4153-bb92-4350eb1f5f95","size":119}},"body":{"filename":"CSV_SAILOR_TEST2.csv","size":119},"headers":{},"metadata":{},"passthrough":{"step_1":{"id":"01a1854e-356c-4152-b548-e0b8d7038c4a","attachments":{"CSV_SAILOR_TEST2.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/6f61d8e4-47d5-4153-bb92-4350eb1f5f95","size":119}},"body":{"filename":"CSV_SAILOR_TEST2.csv","size":119},"headers":{},"metadata":{}}}} Found attachment key=CSV_SAILOR_TEST2.csv attachment={"url":"http://steward-service.platform.svc.cluster.local:8200/files/6f61d8e4-47d5-4153-bb92-4350eb1f5f95","size":119} Wed, 10 Apr 2019 08:38:07 GMT csv Sending GET request to url=http://steward-service.platform.svc.cluster.local:8200/files/6f61d8e4-47d5-4153-bb92-4350eb1f5f95 Error happened: getaddrinfo ENOTFOUND steward-service.platform.svc.cluster.local steward-service.platform.svc.cluster.local:8200 Error: getaddrinfo ENOTFOUND steward-service.platform.svc.cluster.local steward-service.platform.svc.cluster.local:8200 at errnoException (dns.js:50:10) at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:91:26) [INFO tini (1)] Main child exited normally (with status '1')

Improve CSV Write functionality

The CSV write functionality should do the following:

The component should produce a message with an attachment. The attachment should be a link to the CSV file.
In the configuration, the user is able to define the CSV's header columns as well as the type for each column.
The component should place multiple inbound events in the same CSV file (attachment). Event sequence end should be determined based on the inactivity timeout.
The cells in the CSV file should be appropriately formatted based on the set type for the column. This formatting should be set so that Excel users in various locals (USA & Germany) can easily open the file.

CSV. Update component to new sailor version / use new logger functionality / dockerize

DoD
Development:

use new logger
component should be dockerised, add: "buildType":"docker" line into component.json
sailor version is changed from -.-.- to -.-.-
CHANGELOG.md is updated
QA:
Smoke test is finished
Regression testing is finished (optional)
Component is deployed to the production stage (app.elastic.io) and/or another required stage

Write CSV should function allow users to select whether the header row is included in the csv file

CSV Write Functionality should respect formatting

When using the CSV write functionality the cells in the CSV file should be appropriately formatted based on the set type for the column. This formatting should be set so that Excel users in various locals (USA & Germany) can easily open the file.

Make sure the SAPbyDesign Connector follows the OIH standard

We have to make sure that SAPbyDesign Connector follows the OIH standard.
Tasks:
1.) Examine the actual state of the SAPbyDesign Connector
2.) Describe issues that need to be solved
3.) Estimate efforts

Failed to create CSV file with 20000 rows

I've build a simple integration flow:

then using simple load-test tool loaded 2000 request to it:

#!/bin/bash
loadtest -c 5 -n 20000 "https://in.elastic.io/hook/5de5199b20ea507fdaf1a542?message=Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum"

It run all well:

[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Max requests:        20000
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Concurrency level:   5
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Agent:               none
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Completed requests:  20000
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Total errors:        0
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Total time:          837.860013091 s
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Requests per second: 24
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Mean latency:        209.4 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO Percentage of the requests served within a certain time
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   50%      201 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   90%      213 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   95%      235 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO   99%      400 ms
[Mon Dec 02 2019 15:34:23 GMT+0100 (Central European Standard Time)] INFO  100%      723 ms (longest request)

at the end I expected a file to be uploaded to SFTP to have 20.000 rows in it:

[2019-12-02 14:34:33.287]: The resulting CSV file contains 20000 rows
[2019-12-02 14:34:33.287]: Closing the stream due to inactivity
[2019-12-02 14:34:33.290]: Emitting message {"id":"f705ff7c-b18a-4e91-83ac-5a879745d92b","attachments":{"f705ff7c-b18a-4e91-83ac-5a879745d92b.csv":{"content-type":"text/csv","url":"http://steward-service.platform.svc.cluster.local:8200/files/0375788e-5fcd-4931-8208-811c28d1ac3d"}},"body":{"rowCount":20000},"headers":{},"metadata":{}}

unfortunately resulting file was only 1 MB large and had 1795 rows in it, not 20000

Attaching result and log of the execution as ZIP file.

Archive.zip

Convert JSON into CSV

Required to create two new actions:

Write CSV attachment from JSON Object
Write CSV attachment from JSON Array

Refactor CSV using new csv parse library

The csv-parse library is outdated and does some weird things.
After discussion with @jhorbulyk also regarding the dictionary-component, it was decided it makes sense to revitalize the csv-component using papaparse as the csv parser instead.

This adds many useful features such as effective header recgonization and automatic determination of delimiter from the csv data, allowing us to simplify the metadata configuration required as well.

Add Redesigned Action: Create CSV From Message Stream

We would deprecate the existing actions of Write CSV attachment and Write CSV attachment from JSON Object in favor of this redesigned action.

Config Fields

Upload CSV as file to attachments (checkbox, optional): If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
Separator (string, optional): A single char used to delimit the CSV file. Default to ,
Column Order (string, optional): A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic.

Input Metadata

Include Headers (boolean, required): Indicates if a header row should be included in the generated file.
Input Object (object, required): Object to be written as a row in the CSV file. If the Column Order is specified, then individual properties can be specified.

Output Metadata

If Upload CSV as file to attachments is checked:
- csvString (string, required): The output CSV as a string inline in the body
If Upload CSV as file to attachments is not checked:
- attachmentUrl (string, required): A URL to the CSV output
- type (string, required): Should always be set to .csv
- size (integer, required): Size in bytes of the resulting CSV file
- attachmentCreationTime (string - ISO timestamp, required): When the attachment was generated (approximately)
- attachmentExpiryTime (string - ISO timestamp, required): When the attachment is set to expire (approximately)
- contentType (string, required): Should always be set to text/csv

Implementation Notes

This should apply similar batching logic that is applied in the old Write CSV attachment action.
We should deprecate the other two similar actions in favor of this action.

Steps to Reproduce

Create a flow of the following form. Note that executing samples for each step will function as expected.

Code component:

async function run(msg, cfg, snapshot) {
    for(let i = 0; i < 2; i++) {
      const arr = [];
      for(let  j = 0; j < 5; j++) {
        arr.push({
          hello: 'world',
          i,
          j        
        });
      }
      await this.emit('data', messages.newMessageWithBody({
        results: arr
      }));
    }
}

CSV.Write CSV Attachment from JSON array
- Include Headers: yes,
- Separators: ,
- inputArray: (JSONata mode) results
Utility Component.Attachment to String
- Attachment Id or URL: (JSONata mode) url
Email component:
- TO:
- Subject: CSV Bug
- Body: (JSONata mode) result

Publish the flow and run the flow

Expected Result

Two emails arrive with the following contents

hello,i,j
world,0,0
world,0,1
world,0,2
world,0,3
world,0,4

and

hello,i,j
world,1,0
world,2,1
world,3,2
world,4,3
world,5,4

Actual Result

Two blank emails arrive

Allow CSV files to be written with pipe separators

A customer needs to create CSV files using the ¦ character (HexCode: 0xa6) from the actions write_attachment_from_array and write_attachment_from_json.

`Error: write after end` thrown in `Write CSV attachment` action during runtime

DESCRIPTION: Error: write after end thrown in Write CSV attachment action during runtime.
Usually, it takes up to 24 hrs to reproduce the current bug

STR:

Create flow: step1 - simple trigger, generate sample manually.
step2: CSV: Write CSV attachment Action, input:
CSV Header header_1,header_2,header_3,header_4,header_5,header_6,header_7,header_8,header_9,header_10,header_11,header_12,header_13,header_14,header_15,header_16,header_17,header_18,header_19,header_20

Separators Comma

Input some values like "ajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfwajkexhwmufhwiuxfw" in generated fields

Save flow and start it.

AR:
Error: write after end error thrown in Write CSV attachment action during runtime
logs:

Error: write after end
    at writeAfterEnd (_stream_writable.js:220:12)
    at Stringifier.Writable.write (_stream_writable.js:271:5)
    at Stringifier.write (/home/apprunner/node_modules/csv-stringify/lib/index.js:199:43)
    at TaskExec.ProcessAction (/home/apprunner/lib/actions/write.js:95:17)
    at Promise (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/executor.js:22:58)
    at Promise (<anonymous>)
    at TaskExec.process (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/executor.js:21:5)
    at processMessageWithModule (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/sailor.js:285:22)
    at Sailor.doProcess (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/sailor.js:248:20)
    at doProcess.next (<anonymous>)
    at onFulfilled (/home/apprunner/node_modules/co/index.js:65:19)
    at _fulfilled (/home/apprunner/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/home/apprunner/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/home/apprunner/node_modules/q/q.js:796:13)
    at /home/apprunner/node_modules/q/q.js:857:14
    at runSingle (/home/apprunner/node_modules/q/q.js:137:13)

ER:
CSV has written successfully according to Write CSV attachment Action input data with no errors

SOAP component should support the ability to generate SOAP responses given a WSDL #3

Please refer to ...

openintegrationhub/soap-component#3

Delimiter is fixed not configurable

When using the CSV and selecting the delimiter ";" this is ignored when the execution runs:

From Logs:
[2018-12-17 11:45:53.358]: Using delimiter: ','

"Write CSV attachment from JSON Array" input array path should be configurable.

Action "Write CSV attachment from JSON Array" should have configuration for the path to the array in input message.

Disappeared icons of the CSV-component on the flow

Precondition: flow CSV_read=>CSV_write=>SFTP exists
version of CSV=old version
STR:

Make git push new version of the CSV-component
Open our flow

AR: Icons and settings of the CSV-steps are empty. Icor of SFTP-step is present. see the screen

But flow is still work correct
ER: everything looks as usual

Request CA system reset to default values

we need to ask Channel Advisor to reset the Test our accounts

TypeError: timeout is not a function

The following line produces the mentioned error:

https://github.com/elasticio/csv-component/blob/master/lib/actions/write.js#L55

See how it looks in the UI

Update the readme

Update the readme file to include all the necessary configurations and remove/change the dead links.

How about to emit "error" in case of issue with attachments API?

There is a flow: CSV(read)-CSV(write)-SFTP
We had a situation with csv(write), when step looks like successful executed - with green number of messages, but an error was in the logs:

[2018-11-16 14:10:11.644]: Starting slug download ...
[2018-11-16 14:10:13.342]: Successfully downloaded and extracted slug file
[2018-11-16 14:10:13.386]: Writing config vars to .profile.d/config_vars
[2018-11-16 14:10:13.413]: Detected 256 MB available memory
[2018-11-16 14:10:13.413]: Detected 0.1 CPU available
[2018-11-16 14:10:13.418]: Starting the application with node ./node_modules/elasticio-sailor-nodejs/run.js
[2018-11-16 14:10:15.049]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"msg":"About to initialize component for execution","time":"2018-11-16T12:10:15.047Z","v":0}
[2018-11-16 14:10:15.201]: Using delimiter: ','
[2018-11-16 14:10:15.201]: Configured column names: [ 'message', 'time' ]
[2018-11-16 14:10:15.339]: CSV file to be uploaded file to uri=http://steward-service.platform.svc.cluster.local:8200/files/57356501-c76a-4a1c-baf4-6e26de74a4ac
[2018-11-16 14:10:15.341]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"msg":"Component execution initialized successfully","time":"2018-11-16T12:10:15.341Z","v":0}
[2018-11-16 14:10:15.351]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":1,"messageProcessingTime":0,"msg":"processMessage received","time":"2018-11-16T12:10:15.350Z","v":0}
[2018-11-16 14:10:15.353]: Writing Row: {"message":"hello world60","time":"2018-11-16T12:08:36.477Z"}
[2018-11-16 14:10:15.353]: Incoming data: {"time":"2018-11-16T12:08:36.477Z","message":"hello world60"}
[2018-11-16 14:10:15.356]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":0,"errorCount":0,"promises":0,"messageProcessingTime":5,"msg":"processMessage emit end","time":"2018-11-16T12:10:15.355Z","v":0}
[2018-11-16 14:10:23.491]: at IncomingMessage.<anonymous> (/app/node_modules/request/request.js:1085:12)
[2018-11-16 14:10:23.491]: at Object.onceWrapper (events.js:314:30)
[2018-11-16 14:10:23.491]: at endReadableNT (_stream_readable.js:1045:12)
[2018-11-16 14:10:23.491]: at Request.self.callback (/app/node_modules/request/request.js:186:22)
[2018-11-16 14:10:23.491]: at Request.emit (events.js:213:7)
[2018-11-16 14:10:23.491]: at Request.<anonymous> (/app/node_modules/request/request.js:1163:10)
[2018-11-16 14:10:23.491]: at emitOne (events.js:115:13)
[2018-11-16 14:10:23.491]: at _combinedTickCallback (internal/process/next_tick.js:102:11)
[2018-11-16 14:10:23.491]: at process._tickCallback (internal/process/next_tick.js:161:9)
**[2018-11-16 14:10:23.491]: Unhandled rejection StatusCodeError: 500 - "Internal Server Error"**
[2018-11-16 14:10:23.491]: at new StatusCodeError (/app/node_modules/request-promise-core/lib/errors.js:32:15)
[2018-11-16 14:10:23.491]: at Request.plumbing.callback (/app/node_modules/request-promise-core/lib/plumbing.js:104:33)
[2018-11-16 14:10:23.491]: at Request.RP$callback [as _callback] (/app/node_modules/request-promise-core/lib/plumbing.js:46:31)
[2018-11-16 14:10:23.491]: at emitTwo (events.js:125:13)
[2018-11-16 14:10:23.491]: at Request.emit (events.js:210:7)
[2018-11-16 14:10:23.491]: at emitNone (events.js:110:20)
[2018-11-16 14:10:23.491]: at IncomingMessage.emit (events.js:207:7)
[2018-11-16 14:10:25.357]: Closing the stream due to inactivity
[2018-11-16 14:10:25.357]: The resulting CSV file contains 1 rows
[2018-11-16 14:10:25.358]: Emitting message {"id":"997347d0-e998-11e8-a704-21ae36f35c64","attachments":{"997347d0-e998-11e8-a704-21ae36f35c64.csv":{"content-type":"text/csv","url":"http://steward-service.platform.svc.cluster.local:8200/files/57356501-c76a-4a1c-baf4-6e26de74a4ac"}},"body":{"rowCount":1},"headers":{},"metadata":{}}
[2018-11-16 14:10:25.358]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":0,"messageProcessingTime":10007,"msg":"processMessage emit data","time":"2018-11-16T12:10:25.357Z","v":0}
[2018-11-16 14:10:25.362]: Configured column names: [ 'message', 'time' ]
[2018-11-16 14:10:25.362]: Using delimiter: ','
[2018-11-16 14:10:25.515]: CSV file to be uploaded file to uri=http://steward-service.platform.svc.cluster.local:8200/files/5c5b17c3-2a97-4d71-afe9-e1724e631d59
[2018-11-16 14:11:12.375]: Received SIGTERM
[2018-11-16 14:11:12.376]: Disconnecting
[2018-11-16 14:11:12.378]: Successfully disconnected
[2018-11-16 14:11:12.386]: [INFO  tini (1)] Main child exited normally (with status '0')
[2018-11-16 14:11:12.386]: [INFO  tini (1)] Spawned child process '/runner/init' with pid '7'

Component must report errors

CSV written with invalid data type

Description: usage new library instead of current one should probably fix this behavior: #46

STR:

write file, input:
CSV Header*: MATERIAL,GRID,NAME,WHOLESALE,CURRENCY,ATP,ATS,IS_NEW_MATERIAL,QUANTITY,,,,,,,,,,IMAGE,,
all data types are default except for:
retrieve sample
add next step with Read action: use same headers and data types as in write CSV action
retrieve sample
AR:
{
"material": "wdwedwe",
"grid": "uyuuiiuiu",
"name": "iujuijiujuih",
"wholesale": "iyuyuyunh",
"currency": "ewcweciu",
"atp": "uiyiuyuy",
"ats": "yuiyuyuiyui",
"is_new_material": "uiyuiyuyiu",
"quantity": "uuhuihuh",
"column_9": "ntntnt",
"column_10": "",
"column_11": "tytyf",
"column_12": "tyfytft",
"column_13": "tfyf",
"column_14": 1234567,
"column_15": "1",
"column_16": "",
"column_17": "hghg",
"image": "vgvgv",
"column_19": "7887787y7h",
"column_20": "~!@#$%^&*()"
}

ER:

{
"material": "wdwedwe",
"grid": "uyuuiiuiu",
"name": "iujuijiujuih",
"wholesale": "iyuyuyunh",
"currency": "ewcweciu",
"atp": "uiyiuyuy",
"ats": "yuiyuyuiyui",
"is_new_material": "uiyuiyuyiu",
"quantity": "uuhuihuh",
"column_9": "ntntnt",
"column_10": "",
"column_11": "tytyf",
"column_12": "tyfytft",
"column_13": "tfyf",
"column_14": 1234567,
"column_15": true,
"column_16": false,
"column_17": "hghg",
"image": "vgvgv",
"column_19": "7887787y7h",
"column_20": "~!@#$%^&*()"
}

Fix issue "404 - "File's metadata is not found" with bulk cud action (SalesForce component)

#73

Add Redesigned Action: Create CSV From JSON Array

We should deprecate write_attachment_from_array with this action once this action is added.

Config Fields

Upload CSV as file to attachments (checkbox, optional): If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
Separator (string, optional): A single char used to delimit the CSV file. Default to ,
Column Order (string, optional): A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic.

Input Metadata

Include Headers (boolean, required): Indicates if a header row should be included in the generated file.
Input Array (Array of Objects, required): Array of objects to be written as rows in the CSV file. (One row per object + headers) If the Column Order is specified, then individual properties can be specified.

Output Metadata

If Upload CSV as file to attachments is checked:
- csvString (string, required): The output CSV as a string inline in the body
If Upload CSV as file to attachments is not checked:
- attachmentUrl (string, required): A URL to the CSV output
- type (string, required): Should always be set to .csv
- size (integer, required): Size in bytes of the resulting CSV file
- attachmentCreationTime (string - ISO timestamp, required): When the attachment was generated (approximately)
- attachmentExpiryTime (string - ISO timestamp, required): When the attachment is set to expire (approximately)
- contentType (string, required): Should always be set to text/csv

Implementation Notes

Related API Documentation

https://www.papaparse.com/docs#json-to-csv

Sync handling for creating attachments

Required to add sync method - to wait response from steward, when the attachment is ready to download.

Update Sailor to 2.6.7

PR #69

Trouble with using write csv component

STR:

Create the first step for flow. In my case, it is Nodejs.code:
// Create message to be emitted
function send(i, name, delay) {
setTimeout(() => {
const data = {
message: 'hello world',
name,
i
};
console.log('\n emitting', data);
// Emit the data event
emitter.emit('data', messages.newMessageWithBody(data));
}, delay);
}

send(1, 'f-1', 100);
send(2, 'f-1', 3000);
send(3, 'f-1', 5000);

send(4, 'f-2', 16000);

send(5, 'f-3', 28000);

// No need to emit end
console.log('Finished execution');

Add next step - CSV-component. You can see Configure Input in the screen
Click "Test this step"

AR: there is an error in the log and I can't finish my flow
Starting your Docker container ...
Starting application in apprunner
Not Available
Starting slug download ...
Successfully downloaded and extracted slug file
Writing config vars to .profile.d/config_vars
Detected 0.01 CPU available
Starting the application with node ./node_modules/elasticio-sailor-nodejs/run.js
Initializing component for execution
Initializing component for execution
Incoming message is {"passthrough":{"step_1":{"metadata":{},"headers":{},"body":{"i":1,"name":"f-1","message":"hello world"},"attachments":{},"id":"615ecfdc-33db-4b8c-ab64-a67164b48f04"}},"metadata":{},"headers":{},"body":{"i":1,"name":"f-1","message":"hello world"},"attachments":{},"id":"615ecfdc-33db-4b8c-ab64-a67164b48f04"}
URL of the CSV is missing
Process function is a Promise/generator/etc
at TaskExec.onData (/app/node_modules/elasticio-sailor-nodejs/lib/sailor.js:227:38)
TypeError: Cannot create property 'passthrough' on boolean 'true'
at TaskExec.emit (events.js:188:7)
WARNING: End was emitted more than once!
at onPromisedData (/app/node_modules/elasticio-sailor-nodejs/lib/executor.js:31:18)
at emitOne (events.js:96:13)
at process._tickCallback (internal/process/next_tick.js:103:7)

Error happened: Cannot read property 'separator' of undefined

The flow:

With the latest version of the component ( v19 ), when you retrieve a sample, you get


at onFulfilled (/home/apprunner/node_modules/co/index.js:65:19)

at runSingle (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:137:13)

[INFO  tini (1)] Main child exited normally (with status '1')

Error happened: Cannot read property 'separator' of undefined

at Sailor.gen (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/sailor.js:147:20)

at gen.next (<anonymous>)

at _fulfilled (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:834:54)

at flush (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:125:13)

at _combinedTickCallback (internal/process/next_tick.js:95:7)

at Promise (/home/apprunner/node_modules/elasticio-sailor-nodejs/lib/sailor.js:149:51)

at self.promiseDispatch.done (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:863:30)

at /home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:857:14

at process._tickCallback (internal/process/next_tick.js:161:9)

at Object.init (/home/apprunner/lib/actions/write.js:25:31)

at Promise (<anonymous>)

at Promise.promise.promiseDispatch (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:796:13)

TypeError: Cannot read property 'separator' of undefined

at onFulfilled (/home/apprunner/node_modules/co/index.js:65:19)

at runSingle (/home/apprunner/node_modules/elasticio-sailor-nodejs/node_modules/q/q.js:137:13)

[INFO  tini (1)] Main child exited normally (with status '1')```

Add unit tests for write CSV

Modernize tests and add them to build process

Add `Emit all messages` feature to Read CSV file from URL trigger

You may see how this feature is implemented in Read CSV action

Emit all property for CSV component

CSV component should have the opportunity to emit all rows in one message:

{
 "result": [ 
    {"Column1":"1"},
    {"Column1":"2"}
  ]
}

PR: #34

Steward responds with 404, even a file is there

The problem is with Read action of CSV from an attachment. The steward returns signed-URL faster than file is really created.

The actual file exists on the server storage and confirmed to be identical to the original file.

As a solution, we should add retry mechanism into the action and/or upgrade version of sailor-nodejs to implement in-built retry mechanism.

[2019-06-24 14:02:01.394]: Found attachment key=input/depotbestaende/full/Depotbestaende_2019-06-20.csv attachment={"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}
[2019-06-24 14:02:01.394]: Incoming message is {"id":"3d2b27ff-8cf0-4711-b7fd-d750bdeaf896","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee","passthrough":{"step_1":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"Filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv","_query":{}},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"},"mapper:step_1:step_2":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"}}},"headers":{},"metadata":{},"passthrough":{"step_1":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"Filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv","_query":{}},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"},"mapper:step_1:step_2":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"},"step_2":{"id":"3d2b27ff-8cf0-4711-b7fd-d750bdeaf896","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee","passthrough":{"step_1":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"Filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv","_query":{}},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"},"mapper:step_1:step_2":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"}}},"headers":{},"metadata":{}},"step_8":{"id":"3d2b27ff-8cf0-4711-b7fd-d750bdeaf896","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{"input/depotbestaende/full/Depotbestaende_2019-06-20.csv":{"url":"http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f","size":58224,"content-type":"text/csv"}},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee","passthrough":{"step_1":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"Filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv","_query":{}},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"},"mapper:step_1:step_2":{"id":"7a4c7c80-966f-11e9-be5e-d50432ac7d22","attachments":{},"body":{"filename":"input/depotbestaende/full/Depotbestaende_2019-06-20.csv"},"headers":{"host":"in.elastic.io","x-request-id":"d0a36ec0d3e9a2169a7e58a0c9e54df1","x-real-ip":"10.4.20.218","x-forwarded-for":"10.4.20.218","x-forwarded-host":"in.elastic.io","x-forwarded-port":"443","x-forwarded-proto":"https","x-original-uri":"/hook/5c9dff988d9cbc00132a5fee","x-scheme":"https","content-length":"70","content-type":"application/json","authorization":"Basic ZWxhc3RpYzpEMkhoSnRSaFNWNWtCRVduTkduQQ==","accept-encoding":"gzip, deflate"},"metadata":{},"url":"/hook/5c9dff988d9cbc00132a5fee","method":"POST","originalUrl":"/hook/5c9dff988d9cbc00132a5fee","query":{},"params":{},"pathSuffix":"/hook/5c9dff988d9cbc00132a5fee","taskId":"5c9dff988d9cbc00132a5fee"}}},"headers":{"X-EIO-Routing-Key":"5c7d1a5998ba4a0011e801c6.5c9dff988d9cbc00132a5fee/ordinary.step_13.input"},"metadata":{}}}}
[2019-06-24 14:02:01.395]: Mon, 24 Jun 2019 11:02:01 GMT csv Sending GET request to url=http://steward-service.platform.svc.cluster.local:8200/files/5ad8de16-92d3-4821-9a64-bd87b930e36f
[2019-06-24 14:02:01.458]: Mon, 24 Jun 2019 11:02:01 GMT csv Have got response status=404 headers={"content-type":"text/plain; charset=utf-8","content-length":"28","date":"Mon, 24 Jun 2019 11:02:01 GMT","connection":"close"}

Add action function to download/read CSV from URL

At the moment the CSV component can read CSV file from a remote URL as an Trigger function. This is limiting the usability of t he component to read the CSV file only when triggered by the cron job of flow scheduling.

We need to extend the component so an Action function can also read the CSV from a remote URL. This would be beneficial in cases when the read is triggered by an external event like a call to webhook to read the file and process is it further.

Unnecessary configuration option for CSV Write

When configuring CSV Write action, skip-rows option is not required:

It was needed when reading CSV file as it might contain a header row(s) that have to be ignored. When writing CSV this option is not necessary, and potentially could be replaced with the checkbox that will configure if header row would be generated or not.

Add the steward URL to the body of the outgoing message

Additional feature that can be useful for different use cases

CSV Write: We should allow users to add headers which contain spaces and capital letters

Component should be able to handle attachment information via the message body

Add Redesigned Action: Read

Config Fields

Emit Behavior (Dropdown with Fetch All & Emit Individually, required): Indicates if rows should be emitted in a single message with an array or one at a time.

Input Metadata

Attachment URL (string, required): A URL to an attachment which should be a CSV file
The following CSV parse options:
- First row contains headers (boolean, required): Corresponds to the header option in papaparse
- Delimiter (string, optional, default to autodetect):

Output Metadata

When Emit Behavior is Fetch All, the output should be an object with a single property results which is an array where each item in the array is an object representing a row in the CSV file.
When Emit Behavoir is Emit Individually, each message body should be an object representing a row of the CSV file.

Implementation Notes

We should deprecate the existing read action in favor of this action.
It is suggested that we migrate to papaparse when building this new action.

Implement verification that incoming is CSV for Read CSV trigger and action

Description:
First task purpuse: throw crear error when specified URI does not return .csv file (currently - timeout error)
Second task purpuse: currently action may try to read incoming non csv attachment, obviosly it does not return something usefull:

{
  "test csv header": "%PDF-1.4"
}

So we may add this error handling for case when user tries to pass something that is not csv attachment by mistake

as soon as e.io mapping will adopt it (see https://github.com/elasticio/mapper/issues/18) we should test if we could use it in CSV Write - formatting numbers with e.g. '$' signs so that 10.248 would become $10,25

elasticio / csv-component Goto Github PK

csv-component's People

Contributors

Stargazers

Watchers

Forkers

csv-component's Issues

Add Redesigned Action: Create CSV From Message Stream

Config Fields

Input Metadata

Output Metadata

Implementation Notes

Related API Documentation

Steps to Reproduce

Expected Result

Actual Result

Add Redesigned Action: Create CSV From JSON Array

Config Fields

Input Metadata

Output Metadata

Implementation Notes

Related API Documentation

Config Fields

Input Metadata

Output Metadata

Implementation Notes

Recommend Projects

Recommend Topics

Recommend Org