Git Product home page Git Product logo

bootstrap-archived's Introduction

[ARCHIVED] Stedi EDI Bootstrap

Note This repository should only be relied on for existing users. New users should not deploy this version of bootstrap, and instead should use Stedi Core to send and receive EDI files. Please don't hesitate to contact us with any questions!

Previous README contents

This repository contains an end-to-end configuration for building an X12 EDI system using Stedi products. This implementation demonstrates one way to build an integration for common EDI read and write use cases. Your solution may differ depending on your systems and requirements.

We strongly recommend reviewing the documentation for Stedi Core before deploying the bootstrap implementation.

Hands-on support

We'd like to help set up and customize the bootstrap repository with you. Working together helps us understand what Stedi customers need and helps get your solution into production as quickly as possible. We offer free hands-on support that includes:

  • Help deploying the bootstrap workflows and customizing them for your use cases
  • Best practices for designing scalable connections between Stedi and your systems
  • EDI experts to answer your questions
  • Live troubleshooting over Slack or video call

Contact us to get started.

Bootstrap read and write workflow

The Stedi Core module ingests data and emits events with the results of its conversion and validation processing. For example, Core emits an event when it receives a new file or successfully processes a transaction set.

To create a custom end-to-end EDI system on Stedi, you need to automate tasks like adding files from your input buckets and reacting to the emitted events. For example, you may want to automatically forward translated EDI files to an API, FTP server, AS2 server, or a Stedi function to run custom code.

Bootstrap contains opinionated Stedi functions that you can customize through configuration. For example, you can add Destinations where the bootstrap workflows will send incoming and outgoing data. The following sections describe these built-in functions.

Inbound EDI workflow

The edi-inbound function listens to Stedi Core transaction.processed events, which contain the partnership, document direction, location of the translated document, and document transaction set ID for a single EDI transaction set. When it receives an event, it performs the following steps:

  1. Read the translated EDI-like JSON data from the Stedi bucket configured to receive Core output.
  2. Look up configured destinations for the specific partnership and transaction set ID. Refer to Destinations for details.
  3. If a destination has a Stedi Mapping configured, apply the mapping transformation the JSON.
  4. Send the JSON to each destination.
  5. Send failures (such as invalid mappings or missing guides) to Execution Error Destinations.
  6. Retry function execution failures 2 more times. These retries can result in destinations receiving multiple messages, so you must handle at-least-once message delivery separately. We recommend using payload control numbers and message timestamps for deduplication.

Outbound EDI workflow

The edi-outbound function performs the following steps when it receives a payload and a metadata object. The payload must match the shape of the Guide's JSON Schema for writing EDI. Or, if a mappingId is specified, then the payload is the input for a mapping which will output valid Guide JSON data.

  1. Use the metadata to look up the configuration values required to construct an EDI envelope. The partnershipId is the only required field.
  2. If a Stedi Mapping is specified, apply the mapping transformation to the JSON.
  3. Call Stedi EDI Translate to transform the JSON payload into an EDI file.
  4. Look up configured destinations for the specific partnership and transaction set ID. Refer to Destinations for details.
  5. Send failures to Execution Error Destinations.
  6. Retry function execution failures 2 or more times. These retries can result in destinations receiving multiple messages, so you must handle at-least-once message delivery separately. We recommend using payload control numbers and message timestamps for deduplication.

Processed functional groups workflow

The edi-acknowledgement function listens to Stedi Core functional_group.processed inbound events, which contain the partnership, document direction, and envelope data for a single functional group. When it receives an event, the function performs the following steps:

  1. If the direction is RECEIVED, look up up 997 acknowledgment configuration for the specific partnership and transaction set Ids in the functional group. Refer to Acknowledgments for details.
  2. If transaction sets are in the functional group with 997 acknowledgments configured, generate a 997 EDI-like JSON file and send it to the edi-outbound function for processing.

File error workflow

The events-file-error function listens to Stedi Core file.failed events, which Stedi emits when there is an error processing a file. When it receives an event, the function performs the following steps:

  1. Look up the configured file error destinations. Refer to File Error Destinations for details.
  2. Forward the errors to each destination.

Requirements

  1. Install Node.js (minimum version: 18)

  2. Clone the bootstrap repository and install the necessary dependencies:

    git clone https://github.com/Stedi-Demos/bootstrap.git
    cd bootstrap
    npm ci
  3. Create a Stedi account and enable Core. Bootstrap does not overwrite existing Core settings or data.

  4. Rename the bootstrap's .env.example file to .env and update the following environment variables:

    • STEDI_API_KEY: A Stedi API key is required for authentication. You can generate an API key in your Stedi account.
    • DESTINATION_WEBHOOK_URL: Go to webhook.site and copy the unique URL. The bootstrap workflow sends output to this webhook.

    Example .env file

    STEDI_API_KEY=<YOUR_STEDI_API_KEY>
    DESTINATION_WEBHOOK_URL=<YOUR_WEBHOOK_URL>
    

Deploying bootstrap resources

Run the following command in the bootstrap directory:

npm run bootstrap

Bootstrap creates resources in your Stedi account. It may take several minutes for Stedi to deploy all required resources. When deployment is finished, your CLI displays a message similar to the following.

Example CLI output (click to expand):
[email protected] bootstrap
npm run configure-storage && npx ts-node-esm ./src/setup/bootstrap.ts && npm run deploy


[email protected] configure-storage
npm run ensure-keyspaces-exist && npm run configure-buckets

[email protected] ensure-keyspaces-exist
ts-node-esm ./src/setup/bootstrap/ensureKeyspacesExist.ts

Creating keyspaces...
Waiting for keyspaces to become active...
Waiting for keyspaces to become active...

[email protected] configure-buckets
ts-node-esm ./src/setup/configureBuckets.ts

Configuring buckets...
done
Guide created: 01GZM3TCTWE94FQZWTHDR1HP8K
Guide created: 01GZM3TD8C4233TNVZFBADNS7E
Creating X12 Trading Partner Profile in Partners API

[email protected] deploy
ts-node-esm ./src/setup/deploy.ts

Waiting for function deploys to complete
Finished deploying function: edi-acknowledgment
Finished deploying function: events-file-error
Finished deploying function: edi-outbound
Finished deploying function: csv-to-json
Finished deploying function: ftp-external-poller
Finished deploying function: csv-from-json
Finished deploying function: edi-inbound
Creating event bindings
Waiting for event binding deploys to complete
Deploy completed at: 5/4/2023, 3:35:21 PM

Resources

Bootstrap deploys the following resources to your account:

  • A Stedi bucket with directories for you _stedi and trading partners trading_partners. In the trading_partners directory is a single directory for a fictional trading partner called ANOTHERMERCH. Within ANOTHERMERCH, are directories for inbound and outbound files. The inbound directory is where your partners would drop new files to send to you and the outbound directory is where Stedi sends EDI files generated for that trading partner.
  • A Stash keyspace called partners-configuraion. This keyspace contains configuration for bootstrap, including destinations for incoming and outgoing files. If you click the destinations|this-is-me_another-merchant|855 key/value pair, you'll see that bootstrap is configured to send incoming 855 EDI documents to your webhook.
  • Several Stedi functions, including functions required to read and write EDI and generate functional acknowledgements.

Testing the workflows

Inbound EDI

Core automatically processes new files in the designated bucket for incoming data. When Core processes a file, it emits events that automatically invoke the edi-inbound function for each processed transaction set.

  1. Go to the Buckets UI and navigate to the inbound directory for your trading partner: <SFTP_BUCKET_NAME>/trading_partners/ANOTHERMERCH/inbound

  2. Upload the input X12 5010 855 EDI document to this directory.

  3. Look for the output of the function wherever you created your test webhook. The function sends the translated JSON payload to the endpoint you configured.

    Example webhook output (click to expand):
    {
      "envelope": {
        "interchangeHeader": {
          "authorizationInformationQualifier": "00",
          "authorizationInformation": "          ",
          "securityQualifier": "00",
          "securityInformation": "          ",
          "senderQualifier": "14",
          "senderId": "ANOTHERMERCH   ",
          "receiverQualifier": "ZZ",
          "receiverId": "THISISME       ",
          "date": "2022-09-14",
          "time": "20:22",
          "repetitionSeparator": "U",
          "controlVersionNumber": "00501",
          "controlNumber": "000001746",
          "acknowledgementRequestedCode": "0",
          "usageIndicatorCode": "T",
          "componentSeparator": ">"
        },
        "groupHeader": {
          "functionalIdentifierCode": "PR",
          "applicationSenderCode": "ANOTAPPID",
          "applicationReceiverCode": "MYAPPID",
          "date": "2022-09-14",
          "time": "20:22:22",
          "controlNumber": "000001746",
          "agencyCode": "X",
          "release": "005010"
        },
        "groupTrailer": {
          "numberOfTransactions": "1",
          "controlNumber": "000001746"
        },
        "interchangeTrailer": {
          "numberOfFunctionalGroups": "1",
          "controlNumber": "000001746"
        }
      },
      "transactionSets": [
        {
          "heading": {
            "transaction_set_header_ST": {
              "transaction_set_identifier_code_01": "855",
              "transaction_set_control_number_02": 1
            },
            "beginning_segment_for_purchase_order_acknowledgment_BAK": {
              "transaction_set_purpose_code_01": "00",
              "acknowledgment_type_02": "AD",
              "purchase_order_number_03": "365465413",
              "date_04": "2022-09-14",
              "date_09": "2022-09-13"
            },
            "reference_information_REF": [
              {
                "reference_identification_qualifier_01": "CO",
                "reference_identification_02": "ACME-4567"
              }
            ],
            "party_identification_N1_loop_ship_to": [
              {
                "party_identification_N1": {
                  "entity_identifier_code_01": "ST",
                  "name_02": "Wile E Coyote",
                  "identification_code_qualifier_03": "92",
                  "identification_code_04": "DROPSHIP CUSTOMER"
                },
                "party_location_N3": [
                  {
                    "address_information_01": "111 Canyon Court"
                  }
                ],
                "geographic_location_N4": {
                  "city_name_01": "Phoenix",
                  "state_or_province_code_02": "AZ",
                  "postal_code_03": "85001",
                  "country_code_04": "US"
                }
              }
            ],
            "party_identification_N1_loop_selling_party": [
              {
                "party_identification_N1": {
                  "entity_identifier_code_01": "SE",
                  "name_02": "Marvin Acme",
                  "identification_code_qualifier_03": "92",
                  "identification_code_04": "DROPSHIP CUSTOMER"
                },
                "party_location_N3": [
                  {
                    "address_information_01": "123 Main Street"
                  }
                ],
                "geographic_location_N4": {
                  "city_name_01": "Fairfield",
                  "state_or_province_code_02": "NJ",
                  "postal_code_03": "07004",
                  "country_code_04": "US"
                }
              }
            ]
          },
          "detail": {
            "baseline_item_data_PO1_loop": [
              {
                "baseline_item_data_PO1": {
                  "assigned_identification_01": "item-1",
                  "quantity_02": 8,
                  "unit_or_basis_for_measurement_code_03": "EA",
                  "unit_price_04": 400,
                  "product_service_id_qualifier_06": "VC",
                  "product_service_id_07": "VND1234567",
                  "product_service_id_qualifier_08": "SK",
                  "product_service_id_09": "ACM/8900-400"
                },
                "product_item_description_PID_loop": [
                  {
                    "product_item_description_PID": {
                      "item_description_type_01": "F",
                      "description_05": "400 pound anvil"
                    }
                  }
                ],
                "line_item_acknowledgment_ACK_loop": [
                  {
                    "line_item_acknowledgment_ACK": {
                      "line_item_status_code_01": "IA",
                      "quantity_02": 8,
                      "unit_or_basis_for_measurement_code_03": "EA"
                    }
                  }
                ]
              },
              {
                "baseline_item_data_PO1": {
                  "assigned_identification_01": "item-2",
                  "quantity_02": 4,
                  "unit_or_basis_for_measurement_code_03": "EA",
                  "unit_price_04": 125,
                  "product_service_id_qualifier_06": "VC",
                  "product_service_id_07": "VND000111222",
                  "product_service_id_qualifier_08": "SK",
                  "product_service_id_09": "ACM/1100-001"
                },
                "product_item_description_PID_loop": [
                  {
                    "product_item_description_PID": {
                      "item_description_type_01": "F",
                      "description_05": "Detonator"
                    }
                  }
                ],
                "line_item_acknowledgment_ACK_loop": [
                  {
                    "line_item_acknowledgment_ACK": {
                      "line_item_status_code_01": "IA",
                      "quantity_02": 4,
                      "unit_or_basis_for_measurement_code_03": "EA"
                    }
                  }
                ]
              }
            ]
          },
          "summary": {
            "transaction_totals_CTT_loop": [
              {
                "transaction_totals_CTT": {
                  "number_of_line_items_01": 2
                }
              }
            ],
            "transaction_set_trailer_SE": {
              "number_of_included_segments_01": 17,
              "transaction_set_control_number_02": "0001"
            }
          }
        }
      ],
      "delimiters": {
        "element": "*",
        "composite": ">",
        "repetition": "U",
        "segment": "~"
      }
    }

Outbound EDI

You can invoke the edi-outbound function through the UI for testing.

  1. Navigate to the edi-outbound function in the (Functions UI)https://www.stedi.com/app/functions/edi-outbound/edit.

  2. Click the Edit execution payload link, and paste the contents of src/resources/X12/5010/850/outbound.json into the payload modal, and click save.

  3. Click Execute and choose the Synchronous option. If successful the Output should look similar to the following:

    Example function output (click to expand):
    {
      "statusCode": 200,
      "deliveryResults": [
        {
          "type": "bucket",
          "payload": {
            "bucketName": "<STEDI_ACCOUNT_ID>-sftp",
            "key": "trading_partners/ANOTHERMERCH/outbound/1-850.edi",
            "body": "ISA*00*          *00*          *ZZ*THISISME       *14*ANOTHERMERCH   *230113*2027*U*00501*000000005*0*T*>~GS*PO*MYAPPID*ANOTAPPID*20230113*202727*000000005*X*005010~ST*850*0001~BEG*00*DS*365465413**20220830~REF*CO*ACME-4567~REF*ZZ*Thank you for your business~PER*OC*Marvin Acme*TE*973-555-1212*EM*[email protected]~TD5****ZZ*FHD~N1*ST*Wile E Coyote*92*123~N3*111 Canyon Court~N4*Phoenix*AZ*85001*US~PO1*item-1*0008*EA*400**VC*VND1234567*SK*ACM/8900-400~PID*F****400 pound anvil~PO1*item-2*0004*EA*125**VC*VND000111222*SK*ACM/1100-001~PID*F****Detonator~CTT*2~AMT*TT*3700~SE*16*0001~GE*1*000000005~IEA*1*000000005~"
          }
        }
      ]
    }
  4. You can view the file using the Buckets UI. The output of the function includes the bucketName and key (path within the bucket) of where the function saved the generated EDI.

Customizing the workflows

The bootstrap workflow uses sample Partners, a Partnership associating the two partners, and configuration values for destinations configured in Stash to set up and test the read and write EDI workflows. You can customize the bootstrap workflow by doing one or all of the following:

  • Edit a partner profile to replace the test trading partner with your real trading partners' details and requirements.
  • Customize configuration in Stash. Add partnership and transaction set configurations for partnerships, set one or more destinations for a given configurations, forward errors to external services or archive in buckets, configure mappings, and send 997 acknowledgments.
  • Create Stedi mappings. Add a mappingId property to a Stash destination configuration to transform the inbound payload before sending to a destination. Or, when sending EDI, the mappingId can transform the event payload into the JSON schema required for translating to EDI.
  • Create SFTP users for your trading partners, so they can send and retrieve EDI documents from Stedi Buckets.

You may want to use additional Stedi products to further optimize your EDI workflows. We can help you customize the bootstrap workflow and determine which products and approaches are right for your use cases. Contact us to set up a meeting with our technical team.

Poll remote FTP / SFTP servers

You can poll remote FTP and SFTP servers to download files from your trading partners. Visit the External FTP / SFTP poller README for details.

Clean up bootstrap resources

To delete all the resources created by the bootstrap, run the following command:

npm run destroy

Stash configuration

Stedi Stash is a key/value store. You can add and edit Stash key-value pairs to configure destinations for incoming and outgoing documents, destinations for errors, and which transaction sets require functional acknowledgements.

Destinations

Transaction set destination

key: destinations|${partnershipId}|${transactionSetId}

value: JSON Schema

Execution error destinations

key: destinations|errors|execution

value: JSON Schema

File error destinations

key: destinations|errors|execution

value: JSON Schema

Acknowledgment configuration

key: functional_acknowledgments|${partnershipId}

value: JSON Schema

Troubleshooting

There was an issue installing the dependencies using your local npm installation, please check your .npmrc and try again.

If you created a .npmrc in this repository, please remove it.

If you still see this error, you may have a registry override in your npm config. Run npm config list and search for registry like below. Comment out that line and try again.

@stedi:registry = "https://npm.pkg.github.com/" 

bootstrap-archived's People

Contributors

bahrmichael avatar bdq avatar dkanter14 avatar eliotslevin avatar joost-basic-bits avatar kasiafojucik avatar lamarrd avatar laurapacilio avatar pdiazvargas avatar rosswilliams avatar rsioss avatar zackkanter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

bootstrap-archived's Issues

add support for `function` destination type

  • a Stedi Function can be invoked as a destination to allow an escape hatch for other things that aren't supported out of the box (such as sending data to a webhook as FormData, for example)

[bug] `edi-acknowledgment` function should raise exception when a destination is not configured for 997s

Currently generating 997s requires two Stash keys, if keyfunctional_acknowledgments|this-is-me_another-merchant is defined, then the edi-acknowledgment function will be successful. It will called edi-outbound which will fail if key destinations|this-is-me_another-merchant|997 is undefined, or invalid.

It would be helpful if edi-acknowledgment generates an error if the key destinations|this-is-me_another-merchant|997 is undefined or invalid.

Add support for user-defined headers for `webhook` destinations

We currently support basic webhook destinations (unauthenticated). It should be expected that customers will want to include support for some form of authentication to their internal API webhook destinations. Adding support for user-defined headers seems like a fairly straightforward initial solution for this.

Support dynamically choosing a guideId based on the GS-08

For X12 HIPAA Guides, there are multiple "types" of transaction sets. For example, the 837 has a "professional" and "institutional" 837. These are indicated in the GS-08:

GS-08 for professional = "005010X222"
GS-08 for institutional = "005010X223"

The bootstrap repo cannot dynamically choose which guide to use based on the full release with industry extensions.

  • Allow customer to add the release (with extension) in partnership config in Stash when they want to select a guide for a specific release

CC @RossWilliams

[bug] .env file creation causes confusion

Steps to reproduce:

  1. Clone the bootstrap repository and install the necessary dependencies:
git clone https://github.com/Stedi-Demos/bootstrap.git
cd bootstrap
npm ci
  1. Store your Stedi API key and webhook destination URL as environment variables:
export STEDI_API_KEY=<YOUR_STEDI_API_KEY>
export DESTINATION_WEBHOOK_URL=<YOUR_WEBHOOK_URL>
  1. Run npm bootstrap
  2. Bootstrap will deploy your Functions without the STEDI_API_KEY environment variable, and your local .env file will only have the following:
SFTP_BUCKET_NAME=<SFTP_BUCKET_NAME>
EXECUTIONS_BUCKET_NAME=<EXECUTIONS_BUCKET_NAME>
CORE_INGESTION_BUCKET_NAME=<INGESTION_BUCKET_NAME>

Add bulk retry support for inbound file processing

For inbound files that get "stuck" (due to validation errors when translating, for example), it would be great to offer a mechanism for bulk retries. There are multiple approaches that could be used for this:

  • bulk download / re-upload objects
  • construct new bucket notification event payloads for each file (or even a batch of files) and directly invoke the inbound function with those payloads

[enhancement] templatize AS2 provisioning

Rough outline of how AS2 provisioning could work:

export PARTNERSHIP=this-is-me_another-merch
export YOUR_AS2_ID=A1234567890
export PARTNER_AS2_ID=B4567890123
export PARTNER_AS2_SERVER_URL=https://as2-server.com
export PARTNER_ENCRYPTION_ALGORITM=AES256_CBC
export PARTNER_MDN_RESPONSE=SYNC
export MY_PARTNER_SIGNING_PUBLIC_KEY=<partner_public_signing_key_filename.pem>
export MY_PARTNER_ENCRYPTION_PUBLIC_KEY=<partner_public_encryption_key_filename.pem>

# create local certs

openssl req -x509 -newkey rsa:4096 -keyout ${LOCAL_PROFILE}_private.pem \
    -out ${LOCAL_PROFILE}-public.pem -sha256 -days 365 -nodes

# import local certs

stedi as2 import-certificate --name ${LOCAL_PROFILE}_signing \
    --description "${LOCAL_PROFILE}_signing" \
    --usage SIGNING \
    --private-key file://${LOCAL_PROFILE}_private.pem \
    --certificate file://${LOCAL_PROFILE}_public.pem

stedi as2 import-certificate --name ${LOCAL_PROFILE}_encryption \
    --description "${LOCAL_PROFILE}_signing" \
    --usage ENCRYPTION \
    --private-key file://${LOCAL_PROFILE}_private.pem \
    --certificate file://${LOCAL_PROFILE}_public.pem

stedi as2 import-certificate \
    --name ${PARTNERSHIP}_signing \
    --usage SIGNING \
    --description "${PARTNERSHIP}_signing" \
    --certificate file://${MY_PARTNER_SIGNING_PUBLIC_KEY}

stedi as2 import-certificate \
    --name ${PARTNERSHIP}-encryption \
    --usage ENCRYPTION \
    --description "${PARTNERSHIP}_encryption" \
    --certificate file://${MY_PARTNER_ENCRYPTION_PUBLIC_KEY}

stedi as2 create-profile \
    --name ${LOCAL_PROFILE} \
    --profile-type LOCAL \
    --as2-id ${YOUR_AS2_ID} \
    --certificate-ids <SIGNING_CERT_ID>,<ENCRYPTION_CERT_ID> # ID is different from name, so this must be retreived using `stedi as2 describe-certificate --name <CERTIFICATE_NAME>`

stedi as2 create-profile \
    --name ${PARTNER_PROFILE} \
    --profile-type PARTNER \
    --as2-id ${PARTNER_AS2_ID} \
    --certificate-ids <SIGNING_CERT_ID>,<ENCRYPTION_CERT_ID> # ID is different from name, so this must be retreived using `stedi as2 describe-certificate --name <CERTIFICATE_NAME>`

stedi as2 list-profiles

# describe each profile to get ID

stedi as2 describe-profile --name 

stedi as2 create-connector \
    --name ${PARTNERSHIP} \
    --url ${PARTNER_AS2_SERVER_URL} \
    --local-profile-id ${LOCAL_PROFILE_ID} \
    --partner-profile-id ${PARTNER_PROFILE_ID} \
    --encryption-algorithm ${PARTNER_ENCRYPTION_ALGORITM} \
    --mdn-response {$PARTNER_MDN_RESPONSE} \

# server only required if receiving AS2 messages

stedi as2 create-server --name bootstrap

stedi as2 describe-server --name bootstrap

stedi as2 create-agreement \
    --name ${PARTNERSHIP} \
    --base-directory "/as2-inbound-${STEDI_ACCOUNT_ID}/${PARTNERSHIP}" \
    --description "${PARTNERSHIP}" \
    --local-profile-id ${LOCAL_PROFILE_ID} \
    --partner-profile-id ${PARTNER_PROFILE_ID} \
    --server-id ${AS2_SERVER_ID} \
    --status ACTIVE

[Engine] Log all failure events to a Stash keyspace

Create a new function that subscribes to all failure events, logging them to a Stash keyspace for now. This function can act as a kickstart for customers to generate emails or other alerts for Engine failure events.

[bug] buckup/restores scripts should include mappings

Sometimes when testing for a V1 => V2/Core migration we are also moving to a fresh Stedi account, we currently do not backup or restore mappings as part of these scripts, so the copied config is not a complete clone.

Note, when restoring Mappings we need to update all mappingId values in the Stash config to the new ids in the target account (the restore script already does this for Guides).

[feature request] Support setting timeouts on webhook destinations

A customer recently had an issue where they were making multiple webhook destination deliveries and one of them was taking a long time to reply, resulting in the edi-inbound function timing out (so execution tracking was not able to complete).

We should support configuring, and set a default timeout on webhook destinations, I'd suggest 20 seconds default?

This would be helpful on both legacy + main branches.

Format Outbound EDI Files with New Lines

Hey! I am super enjoying getting up to speed with Stedi. I have followed the tutorials here. I am wondering whether there is a way to have Stedi functions format outbound EDI files with new lines in them so that they look a little more readable than what I have below (the output of the EDI outbound translation from the bootstrap).

image

move bucket env vars to stash

they can now be retrieved from the bootstrap resource metadata (introduced in order to support destroy cleanup script)

add support for bucket path template substitution

related to: #55. a pattern suggested in a related slack conversation was as follows:

you can create a new bucket for archived data. I recommend structuring the archive folders to be similar to:
${partner_name}/${year}/${month}/${day}/${original_file_name}-${timestamp}

in order to accomplish this, we would need to support some form of template substitution on the path for bucket destinations

check response status for `webhook` destinations

fetch does not throw errors for 4xx / 5xx responses to requests. our webhook destination implementation should check the response to make sure it was ok, and throw an error if not.

if a customer is trying to ingest data from an inbound EDI file, but the call to the API that is going to process the data fails, we should treat this as a failure (and the input file should not be deleted)

add support for tunneling SFTP connections through proxy

Customers frequently need to connect to their trading partner's SFTP server via allow-listed static IP addresses. One workaround that we frequently suggest to customers is to connect through an external proxy server that has a static address (or set of addresses).

In order to support this in bootstrap, we need to support proxy-related options in the SFTP configuration schema. At a minimum, this would be the proxy host and port, but other options may be useful as well. Some example code for connecting via a Socks v5 proxy can be found here:

theophilusx/ssh2-sftp-client#39 (comment)

Add support for error handler destination

It would be great to add support for a generic (global -- not associated with any particular partnership) error handler webhook destination (including support for custom headers as mentioned in #22). Customers could configure this generic error handler destination to hit a Slack webhook, PagerDuty, etc.

Add support for processing AS2 documents (inbound)

the inbound workflow needs to support documents that arrive in the processed directory of whatever bucket is associated with the AS2 config (could require this to be the SFTP bucket for simplicity / consistency)

add automated deployments and canary

we should replace the existing "demo loop" which is currently running the read-edi-demo and write-edi-demo functions with the bootstrap implementation.

  • add automated deployment of bootstrap resources to the two demo loop accounts ([Write|Read] EDI Demo Test Account, available under support@)
    • each deployment may also need some customization of the stash partnership config? TBD
  • add scheduled task to invoke write workflow to drop a file in an outbound directory
  • existing SFTP poller workflow should pick up file and copy to the read account
  • configure some form of validation of webhook requests to catch any regressions by rejecting requests that don't match expected shape: https://docs.webhook.site/webhookscript/examples.html#validate-request

[enhancement] make `edi-outbound` backwards compatible with legacy (pre-Core)

The edi-outbound function on main currently supports this input shape:

{
  "metadata": {
    "transactionSet": "214",
    "partnershipId": "this-is-me_another-merchant",
    "usageIndicatorCode": "T"
  },
  "payload": {}
}

The same function on legacy expects:

{
  "metadata": {
    "transactionSet": "214",
    "sendingPartnerId": "this-is-me",
    "receivingPartnerId": "another-merchant",
    "release": "005010"
  },
  "payload": {}
}

The edi-outbound function should be updated to support the legacy input, and it can try to resolve the partnershipId using both the sendingPartnerId & receivingPartnerId that legacy expects. This will allow upgrades from legacy to main to not require updates to edi-outbound invokes as part of migration.

We should also log a warning regarding the old syntax, and indicate it will be deprecated eventually, so customers can upgrade at their leisure.

refactor outbound edi workflow to use new `inputMappingId` parameter

Context from a related Slack discussion:

For outbound, having the mappingId on the destination but actually using it BEFORE translate feels wrong, there should be a top level inputMappingId instead (having a map per destination after translate doesn’t make sense as it’s a string at that point)

with this it would be much cleaner -- we currently call Translate N times in the outbound flow (one for every destination). if we introduce inputMappingId we can move to a single Translate call, and then just deliver that EDI string to N destinations (such as an SFTP bucket and an archive bucket)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.