djsuperchief / kyameru Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 972 KB

Kyameru is a business processing engine inspired by Apache Camel and built for .Net Core.

Home Page: https://djsuperchief.github.io/Kyameru/

License: MIT License

C# 98.79% Shell 0.17% Dockerfile 0.03% HCL 0.58% HTML 0.09% Ruby 0.34%

kyameru's Introduction

Kyameru

About

Kyameru is a processing engine built with heavy inspiration from Apache Camel. It is currently in very Alpha stage and contains very few components but more will be added over the coming weeks and months.

Basic Overview

Architecture

Structure

Kyameru is built on the Chain Of Responsibility pattern. It is built up using the three main construction objects:

Route
RouteBuilder
Builder

Each of the objects above are instantiated in the order specified. Route provides the start of the construction by allowing the definition of a From component. This returns a RouteBuilder object allowing you to add Processing components, add headers or a number of other tasks related to message processing. The To component is also created here which in turn returns the final Builder object which allows you to add more To components for final processing and an Error component to handle any chain errors.

The Builder component is responsible for injection into the DI container of .NET Core (which is in turn responsible for resolving ILogger instances) and creates an IHosted service for what is known as a Route.

Workflow

The workflow is fairly simple. Everything starts at the From component. This raises an OnAction event which is picked up by the From chain object that then starts the process off. The Routable message is then passed from component to component until it reaches the last To component in the chain. What is important is each of the declared components sit within a core Chain object responsible for handling logging, errors and passing execution to injected components during the build.

If at any point the Routable message is deemed to be in error, all processing stops and the only component that continues to process is an Error component that is entirely optional.

Atomicity

Atomic Option

Currently under construction, each route can be configured to be atomic. The From component is responsible for the atomic nature of the route for insance if you use the Ftp component and specify that it needs to be atomic, it will not delete the downloaded file from the Ftp server until the To component executes successfully.

Current Atomicity.

Each component is considered atomic as opposed to the entire route. The engineer is expected to be able to handle failures through the use of the Error component to ensure the route is atomic. The reason for this is that a From component cannot assume what the end To component will do and as such cannot assume what action needs to be taken.

FTP Example

For instance an FTP download will download a file to a temporary location to be processed. Whilst it can assume some sort of processing on the data of that file will happen it is not known what the end state of the file will be. The FTP component will clean up after itself but it is up to the engineer to decide if any processing failed on the file, what should they do with the temporary file downloaded? Should they move it to an error folder? As far as the FTP component is concerned, its job was to download the file, delete the source and raise the internal message for processing. Its process has been atomic and it now plays no further part in the processing of the file.

Basic Syntax

URI Format

As stated before, Kyameru is inspired by Apache Camel and the URI format of injecting components seemed like a sensible approach as it is highly descriptive and a format we have been using for decades. The Uri is split into several parts:

Scheme = Kyameru component
Path = Kyameru "target" header
Query = Kyameru headers

An example of this (file component)

Kyameru.Route.From("file:///C:/Test?Notifications=Created&SubDirectories=true&Filter=*.*)
.To("file:///C:/backup?Action=Move)
.Build(services);

The above example is very simple but you can see from the syntax that the From construction that:

Scheme is file -> this intializes the from component Kyameru.Component.File
Path is C:/Test -> this sets the component header "Target" to C:/Test
Query -> Adds the headers Notifications, SubDirectories and Filter

Current Status

This project is still very much in beta but has been released to the wider community early for feedback and to be used.

Wiki (WIP)

Wiki Home

Components In Progress

MSSQL (First)
RabbitMQ
SQL (GENERIC)
sFTP

kyameru's People

Contributors

Stargazers

Watchers

Forkers

eramax

kyameru's Issues

Add localstack & terraform container stack

Add a localstack and terraform container stack so that AWS components can be developed locally.
The reason for a terraform container is so that it executes in isolation to any AWS accounts on the local machine

Entity body needs a body read delegate

The body of the message (Routable) can be anything. The body could do with a way of reading it rather than rely on boxing and unboxing. This could be done with some sort of delegate up front.
This could be specified at route time build and allow users to specify a delegate to read the body of the message.

Move components into solution

Sort out solution files

There are "multiple" solution files. There should be tops:

Kyameru Full - All components + Core
Kyameru Components - Just components
Kyameru Core

Package and publish should be done from one of these. Also decide where you want these to be stored (in build or at root)

Auto inject processing components

Components should be able to inject their own processing components automatically at the start or end of the chain. For instance, SQS having a component that can convert a standard string body into a message to process. This means separation of concerns so that the sending of a message can be concerned only with sending a known message type.
This would mean that the user does not have to be concerned with changing data types or worrying about having to convert the routable item during processing.

Agreed, this could be done inside the component BUT it would make sense for this to be configurable.

Add Nuget Package To Workflow

Nuget package needs to be done to the workflow

Release from main

Maintaining a release branch is great but I think the release version should be what triggers a release to nuget and tag.
Main is always releasable so we should release from that.

Tasks:

Merge to main will tag the repo and bump the version
Create release will then also do the nuget package and push

Also:

Need to ensure there is a readme file that gets put in as part of the nuget pack.

DI Injection - Make it more configurable

Currently each component is involved in an either or scenario. It is either injected by DI or created as a concrete implementation. The component should be able to allow the user to choose whether the component is pulled from DI or concrete implementation.

Exit Clause needed

If you need to exit a route early you need to be able to trigger this at a component level.
Whilst this can be done at the application level it seems silly to ask every component to check if the message can be processed and rely on the application to set this.

Suggest a property exists on the routable that can be set which indicates that the message should stop processing and a final handler indicates that this is the case (with reason)

Nuget build

Create nuget build action and work with tags

Critical: URI route attribute incorrectly interprets URI

The URI builder is not interpreting the Kyameru routes correctly. It is skipping host entirely and going straight to path which means that Path forms the target rather than the host and then the path.

I believe this was changed / not done correctly when building out the file URI.

This needs to be changed so that:

component://target/path?header=value&header=value is correct.

Each component should be responsible for either taking the host and path as different inputs, combining them to a single or treating them as separate usable parts. It should not be up to the core framework to do so.

This will break all components and is considered a blocker until this works.

Release core as V0.1 / No longer beta

I don't think there is any reason why the main core engine and the current components can't be released as "live". Waiting for every component to be built will take some time and the engine will never get released!

Core components that need to be ready to go before V1:

Create component builds

Entity headers able to be added to directly

The headers in the routable entity are a straight dictionary which means that you can add a value directly and manipulate it. Routable entity headers should be immutable and therefore not able to be edited. This is not currently the case and means they can be manipulated and modified runtime.

This needs to be prevented to be more in line with current thoughts.

Dependency Injection For Processing Component

When adding a processing component you currently add the concrete implementation and carry on regardless.
Now there may be a need to rely on DI so that all of this can be resolved at route build time.

So the Process command has to be able to accept a type of IProcessingComponent but also has to be able to get it from DI.

This will allow several things such as mocking in implemented apps, later binding for components.
For note to the above, the construction of the Process component should probably move down to the build but there is something to consider here. We need to think carefully of ordering:

From
Component - Concrete
Component - DI
Component - Concrete
To

We need to make sure the order in execution is kept given that the way it currently works is that concrete implementations are already ready to go before the DI version would have had chance. We do not want to go all one way or the other, it needs to be flexible.

BUG: Release tag doesn't take into account -beta

As the title says

Add prometheus exporter

Adding a default prometheus exporter component would be good. How this interacts is going to be interesting because it would need to be part of the core. Also what stats should it export? Is it the responsibility of this framework to do so?

Probably spike this one.

Add async to all components

All components should be able to do things async.....nough said

SQS Component

Add SQS component for AWS

Component should be able to make use of the usual access key shenanigans but also be able to load profiles from the user aws profile folder.

Packaging For Release

Things to do:

Test build and pack
Mark non packagable projects as such to exclude from pack command
Add readme to all packed projects
Add release pipeline so that on release, nuget pack and push is performed
Add dependency on build pipeline
publish to nuget

Processors need to be able to use URI format

Processing components need to be able to be added via URI or injection like the to or from.
There might be processing components in a in house component but equally you cannot make use of DI of a host application if we do not allow injection of Processors.

Create github automated github release

Even though this is not going to go back to nuget (yet), need to automate the github release process when pushed to the release branch (or github release is done).

Process component by function delegate

Currently the only options to create a processing component are:

TypeName -> Used when routes are specified in config
iprocessComponent -> Concrete implementation
Through DI -> as it says

Users may not want to create objects or use DI to do processing and may want to choose function delegation to do so. We need to include this as an option.

Upgrade to .NET 7

Upgrade to support .NET 7

BREAKING: Only one route is able to be run per app

DI only results in a single route being created and being able to be run. This is breaking functionality and needs to be fixed before this can be considered ready and fit for purpose.

Replace Moq with NSubstitute

Given the questionable nature of a change some time ago, it seems wise to switch mocking frameworks

Implement logger message for enhanced logging

Each route should be named (or randomly named) and this should be reflected in the logs. Logger message will enable this to be a much easier.

Remove duplicate sln files

The repo contains multiple SLN files. Should remove any duplicates and use SLN files at the root

Add security scan to build pipeline

Create pages site

Remove global calls in core tests

There is a "Global Calls" which is particularly bad practice and means we get flaky tests. This should be removed so that tests are predictable and work better.

Suggest injection of objects outside of fake component (See SQS component for IExtractor)

We can also now use post processing to better test the core instead of global calls

Event trigger mechanism

Not all components will be able to rely on a hook to an external event and will need a push to trigger the route.
The SQL component will need a trigger to start the route going and without it is a non-starter.

MVP should be an injected singleton with events that are subscribed to in several from components.

A later stage should be more an observable pattern but not required for an initial implementation.

Nuget security scan

Add security scan for nuget packages

Components able to be injected

There is a use case where from or to components need to be injected because they may need to pull on DI components outside of the core framework. This could be used to leverage say a database connection factory or sql client so that the components responsible for connecting or executing commands are outside of the framework.

We already have the ability to inject processing components in so it might be worth considering that creation of the From or To could be done through DI which would then allow us to pull in those components when build occurs.

Atomic Route

An option to enforce that a route is atomic. This might be useful where say an FTP component is deleting files in the source FTP site. If the whole route fails, the FTP source file may need to stay in source instead of being deleted immediately. The user should be able to specify that the whole route is atomic and that the file should not be deleted until the To is complete.

The atomic actions of the From component are in controlled by the from component as Processing is specified by the user and To is at the end anyway.

Update build so only on PR

Update the build so it only actions on a PR.
Also consolidate them into actions...we don't need 4 builds ffs

S3 Component

Create a new S3 component that allows a route to finish by putting a file in an S3 bucket.

It is not required to allow a file / event to be triggered by a S3 file event as it is thought that this kind of flow will be handled within AWS itself. It doesn't seem wise to have a poll against an S3 bucket for a file (but should anyone need it, then this can be done at a later stage).
For now, only the to route is needed in the component.

This will also serve as the foundation for AWS components in terms of auth etc.

All AWS components should rely on DI for setup so that a localstack injection can be done especially with integration testing. However no unit tests should rely on localstack. We should also investigate if there is any value in running the integration tests as part of CI-CD or whether we do PACT testing (pact may leverage better results).

SQL Component FROM

Create a SQL component with FROM only.

The SQL component should only be implemented using the base interfaces for database connectivity. It should take into account SQL injection and make sure that if queries are not parameterised then they should be.

It is also entirely possible that the onus is on the user to ensure that any queries are not SQL injection prone by using the inbuilt functionality of the component.

Kyameru is not responsible for managing the connection context but should at least provide all the relevant interfaces to do so.

It should at a minimum support MSSQL, PostgreSQL, MySQL.

Thoughts

So whilst this is an essential component, we need to consider security and the part Kyameru plays in it. We should consider that using an ORM will greatly reduce the security concern but we also need to consider that and ORMs power is by mapping DB queries to objects.

DI Register methods not implemented correctly

Change instantiation so that it makes proper use of DI injection (Components were never updated to utilise the Register From and To).

Headers and components need new registration methods.

Immutable headers from components shouldn't be automatic

Each component (from) should be in control as to what it considers to be immutable.
Currently each header from a from component is considered to be immutable but something later down the line (or even a to component) may need a slightly modified version of that header.

As such automatic immutability should not be done and each component should control the immutability of the header.

Second, the SetHeader of the message needs to be public, not internal.

Initialise class for to and from

There may be a situation where headers do not cut it for initialisation data. For instance, you may need to be able to specialise a payload. This could probably be done with extension methods but it is worth investigating whether there is going to be much call for initialisation class and data as part of the core or as part of a component.

Example, REST component will need to be able to specify a data set to post etc. This cannot be done in headers easily or in an easily readable manor.

Non standard headers need a way to be read

Non standard headers in the Routable message need a way to be read and they also need a way to be marked as non standard so that they can be easily picked out of the headers without the need for an overly complicated for statement.

Nuget Builds For Components

Not sure if this should be separated out into separate projects. I mean, do the core components get included but built separately? They need to reference the nuget project to be able to be pulled down...so.

This needs to be figured out BEFORE going live with the open source part.

Move tests to xunit

Routes to be configurable via config

Routes should be able to be configured via a config if the user so chooses. This could be done through the appConfig.json file or through an XML / json file sat outside of the application or in an embedded resource.
This saves on actual code needing to be written.

I think this would work well and could make code a little clearer. Food for thought but once the first set of components are done then this should be considered for development.

Atomic route creation implementation is not know to From or To

Because of the way that atomic routes are created the from component will not know (at the time of initialisation) that the route is meant to be atomic. The activation of the from component needs to be delayed until the Build command has been run to allow atomic route creation to be specified first.