Parent Item

CORE

this module will allow user keep track the status changing history of every object in their shop. For example, Product A was updated price at 10AM, 20/11/2023, from 10$ to 20$

Child Item

• Event sourcing for Offer domain
• Event sourcing for Order domain
• Event sourcing code base
• Event sourcing for Product domain

For our new public roadmap it is necessary, that historically since Jan 01 2023, our Azure Devops Items (like Epics, Product Backlog Item, Features, Impediments, Bugs, ...) are revised. 

Please read the acceptance criteria!

It would be also nice, if you can set a SubArea - if you are missing some, please write Marc C on slack which sub area needs to be inserted in Azure Devops.

After revising an item, please switch the Roadmap to true,.

PS: Comments must not be revised, they are not transfered to our public roadmap.

PPS: Images that are copy & pasted into the description could not be seen in the public roadmap (not possible due to legal reasons).

Acceptance Criteria

Each item must have:

• Remove sensible information (like logins, tokens, etc.) from the item
• Description so on external developer knows what this item is about
• Acceptance Criteria
• Start Date
• Target Date
• Effort
• Area

Parent Item

Public Roadmap

1. Check which Roadmap systems exist on the market
2. check its prices and compare features vs prices

Acceptance Criteria

• must connect to Azure DevOps or have an API (so we can do the connector by our own)
• must have a public view for website visitors without any login
• should be customizable in its design, so we do not have an UI breach.

Parent Item

Public Roadmap

The internal items must be uploaded (CRUD) to GitHub Issues, so they are visible in the public roadmap. Pay attention to GraphQL API from Github.

Parent Item

Sync between Azure DevOps and GitHub

This feature enable users manage dataflow as they needs

Parent Item

Data Orchestrator Engine

The ingression module is the module that will receive data from ERP or Marketplace, and then insert data to the Qilin database. 

When the data comes to the ingression module it will be analyzed, compared with existing data, and broken into tasks such as creating or updating

Child Item

• Comparision component
• Event Grid
• Service bus
• Command storage
• Importing Component

Parent Item

Qilin UI

This feature fixes all problems in the Product project that are scanned and detected by the sonar cloud such as code coverage, duplicated code, code smell, and so on, so that, our code will meet the quality gate:

• UT code coverage >= 80%
• Duplication: < 3%
• Code smell: 0
• Vulnerability: 0

Parent Item

Code Quality

The synchronisation between Azure Devops and GitHub Issues should be full automatically, so no one in the team must do this by hand.

This is a hard criteria, to have the safety that the public roadmap is always up to date with our internal state.

Acceptance Criteria

• Works completly without any manual help
• sync at least once per hour (more often = better)

Parent Item

Public Roadmap

Child Item

• Save Ids in Database
• Upload Items to GitHub Issues
• Filter
• Get Items from Azure API

After an item is synced to Github, its state and ids must be stored in an internal database, so we can perform udpate and deletion on this item in the future.

Parent Item

Sync between Azure DevOps and GitHub

This feature fixes all problems in the Order project that are scanned and detected by the sonar cloud such as code coverage, duplicated code, code smell, and so on, so that, our code will meet the quality gate:

• UT code coverage >= 80%
• Duplication: < 3%
• Code smell: 0
• Vulnerability: 0

Parent Item

Code Quality

Parent Item

Qilin UI

The event processor is a component that handles the events which were created after the data is routed by data orchestrator engine to detect the destination channel and create list of events need to be done and push them to queue. The event processor will read events list in queue and decide handle them in order or parallelly

Parent Item

Egression Module

In this feature we implement a mechanism to limit using rate of subscribers based on their pricing plan. 

For example:

• Subscriber A can send 100 request/s
• Subscriber B can send a maximum 5000 requests/month
• Subscriber C can 100 input/output actions per second

Parent Item

CORE

this is a component of the ingression module, it will listen to blob storage and trigger the workflow when a new JSON file is uploaded

Parent Item

Ingression Module

Parent Item

Qilin UI

In this epic, we fix problems that are scanned and detected by Sonar Cloud to make our code base meet quality gate:

• UT code coverage >= 80%
• Duplication: < 3%
• Code smell: 0
• Vulnerability: 0

Child Item

• Offer project
• Product project
• Order project
• Cache project

As Qilin Egression Module, I want to make sure my requests to output connector must be protected & anti-fraud, so that, I need an authentication method.

Parent Item

Egression Module

This feature fixes all problems in the Cache project that are scanned and detected by the sonar cloud such as code coverage, duplicated code, code smell, and so on, so that, our code will meet the quality gate:

• UT code coverage >= 80%
• Duplication: < 3%
• Code smell: 0
• Vulnerability: 0

Parent Item

Code Quality

this feature will develop a process mining module or integrate with a 3rd party tool to help to analyze data flow and give suggestions to improve the data flow to the end users

Parent Item

Data Orchestrator Engine

Filter out all items, that should not be forwarded to the public roadmap, like

the ones without Roadmap switch = true

Parent Item

Sync between Azure DevOps and GitHub

Parent Item

Qilin UI

this epic includes all features and task related to APIs which are used to manage channel in Qilin, that includes:

• Channel
• Channel mapping

Child Item

• Channel mapping management
• Channel Management

• After being filtered and beautified by NiFi (or done by a 3rd party), objects as jobs need to be processed by Event Processor and sent to a particular Output Connector. The way that the Egression Module (EM) works is similar to a postal station. It looks at the payload of a job and decide that Whose (Subscription), From Where (Source), To Where (Target), When (Scheduled At), and What (Object Type, ObjectIds).
• However, as a greedy person, Nifi (or 3rd party) wants to send a batch of jobs to save the time and cost and requires the EM to send jobs orderly or parallelly because of the dependencies of them. For ex, Category and Brand need to be processed ahead Product.
• Therefore, in the postal station, we need to implement multiple conveyor-belts running concurrently and items in each belt will go by its order.

Explanation

https://dev.azure.com/marcossoftware/Qilin/_wiki/wikis/Qilin.wiki/748/Flows-In-Egression-Module?anchor=explanation

1. NiFi (or 3rd party) will send Http requests to create jobs containing ObjectType and ObjectIds as well as information about SubscriptionId and Action to these jobs(Create, Update, Delete) to an API endpoint in Egression Module.
2. By calling to the endpoint, jobs are pushed from DB to Event Processor
   3 and 4. EventProcessor will process jobs sequentially or parallel based on their partitionKey and do chain of business action. In every single job, it will check whether the Output Connector is over rate limit. If yes, the job will be delayed. Otherwise, it will send requests to Metadata APIs to get the payload (content) of objects in the job, and the Target Channel to determine correct Connector endpoint as well as the authentication method between Subscription and Target Channel (stored as Channel Authentication in Channel).
3. After collecting all the information about content of objects and the Output Connector endpoint, it will send a Http Request which contains object bodies with action, as well as authentication information.
4. Once Output Connector responses the request, the status of that job will be updated, report to Event Processor DB that the job is completely processed with Success or Fail. If the response is a fail, the job will be re-scheduled based on the status code in order to retry automatically.

Child Item

• Event Processor
• Authentication for Egression module
• Exporting component

As a Qilin supporter,

I would like to have logs where I can see the data flow from input to output

so that, I can trouble shoot and analyze if the data is not transferred as expected

Acceptance Criteria

• The status of data from input to output is tracked step by step
• there is an API to get the logs of a specific transaction
• the log records include meaningful full information such as:
  • transaction ID
  • step name
  • status (passed or failed)
  • error message in case the step failed

Parent Item

CORE

Apply event sourcing for entities in offer domain

Parent Item

Event Sourcing

This component has responsibility to send data to output connector based on the events that processed by the event proccesor. It also check the request and response status to output connector to take action accordingly, such as resending request if the output connector is bussy temporarily.

Parent Item

Egression Module

this feature create a list of APIs that will be used to manage channels in Qilin:

• Create channel
• Read channel
• Update channel
• Delete channel

Channel is a configuration that will be used to connect to ERP or marketplace by Qilin such as an authentication token, user name, password

Parent Item

Channel APIs

Parent Item

Qilin UI

as FE dev, I need UI wireframes for all UI functions of Qilin

so that I can based on  them to develop the FE (website)

Acceptance Criteria

• All UI functions of Qilin have UI wireframe
• All wireframes are documented and reviewed by Team

Parent Item

Qilin UI

the services bus is where we store messages that include object type, action, and object payload to call the API and insert data into the database. There is another component called importing component that will listen to the service bus and take action according to messages

Parent Item

Ingression Module

setup data orchestrator engine for development and testing

Parent Item

Data Orchestrator Engine

Apply event sourcing for entities in order domain

Parent Item

Event Sourcing

This feature fixes all problems in the Offer project that are scanned and detected by the sonar cloud such as code coverage, duplicated code, code smell, and so on, so that, our code will meet the quality gate:

• UT code coverage >= 80%
• Duplication: < 3%
• Code smell: 0
• Vulnerability: 0

Parent Item

Code Quality

This component will compare importing data and existing data in the database and decide which object should be updated. It also breaks complex object into smaller tasks and executes them in sequence or parallelly, depend on business logic of the object. For example, categories have to be inserted before products

Parent Item

Ingression Module

This feature focuses on routing data functions. it will try to map and navigate the data based on user configuration and figure out the source and destination channel

Parent Item

Data Orchestrator Engine

As Qilin,

I need to manage users and their permissions

so that, I can be aware of who is using my service and allow them to take the necessary action only.

The mechanism were using to manage user permission is Role Base Access Control (RBAC)

as Qilin user,

I need a user interface to access and configure Qilin

so that, I can customize and manage my data in Qilin

Child Item

• Setup WebUI Codebase skeleton template
• Implemen Subscription and User Management UI
• Wireframe
• Front-End Ingtegration with Qilin APIs
• UI/UX Designing
• Front-End Data flow management UI
• Front-end Connector management UI

Every item from azure devops, that has the switch Roadmap = true must be downloadet from the Microsof Azure Devops API and should be converted into a legacy internall class.

Parent Item

Sync between Azure DevOps and GitHub

This feature will filter the data before it comes into the Data Orchestrator Engine, so that, we can make sure the incoming data is valid and dont make the flow crash

Parent Item

Data Orchestrator Engine

Child Item

• API quota and limitation (rate limit and IOPD)
• Cache
• Data flow tracking
• Transfer status engine
• Activity logs for user

this is the engine that routes data from the input connector to the output connector

Child Item

• Data routing
• Data flow management
• Data processing mining
• Setup Engine
• Data filtering

As Qilin, 

I want to know id mapping (mapping between object IDs that are managed by Qilin and IDs that are managed by channels) & Transfer Status of those objects between Qilin & n-Channels

so that, I can know if the objects are transferred to destinations successfully. 

For example:

Product A is managed by Qilin with ID qilin-product-id-a, which is synced to OTTO where product A is managed with ID OTT-product-id-a. Qilin needs to know if qilin-product-id-a and OTT-product-id-a are the same product and if the product data is synchronized between Qilin and OTTO

Thats why we need this engine

Parent Item

CORE

as a Qilin user

I want to track activities history of my customerID

so that I can trouble shoot by myself

Example: when some products are unexpectedly deleted, I need to know why it disappears

Acceptance Criteria

This log just contains

• UserID
• Timestamp
• TransactionID (for connecting the activity log to the detail)
• ObjectId (which object is affected)
• ActiongroupID(like Product, Offer etc)
• Actioncode (a numeric value mapped to an action…e.g. 15556 = “changed Price”..this ensures a multilanguage support organization.
• Payload (additional data)

have API to get the logs from Qilin return in Json or CSV

API should return the last 7    30 days logs

Standard CSV format of Qilin should be used

user can see activities logs from their customerID only

User can see activities logs based on the role/user permission.

Parent Item

CORE

This feature creates a list of APIs that will be used to manage and configure channel mapping in Qilin:

• Create channel mapping
• Read channel mapping
• Update channel mapping
• Delete channel mapping

Channel mapping is a configuration that defines the route for data from a channel to other channels and versa.

For example: products in ERP A will be transferred to marketplace MP1 and MP2, and products in ERP B will be transferred to marketplace MP3. Versa, orders from MP3 will be sent to ERP B, and orders from MP1 and MP2 will be transferred to ERP 1

Parent Item

Channel APIs

this component will receive data from input connectors in JSON file format and start importing workflow of Qilin

Parent Item

Ingression Module

This feature will implement Cosmos DB as command storage which will replace service bus where contains list of command need to be done when importing data

Parent Item

Ingression Module

Focus on UI/UX of subscription/user management and Data flow management (Drag and drop)

Parent Item

Qilin UI

As a external developer, i want to know what projects Qilin.Cloud is currently working on, so I can plan my own development and communicate it to my users.

Therefore I need to see a public roadmap of Qilin.Cloud.

After some POC with Roadmunk, AHA and other systems, we decidet to use GitHub Projects with its new Roadmap view for this purpose.

Acceptance Criteria

• The connection between internal Azure DevOps and GitHub Roadmap must be full automatically
• The roadmap must be public viewable without any login
• Only internall staff should make CRUD operations on the roadmap an its items
• The roadmap should show items from the past (1 year) and the future (1 year)

Child Item

• POC Different Roadmap Systems
• Sync between Azure DevOps and GitHub
• Revise existing items