• allow configuring in property file the baseUrl to find the schema.json validation file for a JSON DPD
• allow supporting a minVersion and a maxVersion of schema.json validation file for a JSON DPD through property files
• add schema files also locally, so the application could rollback and take that from their resources in a scenario where it's impossible to fetch fhe schema.json validation file from the combinato of baseUrl and version

Allow Git protocol to work also with Username and Password and HTTPS instead of only SSH

Add the possibility of returning the actual task status and additional information in the callback call to ODM. Currently, it's only possible to STOP the task without the ability to declare the status (it's assumed it has been completed successfully).

Below is a proposed structure for the body to be added to the callback call:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Generated schema for Callback API",
  "type": "object",
  "properties": {
    "errors": {
      "type": ["string", "null"]
    },
    "results": {
      "type": ["object", "null"],
      "properties": {},
      "required": []
    },
    "status": {
      "type": "string",
      "enum": ["PROCESSED", "FAILED"]
    }
  },
  "required": [
    "status"
  ]
}

The "results" attribute can be used to return additional information generated by the task to ODM, such as pipeId, name, outputs generated by processes, and so on.

Let me know what you think.

Thanks,
Giandomenico

Refactor project's structure in order to create a dedicated module for core classes

Implementing the full CRUD stack for the domain entity type, including the client and the IT

https://github.com/opendatamesh-initiative/odm-adr/blob/main/ADR-0004-devops-callback.md

Add application and infrastructure API in DataProductVersionComponentsController

Integrate a CI/CD process in the odm-platform module.

• Trigger integration tests execution during build
• Re-organize docker files in repo
• Setup a baseline CI/CD process with GitHub Actions

Use StandardDefinitionDPDS to handle templates in DTO but then map them on the dedicated domain entity TemplateDefinition. Do the same for API mapping the DTO on the dedicated domain entity ApiDefinition. Rename then the rest collection definitions in apis

REGISTRY

• Add a VARIABLES entity (so, also a VARIABLES table in the ODMREGISTRY schema) which will contain variables defined through the descriptor and their value.
• At the moment of the READ of a resource, the descriptor will be serialized with the values of the variables of the new table and the variables section won't be returned.
• At the moment of the creation, the descriptor will be scanned searching for variables in the syntax ${variableName}

DEVOPS

• when a devops provider call the stopTask endpoint returning the results of the Task execution, check for variables in the TaskResultsResource and if founded, update the VARIABLES table

Add TestContainers to allow Integration Tests without requiring manual creation of docker containers

• minor fixes Azure
• Improve error handling: add Unathorized, ...

• Add a command to create a blueprint from file.
• Reduce verbosity for the CLI

Refactor the custom properties structure in application profiles to be more flexible for future properties

If the descriptor files are stored in a Git repository, no meter if private or public, the repository must be cloned on the host machine, and then files fetched locally. Authentication will be performed during the cloning operation using the Git protocol (https or ssh). Acting at the level of git protocol and not at the API level will allow having just only one implementation for all git servers (ex. github, gitlab, azure devops, ecc...).

• Try GitHubApp / OAuthApp instead of PAT for accessing the GitHub API
• Allow different GitServices active at the same time and choose which to use based on the blueprint
• Add an "instances" collection/entity to keep track of correctly created projects from blueprints
• Mock Git providers in test for /instances endpoint and add blueprint initialization as test
• minor fixes

• creation of a in the pom of the server modules with the maven plugin to generate the static documentation through redocly/cli + script for redocly command
• creation of a script on root level to generate the documentation for each server module
• Fix APIdoc.md

Error example: DataProductVersionIT - testDataProductVersionsReadOne, errors only if executed with MySql as DB

• Centralization of all OAuth features in a single module
• Replace actual usages of OAuth with functions from the new module

• Create a PolicyClient inside the policy-api module
• Refactor the ITs of the project https://github.com/opendatamesh-initiative/odm-platform-up-services-policy-opa to use the PolicyClient instead of a custom RestTemplate

• Create a RegistryClient inside the module registry-api
• Refactor ITs of registry-server to use the RegistryClient instead of a custom Rest Template

Create a sparate DB Schema for each module

The initial utilization of Jackson Annotations helped expedite the parser implementation, but it also imposes limitations on the overall process control and adds complexity to the parser's evolution. As a result, it becomes essential to directly manage the parsing process, emulating the code structure employed by the swagger-parser.

Move the server of the project https://github.com/opendatamesh-initiative/odm-platform-up-services-policy-opa inside the utility-plane-services of the platform.

• Lifecyle entity + DDL + resource
• Lifecycle Repository
• Lifecycle Mapper
• Lifecycle Service
• Lifecycle Controller + CRUD (only GET endpoints)
• Extend client
• IT

• Create a separate module for ODMClient and for an Interface / Class for the Routes
• Refactor RegistryClient & PolicyClient in according to the previous task

• Centralization of all Git features in a single module
• Replace actual usages of Git with functions from the new module
• Depends on #128

Store in the database sall chemas exposed by data services endpoints (i.e. endpoints of API associated to output ports and observability ports) then create CRUD services in the registry module to operate on this new collection.

Replace the custom code used to parse OpenAPI definitions with swagger-parser

• Add Postgresql to docker-compose
• Reduce number of mandatory build args but let them be overridable

Upload endpoint fail to resolve reference with a depth grater than the first level. It resolves first level references, but fail to resolve 2nd level references.

Example:
| basedir
|— dp1.0.0.json
|— subdir
|—— port1.json
|—— subcomponentport1.json

Where dp1.0.0.json contains:
"$ref": "subdir/port1.json"
That reference is correctly resolved, but then, resolving it, also port1.json have a reference, that is "$ref": "subcomponentport1.json" . This last reference CANNOT be resolved because the application consider basedir as base URL even when resolving references inside subdir , so it looks for a file named "basedir/subcomponentport1.json" instead of "basedir/subdir/subcomponentport1.json" , due to the reference of the port. It works if port1.json use "$ref": "subdir/subcomponentport1.json" , but it's logically wrong to use it in the file.

• Refactoring
• Minor fixes
• Add a params.json default file in blueprint remote repository, fetch it on creation and store in a "paramsDescription" column of the blueprint entity (it must be a file with a fixed JSON structure to describe parameters and their default values)

Develop:

• the entire Owner CRUD stack (resource, repository, controller, mapper, ...)
• the client
• IT tests (happy and error path)

Replace the custom code used to parse AsyncAPI definitions with asyncapi-parser

When multiple versions of a Data Product are stored in the DB, deletion of a single version will lead to the deletion of all the stored versions

The API of the policy-service is not factored into a dedicated project at the moment. Extract them from odm-platform-up-services-policy-opa and create a dedicate sub-modules of utility-plane-services to store them.

Refactor odm-cli to add a level of subcommands when needed + minor fixes + style.

Example of refactor to add a level:

• from: odm-cli registry listDP
• to: odm-cli registry list dp

Add a odm-blueprint module

• Extend MetaServiceClient to map each NotificationController endpoint
• Extend ITs in odm-platform-up-services-meta-blindata: add all test scenarios in NotificationIT

Add two endpoints to the DataProductVersion controller for the internal components:

• GET application components
• GET infrastructure components

Modify the client and the IT accordingly

• Extend RegistryClient to cover missing endpoints
• Extend basic ITs to cover missing basic scenario (CRUD)

Create a dedicated sub-module for DTO to be shared with client services.

The DevOps module allows platform users to execute DevOps activities over a data product version like provision, build, and deployment. It decomposes the activity in tasks and calls an executor service exposed by the utility plane to execute them. The executor's API implementation works as an abstraction layer that decouples the DevOps service from the underlying tools used to provision infrastructure, build code and deploy artifacts.

Resolve and parse the API definition to extract ApiDefinitionEndpointDPDS from path objects that expose a GET operation

• edit descriptions to be limitless
• replace varchar(5000) with text for mysql