dbomproject / enhancements Goto Github PK

Enhancement Description

It would be highly beneficial for rich query support to be part of the API provided by the open source DBoM node. It would allow for channel wide searches based on attestation fields which has been a common use case for various PoCs/Prototypes.

This would mean that:

• There is a feature complete rich query dialect that the gateway understands

• Existing agents would gain the capability to translate this dialect to the native querying dialect that exists on the repositories they operate on

• Discussion Link: N/A

• Architecture/Reference documentation links:

  • https://dbom-project.readthedocs.io/en/dep-1/query.html
  • https://github.com/DBOMproject/api-specs/tree/dep-1

• Primary contact (assignee): @mathisonryan

• Planned Release Milestone:

  • 1.1.0 (Gateway)
  • 1.1.1 (Database Agent)

• Related Issues

  • DBOMproject/chainsource-gateway#37
  • DBOMproject/database-agent#91

Please keep this description up to date. This will help the DBoM Team to track the evolution of the enhancement efficiently.

Enhancement Description

Motivation and Scope

Over the last two years, we have gained a deep understanding of the applicability of DBoM Channel-based attestation sharing. We have also invalidated some initial assumptions.

These are the primary pain points that we want to address with this re-architecture:

• DBoM Channel establishment is clunky and requires direct connections to repositories
• The current channel access control mechanism is based on using a third-party trust anchor and communication mechanism (Matrix). We leverage Matrix rooms and private messages to allow DBoM nodes to generate and receive both the secrets and instructions to directly connect to the repository. This is relatively insecure and hard to standardize across repositories due to:
• Direct access to the repository port is provided to the remote node (this is dangerous from a security perspective as it is not
  recommended exposing a database endpoint to the outside world with just database authentication
• Revocation and rotation of repository credentials require a unique implementation for each repository. Several implementations lead to more lines of code that we need to maintain and audit for security purposes.
• Any vulnerability that exists in the repository will be exploitable by the remote node. Since we do not intercept the connection between the remote node and the repository, we cannot really apply any mitigation measures.
• Use of Matrix as a handshaking protocol requires the setup of several heavy Matrix services such as the home server, identity server, and more. Matrix (and other alternatives) are full-blown chat servers, with tons of functionality that we will never need to use. DBoM, as a security product, must strive to reduce its attack surface by only implementing the core that is necessary for functionality.
• There is no way to enable Read Audit or implement complex policy rules
• Currently, DBoM implements policies by configuring the RBAC capabilities of the underlying datastore. This means that complex policy rules such as requiring read audit, permission-per-read, and time-limited access to channels are not possible.
• Subscribers cannot truly "subscribe" to a channel in a pub-sub manner. A remote DBoM node cannot scale-ably listen to updates (new/updated) assets on a channel without directly utilizing the change stream functionality presented by the underlying repository. This, again, is difficult to audit as we are limited by the capabilities presented by the repository itself. This also means that the remote DBoM Node needs to care about the type of repository initialized.
• The notarization and storage capabilities of the DBoM node are tightly coupled. The reason we have repository agents for distributed ledgers and transparency logs is exclusively to take advantage of their notarization capabilities. However, most distributed ledgers & transparency logs (with a few exceptions) do not have great query capabilities or performance.

We have had to work around these extensively for both IOTA-Agent and Trillian Agent by either maintaining local state or working around the storage APIs. We also do not have rich query capabilities for any of these agents.

In retrospect, there is no need to couple the notary and storage. You could choose to store your assets on MongoDB, Postgres, or any other compatible database and still choose to notarize them on platforms like Hyperledger Fabric, Ethereum, IOTA, etc. This also means we will get excellent query and commit performance (as databases are built for this) while also having the capability to asynchronously notarize and even have different notaries per channel. This also means that you can only notarize for a subset of critical channels (for instance, why would you want to notarize your dev channel if notarization comes at a cost).

This document describes a high-level summary of changes coupled with a high-level architecture required to achieve these changes. It does not go into implementation specifics such as the API schema.

Summary Of Changes

1. New federation-based channel management model
   All requests to channels from other organizations will now go through their DBoM Node to enable the new policy engine and completely abstract the repository used by the other organization. This is explained in depth in the "Channel Establishment" section of the architecture document.

2. Separation of the data repository and the notary services
   The data repository will now be independent of the choice of the notary. This means that the notarized attestation of data that is stored in MongoDB, PostgreSQL, or any other database can now reside separately on a distributed ledger, transparency log, or any other notary.

3. DBoM Nodes will have universally resolvable identities
   DBoM nodes will have a way to establish their identity, by utilizing a multi-factor authentication mechanism that can involve mTLS PKI-based Cryptography and Decentralized Identities, described in the "DBoM Node Identities and URIs for channels" section.

4. DBoM Nodes will support complex policies and read audits
   With the new federation-based channel management model, all requests to a channel will go through the DBoM node and they no longer hit the repository directly. This means that we can now have complex policies (OpenPolicyAgent-Rego-like) and capabilities like read-audit.

5. Introduction of a pub-sub service
   In this iteration, remote DBoM nodes (and applications leveraging them) can subscribe to changes on a channel in real time with a repository-agnostic streaming API.

6. A new iteration of the asset envelope
   The asset envelope is the schema that needs to be followed to commit an asset to a DBoM channel. There is an overhauled schema that is use-case independent. See more in Asset Envelope V2

7. Complete removal of the RepoID concept at a DBoM Node level
   With the new channel management model, it makes the most sense for one DBoM node to have precisely one local repository configured at a given time. To be clear, the node will still support a variety of local repositories, but will only be associated with one at any given time. This will remove complexities arising from cross-repository operations such as aggregations.

---

• Discussion Link: This Issue
• Architecture/Reference documentation links:
  • High-level Architecture Document
• Primary contact (assignee): @DBOMproject/unisys-technical