See #4 for info on why this is in Git

Off-chain / on-chain private sequenced transfer coordination

This gen2 architecture provides an extension over the first generation, to solve two interconnected problems:

1. When you miss the private (off-chain) data for a message, you don't know which messages this blocks

• This can lead to "stop the world" scenarios, where the only safe thing to do is to stop processing anything
• Most business solutions can use a "topic" (such as a customer ID, or business transaction ID) to discriminated sequences that need to be ordered
• By specifying a topic in the message, an app can state which sequence it's involved in, so if you miss a message, then only that sequence becomes blocked - the other sequences can continue.

2. Putting an identifier in all messages on a topic leaks metadata

• Information can be gleaned by seeing that X messages happened on a given topic
• Even if the topic is hashed, you can still see that the message were interconnected
• In the architecture, we mask this strongly using:
  • A monotonically increasing nonce unique to each sender
  • An ID for the group (calculated as a hash of the membership list and optional name)
• To allow global ordering without leaking the group ID to the chain, we calculate the next-hash of all members, and match new message arrivals against that

Note this issue does not cover anonymity of the sender of the transaction. The integration of HD Wallets, ID mixers etc. is protocol specific, and handled in ethconnect and other Blockchain Interface implementations.

There are a few panels in the Firefly UI that require the count of a resource. This is not returned in the current API response, and can be a very expensive operation.

A few questions/options to consider:

1. Is count always returned? Requires more sweeping changes to API response, should be done sooner than later
2. Only provide count when a query parameter is supplied
3. Aggregate and store counts in a separate table

Operations can fail, such as a private send, or a pin against a blockchain - even with resilient plugins.
At the point the operation is initiated it's locked in a bunch of locally sequenced messages into a batch, and in the case of a pinned private transfer, those messages could arrive over the blockchain and not over private messaging (or vs. versa).

This means that the topics of messages in that batch are stuck - they can't proceed past the blockage.
The operation that failed needs to be re-initiated once the underlying failure is resolved.

So we need to prove a REST API to attempt re-submission, and the UI will be able to add a simple 'retry' for failed operations.

All the data should be available to re-submit, and the processing surrounding the operations is all designed to be idempotent.
So double-sending, or double-pinning, does not actually break the model.
The first sequence to be confirmed on the blockchain wins.

In FireFly explorer, exposing counts of messages, tx's, organizations is required. Since we don't include count in API responses (see https://github.com/hyperledger-labs/firefly/issues/13), a new API to return general count and other statistics would be useful.

• Create a walk through for the community call that builds up to ^^^ with info
• Create a useful set of introductory docs

The confirmed time is set when a message has been confirmed according to all the conditions on that message:

• If is blockchain pinned:
  • It's been confirmed by the blockchain
  • All data is available
  • All previous messages on the same topic+ group + ledger combination have been confirmed

The order by confirmed (nanosecond resolution) timestamp is assured to be the same as the order in which the message_confirmed events are delivered to the application.

So it makes sense for any application, and the UI in particular, should want to query messages:

• With a simple option for "confirmed only"
• With all unconfirmed messages (if included) separated from confirmed messages
• In confirmed order

See #4 for info on why this is in Git

FireFly Multi-Party Event Sequencing

One of the most common problems solved with blockchain in enterprise multi-party networks, tackled head-on with FireFly's core architecture.

See #4 for info on why this is in Git

What is a FireFly Node?

There are three zoom levels of this question, where the architecture matters:

1. What fundamentally is a node - left on the picture
   • It is a multiple runtimes with a single unified HTTPS/Websocket API (exposed by the Core)
   • It has a private database, containing your private data, and data received from others in the network
   • It has connectivity out to other parties in the network, through runtimes (Blockchain, Shared Filesystems, Messaging etc.)
2. What are the core runtime responsibilities, and pluggable elements - right on the picture
   • The core elements of function that FireFly performs, and which runtime is responsible
   • This means some insight into core itself, and the jobs it performs, but not full code structure
   • More importantly, what the split of responsibilities is between Connectors and Infrastructure Runtimes
     • Connectors are the bridging runtimes, that know how to talk to a particular runtime.
       • They run separately to the core (like a microservice architecture of an app)
       • They can be written in any language (not just Go) - Java, TypeScript, Rust, Python, .NET etc.
       • They can use any network transport (not just HTTPS/Websockets) - GRPC, AMQP, UDP etc.
       • They connect to the core with a Golang shim - see separate Plugin Architecture discussion

         • In some special cases (like the Database) the Golang shim does not need a connector runtime

     • Infrastructure Runtimes are the core runtimes for multi-party system activities
       • Blockchain nodes - Ethereum (Hyperledger Besu, Quorum, Geth), Hyperledger Fabric, Corda etc.
       • Public strorage - IPFS etc.
       • Database - PostreSQL, CouchDB etc.
       • etc.
3. What is the code structure inside the core

• The README.md is the reference for this
• Developers contributing to FireFly, on the core, or building new plugins, need this level of detail

• A reconciliation is underway to ensure the medium-level view correlates well with this code structure

We recently deprecated the broadcast/message and send/message endpoints in favor of a new format. The Getting Started tutorials https://labs.hyperledger.org/firefly/gettingstarted/gettingstarted.html and possibly other places should be updated to reflect that, so new developers don't start implementing new projects against the deprecated paths.

It would be really nice to have the latest version of the swagger document published somewhere that people could just go and browse it, without having to actually start a FireFly instance. Perhaps this could be built into an existing GitHub action, or perhaps it is its own action that triggers after a successful merge to main.

If I create a subscription with the following options:

{
  "transport": "webhooks",
  "name": "myapp",
  "options": {
    "url": "http://localhost:9999"
  }
}

And then broadcast a message, FireFly makes a webhook request to that URL for the event, as expected. However, if I broadcast another message, the second message is never received. I think this is due to the fact that the first event is never acknowledged, despite the webhook handler returning an HTTP 200.

Stopping FireFly and starting it again results in the first event being sent again which leads me to believe the event is never being acknowledged.

Expected behavior:
Webhook subscriptions work the same before/after a restart.

Observed behavior:
Webhook subscriptions appear not to send HTTP requests after restarting FireFly core

Steps to reproduce:

• Start a new stack
• Create a webhook subscription
• Send a message to test that a webhook is successfully received from the other node
• Stop/start the stack again
• Send another message that should be sent to the webhook - it appears to never arrive

Deleting and re-creating the webhook subscription again appears to work

When Firefly is returning an empty response, there are two ways it's different from a standard HTTP that another connection is producing.

We want to start building out robust metrics in FireFly for better operations. As a start, we need to create a metrics endpoint that Prometheus can scrape. This should be pretty easy with the HTTP handler built into the Prometheus Go library: https://github.com/prometheus/client_golang.

As a basic demonstration of the new metrics capabilities, we could just add a simple counter for the number of API calls that have been made to the node. Any time an API endpoint is called, it would simply increment a counter.

As a demo of this functionality, it would be great to see the metrics being reported in Prometheus itself. This could simply be a local instance of Prometheus scraping logs from FireFly nodes running locally, set up by the CLI. The FireFly CLI does not need to set up Prometheus.

In Firefly Explorer, it would be useful to be able to fetch the config of my firefly node.

One use case I'm currently facing is determining the default namespace, this would be trivial if there was a way to access the config.

Currently you can implicitly create a group within a private message send, by specifying the group members and name in-line in the message itself. This will hash to ensure the same group is re-used when the same list of recipients is specified, with the same name, on the same ledger (on any member of the group).

However, it should be possible to create the groups explicitly, and it definitely needs to be possible to query the state of groups.

There's also potentially sub-state filtered by groups that might be very useful. Specifically:

• The contexts that have history on that group, and the nexthashes of each
• The sequence of pins for that group

... ideally it should be straightforward to use a set of instructions, to work out if a group is blocked, and hopefully work out which message is blocking a group.

Tracking the status of the asynchronous blob transfer feature, which still has some items remaining:

• Data exchange plugin points (streaming optimized)
  • Upload blob
  • Download blob
  • Transfer blob
  • New blob received notification
• HTTPS implementation
  • Upload blob
  • Download blob
  • Transfer blob
  • New blob received notification
• Core API
  • Upload blob
  • Download blob
  • Attach blob to broadcast message send
  • Attach blob to private message send
• Aggregator
  • Do not mark message confirmed until blob data is available
  • Rewind on arrival of blob data
• e2e test
  • Broadcast with blob data
  • Private message with blob data

*** Needs diagram ***

See discussion in: hyperledger/firefly-dataexchange-https#22
See engineering in: https://github.com/hyperledger-labs/firefly/pull/80

@vipinsun made a great observation on a community call on 30th June, that there's a whole space of innovation (such as The Graph) that listen to events from a blockchain, and build a complete (or semi-complete) off-chain GraphQL queryable store of the on-chain information.

The unanswered questions on the call was:

• Is the state of the technology such that such a layer could be introduce into FireFly in a way that's practical, and can support multiple protocols?
• Should this be a plugin capability that's orthogonal to the fundamental constructs that FireFly models (like tokens), or should we consider such a layer as a base layer under that modelling?

The full code history is in this repo, including Generation 1.
However, the architecture design of Generation 2, and collaboration history on things like PRs etc. isn't.

So I'm making this task (while we are in the pre-foundation process evaluation phase with Hyperledger) to take architecture diagrams and notes and put them into issues..

Particular on areas where in Generation 2 there is active code development happening, that is evolving quickly.

Expected behavior:
If a message is accepted by a FireFly node, whatever is in the value field should be put in the HTTP request body for a webhook subscription

Observed behavior:
When putting a plain string in the value field of a FireFly message, the HTTP request body on the received webhook is an empty object

Fungible and non-fungible token support is planned for FireFly across chains, beginning with Ethereum. One of the first architectural decisions involves selecting a Solidity contract as the reference implementation for tokens. Users may want (and may be encouraged) to define a custom contract in production, but the reference implementation should at a minimum:

1. be good enough for development usage
2. define the expected ABI for core token operations

Planned token functionality includes:

1. fungible and non-fungible token support
2. creation of multiple token types on a single smart contract (if possible)

For these requirements, ERC-1155 seems like the best starting point. However, due to its being a very general contract by nature, it may be difficult to find an existing implementation that exactly aligns with FireFly.

Summary of known ERC-1155 implementations:

• OpenZeppelin ERC1155: Extremely generic and not useful on its own. Implements safe transfers and balance tracking, but no mint or burn, no enforcement of supply or fungibility. Originally based on code by Enjin.
• OpenZeppelin ERC1155 Preset - Minter/Pauser: Extends the base contract with mint and pause functionality (and access control for those roles). Still no enforcement of supply or fungibility.
• OpenZeppelin ERC1155 Extension - Supply: Mix-in for tracking total supply of each token. Doesn't explicitly track fungible vs. non-fungible, but gives some control for creating a token with a supply of 1 and using that knowledge to infer that the token is non-fungible. Did not find any preset/concrete implementation that brings in this extension.
• Enjin ERC1155: Original reference implementation. Basis for the OpenZeppelin implementation. Also not useful on its own.
• Enjin ERC1155: Mixed Fungible/Mintable: Extension to the Enjin base contract. Seems to do exactly what we want - explicitly tracks fungible vs. non-fungible tokens. Has not been updated in some time and was written against solidity 0.5.0.
• Sequence ERC1155: Another base implementation (not connected to Enjin or OpenZeppelin). Also not useful on its own.
• OpenSea ERC1155 Tradable: Extension to the Sequence base contract. Provides mint, burn, and supply tracking. Has not been updated in some time and was written against solidity 0.5.12.

A convenience method that waits for the message_confirmed (or and error/timeout) before it returns from the synchronous POST call.

• Add a ?confirm query parameter to the following endpoints, which are all currently 202 Accepted only style
  • POST /namespaces
  • POST /namespaces/{ns}/broadcast/datatype
  • POST /namespaces/{ns}/broadcast/datatype
  • POST /namespaces/{ns}/send/datatype
  • POST /network/register/node
  • POST /network/register/node/organization
  • POST /network/register/organization
• Enhance the syncasync.Bridge implementation to support the following, or simlar:
  • A PrepareSend operation, that allocates an inflight ID
  • A ConfirmSend operation, that waits for a message_confirmed (or message_rejected or timeout)
  • A SendConfirm operation, that sends a prepared message and blocks for the response

Just like in ethconnect where we use ldflags to add build metadata (commit sha, tags, etc.) to the binary and log it at startup, we should then take it a step further and add a version command (similar to what helm, kubectl, go have) that then will output the same information.

Background

Firefly already solves the challenges of multi-party data flows, such as ensuring the consistency of a company's internal records when using a consortium's shared ledger as source of truth for these records. Users of public blockchains have similar challenges when it comes to ensuring the consistency of offchain applications with an onchain reference state. However, instead of agreed data formats and flows, what's agreed upon is the interface to smart contracts, such as ERC20 for tokens, or other dApps such as DeFi products.
This is currently under development https://labs.hyperledger.org/firefly/gettingstarted/custom_blockchain.html.

Motivation

Very high security guarantees for multi-party process flows come when the logic is moved on-chain.
While this introduces some issues with regards to data privacy, there are still a number of addressable use-cases.
In particular, Firefly could address the needs of existing public blockchain smart contract users (e.g. DeFi) when it comes to offchain integration.
A possible challenge will be the different state and execution models of different blockchains, and this issue is written in the context of Ethereum.

Requirements

Contracts

• Contracts exist on a network, at an address, with a specific ABI
• Any function of the ABI should be available to the Firefly clients
• To support proxy patterns, we need to allow any ABI for any address
• For public networks, there is no need for Firefly to deploy contracts
• For public networks, there is no need for permissioned access to contracts

Calls

• Smart contracts can return data without a transaction via calls
• Firefly clients should be able to make contract calls

Transactions

• All functions of any contract should be executable via transactions
• Firefly clients should be able to send contract transactions, and receive the associated events if any

Events

• Firefly clients should be able to watch contract addresses for desired events (e.g. ERC20 transfer)
• An 'event interface' should be defined so that offchain applications can interact with this data

Outline

To be discussed with the community on how the above can be implemented.

See #4 for info on why this is in Git

FireFly Internal Event Sequencing Model

One of the most important roles FireFly has, is to take actions being performed by the local apps, process them, get them confirmed, and then deliver back as "stream of consciousness" to the application alongside all the other events that are coming into the application from other FireFly Nodes in the network.

You might observe the problems solved in this architecture are similar to those in a message queuing system (like Apache Kafka, or a JMS/AMQP provider like ActiveMQ etc.).

However, we cannot directly replace the internal logic with such a runtime - because FireFly's job is to aggregate data from multiple runtimes that behave similarly to these:

• Private messaging in the Data Exchange
• The blockchain ledger(s) themselves, which are a stream of sequenced events
• The event dispatcher delivering messages to applications that have been sequenced by FireFly

So FireFly provides the convenient REST based management interface to simplify the world for application developers, by aggregating the data from multiple locations, and delivering it to apps in a deterministic sequence.

The sequence is made deterministic:

• Globally to all apps within the scope of the ledger, when a Blockchain ledger is used to pin events (see #10)
• Locally for messages delivered through a single FireFly node into the network
• Locally for all messages delivered to applications connected to a FireFly node, across blockchain

We have some problems in the codebase today with a half-formed identity approach:

• A Message contains a single Author field, and it is ambiguous whether this is the identity or the signing key
• A shim Identity plugin designed to resolve an opaque string (such as a DID) into an identity on the platform... but without a definition of what that identity is
• The requirement for us to properly solve Fabric signing keys and round-trip these to events like we do in ethereum - currently blocking merge of #184
• The fact that both Ethereum and Fabric connectors have the concept of abstract lookup keys for actual signing keys (HD Wallets in Ethereum, and CN shortnames in Fabric)
• The requirement to handle anonymous signing of transactions for privacy preservation
• The requirement that the database representation of identity in a Message / Token Pool or On-chain event (coming soon) etc. must be consistent in every database of members allowed to see the object, even if they didn't submit the transaction

This issue will track a number of threads needed to resolve this.

The E2E tests are fantastic for validating that a set of code changes maintains functionality from an end user perspective. They would be even better if they could be run against any arbitrary environment running outside a local dev machine (or CI instance). This would be very useful for validating that a specific FireFly environment is set up correctly and functions properly.

There is a bug in the latest code in master which causes a member to fail to insert a datatype into their database when another member has broadcasted a datatype.

Steps to reproduce:

• Start a new environment
  • The cli can be used for this: ff init test 2 && ff start test
• Make a POST to POST ​/namespaces​/{ns}​/broadcast​/datatype or using the swagger UI at http://127.0.0.1:5000/api#/default/postBroadcastDatatype
• View logs from each member. The receiving member will fail to insert the transaction.

firefly_core_1_1  | [2021-05-26T19:10:05.322Z] DEBUG SQL-> insert: INSERT INTO transactions (id,ttype,namespace,msg_id,batch_id,author,hash,created,protocol_id,status,confirmed,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)  RETURNING seq pid=1 role=event-manager
postgres_1_1      | 2021-05-26 19:10:05.324 UTC [144] ERROR:  null value in column "created" of relation "transactions" violates not-null constraint
postgres_1_1      | 2021-05-26 19:10:05.324 UTC [144] DETAIL:  Failing row contains (5, 40d04960-ba70-4944-a11b-e4d51a026290, pin, default, null, 12f427a6-b4e6-4078-8120-9bb55b8e0496, 0xf572721117c1271d4269675208ac7e97fa48ec74, 671171ff0e3f6b7f38e2f0eb80ac7a17f64e64bb608c4dc83bb1ac31a243900d, null, 0xd783030c47b48f5482d0d0e626ec4c2cdf4846f067cd9c3266bd0a6f91dd79..., confirmed, 1622056205314032707, {"data": {"txnId": "0x40d04960ba704944a11be4d51a0262900000000000...).
postgres_1_1      | 2021-05-26 19:10:05.324 UTC [144] STATEMENT:  INSERT INTO transactions (id,ttype,namespace,msg_id,batch_id,author,hash,created,protocol_id,status,confirmed,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)  RETURNING seq
firefly_core_1_1  | [2021-05-26T19:10:05.324Z] ERROR SQL insert failed: pq: null value in column "created" of relation "transactions" violates not-null constraint sql=[ INSERT INTO transactions (id,ttype,namespace,msg_id,batch_id,author,hash,created,protocol_id,status,confirmed,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)  RETURNING seq ]: pq: null value in column "created" of relation "transactions" violates not-null constraint pid=1 role=event-manager

We need a documentation page on how to use Webhooks with FireFly, similar to what we have for WebSockets. Specifically we should also call out on this page to use host.docker.internal rather than localhost when setting a callback URL to send webhooks to a server running on the developer's local machine.

Background

Blockchains, coupled with trusted compute techniques, and advances in cryptography allows us to build applications that are inherently secure, running on a new type of virtual machine with inherent security properties.

Motivation

A lot of members of the Hyperledger community have a deep understanding of the capabilities offered by the upcoming 'secure compute' revolution. Members of the Ethereum, ZKP, MPC, FHE communities share a deep understanding of the new type of applications, services and computers we can now build.
By working closely together, and agreeing on an architecture for secure computing which goes beyond what each of us is working on, we can hopefully build products and tools that interact well with one another. Kind of like what the OSI model has achieved for the internet.

Call to action

We're trying to outline a vision that the community agrees on, so that we can establish an architecture for secure compute.
The goal is to expand on the key concepts introduced by Firefly, https://labs.hyperledger.org/firefly/keyconcepts/multiparty.html, and get the word out to assemble a community who wants to make secure compute a reality.
For this purpose, we're releasing the https://www.securecompute.org/ manifesto and it seems very much aligned with the vision of the Firefly community.
We'd love to get feedback, especially on the proposed layer model, and practical proposals for problems we need to solve, these are outlined here. https://www.securecompute.org/atato-manifesto/how-we-will-solve-it

It's a long read, so we really appreciate anyone who'll take the time to read through this!

Checklist:

• Webhooks implementation
• Front-end request/reply
• Docs

Current FireFly support using REST to submit messages to a remote party, then a pluggable interface for events to trigger in that party where the only implementation right no is WebSockets (long lived bi-directional pipe from a remote runtime).

This issue proposes adding two features, that combine to achieve the above pattern:

1. A REST interface on the front-end that is a sync<->async bridge

• Takes the input payload and packages it into a message+data
• Sends to a single participant, or group, as specified in the URL of the call
• Blocks the REST call until a response with a cid (correlation ID) matching the request ID is received as an event
• To begin with will be RPC style input/output POST rather than a full generated REST interface

2. A WebHook plugin for events

• Takes messages that match the subscription, and extracts the data
• Packages that data in a HTTP POST to a configured server
• Waits for the response to the HTTP call
• Packages that response with the correct cid into a message on the same group hash that the message arrived on

Some users are brand new to Go, and struggle with the go install 1.16 route to getting the CLI up.
It would be good to have binary distributions published of the CLI for download, cross compiled for a number of platforms.

Hoping there's a git action off the shelf for this, that can be wired to creation of tags (similar to how the UI builds today).

Note this is a small step towards integration with binary package managers like brew and apt

It would be great if tag could be added to /data endpoint, like the way it works on /message. It will be easier to categorize and retrieve data.

See #4 for info on why this is in Git

FireFly Three Layered Plugin Architecture

In #6 the extensible nature of the FireFly node is described, with separate pluggable runtimes orchestrated into a common API for developers.

Here the mechanic of that pluggability for developers of new connectors is explained:

This architecture is designed to provide separations of concerns to account for:

• Differences in code language for the low-level connection to a backend (Java for Corda for example)
• Differences in transports, particularly for delivery of events:
  • Between FireFly Core and the Connector
    • Different transports other than HTTPS/WebSockets (GRPC etc.), and different wire protocols (socket.io, etc.)
  • Between the Connector and the underlying Infrastructure Runtime
    • Often this is heavy lifting engineering within the connector
• Differences in High Availability (HA) / Scale architectures
  • Between FireFly Core, and the Connector
    • Often for event management, and active/passive connector runtime is sufficient
  • Between the Connector and the Infrastructure Runtime
    • The infrastructure runtimes have all kinds of variation here... think of the potential landscape here from PostreSQL through Besu/Fabric/Corda, to Hyperledger Avalon and even Main-net ethereum

The "register node" and "register org" APIs don't take any parameters, but they require passing a JSON body of {} or they will return an error. It seems like these should be adjusted to allow an empty body.

Seeing the following in goroutine output:

Thread stuck waiting for close

goroutine 133 [chan receive, 7 minutes]:
github.com/hyperledger-labs/firefly/internal/events.(*eventDispatcher).close(0xc0001cc000)
	/firefly/internal/events/event_dispatcher.go:412 +0x11e
github.com/hyperledger-labs/firefly/internal/events.(*subscriptionManager).registerConnection(0xc000434960, 0xe862b8, 0x12c5200, 0xc000618a20, 0x24, 0xc00022e700, 0x0, 0x0)
	/firefly/internal/events/subscription_manager.go:333 +0x25e
github.com/hyperledger-labs/firefly/internal/events.(*boundCallbacks).RegisterConnection(0xc000426330, 0xc000618a20, 0x24, 0xc00022e700, 0x16, 0x1)
	/firefly/internal/events/bound_events_callbacks.go:30 +0x5e
github.com/hyperledger-labs/firefly/internal/events/websockets.(*WebSockets).start(0x12c5200, 0xc000618a20, 0x24, 0xc00049c4d0, 0xc49a20, 0xc00049c4d0)
	/firefly/internal/events/websockets/websockets.go:115 +0x10f
github.com/hyperledger-labs/firefly/internal/events/websockets.(*websocketConnection).handleStart(0xc0003d0380, 0xc00049c4d0, 0x200, 0xc49a20)
	/firefly/internal/events/websockets/websocket_connection.go:222 +0xac
github.com/hyperledger-labs/firefly/internal/events/websockets.(*websocketConnection).receiveLoop(0xc0003d0380)
	/firefly/internal/events/websockets/websocket_connection.go:147 +0x52a
created by github.com/hyperledger-labs/firefly/internal/events/websockets.newConnection
	/firefly/internal/events/websockets/websocket_connection.go:61 +0x20e

Multiple threads waiting in handleStart to get the mutex held by 133

goroutine 147 [semacquire, 7 minutes]:
sync.runtime_SemacquireMutex(0xc0004349b4, 0x0, 0x1)
	/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc0004349b0)
	/usr/local/go/src/sync/mutex.go:138 +0x105
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:81
github.com/hyperledger-labs/firefly/internal/events.(*subscriptionManager).registerConnection(0xc000434960, 0xe862b8, 0x12c5200, 0xc0002eacc0, 0x24, 0xc00022f180, 0x0, 0x0)
	/firefly/internal/events/subscription_manager.go:318 +0x5a9
github.com/hyperledger-labs/firefly/internal/events.(*boundCallbacks).RegisterConnection(0xc000426330, 0xc0002eacc0, 0x24, 0xc00022f180, 0x16, 0x1)
	/firefly/internal/events/bound_events_callbacks.go:30 +0x5e
github.com/hyperledger-labs/firefly/internal/events/websockets.(*WebSockets).start(0x12c5200, 0xc0002eacc0, 0x24, 0xc0004100b0, 0xc49a20, 0xc0004100b0)
	/firefly/internal/events/websockets/websockets.go:115 +0x10f
github.com/hyperledger-labs/firefly/internal/events/websockets.(*websocketConnection).handleStart(0xc000490280, 0xc0004100b0, 0x200, 0xc49a20)
	/firefly/internal/events/websockets/websocket_connection.go:222 +0xac
github.com/hyperledger-labs/firefly/internal/events/websockets.(*websocketConnection).receiveLoop(0xc000490280)
	/firefly/internal/events/websockets/websocket_connection.go:147 +0x52a
created by github.com/hyperledger-labs/firefly/internal/events/websockets.newConnection
	/firefly/internal/events/websockets/websocket_connection.go:61 +0x20e

• Support field-level i18n description keys on structures
  • Needs FireFly extensions to the Struct->JSON Schema parser
  • Could be contributed back openapi3gen in that community: https://github.com/getkin/kin-openapi
• Enhance enums (types etc.) so they show the possible values
  • Translation keys for each value
• Go through all the fftypes structures and add i18n keys

  • This could be a separate work item to the others, done by a different person, as it needs field level knowledge

@gabriel-indik was pointing out the importance of this one in a conversation, as he was exploring the Swagger

Currently you get a ganache CLI node, but there's no way to ask the CLI to use/create an existing Ganache UI node.
So if you want to see what's going on with the blockchain, it's a little hard - tailing logs.

Would be nice to have an option that lets you have a block explorer.

The function SubmitPinnedBatch lives in the broadcast package but is used by both broadcast and privatemessaging. Need a better place for this (and possibly any other common code between the two).

https://github.com/hyperledger-labs/firefly/blob/8fef7ce0e03899a097167e80b6d1cf0d938b313c/internal/broadcast/manager.go#L97

Running the npm i during the build of the firefly core docker is slow, and I've found hard to get reliable.

I think it might be better to have the firefly-ui repo do the build of the UI package, and then publish it into the release via a Git action.

Looks like a pretty straight forward example of the upload action here as an example:
https://trstringer.com/github-actions-create-release-upload-artifacts/

If the config file ~/.firefly/firefly.core exists, then make test fails (presumably because the config in that file overrides the config in the tests).

See #4 for info on why this is in Git

Tramlines view of the ping-pong between components

This demonstrates the problem that at it's core FireFly is there to solve. The internal plumbing complexity of just a very simple set of Enterprise blockchain / multi-party system interactions.

• Party A: Establish existence of a digital asset
  • Nothing more than some binary data (an image, a document, a specification etc. etc.)
• Party A: Broadcast some information about that asset to everyone, using blockchain to record, sequence and propagate
  • So people can find it, or part of a more sophisticated workflow
• Party B: Request the actual data - with evidence of that request tied to the blockchain
  • Including some private data that's sent to the Party A, reliably off-chain
• Party A: Authorize the request, and send the data privately to Party B
  • In this example there's no blockchain involved in this step

This is the kind of thing that enterprise projects have been solving ground-up since the dawn of enterprise blockchain, and the level of engineering required that is completely detached from business value, is very high.

The "tramlines" view shows how FireFly's pluggable model makes the job of the developer really simple:

• A few simple API calls from a modern web app
• Event triggered execution of application logic

This is deliberately a simple flow, and all kinds of additional layers might well layer on (and fit within the FireFly model):

• NFTs to track ownership etc. related to the digital asset
• Tokenized rewards/payments integrated with the authorization of the transfer of data
• Proof of deterministic execution of the logic to perform the authorization (on-chain, TEEs, ZKPs)
• Human workflow, that is of course completely non-deterministic
• Multiple additional process steps, deterministic or not
• Inclusion of multiple additional parties (maybe it's a request-for-tender, submit-tender flow for example)
• etc. etc.

@jimthematrix will be publishing a separate issue that discusses Delivery vs. Payment flows, which are a particularly interesting category of scenarios where FireFly APIs (in Generation 1) already add some simplification for users, and we expect to extend those (through the evolution of Generation 2).

Currently we use in-memory database change events (connected via Go channels in-process from the SQLCommon DB code, through the Database Interface callbacks, to the various components like the batch assembly, event dispatch etc.).

The following problems exists with the current code:

• They are only implemented for a couple of key data objects
• There's no way for the UI to subscribe externally to these local Database events to live update
• As we evolve to active/active FireFly nodes, there's no way for events to propagate to other servers in the cluster

There are two problems that such change events need to solve:

1. How the UI detects changes that might cause views to be refreshed (this issue)
2. How an active/active FireFly environment performs low-latency propagation of "new message"/"new pin"/"new event" information to other FireFly servers in the cluster (future issues)

Having investigated change events in different database technologies, it seems like they are cumbersome and complex to implement.

• Most only support the native primary key of a database
• They require database specific features to be used
• They can cause overhead on the database

So actually both (1) and (2) above can be supported by some slightly smaller extensions to the approach of in-memory go channel events that come from the FireFly node that makes a change. It can simply propagate those to the UI, or other servers over a WebSocket as follows:

• Add ephemeral WebSocket (only) events for the UI to listen externally to these database triggers
  • Emitted for any local database change that might be significant, with as much information as we have
    • The information might be very minimal in some cases (like something updated on a tablee)
  • Lightweight events that can only be subscribed to by ephemeral websocket
    • Do not go into the events table - which itself is a database update, that would cause such an event

Checklist:

• Design (in Git issue)
• Implement
• Docs

WORK IN PROGRESS

... output of this piece of work will include organizing and tagging issues in Git into a milestone.

Line item list

Below is a list of line items that are known requirements, and an initial mapping on which would be required to consider a version of FireFly to be a v1.0.0 release.

The high level requirements of what a v1.0.0 release is would include are:

• Stable API proven to solve a set of real-world problems - on-chain/off-chain coordination is the primary candidate here
• Proven on top of at least two blockchain technologies - Ethereum and Fabric are the primary candidates
• Sufficient non-functional testing of throughput and resiliency, including in fail-over scenarios
• Enough reference plugins for non-core technologies to allow proof of viability for production deployment

There are items below that are under active development, that are not considered a pre-req for meeting the requirements of a v1.0.0 release. Depending on readiness those items might obviously might be available functions at the point the milestone criteria are met.

... due to a PSQL constraint on the table that the data must exist when the message arrives

firefly_core_0_1  | [2021-06-21T05:21:04.275Z] DEBUG SQL-> insert: INSERT INTO messages_data (message_id,data_id,data_hash,data_idx) VALUES ($1,$2,$3,$4)  RETURNING seq pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.276Z] ERROR SQL insert failed: pq: insert or update on table "messages_data" violates foreign key constraint "messages_data_data_id_fkey" sql=[ INSERT INTO messages_data (message_id,data_id,data_hash,data_idx) VALUES ($1,$2,$3,$4)  RETURNING seq ]: pq: insert or update on table "messages_data" violates foreign key constraint "messages_data_data_id_fkey" pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.276Z]  WARN SQL! transaction rollback pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.276Z] ERROR Failed to insert message entry 0 in batch '3903f1ce-21e2-45a5-b697-a2313de03041': FF10116: Database insert failed: pq: insert or update on table "messages_data" violates foreign key constraint "messages_data_data_id_fkey" pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.277Z] ERROR Batch received from 0x426e73059b3c49b4a3fe0c778b0d8568610c7320/node_1 invalid: FF10116: Database insert failed: pq: insert or update on table "messages_data" violates foreign key constraint "messages_data_data_id_fkey" pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.277Z] ERROR private batch received attempt 69: FF10116: Database insert failed: pq: insert or update on table "messages_data" violates foreign key constraint "messages_data_data_id_fkey" pid=1 role=event-manager
firefly_core_0_1  | [2021-06-21T05:21:04.455Z] DEBUG Woken after poll timeout pid=1 role=batchmgr

The SQLCommon layer is very important across SQL databases as it:

• Reduces DB specific code
• Contains detailed e2e tests that are specific to the rules of each collection

However, there are some challenges when you look at taking the next step:

• The e2e tests run as UTs, and can only be run against the ql database
  • Cannot be used test migrations for other databases like PostgreSQL, SQLite etc.
  • Cannot be used to test non-SQL databases
• There is a lot of resource specific logic/optimization that must be applied to all databases
  • Collections with name uniqueness (namespaces, datatypes)
  • Collections with sequences that are used directly by code as int64 (sequence is universal for natural sort order)
  • Collections with UUID as the key on the API (common, but not universal - note sequence is primary key)
  • Collections with Hash as the key on the API (rare, just group at time of writing - note sequence is primary key)

I note that some detailed exploration was performed of various ways to reduce the boilerplate in the SQLCommon layer (even further than was possible with the common insert/update functions and filtering).
Something like an ORM, where there's metadata describing each resource-collection, that is sufficient to allow a generic function to handle all the various types.
However, every approach tried came with pretty horrible trade-offs, where the caller would have to do type coercion, or the boilerplate just moved to a different layer. As such this was paused to be re-visited as part of this issue.

Tried to build it on WSL 2 (Ubuntu 20.04) Win 10

Logs

$ docker build .
[+] Building 159.3s (4/9)                                                                                                                                                                                                                                                                          
[+] Building 351.6s (8/9)                                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                          0.1s
 => => transferring dockerfile: 164B                                                                                                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                             0.0s
 => => transferring context: 82B                                                                                                                                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/golang:1.16.4-alpine3.13                                                                                                                                                                                                                  13.5s
 => [internal] load build context                                                                                                                                                                                                                                                             0.4s
 => => transferring context: 8.58MB                                                                                                                                                                                                                                                           0.3s
 => [1/5] FROM docker.io/library/golang:1.16.4-alpine3.13@sha256:4dd403b2e7a689adc5b7110ba9cd5da43d216cfcfccfbe2b35680effcf336c7e                                                                                                                                                           335.1s
 => => resolve docker.io/library/golang:1.16.4-alpine3.13@sha256:4dd403b2e7a689adc5b7110ba9cd5da43d216cfcfccfbe2b35680effcf336c7e                                                                                                                                                             0.0s
 => => sha256:722a834ff95bfd3dac4bc5aae498c245eb76b98108ed9de4293df2596e60cf1a 5.16kB / 5.16kB                                                                                                                                                                                                0.0s
 => => sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba 2.81MB / 2.81MB                                                                                                                                                                                              154.5s
 => => sha256:adcc1eea9eeabb6de296adb3e0c1b0722cf13251ff3e4e2d0a5f7ed8e3d48342 281.27kB / 281.27kB                                                                                                                                                                                           11.8s
 => => sha256:4c4ab2625f07be8d5c6e48046a05ff3ecc7f374b794a926fb62247b66b511909 154B / 154B                                                                                                                                                                                                    3.8s
 => => sha256:4dd403b2e7a689adc5b7110ba9cd5da43d216cfcfccfbe2b35680effcf336c7e 1.65kB / 1.65kB                                                                                                                                                                                                0.0s
 => => sha256:9dd1788d4bd0df3006d79a88bda67cb8357ab49028eebbcb1ae64f2ec07be627 1.36kB / 1.36kB                                                                                                                                                                                                0.0s
 => => sha256:c5e7595549f7536d76f08a8a23fb67e3e6fae08ccf3add715c5c1c956f9445d2 105.75MB / 105.75MB                                                                                                                                                                                          319.6s
 => => sha256:3df88182f7acff97ffde9f614a0fd86e8a26590e445aa76e442c3a79d9e4c4f4 155B / 155B                                                                                                                                                                                                   13.7s
 => => extracting sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba                                                                                                                                                                                                     0.6s
 => => extracting sha256:adcc1eea9eeabb6de296adb3e0c1b0722cf13251ff3e4e2d0a5f7ed8e3d48342                                                                                                                                                                                                     0.5s
 => => extracting sha256:4c4ab2625f07be8d5c6e48046a05ff3ecc7f374b794a926fb62247b66b511909                                                                                                                                                                                                     0.0s
 => => extracting sha256:c5e7595549f7536d76f08a8a23fb67e3e6fae08ccf3add715c5c1c956f9445d2                                                                                                                                                                                                    14.5s
 => => extracting sha256:3df88182f7acff97ffde9f614a0fd86e8a26590e445aa76e442c3a79d9e4c4f4                                                                                                                                                                                                     0.0s
 => [2/5] WORKDIR /firefly                                                                                                                                                                                                                                                                    1.2s
 => [3/5] ADD . .                                                                                                                                                                                                                                                                             0.3s
 => ERROR [4/5] RUN apk add make gcc build-base                                                                                                                                                                                                                                               1.2s
------
 > [4/5] RUN apk add make gcc build-base:
#8 0.679 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
#8 0.961 139758786403144:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#8 0.966 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/main: Permission denied
#8 0.966 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory
#8 0.966 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
#8 1.149 139758786403144:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#8 1.155 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/community: Permission denied
#8 1.155 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory
#8 1.156 ERROR: unable to select packages:
#8 1.159   build-base (no such package):
#8 1.159     required by: world[build-base]
#8 1.159   gcc (no such package):
#8 1.159     required by: world[gcc]
#8 1.159   make (no such package):
#8 1.159     required by: world[make]
------
executor failed running [/bin/sh -c apk add make gcc build-base]: exit code: 3

Docker Version

n$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:17:50 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Info

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.6.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 49.99GiB
 Name: docker-desktop
 ID: LQHA:FUMR:B7GW:5PW3:2KQM:DI7U:OYGG:KHNQ:D2OK:57BZ:3ZV4:RF3K
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 41
  Goroutines: 45
  System Time: 2021-05-13T23:29:07.0781329Z
  EventsListeners: 3
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Requirement

The identity plugin needs to be enhanced to handle requirements from different blockchain protocols. While the Ethereum identities are global, a single address string representing the unique account, Fabric identities are scoped to organizations.

• example Ethereum identity: 0x170dcf0187fb2f8884b63cfa2b1573c37f32eeb8
• example Fabric identity: Org1MSP::x509::CN=user001,OU=client::CN=fabric_ca.org1.example.com,OU=Hyperledger Fabric,O=org1.example.com,L=San Francisco,ST=California,C=US, which contain these segments:
  • MSP ID
  • certificate type
  • user id
  • CA ID

While both types of identities can be represented in a string format, it'd be very cumbersome to require developers to input the full Fabric string as the signer property in the API payload.

Practically, a Fabric client would be tied to a single Fabric CA with a particular org MSP, so the API payload only needs the user ID portion to be specified, user001 in the above example. But as soon as the identity gets into the FireFly core, it should be resolved to the fully qualified string before it gets replicated to other FireFly nodes.

In addition, the Fabric chaincode spits out the signer identity in the chaincode event in the fully qualified format, which means the matching logic in the aggregator won't work unless FireFly core resolves the supplied signer identity to use the same format.

Proposed Solution 1

Provide separate identity plugins for each protocol. this means a new protocol support requires at least 2 plugins, the blockchain plugin and the identity plugin. there will be some overlap between the two because the blockchain plugin is also responsible for validating the identity strings

Proposed Solution 2

Have the OnChain identity plugin consult the blockchain plugin for resolving the identity. this requires inter-plugin dependencies as the blockchain plugin must be initialized ahead of the identity plugin