If there is a new version of firefly or other components of the stack available, I want to be able to upgrade my local dev stack to use the new version, without losing any of my data.

It's important that I be able to upgrade without data loss, and test my app against the new version because this is what developers will need to do in a production environment. Verifying that the new version is backwards compatible with existing data is very important.

There are a lot of things that need to happen the first time a stack starts up.

• Docker volumes get created and config files get copied to them
• A data directory gets created
• Smart contracts are deployed
• Network members are registered

A failure at any one of these steps could result in a stack that's just not usable. I'm wondering if we want to catch errors that are thrown, and automatically run a resetStack so the user can just try the start command again?

When running ff start, nodes intermittently fail to register with the following error:

Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp 127.0.0.1:5000: connect: connection refused

Furthermore, the registration is not re-attempted on subsequent starts, so you generally have to remove and recreate the stack.

OS: macOS Version 11.3

commands run:

1. ff init trial
2. ff start trial

Snippet from the terminal:

firefly % ff start trial --verbose
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
starting FireFly stack 'trial'... initializing blockchain node
/usr/local/bin/docker-compose down --volumes
trial Warning: No resource found to remove
Error: docker run --rm -v trial_geth:/data ethereum/client-go:release-1.9 --datadir /data --nousb init /data/genesis.json
Failed [1] INFO [10-01|19:14:55.834] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:14:55.834] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:14:55.835] Set global gas cap cap=25000000
INFO [10-01|19:14:55.835] Allocated cache and file handles database=/data/geth/chaindata cache=16.00MiB handles=16
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
INFO [10-01|19:14:55.844] Persisted trie from memory database nodes=3 size=455.00B time="19.156µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)

• all changes rolled back
  Usage:
  ff start <stack_name> [flags]

Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
-b, --no-rollback Do not automatically rollback changes if first time setup fails

Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output

Error: docker run --rm -v trial_geth:/data ethereum/client-go:release-1.9 --datadir /data --nousb init /data/genesis.json
Failed [1] INFO [10-01|19:14:55.834] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:14:55.834] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:14:55.835] Set global gas cap cap=25000000
INFO [10-01|19:14:55.835] Allocated cache and file handles database=/data/geth/chaindata cache=16.00MiB handles=16
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
INFO [10-01|19:14:55.844] Persisted trie from memory database nodes=3 size=455.00B time="19.156µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)

• all changes rolled back

firefly logs command is broken. Please find the detailed error below.

ff logs dev1
getting logs...
Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

The CLI needs to perform special one-time initialization the first time a stack is started. To detect if the stack has been run before, currently it checks for the existence of a cert.pem file used by the dataexchange service. We should not really rely on that, but should instead define a specific config value or touch a file to note that first-time setup is complete. The value must also be reset during an ff reset command.

Error saying volume already exists but was not created by Docker Compose. Upgrading to docker compose v2.2.2 solves the issue.

https://github.com/docker/compose/releases/tag/v2.2.2

It would be nice to have a cli command to generate the firefly.core skeleton.

We should add an ff info command to the CLI that will show which version of each component is in a stack

When I execute:

ff init -b besu dev1

and then:

ff start dev1

I get:

reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
⡿ Error: an error occurred - rolling back changes... Error: docker-compose up -d
Failed [1] Creating network "dev1_default" with the default driver
Creating volume "dev1_ethconnect_abis_0" with default driver
Creating volume "dev1_ethconnect_abis_1" with default driver
Creating volume "dev1_ethconnect_events_0" with default driver
Creating volume "dev1_ethconnect_events_1" with default driver
Creating volume "dev1_ipfs_data_0" with default driver
Creating volume "dev1_ipfs_data_1" with default driver
Creating volume "dev1_ipfs_staging_0" with default driver
Creating volume "dev1_ipfs_staging_1" with default driver
Creating volume "dev1_member1tessera" with default driver
Creating volume "dev1_public-keys" with default driver
Creating dev1_validator1_1 ... 
Creating dev1_ipfs_0       ... 
Creating dev1_member1tessera_1 ... 
Creating dev1_ipfs_1           ... 
Creating dev1_dataexchange_0   ... 
Creating dev1_dataexchange_1   ... 
Creating dev1_ipfs_1           ... done
Creating dev1_member1tessera_1 ... done
Creating dev1_ipfs_0           ... done
Creating dev1_validator1_1     ... done
Creating dev1_rpcnode_1        ... 
Creating dev1_dataexchange_0   ... done
Creating dev1_dataexchange_1   ... done
Creating dev1_rpcnode_1        ... done

ERROR: for firefly_core_0  Container "3d03ad4e2b1a" is unhealthy.

ERROR: for ethsigner  Container "3d03ad4e2b1a" is unhealthy.

ERROR: for member1besu  Container "3d03ad4e2b1a" is unhealthy.
Encountered errors while bringing up the project.
 - all changes rolled back
Usage:
  ff start <stack_name> [flags]

Flags:
  -h, --help          help for start
  -b, --no-rollback   Do not automatically rollback changes if first time setup fails

Global Flags:
      --ansi string   control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
  -v, --verbose       verbose log output

Error: docker-compose up -d
Failed [1] Creating network "dev1_default" with the default driver
Creating volume "dev1_ethconnect_abis_0" with default driver
Creating volume "dev1_ethconnect_abis_1" with default driver
Creating volume "dev1_ethconnect_events_0" with default driver
Creating volume "dev1_ethconnect_events_1" with default driver
Creating volume "dev1_ipfs_data_0" with default driver
Creating volume "dev1_ipfs_data_1" with default driver
Creating volume "dev1_ipfs_staging_0" with default driver
Creating volume "dev1_ipfs_staging_1" with default driver
Creating volume "dev1_member1tessera" with default driver
Creating volume "dev1_public-keys" with default driver
Creating dev1_validator1_1 ... 
Creating dev1_ipfs_0       ... 
Creating dev1_member1tessera_1 ... 
Creating dev1_ipfs_1           ... 
Creating dev1_dataexchange_0   ... 
Creating dev1_dataexchange_1   ... 
Creating dev1_ipfs_1           ... done
Creating dev1_member1tessera_1 ... done
Creating dev1_ipfs_0           ... done
Creating dev1_validator1_1     ... done
Creating dev1_rpcnode_1        ... 
Creating dev1_dataexchange_0   ... done
Creating dev1_dataexchange_1   ... done
Creating dev1_rpcnode_1        ... done

ERROR: for firefly_core_0  Container "3d03ad4e2b1a" is unhealthy.

ERROR: for ethsigner  Container "3d03ad4e2b1a" is unhealthy.

ERROR: for member1besu  Container "3d03ad4e2b1a" is unhealthy.
Encountered errors while bringing up the project.
 - all changes rolled back

We use a string for member identifiers throughout the codebase. However, right now the strings are generated by simply incrementing a counter as each member's config is generated. Users should be able to customize the name of each member both interactively and by passing flags.

The IPFS nodes that FireFly CLI creates should be part of their own private network, and not be connected to public IPFS. We do not want data uploaded to a local FireFly to be accessible to the public internet. This will require some work on the generated docker-compose.yaml file.

Expected Behavior

ff start [stack_name] should start up stack completely per user prompted information

Current Behavior

Upon initializing the stack and issuing the ff start [stack_name] command, containers are spinning up but the postgres containers are failing with apparent permissions errors which cause other containers to fail due to inability to connect to the database container.

Context (Environment)

• OS: Windows 10
• Terminal: Git Bash
• Go version: go1.16.5 windows/amd64
• Docker version: 20.10.6
• Firefly versions:
  

Attempted Manual Corrections

I have executed chmod -R 0755 ~/.firefly/stacks/[stack_name]/data/postgres_[N] on each of the indicated node folders.

macOS

zsh: bad CPU type in executable: ff

Some users are brand new to Go, and struggle with the go install 1.16 route to getting the CLI up.
It would be good to have binary distributions published of the CLI for download, cross compiled for a number of platforms.

Hoping there's a git action off the shelf for this, that can be wired to creation of tags (similar to how the UI builds today).

Note this is a small step towards integration with binary package managers like brew and apt

This has lead to sporadic timeout failures in FireFly E2E tests such as this one: https://github.com/hyperledger/firefly/runs/4026832368?check_suite_focus=true

The pull command should pull all images required to run the stack.

It would be awesome to have the cli distributed & managed through homebrew (and other package managers), so users do not need to manually upgrade versions, etc.

I'm seeing the blow error with increasing frequency when running make e2e in FireFly Core.
I believe the problem is the CLI is terminating the FireFly core process while it's competing the automigrations.

Few potential enhancements I can see that might resolve this:

1. Move to the new admin.preinit feature, so the restart is not necessary
2. Implement a healthcheck on the FireFly core such that it only reports "up" once it's ready
3. Don't start FireFly core at all during the Ethconnect initialization step - leave it out of docker-compose.yml until later

Symptom in CLI (after long delay)

...
registering org_0 and node_0
Error: Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp [::1]:5000: connect: connection refused
Usage:
  ff start <stack_name> [flags]

Flags:
  -h, --help      help for start
  -n, --no-pull   Do not pull latest images when starting

Global Flags:
      --ansi string   control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
  -v, --verbose       verbose log output

Error: Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp [::1]:5000: connect: connection refused
make: *** [e2e] Error 1

Underlying logs in core:

 docker compose logs -f firefly_core_0
firefly_core_0_1  | [2021-06-21T02:52:37.294Z] DEBUG Log level: debug pid=1
firefly_core_0_1  | [2021-06-21T02:52:37.295Z]  INFO Project Firefly pid=1
firefly_core_0_1  | [2021-06-21T02:52:37.295Z]  INFO © Copyright 2021 Kaleido, Inc. pid=1
firefly_core_0_1  | [2021-06-21T02:52:37.295Z] DEBUG Debug HTTP endpoint listening on localhost:6060 pid=1
firefly_core_0_1  | [2021-06-21T02:52:39.525Z]  INFO Shutting down due to terminated pid=1
firefly_core_0_1  | [2021-06-21T02:52:44.156Z] DEBUG Log level: debug pid=1
firefly_core_0_1  | [2021-06-21T02:52:44.156Z]  INFO Project Firefly pid=1
firefly_core_0_1  | [2021-06-21T02:52:44.156Z]  INFO © Copyright 2021 Kaleido, Inc. pid=1
firefly_core_0_1  | [2021-06-21T02:52:44.157Z] DEBUG Debug HTTP endpoint listening on localhost:6060 pid=1
firefly_core_0_1  | Error: FF10163: Database migration failed: FF10163: Database migration failed: Dirty database version 19. Fix and force version.
firefly_core_0_1  | Usage:
firefly_core_0_1  |   firefly [flags]
firefly_core_0_1  |   firefly [command]
firefly_core_0_1  | 
firefly_core_0_1  | Available Commands:
firefly_core_0_1  |   help        Help about any command
firefly_core_0_1  |   showconfig  List out the configuration options
firefly_core_0_1  | 
firefly_core_0_1  | Flags:
firefly_core_0_1  |   -f, --config string   config file
firefly_core_0_1  |   -h, --help            help for firefly
firefly_core_0_1  | 
firefly_core_0_1  | Use "firefly [command] --help" for more information about a command.
firefly_core_0_1  | 
firefly_core_0_1  | FF10163: Database migration failed: FF10163: Database migration failed: Dirty database version 19. Fix and force version.

ipfs containers fail on startup after using ff cli with ipfs/go-ipfs:latest image as referenced in this issue. The temporary fix mentioned in the issue is to revert to v0.10.0.

ff cli workaround is manually editing the docker-compose file to v0.10.0 in order for the stack to start. Local firefly stack running with workaround appears to function but runs into 2 E2E test failures TestE2EBroadcastBlob and TestE2EPrivateBlobDatatypeTagged.

E2E Test Output below
ipfs_issue_e2e_logs.txt

We should add unit tests to verify CLI functionality. We should probably also move some of the logic out of the cmd module and into the internal module.

In order to make sure PRs are acceptable, we should be testing that the code compiles and passes all the linting rules added in #42

A FireFly stack uses Docker to expose quite a few ports. Unfortunately when docker-compose brings up a stack, if there is something already listening on a port that it is exposing, it silently fails to forward the port to the container. The stack appears to come up correctly, but some services likely will not work correctly.

The CLI should check all the ports required by the stack before starting it to make sure they are free. If the port is not free, the CLI should print an error explaining which port needs to be freed and then exit.

If any of the docker/docker-compose commands wrapped by ff exit with an error (non-zero) status, this should be treated as an error and surfaced by the CLI. Currently it checks for errors at the Go level but does not check exit codes.

Previously, if a named volume was created before running docker-compose up it would still be removed when running docker-compose down --volumes. In docker-compose 2.0.x this no longer appears to be the case. This causes all kinds of problems and confusion when the CLI reports that it has rolled changes back, but leftover volumes still have old data on them, causing subsequent runs to fail.

I just generated a 2 party stack (as part of running the firefly E2E tests), and some of the URLs in the config are mismatched. Notice in the file below, it is for org_0 and points to ipfs_0 and dataexchange_0, but also points to ethconnect_1 and tokens_1. This causes some strange failures since certain actions go to the wrong container.

firefly_core_0.yaml

log:
  level: debug
debug:
  port: 6060
http:
  port: 5000
  address: 0.0.0.0
  publicURL: http://127.0.0.1:5000
admin:
  port: 5101
  address: 0.0.0.0
  enabled: true
  preinit: true
  publicURL: http://127.0.0.1:5101
ui:
  path: ./frontend
node:
  name: node_0
org:
  name: org_0
  identity: 0xee08e985af4544e46dd735cf13080a39a8761146
blockchain:
  type: ethereum
  ethereum:
    ethconnect:
      url: http://ethconnect_1:8080
      instance: /contracts/firefly
      topic: "1"
database:
  type: sqlite3
  sqlite3:
    url: /etc/firefly/db?_busy_timeout=5000
    migrations:
      auto: true
publicstorage:
  type: ipfs
  ipfs:
    api:
      url: http://ipfs_0:5001
    gateway:
      url: http://ipfs_0:8080
dataexchange:
  https:
    url: http://dataexchange_0:3000
tokens:
- connector: https
  name: erc1155
  url: http://tokens_1:3000

When I try to get /webui/ on the IPFS container, it redirects me to:
/ipfs/bafybeif4zkmu7qdhkpf3pnhwxipylqleof7rl6ojbe7mq3fzogz6m4xk3i/

However, it then doesn't return, and I assume that's because that needs to be published to the IPFS network as part of the CLI initialization ❔

https://github.com/ipfs/go-ipfs/blob/2c431eb825b3a8ab39eafdf29c41fcc4c25c529d/core/corehttp/webui.go#L4

Would be nice to:

• list which stack is running (if any)
• get a friendly error if you attempt to start a stack when another one is already running

Could also discuss if running multiple stacks simultaneously should be supported - right now every stack picks the same port numbers so they will always conflict.

This needs a bit more investigation. It seems like sometimes FireFly may be closing the HTTP connection before responding when the /config/reset endpoint is called, but FireFly has code to specifically make sure it finishes the response before restarting the HTTP server. It may be easier to gracefully handle the EOF in the CLI, but I'm not sure that's the "right" answer here.

This behavior ends up resulting in the CLI rolling back the stack. If the user simply tries again it usually works, but this happens often enough that it's really annoying.

ff start <stack_name>
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
⣟ Error: an error occurred - rolling back changes... Error: Post "http://localhost:5201/admin/api/v1/config/reset": EOF - all changes rolled back
Usage:
  ff start <stack_name> [flags]

Flags:
  -h, --help          help for start
  -n, --no-pull       Do not pull latest images when starting
  -b, --no-rollback   Do not automatically rollback changes if first time setup fails

Global Flags:
      --ansi string   control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
  -v, --verbose       verbose log output

Error: Post "http://localhost:5201/admin/api/v1/config/reset": EOF - all changes rolled back

We should add an ff ls command to list the stacks that currently exist. It would also be nice to indicate if any of them are running or not.

There are a couple of potential issues with the init command right now:

• The stack name can be created with " " which causes all kinds of problems, including files being written to the wrong directory
  - Somehow it is possible to create a stack with only one member. We need to test and improve the validation logic here.

Windows 10 19041.vb_release.191206-1406
Docker 20.10.7
Ubuntu 20.04.2 LTS
WSL2

user@WinDev2104Eval:/mnt/c/Users/User$ ff start dev
reading stack config... done
starting FireFly stack 'dev'...
this will take a few seconds longer since this is the first time you're running this stack...
doneError: docker-compose up -d
Failed [1] Starting dev_dataexchange_1_1 ...
Recreating dev_ganache_1      ...
Recreating dev_ipfs_1_1       ...
Starting dev_dataexchange_0_1 ...
Recreating dev_ipfs_0_1       ...
Starting dev_postgres_0_1     ...
Starting dev_postgres_1_1     ...
Starting dev_postgres_1_1     ... error

ERROR: for dev_postgres_1_1  Cannot start service postgres_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/b897274b7ee0a84e4b7a08f01135f7abf252252387a6bfd4a83f4d48be882275" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_dataexchange_0_1 ... error

ERROR: for dev_dataexchange_0_1  Cannot start service dataexchange_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f6af0a37e04f2cabccc680a8ec01fefbb2e6b72f8af9fc12ac2b418d1b6f11ab" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_postgres_0_1     ... error

ERROR: for dev_postgres_0_1  Cannot start service postgres_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/6ff0531043d38f4b8ce90f6bbcf311b1211f9b53d3de8d3fc725d6472b2bc693" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_dataexchange_1_1 ... error

ERROR: for dev_dataexchange_1_1  Cannot start service dataexchange_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/aafe84939634b8d96987e93acabc4150d998cb9829da0f60274d65768945ca13" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
Recreating dev_ipfs_1_1       ... done
Recreating dev_ipfs_0_1       ... done
Recreating dev_ganache_1      ... done

ERROR: for postgres_1  Cannot start service postgres_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/b897274b7ee0a84e4b7a08f01135f7abf252252387a6bfd4a83f4d48be882275" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown

ERROR: for dataexchange_0  Cannot start service dataexchange_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f6af0a37e04f2cabccc680a8ec01fefbb2e6b72f8af9fc12ac2b418d1b6f11ab" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown

ERROR: for postgres_0  Cannot start service postgres_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/6ff0531043d38f4b8ce90f6bbcf311b1211f9b53d3de8d3fc725d6472b2bc693" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown

ERROR: for dataexchange_1  Cannot start service dataexchange_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/aafe84939634b8d96987e93acabc4150d998cb9829da0f60274d65768945ca13" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown

ERROR: for ethconnect_0  Container "d2b2ab1a1280" is unhealthy.

ERROR: for ethconnect_1  Container "d2b2ab1a1280" is unhealthy.
Encountered errors while bringing up the project.

Usage:
  ff start <stack_name> [flags]

Flags:
  -h, --help      help for start
  -n, --no-pull   Do not pull latest images when starting

Global Flags:
      --ansi string   control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
  -v, --verbose       verbose log output

Error: docker-compose up -d

Would mean syntax highlighting and other helpful features in editors.

Steps to Reproduce:

• Create two firefly stacks, both with prometheus enabled: ff init stackone 2 --prometheus-enabled; ff init stacktwo 2 --prometheus-enabled
• Start and stop stack one: ff start stackone; ff stop stackone
• Try to start stack two: ff start stacktwo
• See error: ERROR: for prometheus Cannot create container for service prometheus: Conflict. The container name "/prometheus" is already in use by container <container_name>. You have to remove (or rename) that container to be able to reuse that name.

To keep consistency across codebases, we should add the linters to the Makefile to this repo from https://github.com/hyperledger-labs/firefly/blob/main/Makefile

the command is not deleting local data and configuration even after executing the command

Using Git Bash on Windows, the promptui functions seem to get skipped over, including validation which leads to problems. For instance, it's possible to create a stack with no name, or with an invalid number of members. More testing is needed on this to determine why this shell behaves differently.

A few releases back (can't remember exactly when) this config was deprecated:

It's now org.key.
Would be good for the CLI to switch over, as it will print a warning on every start up that the old config is used.

Note that the same is now true in v0.13.0 for dataexchange.https here (now dataexchange.ffdx):

firefly % ff start trial --verbose
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
starting FireFly stack 'trial'... initializing blockchain node
/usr/local/bin/docker-compose down --volumes
trial Warning: No resource found to remove
Error: docker run --rm -v /Users/kaulgudramer/.firefly/stacks/trial/blockchain:/geth -v trial_geth:/data ethereum/client-go:release-1.9 --nousb account import --password /geth/password --keystore /data/keystore /geth/0/keyfile
Failed [1] INFO [10-01|19:15:54.069] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:15:54.069] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:15:54.070] Set global gas cap cap=25000000
Fatal: Could not create the account: account already exists
Fatal: Could not create the account: account already exists

• all changes rolled back
  Usage:
  ff start <stack_name> [flags]

Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
-b, --no-rollback Do not automatically rollback changes if first time setup fails

Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output

Error: docker run --rm -v /Users/kaulgudramer/.firefly/stacks/trial/blockchain:/geth -v trial_geth:/data ethereum/client-go:release-1.9 --nousb account import --password /geth/password --keystore /data/keystore /geth/0/keyfile
Failed [1] INFO [10-01|19:15:54.069] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:15:54.069] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:15:54.070] Set global gas cap cap=25000000
Fatal: Could not create the account: account already exists
Fatal: Could not create the account: account already exists

• all changes rolled back

CLI version: 0.0.44

Running into an issue unlocking accounts when creating a stack with 3 members, geth seems to be crashing

INFO [02-23|21:36:24.190] Maximum peer count                       ETH=50 LES=0 total=50
INFO [02-23|21:36:24.191] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [02-23|21:36:24.191] Sanitizing cache to Go's GC limits       provided=1024 updated=662
INFO [02-23|21:36:24.192] Set global gas cap                       cap=50,000,000
WARN [02-23|21:36:24.192] Sanitizing invalid miner gas price       provided=0    updated=0
INFO [02-23|21:36:24.193] Allocated trie memory caches             clean=99.00MiB dirty=165.00MiB
INFO [02-23|21:36:24.193] Allocated cache and file handles         database=/data/geth/chaindata cache=329.00MiB handles=524,288
INFO [02-23|21:36:24.247] Opened ancient database                  database=/data/geth/chaindata/ancient readonly=false
INFO [02-23|21:36:24.248] Initialised chain configuration          config="{ChainID: 2021 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: <nil>, Berlin: <nil>, London: <nil>, Arrow Glacier: <nil>, MergeFork: <nil>, Engine: clique}"
INFO [02-23|21:36:24.248] Initialising Ethereum protocol           network=2021 dbversion=<nil>
INFO [02-23|21:36:24.249] Loaded most recent local header          number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
INFO [02-23|21:36:24.249] Loaded most recent local full block      number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
INFO [02-23|21:36:24.249] Loaded most recent local fast block      number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
WARN [02-23|21:36:24.249] Failed to load snapshot, regenerating    err="missing or corrupted snapshot"
INFO [02-23|21:36:24.250] Rebuilding state snapshot
INFO [02-23|21:36:24.250] Resuming state snapshot generation       root=0f51ed..0d5486 accounts=0 slots=0 storage=0.00B elapsed="394.166µs"
INFO [02-23|21:36:24.251] Generated state snapshot                 accounts=3 slots=0 storage=210.00B elapsed=1.102ms
INFO [02-23|21:36:24.252] Regenerated local transaction journal    transactions=0 accounts=0
INFO [02-23|21:36:24.253] Stored checkpoint snapshot to disk       number=0 hash=c408a8..dce2ac
INFO [02-23|21:36:24.253] Gasprice oracle is ignoring threshold set threshold=2
WARN [02-23|21:36:24.254] Error reading unclean shutdown markers   error="leveldb: not found"
INFO [02-23|21:36:24.254] Starting peer-to-peer node               instance=Geth/v1.10.16-stable-20356e57/linux-arm64/go1.17.7
INFO [02-23|21:36:24.265] New local node record                    seq=1,645,652,184,264 id=1b6f59cf886b6815 ip=127.0.0.1 udp=0 tcp=30311
INFO [02-23|21:36:24.265] Started P2P networking                   self="enode://b910ef363120bcf2b33df7baa44078acdce7c2c0385974ae6cc692fc45629a942cde2edfbdc9a7de73673570d33b847c9ad725525cd82fadbbc6c08dc311bdbf@127.0.0.1:30311?discport=0"
INFO [02-23|21:36:24.267] IPC endpoint opened                      url=/data/geth.ipc
INFO [02-23|21:36:24.267] HTTP server started                      endpoint=[::]:8545 prefix= cors= vhosts=*
INFO [02-23|21:36:24.267] Transaction pool price threshold updated price=0
INFO [02-23|21:36:24.267] Updated mining threads                   threads=0
INFO [02-23|21:36:24.267] Transaction pool price threshold updated price=0
INFO [02-23|21:36:24.267] Etherbase automatically configured       address=0xa24220382c330594Cce66469eD33C913a3A6d991
INFO [02-23|21:36:24.267] Commit new sealing work                  number=1 sealhash=4a0634..06b743 uncles=0 txs=0 gas=0 fees=0 elapsed="214.458µs"
WARN [02-23|21:36:24.268] Block sealing failed                     err="sealing paused while waiting for transactions"
INFO [02-23|21:36:24.268] Commit new sealing work                  number=1 sealhash=4a0634..06b743 uncles=0 txs=0 gas=0 fees=0 elapsed="339.291µs"

Due to licensing issues, we should remove dependencies on go-ethereum packages. These should be fairly easy to remove as the only thing they are used for right now is generating key pairs and public addresses.

There is no requirement for the messaging blockchain and the tokens blockchain to be the same - FireFly can fully support using Fabric as the primary blockchain but using Ethereum ERC1155 for tokens. Currently the CLI blocks this combination.

The setup logic should be adjusted to properly handle this combination - which means spinning up containers for both ethconnect and fabconnect.

This issue was reported by ksanjayk in Rocket Chat:

i think this error is due to ethconnect starting before than ganache and its unable to connect to ganche, maybe fixing the starting of container order will solve it

For developers working on core, it would be handy to have a mode that generates the YAML files for running core outside of docker-compose using the exposed ports.

This is a manual activity today, that has to be re-done each time on the ff remove / ff init cycle:

• ff init / ff start -n
• Edit docker-compose.yml to comment out one or more firefly_core_0 services
• Edit firefly_1.core etc. to change all the docker link hostnames, into localhost:510x hostnames
• ff stop / ff start
• Run FireFly core (using VSCode launch.json etc)

On setting up a new dev environment it's easy to miss steps, especially if you're in a hurry, or if you're like me and think you know what you're doing without reading the instructions. For example, if you install docker but not docker-compose (on Linux they're two different packages), some things will work, but other things will not. It would be great to have some checks to make sure prerequisites are in place before starting to try to use docker or docker-compose. If things are not set up correctly, it would be nice for the CLI to print out a helpful error message. Right now, if docker-compose is not installed, usually the error that gets printed is very cryptic.

The docker daemon not running in the background is another common issue that people sometimes run into.

Lastly, the current user might not have permission to run docker commands without sudo if they're not in the right group. It would be great if the CLI could run quick pre-check to make sure all these things are set up correctly. I think this would really help new devs setting up their environment for the first time.

If the docker compose version is v2.0.0, the name of the container is prefixed with a '-' instead of '_' causing the below error-

Error: docker cp trial2_firefly_core_0_1:/firefly/contracts /Users/isriniva/.firefly/stacks/trial2
Failed [1] Error: No such container:path: trial2_firefly_core_0_1:/firefly/contracts

Actual container created -

docker container ls
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED         STATUS                   PORTS                                                                          NAMES
96d2ade0afc4   ghcr.io/hyperledger/firefly:latest                      "firefly"                5 minutes ago   Up 4 minutes             0.0.0.0:5000->5000/tcp, 0.0.0.0:5101->5101/tcp                                 trial2-firefly_core_0-1
04b5732a640d   ghcr.io/hyperledger/firefly:latest                      "firefly"                5 minutes ago   Up 4 minutes             0.0.0.0:5001->5001/tcp, 0.0.0.0:5201->5201/tcp                                 trial2-firefly_core_1-1

➜  firefly-cli git:(main) docker --version
Docker version 20.10.8, build 3967b7d
➜  firefly-cli git:(main) docker-compose --version
Docker Compose version v2.0.0
➜  firefly-cli git:(main)

The "usage" info for the active CLI command is always dumped when an error is encountered. This should only be dumped for errors that stem from invalid syntax, not from functional errors executing the command.

If you point the CLI to a local manifest with -m manifest.json, and that manifest contains a local Docker definitions such as:

  "tokens-erc1155": {
    "image": "ghcr.io/hyperledger/firefly-tokens-erc1155",
    "local": true
  }

When running ff start, the CLI still pulls the latest image from the server, overwriting the local image.

This seems to be due to a duplication in the PullStack method here. It first iterates the images defined in the manifest, and does properly skip those tagged as local. But then it goes on to pull the latest for all images defined by the blockchain and token plugins, defeating the purpose of the earlier loop.

Side node: because it first pulls all the images defined in the manifest, it will pull images even if you're ultimately not using them (ie it pulls fabconnect even if your blockchain provider is Ethereum).

Conclusion from all this: we need better "pull" logic to intersect the info from these two places - it should only pull the Dockers actually used in the stack, but it should also honor the versions and "local" flag defined for those images in the manifest.

With hyperledger/firefly#321, firefly will now have a metrics section in its config file for configuring the Prometheus metrics server it comes with.

ff needs to be updated to include templating / generating this section of config. And then as a potential enhancement to the local dev experience, I think it would be great if we had a CLI option to also standup a Prometheus and Grafana via docker-compose when starting a stack. These would automatically know to scrape all FireFlys w/in a network for metrics.

CLI README should have advanced usage examples:

• Initialize the stack with the custom manifest.json file
• Running a specific release of FireFly

like in "https://hyperledger.github.io/firefly/contributors/advanced_cli_usage.html" link

The ff version command is currently not printing the correct information. This is likely a build time issue with some flags not being set correctly.