Comments (13)
nguyer
commented on July 17, 2024
1
Hey, sorry I have been heads down on some other stuff hand haven't had a chance to look at this one yet. Perhaps @sukrut29 is able to take a look at it as he is the author of a lot of the besu-related code in the FireFly CLI?
from firefly-cli.
sukrut29
commented on July 17, 2024
1
Hey @nguyer thanks for pointing it out. I can take a look. I was myself using a firefly stack with besu a couple of days ago, and it was working fine. @paul-snively which version of firefly-cli are you using?
I can fire up a new stack with besu and test it myself. I ll keep you posted.
from firefly-cli.
nguyer
commented on July 17, 2024
Thanks for reporting this @paul-snively. I'm going to move this issue over to the firefly-cli repo, because this is likely a configuration problem and needs to be addressed in that codebase.
from firefly-cli.
paul-snively
commented on July 17, 2024
Hi! Just wanted to ask if anyone has had any bandwidth to look into this. Thanks!
from firefly-cli.
paul-snively
commented on July 17, 2024
@nguyer, any thoughts on this? It appears to have something to do with liveness checks on the Besu containers, but I can't see how.
from firefly-cli.
sukrut29
commented on July 17, 2024
from firefly-cli.
sukrut29
commented on July 17, 2024
works fine with me. That's what I was wondering, because I have been using Firefly stack with Hyperledger Besu, since we made that intergration. @paul-snively can you maybe try to start a stack with -v flag for verbose, so we can dig a bit deeper into what's happening.
According to your error message it seems like the health checks aren't working properly. And that the docker-compose cannot start up the network. Either case, with verbose logs, we can maybe try to find out what's wrong.
from firefly-cli.
nguyer
commented on July 17, 2024
For me, the ff start command just hangs at a certain point waiting for besu nodes to come up. Here's the verbose log:
ff start besu --verbose --no-rollback
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
starting FireFly stack 'besu'... initializing blockchain node
/usr/local/bin/docker volume create besu_ethsigner_keys
besu_ethsigner_keys
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/config/ethsigner:/ethSigner -v besu_ethsigner_keys:/usr/local/bin/keyFiles --entrypoint /ethSigner/Nodejs.sh node:latest
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
added 363 packages, and audited 364 packages in 18s
57 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
npm notice
npm notice New patch version of npm available! 8.5.1 -> 8.5.3
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.5.3>
npm notice Run `npm install -g [email protected]` to update!
npm notice
added 1 package, and audited 365 packages in 558ms
57 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Files: [ '0', '1' ]
Inside keygen
Keyfile: {"version":3,"id":"daa5364a-1858-4ef2-acb7-f71d94a7dd94","address":"50cc0f39e0e0a5bebad021c3e87a48072160bd08","crypto":{"ciphertext":"056b2abe0132c4d85f24fa2e0ee9e96d25314b18e6bb29861c7a41643a8f8f40","cipherparams":{"iv":"79be23240f1b685a95cf1bda9a214c75"},"cipher":"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"9a2ac2365ef3f82eec68436005d705752b625d07b7a2c1b9572c1c9af7aa00eb","n":8192,"r":8,"p":1},"mac":"775dba323f8edd9cbae880d58068fc94d4f754ad31f216b7d60c85d0d36d9866"}}
Inside keygen
Keyfile: {"version":3,"id":"a68571b5-3106-4890-b5a6-83be4fb2b62a","address":"fb3ba3ad729d155d9dfd2475511015c8b5bb1845","crypto":{"ciphertext":"9967bbefbd7b069dc8e9d6017c39b68f32da196faca3a3351d459356da714c1e","cipherparams":{"iv":"a325bfdc5fcd9b2ddcddc155d51ad06f"},"cipher":"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"8ec7d6de543945589fb05727b1e2f9b23f7f8bfc73ade695336a7d113eb3109b","n":8192,"r":8,"p":1},"mac":"d62aa7ed658a84f1703d66ac5c1b34d162d8959c3bad04c7884561ba58ba1372"}}
Success
Success
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/configs/ethconnect_0.yaml:/source/ethconnect_0.yaml -v besu_ethconnect_config_0:/dest alpine cp /source/ethconnect_0.yaml /dest/config.yaml
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/configs/ethconnect_1.yaml:/source/ethconnect_1.yaml -v besu_ethconnect_config_1:/dest alpine cp /source/ethconnect_1.yaml /dest/config.yaml
writing data exchange certs
/usr/local/bin/docker run --rm -v besu_dataexchange_0:/dest alpine mkdir -p /dest/peer-certs
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_0/config.json:/source/config.json -v besu_dataexchange_0:/dest alpine cp /source/config.json /dest/config.json
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_0/cert.pem:/source/cert.pem -v besu_dataexchange_0:/dest alpine cp /source/cert.pem /dest/cert.pem
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_0/key.pem:/source/key.pem -v besu_dataexchange_0:/dest alpine cp /source/key.pem /dest/key.pem
/usr/local/bin/docker run --rm -v besu_dataexchange_1:/dest alpine mkdir -p /dest/peer-certs
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_1/config.json:/source/config.json -v besu_dataexchange_1:/dest alpine cp /source/config.json /dest/config.json
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_1/cert.pem:/source/cert.pem -v besu_dataexchange_1:/dest alpine cp /source/cert.pem /dest/cert.pem
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/data/dataexchange_1/key.pem:/source/key.pem -v besu_dataexchange_1:/dest alpine cp /source/key.pem /dest/key.pem
copying firefly.core to firefly_core_0
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/configs/firefly_core_0.yml:/source/firefly_core_0.yml -v besu_firefly_core_0:/dest alpine cp /source/firefly_core_0.yml /dest/firefly.core
copying firefly.core to firefly_core_1
/usr/local/bin/docker run --rm -v /Users/nguyer/.firefly/stacks/besu/configs/firefly_core_1.yml:/source/firefly_core_1.yml -v besu_firefly_core_1:/dest alpine cp /source/firefly_core_1.yml /dest/firefly.core
starting FireFly dependencies
/usr/local/bin/docker-compose up -d
Network besu_default Creating
Network besu_default Created
Volume "besu_ipfs_staging_0" Creating
Volume "besu_ipfs_staging_0" Created
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_ethconnect_config_1\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_firefly_core_1\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_dataexchange_0\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_firefly_core_0\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_dataexchange_1\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
Volume "besu_ipfs_data_1" Creating
Volume "besu_ipfs_data_1" Created
Volume "besu_ethconnect_events_1" Creating
Volume "besu_ethconnect_events_1" Created
Volume "besu_ethconnect_abis_0" Creating
Volume "besu_ethconnect_abis_0" Created
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_ethsigner_keys\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
Volume "besu_member1tessera" Creating
Volume "besu_member1tessera" Created
Volume "besu_ipfs_data_0" Creating
Volume "besu_ipfs_data_0" Created
time="2022-03-04T08:01:38-05:00" level=warning msg="volume \"besu_ethconnect_config_0\" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
Volume "besu_ipfs_staging_1" Creating
Volume "besu_ipfs_staging_1" Created
Volume "besu_ethconnect_abis_1" Creating
Volume "besu_ethconnect_abis_1" Created
Volume "besu_ethconnect_events_0" Creating
Volume "besu_ethconnect_events_0" Created
Volume "besu_public-keys" Creating
Volume "besu_public-keys" Created
Container besu_ipfs_1 Creating
Container besu_ipfs_0 Creating
Container besu_dataexchange_0 Creating
Container besu-member1tessera-1 Creating
Container besu_dataexchange_1 Creating
Container besu-validator1-1 Creating
Container besu_dataexchange_0 Created
Container besu_dataexchange_1 Created
Container besu_ipfs_0 Created
Container besu_ipfs_1 Created
Container besu-member1tessera-1 Created
Container besu-validator1-1 Created
Container besu-rpcnode-1 Creating
Container besu-member1besu-1 Creating
Container besu-rpcnode-1 Created
Container besu-ethsigner-1 Creating
Container besu-member1besu-1 Created
Container besu-ethsigner-1 Created
Container besu_ethconnect_0 Creating
Container besu_ethconnect_1 Creating
Container besu_ethconnect_0 Created
Container besu_tokens_0_0 Creating
Container besu_ethconnect_1 Created
Container besu_tokens_1_0 Creating
Container besu_tokens_0_0 Created
Container besu_tokens_1_0 Created
Container besu_firefly_core_0 Creating
Container besu_firefly_core_1 Creating
Container besu_firefly_core_0 Created
Container besu_firefly_core_1 Created
Container besu_dataexchange_0 Starting
Container besu-member1tessera-1 Starting
Container besu_dataexchange_1 Starting
Container besu_ipfs_0 Starting
Container besu_ipfs_1 Starting
Container besu-validator1-1 Starting
Container besu_dataexchange_1 Started
Container besu_dataexchange_0 Started
Container besu_ipfs_1 Started
Container besu_ipfs_0 Started
Container besu-validator1-1 Started
Container besu-member1tessera-1 Started
Container besu-rpcnode-1 Starting
Container besu-rpcnode-1 Started
The RPC node and validator health checks are never passing:
ffe9283f6e23 hyperledger/besu:latest "/config/validator_n…" 51 seconds ago Up 49 seconds (health: starting) 8545-8547/tcp, 30303/tcp, 0.0.0.0:8555-8556->8555-8556/tcp besu-rpcnode-1
c8186b1c1f68 hyperledger/besu:latest "/config/bootnode_de…" 52 seconds ago Up 50 seconds (health: starting) 8545-8547/tcp, 30303/tcp besu-validator1-1
cca6d3b30b12 quorumengineering/tessera:21.7.0 "/config/tessera_def…" 52 seconds ago Up 50 seconds (healthy) 9000/tcp, 9101/tcp, 0.0.0.0:9081->9080/tcp besu-member1tessera-1
59efa645f763 ghcr.io/hyperledger/firefly-dataexchange-https "docker-entrypoint.s…" 52 seconds ago Up 50 seconds 3001/tcp, 0.0.0.0:5205->3000/tcp besu_dataexchange_1
221141940c28 ipfs/go-ipfs:v0.10.0 "/sbin/tini -- /usr/…" 52 seconds ago Up 50 seconds 4001/tcp, 8081/tcp, 4001/udp, 0.0.0.0:5206->5001/tcp, 0.0.0.0:5207->8080/tcp besu_ipfs_1
76684e1275d6 ipfs/go-ipfs:v0.10.0 "/sbin/tini -- /usr/…" 52 seconds ago Up 50 seconds 4001/tcp, 8081/tcp, 4001/udp, 0.0.0.0:5106->5001/tcp, 0.0.0.0:5107->8080/tcp besu_ipfs_0
b0d3b648c7e9 ghcr.io/hyperledger/firefly-dataexchange-https "docker-entrypoint.s…" 52 seconds ago Up 50 seconds 3001/tcp, 0.0.0.0:5105->3000/tcp besu_dataexchange_0
I've attached logs from the RPC node and the validator.
rpc_node_log.txt
validator_logs.txt
from firefly-cli.
nguyer
commented on July 17, 2024
It's also worth pointing out that I've recently moved to working on an M1 based Mac, so my machine has a native CPU architecture of ARM_64 now. I'm not sure if that's related, but I have seen that cause some issues with other things because docker ends up pulling an ARM image which is slightly different or has different packages available from the expected x86_64 image.
from firefly-cli.
sukrut29
commented on July 17, 2024
that could be, because on my Razor Blade-15 with ubuntu 20.04 it works perfectly fine.
I can't seem to recreate this error, no matter how many stacks I spin up
from firefly-cli.
nguyer
commented on July 17, 2024
I had some cycles to re-test this today on an x86_64 Linux machine and I'm seeing the same behavior as on my Mac, so I don't think that's the issue. I think something changed and the healthchecks just aren't passing anymore. I started with a completely clean slate (removed ~/.firefly/stacks and did a docker system prune -a), and was unable to bring up a new besu stack.
I have attached the logs from ff as well as the RPC node and validator nodes, both of which Docker is reporting as unhealthy.
@sukrut29 Do you mind taking another look at this please?
ff_start.log
docker_ps.log
rpcnode_1.log
validator_1.log
from firefly-cli.
nguyer
commented on July 17, 2024
A couple of things I noticed when digging deeper into this
- The besu images in the docker config are using
latest- which means that the exact image that gets pulled is unpredictable. It could work for one person one day, and then not for someone else on the next day, if a new image was published that changes something. I'm pretty sure that's what happened here. - It looks like
curlis no longer in these images, which is being used for the health checks. I did some poking around and it looks like at least the Besu RPC node image has a healthcheck defined in its Dockerfile. I'm not sure if that's sufficient for us to use here, or if we need to get creative on the healthcheck.
So I'd recommend the following actions:
- Do not use
latesttags on any images we're pulling, but rather stick to a specific version that we know works. That way we can control when the stack is upgraded to a new version and make sure it's tested before we roll it out to people. - Fix the health checks :)
@sukrut29 Do you think you'd be able to tackle the above?
from firefly-cli.
nguyer
commented on July 17, 2024
Quick update on this one: I've opened a PR which significantly simplifies the Besu stack setup and resolves the issues described above: #166
from firefly-cli.
Related Issues (20)
- Besu networks only: 'ff upgrade' results in unusable stack. The ff_core-nodes fail to start - Log message: FF10412 - Bad network version: %!!(MISSING)s(<nil>)
- ff info command only works with --verbose flag HOT 1
- Recipients don't appear on sandbox homescreen, for sending private messages HOT 6
- Documentation for "--release" option to init is incorrect HOT 1
- 500 - failed to invoke backing implementation of 'InstallChaincode' HOT 2
- Builds failing due to goreleaser deprecation/removal of `replacements`
- Add --path to "ff" commands HOT 8
- Fabconnect exited(1) HOT 11
- Redundant command to list stack on local machine
- Add support for Fabric v2.5 LTS release
- info command should output stack information
- runtime shell completion for stack names
- looking for help for setup evm connect only connect to hyperledger besu HOT 1
- Failed to find installed chaincode HOT 2
- Failing Unit Tests
- Enhance CLI arguments
- Environment deletion fails with FF CLI built from `main` HOT 6
- More control over docker-compose content HOT 5
- Error: failed to find installed chaincode HOT 1
- ff init command does not work HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from firefly-cli.