Comments (8)
SamMayWork
commented on June 26, 2024
RC1 - Migration Testing
Some pre-RC testing has already been done in this space, so the aim of this testing is to double check the existing testing and other permutations of FireFly configurations which have not yet been checked. From the original comment we know we don't have coverage for migration testing in these areas:
- Testing in multi-party and gateway mode
- PostgreSQL as DB
- ETHConnect as blockchain connector
Additionally, we'll also need to do migration testing in the areas we have covered pre-RC:
- ERC20/721/1155 tokens connectors
- EVM Connector as blockchain connector
- Fabric
Using this issue to track permutations of testing...
General steps to migrate a FireFly stack
- A freshly built CLI from source (to ensure you have the latest commits)
- Create and run a normal stack
- Run E2E tests (using tests from the commit your stack is based from)
- Build new images using commits from the release you want to move to
- Update the Docker Compose Override file with the new images
- Upgrade the batch pin contract (see below)
- Restart the containers
- Run the E2E tests (using tests from the commit you've moved to)
- Verify that all data from the previous run of the E2E is still available
To upgrade the batch pin contract:
- Deploy the new contract
- Update the config file for each node with the new address and block number
- Restart the nodes
POSTto/network/actionswith payload{"type": "terminate"}- Verify with a
GET /statuscall that the new contract is in use
...or alternatively use the script from the original comment to automate this process.
Very quick and dirty migration script
Will contribute this formally into a script somewhere when it's not thrown-together hackiness but this script does a semi-automatic migration.
#!/bin/bash
export STACK_NAME=migration
export CREATE_STACK=false
ff init migration
ff start migration
sleep 10
cd ./firefly
make e2e
cd ..
ff stop migration
cat >>~/.firefly/stacks/migration <<EOL
...
EOL
services:
dataexchange_0:
image: localhost:7000/firefly-dataexchange-https:latest
dataexchange_1:
image: localhost:7000/firefly-dataexchange-https:latest
evmconnect_0:
image: localhost:7000/firefly-evmconnect:latest
evmconnect_1:
image: localhost:7000/firefly-evmconnect:latest
firefly_core_0:
image: localhost:7000/firefly:latest
firefly_core_1:
image: localhost:7000/firefly:latest
tokens_0_0:
image: localhost:7000/firefly-tokens-erc20-erc721:latest
tokens_1_0:
image: localhost:7000/firefly-tokens-erc20-erc721:latest
ff start migration
./contract-migration migration ~/.firefly/stacks ~/firefly
cd ~/firefly
git checkout main
git pull
make e2e| Multi-party | DB Provider | Blockchain Connector | Tokens Connector | Passed? | Tested by |
|---|---|---|---|---|---|
| N | PostgreSQL | EVMConnect | None | ✅ | @nguyer |
| Y | PostgreSQL | EVMConnect | None | ✅ | @nguyer |
| N | PostgreSQL | EVMConnect | ERC20/721 | ✅ | @nguyer |
| Y | PostgreSQL | EVMConnect | ERC20/721 | ✅ | @nguyer |
| Y | PostgreSQL | EVMConnect | ERC1155 | ✅ | @nguyer |
| N | PostgreSQL | EVMConnect | ERC1155 | ✅ | @nguyer |
| N | PostgreSQL | Fabconnect | None | ✅ | @nguyer |
| Y | PostgreSQL | Fabconnect | None | ✅ | @nguyer |
from firefly.
SamMayWork
commented on June 26, 2024
RC1 - Functionality Check
from firefly.
SamMayWork
commented on June 26, 2024
RC1 - Performance Testing
Will start looking at the performance of RC1 soon, but in the mean time, I've kicked a test of 1.2.2 to gather some performance metrics as a reference point, I'll put the configuration and results below.
1.2.2 Release Commit
1.3-rc1 Release Commit
Reference performance testing for 1.2.2
nohup ./start.sh &> ffperf.log &
core-config.yml
log:
level: debug
broadcast:
batch:
size: 200
timeout: 1s
privatemessaging:
batch:
size: 200
timeout: 1s
message:
writer:
count: 5
download:
worker:
count: 100
publicstorage:
ipfs:
api:
requestTimeout: 2s
gateway:
requestTimeout: 2s
ethconnect.yml
rest:
rest-gateway:
maxTXWaitTime: 120
maxInFlight: 200
alwaysManageNonce: true
attemptGapFill: true
sendConcurrency: 3
gasEstimationFactor: 2.0
confirmations:
required: 5
debug:
port: 6000
instances.yml
stackJSONPath: /home/ubuntu/.firefly/stacks/1-2-2-perf-test/stack.json
wsConfig:
wsPath: /ws
readBufferSize: 16000
writeBufferSize: 16000
initialDelay: 250ms
maximumDelay: 30s
initialConnectAttempts: 5
heartbeatInterval: 5s
instances:
- name: long-run
tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
length: 500h
sender: 0
recipient: 1
messageOptions:
longMessage: false
tokenOptions:
tokenType: fungible
contractOptions: {"address": "0xfe1a8867fc460fe5696cb316b2649788b74ec46d"}
FireFly git commit:
d0fb82d64cfeb2848b0a32a6bc286d5b9ade87ea
from firefly.
nguyer
commented on June 26, 2024
Steps that I am following for validating migration scenarios:
Multiparty Tests
- Deploy a v1.2.2 stack
- Enable multiparty mode (run the
./hack/multiparty.shscript) - Send a broadcast (using the sandbox)
- Send a private message
- Upgrade to v1.3.0
- Send a broadcast (still using the old batch pin contract)
- Upgrade the contract (run
./hack/multiparty.shagain) - Send a broadcast
- Send a private message
- Verify transactions in FireFly UI
Tokens Tests
- After installing, deploy a token contract (via FF API)
- Mint some tokens
- Transfer one
- Burn one
- Transfer one with a message attached
- Verify transactions and balances in FireFly UI
- If using ERC20/721 connector, repeat for each
from firefly.
EnriqueL8
commented on June 26, 2024
RC4 - Performance Testing
Have started a RC4 performance testing with the below options
Reference Performance test options
nohup ./start.sh &> ffperf.log &
core-config.yml
log:
level: debug
broadcast:
batch:
size: 200
timeout: 1s
privatemessaging:
batch:
size: 200
timeout: 1s
message:
writer:
count: 5
download:
worker:
count: 100
publicstorage:
ipfs:
api:
requestTimeout: 2s
gateway:
requestTimeout: 2s
ethconnect.yml
rest:
rest-gateway:
maxTXWaitTime: 120
maxInFlight: 200
alwaysManageNonce: true
attemptGapFill: true
sendConcurrency: 3
gasEstimationFactor: 2.0
confirmations:
required: 5
debug:
port: 6000
instances.yml
stackJSONPath: /home/ubuntu/.firefly/stacks/enrique-test/stack.json
wsConfig:
wsPath: /ws
readBufferSize: 16000
writeBufferSize: 16000
initialDelay: 250ms
maximumDelay: 30s
initialConnectAttempts: 5
heartbeatInterval: 5s
instances:
- name: long-run
tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
length: 500h
sender: 0
recipient: 1
messageOptions:
longMessage: false
tokenOptions:
tokenType: fungible
contractOptions: {"address": "0xf4e5e921cf78de3c503623bb91230c4e54cf91cb"}
FireFly git commit:
577e8c47680c6230209a74829921a9c427766af8
from firefly.
EnriqueL8
commented on June 26, 2024
Run Report RC4
Started: 10/04/24
Duration: ~5 hours
Git commit: [49410c52653e143e8a17bd9ab58ba2423f564714]
Node Configuration
2 FireFly nodes on one virtual server (EC2 m4.xlarge)
Entire FireFly stack is local to the server (ie both blockchains, Postgres databases, etc)
Single geth node with 2 instances of ethconnect
Maximum time to confirm before considering failure = 1 minute
Reference Performance test options
core-config.yml
log:
level: debug
broadcast:
batch:
size: 200
timeout: 1s
privatemessaging:
batch:
size: 200
timeout: 1s
message:
writer:
count: 5
download:
worker:
count: 100
publicstorage:
ipfs:
api:
requestTimeout: 2s
gateway:
requestTimeout: 2s
ethconnect.yml
rest:
rest-gateway:
maxTXWaitTime: 120
maxInFlight: 200
alwaysManageNonce: true
attemptGapFill: true
sendConcurrency: 3
gasEstimationFactor: 2.0
debug:
port: 6000
instances.yml
stackJSONPath: /home/ubuntu/.firefly/stacks/latest/stack.json
wsConfig:
wsPath: /ws
readBufferSize: 16000
writeBufferSize: 16000
initialDelay: 250ms
maximumDelay: 30s
initialConnectAttempts: 5
heartbeatInterval: 5s
instances:
- name: long-run
tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
length: 500h
sender: 0
recipient: 1
messageOptions:
longMessage: false
tokenOptions:
tokenType: fungible
contractOptions: {"address": "0x528adc5c826721ba6a40342ad5918a3499f9663c"}
FireFly git commit:
49410c52653e143e8a17bd9ab58ba2423f564714
NOTE: confirmations set to 0
Results
Broadcast messages: 199,448
Private messages: 235,242
Token mints: 18,443
Transactions: 89,198
No errors
Summary result:
INFO[2024-04-09T16:32:43.924] Shutdown summary:
INFO[2024-04-09T16:32:43.924] - Prometheus metric sent_mints_total = 18447.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric sent_mint_errors_total = 0.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric mint_token_balance = 0.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric received_events_total = 1097482.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric incomplete_events_total = 0.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric delinquent_msgs_total = 0.000000
INFO[2024-04-09T16:32:43.924] - Prometheus metric actions_submitted_total = 530299.000000
INFO[2024-04-09T16:32:43.924] - Test duration: 4h58m57.296804144s
INFO[2024-04-09T16:32:43.924] - Measured actions: 1097105
INFO[2024-04-09T16:32:43.924] - Measured send TPS: 61.167763
INFO[2024-04-09T16:32:43.924] - Measured throughput: 61.163341
INFO[2024-04-09T16:32:43.924] - Measured send duration: min: 11.556354ms, max: 1.485517803s, avg: 151ms
INFO[2024-04-09T16:32:43.924] - Measured event receiving duration: min: 2.030708026s, max: 1m4.571629481s, avg: 6.414s
INFO[2024-04-09T16:32:43.924] - Measured total duration: min: 2.030708026s, max: 1m4.571629481s, avg: 6.414s
Grafana results:
(Note I modified the Grafana dashboard to add the transfer submitted to the broadcast submitted ahead of #1490 getting merged)
I find the heatmap not particularly useful, so this a view with histograms to see on average how long it takes to confirm:
Compared to the testing from 1.2, the number look slightly better based on the above testing.
I did notice that the TPS and time to confirm grows overtime
from firefly.
EnriqueL8
commented on June 26, 2024
Run Report RC2
Started: 24/05/24
Duration: ~27 hours
Git commit: [b2f86880a109d17751c3481ea1a72c9a2e94dd28]
Node Configuration
2 FireFly nodes on one virtual server (EC2 m4.xlarge)
Entire FireFly stack is local to the server (ie both blockchains, Postgres databases, etc)
Single geth node with 2 instances of ethconnect
Maximum time to confirm before considering failure = 1 minute
Reference Performance test options
core-config.yml
log:
level: debug
broadcast:
batch:
size: 200
timeout: 1s
privatemessaging:
batch:
size: 200
timeout: 1s
message:
writer:
count: 5
download:
worker:
count: 100
publicstorage:
ipfs:
api:
requestTimeout: 2s
gateway:
requestTimeout: 2s
ethconnect.yml
rest:
rest-gateway:
maxTXWaitTime: 120
maxInFlight: 200
alwaysManageNonce: true
attemptGapFill: true
sendConcurrency: 3
gasEstimationFactor: 2.0
debug:
port: 6000
instances.yml
stackJSONPath: /home/ubuntu/.firefly/stacks/rc2-latest/stack.json
wsConfig:
wsPath: /ws
readBufferSize: 16000
writeBufferSize: 16000
initialDelay: 250ms
maximumDelay: 30s
initialConnectAttempts: 5
heartbeatInterval: 5s
instances:
- name: long-run
tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
length: 500h
sender: 0
recipient: 1
messageOptions:
longMessage: false
tokenOptions:
tokenType: fungible
contractOptions: {"address": "0x4c05f4e749304da29017e4eb3e5e2a4aaa84e637"}
subscriptionOptions:
batch: true
batchTimeout: 250ms
readAhead: 50
FireFly git commit:
b2f86880a109d17751c3481ea1a72c9a2e94dd28
NOTE: confirmations set to 0
Results
Broadcast messages: 1,000,039
Private messages: 1,424,155
Token mints: 96,353
Transactions: 540K
No errors
Summary result:
INFO[2024-04-25T13:57:39.019] Shutdown summary:
INFO[2024-04-25T13:57:39.020] - Prometheus metric sent_mints_total = 96373.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric sent_mint_errors_total = 0.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric mint_token_balance = 0.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric received_events_total = 6008344.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric incomplete_events_total = 0.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric delinquent_msgs_total = 0.000000
INFO[2024-04-25T13:57:39.020] - Prometheus metric actions_submitted_total = 2907814.000000
INFO[2024-04-25T13:57:39.020] - Test duration: 27h32m18.966742004s
INFO[2024-04-25T13:57:39.020] - Measured actions: 6008025
INFO[2024-04-25T13:57:39.020] - Measured send TPS: 60.604479
INFO[2024-04-25T13:57:39.020] - Measured throughput: 60.602054
INFO[2024-04-25T13:57:39.020] - Measured send duration: min: 8.687266ms, max: 6.013594408s, avg: 174ms
INFO[2024-04-25T13:57:39.021] - Measured event receiving duration: min: 2.007866751s, max: 54.132299131s, avg: 6.473s
INFO[2024-04-25T13:57:39.021] - Measured total duration: min: 2.007866751s, max: 54.132299131s, avg: 6.473s
Grafana results:
I find the heatmap not particularly useful, so this a view with histograms to see on average how long it takes to confirm:
from firefly.
nguyer
commented on June 26, 2024
This is great! Thank you so much for all the work on this, @EnriqueL8
from firefly.
Related Issues (20)
- Status call returns error when node is not registered to a multiparty namespace HOT 2
- Protocol ID behaviour inconsistent with docs HOT 3
- Migration Issue with ERC-1155 Token Pools HOT 5
- FireFly gets stuck processing events from an unknown listener HOT 1
- FireFly 1.3 Performance Testing HOT 7
- Transaction failure reason not visible when using fixed gas amount HOT 8
- Intermittent Unit Test - TestDownloadManagerStartupRecoveryCombinations
- Error for "unable to resolve contract method" has a missing string
- Enhance documentation with signing knowledge
- Enhance multiparty documentation HOT 2
- FireFly does not attempt to re-connect to token connectors on Namespace startup retry
- FireFly requires a signing key for "query" operations
- Broadcast are stuck whilst running performance testing against 1.3.0-rc1 HOT 2
- Need to document the FF1.3 revert error handling, and make the default safer HOT 12
- GET API(s) response doesn't include a REST endpoint in urls
- Fix `latest` tag in Docker image workflows
- Improve multi-party status information (if enabled) to aid registration automation HOT 3
- Build broken on main HOT 1
- Google search results for docs not going to /latest HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from firefly.