axelarnetwork / axelarate-community Goto Github PK

Describe the bug
Exercise 2 - Burn ERC20 wrapped Bitcoin tokens and obtain native Satoshi
When the burn transaction is detected, an error occurs stating that the transaction type is not supported. It is therefore not possible for the CLI to retrieve the 30 confirmations. This is an EIP-1559 transaction.

To Reproduce
Steps to reproduce the behavior:

1. Start with fresh system.
2. Complete prerequisite of exercise 2
3. Complete EX2 until Burn ERC20 wrapped Bitcoin tokens and obtain native Satoshi
4. Start Burn ERC20 wrapped Bitcoin tokens and obtain native Satoshi by typing in for example c2d2cli transfer satoshi tb1qn267dum8dvgudl4u7x7dpk6uts9k5pc5cv8d3u --source-chain ethereum --dest-chain bitcoin --gas=auto --gas-adjustment=1.4
5. Transfer satoshi to the provided ethereum address with Metamask
6. ERROR

Expected behavior
After the tx got recognized it should start analyzing the confirmations.

Screenshots/Log excerpts

bash-5.1# c2d2cli transfer satoshi tb1qn267dum8dvgudl4u7x7dpk6uts9k5pc5cv8d3u --source-chain ethereum --dest-chain bitcoin --gas=auto --gas-adjustment=1.4
| Using AxelarGateway at 0x2A1165BA03e10F2ce86c011f7771b9fA11f26274
| Using satoshi token at 0x817CB378496E5dA59b7D56ae0C91E89B19E821cE

info: (3/5) Confirming Ethereum tx 0x43e8db8bd314d8e919be6c7a304db4c647e9fbcdb450aa68c9334035034d8296 on Axelar

| ✗ 0 of 30 required confirmations on Ethereum
Error: failed to confirm withdrawal: failed to get transaction by hash: transaction type not supported
failed to confirm withdrawal: failed to get transaction by hash: transaction type not supported [/build/pkg/bridges/evm/provider/client.go:219]

Additional context
Burn tx: https://ropsten.etherscan.io/tx/0x43e8db8bd314d8e919be6c7a304db4c647e9fbcdb450aa68c9334035034d8296

Setup
Bitcoin Wallet: Electrum Testnet
ETH Wallet: Metamask
Hardware: Bare metal server (Hetzner AX101)
OS: Ubuntu-2004-focal-64-minimal

Describe the bug

I am not sure if this is correct behaviour or not so i will just describe what i did.

1. I minted 0.00087 tbtc and sent it all to ethereum
2. I then sent only 0.0004 tbtc back to bitcoin network here: https://ropsten.etherscan.io/tx/0x2d3cfba30d9b495b1d66bd147ff61725ab9437c747179040f91f5bed63498d78
3. After i confirmed, signed and submitted the tx via Blockcypher, i saw that it sent back the full 0.00087 tbtc somehow.. the tx for it is here: https://blockstream.info/testnet/tx/d19b75d38b626dacbb1fdba5ee29553d467ce2e1184a599e2da4c0b96e148f07

ETH addr: 0x2c2822031e9CAf1C031e71BB579fC796551F7BEE
BTC addr: tb1qwe68q8zwdlhky2eg54a9cnlh7pmypgaqqnwne7fthjhthxl6x6zsgddvsg

should add a warning on the flag and the script when it's used.
it's a dangerous flag to use as it erases all secret keys also.

i.e., when reset-chain flag is used, we pause, ask a question, are you sure you want to erase all previous blockchain state?, press Y to continue.

Describe the bug
See picture :

To Reproduce
Steps to reproduce the behavior:

1. Execute command axelarcli q bitcoin txInfo {blockhash} {txID}:{voutIdx}
2. use an incomplete {blockhash}

Expected behavior
a clearer and handled error message mentioning the blockhash wasn't correctly unformatted

Screenshots/Log excerpts
If applicable, add screenshots or excerpts from the node log to help explain your problem.

Additional context
In this case, the copy paste was missing characters but the error wasn't straight forward pointing to me which field had issues
in the test scenario, most likely worth to also try with an incomplete txID and a non numeric voutIdx

Need to write a proper doc for "common problems and their solutions". No time for that now so pasting rough notes here instead.

On node startup, node immediately shuts down with log:

panic: can't get node FCAA20B905AE004A69C12B3F17E88B8817FD73B10CB3586F1C87AF8B0E9FECC7: open /Users/gus/.axelar_testnet/.core/data/application.db/077294.ldb: too many open files

Fix: Mac OS has a very low limit on the concurrent open file handles. I increased it with ulimit -n 2048.

See cosmos/cosmos-sdk#1394

My testnet validator posts many log messages of the form:

2022-01-17T19:34:58Z INF backing off (retry in 6.950446643s ): rpc error: code = InvalidArgument desc = account sequence mismatch, expected 1623, got 1622: incorrect account sequence: invalid request module=vald

ie. sequence mismatch off by 1 for a couple hundred attempts now for hours. This behaviour persists even after restarting the node. (This puts the validator in missed-too-many-blocks status.)

More logs:

2022-01-17T20:04:58Z INF operator axelarvaloper1ckg9aus8tv9xayyc9pzz8yxvrpty9mmnc09yak sending heartbeat acknowledging keys: [avax-master-1 avax-master-genesis avax-secondary-1 avax-secondary-genesis eth-master-1 eth-master-genesis eth-secondary-1 eth-secondary-genesis ftm-master-1 ftm-master-genesis ftm-secondary-1 ftm-secondary-genesis moonbeam-master-1 moonbeam-master-genesis moonbeam-secondary-1 moonbeam-secondary-genesis poly-master-1 poly-master-genesis poly-secondary-1 poly-secondary-genesis] listener=tss module=vald
2022-01-17T20:04:58Z INF operator axelarvaloper1ckg9aus8tv9xayyc9pzz8yxvrpty9mmnc09yak sending heartbeat acknowledging keys: [avax-master-1 avax-master-genesis avax-secondary-1 avax-secondary-genesis eth-master-1 eth-master-genesis eth-secondary-1 eth-secondary-genesis ftm-master-1 ftm-master-genesis ftm-secondary-1 ftm-secondary-genesis moonbeam-master-1 moonbeam-master-genesis moonbeam-secondary-1 moonbeam-secondary-genesis poly-master-1 poly-master-genesis poly-secondary-1 poly-secondary-genesis] listener=tss module=vald
2022-01-17T20:05:08Z INF backing off (retry in 5s ): RPC error -32603 - Internal error: timed out waiting for tx to be included in a block module=vald
2022-01-17T20:05:08Z INF backing off (retry in 5s ): RPC error -32603 - Internal error: timed out waiting for tx to be included in a block module=vald
2022-01-17T20:05:16Z INF backing off (retry in 6.903285946s ): rpc error: code = InvalidArgument desc = account sequence mismatch, expected 1626, got 1625: incorrect account sequence: invalid request module=vald
2022-01-17T20:05:16Z INF backing off (retry in 6.903285946s ): rpc error: code = InvalidArgument desc = account sequence mismatch, expected 1626, got 1625: incorrect account sequence: invalid request module=vald

...

2022-01-17T20:08:16Z ERR handler goroutine: failure to broadcast outgoing heartbeat msg: aborting after 10 retries: rpc error: code = InvalidArgument desc = account sequence mismatch, expected 1626, got 1625: incorrect account sequence: invalid request module=vald
2022-01-17T20:08:16Z ERR handler goroutine: failure to broadcast outgoing heartbeat msg: aborting after 10 retries: rpc error: code = InvalidArgument desc = account sequence mismatch, expected 1626, got 1625: incorrect account sequence: invalid request module=vald

A while later I also see

2022-01-17T21:16:03Z ERR tx response with error: : tx already in mempool module=vald
2022-01-17T21:16:03Z ERR job panicked:runtime error: invalid memory address or nil pointer dereference module=vald
2022-01-17T21:16:03Z ERR tx response with error: : tx already in mempool module=vald
2022-01-17T21:16:03Z ERR job panicked:runtime error: invalid memory address or nil pointer dereference module=vald

@milapsheth suggests that perhaps there are two instances of vald running. My investigation supports this. A ps inside the vald container shows:

docker exec -it vald sh
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 {vald-entrypoint} /bin/sh /home/axelard/shared/vald-entrypoint.sh startValdProc
   30 root      0:00 {vald-entrypoint} /bin/sh /home/axelard/shared/vald-entrypoint.sh startValdProc
   33 root      0:20 axelard vald-start --tofnd-host tofnd --node http://axelar-core:26657 --validator-addr axelarvaloper1ckg9aus8tv9xayyc9pzz8yxvrpty9mmnc09yak
   44 root      0:00 sh
   51 root      0:00 ps

suggesting that startValdProc is being called twice.

Look at lines 341 and 344 of the following snippet:

344: run startValdProc
341: call vald-entrypoint.sh which does:

Steps to reproduce

As per setup instructions: Testnet Node Setup | Axelar Testnet Docs

1. In axelarate-community repo checkout release tag v0.6.0
2. Change the external_address field in the join/config.toml file and append the rpc port. Make the address malformed eg. by adding a space at the beginning:

external_address = " 123.123.123.123:26656"

3. Run join/join-testnet.sh

Actual behaviour

Script output indicates happy path yet it appears that the container doesn't even start:

• docker ps -aq produces no output
• there are no container logs:

$ docker logs -f axelar-core 2>&1
Error: No such container: axelar-core

Expected behavior

A suitable error message should be printed to the terminal eg. malformed external_address in config.toml. Then either:

1. exit and do nothing further, or
2. axelar-core should proceed as if external_address were the empty string. In this case a suitably clear message should be printed to the terminal eg. proceeding with external_address = ""

I also expect docker logs -f axelar-core 2>&1 to produce some output.

Describe the bug
axelar-core panic running join-testnet.sh on current testnet (toronto). Seems from the panic trace to be an unmarshalling error in the genesis file. Are we pulling the wrong genesis file?

To Reproduce

1. Remove the -d and -rm flags from the docker run command in join-testnet.sh so you can see the logs.
2. Start with a fully synced and running validator node. (Probably don't need it to be a validator or fully synced.)
3. Stop the node docker stop vald tofnd axelar-core
4. remove all data: mv ~/.axelar_testnet ~/.axelar_testnet_backup
5. Start the node again join-testnet.sh

Container panics on start-up. Snippet of output. (I have the whole output in a text file.)

2021-12-28T07:06:17Z INF port binded module=x/ibc/port port=transfer
2021-12-28T07:06:17Z INF claimed capability capability=1 module=transfer name=ports/transfer
panic: json: cannot unmarshal object into Go value of type string

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshalJSON(0xc0012f6530, 0xc000c20780, 0x1cc, 0x1e0, 0x27bbf80, 0xc0012d4d00)
	/go/pkg/mod/github.com/cosmos/[email protected]/codec/proto_codec.go:176 +0x98
github.com/axelarnetwork/axelar-core/x/axelarnet.AppModule.InitGenesis(0x27d6840, 0xc0012c58c0, 0x27b47f0, 0xc000f487b0, 0x27ef9c8, 0xc0012f6530, 0x27ef9c8, 0xc0012f6530, 0xc00120e060, 0x27b47f0, ...)
	/go/axelar/x/axelarnet/module.go:129 +0xa4
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(0xc001117dc0, 0x27d5848, 0xc000138010, 0x27f2ba0, 0xc0001fad00, 0x0, 0x0, 0xc0012f0030, 0x15, 0x0, ...)
	/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:307 +0x2b5
github.com/axelarnetwork/axelar-core/app.(*AxelarApp).InitChainer(0xc000f3b200, 0x27d5848, 0xc000138010, 0x27f2ba0, 0xc0001fad00, 0x0, 0x0, 0xc0012f0030, 0x15, 0x0, ...)
	/go/axelar/app/app.go:576 +0x170
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc0000ae1a0, 0x38e4cdd8, 0xed95c23f3, 0x0, 0xc0012f0030, 0x15, 0xc000142160, 0x3742dc0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:62 +0x315
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc0006a2600, 0x38e4cdd8, 0xed95c23f3, 0x0, 0xc0012f0030, 0x15, 0xc000142160, 0x3742dc0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x115
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0xc000f78ea0, 0x38e4cdd8, 0xed95c23f3, 0x0, 0xc0012f0030, 0x15, 0xc000142160, 0x3742dc0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x78
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc0027bda48, 0xb, 0x0, 0x2298548, 0x7, 0xc0012f0030, 0x15, 0x1, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:319 +0x655
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0027bda48, 0x27f44d8, 0xc0010fd930, 0x1f4, 0xc000e86480)
	/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:268 +0x458
github.com/tendermint/tendermint/node.doHandshake(0x27ef490, 0xc000f78260, 0xb, 0x0, 0x2298548, 0x7, 0xc0012f0030, 0x15, 0x1, 0x0, ...)
	/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:325 +0x1d8
github.com/tendermint/tendermint/node.NewNode(0xc000c3a8c0, 0x27cb610, 0xc0012dfb80, 0xc00051a010, 0x278ca40, 0xc0005203a8, 0xc00051a0f0, 0x25322e8, 0xc00051a270, 0x27d6840, ...)
	/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:733 +0x2105
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000194100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x27e0dd0, 0xc0011157b0, ...)
	/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:263 +0x685
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0011fe780, 0x3742dc0, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:129 +0x209
github.com/spf13/cobra.(*Command).execute(0xc0011fe780, 0x3742dc0, 0x0, 0x0, 0xc0011fe780, 0x3742dc0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc00110f900, 0x0, 0x0, 0xc00107bf70)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc00110f900, 0xc00107bf70, 0xd, 0x4000000000000000, 0x0)
	/go/pkg/mod/github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x285
main.main()
	/go/axelar/cmd/axelard/main.go:70 +0x155

Expected behavior

I expect join-testnet.sh to join the testnet and start syncing the chain from scratch as if it were a fresh new node.

Additional context
Why did I do this? I want to test validator recovery flows. My first attempt also included mnemonic recovery flags for join-testnet.sh and hit the same panic. I soon discovered that the panic occurs even without the recovery flags.

Same result even if you include --reset-chain. The only way I can get my node to run is if I restore the backup I made above

Describe the bug

Changes to ~/axelarate-community/join/config.toml are not propagated to the vald container. Thus, for example, validator nodes cannot connect to external chains.

To Reproduce

Current validator setup flow for external chain registration says:

• stop vald, tofnd containers
• Edit the file ~/axelarate-community/join/config.toml: edit the rpc_addr and start-with-bridge entries corresponding to the external chain you wish to connect. (eg. Ethereum)
• resume vald, tofnd containers with ./join/launch-validator-tools.sh

Attach to the vald container (eg docker exec -it vald sh). View the file ~/.axelar/config/config.toml. The changes you made above are not visible in this file.

Expected behavior

Changes to config.toml should be visible in the vald container. In the example of ethereum external chain registration vald logs should indicate a successful connection to the ethereum RPC. If no such logs appear then your validator node is not connected to your ethereum node. ☹️

On docker for linux, host-gateway is not mapped to host.docker.internal and has to be manually configured to allow access to geth and bitcoin nodes running on the host machine.

Run through the onboarding process on an ubuntu machine and make the appropriate fixes. Some suggested fixes are :

• Adding --http.addr "0.0.0.0" to my geth command
• Setting rpcbind=0.0.0.0 in bitcoin.conf
• Adding '--add-host host.docker.internal:192.168.16.1' to the docker run command in join-testnet.sh (I found this IP by running 'docker network inspect axelarate_default' - it is the gateway IP)
  add '--add-host host.docker.internal:192.168.16.1' to the docker run command for vald

Description

For commit de212f58403dc44c75a088dac2ce584df841ca34 I ran

$ ./scripts/setup-node.sh -n testnet

and followed the instructions. At the end I was told that I could run the node with the following command

/home/sseefried/.axelar_testnet/bin/axelard start [moniker] --home /home/sseefried/.axelar_testnet >> /home/sseefried/.axelar_testnet/logs/axelard.log 2>&1 &

After choosing a suitable [moniker] and trying this I got the following output in ~/.axelar_testnet/logs/axelard.log

2022-10-20T11:02:00+10:00 INF adding 16 seeds from seeds.toml
2022-10-20T11:02:00+10:00 INF starting node with ABCI Tendermint in-process
2022-10-20T11:02:00+10:00 INF Starting multiAppConn service impl=multiAppConn module=proxy
2022-10-20T11:02:00+10:00 INF Starting localClient service connection=query impl=localClient module=abci-client
2022-10-20T11:02:00+10:00 INF Starting localClient service connection=snapshot impl=localClient module=abci-client
2022-10-20T11:02:00+10:00 INF Starting localClient service connection=mempool impl=localClient module=abci-client
2022-10-20T11:02:00+10:00 INF Starting localClient service connection=consensus impl=localClient module=abci-client
2022-10-20T11:02:00+10:00 INF Starting EventBus service impl=EventBus module=events
2022-10-20T11:02:00+10:00 INF Starting PubSub service impl=PubSub module=pubsub
2022-10-20T11:02:00+10:00 INF Starting IndexerService service impl=IndexerService module=txindex
2022-10-20T11:02:00+10:00 INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=0.26.0-patch
2022-10-20T11:02:00+10:00 INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=1/11 module=x/crisis name=gov/module-account
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=2/11 module=x/crisis name=staking/module-accounts
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=3/11 module=x/crisis name=staking/nonnegative-power
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=4/11 module=x/crisis name=staking/positive-delegation
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=5/11 module=x/crisis name=staking/delegator-shares
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=6/11 module=x/crisis name=bank/nonnegative-outstanding
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=7/11 module=x/crisis name=bank/total-supply
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=8/11 module=x/crisis name=distribution/nonnegative-outstanding
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=9/11 module=x/crisis name=distribution/can-withdraw
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=10/11 module=x/crisis name=distribution/reference-count
2022-10-20T11:02:00+10:00 INF asserting crisis invariants inv=11/11 module=x/crisis name=distribution/module-account
2022-10-20T11:02:00+10:00 INF asserted all invariants duration=1.124706 height=690489 module=x/crisis
panic: unknown field "gateway_code" in types.Params

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshalJSON(0x0?, {0xc000d94000?, 0x17?, 0xa8939?}, {0x2e7aa40?, 0xc000126b70?})
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/codec/proto_codec.go:176 +0x45
github.com/axelarnetwork/axelar-core/x/evm.AppModule.InitGenesis({{}, {0x2e8f050, 0xc000130d80}, {{0x2e76a10, 0xc000fc99c0}, {0x2e9b4f8, 0xc0005cc890}, {0x2e788b0, 0xc000d15940}, 0xc000d0ce10}, ...}, ...)
	/home/ubuntu/axelar-core-2/x/evm/module.go:134 +0xa7
github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x2e8dfe8, 0xc000136000}, {0x2e9d000, 0xc0012e5000}, {{0x0, 0x0}, {0xc0000598a8, 0x17}, 0xa8939, ...}, ...}, ...)
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:320 +0x29d
github.com/axelarnetwork/axelar-core/app.(*AxelarApp).InitChainer(_, {{0x2e8dfe8, 0xc000136000}, {0x2e9d000, 0xc0012e5000}, {{0x0, 0x0}, {0xc0000598a8, 0x17}, 0xa8939, ...}, ...}, ...)
	/home/ubuntu/axelar-core-2/app/app.go:725 +0x20e
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000cabba0, {{0x218d5b00, 0xed994a1e0, 0x0}, {0xc0000598a8, 0x17}, 0xc001291ce0, {0xc00132bc20, 0x1, 0x1}, ...})
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:62 +0x455
github.com/tendermint/tendermint/abci/client.(*localClient).InitChainSync(0xc0011dbe60, {{0x218d5b00, 0xed994a1e0, 0x0}, {0xc0000598a8, 0x17}, 0xc001291ce0, {0xc00132bc20, 0x1, 0x1}, ...})
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:272 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).InitChainSync(0x0?, {{0x218d5b00, 0xed994a1e0, 0x0}, {0xc0000598a8, 0x17}, 0xc001291ce0, {0xc00132bc20, 0x1, 0x1}, ...})
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:77 +0x55
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0x1f89d0c, 0x7}}, {0xc0000598a8, 0x17}, 0xa8939, 0x0, {{0x0, ...}, ...}, ...}, ...)
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:319 +0xd78
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00100ddd8, {0x2e9dc10, 0xc0005cfad0})
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0x1f89d0c, 0x7}}, {0xc0000598a8, 0x17}, 0xa8939, 0x0, ...}, ...)
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:325 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00052a280, {0x2e8aa90, 0xc000ee9720}, 0xc001327400, {0x2e711c0, 0xc00132a6f0}, 0x0?, 0x0?, 0xc001327700, {0x2e8f050, ...}, ...)
	/home/ubuntu/go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:733 +0x577
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x2e92b00, 0xc001203bc0}, ...}, ...)
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:272 +0x7db
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0012ac500?, {0xc0005bb620?, 0x1?, 0x3?})
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:126 +0x169
github.com/spf13/cobra.(*Command).execute(0xc0012ac500, {0xc0005bb5c0, 0x3, 0x3})
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc001159400)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:911
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1f87fd6?, {0xc000de3d10, 0x17})
	/home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x1eb
main.main()
	/home/ubuntu/axelar-core-2/cmd/axelard/main.go:70 +0x4c5

The key line appears to be:

panic: unknown field "gateway_code" in types.Params

The version of axelard downloaded was axelard-v0.26.0-patch

Is there a chance that this version no longer expects the gateway_code attribute that appears in ~/.axelar_testnet/config/genesis.json?

Describe the bug
vald crashes on transaction broadcast because of insufficient fees

To Reproduce
Setup an Axelar validator node following the official instructions and using the default configuration
After launching vald as instructed it crashes with error:
2022-03-03T23:47:55+01:00 ERR tx response with error: insufficient fees; got: 37uaxl required: 36478uaxl: insufficient fee: insufficient fee module=vald process="retry pipeline"

The broadcaster account is funded as double checked on the explorer
https://axelar-testnet.coinhippo.io/account/axelar13y2svku7xhcqy5n0m0r8pxn3ncagacvrhuq2vs

Expected behavior
vald should run and has sufficient fees to broadcast

Additional context
I might have missed something in which case I apologize but looking for help here since nobody answered on Discord.
Tried changing the gas-adjustment in config.toml without any success.

• add a sentence description what is the "broadcaster" and why we need to register it. "Axelar network propagates messages from threshold multi-party computation protocols via the underlying consensus. Such messages are signed and delivered via the blockchain".
• I believe we need to faucet funds to the "validator" address also, not just broadcaster.
• worth adding a note that the validator needs to have >= 0.5% of the delegated stake to participate in the consensus. users can check what's active by looking at the monitoring dashboard pinned in the discord testnet channel.
• add a command to delegate more stake to a validator.
• add a command to check how much stake the validator has bonded.
• "monkier" -> should ask the testnet users to specify something descriptive for their nodes there.
• rename the exercise to "join" only (i.e. remove "leave" part out of the exercise name).

Describe the bug
When upgrading chains if the script is run once you're never able to override the broadcaster address

To Reproduce
Steps to reproduce the behavior:
Example

1. Execute ./scripts/validator-tools-host.sh once with no mnemonic flag and generate a new broadcaster address "by accident"
2. kill -9 $(pgrep tofnd) && kill -9 $(pgrep -f "axelard vald-start") to stop the services
3. Execute ./scripts/validator-tools-host.sh, but this time attempting to use the broadcaster mnemonic in a .txt file and the -p flag

Expected behavior
The broadcaster address SHOULD be started up with the address generate by the mnemonic text file.

Screenshots/Log excerpts
The printed message will NOT read out with the broadcaster address generated from the mnemonic. It will match whatever address was generated in the first execution of the script no matter what.

Additional context
This has happened on every testnet chain change for us. The mnemonic flags have not worked as intended.

My axelarate-community/c2d2/config.toml file reads:

[bitcoin] rpc_addr = "http://axelar:[email protected]:8332"

I updated the community package version to v0.3.1 and then removed the old config.toml file, but when I run

c2d2cli transfer satoshi 0xfEB31fe1edF6F53e37dd318c9467FdD783422ec0 --source-chain bitcoin --dest-chain ethereum --gas=auto --gas-adjustment=1.4

I get this error:

Error: failed to get bitcoin bridge client: unexpected error when waiting for bitcoin node warmup: Post "http://a6f628019c4194ded9623daaa54834e0-632184776.us-east-2.elb.amazonaws.com:8332": dial tcp: lookup a6f628019c4194ded9623daaa54834e0-632184776.us-east-2.elb.amazonaws.com on 192.168.65.5:53: no such host: connection to the bitcoin node failed failed to get bitcoin bridge client: unexpected error when waiting for bitcoin node warmup: Post "http://a6f628019c4194ded9623daaa54834e0-632184776.us-east-2.elb.amazonaws.com:8332": dial tcp: lookup a6f628019c4194ded9623daaa54834e0-632184776.us-east-2.elb.amazonaws.com on 192.168.65.5:53: no such host: connection to the bitcoin node failed [axelarnetwork/[email protected]/cmd/axelard/cmd/vald/btc/rpc/rpcClient.go:143]

On the daemon, I run:
join/joinTestnet.sh --tofnd v0.4.0 --axelar-core v0.5.1

and I get this error, but then it skips it and seems to mine blocks.

ERR Error dialing seed err="auth failure: conn.ID (0f8b5a12c59040ab9973d17bd8ddc21ce8b15e0b) dialed ID (6e09ca649a15daeaa97f14100cb95f062257bdd8) mismatch" module=p2p seed={"id":"6e09ca649a15daeaa97f14100cb95f062257bdd8","ip":"3.140.154.113","port":26656} 2021-07-22T22:48:40Z ERR Couldn't connect to any seeds module=p2p

https://github.com/axelarnetwork/axelar-core/blob/main/x/nexus/types/params.go#L31=

we have decoupled account management from the scripts and we want to allow people to migrate from the old scripts to the new ones. the migration would involve folder structure changes.

we want to improve documentation. take logic away from the scripts if there is potential for more and we should improve docs to explain exactly what the scripts are doing.

An ecosystem partner sent us the following:

During our initial tests, the docker container was failing to start for no apparent reason.
Upon troubleshooting, checking the contents of the file produced here:
https://github.com/axelarnetwork/axelarate-community/blob/main/configuration/consume-genesis.sh#L29
we saw the following:

$ head ~/.axelar_testnet/shared/validator.txt
password must be at least 8 characters
password must be at least 8 characters
EOF
Error: too many failed passphrase attempts

Obviously, the value of the env variable KEYRING_PASSWORD we were using was shorter than 8 characters.
Our production environment uses a strong password, but initially we were using simple small strings for testing, disregarding this limitation.
It would probably be beneficial for the scripts themselves to perform a validation check, or even mention this requirement in Github README.md.

Describe the bug

You can only run ./join/launch-validator-tools-with-binaries.sh --proxy-mnemonic broadcaster_mnemonic.txt --tofnd-mnemonic tofnd_mnemonic.txt exactly once after resetting the chain.

Killing and attempting to restart after that first time, results in errors and the companion processes will not start. Starting WITHOUT the flags results in the script creating a new broadcaster address and mnemonic (which is not what you're trying to do).

Expected behavior

Reasonable behaviour if mnemonics are supplied multiple times. Examples:

1. Ignore mnemonics and print a message explaining why
2. Ask user if he/she wishes to overwrite existing mnemonics and re-derive accounts. (This probably opens a can of worms.)

Workaround

Validators on discord painstakingly deduced the following workaround that recovers the desired state in the current axelar-core:

kill -9 $(pgrep -f "axelard vald-start") && kill -9 $(pgrep -f "tofnd")`
rm -rf .axelar_testnet/.tofnd/kvstore
rm .axelar_testnet/.vald/keyring-test/*

# This might not have been required
# [Gus: Seems like it is required.  Link below.]
# rm .axelar_testnet/broadcaster.bech

./join/launch-validator-tools-with-binaries.sh --proxy-mnemonic broadcaster_mnemonic.txt --tofnd-mnemonic tofnd_mnemonic.txt

Another discord validator confirms that rm .axelar_testnet/broadcaster.bech is in fact needed.

From #347 (comment)

it's kinda weird that the standard way to start the node always overwrites the config in place?
It feels like overwritting the config with default config should be only happening with a special parameter no?

I agree. We need to overhaul and simplify the join scripts.

old binaries use seeds.txt while new binaries use toml file for seeds. when people try to sync from scratch the script tries to use the new method but the old binaries dont support it. we want to create a method where depending on the verison of the binary the seeds.toml is transferred to the config.toml file.

#350

Describe the bug
axelarcli q account shows a wrong denom stake

see in full output

axelarcli keys show validator -a
axelar10vse7zu90ndfdw20vzelvxt8kjf7lf0uwt4cfu
/ # axelarcli q account axelar10vse7zu90ndfdw20vzelvxt8kjf7lf0uwt4cfu
{
  "type": "cosmos-sdk/Account",
  "value": {
    "address": "axelar10vse7zu90ndfdw20vzelvxt8kjf7lf0uwt4cfu",
    "coins": [
      {
        "denom": "stake",
        "amount": "1000000"
      }
    ],
    "public_key": {
      "type": "tendermint/PubKeySecp256k1",
      "value": "A8Vmg/1T+HY0IMcoNY5fBbrVlEupH+yLF4mvTxcT5fzV"
    },
    "account_number": "20",
    "sequence": "9"
  }
}

To Reproduce

Steps to reproduce the behavior:

1. Execute command axelarcli keys show validator -a axelar10vse7zu90ndfdw20vzelvxt8kjf7lf0uwt4cfu

Expected behavior
Expected to have something different than the denom stake maybe axl or whatever the team was planning to use

Screenshots/Log excerpts
If applicable, add screenshots or excerpts from the node log to help explain your problem.

Additional context
Add any other context about the problem here.

Describe the bug
**join-testnet-with-binaries.sh** script fails to recover using the priv_validator_key.json file specified using **_--tendermint-key_** when there is a chain reset or trying to restore after removing $HOME/.axelar_testnet

To Reproduce
Steps to reproduce the behavior:

1. rm -rf $HOME/.axelar_testnet
2. ./join-testnet-with-binaries.sh --axelar-core v0.9.6 --validator-mnemonic /brightlystake/valdacct --tendermint-key /brightlystake/privkey.json
3. No error returned by the script . It restores the validator account using the keys specified but fails to restore privkey.json
4. The script creates a new key in $HOME/.axelar_testnet/.core/config/priv_validator_key.json which is different from the contents of /brightlystake/privkey.json specified when starting the script.

Expected behavior
The tendermint key specified using the parameter --tendermint-key should be restored to $HOME/.axelar_testnet/.core/config/priv_validator_key.json`

Screenshots/Log excerpts
If applicable, add screenshots or excerpts from the node log to help explain your problem.

Cause of issue

if [ ! -f "$CONFIG_DIRECTORY/genesis.json" ]; then
  "$AXELARD" init "$NODE_MONIKER" --chain-id "$AXELARD_CHAIN_ID" --home "$CORE_DIRECTORY"
  if [ -f "$TENDERMINT_KEY_PATH" ]; then
    cp -f "$TENDERMINT_KEY_PATH" "$CONFIG_DIRECTORY/priv_validator_key.json"
  fi
fi

Since the genesis file is downloaded earlier in the script the priv_validator_key.json file is not restored from the specified file.

Possible solution

Restore the file earlier in the startup script or skip checking of non existence of genesis file.

@kalidax to add exact requirements and specs

Describe the bug
Running join-testnet.sh on main branch fails because the bitcoin and ethereum node RPC endpoints have changed, we replaced the axelar-run RPC endpoints with placeholders that are invalid RPC endpoints.

To Reproduce
checkout axelarate community main branch
run join/join-testnet.sh
fails due to error

Expected behavior
Using the placeholder RPC endpoint should not cause a regular axelar node to fail. For validator nodes, the expected behavior is uncertain due to changes to the bitcoin module support, and the ability to register as a chain maintainer for voting, in the next testnet release

A suitable, more descriptive error message should be logged

Additional context
see #210 for a similar issue, caused by invalid configuration in the config.toml

Describe the bug

$ ./scripts/setup-node.sh -n testnet
FAILED to run the last command

script cleanup running
FAILED to run the last command

script cleanup running

To Reproduce

Example

1. Clone this repo
2. Run $ ./scripts/setup-node.sh -n testnet
3. See the error above

Expected behavior
To set up config for testnet

Screenshots/Log excerpts
If I run this with verbose logging, I get a bit more info:

$ ./scripts/setup-node.sh -n testnet --verbose
+ shift
+ :
+ case "${1-}" in
+ break
+ args=("$@")
+ '[' testnet == mainnet ']'
+ '[' testnet == testnet ']'
+ '[' -z '' ']'
+ chain_id=axelar-testnet-lisbon-3
+ '[' -z '' ']'
+ root_directory=/Users/rootulp/.axelar_testnet
+ '[' -z '' ']'
++ curl -s https://raw.githubusercontent.com/axelarnetwork/axelar-docs/main/pages/resources/testnet.mdx
++ grep axelar-core
++ cut -d '`' -f 4
+++ failure
+++ trap - SIGINT SIGTERM ERR
+++ msg 'FAILED to run the last command'
+++ echo -e 'FAILED to run the last command'
FAILED to run the last command
+++ cleanup
+++ trap - EXIT
+++ msg
+++ echo -e ''

+++ msg 'script cleanup running'
+++ echo -e 'script cleanup running'
script cleanup running
+++ exit 1
+ axelar_core_version=
++ failure
++ trap - SIGINT SIGTERM ERR
++ msg 'FAILED to run the last command'
++ echo -e 'FAILED to run the last command'
FAILED to run the last command
++ cleanup
++ trap - EXIT
++ msg
++ echo -e ''

++ msg 'script cleanup running'
++ echo -e 'script cleanup running'
script cleanup running
++ exit 1

Notice the website it is trying to request isn't found

$ curl -s https://raw.githubusercontent.com/axelarnetwork/axelar-docs/main/pages/resources/testnet.mdx
404: Not Found%

We currently depend on users to clean up their mnemonics after backing them up. However, it is possible that users can forget their data on disk. We need to enforce correct mnemonic backup.

explain what these need to be; give examples of what would work.

gather general information about system/os/filestructure/configuration etc.

I'm trying to run an Axelar node on a non standard port

config.toml under both /home/axelar/.axelar_testnet/config and /home/axelar/.axelar/config has the following section:

[p2p]
# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:26556"

(26556 instead of 26656 that is already used for another project)

But when I launch the node:
./scripts/node.sh -e host -a v0.13.6 -n testnet -d /home/axelar/.axelar_testnet
I get:

2022-02-24T23:44:23+01:00 INF Starting Node service impl=Node
2022-02-24T23:44:23+01:00 INF Starting pprof server laddr=localhost:6060
2022-02-24T23:44:23+01:00 ERR pprof server error err="listen tcp 127.0.0.1:6060: bind: address already in use"
Error: listen tcp 0.0.0.0:26656: bind: address already in use
2022-02-24T23:44:23+01:00 INF Starting RPC HTTP server on [::]:26657 module=rpc-server

Describe the bug
tofnd fails to restart using launch-validator-tools-with-binaries.sh

To Reproduce
Steps to reproduce the behavior:
Example

1. execute launch-validator-tools-with-binaries.sh
2. kill vald and tofnd
3. execute launch-validator-tools-with-binaries.sh again

Expected behavior
tofnd should be able to restart

Additional context
tofnd attempts to be started using the -m create mnemonic command while there is an existing mnemonic.

2021-07-22T14:33:44Z INF New websocket connection module=rpc-server protocol=websocket remote=127.0.0.1:40456
2021-07-22T14:33:44Z INF Starting wsConnection service impl=wsConnection module=rpc-server protocol=websocket remote={"IP":"127.0.0.1","Port":40456,"Zone":""}
2021-07-22T14:33:44Z INF Connected tendermint client at http://localhost:26657 module=vald
2021-07-22T14:33:44Z INF Start listening to events module=vald
2021-07-22T14:33:44Z INF initiate connection to tofnd gRPC server: host.docker.internal:50051 module=vald
panic: failed to create tss manager: context deadline exceeded

goroutine 1 [running]:
github.com/axelarnetwork/axelar-core/cmd/axelard/cmd/vald.createTSSMgr(0x22287c0, 0xc0012b2d80, 0xc001175ef0, 0x14, 0x18, 0xc000c6e9b8, 0xc000f1a7d0, 0x47, 0x0, 0x0, ...)
/go/axelar/cmd/axelard/cmd/vald/start.go:263 +0x186
github.com/axelarnetwork/axelar-core/cmd/axelard/cmd/vald.listen(0xc001175ef0, 0x14, 0x18, 0x2287aa8, 0xc000f36450, 0x1dc7cec, 0x6, 0x226def0, 0xc000de2bc0, 0x2270cf8, ...)
/go/axelar/cmd/axelard/cmd/vald/start.go:176 +0x48a
github.com/axelarnetwork/axelar-core/cmd/axelard/cmd/vald.GetValdCommand.func1(0xc000ee2780, 0xc000bf6280, 0x0, 0x4, 0x0, 0x0)
/go/axelar/cmd/axelard/cmd/vald/start.go:117 +0xa95
github.com/spf13/cobra.(*Command).execute(0xc000ee2780, 0xc000bf6240, 0x4, 0x4, 0xc000ee2780, 0xc000bf6240)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc000e1a280, 0x0, 0x0, 0xc000d47680)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:890
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000e1a280, 0xc000d47680, 0xd, 0x4000000000000000, 0x0)
/go/pkg/mod/github.com/cosmos/[email protected]/server/cmd/execute.go:36 +0x265
main.main()
/go/axelar/cmd/axelard/main.go:70 +0x14f
2021-07-22T14:33:59Z ERR Failed to read request err="websocket: close 1006 (abnormal closure): unexpected EOF" module=rpc-server protocol=websocket remote={"IP":"127.0.0.1","Port":40456,"Zone":""}
2021-07-22T14:33:59Z INF Stopping wsConnection service impl={"Logger":{}} module=rpc-server protocol=websocket remote={"IP":"127.0.0.1","Port":40456,"Zone":""}
2021-07-22T14:33:59Z ERR error while stopping connection error="already stopped" module=rpc-server protocol=websocket

Describe the bug
Stop a healthy node and restart via node.sh and you see

docker: Error response from daemon: Conflict. The container name "/axelar-core" is already in use by container "f28510a0bfafd5d8ddae39d4205412b98dbb4befa23c8077fb921e14290a57ed". You have to remove (or rename) that container to be able to reuse that name.

To Reproduce
Steps to reproduce the behavior:
Example

1. Start with a healthy node.
2. docker stop axelar-core
3. Run ./scripts/node.sh with appropriate flags

Expected behavior
Node should restart.

Additional context
I had to do docker rm axelar-core before step 3 to get node.sh to successfully start the node.

Work on discord feedback as gathered on June 7, 2022.

Describe the bug
A clear and concise description of what the bug is.

1. clone "git clone https://github.com/axelarnetwork/axelarate-community.git"
2. cd axelarate-community
3. ./scripts/setup-node.sh -n testnet
   I got the following error message

"
FAILED to run the last command
script cleanup running
FAILED to run the last command
script cleanup running
"

To Reproduce
Steps to reproduce the behavior:
Example

1. Execute script ...
2. Execute command axelarcli...
3. See returned error ...

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots/Log excerpts
If applicable, add screenshots or excerpts from the node log to help explain your problem.

Additional context
Add any other context about the problem here.