Since we have a new version of bridge contracts (omni/tokenbridge-contracts@d53ae35) which complitely different from the original parity-bridge contracts, it is necessary to change deployment code in the bridge. But in order to be more undependent in maintanance of the deployment code, it is being suggested to implement it in form of a js/python/... script (the langugage is discussible).

The script must:

1. read the configuration toml file
2. deploy contracts with set of parameters given from configuration file
3. write the database file

Currently parity-bridge assumes that full nodes of networks are configured on the same system where the bridge is run. Therefore it communicates to the nodes by using IPC. This cause several limitations:

• Requirements to the network bandwidth and the system performance/capacity should be quite high: the system must have enough space to keep two blockchains and must verify all new appeared blocks. So, it means that maintaining one bridge instance withing the cloud could be quite expensive solution.
• The availability of the bridge software depends on the reliability of the particular Ethereum network client software. If an Ethereum client is not operaing (e.g. it crashes accidentally), the bridge software is not operating as well.

The way to avoid these limitations is to introduce ability for the bridge to communicate to full nodes through RPC. It will increase responsiveness of the bridge slightly which should not cause big issues due to the current nature of transactions confirmation mechanism used in the blockchain (there is no strict guarantee of a transaction inclusion into the next block).

Changes required:

1. Change bridge/src/config.rs to
   • support the configuration parameters rpc_host and rpc_port;
   • introduce the parameter keystore to keep the path to the directory where files with private keys are stored, this parameter is not subordinated to any section of the configuration file since it is common for entire bridge;
   • extend sections [home] and [foregin] with parameter password which contains path to the file with password to decrypt the corrsponding key file to extract the private key of the account responsible for signing transactions related this side of the bridge.
2. Change bridge/src/app.rs to provide the ability to connect through RPC. The following logic of connection channel selection should be implemented: if the parameter ipc is specified the IPC connection must be setup even if rpc_host is specified. If rpc_host specifed but there is no configuration for the parameter rpc_port the port 8545 is used by default.
3. Extend the functionality of sending transactions to sign them by using private key of the corresponding account and sending them through sendRawTransaction functionality. Encrypted key for the bridge authority account must stored in a separate file. This approach requires to use network id of the corresponding blockchain and NOnce (number of operation) of the corresponding account.
4. Introduce new parameters home_network_id, home_account_nonce, foreign_network_id and home_account_nonce in the database in order to get rid of requests to the IPC/RPC nodes to track network id and NOnce required to sign transactions. A new paramter authority_keystore must keep a path to the keystore file with the private key of the bridge authority. This parameter should not belong any node in in the configuration file.

Steps to reproduce:

1. Change the configuration file to reduce gas for deposit_relay to 25400.
2. Run the bridge with RUST_LOG=debug
3. Send ether to the HomeBridge contract

When the bridge is trying to send deposit confirmation to the ForeignBridge contract it reports that transaction was sent and deposit completed even it was not handled properly (in my case it was not included in a new block at all):

INFO:bridge::bridge::deposit_relay: got 1 new deposits to relay
DEBUG:<unknown>: [29] Calling: {"jsonrpc":"2.0","method":"eth_sendTransaction","params":[{"data":"0x26b3293f00000000000000000000000037a30534da3d53aa1867adde26e114a3161b2b1200000000
0000000000000000000000000000000000000000000a8503892e10001b254563acfa39c680314bf887d84561a8dddb3bb832664d0adaa16a2142a896","from":"0xf3ee321df87781864f46f6464e764c2827fca73b","gas":
"0x6338","gasPrice":"0x430e23400","to":"0x6500d471c8973d95493b417c44ab85f31265a2b6"}],"id":29}
INFO:bridge::bridge::deposit_relay: relaying 1 deposits
...
INFO:bridge::bridge::deposit_relay: deposit relay completed

Moreover the counter of checked_deposit_relay is increased in the database file. So, the bridge is in incorrect state after sending of such transaction.

Expected behavior

1. The bridge must track the transactions it is sending and reports if something was wrong with them.
2. If some relay or confirm operation cannot be completed due to failed transactions the corresponding counter in the database file must not be incremented.

Steps to reproduce:

1. Change the configuration file to reduce gas for deposit_relay to 20000.
2. Run the bridge with RUST_LOG=debug
3. Send ether to the HomeBridge contract

When the bridge is trying to send deposit confirmation to the ForeignBridge contract it crashes and the following appears in the console:

Error(Rpc(Error { code: ServerError(-32010), message: "Transaction gas is too low. There is not enough gas to cover minimal cost of the transaction (minimal: 25368, got: 12000). Try increasing supplied gas.", data: None }), State { next_error: None, backtrace: None })

Expected behavior

The bridge produces an error message to console (which is observable even with RUST_LOG=error) and continues working.

According to our initial vision, we need to implement daily limits in case if all node bridges are compromised and it won't trigger a massive attack.

https://medium.com/oracles-network/cross-chain-bridges-paving-the-way-to-internet-of-blockchains-422ac94bc2e5

1. The new name of the repository must be 'poa-bridge'
2. parity-stream upstream must be removed

INFO:bridge: Parsing cli arguments
INFO:bridge: Loading config
Cannot parse config

Caused by:
unknown field ipc, expected one of account, contract, request_timeout, poll_interval, required_confirmations, rpc_host, rpc_port, password for key home

Anyone else getting this? using file as shown on readme:

estimated_gas_cost_of_withdraw = 0
[home]
#account which you specified as validator on Home network
account = "0xETH_ACCOUNT_VALIDATOR_SOKOL"
#Full IPC path to your HOME node
ipc = "/Users/USERNAME/sokol-kovan-bridge/sokol-node/sokol-datadir/jsonrpc.ipc"

How many block confirmation to wait to send a tx

required_confirmations = 0

[home.contract]

bin = "HomeBridge_bytecode.bin"

[foreign]
account = "0xETH_ACCOUNT_VALIDATOR_KOVAN"
ipc = "/Users/USERNAME/sokol-kovan-bridge/kovan-node/kovan-datadir/jsonrpc.ipc"
required_confirmations = 0

[foreign.contract]
bin = "ForeignBridge_bytecode.bin"

[authorities]
accounts = [
"0xETH_ACCOUNT_VALIDATOR_KOVAN"
]
required_signatures = 1

[transactions]
home_deploy = { gas = 3000000, gas_price = 1000000000 }
foreign_deploy = { gas = 3000000, gas_price = 1000000000 }
deposit_relay = { gas = 3000000, gas_price = 1000000000 }
withdraw_relay = { gas = 3000000, gas_price = 1000000000 }
withdraw_confirm = { gas = 3000000, gas_price = 1000000000 }

Background

The basis of the bridge operations is events EVM sends as part of contract execution. Every bridge instance is looking for new events and as soon as an event appears after transaction applying from a new block, the bridge performs an action.

For example, when an account sends ether to the brigde contract in the Home side of the bridge (the left side), Deposit events is produced. The bridge instance catches this event and creates transaction on the Foreign (right) side to invoke deposit() method of the bridge contract. This transaction is considered as a bridge signature under the fact that some user transfers asset to the Foreign network.

In order to achieve availability and reliability, there are more than one bridge instances (N). During these bridges configuration it is settled that just subset (M, M <= N) of them is enough to transit the state between two networks. So, if M bridges confirmed the same state transition operation (deposit), this
state committed on the opposite side of the bridge:

This is a basic scenario which is enough to transite state between two EVM-based networks. The maximum (for the third bridge) gas consumption of deposit operation is about 90000.

The use case considered by parity-bridge (https://github.com/paritytech/parity-bridge, commit ceaf22f) is to connect Ethereum Foundation (Home side) and a private PoA networks (Foreign side). So, it is assumed that we need to keep as cheap as possible transactions on the Home side whereas consumption of the gas on the Foreign side does not make much sense. This is due to the reason that most probably bridge instances are managed/supported by owners of the PoA network so almost infinite resources could be spent to maintain transactions from bridges within the PoA network.

That is why the original development of parity-bridge is focused on the reduce number of transactions and gas consumption for operations made on the Home side of the brige. In order to transit the state from the Home to the Foreign network three bridges will send three transactions. And only one bridge is being chosen to perform state transition from the Foreign side to Home.

It becomes posisble owing to the architecture when confirmations of bridges to transfer the state are collected on the Foreign side (remember that it costs nothing) and as soon as needed number of confirmation gathered the bridge which sent the latest confirmation is responsible for forwarding all signatures (which are part of the confirmations) to the Home side.

The procedure to confirm the state transition (submitSignature) is the most expensive one. It consumes about 270000 gas (if it was sent to provide a final signature) since requires lots of mathematical operations and use the contract storage extensively.

Here is the list gas consumption for all operations performed by bridge:

Handling deposit() on the Foreign side (deposit_relay) = 90861, every bridge instance
Handling submitSignature() on the Foreign side (withdraw_confirm) = 265380, every bridge instance
Handling withdraw() on the Home side (withdraw_relay) = 72444, one bridge instance

Relatively huge gas consumption is the only the reason why patity-bridge is not recommended to be used with Ethereum Foundation network in the right side: the meaningless from the state transaction confirmation operation will cost ~5 USD (exchange rate 1000 USD per 1 ether). But cases when a PoA network is on the Home side and the Ethereum Foundation network is on the Foreign side make sense since they allow to split transactions traffic between two networks which could have positive effect on network performance and transactions fees.

Proposal for costs optimization

From the analysis above it is evident that most costly transactions happens on the right side of the bridge. So, if Ethereum Foundation network is there so it will not be cost effective to send several transaction to confirm the deposit and several transaction to confirm the withdraw.

It means that all operations for confirmation should happens on the left side of the bridge and it equals of mirroring handling of confirmation for the current implementation:

1. As soon as a user deposits coins, validators confirmations must be gathered on the left side of the bridge. It will be called deposit_confirm.
2. When last confirmation received, one validator (one which sent the final confirmation) is responsible for forwarding the confirmation to the right side: deposit_relay.
3. If a user sends request to exchange tokens to coins, the Withdraw event is handled and every bridge instance sends the confirmation directly to the left side of the bridge. It is withdraw_relay.

Changes required:

1. Optimization of the ForeignBridge contract:
   • remove submitSignature(), signature() and message();
   • remove the corresponding maps keeping information about signatures;
   • rework deposit() to work with list of signatures similarly to withdraw() of the original HomeBridge contract;
   • consider to have the code to cover costs of the bridge (isMessageValueSufficientToCoverRelay(), getWithdrawRelayCost() and estimatedGasCostOfWithdraw)
2. Modification of the HomeBridge contract:
   • rework withdraw() to accept signatures directly from bridge validators similar to deposit() method on the original ForeignBridge contract.
   • there is no need any more to cover costs of the bridge on the Home side so isMessageValueSufficientToCoverRelay(), getWithdrawRelayCost() and estimatedGasCostOfWithdraw must be removed.
   • introduce submitSignature(), signature() and message() and corresponding maps to keep information about signatures similarly to the original ForeignBridge contract.
3. Rework bridge/src/bridge/withdraw_relay.rs to send withdraw() to the HomeBridge contract as soon as Withdraw event appeared on the Foreign side. Every bridge instance should invoke this method simiarly deposit() of the original HomeBridge contract.
4. Rework bridge/src/bridge/deposit_relay.rs to wait for CollectedSignatures and forward deposit with the list of signatures to the ForeignBridge contract.
5. Rename bridge/src/bridge/withdraw_confirm.rs to bridge/src/bridge/deposit_confirm.rs and modify it as so Deposit event is being waited from Home side and submitSignature() is invoked by every bridge instance. The corresponding changes must be made in bridge/src/bridge/mod.rs in order to run the confirmation module.
6. Modify bridge/src/database.rs and bridge/src/bridge/mod.rs to not track checked_withdraw_confirm in the database and update checked_deposit_confirm instead.
7. Modify bridge/src/config.rs to
   • get rid of options transaction.withdraw_confirm.gas, transaction.withdraw_confirm.gas_price.
   • introduce options transaction.deposit_confirm.gas, transaction.withdraw_deposit.gas_price.

NFO:bridge:   using RPC connection
WARN:ethstore::accounts_dir::disk: Invalid key file: "keys/authority.json" (ErrorImpl { code: Message("Invalid hash"), line: 1, column: 111 })
WARN:ethstore::accounts_dir::disk: Invalid key file: "keys/authority.json" (ErrorImpl { code: Message("Invalid hash"), line: 1, column: 111 })
INFO:bridge: Acquiring home & foreign chain ids
INFO:bridge: Home chain ID: 77 Foreign chain ID: 42
INFO:bridge: Deploying contracts (if needed)
INFO:bridge: Loaded database
INFO:bridge: Starting listening to events
INFO:bridge::bridge: Retrieved home contract balance
INFO:bridge::bridge: Retrieved foreign contract balance

bridge should be stopped if keystore is missing/invalid

The Travis CI build was introduced recently for the repository but it have never succeeded due to dependency error:

running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-m64" "-I" "etc/hidapi/hidapi" "-Wall" "-Wextra" "-o" "/home/travis/build/poanetwork/poa-bridge/target/debug/build/hidapi-9b646fffdb35f685/out/etc/hidapi/linux/hid.o" "-c" "etc/hidapi/linux/hid.c"
cargo:warning=etc/hidapi/linux/hid.c:44:21: fatal error: libudev.h: No such file or directory
cargo:warning= #include <libudev.h>
cargo:warning=                     ^
cargo:warning=compilation terminated.
exit code: 1

Currently the block number which includes deployment transaction stored in the database file. This number could be used in recovery scenarios to try reapplying all skipped transactions: go through all the blocks since the block the contract was created and filter deploy/withdraw events.
It is also useful to deploy several new bridge instances: every bridge will start filtering events starting from the same block.

In order to simplify deployment of new bridges it could be worth to store this information in the bridge contract instead of storing it in database file: bridge deployment process could get this information from the contracts in order to generate database file with correct checked_deposit_relay, checked_withdraw_relay and checked_withdraw_confirm. So, it will be no necessity to distribute the database file among the target systems.

Proposals for changes

Solidity

1. Introduce deployedAtBlock public field in HomeBridge and ForeignBridge contracts.
2. This field initialized within the contract creation process with the block number.

Rust

1. Change the logic of generating database file as so if there are no checked_deposit_relay, checked_withdraw_relay and checked_withdraw_confirm defined in the file, deployedAtBlock is read from the corresponding contract and the parameters are initialized in the values received from the contract.
2. Remove the code which write home_deploy and foreign_deploy to the database file.

We have to think about who can upgrade contracts and who can deploy them
Those authorities should not be validators.

Currently every bridge instance sends deposit() to the Foreign side to confirm transfer of funds. But only transaction which allow to reach requiredSignatures limit generates an event.

Proposal for changes

In order to simplify issues investigation process it makes sense to generate events on every unique and valid depost() transactions.

The suggestion is to raise SignedForDeposit(address indexed, bytes32) with the bridge address and id of the transaction originating deposit request on the Home (left) side:

• the brigde address will allow to filter transactions from the specific bridge;
• the tx id will allow to match the transaction on the Home side and bridge resposes on the Foreign side.

Prior to merge of these changes it is necessary to understand how gas consumption is being increased and get agreement that it is acceptable.

As per the bridge deployment process it is necessary to create configuration file for every bridge instance. There is a section in the configuration file which allows to define gas consumption limits which will be used in the bridge transactions:

deposit_relay
withdraw_confirm
withdraw_relay

Since these limits are common for every bridge instance it is redundant to distribute this info among all nodes before deployment begins.
Moreover these limits needs to be updated in case of gas consumption changes if the bridge contract is upgraded as per #7. In order to reduce human factor it is worth to update this automatically.
Some big constant value for these parameters is not good option to have - it could cause delays in validation since limits will be too big for block validators especially in the Ethereum Foundation network.

Proposal for changes

Solidity

1. Introduce gasLimitWithdrawRelay in the contract HomeBridge.
2. Introduce gasLimitDepositRelay and gasLimitWithdrawConfirm field in the contract ForeignBridge.
3. Setup this parameter as part of bridge contracts deployment process.
4. Introduce ability to change these parameters as part of bridge contract upgrade process as soon as #7 is implemented. Events must be raised in case of gas limit changes.

Rust

1. Change bridge/src/bridge/deploy.rs to setup gasLimitWithdrawRelay, gasLimitDepositRelay and gasLimitWithdrawConfirm as part of deployment process.
2. Introduce new functionality in bridge/src/bridge/deposit_relay.rs, bridge/src/bridge/withdraw_confirm.rs and bridge/src/bridge/withdraw_relay.rs in order to pickup gas limits from the corresponding contract. In order to optimize number of requests made to the blockchain nodes the limits could be re-read as soon as the corresponding event received from the bridge contracts.

Problem:

• our temp changes are in master branch and it's hard to sync with parity
  Solution:
• move commits from master to your branch
• sync with upstream

As per recommendation from a team provided security audit for POA bridge it is needed to force https connection for RPC communications.

In other words RPC connection must not succeed if HTTP is used and the bridge instance must stop.

For testing purposes a new parameter like force_https could be introduced in the configuration file. It's value should be yes by default. If it is necessary to use HTTP connection instead of HTTPS the parameter needs to be set to no.

The current behavior for logs displayed during the bridge initialization is not consistent - home url is reported whereas foreign url is not.

INFO:bridge: Parsing cli arguments
INFO:bridge: Loading config
INFO:bridge: Starting event loop
INFO:bridge: Home rpc host https://sokol.poa.network
INFO:bridge: Establishing connection:
INFO:bridge:   using RPC connection
INFO:bridge: Acquiring home & foreign chain ids
INFO:bridge: Home chain ID: 77 Foreign chain ID: 3
INFO:bridge: Reading the database
INFO:bridge: Loaded database
INFO:bridge: Starting listening to events
INFO:bridge::bridge: Retrieved home contract balance
INFO:bridge::bridge: Retrieved foreign contract balance

Rust side needs to call isValidator before making any transactions to make sure that account that it's running is valid.
It needs to call BridgeValidators's contract address which should exists in both sides(Home and Foreign)

Currently it is assumed that in order to achieve more security at least 3 bridge instances need to be deployed on different independent systems.

The deployment process of several bridges now is the following:

1. Configure and run the first instance of the bridge.
2. Distribute the database toml-file created by the first bridge to other nodes. This file contains addresses of bridge contracts deployed by the first instance, so other instances needs to work with these contracts as well.
3. As soon as the database file is the same on every nodes, they could be run.

Proposal for changes:

Simplfy deployment process with creation of the bridge contracts with the same address in Home and Foreign networks. It could be achived by using the same address and the same NOnce for the contracts deployment in both networks. Since in most cases it requires to create new account with new keystore/private key, so, there is a neccessity to manage this keystore/private key. That's why another approach could be to use deployment transactions with hardcoded signature as it is implemented here: https://github.com/jbaylina/eip820/blob/master/js/deployment.js#L7-L25 (ability to do this in different networks confirmed in https://gist.github.com/akolotov/7b27902d196da15fde63fd4a484cb407).

It will allow:

• generate the database toml-file easily as part of automatic deployment by passing one address to the deployment scripts.
• if the bridge contracts are extended with ability to return the block number which deployment transaction is included in, deployment script will extract this information from the contracts and use it for the database toml-file generation.

Automatic generation of the database file and changes in the contracts will be covered in separate issues.

Changes required:

1. Modify cli/src/main.rs to support command deploy to send contracts deployment transactions to the networks and exit. The database toml-file must be generated after the command execution.
2. Modify bridge/src/bridge/deploy.rs to deploy contracts with the same address.

ZepplinOs and Aragon presented new way to write smart contract so you can separate the storage and implementation.
https://github.com/zeppelinos/labs
I personally like eternal storage with registry of contracts

As an example, ICO wizard makes an API call to https://gasprice.poa.network/, get the fastest or lowest(@igorbarinov )
IF service is not available, use hardcoded value.
I'd recommend using standard value from https://gasprice.poa.network/

Should be done on Rust side

https://gasprice.poa.network/

@akolotov please approve

There are more situations that can be distinguished -- for example, nonce re-use. This particular error will be conflated with insufficient funds because they share the error code in the JSON-RPC respponse.

Proposed solution: discriminate JSON-RPC responses with 32010 code according to their message.

The validators information is completely configured through bridge contracts and does not depend on authorities.required_signatures parameter of the bridge configuration.
The number of validators also could be changed during run-time and therefore authorities.required_signatures parameter will not reflect actual number of signatures required for a transaction validation.

The number of required signatures is required during withdraw_relay process to fetch signatures from foreign contract:
https://github.com/poanetwork/poa-bridge/blob/009d40aa504e7a48ddae6e965863b0338a8ebb0e/bridge/src/bridge/withdraw_relay.rs#L130-L134
Since the value of this parameter is picked up from configuration file it could be different from the actual number of signatures and it could cause:

1. the bridge cannot fetch enough number of signatures if the the value stored in the contract is greater than the value from the bridge config, so it will not be able to relay withdraws.
2. the bridge will fail on fetching signatures in the line
   https://github.com/poanetwork/poa-bridge/blob/009d40aa504e7a48ddae6e965863b0338a8ebb0e/bridge/src/bridge/withdraw_relay.rs#L51
   if the value stored in the contract is less than the value from the bridge configuration file.

The suggested changes are:

1. To get the address of Bridge Validators contracts from both home and foreign side.
2. To invoke requiredSignatures() on the foreign side for the same block which contains CollectedSignatures (an option could be to listen the event which will be introduced under omni/tokenbridge-contracts#27 as so update the value in the bridge instance on demand instead of polling)
3. To relay withdraw with the number of required signatures got in the step 2.

The situation when numbers of required signatures are different on both sides of bridges will be handled by monitoring. In order to make sure that number of the same the procedure to add/remove validators should make sure that no withdraw transactions are being sent at the moment of changing number of required signatures.

1. Will it be a pre-minted token (hardcoded totalSupply) or Mintable token?
2. If the token is Mintable then possible two options:
   • the bridge contract mints value of tokens every time it receives deposit() and
     burns value tokens every time it receives a request to transfer tokens to the
     PoA network (exchange to coins). This functionality is used now by the original
     Parity bridge.
   • the needed number of tokens is minted just after the contract deployment (the same
     applicable to the scenario when the amount of tokens is hardcoded by totalSupply)
     and ownership of all minted tokens is passed to the bridge contract. So, when
     coins are deposited, tokens owned by the ForeignBridge contract transfered to
     the user, and if user would like to withdraw coins, he/she transfer tokens to the
     bridge contract. If such apporach is chosen, the value of minted tokens must be
     set in the HomeBridge contract as so
     it could control that amount of all deposits is not greater than the amount of
     all tokens. If this mechanism is not introduced the situation with user's funds
     locked on the HomeBridge contract could happen since the ForeignBridge contract
     will try to transfer more tokens than it owns and will fail.
3. What the interface for the user to request exchange tokens to coins (withdraw) is?
   • it seems to be logical to use ERC677
     which describes usage of transferAndCall() method. If a user invokes
     transferAndCall of the token contract, it will call tokenFallback() of the
     ForeignBridge contract to complete transfer of tokens and proceed with withdraw.
     Disadvantages of this approach are that the user needs to know the address of
     the bridge and also the standard requires to specify 'data' in the method call
     which is unnecessary in our case.
   • another way is to define our own method in the token contract (e.g. the original
     bridge specifies transferHomeViaRelay) and inform all users for ABI and the
     correct procedure to withdraw funds. Advantages are the following: no need
     to specify the bridge address (since the token will know it) and 'data', only
     value to withdraw is required.
   • The next option is to use ERC223 token but it's support by exchanges is not clear.

Currently, a static JSON schema is expected to be returned from the gas-price Oracle:

{
    "block_number": u64,
    "block_time": f64,
    "instant": f64,
    "standard": f64,
    "slow": f64,
    "fast": f64,
    "health": bool
}

We should allow users the ability to configure their bridge such that they can query a gas-price Oracle which returns a JSON schema differing from the above schema. This should be implemented using JMESPath expressions.

A new config option home/foreign.gas_price_oracle_expr should be added to account for the User's JMESPath expression.

Something similar to the following psuedo-code should be added to GasPriceStream's impl Stream.

let json_str = /* Use hyper to GET the Oracle's data, then string-ify the response body */
let resp_json = jmespath::Variable::from_json(json_str).map_err(|_| ... )?;
let gas_price = config.<home/foreign>.gas_price_expr
        .search(resp_json).unwrap_or(config.<home/foreign>.default_gas_price);

Currently the bridge could send the same transactions several times if the database is not updated properly.

The following changes could be introduced:

1. To handle Deposit() event and avoid invocation of deposit() several times for the same origin transaction:

For Foreign chain it is necessary to check that

depositsSigned(sha3(ValidatorAddress, sha3(HomeDepositSender, Value, HomeTransactionHash)))

returns false and invoke deposit() in this case only.

2. To handle Withdraw() event on the Foreign side and avoid invocation of submitSignature() several times for the same 'message'

For Foreign chain it is necessary to check that

messagesSigned(sha3(ValidatorAddress, sha3(message)))

returns false and invoke submitSignature() in this case only.

3. To handle CollectedSignatures() event on the Foreign side and avoid invocation of withdraw() several times for the the same origin transaction

For Home chain it is necessary to check that

withdraws(ForeignTransactionHash)

returns false and invoke withdraw() in this case only.

At this moment the bridge contracts are working with pre-defined list of authorities which is passed to the contract constructor during deployment process. There is no possibility to extend the list if new authority is added later or the number of required signatures needs to be changed.

Proposal for changes

1. Introduce a new contract Authorities
2. Change HomeBridge and ForeignBridge:
   • to verify messages sender by using Authorities contract
   • to verify if necessary number of signatures collected
3. Introduce the functionality to add and remove authorities, to change number of required signatures
4. Consider and implement a model to manage authorities list:
   • it can be done by bridge owners;
   • or it can be done by authorities voting.
5. Probably it is a good idea to allow owners of bridge to change address of Authorities contract in order to allow change consensus describing in item 4 above.

I'm running Parity-bridge, based on the current version of the README, using:

./parity-bridge/target/release/bridge --config config.toml --database db.toml

And my db.toml file is like this:

home_contract_address = "0x74A71bFb6C4903d79EbDd77C556D2d43A9346BdD"
foreign_contract_address = "0xD6fE4c32ADDe8e4b5eD6272Ecb5c434Dc7b120c4"
home_deploy = 2025251 # block number at which the contract was created for home contract you can find in `bridgeDeploymentResults.json`
foreign_deploy = 6923247 # block number at which the contract was created for foreign contract you can find in `bridgeDeploymentResults.json`
checked_deposit_relay = 2025251 # last checked deposits events on Home network
checked_withdraw_relay = 6923247 # last checked withdraw events on Foreign network
checked_withdraw_confirm = 6923247 # last checked withdraw events on Foreign network

However, once the parity-bridge process starts, the file gets corrupted. It looks like this:

home_contract_address = "0x74a71bfb6c4903d79ebdd77c556d2d43a9346bdd"
foreign_contract_address = "0xd6fe4c32adde8e4b5ed6272ecb5c434dc7b120c4"
home_deploy = 2025251
foreign_deploy = 6923247
checked_deposit_relay = 2026365
checked_withdraw_relay = 6924041
checked_withdraw_confirm = 6924041
loy = 6923247 # block number at which the contract was created for foreign contract you can find in `bridgeDeploymentResults.json`
checked_deposit_relay = 2025251 # last checked deposits events on Home network
checked_withdraw_relay = 6923247 # last checked withdraw events on Foreign network
checked_withdraw_confirm = 6923247 # last checked withdraw events on Foreign network

The Home and Foreign account balances show as zero, even though I transferred ETH into them, and I suspect this is the cause.

https://github.com/poanetwork/parity-bridge/blob/master/contracts/bridge.sol#L267

    string public name = "ForeignBridge";

it will be shown on every DEX.
we also need to add symbol = "POA" in order to follow ERC20 standard

I propose for "POA ERC20" as a name of the token

@akolotov, @igorbarinov thoughts?

Problem:

current README.md is for paritytech/parity-bridge version of parity
we split and have our own vision and documentation

Solution:

• Update the README.md
• Move part of the documentation to the Wiki https://github.com/poanetwork/parity-bridge/wiki

This is the issue is the same as paritytech/parity-bridge#149 but behavior is even worse due to automatic bridge restart implemented in POA bridge.

Network setup:

• Home: a PoA testnet (Sokol)
• Foreign: Ropsten

There are 1200 deposit transactions sent successfully to HomeBridge contract by a special python script. It took 8 blocks to validate all transactions.
https://sokol-explorer.poa.network/block/1418808
...
https://sokol-explorer.poa.network/block/1418815

The bridge discovered part of these transactions, tried to relay some of them, lost the connection and restarted:

INFO:bridge::bridge::deposit_relay: got 7 new deposits to relay
INFO:bridge::bridge::deposit_relay: relaying 7 deposits
INFO:bridge::bridge::deposit_relay: deposit relay completed
INFO:bridge::bridge::deposit_relay: got 129 new deposits to relay
INFO:bridge::bridge::deposit_relay: relaying 129 deposits
INFO:bridge::bridge::deposit_relay: deposit relay completed
INFO:bridge::bridge::withdraw_relay: got 0 new signed withdraws to relay
INFO:bridge::bridge::withdraw_relay: fetching messages and signatures
INFO:bridge::bridge::withdraw_relay: fetching messages and signatures complete
INFO:bridge::bridge::withdraw_relay: relaying 0 withdraws
INFO:bridge::bridge::withdraw_relay: relaying withdraws complete
INFO:bridge::bridge::withdraw_relay: waiting for signed withdraws to relay
INFO:bridge::bridge::withdraw_confirm: got 0 new withdraws to sign
INFO:bridge::bridge::withdraw_confirm: signing
INFO:bridge::bridge::withdraw_confirm: signing complete
INFO:bridge::bridge::withdraw_confirm: submitting 0 signatures
INFO:bridge::bridge::withdraw_confirm: submitting signatures complete
INFO:bridge::bridge::withdraw_confirm: waiting for new withdraws that should get signed
INFO:bridge::bridge::deposit_relay: got 113 new deposits to relay
INFO:bridge::bridge::deposit_relay: relaying 113 deposits
INFO:bridge::bridge::deposit_relay: deposit relay completed
INFO:bridge::bridge::deposit_relay: got 710 new deposits to relay
INFO:bridge::bridge::deposit_relay: relaying 710 deposits
WARN:bridge: Bridge is down with Request eth_sendTransaction timed out, attempting to restart
WARN:<unknown>: Sending a response to deallocated channel: Ok([Ok(String("0xc47e8427c9eda913b9749bf0904cd8765b39f5448368194cb7aa4330bbe6a44d"))])
WARN:<unknown>: Sending a response to deallocated channel: Ok([Ok(String("0x4d96f7f39c50dd7b19aac9af708bca450fb6c9fb6b72024c7736d9883614845d"))])
...
WARN:<unknown>: Sending a response to deallocated channel: Ok([Ok(String("0x6b16c0ef9e34d65c4431c4e1e8c043564d53cf3ce440d7ea706164838904e209"))])

The database file was not updated so the restart of the bridge thread caused the same error.

INFO:bridge::bridge::deposit_relay: got 951 new deposits to relay
INFO:bridge::bridge::deposit_relay: relaying 951 deposits
WARN:<unknown>: Sending a response to deallocated channel: Ok([Ok(String("0x9b034a82ebf6306933a35848f9d7b1552ababb8be5774e5501e4802013911a01"))])
WARN:<unknown>: Sending a response to deallocated channel: Ok([Ok(String("0x606b57af0b13dbeddf6b4c16833b0365dace777cc4feee82d2de14c5dde53c45"))])

So, the bridge is continuing to send transactions forever.

Even if the bridge process is killed manually, it is necessary to do manual modification of database but it causes lock of funds on HomeBridge contract side since incomplete amount of tokens is transfered by ForeignBridge contract.

It is necessary to develop a procedure and corresponding automation to easily deploy bridge on several nodes.

Key points

1. It is necessary to provide ability to deploy bridge contracts as a separate step of the process. Results of this step: contract addresses for HomeBridge and ForeignBridge, block numbers where these contracts were deployed.
2. Another step should allow easily generate configuration and database files for particular bridge instance. Most sensitive information is:
   • accounts which are used to communicate to the nodes (home.account and foreign.account)
   • communication channel with the node (home.ipc, home.required_confirmations, home.poll_interval, home.request_timeout, foreign.ipc, foreign.required_confirmations, foreign.poll_interval and foreign.request_timeout)
3. bridge process must be run as a service on target system.
4. Logs must be enabled by default, redirected to a file, log rotation must be configured.

https://github.com/poanetwork/parity-bridge/blob/master/contracts/bridge.sol#L165

@akolotov should we make it public?

Reproduced on the latest (3961987) POA master branch.

Steps to reproduce:

1. Run two Parity-based nodes responsible for Home and Foreign chains.
2. Run bridge: RUST_LOG=info bridge --config ... --database ....
3. Kill parity process responsible for Foreign chain.

Expected results:
The bridge handles gracefully death of Parity node: warns about the connection lose, shutdowns all operations (deposit_relay, withdraw_confirm and withdraw_relay) for a while, waits when the connection appears and runs all operations after that.

Actual results:
After killing Parity process the following appear in the terminal where the bridge is running:

WARN:<unknown>: Unexpected IO error: Error { repr: Os { code: 32, message: "Broken pipe" } }

No messages appear from withdraw_confirm and withdraw_relay.
Then after some time (few seconds or few minutes) the following appear on the terminal and the bridge dies:

Request eth_blockNumber timed out

The behaviour to re-establish IPC connection after parity re-run was introduced as part of fix for #22 (9c375cc).

But it caused appearance of minor regression: last set of transactions are being sent twice. Here is the logs from the foreign parity instance before it is killed:

2018-03-10 09:02:53     0/25 peers   87 KiB chain 5 MiB db 0 bytes queue 448 bytes sync  RPC:  0 conn,  1 req/s, 175 µs
2018-03-10 09:03:10  Transaction mined (hash e0ac09e000484637fb5af07649d90103edbe6dc48640caa2e81956ac11d88b36)
|-------- deposit() invocation
2018-03-10 09:03:10  Imported #6501 fa4a…cf4e (1 txs, 0.08 Mgas, 0.99 ms, 0.77 KiB)
2018-03-10 09:03:20  Transaction mined (hash 7914bbbcc10c105823689628b9815c340eb1a923433b8e5c64ebd8664df31eeb)
|-------- approveAndCall() invocation
2018-03-10 09:03:20  Imported #6502 f517…7239 (1 txs, 0.06 Mgas, 1.01 ms, 0.83 KiB)
2018-03-10 09:03:25  Transaction mined (hash 2c6324d205e276170013f8f4450a7379998b4b4fb4b06001d3b296257b84e103)
|-------- submitSignature() invocation
2018-03-10 09:03:25  Imported #6503 50b8…91df (1 txs, 0.26 Mgas, 1.12 ms, 1.03 KiB)
2018-03-10 09:03:28     0/25 peers   98 KiB chain 5 MiB db 0 bytes queue 448 bytes sync  RPC:  0 conn,  2 req/s, 186 µs

The database file contains the following after that:

$ cat ../erc20_db.toml
home_contract_address = "0x2ace2268ed7a96713e6cd18e9b2c2ef0c306c22c"
foreign_contract_address = "0x5e702ea5d81e14ba807fdd0ac923e811a12bfef1"
home_deploy = 6209
foreign_deploy = 6488
checked_deposit_relay = 6218
checked_withdraw_relay = 6503
checked_withdraw_confirm = 6503

As soon as the parity instance is killed and run it produces the following logs:

2018-03-10 09:04:29  Starting Parity/v1.9.2-unstable-0feb0bb-20180201/x86_64-linux-gnu/rustc1.20.0
2018-03-10 09:04:29  Keys path /home/koal/parity/keys/PoA_foreign
2018-03-10 09:04:29  DB path /home/koal/parity/PoA_foreign/chains/PoA_foreign/db/d87bc73279457e60
2018-03-10 09:04:29  Path to dapps /home/koal/parity/PoA_foreign/dapps
2018-03-10 09:04:29  State DB configuration: fast
2018-03-10 09:04:29  Operating mode: active
2018-03-10 09:04:29  Configured for PoA_foreign using AuthorityRound engine
2018-03-10 09:04:30  Public node URL: enode://96b8fa10c0504cb3edb182786fcc8baf6cec3ff5bc4b5f9f82a24c49fe6bd5d09b64a8a2a595ec8d215d0523ec254cf31dda6151cf0a4669edc7bb964fd50af8@192.168.1.15:30343
2018-03-10 09:04:35  Transaction mined (hash 2faba8376df4632da7b4a0ce4cb984b7e51cc90849bf94e8a56a635195253537)
|-------- deposit() invocation
2018-03-10 09:04:35  Transaction mined (hash 00da53a569c639617e23854a2fb3e80745690c69230e07df92e7159f1fe1c958)
|-------- submitSignature() invocation
2018-03-10 09:04:35  Imported #6504 783d…e97e (2 txs, 4.20 Mgas, 1.15 ms, 1.23 KiB)

and the database file contains:

$ cat ../erc20_db.toml
home_contract_address = "0x2ace2268ed7a96713e6cd18e9b2c2ef0c306c22c"
foreign_contract_address = "0x5e702ea5d81e14ba807fdd0ac923e811a12bfef1"
home_deploy = 6209
foreign_deploy = 6488
checked_deposit_relay = 6219
checked_withdraw_relay = 6504
checked_withdraw_confirm = 6504

These transactions are handled properly by the contracts that's why double-spend does not happen. But it increases expenses of bridge authorities by producing more transactions.

Since bridge contracts are in separate project just now and deployed separately, 'solc' is not needed as dependency for the bridge.

If the error like this appears in the logs:

INFO:bridge::bridge::withdraw_confirm: waiting for new withdraws that should get signed
WARN:bridge: Bridge crashed with Error(Transport("Incomplete"), State { next_error: None, backtrace: None })
Error(Transport("Incomplete"), State { next_error: None, backtrace: None })

it is hard to understand which side of the bridge failed. The message must contains type of operation (deposit_relay, withdraw_confirm or withdraw_relay) and side of bridge (URL of RPC channel).

Since the bridge does not configure list of authorities as part of deployment procedure, the parameter authorities.accounts needs to be removed from configuration file and the corresponding pieces of Rust code.

Currently the length of message to withdrawal is hard coded:

https://github.com/poanetwork/poa-bridge-contracts/blob/93969cd934190649d42691f054676d0d16a9777e/contracts/libraries/Message.sol#L65

and it is also hard coded in the bridge code:

https://github.com/poanetwork/poa-bridge/blob/cdb79b9ba72bc7094ab7e03b133d4e03a1b5d48a/bridge/src/message_to_mainnet.rs#L19

So, it is necessary to change the length in both places if format of the message is being changed.

As soon as the issue omni/tokenbridge-contracts#39 is addressed in the bridge contract, it is necessary to change the bridge code as so the bridge calls the new method during startup sequence and gets actual value for the message. It will reduce amount of work when new bridge contracts are being introduced.

Bridge should read from config.toml

gas_price_oracle_url = "https://gasprice.poa.network/"
gas_price_speed_type = "fast"

and receive this value in gwei, convert to hex value in wei for signing tx on foreign.
this value could be cached only for short period of time ( 5-10min)

If a node configured as Foreign for the bridge and it has no validator account unlocked the bridge crashes and produces the following message:

INFO:bridge::bridge::withdraw_confirm: got 1 new withdraws to sign
INFO:bridge::bridge::withdraw_confirm: withdraw is ready for signature submission. tx hash 0x6493…4fa8
INFO:bridge::bridge::withdraw_confirm: signing
WARN:bridge: Bridge crashed with Error(Transport("Unexpected response status code: 405 Method Not Allowed"), State { next_error: None, backtrace: None })
Error(Transport("Unexpected response status code: 405 Method Not Allowed"), State { next_error: None, backtrace: None })

Currently there are two possible situations related to low balance on the account which is used for bridge operations:

1. The account which is used to sign transactions to be addressed by ForeignBridge contract has low balance. So, the bridge is not able to do deposit_relay and withdraw_confirm.
2. The account which is used to sign transactions to be addressed by HomeBridge contract has low balance. So, the bridge is not able to do withdraw_relay.

In both cases bridges hangs silently at the moment of sending transactions and does not proceed with further actions even the operation is intended to be performed in opposite direction (e.g. the bridge hangs at the moment to perform withdraw_relay, so deposit_relay cannot be performed either).

Proposal for changes

Fix the issue - the bridge should not hang if the account has low balance. It should proceed with operations in the opposite direction. It can be achieved by preventing sending transaction if such situation occurs. The error message should be appear in the bridge logs as soon as low balance issue discovered.

Previously messages appeared as part of Withdraw event in ForeignBridge contract were put as is into the map messages. When the new module MessageToMainnet was introduced some integer values from messages were presented with little endian encoding. This causes bug in handling value and homeGasPrice on HomeBridge side and final transfer of funds cannot be completed.

Investigation is required to understand if it is necessary to use little endian encoding at all.

As soon as the issue is fixed changes to withdraw() of HomeBridge contract need to be applied to remove the workaround.

As per discussion with the team which is doing security audit of the bridge we decided to have one account to sign transactions in home and foreign network (https://github.com/secureware/reports/blob/master/POA_Network-POA_Bridge_Security_Assessment.pdf).

So, consider to change handling of the bridge configuration file as per the following structure:

keystore = "/path/to/keystore"
account = "0x006e27b6a72e1f34c626762f3c4761547aff1421"
password = "password.txt"

[home]
rpc_host = "http://localhost"
rpc_port = 8545
required_confirmations = 0

[foreign]
rpc_host = "http://localhost"
rpc_port = 9545
required_confirmations = 0
...

home_deploy and foreign_deploy fields from database toml file are not needed anymore since bridge does not make deployments.

Currently there is huge delay for withdrawal, the bridge only sends 1 tx per block.
It's unacceptable performance

1000 tx = 1000 blocks = 1000 * 5 sec = 5000 sec to process . = 83.33333 minutes.

Now new nonce is being looked for silently:

https://github.com/poanetwork/poa-bridge/blob/83eeb9d51bbf7c0d112ddacb8179e4b730486ced/bridge/src/bridge/nonce.rs#L107-L110

So, we will not see reasons why sending of transaction could hang for some time due to a reason of incorrect nonce.
That's why a warning message needs to be added to inform about this incorrect situation.

Before launching "POA <--> POA20" into production mode we need to have metrics and alerts for the "POA <--> POA20" bridge.

The bridge as a service has several independent parts on validators nodes and on two chains. Let's assume that the prodcution version will use three validators. We need:

• 3 nodes of validators
  • 3 instances of "parity-bridge". No health API (?)
  • 3 instances of Foundation client with an unlocked account. Health API is not available from outside connection
  • 3 instances of Core client with an unlocked account. Health API is not available from outside
• 3 balances on Foundation network
• 3 balances on POA network

Open questions:

• balances of validators on POA on Ethereum.

Do we have a monitor in our monitoring system capable to monitor balances and calculate thresholds?Let's assume that we will change monitors after modifying quotas.

• exceptions in logs

Do we have exceptions we need to create alerts for?

• stop/start of services

How to know that a service is stopped?

• a difference in balances in Home vs Foreign. Can be used to track risks of
  • stopped services (deposited >> minted)
  • unauthorized minting of POA20 (minted >> deposited)

Currently the bridge instance support logging through configuration of the environment variable RUST_LOG. The bridge produces the logs with log level info and allows to get details about JSON-RPC call by setting the level debug.

env RUST_LOG=info ./bridge --config config.toml --database db.toml

example of logs: https://gist.github.com/akolotov/47d951346db3627562acc7e6578dabaa#file-01_info_level-md

or

env RUST_LOG=debug ./bridge --config config.toml --database db.toml

example of logs: https://gist.github.com/akolotov/47d951346db3627562acc7e6578dabaa#file-02_debug_level-md

From examples above it is clear that info level does not provide enough information to investigate possible issues whereas debug level produces lots of lines (not presented in the example above) which are superfluous for investigation of issues related to skipped or missed transactions or hard to use for tracing state of transfers.

Proposal for changes:

It is necessary to extend the logging functionality on 'info' level in order to have information hanlded and generated by the bridge instance more transparent.
Below is the list of information needed for every type of operation:

1. deposit_relay

   • tx hash of the transaction used to send funds to the HomeBridge contract, transaction originator, value of transaction.
   • tx hash of the transaction sent by the bridge to the ForeingBridge contract, a reference tx hash (allow to link this transaction to transaction from previous item)
   • information whether respond from the bridge contract (Deposit() event) recieved with the reference tx hash
   • information about other bridges responds with the reference tx hash

2. withdraw_confirm

   • tx hash of the transaction used to request withdraw from the ForeignBridge contract, transaction originator, value of transaction.
   • tx hash of the transaction sent by the bridge to the ForeingBridge contract, a reference tx hash (allow to link this transaction to transaction from previous item)
   • information whether respond from the bridge contract (event implemented under #9) recieved with the reference tx hash
   • information about other bridges responds with the reference tx hash

3. withdraw_relay

   • tx hash of the transaction used to inform about signatures collected on ForeignBridge contract with decision if this bridge instance is responsible for relaying the withdraw
   • tx hash of the transaction sent by the bridge to the HomeBridge contract, a reference tx hash (allow to link this transaction to transaction from previous item)
   • information whether respond from the bridge contract (Withdraw() event) recieved with the reference tx hash. This should be included even if another instance of bridge sent information to the HomeBridge contract.