paritytech / cumulus Goto Github PK

Write Parachains on Substrate

License: GNU General Public License v3.0

Rust 97.31% Dockerfile 0.15% Shell 1.71% JavaScript 0.26% Ruby 0.18% Handlebars 0.06% HTML 0.33%

cumulus's Introduction

Dear contributors and users,

We would like to inform you that we have recently made significant changes to our repository structure. In order to streamline our development process and foster better contributions, we have merged three separate repositories Cumulus, Substrate and Polkadot into a single new repository: the Polkadot SDK. Go ahead and make sure to support us by giving a star ⭐️ to the new repo.

By consolidating our codebase, we aim to enhance collaboration and provide a more efficient platform for future development.

If you currently have an open pull request in any of the merged repositories, we kindly request that you resubmit your PR in the new repository. This will ensure that your contributions are considered within the updated context and enable us to review and merge them more effectively.

We appreciate your understanding and ongoing support throughout this transition. Should you have any questions or require further assistance, please don't hesitate to reach out to us.

Best Regards,

Parity Technologies

cumulus's People

Contributors

Stargazers

Watchers

Forkers

ltfschoen juniuszhou bikinibottomsveryown jasonzhoupw pillardevelopment en cheme blockchain-it-hr efimovcapital bewater2020 litentry iammelea guoxbin expenses 0xthreebody akru mattsse pscott xutdns kiltprotocol moonbeam-foundation kunzhangs312 airalab iorveth lucas7788 danforbes c0ding aadorjanfi365 laminar-protocol yangmiok playzero justinphamnz codemeit allenchain polka-defi nuke-web3 kylin-network polkaoracle zk-oracle kaicodex web3capital spiderdao vmarkushin atmaramathva lloyd-protocol gguoss cover001 mitchelltesla jeff31521157 zentachain chainx-org mrq1911 astarnetwork whalelephant benwhitejam notlesh patractlabs dejavukong satoshi-kusumoto ljg-cqu mathwallet snowfork sekmet subdao-network sukhavati-labs unknown522 mapdev33 4meta5 mmyyrroonn aadorian radupopa2010 shaunxw uni-arts-chain hging icetea-labs enfipy caosbad sunriseprotocol valoisprotocol jasl cryptofk apiprotocol stjordanis nichlor icodein byron internfts stanly-johnson ultra-pool viragmody titikterang shadowsnetwork setheum-labs kichjang plutosnetwork kroos47 apron-network totem-tech konomi-network c410-f3r

cumulus's Issues

What types to use with Polkadot API

The Polkadot JS API supports interacting with the standard Substrate node out of the box, and with custom Substrate-based nodes by supplying type information about any types that have been customized (docs).

When I connect the API to the cumulus test parachain (eg using apps) and submit an extrinsic, the API reports an error.

submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1002: Verification Error: Execution: Could not convert parameter `tx` between node and runtime: Invalid transaction version: Client(Execution(ApiError("Could

What custom types need to be supplied to interact with the cumulus node?

JustifiedBlockAnnounceValidator requires the Polkadot service which is created later

At the moment there is a DelayedBlockAnnounceValidator that allow to validate data when its internal
BlockAnnounceValidator is set. Ideally we don't want to have that "delayed" mechanism at all.

This is required for now because the BlockAnnounceValidator is set during the build of a
Service (see substrate: client/service/builder.rs) and our JustifiedBlockAnnounceValidator
requires a Polkadot client but we get this client only when the Polkadot service is created
(chicken and egg problem).

One way to solve this would be to split the start_collator into a "prepare" step where we
receive the service with the client and a "build/start" step where we finalize and run the
collator. Unfortunately the type of the service may vary (Polkadot, Kusama, Westend) and it
can't be dynamically dispatched because the services (and the client) have Self as type
parameters in the supertraits (or where-clauses).

No detail on the error when polkadot's essential task fails

It just says:

2020-06-11 14:09:25.769 tokio-runtime-worker ERROR sc_service  Essential task `polkadot` failed. Shutting down service.

It doesn't give any useful error even though there is one.
It doesn't shutdown the process (only the service)

`CODE` is part of the proof

Because of the current design of substrate, CODE is always part of the proof. This is reading CODE and will make the recorder add it.
This adds some extra data to the proof that is not required and just increases the size of the proof.

Substrate requires that we provide a type for the light client

With this pr we start a full client, when a light client should be started. This is required to make the interface happy, as it wants a type for the light client.
As we currently don't support light clients, we should return to panic when someone wants to start a light client.

[DOC] Unable to register a parachain

I'm trying to register a parachain following the part 'Running a collator'. However, when i try to register a parachain, point 6, i'm not able to do it since the instructions does not corrispond to the current web UI version (i think the documentation is not updated)

Peer set cultivation

When synchronizing the parachain, we will have a reference to a relay chain which is fully to completely synchronized. We want to sync our parachain blocks from peers who have the same chain as is in the relay chain.

We want to compute a list of blocks used to check peers.

If the parachain supports log2-ancestors in the state, we can compute a list of "valid blocks" by following the last finalized block and looking at the log2-ancestors within its state.

Otherwise, we just watch the parachain block hashes coming in recent polkadot blocks and sync only from peers who have those.

Collator Implementation

We want to provide an implementation of Polkadot's ParachainContext. As a first step, this should generate a block, a state proof of execute_block and produce a BlockData which is the block body, the proof, and the next header. We will ignore the incoming messages for now.

The next phase is to extend the state proof to cover some inherent, generated from incoming messages. This will depend on #5

Log filtering for the Polkadot node

The logs of the embedded Polkadot node are interleaved with the logs of the Cumulus node. There should be some filtering applied so that we see mostly the logs of the Cumulus node and only warnings / periodic status updates from the Polkadot chain.

Transactions rollback

Motivation

I'm implementing a collator using a cumulus example parachain at this branch. It really works, catchup Polkadot local testnet updates and follow polkadot consensus. Thank you for that!

Problem

During using a parachain registered in my testnet I see a lot of rolled-back transactions. It looks like transaction applied in UI but after some time state changes come back to initial. It happens in 1 of 5 cases that transactions successfully applied for a long time. In the collator log I a lot of reorgs, probably it's the core of the problem.

Test parachain does not compile

On commit 7fb8b2f (latest master at time of writing), I can't compile the test parachain.

$ cargo check -p cumulus-test-parachain-collator
    Updating git repository `https://github.com/paritytech/polkadot`
error: failed to load source for a dependency on `polkadot-primitives`

Caused by:
  Unable to update https://github.com/paritytech/polkadot?branch=cumulus-branch#b43b0455

Caused by:
  revspec 'b43b04552ac3279e446558aa9db33063deb70f11' not found; class=Reference (4); code=NotFound (-3)

XCMP

The Cross Chain Message Passing system will be implemented in Polkadot, for more information see: paritytech/polkadot#597

Cumulus will need to adapt to the interfaces provided by Polkadot on the client and runtime side.

We will also require specialized networking to support sending the XCMP messages between different Parachains.

Cumulus on substrate crates release

Motivation

I'm interested in discard using custom branches like cumulus-branch as soon as possible. Because it produce double dependencies in the runtime code that makes incompatible by other parts of the node. As an alternative probably is possible to use the latest substrate crates release like https://github.com/paritytech/substrate/releases/tag/v2.0.0-alpha.8

Switch from `Proposer` to `Slots`

Instead of using Proposer as interface for building the blocks in the collator, we probably want to switch over to Slots as the interface. At the point when we want to support custom authoring, using Slots is easier as Babe and Aura already implemented using this trait.

We would need to change the trait in Substrate to return the Proposal, because we need this in the collator.

Process is not stopping with SIGINT/SIGTERM when mdns is active

When pressing CTRL+C or sending SIGTERM the process stops any activity but doesn't end.

For some reason this problem does not occurs with the --dev parameter which makes the test pass.

This issue exist only on cumulus. Polkadot and substrate are working fine.

Collator node throws incorrect warnings on startup

When starting the collator node, it reliably prints errors like

Unable to bind server to 127.0.0.1:9955. Trying random port.
Not a valid bootnode address: /ip4/127.0.0.1/tcp/30333/QmbQMG8VWKumRyXg47UixQiosUMP2nWRsmWRojqDqJTG9t

These messages do not correspond to actual underlying errors. UIs still connect to the websocket port correctly, and the collator still peers with the bootnode at the supposedly invalid address. Possibly related to #19

IncompleteDatabase runtime failure

Traceback

2020-05-11 13:30:06 panicked at 'Calculates storage root: IncompleteDatabase(0x82faa27dd970cd50c955d0c97605fae0e5a3f505db0ed812955e8beb7842647d)', /home/akru/.cargo/git/checkouts/cumulus-59522f43471fa161/e810cf3/runtime/src/validate_block/implementation.rs:209:20

Description

This failure reached after two days of stable work as parachain in local testnet.

Implement `storage_changes_root` in validation context

We need a proper implementation of https://github.com/paritytech/cumulus/blob/master/runtime/src/validate_block/implementation.rs#L261-L264 (host_storage_changes_root).

CC @svyatonik could we make the implementation of storage_changes_root no_std compatible in Substrate? Than we could reuse it here directly.

Provide randomness from relay chain VRF output. Also clock from median algorithm

We expect that parachains might lack enough collators to produce secure randomness, even if they run sensible BABE configurations. If they run AURA with collective flip then their randomness could be significantly worse than their own blockhash. We should provide randomness from the relay chain VRF for use in parachains.

We need each parachain block to acknowledge some parent-like block on relay chain, yes? I suppose the watermark in XCMP gives one example, but way too loose for this use case. Our new A&V design has two relay chain blocks R_2 > R_1 associated to each a prachain block B_1, the second R_2 when B_1 actually runs may not yet exist when B_2 > B_1 gets created, but R_1 exists.

Anyways..

There is always some relay chain parent-like block so it's VRF would provide good randomness for the parachain.

Afiak, there is not much reason to work on this until A&V spec and implementation get more nailed down.

cc @AlistairStewart @rphmeier

Collation failure: timeout

With this chainspec I started polkadot node, which connected with other nodes at substrate.seminar.

Then I have built parachain collator and ran it with the same chainspec.
At this point there was no collators, so it should be the first one.
As a result collator was not introducing any blocks due to timeout error (maybe ralated to #27 ).

Output:

Cumulus Test Parachain Collator
2019-12-03 17:32:33   version 0.1.0-439b4e7-x86_64-linux-gnu
2019-12-03 17:32:33   by Parity Technologies <[email protected]>, 2019
2019-12-03 17:32:33 Chain specification: Local Testnet
2019-12-03 17:32:33 Node name: direful-berry-6194
2019-12-03 17:32:33 Roles: FULL
2019-12-03 17:32:33 Parachain id: Id(0)
2019-12-03 17:32:33 Highest known block at #0
2019-12-03 17:32:33 Using default protocol ID "sup" because none is configured in the chain specs
2019-12-03 17:32:33 Local node identity is: QmaD1tPTMSddqZnGaMA8sA9aUN15G4ECTEHdvCjUJr6fB8
2019-12-03 17:32:33 Unable to bind server to 127.0.0.1:9944. Trying random port.
2019-12-03 17:32:33 Not registering Substrate logger, as there is already a global logger registered!
2019-12-03 17:32:34 Cumulus Test Parachain Collator
2019-12-03 17:32:34   version 0.1.0-439b4e7-x86_64-linux-gnu
2019-12-03 17:32:34   by Parity Technologies <[email protected]>, 2017-2019
2019-12-03 17:32:34 Chain specification: Cumulus Relay
2019-12-03 17:32:34 Node name: spiky-jail-4621
2019-12-03 17:32:34 Roles: FULL
2019-12-03 17:32:34 Highest known block at #1799
2019-12-03 17:32:34 Local node identity is: QmXEdFrCUxjthf5sHo5oHZ1HffSTpiPeFK1W5mQL5uC8qu
2019-12-03 17:32:34 Can't listen on /ip4/0.0.0.0/tcp/30333 because: Other(Custom { kind: Other, error: Other(A(A(A(B(Underlying(B(Os { code: 98, kind: AddrInUse, message: "Address already in use" }))))))) })
2019-12-03 17:32:34 Unable to bind server to 127.0.0.1:9944. Trying random port.
2019-12-03 17:32:36 Imported #1805 (0x9b41…32b2)
2019-12-03 17:32:36 Starting consensus session on top of parent 0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a
2019-12-03 17:32:36 Proposing failed: UnknownBlock("Hash(0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a)")
2019-12-03 17:32:36 Collation failure: timeout
2019-12-03 17:32:39 Idle (3 peers), best: #1805 (0x9b41…32b2), finalized #1804 (0xa4ef…98eb), ⬇ 5.1kiB/s ⬆ 2.8kiB/s
2019-12-03 17:32:42 Imported #1806 (0x764a…27a2)
2019-12-03 17:32:42 Starting consensus session on top of parent 0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a
2019-12-03 17:32:42 Proposing failed: UnknownBlock("Hash(0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a)")
2019-12-03 17:32:42 Collation failure: timeout
2019-12-03 17:32:44 Idle (3 peers), best: #1806 (0x764a…27a2), finalized #1805 (0x9b41…32b2), ⬇ 1.6kiB/s ⬆ 0.9kiB/s
2019-12-03 17:32:48 Imported #1807 (0xad24…1e5d)
2019-12-03 17:32:48 Starting consensus session on top of parent 0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a
2019-12-03 17:32:48 Proposing failed: UnknownBlock("Hash(0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a)")
2019-12-03 17:32:48 Collation failure: timeout
2019-12-03 17:32:49 Idle (3 peers), best: #1807 (0xad24…1e5d), finalized #1805 (0x9b41…32b2), ⬇ 1.7kiB/s ⬆ 1.2kiB/s
2019-12-03 17:32:54 Idle (3 peers), best: #1807 (0xad24…1e5d), finalized #1806 (0x764a…27a2), ⬇ 198.8kiB/s ⬆ 132.4kiB/s
2019-12-03 17:32:56 Imported #1808 (0xbad8…3445)
2019-12-03 17:32:56 Starting consensus session on top of parent 0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a
2019-12-03 17:32:56 Proposing failed: UnknownBlock("Hash(0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a)")
2019-12-03 17:32:56 Collation failure: timeout
2019-12-03 17:32:59 Idle (2 peers), best: #1808 (0xbad8…3445), finalized #1807 (0xad24…1e5d), ⬇ 131.6kiB/s ⬆ 0.2kiB/s
2019-12-03 17:33:00 Imported #1809 (0xff5f…6bdd)
2019-12-03 17:33:00 Starting consensus session on top of parent 0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a
2019-12-03 17:33:00 Proposing failed: UnknownBlock("Hash(0xaee8680b63df8ee7488a0018c8614ddd5095c67c323e2bc15a894f14e82e7f1a)")
2019-12-03 17:33:00 Collation failure: timeout

Cumulus Runtime

We will want a runtime component for parachains to handle incoming messages.

The module will look something like this.

trait OnMessages {
    fn on_messages(messages: Vec<polkadot::IncomingMessage>);
}

trait Trait {
    type OnMessages: OnMessages;
}

struct<T: Trait> Module<T> {
    // defines an inherent for the incoming messages to be passed. the incoming messages are passed to `T::OnMessages::on_messages`.
}

Test collator does not currently compile

   Compiling cumulus-collator v0.1.0 (/paritytech/cumulus/collator)
error[E0603]: module `client` is private
  --> test/parachain/src/command.rs:26:17
   |
26 | use sc_service::client::genesis;
   |                 ^^^^^^ this module is private
   |
note: the module `client` is defined here
  --> /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/39560f8/client/service/src/lib.rs:33:1
   |
33 | mod client;
   | ^^^^^^^^^^^

error: aborting due to previous error

Presumably substrate has simply shifted out from under us.

CLI relay chain arguments parsing problems

The following CLI should work:

-d cumulus_collator -- --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/Qmbx43psh7LVkrYTRXisUpzCubbgYojkejzAgj5mteDnxy --bootnodes /ip4/127.0.0.1/tcp/50500/p2p/Qma6SpS7tzfCrhtgEVKR9Uhjmuv55ovC3kY6y6rPBxpWd

However, the parser requires extra =:

-d cumulus_collator -- --bootnodes=/ip4/127.0.0.1/tcp/30333/p2p/Qmbx43psh7LVkrYTRXisUpzCubbgYojkejzAgj5mteDnxy --bootnodes=/ip4/127.0.0.1/tcp/50500/p2p/Qma6SpS7tzfCrhtgEVKR9Uhjmuv55ovC3kY6y6rPBxpWd

The behavior without = normally works for CLI arguments.

Switch to Rust async

Code like this: https://github.com/paritytech/cumulus/blob/master/collator/src/lib.rs#L114-L268

Should be converted to proper rust async to make it look better and hopefully to make it better maintainable.

(The code should be scanned for other opportunities)

Weak-subjectivity resistant synchronization

Here are our considerations:

full nodes need to execute all blocks
full nodes are somewhat limited in reorganization capability
we don't have warp sync
it's difficult to evaluate the fork-choice until we've already synced a bunch.
the Polkadot light client has some kind of protection against weak subjectivity that we want to extend to our parachain.

Let's say we are synchronizing a generalized Substrate chain called Local.

Nodes on Local should attach a little piece of metadata (an inclusion-proof) to every Nth Local block called an "inclusion proof". It's a small state proof of the storage changing for the parachain head. There might be multiple of these blocks.

Light and full polkadot nodes can easily tell you the block hash of the ancestor of the finalized block, so the inclusion proofs will be useful.

Who generates these proofs? Any node at the head of the chain can probably do it. After we import a Polkadot block which updated the Local head, we want a merkle proof of the new storage entry. This is easy to check. We keep a list for every Local block with number % N == 0. These lists can be pruned on finality.

On block import of a Local block LB with number % N == 0, we query inclusion proofs for that block where the Polkadot block within is in the finalized chain. If there are none or the Polkadot block is not finalized, we simply import (for now). This is fine assuming that there is not a larger lag in finality than the client can cope with reorganizations.

Tests, tests, we need tests

Yeah, we need tests. We already have "some" unit tests, but we need more. The biggest problem currently is that we lack test primitives.

It would also be nice to have real integration tests that probably spawn some docker containers that test setting up a parachain and making sure that the collator is working properly. A first implementation could just query the heads and make sure that it changes in the relay-chain.

Runtime panics with enabled upgrade pallet

Traceback

2020-05-15 18:30:32 panicked at 'VFPs must be updated once per block', /rustc/20fc02f836f3035b86b56a7cedb97c5cd4ed9612/src/libcore/macros/mod.rs:34:9
2020-05-15 18:30:32 Proposing failed: Execution(Wasmi(Trap(Trap { kind: Unreachable })))

After enabling cumulus-parachain-upgrade pallet runtime panics with errow above.

Possible Safety Violation ...

Following the instructions in the readme (along with the workaround described in #25). I'm able to start a collator and register it with the relay chain.

Upon successful registration, the collator starts producing blocks, but also starts printing the log message Possible safety violation: attempted to re-finalize last finalized block 0x1ba54575ff754aafb1b468b7da21d7c6555daf0fc292cf80565960797dbdd981.

Full logs:

Cumulus Test Parachain Collator
2019-11-29 12:20:38   version 0.1.0-x86_64-linux-gnu
2019-11-29 12:20:38   by Parity Technologies <[email protected]>, 2019
2019-11-29 12:20:38 Chain specification: Local Testnet
2019-11-29 12:20:38 Node name: standing-rings-5270
2019-11-29 12:20:38 Roles: FULL
2019-11-29 12:20:38 Parachain id: Id(0)
2019-11-29 12:20:38 Highest known block at #0
2019-11-29 12:20:38 Using default protocol ID "sup" because none is configured in the chain specs
2019-11-29 12:20:38 Local node identity is: QmVi1qo3rrcqVAFZtJKETnQ3SmEdZcHuQ19BAm3dMt9k3C
2019-11-29 12:20:38 Not registering Substrate logger, as there is already a global logger registered!
2019-11-29 12:20:39 Cumulus Test Parachain Collator
2019-11-29 12:20:39   version 0.1.0-x86_64-linux-gnu
2019-11-29 12:20:39   by Parity Technologies <[email protected]>, 2017-2019
2019-11-29 12:20:39 Chain specification: Local Testnet
2019-11-29 12:20:39 Node name: well-off-hall-2350
2019-11-29 12:20:39 Roles: FULL
2019-11-29 12:20:39 Highest known block at #323
2019-11-29 12:20:39 Local node identity is: QmVj4EZKV9AEHeDs43JLFauDdGAU6Lho83xyaZQXeigUFN
2019-11-29 12:20:39 Can't listen on /ip4/0.0.0.0/tcp/30333 because: Other(Custom { kind: Other, error: Other(A(A(A(B(Underlying(B(Os { code: 98, kind: AddrInUse, message: "Address already in use" }))))))) })
2019-11-29 12:20:39 Unable to bind server to 127.0.0.1:9955. Trying random port.
2019-11-29 12:20:42 Imported #330 (0x3a6a…c9a8)
2019-11-29 12:20:44 Idle (1 peers), best: #330 (0x3a6a…c9a8), finalized #329 (0xf2c7…9bd3), ⬇ 1.9kiB/s ⬆ 0.9kiB/s
2019-11-29 12:20:48 Imported #331 (0xb4eb…3291)
2019-11-29 12:20:49 Idle (1 peers), best: #331 (0xb4eb…3291), finalized #330 (0x3a6a…c9a8), ⬇ 0.9kiB/s ⬆ 0.4kiB/s
2019-11-29 12:20:54 Idle (1 peers), best: #331 (0xb4eb…3291), finalized #330 (0x3a6a…c9a8), ⬇ 65.8kiB/s ⬆ 0.2kiB/s
2019-11-29 12:20:55 Imported #332 (0xaaf8…3385)
2019-11-29 12:20:55 Starting consensus session on top of parent 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372
2019-11-29 12:20:55 Imported #332 (0x7bce…5de3)
2019-11-29 12:20:55 Prepared block for proposing at 1 [hash: 0xa8b930e1297c2ddd4afaf5499ecd6b62d199a621cd16a592150cdf6a405adfc7; parent_hash: 0x6a09…b372; extrinsics: [0xbfe3…a91c]]
2019-11-29 12:20:55 Starting consensus session on top of parent 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372
2019-11-29 12:20:55 Prepared block for proposing at 1 [hash: 0x0cd7ad590f40364289d7eb5934841c9c25f6a3719c769d01b890d7acb98d97e5; parent_hash: 0x6a09…b372; extrinsics: [0xec3b…3d76]]
2019-11-29 12:20:59 Idle (1 peers), best: #332 (0xaaf8…3385), finalized #331 (0xb4eb…3291), ⬇ 197.5kiB/s ⬆ 0.5kiB/s
2019-11-29 12:21:00 Imported #333 (0x7425…133e)
2019-11-29 12:21:00 Starting consensus session on top of parent 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372
2019-11-29 12:21:00 Prepared block for proposing at 1 [hash: 0xd2e5e107f55115a2d111befee92a53b98235b285cedcc4c4a204a701c17c28ac; parent_hash: 0x6a09…b372; extrinsics: [0x3b29…325a]]
2019-11-29 12:21:02 Possible safety violation: attempted to re-finalize last finalized block 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372 
2019-11-29 12:21:04 Idle (1 peers), best: #333 (0x7425…133e), finalized #332 (0xaaf8…3385), ⬇ 66.3kiB/s ⬆ 66.2kiB/s
2019-11-29 12:21:06 Imported #334 (0xadb4…849d)
2019-11-29 12:21:06 Starting consensus session on top of parent 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372
2019-11-29 12:21:06 Prepared block for proposing at 1 [hash: 0x4d228924a1048d00c9bb4c7818f72cf5915397c8312e4d32e948d918dc38549c; parent_hash: 0x6a09…b372; extrinsics: [0xf17e…a93f]]
2019-11-29 12:21:07 Possible safety violation: attempted to re-finalize last finalized block 0x6a09b77fced72eec69dd1dd656950fc2406853e5864effe06c7befa2f1c8b372

Support Babe

We should support Babe for Block authoring.

Blocked on: #36

Pass CLI arguments to parachain/relay chain node

When running the collator, I want to pass CLI arguments that are related to the parachain and to the relay chain. As both are substrate projects, we need to make it possible to pass the same arguments twice to the same binary.

We want the following syntax:

collator --name parachain -- --name relaychain

The first arguments before the -- are the parachain parameters and everything after -- is a relay chain parameter.

--help should make it clear how we expect parameters to be passed.

Maybe we could also include some reasonable defaults for ports, so that they don't clash: #24

Like putting the relay chain ports at default + 1 (if they are not overwritten by a cli arg)

Change Polkadot `run_collator` to `start_collator`

Currently to start a Collator with Polkadot, we use run_collator. I think we should add a function start_collator, that takes almost the same arguments, but returns a future. I think it would make the setup stuff easier, when we are driving the Polkadot future/service and not the other way around (like it is done now). This also replicates more of the "default" model of starting a service in Substrate.

Parachains registered at IDs other than 0 do not collate

The example parachain node has a hardcoded ID of 100. When I follow the instructions to register the parachian, I use id 100 as instructed, and the registration extrinsic completes successfully. However, the collator does not begin collating. One hint from the relay chain node logs is Local parachain duty is Some(LocalDuty { validation: Parachain(Id(0)) }). I expected that value to be 100, not 0.

Second Attempt
Kill all parachain and relay chain nodes, and purge all chains. Start the relay chain again. Start the Collator again, this time try to register at id 0 despite the hard-coded id 100 in the collator code.
Same behavior as before. Now, the collator actually starts producing blocks despite the mismatch between hard-coded id and registered id.

Third Attempt
Modify the collator node to hard-code id 0, and recompile. Kill all nodes and purge all chains. Re-start the relay chain. Re-start the collator. Register at id 0. As in the second attempt, the collator now begins producing blocks as expected.

Split validation and runtime wasm blobs

For the time being, the WASM-blob registered for a parachain on the relay-chain is the same as the runtime WASM-blob of the parachain. The only difference to a normal Substrate chain is that the runtime is augmented with the validate_block function. After some recent discussion, we came to the conclusion to split these WASM-blobs. This will mainly bring us the advantage that the WASM-blob of the parachain living on the relay-chain will hopefully never require an update and thus, can be used to validate any POV block of a parachain (before and after runtime updates). The validation will work in the following way:

The parachain-validator calls validate_block in the parachain WASM-blob.
validate_block will extract the :code (the runtime WASM-blob) out of the WitnessData in the POV block (currently :code is always present in the WitnessData, nevertheless we should make sure that some logic in the block building process reads the blob once to make sure the data is present.)
validate_block instantiates a WASM-sandbox with the :code of the runtime. The storage functions for the runtime will be provided by the validate_block WASM-blob and the other host functions imported by validate_block will be passed exported to the runtime.
validate_block executes Core_execute_block of the runtime.
validate_block returns the result.

To implement this, roughly the following needs to be done:
[] Implement support for passing through host functions to the sandbox (in Substrate)
[] Split the runtime and the validate_block blobs into 2 seperate WASM-blobs. Should be mainly doable in Cumulus.
[] Make sure the runtime WASM-blob is included in the WitnessData.

CC @rphmeier @pepyakin

Runtime upgrade

Substrate supports to upgrade the runtime from within the Blockchain. We need to implement some extended functionality to make the upgrade fit into the Polkadot model of upgrading Parachain runtimes, described here.

validate_block needs to check that :code is only altered when allowed (see the Polkadot issue for the requirements).
Provide a pallet in Cumulus that makes the process of registering an upgrade/checking if an upgrade is doable easy to use.
It would probably be nice to augment set_code etc to fail on applying, if the requirements are not allowed (instead of failing in validate_block). Users can for example send a proposal to upgrade the runtime, this should fail to dispatch if it does not follow the correct path.

Parachain upgrade module compilation fail

Backtrace

   Compiling cumulus-parachain-upgrade v0.1.0 (https://github.com/paritytech/cumulus#d6ab13c0)             
error[E0412]: cannot find type `Vec` in this scope                                                                             
  --> /home/akru/.cargo/git/checkouts/cumulus-59522f43471fa161/d6ab13c/parachain-upgrade/src/lib.rs:48:31
   |                                                                                                                           
48 | pub type ValidationFunction = Vec<u8>;                    
   |                               ^^^ not found in this scope                                                                 
   |                                                           
help: possible candidates are found in other modules, you can import them into scope
   |                                                                                                                           
32 | use frame_support::dispatch::Vec;                                                                                         
   |
32 | use frame_support::inherent::Vec;
   |

error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
   --> /home/akru/.cargo/git/checkouts/cumulus-59522f43471fa161/d6ab13c/parachain-upgrade/src/lib.rs:132:30
    |
132 |             if let Some((apply_block, validation_function)) = PendingValidationFunction::get() {
    |                                       ^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `core::marker::Sized` is not implemented for `[u8]`
    = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-si
zed-trait>
    = note: all local variables must have a statically known size
    = help: unsized locals are gated as an unstable feature

Description

Unfortunately latest version of parachain upgrade pallet failure on compilation.

Fix integration test

The integration test was disabled because it used the Polkadot runtime. We need to switch the test to use the test runtime, because Polkadot for example currently has almost all transactions disabled an in the future it will also drop the sudo module. We really need to use the test runtime to have something stable.

Keep some un-pinned parachain blocks

A "pinned" parachain block is one who's been referenced by a block in the relay chain. Blocks that have been freshly authored by collators are initially are unpinned, while they are circulated and gather attestations from validators. We don't want to keep an unbounded number of unpinned blocks, because it can be a DoS vector and complicates fork-choice.

However, assuming that collators are good and broadcast out the block data for unpinned blocks in a way which is easily obtainable by peers, it is more efficient for other full nodes to keep the unpinned block with the expectation that it likely will be pinned. This means that as soon as the block is pinned, a collator can build on top of it for the next block, rather than having to fetch it and then start building.

Every parachain block needs to reference some data from Polkadot (#5) including incoming messages and relay chain block hash.

There should be an import queue verifier which accepts new blocks if:

they are pinned
they are unpinned and the Polkadot block hash referenced is a leaf.

Block production stuck

Backtrace

After one day the latest version of parachain block production stuck with error:

Verification failed for block 0x53ec2277af2c11036a35dbe743605fb851881318df1998e934cae8c765c7a2d1 received from peer: 12D3KooWLAG1N4LSNiEbCmvAu94ByZoZMTNPWPHzKJF5B15CaCKE, "Execution: Memory: trying to access region [1048576..1048604] in memory [0..0]"

when I fetch parachain from scratch I get:

2020-06-15 18:42:26 Proposing failed: UnknownBlock("Expect block number from id: BlockId::Hash(0x9c0041f7d6616f81d49079fc0af9ba91caa9b9bfb20ce2cf1885371dc51b2f07)")

Collator source code available at https://github.com/airalab/robonomics/blob/df3ca25348905bb578e59dd15b634d824ab79516/bin/node/cli/src/parachain/collator.rs

Public node available at https://parachain.robonomics.network

Prepare test net

Prepare a test net for Parachains.

Need a way for democracy to signal that something will be applied in the future

See https://github.com/paritytech/cumulus/pull/70/files#r412915925.

This would enable a parachain to vote on a validation function upgrade, instead of having it applied by a superuser.

Non-collating Cumulus light/full node

At the moment, it is only possible to run a Cumulus collator node. We want to supplement this with non-collating full nodes and later on, light nodes.

Change `validate_block` signature

validate_block needs to get the last header hash and check that the hash matches the hash in the generated header.

Unwrap/Expect `TrieError`

Currently we unwrap TrieError manually here:

cumulus/runtime/src/validate_block/implementation.rs

Lines 147 to 160 in dcde10c

 fn unwrap_trie_error<R, T, E>(result: Result<R, Box<trie_db::TrieError<T, E>>>) -> R { 

 match result { 

 Ok(r) => r, 

 Err(error) => match *error { 

 trie_db::TrieError::InvalidStateRoot(_) => panic!("trie_db: Invalid state root"), 

 trie_db::TrieError::IncompleteDatabase(_) => panic!("trie_db: IncompleteDatabase"), 

 trie_db::TrieError::DecoderError(_, _) => panic!("trie_db: DecodeError"), 

 trie_db::TrieError::InvalidHash(_, _) => panic!("trie_db: InvalidHash"), 

 trie_db::TrieError::ValueAtIncompleteKey(_, _) => { 

 panic!("trie_db: ValueAtIncompleteKey") 

 } 

 }, 

 } 

 }

After paritytech/trie#59 is merged and published, we should switch to expect.

Support Aura

We should support Aura when authoring a block.

Blocked on: #36

Thread panicked at 'Storage root must match that calculated.'

Following the README instructions for running the project leads to this issue with the collator node. Reproducible by manually following Running a collator or by simply executing the built-in scripts referenced in Running the collator automatically.

Thread 'import-queue-worker-0' panicked at 'Storage root must match that calculated.', <::std::macros::panic macros>:2

A complete trace can be seen in this gist.

Fix multiple collator setup

Running multiple collators lead to problems of collators not receiving Parachain blocks on time. This needs to be investigated and fixed.

Broken Pipe

I was looking into cumulus --help using the grep command. I erroneously typed cumulus --help | grep and this happened:

$> ./target/release/cumulus-test-parachain-collator --help | grep
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.

====================

Version: 0.1.0-e025dab-x86_64-linux-gnu

stack backtrace:
   0: substrate_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:481
   2: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   3: rust_begin_unwind
             at src/libstd/panicking.rs:311
   4: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   5: core::result::unwrap_failed
             at src/libcore/result.rs:1084
   6: clap::errors::Error::exit
   7: clap::app::App::get_matches_from
   8: substrate_cli::parse_and_prepare
   9: cumulus_test_parachain_collator::cli::run
  10: cumulus_test_parachain_collator::main
  11: std::rt::lang_start::{{closure}}
  12: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:49
      std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  13: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  14: std::panicking::try
             at src/libstd/panicking.rs:275
      std::panic::catch_unwind
             at src/libstd/panic.rs:394
      std::rt::lang_start_internal
             at src/libstd/rt.rs:48
  15: main
  16: __libc_start_main
  17: _start


Thread 'main' panicked at 'Error writing Error to stdout: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', src/libcore/result.rs:1084

This is a bug. Please report it at:

        https://github.com/paritytech/cumulus/issues/new

Collation failure on Wasmi

Traceback

Proposing failed: Execution(Wasmi(Memory("trying to access region [1048576..1048660] in memory [0..0]")))

Description

Maybe it's connected to #95.

During massive collation testing with 30 collators, block producing stuck on 2076 block with the error above.

Cannot add second collator to parachain (Collation failure: timeout)

Following the instructions in the readme (along with the workaround described in #25). I'm able to start a collator and register it with the relay chain.

I then try to start a second collator by running the same command with a different data directory. The first collator continues to work as well as it ever did (see #26). But the second collator is not able to collate. It give the log message Collation failure: timeout.

Full logs:

$ ./target/release/cumulus-test-parachain-collator --base-path /tmp/p1c2 --chain=../polkadot/polkadot_chainspec.json --bootnodes /ip4/127.0.0.1/tcp/30333/p2p/QmbQMG8VWKumRyXg47UixQiosUMP2nWRsmWRojqDqJTG9t --ws-port 9966
2019-11-29 12:58:18 Cumulus Test Parachain Collator
2019-11-29 12:58:18   version 0.1.0-x86_64-linux-gnu
2019-11-29 12:58:18   by Parity Technologies <[email protected]>, 2019
2019-11-29 12:58:18 Chain specification: Local Testnet
2019-11-29 12:58:18 Node name: intelligent-balls-4461
2019-11-29 12:58:18 Roles: FULL
2019-11-29 12:58:18 Parachain id: Id(0)
2019-11-29 12:58:18 Initializing Genesis block/state (state: 0x80d8…a364, header-hash: 0x6a09…b372)
2019-11-29 12:58:18 Highest known block at #0
2019-11-29 12:58:18 Using default protocol ID "sup" because none is configured in the chain specs
2019-11-29 12:58:18 Local node identity is: QmaqWJguAo8RbVpqoVxrSEEdGdki9pwZfpH8DUB6tWxzGD
2019-11-29 12:58:19 Not registering Substrate logger, as there is already a global logger registered!
2019-11-29 12:58:19 Cumulus Test Parachain Collator
2019-11-29 12:58:19   version 0.1.0-x86_64-linux-gnu
2019-11-29 12:58:19   by Parity Technologies <[email protected]>, 2017-2019
2019-11-29 12:58:19 Chain specification: Local Testnet
2019-11-29 12:58:19 Node name: entertaining-cord-8336
2019-11-29 12:58:19 Roles: FULL
2019-11-29 12:58:20 Initializing Genesis block/state (state: 0x09d4…4b97, header-hash: 0x680f…0321)
2019-11-29 12:58:20 Loading GRANDPA authority set from genesis on what appears to be first startup.
2019-11-29 12:58:20 Loaded block-time = BabeConfiguration { slot_duration: 6000, epoch_length: 2400, c: (1, 4), genesis_authorities: [(Public(8eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a48 (5FHneW46...)), 1), (Public(d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d (5GrwvaEF...)), 1)], randomness: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], secondary_slots: true } seconds from genesis on first-launch
2019-11-29 12:58:20 Creating empty BABE epoch changes on what appears to be first startup.
2019-11-29 12:58:20 Highest known block at #0
2019-11-29 12:58:20 Local node identity is: QmQPawAffS3Ta1z7dHG6DzBCcW16p9UySF6JAamFLemwq6
2019-11-29 12:58:20 Can't listen on /ip4/0.0.0.0/tcp/30333 because: Other(Custom { kind: Other, error: Other(A(A(A(B(Underlying(B(Os { code: 98, kind: AddrInUse, message: "Address already in use" }))))))) })
2019-11-29 12:58:21 Unable to bind server to 127.0.0.1:9966. Trying random port.
2019-11-29 12:58:21 New epoch 0 launching at block 0xb5f4…5022 (block slot 262507676 >= start slot 262507676).
2019-11-29 12:58:21 Next epoch starts at slot 262510076
2019-11-29 12:58:24 Imported #707 (0xa728…d6bb)
2019-11-29 12:58:24 Starting consensus session on top of parent 0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88
2019-11-29 12:58:24 Proposing failed: UnknownBlock("Hash(0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88)")
2019-11-29 12:58:24 Collation failure: timeout
2019-11-29 12:58:26 Idle (1 peers), best: #707 (0xa728…d6bb), finalized #706 (0xe451…44be), ⬇ 111.5kiB/s ⬆ 0.9kiB/s
2019-11-29 12:58:30 Imported #708 (0x0532…9907)
2019-11-29 12:58:30 Starting consensus session on top of parent 0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88
2019-11-29 12:58:30 Proposing failed: UnknownBlock("Hash(0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88)")
2019-11-29 12:58:30 Collation failure: timeout
2019-11-29 12:58:31 Idle (1 peers), best: #708 (0x0532…9907), finalized #706 (0xe451…44be), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
2019-11-29 12:58:36 Idle (1 peers), best: #708 (0x0532…9907), finalized #707 (0xa728…d6bb), ⬇ 0.9kiB/s ⬆ 0.4kiB/s
2019-11-29 12:58:36 Imported #709 (0x9ff5…9317)
2019-11-29 12:58:36 Starting consensus session on top of parent 0x3253a5215fafb8fcabadf6aa27bd7394ef8744b831ab38ea275bb648fcbbaeb3
2019-11-29 12:58:36 Proposing failed: UnknownBlock("Hash(0x3253a5215fafb8fcabadf6aa27bd7394ef8744b831ab38ea275bb648fcbbaeb3)")
2019-11-29 12:58:36 Collation failure: timeout
2019-11-29 12:58:36 Imported #709 (0xa319…5e13)
2019-11-29 12:58:36 Starting consensus session on top of parent 0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88
2019-11-29 12:58:36 Proposing failed: UnknownBlock("Hash(0xec571ca6ed84a69f305de6cabc233e7daa362ec911c313a04d187b4c206dbf88)")
2019-11-29 12:58:36 Collation failure: timeout
2019-11-29 12:58:41 Idle (1 peers), best: #709 (0x9ff5…9317), finalized #708 (0x0532…9907), ⬇ 1.2kiB/s ⬆ 0.5kiB/s
2019-11-29 12:58:42 Imported #710 (0x0c33…5335)
2019-11-29 12:58:42 Starting consensus session on top of parent 0x3253a5215fafb8fcabadf6aa27bd7394ef8744b831ab38ea275bb648fcbbaeb3
2019-11-29 12:58:42 Proposing failed: UnknownBlock("Hash(0x3253a5215fafb8fcabadf6aa27bd7394ef8744b831ab38ea275bb648fcbbaeb3)")
2019-11-29 12:58:42 Collation failure: timeout
2019-11-29 12:58:46 Idle (1 peers), best: #710 (0x0c33…5335), finalized #709 (0x9ff5…9317), ⬇ 1.0kiB/s ⬆ 0.4kiB/s
2019-11-29 12:58:48 Imported #711 (0x8d75…c160)
2019-11-29 12:58:48 Starting consensus session on top of parent 0x744f4f97067d635274a57984d1f2396a954b859865511b03b522ef8de1705d65
2019-11-29 12:58:48 Proposing failed: UnknownBlock("Hash(0x744f4f97067d635274a57984d1f2396a954b859865511b03b522ef8de1705d65)")
2019-11-29 12:58:48 Collation failure: timeout
2019-11-29 12:58:51 Idle (1 peers), best: #711 (0x8d75…c160), finalized #710 (0x0c33…5335), ⬇ 0.8kiB/s ⬆ 0.1kiB/s
2019-11-29 12:58:54 Imported #712 (0x7ae3…3abf)
2019-11-29 12:58:54 Starting consensus session on top of parent 0x744f4f97067d635274a57984d1f2396a954b859865511b03b522ef8de1705d65
2019-11-29 12:58:54 Proposing failed: UnknownBlock("Hash(0x744f4f97067d635274a57984d1f2396a954b859865511b03b522ef8de1705d65)")
2019-11-29 12:58:54 Collation failure: timeout

Try to remove Polkadot logic

Origin discussion: #53 (comment)

Ideally we don't want to have cumulus code that come from polkadot. This could probably be fixed with #44

	fn unwrap_trie_error<R, T, E>(result: Result<R, Box<trie_db::TrieError<T, E>>>) -> R {
	match result {
	Ok(r) => r,
	Err(error) => match *error {
	trie_db::TrieError::InvalidStateRoot(_) => panic!("trie_db: Invalid state root"),
	trie_db::TrieError::IncompleteDatabase(_) => panic!("trie_db: IncompleteDatabase"),
	trie_db::TrieError::DecoderError(_, _) => panic!("trie_db: DecodeError"),
	trie_db::TrieError::InvalidHash(_, _) => panic!("trie_db: InvalidHash"),
	trie_db::TrieError::ValueAtIncompleteKey(_, _) => {
	panic!("trie_db: ValueAtIncompleteKey")
	}
	},
	}
	}

paritytech / cumulus Goto Github PK

cumulus's Introduction

cumulus's People

Contributors

Stargazers

Watchers

Forkers

cumulus's Issues

Recommend Projects

Recommend Topics

Recommend Org