The rusqlite version we use is quite old and has some problems for Windows builds: @danielSanchezQ reported it prevents fixing input-output-hk/jormungandr#1699, while my standalone experiment with 0.21 and "bundled" worked well with crt-static.

Update the rusqlite dependency to the latest version (keep "bundled" on or optionally-default for the time being). There is a use of r2d2_sqlite backend in chain-storage-sqlite, which is ironically the more outdated crate, so that needs to be dropped and chain-storage-sqlite-old renamed as the only surviving storage implementation.

If we start with consensus genesis praos in block0, we want to check that the system is in a working state. given that only the one that control the stake, control creation of the block, we need to make sure the system is defined correctly:

1. check that there's some stake defined in the system, by calculating the stake distribution
2. check that there's >= 1 stake pool certificates
3. check there's some overlap between the stake pool certificate and the stake distribution

This is more like a unit test. We need to check that the ledger, when it applies a new fragment (a new transaction), verify the discrimination.

This is the code in here:

https://github.com/input-output-hk/rust-cardano/blob/master/chain-impl-mockchain/src/ledger.rs#L672-L674

We need to check that:

• if address 's discrimination is the same as the one in the params, it passes;
• if it differs it fails

Relates to #727

UpdateState::process_proposals has bug in my opinion, which leads to problem in which proposal which was not accepted before proposal_expiration period won't be removed from proposal collection at all.

there are two conditions in mentioned method:

 if prev_date.epoch < new_date.epoch

and

else if proposal_state.proposal_date.epoch + settings.proposal_expiration
                    > new_date.epoch

if we apply proposal in the current epoch (let's say epoch 0 and our proposal_expiration is any value different than 0, then it's impossible to satisfy condition proposal_state.proposal_date.epoch + settings.proposal_expiration < new_date.epoch => 0 + 1 > 1 = false and then new_date.epoch will be increased i believe if we time proceeds, so above condition will be never met and therefore proposal will stay in collection forever).

In different scenario in which we apply proposal for epoch in the future let's say epoch 5 (while we are at epoch 0) then if proposal won't be accepted in epoch 0, then it will be immediately removed from proposals in process_proposals . (5 + 0 > 0 = true)

To fix this situation we need to reverse condition:

else if proposal_state.proposal_date.epoch + settings.proposal_expiration
                    < new_date.epoch

This will solve first scenario in epoch 2 (0+ 1 < 2 = true) and in second scenario proposal won't be removed from proposal before its time will come (5 + 0 < 0 =false )

Testing improvements in the chain library

There is a lot of problem that are being reported that could be have
been discovered with better testing of the library.

Testing the blockchain property is an incredibly complex task that may
take a lot of CI time. In order to facilitate writing the testing and
not make the developer life too hard we will split the testing into
different layer of levels.

Level1

we need to be able to work on the small properties.

• Encoding then Decoding the blockchain should returns the original content;
• Addresses Discrimination should be the same as ledger (#16)
• Simple property of the different components:
  • transaction (inputs == outputs + fee) (#728 )
  • the blockchain state's total value is constant (#729);
  • when building the stake distribution we need to retrieve the total value;
  • for a total of N value in the blockchain, a stake key with M (M < N),
    the stake distribution should be consistent.

Most of these tests can be property tests (utilising quickcheck). This is important
as proving these properties if hold independently, the higher composition of these
properties will be easier to test.

Level 2

These tests are still property tests, but may take a bit longer to execute as they
are testing small scenarios:

• when applying a transaction to the blockchain the state needs to change
  • the total value remains the same;
  • the spent utxo are removed, a new one is added;
  • the accounts state are updated; #45
  • the stake distribution has changed;
• we need more thorough testing on the update mechanism too;

Level 3

theses are slightly more complex tests that may takes some time to execute on the
CI. They intent to test long running properties:

• leader election through an epoch is on average consistent with the stake distribution.
  if a leader as 30% of the total stake, they should be elected 30% of the time; #41
• update and voting, we need to check that changes on the blockchain settings actually
  changes the settings when they are voted and adopted. #49

This is needed for input-output-hk/jormungandr#348.

Since we only need the ability to restore a single state (namely the current tip minus some number of blocks), we probably don't need any kind of fancy delta storage (like we had for cardano::ChainState).

The current storage implementation has issues with consistency, including:

• It is possible to put a block while its parent is not present in the database.
• It is possible to put a tag on a non-existent header.

Consider adding checks to the database queries to prevent inconsistency in an atomic way.

add specialisation fee for the vote plan certificate

Dependabot couldn't parse the Cargo.toml found at /chain-impl-mockchain/Cargo.toml.

The error Dependabot encountered was:

Dependabot::DependencyFileNotParseable

View the update logs.

https://github.com/input-output-hk/chain-libs/blob/master/chain-impl-mockchain/src/ledger/ledger.rs#L484

add the necessary auth parameter for the vote plan certificate so we can create registration certificate, sign them and propagate them within transactions

Now there are two different implementations of bitmaps:

• One in imhamt (32 bits long)
• Another one in sparse-array (256 bits long)

We need to implement a bitmap that will satisfy different corner-cases (8, 16, ... 256 bits) and be reusable so that we can remove functionality duplication.

Before and after finalize operation:

• each add input/ add output / add certificates changes transaction Id
• After finalize it does not change

You can seal only transaction with the sufficient amount of witnesses and corresponding types (does order of witnesses matter?)

We used to run with the log crate. But we have a requirement for structural logging coming from different end users. log does not provide this feature yet. For simplicity we will be using slog-scope crate (and derivative).

There is an issue in code:

https://github.com/input-output-hk/chain-libs/blob/master/chain-impl-mockchain/src/block/leaderlog.rs#L41

It first increment total and then check if insert is correct which can leads to wrong counter values for participants record. See test:

 #[test]
    pub fn test_set_new_value_for_existing_stake_pool() {
        let stake_pool_id = new_stake_pool_id();

        let mut leaders_participation_record = create_log(vec![(stake_pool_id.clone(), 1)]);

        assert_eq!(
            InsertError::EntryExists,
            leaders_participation_record
                .set_for(stake_pool_id.clone(), 10)
                .err()
                .unwrap()
        );

        assert_eq!(
            1,
            get_count(&leaders_participation_record, &stake_pool_id).unwrap(),
            "wrong count for new stake pool"
        );

        verify_participants_count(&leaders_participation_record, 1);
        //correct expected value: 1
        verify_total(&leaders_participation_record, 11);
    }

in PR: #223

• get rid of chain-core (some of the) properties in storage (optional)
• remove sqlite indices and replacing by in-memory indices in btree-map (BlockHash -> ..) and Sequential / Geographical data structure (data height, date)
• remove the storage trait, in favor of a simple implementation with possibly only 3 backends (dynamic branching would do) that we care about for storing blocks : memory, sqlite, hdb (branch in chain-libs). it's possible that some backends could cache, e.g memory cache of blocks then other backend also.

Certificate might have different overhead on the chain, and thus might require more fees.

Introduce a certificate trait that allow to define extra fees depending on the certificate type and content, effectively Certificate -> ExtraFee

related to #427

general improvement to signing

• cleanup signing capability
• overhaul signing to be flexible to multi owners and different algorithms
• transaction binding mechanism
• document the security guarantees

Just like Header and Transaction, refactor Content / Fragment and add ContentSlice / FragmentSlice to be cheaper to extract / parse.

• core should be splitted in blockchain-traits (questionable need) and serialization (still needed)
• remove use of the traits in network and existing storage
• alternatively remove the trait as a dependency of chain-impl-mockchain and invert it to a package that consume the chain-impl-mockchain and add the trait to gradually get rid of things

track down the votes in the ledger's vote plan state manager

• a vote is valid is in a block within the allocated time for the vote
• no duplicate vote, new votes erase the previous vote

• contains a proposal ID: (Hash, u8), the Hash of the vote plan and the index (u8) of the specific proposal
• contains the vote choice (TODO: define the choice with the vote crypto)

a vote is always associated to an account so we can track down the stake too. Stake Delegation certificate should be a good example starting point for this

version: newest master

method: assign_account_value in stake/distribution.rs file. If we have a DelegationType::Ratio which delegates e.g. Value(100) to non existing 4 pool stakes it is expected that StakeDistribution struct will hold dangling value equal to 100, however it is equal to 800 (this is due to the face that
assign_account_value mutliply incoming value by 2 and does it 4 times (4 is the number of pools)).

Please run test: dangling_stake_multiplied in stake/distribution.rs to observe mentioned behavior (it is possible to uncomment commented assertion to get proper expected value).

leader election through an epoch is on average consistent with the stake distribution.
if a leader as 30% of the total stake, they should be elected 30% of the time;

In the module mempack of chain-core, ReadBuf looks like a statically lifetime-bound analog of Bytes from crate bytes. Switching the Readable/Writable traits to working with Bytes/BytesMut may enable more flexible zero-copy processing of asynchronously streamable data in the style of tokio-io.

Dependabot couldn't parse the Cargo.toml found at /chain-core/Cargo.toml.

The error Dependabot encountered was:

Dependabot::DependencyFileNotParseable

View the update logs.

The method DigestOf::digest_with is arguably too loose for library API: it allows any closure to calculate a digest of an object, which is however, tied to the object type by its type parameter.

To reduce possibility of sneaky errors, this method should be replaced by a different method using a new trait similar to std::hash::Hash, which object types would need to implement to support calculation of a digest on their contents in a single canonical way.

DigestOf::coerce suffers from similar looseness, but at least its intent is explicit in the code. Its only usage so far is in a quickcheck Arbitrary impl for DigestOf, which is rather hackish.

All dependencies of tower-grpc in its git master branch have graduated to be versioned on crates.io. It can be used with a crates.io version of tower-hyper as the HTTP stack. This means we can achieve build stability with revision-locked tower-grpc, but the HTTP stack crate needs to be switched from tower-h2 to tower-hyper.

Things to do

Blockers in tower-hyper

• Introduce client::Background (tower-rs/tower-hyper#46)
• Make client::ConnectFuture public (tower-rs/tower-hyper#48)
• Make client::ResponseFuture public (tower-rs/tower-hyper#47)
• Wait for a crates.io release incorporating the above changes.

network-grpc

• Finalize the porting branch after tower-hyper issues have been resoved and submit a PR.

In many immutable places, we keep Vec<T> instead of using Box<[T]> which save 8 bytes and also represent immutability better than Vec.

Dependabot couldn't parse the Cargo.toml found at /chain-core/Cargo.toml.

The error Dependabot encountered was:

Dependabot::DependencyFileNotParseable

View the update logs.

the uniqueness property of utxo is hard to satisfy when the component are just accounts, since the nonce is associated with the witness only.

We need to be able to check that, for any new transaction applied to the state, the total value remains constant.

This task is a bit more complex as you need to create only valid transaction for a given state of the blockchain. but still possible:

• for an arbitrary state of the blockchain (random UTxO, accounts, ledger parameters...);
• for an arbitrary valid transaction;

apply the transaction on the blockchain state;

• the total value should not have changed

relates to epic #727

property test which verifies that changes on the blockchain settings actually
changes the settings when they are voted and adopted.

and it is not checked from decoding the value from the config param either because we use Milli:

At the moment the unspent is implemented using a BtreeMap:

struct TransactionUnspents<OutAddress>(BTreeMap<TransactionIndex, Output<OutAddress>>);

whereas it could do with an optimised contiguous implementation with a bitmap and special cases (likely 1 index, 2 indices, 3 indices)

track down the different active vote plans and their different timelines within the ledger

need to restructure the ledger checks / applications to separate some parts, which would give the ability to do more check on fragments ahead of application.

Errors need to splitted in two categories:

• ErrorStateless: the fragment is not valid and never will be : e.g. structure is invalid, some cryptographic signature return Failed, ...
• ErrorLedgerState: the fragment is not currently valid : it doesn't apply to the current ledger state (e.g. account doesn't have enough value), but could validly apply to a different ledger state.

Operations need to separate the stateless checks on a fragment from the stateful checks

Stateless:

• Structural/Consistency checks
• Limits
• Hash calculation
• Cryptographic signatures 1/2 (public key available)

Stateful:

• Cryptographic signatures 2/2 (public key need to be retrieved from state)
• Application to the state (e.g. spending money, creating a new entry)

strongly related to #7

When using max_limit parameter along with fixed tax, reward mechanism does not take max_limit parameter value into account.

Example:
Given stake pool registration:

StakePool { ..., 
     rewards: TaxType { 
          fixed: Value(100), 
          ratio: Ratio { numerator: 0, denominator: 1 }, 
          max_limit: Some(30) 
     }, ....
 }

Actual reward: 100
Expected reward: 30

• mutable mode should kept the tree mutable and modify the tree on demand (Cow)
• freezing: should replace all mutated cells, back to frozen nodes
• replace recursive functions by explicit iterative functions
• optimise node internals (unsafe / pointers / ..)

the BlockStore method name is_ancestor is misleading. Ok(None) is returned when there are no ancestral relations.

Perhaps the method should be renamed and a purpose-specific enum can be introduced for self-descriptive explanation of the success cases.

Separate the ledger checks for transaction (w or w/ certificate) that have to do with self-validation (making sure the format is ok, and that basic property is enforced correctly), and the context based validation.

e.g.:

X.valid_self() -> Result<(), Error>
X.apply_ledger(ledger: &Ledger) -> Result<Ledger, Error>

this is useful when wanting to make simple verification on transaction without having the Ledger available.

for epic input-output-hk/jormungandr#1972

• contains an array of up to 255 proposals;
• starts being active at the next epoch;
• has a voting length
• has a committee length

The intent is to optionally be able to bring an old blockchain latest hash as data of the chain. This can be thought as a logical block0.prev.

One alternative option, that I don't think make sense, is to use the official block0.prev which is right now set to 0, but it's only working if/when we have the same hash size.

• when missing setting should be maybe set to block0 hash instead of ::zero()
• allow to override set a 32 bytes value from block0 config param

epic input-output-hk/jormungandr#1972

we need a message and a subscription in order to allow a new class of gossip to be broadcasted through the network.

The content of the message is not specified yet, so a byte array is expected at the moment.

Allow the system to start at a defined offset epoch=N instead of assuming epoch=0, which need to de-hardcode couple of place that expect to start at date=0.0