granola-team / mina-indexer Goto Github PK

when running the regression testing suite, upon encountering a failure, the local file mina-indexer.sock is not removed. The user must then manually remove the file to allow the tests to pass

There have been on occasion instances where modifying the Cargo.toml broke hermeticity in the nix build

Add documentation to the indexer for code contribution and it's potentially GitHub templates

Consider the following entry in the genesis ledger:

{
    "pk": "B62qqDJCQsfDoHJvJCh1hgTpiVbmgBg8SbNKLMXsjuVsX5pxCELDyFk",
    "balance": "148837.2",
    "delegate": "B62qp8Vq6n4VHq1LUm9Wd5QKjpKb7umoZ2oU9gpJYuHNUc7t2HGhGUA",
    "timing": {
        "initial_minimum_balance": "148837.2",
        "cliff_time": "86400",
        "cliff_amount": "148837.2",
        "vesting_period": "1",
        "vesting_increment": "0"
}

Start an indexer with no blocks

target/release/mina-indexer server cli --initial-ledger ~/mainnet.json -s ~/.mina-indexer/blocks/empty

Query the account balance for the above public key

target/release/mina-indexer client account -p B62qqDJCQsfDoHJvJCh1hgTpiVbmgBg8SbNKLMXsjuVsX5pxCELDyFk
{
  pk:       B62qqDJCQsfDoHJvJCh1hgTpiVbmgBg8SbNKLMXsjuVsX5pxCELDyFk
  balance:  0
  nonce:    0
  delegate: B62qp8Vq6n4VHq1LUm9Wd5QKjpKb7umoZ2oU9gpJYuHNUc7t2HGhGUA
}

The balance should be 148837.2

Proof read and re-evaluate the structure of the README

Investigate file watcher error and improve handling

Lots of complicated code for a solved problem. Investigate current logging code documentation for file rotation.

AC:

• Have GitHub actions run the regression test suite

Create a CODEOWNERS file so we may add maintainers

Running the command

docker build .

fails with the following error

#0 80.20 error: failed to run custom build command for `librocksdb-sys v0.10.0+7.9.2`
#0 80.20 
#0 80.20 Caused by:
#0 80.20   process didn't exit successfully: `/usr/src/mina-indexer/target/release/build/librocksdb-sys-285bd6ad0168a261/build-script-build` (exit status: 101)
#0 80.20   --- stderr
#0 80.20   thread 'main' panicked at 'Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/bindgen-0.64.0/./lib.rs:2393:31
#0 80.20   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
#0 80.20 warning: build failed, waiting for other jobs to finish...
#0 80.29 error: failed to compile `mina-indexer v0.1.1 (/usr/src/mina-indexer)`, intermediate artifacts can be found at `/usr/src/mina-indexer/target`
------
Dockerfile:4
--------------------
   2 |     WORKDIR /usr/src/mina-indexer
   3 |     COPY . .
   4 | >>> RUN cargo install --path .
   5 |     
   6 |     FROM debian:bullseye-slim
--------------------
ERROR: failed to solve: process "/bin/sh -c cargo install --path ." did not complete successfully: exit code: 101

Problem:

The witness tree is highly tightly coupled to the persistent datastore.

Proposed Solution

Disentangle the witness tree logic from the database code. The goal for canonical chain discovery should be accept a stream of block added events and emit it's own lifecycle events. E.g. CanonicalChainExtended(state_hash), CanonicalChainReorg(height, Vec), etc.

Solution: Refactor each test cases to be it's own function and add some assert statements

Having a regression test suite ensures software quality, enables catching bugs early, supports CI/CD, etc.

Solution: Return the string representation for the public key and delegate.

{"public_key":{"version":1,"t":{"version":1,"t":{"x":[201,165,50,75,130,18,218,94,196,34,20,242,197,48,15,142,191,104,117,101,78,219,253,231,66,226,185,129,166,6,3,50],"is_odd":false}}},"balance":148837200000000,"nonce":0,"delegate":{"version":1,"t":{"version":1,"t":{"x":[169,86,193,110,107,79,82,0,135,202,60,98,133,10,246,24,130,232,167,213,231,130,99,216,1,197,212,231,72,229,224,32],"is_odd":false}}}}

Currently, we can create an indexer state from genesis with an optional ledger.
I propose we provide three ways to create an indexer state:
new(root_hash, genesis_ledger, blocks_path) create a state from genesis state hash and ledger
new_non_genesis(precomputed_block, ledger, blocks_path) create a state from a non-genesis precomputed block and ledger
new_testing(precomputed_block, ledger, blocks_path) create a state from a block, optional ledger, and optional block store. This would only be for use in tests.
With this, we could also remove the Option from the root_branch

The indexer isn't correctly initialized if the startup-blocks doesn't exist. The domain socket server will never get started meaning it never can accept commands.

rm -r $HOME/.mina-indexer/startup-blocks
./target/release/mina-indexer server cli \
    --initial-ledger tests/ledgers/mainnet-genesis.json \
    --is-genesis-ledger \
    --root-hash 3NKeMoncuHab5ScarV5ViyF16cJPT4taWNSaTLS64Dp67wuXigPZ

2023-10-08T17:33:05.373775Z  INFO actix_server::builder: starting 10 workers
2023-10-08T17:33:05.374106Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2023-10-08T17:33:05.374124Z  INFO mina_indexer::server: Starting mina-indexer server
2023-10-08T17:33:05.374144Z  INFO mina_indexer::server: Initializing indexer state from blocks in /Users/tbernard/.mina-indexer/startup-blocks

Run the following query against the indexer and cross reference the data from the Archive Node.

query MIP4 {
  transactions(limit: 10000, query: {canonical: true, OR: [{memo: "E4YVPwLUR2LrP9tSSi3fjw1svcZys1gJHrGvRefwVTCMbP2NQRqdW"}, {memo: "E4YVe5wRALCJJ2dGEwRMyH7Z8y1QxzM76M8rXivFo5XbeBJdKryV6"}, {memo: "E4YbUmaZjNgLgezBD3JzyGKuCn4iugZ5EcXT1JuNTudm5tT4MHvKz"}, {memo: "E4YbUmaZZqAoUdTZYvZkSmLjHfccTMbb5RnTQHixwRWq2YqLdLZyE"}], dateTime_gte: "2023-05-20T06:00:00Z", dateTime_lte: "2023-05-28T06:00:00Z"}, sortBy: NONCE_DESC) {
    memo
    canonical
    from
    to
    hash
    blockHeight
    dateTime
  }
}

In order to unlock the full potential for the client server protocol, it should be changed to be texted based. Replace BCS serialization with JSON as the network payload. This enables us to use unix cli tooling like nc and jq

Add the necessary hooks and methods to cleanly shut down the Mina Indexer. Things to consider: Indexer state, rocksdb, graphql webserver, UDS/IPC socket files, etc.

Return ledgers (best tip, canonical best tip, etc.) as JSON to the mina-indexer client CLI.

The MinaIndexer object has access to a state variable that represents the current phase of the indexer: Intialization, Watching for Blocks, Opening an IPC connection, etc. However, this information is not available to the Indexer Client. Solution will add a new field to the SummaryVerbose and/or SummaryShort records so that this data is accesible to the client.

Things like executing docker builds, running regression suites, linters are all executed manually. Use just to create a better developer user experience.

AC:

• Add just support to nix
• Add docker build to justfile
• Add running unit tests to justfile
• Add linters to justfile
• Add regression suite to justfile

The Mina Indexer does not currently clean up its rocksdb journal after a test has executed, leading to multiple executions of the testing suite failing with a LOCK in the database dir

When running the save-state client command, the client always receives back an error, even when the snapshot is created successfully. Fix this so that the save-state command reports its success as well as a summary of its result

Add testing phase to flake.nix that runs cargo audit and cargo nextest run

Using mina-indexer --version mina-indexer 0.1.1

Trying to understand the process for importing blocks. As I understand the precomputed blocks have to be in a local directory, and that directory is passed via the --logs-dir flag. Is the naming/format of the blocks as is currently in Gcloud?

If so, can I import an arbitrary block for testing purposes, e.g., this block https://storage.googleapis.com/mina_network_block_data/mainnet-100001-3NLMfBr61BrTrZ86BKsJBUtWvRaGEyCBYTwywL5Ut7Jqye2qkfvv.json which is stored in the /home/gareth/logs/ directory.

I don't get any meaningful output when doing this, so I assume I am missing something.

=== Run ===
Starting mina-indexer with logs dir "/home/gareth/logs/"

Observability and debuggability into the indexer is lacking outside the GraphQL interface. Create tooling to inspect blocks and heights at specific heights and state hashes.

on the latest commit for main (05f5221), the regression tests do not pass on my local machine. the consistently failing test is test_best_canonical_tip which is failing with the error Error: invalid type: string "No summary available yet", expected struct SummaryShort at line 1 column 26

Remove the Google Cloud module from the codebase as it's unused. Also remove unused dependencies in Cargo.toml caused by removing the Google Cloud module.

when running the mina-indexer, one can pass in a flag canonical_threshold that should determine how many blocks after (confirmations) from the current Best Tip to consider as a Canonical block. Currently, the indexer does not respect this flag, and simply uses a constant CANONICAL_UPDATE_THRESHOLD defined in src/lib.rs. Fix will include removing this constant from all code excepting as a default in the clap command line parser.

Currently, the formatting for displaying account balances in the mina-indexer client uses a floating point representation. Change this to use NanoMina in any output that is designed to be consumed by another program, to prevent rounding errors.

Asana Tasks referenced:

• https://app.asana.com/0/1203669416123378/1205103570662676/f

Prefixing the socket with an @ causes the socket to be put in the abstract namespace which does not create a file. The regression suite assumes the UDS creates a file and manages it this way.

Solution: Remove the @ prefix and update any dependant code

Whenever one wants to add a new feature or fix a bug, one needs to write a regression test demonstrating the feature or bugfix working; however, when writing these tests, all other tests in the suite must be run in order before the new test can be ran. As the regression suite grows, this will bottleneck the development feedback loop's velocity. A fix would involve either caching passing tests, or allowing a developer to run only a subset from their local environment

The TransactionStore implementation does not use the common interface described by the BlockStore and CanonicityStore implementations. Fix will include removing TransactionKey struct and impl and creating a TransactionStore trait with an implementation in src/store.rs

This is slowing down the feedback loop. We should cache the compiled librocksdb or add the dependency to our nix flake.

A manual kill of the indexer's PID is required. This should be handled by ctrl-c.

Create a specification document that describes the mina-indexer client server protocol. Create ebnf grammar for the protocol.

The issue:

Asking the mina-indexer for account information after initialization on mainnet blocks results in a file error

Error: IO error: While open a file for random read: /mnt/indexer-block-storage/rocksdb/005690.sst: Too many open files

Steps to reproduce:

1. Initialize the mina-indexer on mainnet blocks
2. run mina indexer client account --public-key <public-key> in another terminal

Add the regression test suite to the github actions workflow

The IndexerState represents the data that the mina-indexer server needs in order to operate properly. Currently, the initialization for this state is highly coupled to the functionality that actually runs the indexer process.

Add a shutdown command to the mina-indexer client CLI API.

The mina-indexer server can be run using a YAML configuration file with the config subcommand, however, all paths present in that file must be absolute, and cannot use environment variables like $HOME to make the file more portable.

Provide documentation to the design of the indexer so that people can more easily understand how to contribute

This is the only file in the blocks dir: mainnet-2-3NLyWnjZqUECniE1q719CoLmes6WDQAod4vrTeLfN7XXJbHv6EHH.json

target/release/mina-indexer server cli -l ~/mainnet.json -s ~/.mina-indexer/blocks/borked
2023-09-13T12:44:03.309300Z  INFO actix_server::builder: starting 10 workers
2023-09-13T12:44:03.309309Z  INFO mina_indexer::server: Starting mina-indexer server
2023-09-13T12:44:03.309341Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2023-09-13T12:44:03.309377Z  INFO mina_indexer::server: Initializing indexer state from blocks in /Users/tbernard/.mina-indexer/blocks/borked
2023-09-13T12:44:03.361542Z  INFO mina_indexer::block::parser: Sorting startup blocks by length
2023-09-13T12:44:03.361559Z  INFO mina_indexer::block::parser: 1 blocks sorted by length in 167ns
2023-09-13T12:44:03.361569Z  INFO mina_indexer::block::parser: Searching for canonical chain in startup blocks
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', src/block/parser.rs:166:70
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace