dewi-alliance / helium-etl-lite Goto Github PK

Hi,

First of all, nice initiative / tool!

I'm having an issue getting it to run. I followed the quick start guide and blockchain-node seems to be syncing. Once I run helium-etl-lite I keep getting following error (ever after syncing for 24+ hours) ->

target/release/helium_etl_lite start

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NodeError("No transaction: <<\"Wh40ejvFhSY65XO5lo7sfZuIdNIUSpCC2wD4SjA3mPA\">>", -100)', /home/arnout/repos/helium-etl-lite/src/follower.rs:194:79
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1599:5
   3: helium_etl_lite::main::{{closure}}::run::{{closure}}
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: tokio::park::thread::CachedParkThread::block_on
   6: tokio::runtime::thread_pool::ThreadPool::block_on
   7: helium_etl_lite::main
note: Some details are omitted, run

Oct 17 15:46:47.139 INFO hello!
Oct 17 15:46:47.154 INFO no follower_info data
Oct 17 15:46:47.154 INFO Scanning blocks by epoch to get first block on node.
Oct 17 15:46:49.543 INFO Getting start_epoch from block 1048423
Oct 17 15:46:51.516 INFO Getting start_epoch from block 1048389
Oct 17 15:46:53.110 INFO Getting start_epoch from block 1048351
Oct 17 15:46:54.575 INFO Getting start_epoch from block 1048316
Oct 17 15:46:55.962 INFO Getting start_epoch from block 1048283
Oct 17 15:46:57.397 INFO Getting start_epoch from block 1048250
Oct 17 15:46:59.091 INFO Getting start_epoch from block 1048217
... some 60 more blocks until it panics

Do you maybe know what might be causing this / how to fix it?

Recently blockchain_node has been struggling to keep up with the chain when store_json is on (see issue helium/blockchain-node#121). This causes rewards transactions to return with only account-level data versus gateway-level, which is not ideal for the etl-lite. When this is happening, rewards come in with type reward_v2. We need to see a warning in the logs any time rewards are being loaded with type reward_v2, since this means we're not getting all the details we are expecting from blockchain_node.

This post is not really an issue with this project or its code. It is intendent as an inside in to my use case for it and obstacles I am dealing with. And also to help anyone else that wants to use it in same or similar way as I do. ( getting out challenges for my hotspots)

I was having problem when querying transactions table by witness address. Since that value is in array, inside jsonb it took long time.
Querying time for transaction table by address of challengee was ok, but got great when I added index on ((fields #> '{path,0}') ->> 'challengee'). I had no luck with trying to index witness gateway.

I hoped filters would also filter transactions table witch would reduce database size. It does not. I guess that is not what this etl was designed for, and I am ok with it. I still love this project!

Since I don't know rust to add filtering of transactions to this code, I added trigger to database that prevents inserting rows not containing data for my hotspots. It uses filter table but values must be gateway addresses.

here is the sql code for trigger:

 CREATE OR REPLACE FUNCTION filter_transactions()
                          RETURNS trigger AS
 $BODY$
 DECLARE
      i json;
      pass boolean := false;
      fieldsjson json := NEW.fields#>'{path,0}';
 BEGIN
      IF NEW.type::text LIKE 'poc_receipts_v1' THEN
           IF  (Select value from filters where value LIKE fieldsjson->>'challengee' ) IS NULL THEN
                FOR i IN SELECT * FROM json_array_elements(fieldsjson ->'witnesses')
                LOOP 
                     IF  (Select value from filters where value LIKE i->>'gateway' ) IS NOT NULL THEN
                          pass := true;
                     END IF;  
                END LOOP;
           else
                pass := true;
           END IF; 

      END IF;
      IF pass = false THEN
           RETURN NULL;
      else
           RETURN NEW;
      END IF;
      
 END;
 $BODY$
 LANGUAGE plpgsql;




 create trigger filter before insert on transactions
 FOR EACH ROW
 EXECUTE PROCEDURE filter_transactions();

I have tried on two different ubuntu 20.04 machines and on both I get same errors when building.

error[E0624]: associated function new` is private
--> src/settings.rs:52:27
|
52 | let mut con = Config::new();
| ^^^ private associated function
|
::: /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/config-0.12.0/src/config.rs:39:5
|
39 | pub(crate) fn new(value: Value) -> Self {
| --------------------------------------- private associated function defined here

error[E0061]: this function takes 1 argument but 0 arguments were supplied
--> src/settings.rs:52:19
|
52 | let mut con = Config::new();
| ^^^^^^^^^^^-- supplied 0 arguments
| |
| expected 1 argument
|
note: associated function defined here
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/config-0.12.0/src/config.rs:39:19
|
39 | pub(crate) fn new(value: Value) -> Self {
| ^^^

warning: use of deprecated associated function config::Config::merge: please use 'ConfigBuilder' instead
--> src/settings.rs:54:9
|
54 | con.merge(File::with_name(path.to_str().expect("file name")))?;
| ^^^^^
|
= note: #[warn(deprecated)] on by default

error[E0599]: no method named try_into found for struct config::Config in the current scope
--> src/settings.rs:55:9
|
55 | con.try_into().map_err(|e| e.into())
| ^^^^^^^^ method not found in config::Config
|
= help: items from traits can only be used if the trait is in scope
= note: 'std::convert::TryInto' is included in the prelude starting in Edition 2021
help: the following trait is implemented but not in scope; perhaps add a use for it:
|
1 | use std::convert::TryInto;
|

Some errors have detailed explanations: E0061, E0599, E0624.
For more information about an error, try rustc --explain E0061.
warning: helium-etl-lite (lib) generated 1 warning
error: could not compile helium-etl-lite due to 3 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
error: build failed
`

I get 'thread 'main' panicked at 'called Result::unwrap() on an Err value: Config(unsupported etl mode: "full")', src/main.rs:28:36'

Once the etl runs for a few weeks, the rewards table is getting really slow with queries that are super fast on DeWi ETL. Need to troubleshoot what is causing this and implement optimizations that will fix.

Not shure if i understand this correctly.

"Because of the way blockchain-node stores rewards info, the first ~300 blocks after the snapshot height won't incldue gateway or type information for specific rewards. All rewards with type = 'rewards_v2' are the total rewards paid to that account vs individual rewards that you will see being loaded into the rewards db after the first ~300 blocks after the snapshot height of the node."

After ~300 blocks it shuld start filling individual rewards in table rewards with gateway address included?
Beacause after 5000 blocks i still cant see it.

Currently, you have to manually clear tables if the etl is being reset (because of new snapshot or blockchain-node problems)

Currently the program crashes if it can't connect to the node. It should at least pause for 30 seconds and try again a couple times before crashing

currently the blockchain follower only accounts for when its own height is either:

A) The same as Node's height
B) Not the same as Node's height

It should respond differently when self.height is > Node vs when self.height is < Node.
Right now it assumes if self.height != Node, then self.height < Node height.

It needs to account for the scenario where it is ahead of node if, for example a new snapshot was loaded to Node that is older than self.height

When follower cannot get a block from the node ("Block not found"), follower should not increment to trying to get next block.

This tool probably needs to run using a db pool such as bb8
Need to test speed and performance using a db vs single connections. This will be part of the benchmark/optimization milestone of the project.

*mostly needed for Full mode

example:
https://github.com/djc/bb8/blob/main/postgres/examples/async_await.rs

I'm curious about using this library as a backend for a personal project with Helium that I'm hacking on in my free time. How much data (rough estimate) should I expect to need to make room for with a db hosting plan? Would it be possible to add this (rough) figure somewhere to the README? Or would it be changing too often to hard code it anywhere?

Thanks!