dewi-alliance / helium-etl-lite Goto Github PK
View Code? Open in Web Editor NEWA light ETL for the Helium blockchain
A light ETL for the Helium blockchain
Hi,
First of all, nice initiative / tool!
I'm having an issue getting it to run. I followed the quick start guide and blockchain-node seems to be syncing. Once I run helium-etl-lite I keep getting following error (ever after syncing for 24+ hours) ->
target/release/helium_etl_lite start
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NodeError("No transaction: <<\"Wh40ejvFhSY65XO5lo7sfZuIdNIUSpCC2wD4SjA3mPA\">>", -100)', /home/arnout/repos/helium-etl-lite/src/follower.rs:194:79
stack backtrace:
0: rust_begin_unwind
at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
1: core::panicking::panic_fmt
at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
2: core::result::unwrap_failed
at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1599:5
3: helium_etl_lite::main::{{closure}}::run::{{closure}}
4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
5: tokio::park::thread::CachedParkThread::block_on
6: tokio::runtime::thread_pool::ThreadPool::block_on
7: helium_etl_lite::main
note: Some details are omitted, run
Oct 17 15:46:47.139 INFO hello!
Oct 17 15:46:47.154 INFO no follower_info data
Oct 17 15:46:47.154 INFO Scanning blocks by epoch to get first block on node.
Oct 17 15:46:49.543 INFO Getting start_epoch from block 1048423
Oct 17 15:46:51.516 INFO Getting start_epoch from block 1048389
Oct 17 15:46:53.110 INFO Getting start_epoch from block 1048351
Oct 17 15:46:54.575 INFO Getting start_epoch from block 1048316
Oct 17 15:46:55.962 INFO Getting start_epoch from block 1048283
Oct 17 15:46:57.397 INFO Getting start_epoch from block 1048250
Oct 17 15:46:59.091 INFO Getting start_epoch from block 1048217
... some 60 more blocks until it panics
Do you maybe know what might be causing this / how to fix it?
Recently blockchain_node has been struggling to keep up with the chain when store_json
is on (see issue helium/blockchain-node#121). This causes rewards transactions to return with only account-level data versus gateway-level, which is not ideal for the etl-lite. When this is happening, rewards come in with type reward_v2
. We need to see a warning in the logs any time rewards are being loaded with type reward_v2
, since this means we're not getting all the details we are expecting from blockchain_node.
This post is not really an issue with this project or its code. It is intendent as an inside in to my use case for it and obstacles I am dealing with. And also to help anyone else that wants to use it in same or similar way as I do. ( getting out challenges for my hotspots)
I was having problem when querying transactions table by witness address. Since that value is in array, inside jsonb it took long time.
Querying time for transaction table by address of challengee was ok, but got great when I added index on ((fields #> '{path,0}') ->> 'challengee'). I had no luck with trying to index witness gateway.
I hoped filters would also filter transactions table witch would reduce database size. It does not. I guess that is not what this etl was designed for, and I am ok with it. I still love this project!
Since I don't know rust to add filtering of transactions to this code, I added trigger to database that prevents inserting rows not containing data for my hotspots. It uses filter table but values must be gateway addresses.
here is the sql code for trigger:
CREATE OR REPLACE FUNCTION filter_transactions()
RETURNS trigger AS
$BODY$
DECLARE
i json;
pass boolean := false;
fieldsjson json := NEW.fields#>'{path,0}';
BEGIN
IF NEW.type::text LIKE 'poc_receipts_v1' THEN
IF (Select value from filters where value LIKE fieldsjson->>'challengee' ) IS NULL THEN
FOR i IN SELECT * FROM json_array_elements(fieldsjson ->'witnesses')
LOOP
IF (Select value from filters where value LIKE i->>'gateway' ) IS NOT NULL THEN
pass := true;
END IF;
END LOOP;
else
pass := true;
END IF;
END IF;
IF pass = false THEN
RETURN NULL;
else
RETURN NEW;
END IF;
END;
$BODY$
LANGUAGE plpgsql;
create trigger filter before insert on transactions
FOR EACH ROW
EXECUTE PROCEDURE filter_transactions();
I have tried on two different ubuntu 20.04 machines and on both I get same errors when building.
error[E0624]: associated function
new` is private
--> src/settings.rs:52:27
|
52 | let mut con = Config::new();
| ^^^ private associated function
|
::: /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/config-0.12.0/src/config.rs:39:5
|
39 | pub(crate) fn new(value: Value) -> Self {
| --------------------------------------- private associated function defined here
error[E0061]: this function takes 1 argument but 0 arguments were supplied
--> src/settings.rs:52:19
|
52 | let mut con = Config::new();
| ^^^^^^^^^^^-- supplied 0 arguments
| |
| expected 1 argument
|
note: associated function defined here
--> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/config-0.12.0/src/config.rs:39:19
|
39 | pub(crate) fn new(value: Value) -> Self {
| ^^^
warning: use of deprecated associated function config::Config::merge
: please use 'ConfigBuilder' instead
--> src/settings.rs:54:9
|
54 | con.merge(File::with_name(path.to_str().expect("file name")))?;
| ^^^^^
|
= note: #[warn(deprecated)]
on by default
error[E0599]: no method named try_into
found for struct config::Config
in the current scope
--> src/settings.rs:55:9
|
55 | con.try_into().map_err(|e| e.into())
| ^^^^^^^^ method not found in config::Config
|
= help: items from traits can only be used if the trait is in scope
= note: 'std::convert::TryInto' is included in the prelude starting in Edition 2021
help: the following trait is implemented but not in scope; perhaps add a use
for it:
|
1 | use std::convert::TryInto;
|
Some errors have detailed explanations: E0061, E0599, E0624.
For more information about an error, try rustc --explain E0061
.
warning: helium-etl-lite
(lib) generated 1 warning
error: could not compile helium-etl-lite
due to 3 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
error: build failed
`
I get 'thread 'main' panicked at 'called Result::unwrap()
on an Err
value: Config(unsupported etl mode: "full")', src/main.rs:28:36'
Once the etl runs for a few weeks, the rewards table is getting really slow with queries that are super fast on DeWi ETL. Need to troubleshoot what is causing this and implement optimizations that will fix.
Not shure if i understand this correctly.
"Because of the way blockchain-node stores rewards info, the first ~300 blocks after the snapshot height won't incldue gateway or type information for specific rewards. All rewards with type = 'rewards_v2' are the total rewards paid to that account vs individual rewards that you will see being loaded into the rewards db after the first ~300 blocks after the snapshot height of the node."
After ~300 blocks it shuld start filling individual rewards in table rewards with gateway address included?
Beacause after 5000 blocks i still cant see it.
Currently, you have to manually clear tables if the etl is being reset (because of new snapshot or blockchain-node problems)
Currently the program crashes if it can't connect to the node. It should at least pause for 30 seconds and try again a couple times before crashing
currently the blockchain follower only accounts for when its own height is either:
A) The same as Node's height
B) Not the same as Node's height
It should respond differently when self.height is > Node vs when self.height is < Node.
Right now it assumes if self.height != Node, then self.height < Node height.
It needs to account for the scenario where it is ahead of node if, for example a new snapshot was loaded to Node that is older than self.height
When follower cannot get a block from the node ("Block not found"), follower should not increment to trying to get next block.
This tool probably needs to run using a db pool such as bb8
Need to test speed and performance using a db vs single connections. This will be part of the benchmark/optimization milestone of the project.
*mostly needed for Full mode
example:
https://github.com/djc/bb8/blob/main/postgres/examples/async_await.rs
I'm curious about using this library as a backend for a personal project with Helium that I'm hacking on in my free time. How much data (rough estimate) should I expect to need to make room for with a db hosting plan? Would it be possible to add this (rough) figure somewhere to the README? Or would it be changing too often to hard code it anywhere?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.