gcarq / rusty-blockparser Goto Github PK

Hello. Ubuntu 16, 32gb ram, 500gb ssd, 80gb swap

[17:36:43] INFO - dispatch: Status: 519565 Blocks processed. (left: 8881, avg: 19 blocks/sec)
[17:36:53] INFO - dispatch: Status: 519577 Blocks processed. (left: 8869, avg: 19 blocks/sec)
[17:37:04] INFO - dispatch: Status: 519589 Blocks processed. (left: 8857, avg: 19 blocks/sec)
[17:37:14] INFO - dispatch: Status: 519602 Blocks processed. (left: 8844, avg: 19 blocks/sec)
Segmentation fault

Any ideas?

$ cargo build
warning: profile doc is deprecated and has no effect
Compiling cfg-if v0.1.6
Compiling unicode-width v0.1.5
Compiling rustc-serialize v0.3.24
Compiling strsim v0.6.0
Compiling bitflags v0.9.1
Compiling ansi_term v0.9.0
Compiling vec_map v0.8.1
Compiling byteorder v1.1.0
Compiling seek_bufread v1.2.2
Compiling log v0.4.6
Compiling num-traits v0.2.6
Compiling libc v0.2.47
Compiling log v0.3.9
Compiling term_size v0.3.1
Compiling rand v0.4.5
Compiling atty v0.2.11
Compiling time v0.1.42
Compiling textwrap v0.8.0
Compiling num-integer v0.1.39
Compiling num-complex v0.2.1
Compiling clap v2.26.2
Compiling rand v0.3.22
Compiling num-bigint v0.2.2
Compiling num-iter v0.1.37
Compiling rust-crypto v0.2.36
Compiling num-rational v0.2.1
Compiling num v0.2.0
Compiling rust-base58 v0.0.4
Compiling rusty-blockparser v0.6.1 (/home/chawla/rusty-blockparser)
warning: use of deprecated item 'std::env::home_dir': This function's behavior i s unexpected and probably not what you want. Consider using the home_dir functio n from https://crates.io/crates/dirs instead.
--> src/blockchain/utils/mod.rs:131:19
|
131 | PathBuf::from(env::home_dir().expect("Unable to get home path from env !"))
| ^^^^^^^^^^^^^
|
= note: #[warn(deprecated)] on by default

Finished dev [unoptimized + debuginfo] target(s) in 40.01s

Speed is decreasing from X to 0 and the process get killed at the second phase. How can I increase the debug level to 2. I can't put a option on flag -v ?

Hi, I'm getting this error after processing is done:

[08:04:07] ERROR - dispatch: Genesis hash for Bitcoin does not match:
Got: 00000000000091108032fce57099b7dad0cc42444e400f540db05c18a88d10b2
Exp: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f Validation Error

If I run:

./rusty-blockparser -n -d /mnt/c/Users/peter/AppData/Roaming/Bitcoin/blocks unspentcsvdump /home/peter/data

I get:

[08:03:38] INFO - dispatch: Status: 381134 Blocks added to index. (avg: 328 blocks/sec)
[08:03:49] INFO - dispatch: Status: 382405 Blocks added to index. (avg: 326 blocks/sec)
[08:04:00] INFO - dispatch: Status: 383670 Blocks added to index. (avg: 324 blocks/sec)
[08:04:05] INFO - dispatch: All threads finished.
[08:04:05] INFO - dispatch: Done. Processed 384321 blocks in 19.79 minutes. (avg: 323 blocks/sec)
[08:04:05] INFO - dispatch: Saving block headers as chain.json ...
[08:04:07] ERROR - dispatch: Genesis hash for Bitcoin does not match:
Got: 00000000000091108032fce57099b7dad0cc42444e400f540db05c18a88d10b2
Exp: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f Validation Error

My data directory looks like this with empty files:

peter@peter-THINK:~/data$ ls -la
total 0
drwxrwxrwx 0 peter peter 512 Jan 7 17:59 .
drwxr-xr-x 0 peter peter 512 Jan 7 18:41 ..
-rw-rw-rw- 1 peter peter 0 Jan 7 17:59 blocks.csv.tmp
-rw-rw-rw- 1 peter peter 0 Jan 7 17:59 transactions.csv.tmp
-rw-rw-rw- 1 peter peter 0 Jan 7 17:59 tx_in.csv.tmp
-rw-rw-rw- 1 peter peter 0 Jan 7 17:59 tx_out.csv.tmp
-rw-rw-rw- 1 peter peter 0 Jan 15 07:44 unspent.csv.tmp

Any ideas? I just downloaded a fresh copy of bitcoin core over the weekend, and re-downloaded the entire blockchain. Thanks!

Hi,

It is possible to add skeincoin?
https://github.com/skeincoin/skeincoin

Hello,

Im using your program to parse the bitcoin blockchain, everything goes smooth until i get this error. Im not able to identify which is the exact block causing the problem, but probably is located between blk00308.dat and blk00311.dat

[15:44:25] INFO - dispatch: Status: 367219 Blocks added to index. (avg: 307 blocks/sec)
[15:44:25] DEBUG - worker-3: Got 0x00000000 as magic number. Finished.
[15:44:25] DEBUG - dispatch: worker-3 completed
[15:44:25] ERROR - worker-2: Got invalid magic value for Bitcoin: 0x36b2e95, expected: 0xd9b4bef9 Validation Error
[15:44:25] ERROR - dispatch: Got invalid magic value for Bitcoin: 0x36b2e95, expected: 0xd9b4bef9 Validation Error

Is it a program configuration related error or is my blockchain corrupted?

Any help would be greatly appreciated!

And thanks for your great program!

All txid's output in the unspentcsvdump are missing a character, due to this mistake:
let txid = &key[0..63];

Instead it should be:
let txid = &key[0..64];

Hi there,

first of all, good work, this is what I have been after for some time,

I wonder what is a lockTime for given transaction

is that possible to make an option to dump all address with unspent balances like another aged blockparser did?

Hi gcarq,

I would like to verify with you how compatible rusty-blockparser currently is with segwit blocks. It looks like required modifications should be minimal, if any.

If you need any help in making this happen, please let me know :) .

Just wondering if there is a way to resume without indexing or any idea for optimizing the indexing ?
Great tool BTW !

The bitcoin blockchain is about 145GB. I was just wondering how much space I need to have in addition to that.

[seems to be a duplicate of #13 - the issue is still actual]

Hi again ;)

I managed to start processing callbacks (and dumping) on the different hardware.

The issue with the program is that it seems to do no memory management and dies as soon as it consumes all available memory and all swap space. Linux process manager sends it a kill signal and it dies with 'Killed' message.

Looking at the performance (for "csvdump"), as soon as memory runs out and swap space is used, the performance degrades considerably (from 600% CPU load it goes to 20% ..etc - the I/O become a bottleneck).

Is there a way to release memory when dumping blocks? Does it have to keep such a large allocation (it got killed after consuming 32 GB RAM and 16 GB swap). When dumping to CSV, why does it need to keep so much data in memory, when what it does is basically doing a data passthru from block files to csv files?

Cheers
vZ

Hi! Thank you for work on this program, its cool. I found bug: in csvdump mode block height cell calculated as count processed blocks and ignoring --start option. Other block information correct,

Two first lines from file blocks-0-300000.csv:
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f;0;1;285;0000000000000000000000000000000000000000000000000000000000000000;4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b;1231006505;486604799;2083236893 00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048;1;1;215;000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f;0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098;1231469665;486604799;2573394689

Two first lines from file blocks-300000-400000.csv:
000000000000000082ccf8f1557c5d40b21edabb18d2d691cfbf87118bac7254;0;2;128810;000000000000000067ecc744b5ae34eebbde14d21ca4db51652e4d67e155f07e;915c887a2d9ec3f566a648bedcf4ed30d0988e22268cfe43ab5b0cf8638999d3;1399703554;419465580;222771801 000000000000000049a0914d83df36982c77ac1f65ade6a52bdced2ce312aba9;1;2;241334;000000000000000082ccf8f1557c5d40b21edabb18d2d691cfbf87118bac7254;7cbbf3148fe2407123ae248c2de7428fa067066baee245159bf4a37c37aa0aab;1399704683;419465580;3476871405

Version column (2nd) in file with blocks started from 300000 start counting from 0.

Hi.
Run with parametrs:
rusty-blockparser -s 655239 -e 655240 -d y:\blocks\ csvdump t:\CSV

ruturn files' names
tx_out-0-1.csv
transactions-0-1.csv
tx_in-0-1.csv
blocks-0-1.csv
can be: 655239-655240

into file: blocks-0-1.csv
00000000000000000004a1bbdf5771f90c7a3b94b573fbf3993f4eb47d45dfa0;0;536870912;1272426;0000000000000000000d6a44b46922ebf506f4fb1b9d7914ad3d02f9d5e24c62;e50b08133ad1d208043bea23e6313de79c530f21295c6fe1bbda5500745486bc;1604387714;386798414;237847184

where hash 00000000000000000004a1bbdf5771f90c7a3b94b573fbf3993f4eb47d45dfa0 - it block 655191!!!!!
https://www.blockchain.com/ru/btc/block/00000000000000000004a1bbdf5771f90c7a3b94b573fbf3993f4eb47d45dfa0

and into file: blocks-0-1.csv begin block 0!!!!!!!
can be 655239 and 655240

it's case bigin with 461247 block
hash - 0000000000000000003c6fe479122bfa4a9187493937af1734e1e5cd9f198ec7 - !!!!!!!!!!!!

Hi,

I'm using your parser and the sql schema for analyzing the blockchain. However, the queries I'm using are extremly slow.

For example the following one:
Select WEEK(from_unixtime(nTime)) as week,
YEAR(from_unixtime(nTime)) as year, Count(*) INTO OUTFILE '/var/lib/mysql/stealth.txt'
FROM (
Select ot.txid from tx_out ot where out.address LIKE '3%'
) ot, transaction t, blocks b WHERE
t.txid = ot.txid AND b.hash = t.hashBlock
Group BY year, week;

I'm using an Index on ot.txid, t.txid, hash, hashBlock and addresses. The database is deployed on an iMac with 8GB RAM of 2013, so the performance should be ok.

Are there any possible problems with joining binaries (txid, hash). Or are there other possibilities to speed up the queries?

Thanks in advance!!!
Jonas

Ubuntu 18.04, built from sources and installed from repo.
470597 blocks, then quit. There is 594248 blocks in the blockchain. Thoughts?

{ "chain": "main", "blocks": 594248, "headers": 594248, "bestblockhash": "0000000000000000000fe03dc34486f74b05bfa7803d5a3ea6d3c363cda37e60", "difficulty": 10771996663680.4, "mediantime": 1568151710, "verificationprogress": 0.9999961678274221, "initialblockdownload": false, "chainwork": "00000000000000000000000000000000000000000865da7e399526255f6520d3", "size_on_disk": 271704376342, "pruned": false,

[18:56:47] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:56:52] DEBUG - worker-0: Parsing blk01758.dat (132.95 Mb) [18:56:56] DEBUG - worker-0: Parsing blk01759.dat (133.64 Mb) [18:56:57] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:01] DEBUG - worker-0: Parsing blk01760.dat (133.41 Mb) [18:57:06] DEBUG - worker-0: Parsing blk01761.dat (134.14 Mb) [18:57:07] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:12] DEBUG - worker-0: Parsing blk01762.dat (134.06 Mb) [18:57:17] DEBUG - worker-0: Parsing blk01763.dat (133.97 Mb) [18:57:17] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:22] DEBUG - worker-0: Parsing blk01764.dat (133.70 Mb) [18:57:27] DEBUG - worker-0: Parsing blk01765.dat (133.34 Mb) [18:57:27] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:32] DEBUG - worker-0: Parsing blk01766.dat (132.97 Mb) [18:57:37] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:38] DEBUG - worker-0: Parsing blk01767.dat (133.78 Mb) [18:57:43] DEBUG - worker-0: Parsing blk01768.dat (133.53 Mb) [18:57:47] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:48] DEBUG - worker-0: Parsing blk01769.dat (133.09 Mb) [18:57:52] DEBUG - worker-0: Parsing blk01770.dat (133.04 Mb) [18:57:57] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:58] DEBUG - worker-0: Parsing blk01771.dat (134.05 Mb) [18:58:03] DEBUG - worker-0: Parsing blk01772.dat (133.63 Mb) [18:58:07] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:58:07] DEBUG - worker-0: Parsing blk01773.dat (133.70 Mb) [18:58:12] DEBUG - worker-0: Parsing blk01774.dat (132.92 Mb) [18:58:13] DEBUG - worker-0: Got 0x00000000 as magic number. Finished. [18:58:13] DEBUG - dispatch: worker-0 completed [18:58:13] INFO - dispatch: All threads finished. [18:58:13] INFO - dispatch: Done. Processed 470597 blocks in 125.37 minutes. (avg: 62 blocks/sec)
bitcoind version:
{ "version": 180000, "subversion": "/Satoshi:0.18.0/", "protocolversion": 70015, "localservices": "000000000000040d", "localrelay": true, "timeoffset": -1, "networkactive": true, "connections": 16,

[00:04:59] INFO - dispatch: All threads finished. [00:04:59] INFO - dispatch: Done. Processed 594248 blocks in 7.26 minutes. (avg: 1365 blocks/sec) [00:04:59] INFO - dispatch: Saving block headers as ./chain.json ... [00:05:00] DEBUG - chain: Longest chain: -> height: 470596 -> newest block: 00000000000000000081e7ef68029833ebf5db8c458948cc911cf6e2b8fa9a40 -> genesis block: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f [00:05:00] DEBUG - chain: Genesis hash is valid. [00:05:00] DEBUG - chain: Inserted 470597 new blocks ... [00:05:01] DEBUG - chain: Serialized 470597 hashes to ./chain.json. Latest processed block height: 0 ... (latest blk.dat index: 0) [00:05:01] DEBUG - main: Iteration 1 finished. [00:05:02] DEBUG - chain: Imported 470597 hashes from ./chain.json. Current block height: 0 ... (latest blk.dat index: 0)

Thoughts?

[18:37:50] INFO - dispatch: Status: 445643 Blocks processed. (left: 40221, avg: 104 blocks/sec)
[18:37:58] ERROR - worker-9: I/O Error: failed to fill whole buffer
[18:37:59] ERROR - worker-17: I/O Error: failed to fill whole buffer
[18:38:00] ERROR - worker-10: I/O Error: failed to fill whole buffer
[18:38:00] INFO - dispatch: Status: 445888 Blocks processed. (left: 39976, avg: 104 blocks/sec)

thread 'worker-7' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

Hello, Hello! the program gives an error. How to fix?

Hi,

are there any plans on adding support for parsing raw ethereum blockdata?

-tin

I observed that it only parses up to 365117 blocks. How i can parse up to the latest block?

For what reason I can get csv dump just up to a certain block (516957) on Mac OS 10.13.6 while I have a fully synced database?

[21:31:48] INFO - dispatch: Status: 520973 Blocks added to index. (avg: 1315 blocks/sec)
[21:31:58] INFO - dispatch: Status: 526319 Blocks added to index. (avg: 1296 blocks/sec)
[21:32:08] INFO - dispatch: Status: 531076 Blocks added to index. (avg: 1276 blocks/sec)
[21:32:19] INFO - dispatch: Status: 535735 Blocks added to index. (avg: 1254 blocks/sec)
[21:32:19] INFO - dispatch: All threads finished.
[21:32:19] INFO - dispatch: Done. Processed 536299 blocks in 7.13 minutes. (avg: 1255 blocks/sec)
[21:32:19] INFO - dispatch: Saving block headers as chain.json ...
[21:32:23] INFO - blkfile: Reading files from /Volumes/Bitcoin/blocks ...
[21:32:28] INFO - rusty_blockparser: All 516957 known blocks are processed!

I get the same result even if I start the process from the scratch (without .json file).

I was unable to successfully parse TestNet3 block data using the TestNet3 flag.

I used the following command to run on TestNet block data:
./rusty-blockparser -t 1 -c testnet3 --blockchain-dir <dir> csvdump ... <dir>

With the following error message:

[19:17:28] INFO - main: Starting rusty-blockparser v0.5.0 ...
[19:17:28] INFO - blkfile: Reading files from (dir) ...
[19:17:28] INFO - parser: Parsing TestNet3 blockchain ...
[19:17:28] INFO - parser: Parsing with mode HeaderOnly (first run).
[19:17:28] ERROR - worker-0: Got invalid magic value for TestNet3: 0xd9b4bef9, expected: 0x709110b Validation Error
[19:17:28] ERROR - dispatch: Got invalid magic value for TestNet3: 0xd9b4bef9, expected: 0x709110b Validation Error

I was able to successfully parse the testnet data using the -c bitcoin flag.

I am unable to run the parser on the current Litecoin blockchain. It finds the data (showing the total blocks) but never processes any. Other coins seem to work fine, its just Litecoin that fails like this.

~/rusty-blockparser/target/release$ ./rusty-blockparser -t 3 --blockchain-dir /litecoindata/blocks/ --coin litecoin csvdump /litecoindata/csv
[13:01:49] INFO - main: Starting rusty-blockparser v0.6.1 ...
[13:01:56] INFO - blkfile: Reading files from /litecoindata/blocks/ ...
[13:01:56] INFO - parser: Parsing Litecoin blockchain ...
[13:01:56] INFO - parser: Parsing 2053526 blocks with mode FullData.
[13:01:56] INFO - callback: Using csvdump with dump folder: /litecoindata/csv ...
[13:02:35] INFO - dispatch: Status: 0 Blocks processed. (left: 2053526, avg: 0 blocks/sec)
[13:02:45] INFO - dispatch: Status: 0 Blocks processed. (left: 2053526, avg: 0 blocks/sec)
[13:02:55] INFO - dispatch: Status: 0 Blocks processed. (left: 2053526, avg: 0 blocks/sec)

Blank address are dumped during a csvdump callback when vout type is witness_v0_keyhash.
For example txid : "f1b2caa3153aa0c37dcfcdff69214911512804c55ee162ba85cc65b1597903a2"
Is there an easy fix for this?

Hi!

Unfortunately, after running 'rusty-blockparser -r -t 6 unspentcsvdump' (or full 'csvdump') process finishes with empty, zero-length files.

Looks like second step of 'FullData' scan never starts?

-rw-r--r-- 1 root root 0 Oct 25 19:31 unspent.csv.tmp

...
[20:18:50] INFO - dispatch: Status: 464264 Blocks added to index. (avg:   162 blocks/sec)
[20:19:00] INFO - dispatch: Status: 464980 Blocks added to index. (avg:   161 blocks/sec)
[20:19:12] INFO - dispatch: Status: 465810 Blocks added to index. (avg:   161 blocks/sec)
[20:19:24] INFO - dispatch: Status: 466332 Blocks added to index. (avg:   160 blocks/sec)
[20:19:34] INFO - dispatch: Status: 466848 Blocks added to index. (avg:   160 blocks/sec)
[20:19:45] INFO - dispatch: Status: 467262 Blocks added to index. (avg:   160 blocks/sec)
[20:19:56] INFO - dispatch: Status: 467706 Blocks added to index. (avg:   159 blocks/sec)
[20:20:09] INFO - dispatch: Status: 468259 Blocks added to index. (avg:   159 blocks/sec)
[20:20:19] INFO - dispatch: Status: 468561 Blocks added to index. (avg:   158 blocks/sec)
[20:20:31] INFO - dispatch: Status: 469178 Blocks added to index. (avg:   158 blocks/sec)
[20:20:42] INFO - dispatch: Status: 469483 Blocks added to index. (avg:   157 blocks/sec)
[20:20:52] INFO - dispatch: Status: 470132 Blocks added to index. (avg:   157 blocks/sec)
[20:21:02] INFO - dispatch: Status: 470404 Blocks added to index. (avg:   157 blocks/sec)
[20:21:13] INFO - dispatch: Status: 470995 Blocks added to index. (avg:   156 blocks/sec)
[20:21:25] INFO - dispatch: Status: 471268 Blocks added to index. (avg:   156 blocks/sec)
[20:21:35] INFO - dispatch: Status: 471745 Blocks added to index. (avg:   155 blocks/sec)
[20:21:47] INFO - dispatch: Status: 472193 Blocks added to index. (avg:   155 blocks/sec)
[20:21:58] INFO - dispatch: Status: 472643 Blocks added to index. (avg:   154 blocks/sec)
[20:22:08] INFO - dispatch: Status: 473026 Blocks added to index. (avg:   154 blocks/sec)
[20:22:25] INFO - dispatch: Status: 473611 Blocks added to index. (avg:   153 blocks/sec)
[20:22:36] INFO - dispatch: Status: 474314 Blocks added to index. (avg:   153 blocks/sec)
[20:22:46] INFO - dispatch: Status: 474539 Blocks added to index. (avg:   153 blocks/sec)
[20:22:57] INFO - dispatch: Status: 475249 Blocks added to index. (avg:   152 blocks/sec)
[20:23:07] INFO - dispatch: Status: 475531 Blocks added to index. (avg:   152 blocks/sec)
[20:23:18] INFO - dispatch: Status: 475937 Blocks added to index. (avg:   151 blocks/sec)
[20:23:29] INFO - dispatch: Status: 476380 Blocks added to index. (avg:   151 blocks/sec)
[20:23:39] INFO - dispatch: Status: 476804 Blocks added to index. (avg:   151 blocks/sec)
[20:23:54] INFO - dispatch: Status: 477320 Blocks added to index. (avg:   150 blocks/sec)
[20:24:05] INFO - dispatch: Status: 477887 Blocks added to index. (avg:   150 blocks/sec)
[20:24:16] INFO - dispatch: Status: 478152 Blocks added to index. (avg:   149 blocks/sec)
[20:24:34] INFO - dispatch: Status: 478973 Blocks added to index. (avg:   149 blocks/sec)
[20:24:45] INFO - dispatch: Status: 479746 Blocks added to index. (avg:   149 blocks/sec)
[20:24:58] INFO - dispatch: Status: 479854 Blocks added to index. (avg:   148 blocks/sec)
[20:25:09] INFO - dispatch: Status: 480576 Blocks added to index. (avg:   148 blocks/sec)
[20:25:19] INFO - dispatch: Status: 480678 Blocks added to index. (avg:   147 blocks/sec)
[20:25:29] INFO - dispatch: Status: 481264 Blocks added to index. (avg:   147 blocks/sec)
[20:25:42] INFO - dispatch: Status: 481611 Blocks added to index. (avg:   147 blocks/sec)
[20:25:53] INFO - dispatch: Status: 482122 Blocks added to index. (avg:   146 blocks/sec)
[20:26:04] INFO - dispatch: Status: 482433 Blocks added to index. (avg:   146 blocks/sec)
[20:26:15] INFO - dispatch: Status: 482921 Blocks added to index. (avg:   145 blocks/sec)
[20:26:27] INFO - dispatch: Status: 483458 Blocks added to index. (avg:   145 blocks/sec)
[20:26:38] INFO - dispatch: Status: 484414 Blocks added to index. (avg:   145 blocks/sec)
[20:26:51] INFO - dispatch: Status: 485235 Blocks added to index. (avg:   145 blocks/sec)
[20:27:03] INFO - dispatch: Status: 485776 Blocks added to index. (avg:   144 blocks/sec)
[20:27:13] INFO - dispatch: Status: 486498 Blocks added to index. (avg:   144 blocks/sec)
[20:27:26] INFO - dispatch: Status: 486959 Blocks added to index. (avg:   144 blocks/sec)
[20:27:36] INFO - dispatch: Status: 487381 Blocks added to index. (avg:   143 blocks/sec)
[20:27:47] INFO - dispatch: Status: 487954 Blocks added to index. (avg:   143 blocks/sec)
[20:27:59] INFO - dispatch: Status: 488492 Blocks added to index. (avg:   143 blocks/sec)
[20:28:11] INFO - dispatch: Status: 488871 Blocks added to index. (avg:   142 blocks/sec)
[20:28:21] INFO - dispatch: Status: 489432 Blocks added to index. (avg:   142 blocks/sec)
[20:28:33] INFO - dispatch: Status: 489938 Blocks added to index. (avg:   142 blocks/sec)
[20:28:43] INFO - dispatch: Status: 490268 Blocks added to index. (avg:   141 blocks/sec)
[20:28:54] INFO - dispatch: Status: 490803 Blocks added to index. (avg:   141 blocks/sec)
[20:29:00] INFO - dispatch: All threads finished.
[20:29:00] INFO - dispatch: Done. Processed 491387 blocks in 57.90 minutes. (avg:   141 blocks/sec)
[20:29:00] INFO - dispatch: Saving block headers as chain.json ...
[20:29:03] INFO - blkfile: Reading files from /home/user/.bitcoin/blocks ...
[20:29:03] INFO - rusty_blockparser: All 491386 known blocks are processed! Try again with `-resume` to scan for new blocks, or force a full rescan with `--reindex`

That's how it ends.

Any clues?

Thanks!

It would be nice to specify a rough estimate of the output data.
I set 400Gb, but it looks like it's not enough. Do you have any estimations?

Howdy,

In reading the README, I gathered that the RAM footprint would easily fit within my VMs resources (12 proc, 2TB storage, 48GB RAM), but I found that in running the example 'blockparser -t 3 csvdump' the system got to the brink and the kernel ended up reaping the process before it panic'ed. I've not investigated the code yet, but it was my understanding that most of the processing (after the chain.json is loaded in RAM) is done via disk in/outs.

Oh, one thing that I may have polluted things with is that I used 8 thread workers (-t 8).

Thanks for any quick guidance on where I should look!

Best,
-ME

No matter what I do, there's no way to get the "balances" option to process data from the blockchain at more than 24 MegaBYTES per second.

I still know the process itself is single-threaded, and by monitoring the CPU, it is clearly not a CPU bottleneck, even tough it's using around 140% of CPU on a 8 core-16 threads Ryzen 7 3700x (shouldn't it be using up to 100% since it's single threaded?)

Memory for sure is not a bottleneck ...

The next one would be the disk.
I have the whole blockchain on an SSD than can do 532 MBPS in read operations when reading sequentially 16 bits blocks in 8 queues on 1 thread ( as measured by CrystealDiskMark 7 ) ... worse case scenario, if the blockparser is jumping around files non sequentially, the SSD can do read 37 MBPS at a Queue Depth of 1, 1 thread for 4KiB blocks.

My thinking is that the software may be not taking full advantage of doing more sequential reads from the disk?

Also, is the software taking any advantage of having the blockchain indexed ? ( TXINDEX = 1 in bitcoin.conf )

If I would be using python, I would run a profiler to see where cycles are being spent or lost, and try to optimize that a little? Maybe run some subroutines in C++ closer to the hardware?

I'm only interested in the last 1 years worth of blocks, is there a way I can specify the block height? It seems that for a lot of people the parser is crashing when working with the entire blockchain.

Hi! I get this result. Someone can explain the problem

[17:38:37] INFO - dispatch: Saving block headers as chain.json ...
thread 'main' panicked at 'assertion failed: best_height > 0', src/blockchain/parser/chain.rs:228:9

Thanks

Hello,
Can we get the clustered addresses using this tool? Does it have that feature in it?
Best Regards

Hello. Dogecoin parsing error. Ubuntu 16. Please help.

[06:16:43] INFO - dispatch: Status: 558752 Blocks processed. (left: 1464140, avg:   401 blocks/sec)
[06:16:53] INFO - dispatch: Status: 756529 Blocks processed. (left: 1266363, avg:   539 blocks/sec)
[06:17:03] INFO - dispatch: Status: 1121588 Blocks processed. (left: 901304, avg:   794 blocks/sec)
[06:17:13] INFO - dispatch: Status: 1623221 Blocks processed. (left: 399671, avg:  1142 blocks/sec)
[06:17:19] ERROR - worker-0: I/O Error: failed to fill whole buffer
[06:17:19] ERROR - dispatch: I/O Error: failed to fill whole buffer

This tool is great and worked without issues till around a couple of weeks ago.
When I use the option unspentcsvdump, I've noticed it doesn't parse beyond block 476592. In the initital blockchain scan it does go through the whole chain up to the current block, 479648 atm. But when it starts parsing it only goes up to block 476592.
I've tested this on a virtualbox VM and physical machine running ubuntu. Built the current version of blockparser. Tried force re-index and resume options.
What I see is that the temporary ChainStorage file chain.json doesn't seem to grow beyond 50MB in size. Could this be the cause? And if so, is this easy to fix in the code? I don't speak Rust and I looked through most of the src files but couldn't really find a size-limit somewhere.
Thanks for any help,
Robert

I've noticed that when running the new "Balances" callback, I get no bc1 address in the output file. Is that because you consider all bc1's as 3x's , or because you can't parse them yet?

Also, another question about Balances ... which is the rationale of how the addresses are sorted in the output file? It's definitively not by balance, and doesn't seem to either be by "last day seen or used".

Ubuntu 18.04
16GB Ram
1TB SSD
Running on 3 threads (-t 3)

Tried both rebuilding and reinstalling, latest version etc.
Parsing always stops at 534836 Blocks
Dispatch displays that it processes up to 549096 blocks yet at the time of writing block height is at 549142 Blocks (Approx. 50-60 blocks difference everytime)

[14:10:29] INFO - dispatch: Status: 548648 Blocks added to index. (avg: 1554 blocks/sec) [14:10:30] INFO - dispatch: All threads finished. [14:10:30] INFO - dispatch: Done. Processed 549096 blocks in 5.91 minutes. (avg: 1551 blocks/sec) [14:10:30] INFO - dispatch: Saving block headers as chain.json ... [14:10:34] INFO - blkfile: Reading files from /home/dirtymachine/.bitcoin/blocks ... [14:10:34] INFO - parser: Parsing Bitcoin blockchain ... [14:10:34] INFO - parser: Parsing 534836 blocks with mode FullData.

Hello, is there any way to extract the input amount from MultiSig transaction?
The example of a transaction I am trying to parse is the following:

https://blockchain.info/tx/3c1d7e82342158e4109df2e0b6348b6e84e403d8b4046d7007663ace63cddb23?show_adv=true

In here there are two inputs from the same address, and in the current implementation, I could not find a way to actually extract the input value information.

Thanks

Hi,

I believe there is a bug with the indexOut field in tx_out.csv.

Running: "./rusty-blockparser -t 4 csvdump csvdump/"

The indexOut listed in the .csv does not match up with the actual output index.

Multiple rows with the same txid have the same indexOut too, e.g. row#172 with txId f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16 has two outputs, both with indexOutput 1.

Some rows have an indexOut bigger than the number of outputs in the transaction. For example, on line #38948 in tx_out.csv:

7db18e491ffd40e2af82be0ea938c9c79e5cb9751f951b0c0fff29b9f4d1a000;7;5000000000;76a91461b66ebb35bf3bf51ed2daf510e4835a2844f89d88ac;19uf6F6EDijkH4ZUaqsi3pZ2SVD6A5RG8X

Looking this transaction up elsewhere shows it only has 1 input - not 7, so an output couldn't have an index of 7.

Am I misunderstanding what indexOut means, or is this actually a bug? I tried to find it but couldn't - not very familiar with Rust :)

Thanks in advance.

Great initiative with this software, congratulations!

I've just run the following to get the full unspent list of addresses as of yesterday:

rusty-blockparser unspentcsvdump .

Then to test the accuracy of the calculations, I've picked a set of random addresses and looked them up in a blockexplorer, and the balances for the accounts never match.

Am I mixing the "Unspent" definition that you use with the "Address Balance" definition I'm looking for? I want to have a list of all the currently used address that have a balance <> 0 , along with the balance itself. Is that possible?

Example, I got this result as one of the last records in the dump ( again, blockchain as of yesterday ) :

78fee78d871f2af02f3beaa348bae6680c488ec8e937ed99688e77a5a0aaed90;3;364963;1000;1rY7jaoyXUEcQ9scHEK4japsUf6eJ38QP

And that address currently holds 0.20+ BTC ( composed of many transactions UTXO's ) :

This other entry, which actually was the last one of the file, is correct, because it had just one TRANSACTION associated with it:
59caede5124946d243471b0a13661005ea5242d6924539faa73717c6e5154e99;0;277966;100000;1A5RJxRF21EaEauzcdah95Er71W9mzpyJm

And it's current balance is 0.001 BTC. ( )

I've seen other parsers that first do the actual parsing of the transactions UTXO's, and then run a bash command to add-up the transactions by address to get to a result... are you doing something like that ( or should?)

Thanks in advance!

Like several others I've been experiencing a gradual drop in performance eventually followed by a crash. This wouldn't be so bad if we could resume csvdump where it left off instead of having to restart every time. Has anyone figured out a way to do this?

I don't speak Rust at all and don't have the time to try to figure this out now, but if I do have some time to figure it out I'll share here

Hi,

Is it possible to parse for address balance or should one have to calculate balances after the parse ?

Thanks

I deterministically get on_block called with block_height 144 and immediately after 146 in all callbacks when starting from scratch. I have put some trace! to help with the debugging.

I have tried wiping the blockchain and redownload it completely, and the issue could still be reproduced.

[14:11:11] TRACE - callback: Block: 144.
[14:11:11] TRACE - callback: Block: Block {
    blk_index: 0,
    blk_offset: 190441,
    header: Hashed {
        hash: "000000007f47b4ef1a99f9a961199a5878852e7695ef07820de4b7fa45bda802",
        value: BlockHeader {
            version: 1,
            prev_hash: "0000000049ec23bf6bfc47b5c8560b15dd52fb57db15ef185d78d4af12871861",
            merkle_root: "70b8b6b9f6b4989ad63a19001c311c141dbb6ecdd060c56ae4f05bb507f95836",
            timestamp: 1231693144,
            bits: 486604799,
            nonce: 412551206
        }
    },
    tx_count: VarUint {
        value: 1,
        buf: [
            1
        ]
    }
}.
[14:11:11] TRACE - callback: Block: 146.
[14:11:11] TRACE - callback: Block: Block {
    blk_index: 0,
    blk_offset: 32724,
    header: Hashed {
        hash: "000000002e76e88e6ca526f324505fae2cc8245af31e92279f031a6019270512",
        value: BlockHeader {
            version: 1,
            prev_hash: "000000007f47b4ef1a99f9a961199a5878852e7695ef07820de4b7fa45bda802",
            merkle_root: "3f44efc23687edae461827db9545731ec7232b1d681f1562911a25c0159e3539",
            timestamp: 1231693700,
            bits: 486604799,
            nonce: 495243799
        }
    },
    tx_count: VarUint {
        value: 1,
        buf: [
            1
        ]
    }
}.

Hello,

I used unspentcsvdump and got this error :

[17:06:55] INFO - dispatch: Status: 367746 Blocks processed. (left: 228451, avg:   108 blocks/sec)
[17:07:06] INFO - dispatch: Status: 367785 Blocks processed. (left: 228412, avg:   107 blocks/sec)
[17:07:16] INFO - dispatch: Status: 367809 Blocks processed. (left: 228388, avg:   107 blocks/sec)
memory allocation of 2181038096 bytes failed[1]    1606 abort (core dumped)  rusty-blockparser

Which is 2GB Ram allocation. But my ram usage was stable until this block

Someone can upload an updated unspentcsvdump pls ?

Thanks a lot

I'm having trouble with the --resume flag.
I have run csv dump once like that:
rusty-blockparser -d /data/.bitcore/data/blocks --chain-storage chain.json -t 3 csvdump .
Once finished I did:
rusty-blockparser --resume -d /data/.bitcore/data/blocks --chain-storage chain.json -t 3 csvdump .

However, it seems the program start over the indexing process.

Thank you for your help.

This is a BIP :)

Change request would be to enable the user to resume from a certain block, so he/she doesn't have to process all the blockchain each time they want to get a new version of the "balances", with a blockchain that have grown.

This seems to be easily achievable by loading the previous balances.csv file, using the naming convention ( balances0..641023.csv ), the software can determine which was the last block that was ingested, then load all the balances into the hash table in memory, and start ingesting from the following one until the end, then dump all of the hash table back to disk.

Parser not found blockchain files

root@ubuntu-minimal:~/rusty-blockparser/target/release# ./rusty-blockparser -vv -c litecoin --chain-storage litecoin --blockchain-dir ~/.litecoin/blocks/ csvdump lite
[19:39:16] INFO - main: Starting rusty-blockparser v0.6.1 ...
[19:39:16] DEBUG - main: Using LogLevel TRACE
[19:39:16] INFO - blkfile: Reading files from /root/.litecoin/blocks/ ...
[19:39:16] TRACE - blkfile: Found 0 blk files
[19:39:16] ERROR - rusty_blockparser: Cannot load blockchain from: '/root/.litecoin/blocks/'. No blk files found! Runtime Error

With what command we can extract only hash160, address and balances?
Please let me know.
Thanks.

thanks. i was unable to figure it out

txid;indexOut;height;value;address
16162f8e21f6b184f60d793f091aec7773e5f6146b56061655304015e2350d2;;558965;1000000;3GbJNPBSWTjq4db6wUZNFyMBfnLYk7fNzj
92351f0ce2c65033efb8d1ad326296f2a2cefa726e3a17db7f23e98d4074237;;559900;0;
68ad83587fb2bb690630d5dcbd205c969f0fa7e737019b61844cc362ebc7ee6;6;559040;5100000;1APeB4CbJEbPit6zCfyPr7b5zRncQKys77
efbaaf612d142f2508749d0a16429ebd478ea4c778697a2e8510a4a78c45596;;559605;13571995;3CQuasFfeEzz38WjH384SrupExYrgoXRug
3d4e325e3782b02d3af0a381cc6b8ee69af51feabf084b3010bce57f7de11c3;;560495;1592228;3DNkb3dXJMwTkuGDPMmpJg47eZ8m5ZSLem
5a734f115f1a6432aa7b177753e956aa20fa526944fd387bfe25b7ee404741a;;559264;0;
9a915e478208ec050744e18b800671488120365b7cf5a17a3096b063f0d36a6;;559843;9980741;3LkoRFpzM6xDMkHXUCVM9wVNw5Kmp1Cj6o
51d6ce31abd4012cfea3b67c80e575915fb07250b2518c6ec924511f0920382;;560132;0;
7255cdc2b0ca39af54d2473197446746557095cd423ab27ed55038e250cfa34;;560183;917877;12hp65EJtxriF69tR6yZDZqQP7jYTNzKNw
f22ce9d611e1e986ccdde36b8232bc25ad1a2b03e13e4e4771a9bbfc64c42af;;559034;95641;37NSajsYqfVvJMrLetw9C8VmNF5WPg3uFn
80a22bb9bf548c629ca6aa92a95a4281e41367c18c4aca7307a43bfcce77c89;;559306;342639;14Ap88Hfa9Py3Wp8qxxdockr9MCqw9rHsZ
e7c428655be162a6aea7592379ff9b00f2d21e6061330f713f8c70b8c220d38;15;560032;400882;17WvxWpbHCw76DFUrrrWYSpv8MupfYAZim
01465f9c2d4369e77b623e379b28b0cba40a71302ed8b69bd7ba3320871db6c;;560349;546;1DURUZXNM7k3fboaHhoPfTDPSYhw4tZQqQ
b0381303b22b9e73cb07bc9e6dd6c8af7cdb25ffa00cc11a89c460eb2e8074e;;559015;28000000;33DJ3vYkrVjYV7BddiZuoivDKw68UqygfE
7db27874a01bfbd056961693439ec9c9ce18137423ea0e15040857169e65f2c;;559372;0;
2cb0ad30b6e15331e8896e467e25ab6212ab4d4fa73a5469dfbaba0e7ad2c43;;560051;1300000;1N3DrFuem5VXHwRRyn17MYhfQgZJy1RfaX
dd34e2b8695657143a24f793f04c36e46b2787ef30a5623f61ec46458885656;2;560126;89569;3F1XMfL8LCQy397c9e4Z7ypbR6xnf48fjm
a089d11f62f7ef6dcbe2c0d8726b84c04c5c7854f26c51132f5ab3c851f9454;;559047;0;
7b693cc57fdac4fc512550bb6665fff821f5e18ac49be3db3815af2117de4af;;559397;37304671;1KcMRh8Jc12LjVC9eF3B1Yj2CHGNen5RBX
c9e13d5f86bc148958cefed5ba2861eba3a8f670cf8ca21ded039e9c5a00e3c;;559138;0;
7bcac9224c630b5c5319119c4cd7f2516b27a670df4c126e3e47b2f4c7e9942;;560128;0;
3912314e4f73327acb9a695f01c94494b42c089bcb58b37dde39ba612e388e0;;559114;10171;15MoKovANartYEGqz63bSqGfRbN87JQunh
99eda53f385f74f03c4612c069de50d62dc194c0b529f42a1587e06a172015e;;559723;0;
0c84ec205fe2515a7c6223ffd1639c72d3c97deda2756f4152c52537ddfef1b;;560511;3285269;3FQmtqhYP4FKpPnGPxjwoHJRCpSjbFpsoL
374f7dc5625c21fcf51f777f836eabeb9422212ec01b7c551aca1d12578eed2;;560510;0;
e8abfd226eb4a8c2ffe74715625419f7bcec23cdd9dbbf990338bd14de44ec3;;558983;0;
ed821acb385a732a0e008b0e53d8498380ae66070d2ad46773886b287b7dfae;;559558;0;
032cea70786a9a019ba101f46f73bc4ce19cbb6427c7be26a82b86079edb60f;;559433;0;
662f483f638cfe0f2f62e84540ae66736131d20090446b93d6698002d36d5e6;;559726;13833669;1n2KRyDfMW1k72Bsqwxqgb6bXy9S14KK7
4fc464283407d533c401d94e488b8b2f2375830e26aaa7bde40ad5c8af60854;;560453;0;

see above its txid last 64 char missing in complete file
and parser status

[03:37:22] INFO - dispatch: Status: 1310 Blocks processed. (left: 323, avg: 13 blocks/sec)
[03:37:31] INFO - dispatch: All threads finished.
[03:37:31] INFO - dispatch: Done. Processed 1633 blocks in 1.82 minutes. (avg: 14 blocks/sec)
[03:37:33] INFO - callback: Done.
Dumped all 560539 blocks:
-> transactions: 3387477
-> inputs: 6918230
-> outputs: 8220615
[03:37:33] INFO - dispatch: Saving block headers as chain.json ...
[03:37:35] INFO - main: Fin.

actuall file contain 2487357 txid and same total lines are 2487357
whats wrong with this parser ??????

I'm wondering about the following verification errors. I find it unlikely both of my blockchains for 2 different coins are corrupted, but I did manage to get dogecoin to parse properly. Are you numbers accurate for BTC and LTC, or any other ideas on what the problem might be?

bash-3.2$ ./litestats
[12:05:02] INFO - main: Starting rusty-blockparser v0.6.1 ...
[12:05:02] INFO - blkfile: Reading files from /Users/burton/Litecoin/blocks/ ...
[12:05:02] INFO - parser: Parsing Litecoin blockchain ...
[12:05:02] INFO - parser: Building blockchain index ...
[12:05:02] ERROR - worker-1: Got invalid magic value for Litecoin: 0xa36ba22d, expected: 0xdbb6c0fb Validation Error
[12:05:02] ERROR - dispatch: Got invalid magic value for Litecoin: 0xa36ba22d, expected: 0xdbb6c0fb Validation Error
bash-3.2$ cat litestats
./target/release/rusty-blockparser --resume -d ~/Litecoin/blocks/ -c litecoin simplestats
bash-3.2$ ./bitstats
[12:12:02] INFO - main: Starting rusty-blockparser v0.6.1 ...
[12:12:02] INFO - blkfile: Reading files from /Users/burton/Bitcoin/blocks/ ...
[12:12:02] INFO - parser: Parsing Bitcoin blockchain ...
[12:12:02] INFO - parser: Building blockchain index ...
[12:12:03] ERROR - worker-0: Got invalid magic value for Bitcoin: 0xac9dd85e, expected: 0xd9b4bef9 Validation Error
[12:12:03] ERROR - dispatch: Got invalid magic value for Bitcoin: 0xac9dd85e, expected: 0xd9b4bef9 Validation Error

Is there a way to import the csv files into a neo4j database?