alecalve / python-bitcoin-blockchain-parser Goto Github PK

View Code? Open in Web Editor NEW

446.0 17.0 187.0 138 KB

A Python 3 Bitcoin blockchain parser

License: Other

Python 100.00%

python-bitcoin-blockchain-parser's People

Stargazers

Watchers

Forkers

ypschr domwoe aznisa07 chakhdin uzlider cpsola llamasoft priestc 64octets jbn agentbond007 kidol admiralsnackbar asutoshpalai gidsteckoll jamesvrt lishixin7 musamaahmad mkvtvseries dphang liaobs ddnn55 brangerbriz losh11 yashbhutwala baylism robdmc donscheme smartree sunshineben salisbury-espinosa carlosf xinqiyang jonlorusso paimpozhil hiddenvs zhengger 1henr1 scriptzteam ecrespo vincentbdb mickeystone roblaw113 themagicmountain tejaskhajanchee sschepis-zuul bryceweiner romali sbuxmmm behrouz-psh blt-soc puchtaw ori129 xvgpedro jamesyoungdigital blockparty-sh thenakulchawla nadahalli mhmystackpath lapolonio jcrawfordor rdbh daedalus nkwarrior84 dendisuhubdy mousewu nbarraille digger3d hzxgoforward cas3003p brainzhang-bitcoin thephez ishotoli hmy626 marcus-beasley schaeferj jameskbowler 5l1v3r1 manhdtran jayavanth dniprox abhishekshuklamath uncleguanghui jeanmonet enricozanardo 0xb10c frstrtr bilaljawed aketurits earce ivanofforks freemocker washgo pouyiouk stardust-xs cirosantilli qalander chrisguida kpaxer starryclc

python-bitcoin-blockchain-parser's Issues

bitcoin.core.script.CScriptTruncatedPushDataError: PUSHDATA(75): truncated data

Thanks for your project. But when I parse blk01400.dat, an error ocurred. How can I fix it?

Traceback (most recent call last):
  File "test_code.py", line 20, in <module>
    print("tx=%s outputno=%d type=%s value=%s" % (tx.hash, no, output.type, output.value))
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser-0.1.4-py3.5.egg/blockchain_parser/output.py", line 101, in type
    if self.is_pubkey():
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser-0.1.4-py3.5.egg/blockchain_parser/output.py", line 84, in is_pubkey
    return self.script.is_pubkey()
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser-0.1.4-py3.5.egg/blockchain_parser/script.py", line 102, in is_pubkey
    return len(self.operations) == 2 \
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser-0.1.4-py3.5.egg/blockchain_parser/script.py", line 73, in operations
    self._operations = list(self.script)
  File "/usr/local/lib/python3.5/dist-packages/bitcoin/core/script.py", line 628, in __iter__
    for (opcode, data, sop_idx) in self.raw_iter():
  File "/usr/local/lib/python3.5/dist-packages/bitcoin/core/script.py", line 613, in raw_iter
    raise CScriptTruncatedPushDataError('%s: truncated data' % pushdata_type, data)
bitcoin.core.script.CScriptTruncatedPushDataError: PUSHDATA(75): truncated data

Investigate using struct to improve parsing speed.

I find the way the DBBlockIndex parses the block header info using struct very interesting. Using it elsewhere in the repo might help improve perf as struct is written in C.

things about print from_public_key

I know that the from_public_key can be obtained from the inputs of transactions, the thing is that there is no definition about how to get the from_public_key. Would it possible to give me some suggestions on that? Thank you very much!

Improve speed of Transaction generation

Hi,

thank you for providing this nice blockchain parsing library!

I am using it to parse the whole blockchain and using debug prints and cProfile, I recognized that parsing one *.blk file roughly takes 40-50 seconds on my setup (I'm reading from an SSD and definitely have enough RAM). Of these 40-50 seconds, my program spends already 28 seconds in the __init__() function of blockchain_parser.Transaction.

Do you happen to know of any means to speed this up?
I also tried to use a C++ library before, which works considerably faster.

Thank you in advance for any response!

Transaction fee

I'm trying to calculate a transaction fee. How should I do it with this parser? Ty

ImportError: No module named core.script

Hi. Wile trying to run example from README I get following error.
How can I fix this?

# python test.py /root/.bitcoin/blocks/
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    from blockchain_parser.blockchain import Blockchain
  File "build/bdist.linux-x86_64/egg/blockchain_parser/blockchain.py", line 16, in <module>
  File "build/bdist.linux-x86_64/egg/blockchain_parser/block.py", line 12, in <module>
  File "build/bdist.linux-x86_64/egg/blockchain_parser/transaction.py", line 13, in <module>
  File "build/bdist.linux-x86_64/egg/blockchain_parser/input.py", line 13, in <module>
  File "build/bdist.linux-x86_64/egg/blockchain_parser/script.py", line 12, in <module>
ImportError: No module named core.script

Calling `addresses` on transaction with invalid script fails

e.g. for transaction ebc9fa1196a59e192352d76c0f6e73167046b9d37b8302b6bb6968dfd279b767 (the first that exhibits this behavior):

  File "./utxo.py", line 31, in <module>
    if output.addresses:
  File "/usr/local/lib/python3.6/site-packages/blockchain_parser-0.1.4-py3.6.egg/blockchain_parser/output.py", line 60, in addresses
    if self.type == "pubkey":
  File "/usr/local/lib/python3.6/site-packages/blockchain_parser-0.1.4-py3.6.egg/blockchain_parser/output.py", line 101, in type
    if self.is_pubkey():
  File "/usr/local/lib/python3.6/site-packages/blockchain_parser-0.1.4-py3.6.egg/blockchain_parser/output.py", line 84, in is_pubkey
    return self.script.is_pubkey()
  File "/usr/local/lib/python3.6/site-packages/blockchain_parser-0.1.4-py3.6.egg/blockchain_parser/script.py", line 103, in is_pubkey
    return len(self.operations) == 2 \
  File "/usr/local/lib/python3.6/site-packages/blockchain_parser-0.1.4-py3.6.egg/blockchain_parser/script.py", line 74, in operations
    self._operations = list(self.script)
  File "/usr/local/lib/python3.6/site-packages/python_bitcoinlib-0.5.0-py3.6.egg/bitcoin/core/script.py", line 622, in __iter__
    for (opcode, data, sop_idx) in self.raw_iter():
  File "/usr/local/lib/python3.6/site-packages/python_bitcoinlib-0.5.0-py3.6.egg/bitcoin/core/script.py", line 607, in raw_iter
    raise CScriptTruncatedPushDataError('%s: truncated data' % pushdata_type, data)
bitcoin.core.script.CScriptTruncatedPushDataError: PUSHDATA(1): truncated data

my workaround is to checkout if output.script.value == 'INVALID_SCRIPT' before accessing addresses.

get_ordered_blocks function is missing

This is the error I get when I try to parse the btc ordered blocks:

Traceback (most recent call last):
File "path.to.file", line 7, in
for block in blockchain.get_ordered_blocks(sys.argv[0] + '/Volumes/BTC_Blockch/blocks/index', end=1):
AttributeError: 'Blockchain' object has no attribute 'get_ordered_blocks'

Also, I had to change from:

sys.argv[1] to sys.argv[0]

because I was getting the following error:

Traceback (most recent call last):
File "path.to.file", line 5, in
blockchain = Blockchain(sys.argv[1])
IndexError: list index out of range

Your library cannot read blocks in blk00977.dat

Your library fails to read all the blocks in blk00977.dat.

I think every blocks after that such as blk00978.dat, blk00979.dat, ...

Could you check? Please.

Thank you.

Bitcoin split support

Hi,
Will the library support the bitcoin split?
Thanks

why there isn't a member variable `self.value` in class `Input`?

Hi,
Why there isn't a member variable self.value or self.satoshi in class Input to indicate how much money the input address (or previous trans) spent? Without this value I could not know the balance of an address. Is that me who make something wrong?
Thanks

How can I get the address from which a transaction originates?

Should input.py have an address?

Hello,

I was wondering if the input should have an associated address. Is there a way to obtain this information with this package?

Thanks,
Sam

TypeError: 'BlockHeader' object is not subscriptable when using get_main_chain()

I tried the the example program with get_main_chain() instead of get_unordered_blocks()

Code:

from blockchain_parser.blockchain import Blockchain
import sys

blockchain = Blockchain(sys.argv[1])
for block in blockchain.get_main_chain():
    for tx in block.transactions:
        for no, output in enumerate(tx.outputs):
            print("tx=%s outputno=%d type=%s value=%s" % (tx.hash, no, output.type, output.value))

Error:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    for block in blockchain.get_main_chain():
  File "/usr/local/lib/python3.4/dist-packages/blockchain_parser-0.0.1-py3.4.egg/blockchain_parser/blockchain.py", line 90, in get_main_chain
TypeError: 'BlockHeader' object is not subscriptable

Cache LevelDB index parsing

Blockchain.get_ordered_blocks(...) relies on Bitcoin's LevelDB database in ~/.bitcoin/blocks/index to produce ordered blocks. This is great, except parsing this LevelDB index can take a few minutes or longer depending on the user's hardware. When testing new code that relies on this function this can be a huge slowdown if you have to wait 10+ minutes each time you run your code.

@alecalve, how would you feel about adding optional params to get_ordered_blocks() to control saving/loading the parsed LevelDB indexes using pickle? I've got a version of this working locally and enabling it while I'm deving is a lifesaver. I wanted to bring it up here and discuss it before making a PR. Caching is always tricky business because if the user doesn't know that it's enabled/how it works it can lead to unexpected results and headache.

I'm imaging an interface like this:

def get_ordered_blocks(self, index, start=0, end=None, cache=None, cache_file='~/.ordered-blocks-index.pickle'):
#...

cache could take the following values:

None: caching is disabled and not used. This is the default behavior.
"save/load" (and "load/save"): Saves cache to cache_file if it doesn't already exists. Otherwise loads it.
delete: Delete cache_file.

Or, perhaps an even simpler interface is this:

def get_ordered_blocks(self, index, start=0, end=None, cache=None):
#...

Where cache can take on the following values:

None: Caching is disabled and not used.
"path/to/cache.pickle": The cache file to create if none exists or to load from if one does. It is then up to the user to manually delete this file to reload the index.

Of course, either of these solutions will require explicit documentation of this feature in the README to avoid some of the pitfalls that could come with not understanding how this caching system works.

I'm happy to take on this issue if we deem it sufficiently useful.

invalid literal for int() with base 10: '\x01'

I'm still downloading the blockchain, but wanted to try get_unordered_blocks with the blocks already downloaded.

I'm getting this exception (using the first example code from the readme):

  File "/home/tshabtay/Git/python-bitcoin-blockchain-parser/blockchain_parser/xxx.py", line 5, in <module>
    for tx in block.transactions:
  File "/usr/local/lib/python2.7/dist-packages/blockchain_parser-0.1.4-py2.7.egg/blockchain_parser/block.py", line 81, in transactions
    self._transactions = list(get_block_transactions(self.hex))
  File "/usr/local/lib/python2.7/dist-packages/blockchain_parser-0.1.4-py2.7.egg/blockchain_parser/block.py", line 26, in get_block_transactions
    n_transactions, offset = decode_varint(transaction_data)
  File "/usr/local/lib/python2.7/dist-packages/blockchain_parser-0.1.4-py2.7.egg/blockchain_parser/utils.py", line 44, in decode_varint
    size = int(data[0])
ValueError: invalid literal for int() with base 10: '\x01'

It's coming from the first block (blk00000.dat), and the data being passed to the decode_varint function is:

"\x01\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xffM\x04\xff\xff\x00\x1d\x01\x04EThe Times 03/Jan/2009 Chancellor on brink of second bailout for banks\xff\xff\xff\xff\x01\x00\xf2\x05*\x01\x00\x00\x00CA\x04g\x8a\xfd\xb0\xfeUH'\x19g\xf1\xa6q0\xb7\x10\\\xd6\xa8(\xe09\t\xa6yb\xe0\xea\x1fa\xde\xb6I\xf6\xbc?L\xef8\xc4\xf3U\x04\xe5\x1e\xc1\x12\xde\\8M\xf7\xba\x0b\x8dW\x8aLp+k\xf1\x1d_\xac\x00\x00\x00\x00"

self.hex from block.py is:

"\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00;\xa3\xed\xfdz{\x12\xb2z\xc7,>gv\x8fa\x7f\xc8\x1b\xc3\x88\x8aQ2:\x9f\xb8\xaaK\x1e^J)\xab_I\xff\xff\x00\x1d\x1d\xac+|\x01\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xffM\x04\xff\xff\x00\x1d\x01\x04EThe Times 03/Jan/2009 Chancellor on brink of second bailout for banks\xff\xff\xff\xff\x01\x00\xf2\x05*\x01\x00\x00\x00CA\x04g\x8a\xfd\xb0\xfeUH'\x19g\xf1\xa6q0\xb7\x10\\\xd6\xa8(\xe09\t\xa6yb\xe0\xea\x1fa\xde\xb6I\xf6\xbc?L\xef8\xc4\xf3U\x04\xe5\x1e\xc1\x12\xde\\8M\xf7\xba\x0b\x8dW\x8aLp+k\xf1\x1d_\xac\x00\x00\x00\x00"

What am I doing wrong?
Thanks.

how can i get list of all addresses having balance ?

i.e unspent outputs.

thanks

IOError on LOCK file when running bitcoind

When trying to run the parser while bitcoind is also running, I receive the following error:

IOError: b'IO error: lock /home/user/.bitcoin/blocks/index/LOCK: Resource temporarily unavailable'

Running the parser while the node has been shut down does work. I've mitigated this issue by copying /.bitcoin/blocks/ to another directory and referencing that with the parser, but I wonder if I'm missing something. May be related to Issue #43.

How can i get list of all addresses and hash160, having positive balance ?

Please let me know.

Indicate wheter an address uses compressed or uncompressed public key

Add a boolean property to the Address class indicating wheter an address uses compressed or uncompressed public key. If the public key for the address is unknown, an exception should be raised.

Setup.py fails on Ubuntu 16.04

Operating System

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

Leveldb:

$ sudo apt-get install libleveldb-dev
[sudo] password for parallels:
Reading package lists... Done
Building dependency tree
Reading state information... Done
libleveldb-dev is already the newest version (1.18-5).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.

running sudo python setup.py install fails with this backtrace:

Running plyvel-1.0.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-hA48Is/plyvel-1.0.4/egg-dist-tmp-1ZRxxS
warning: no files found matching 'plyvel/*.pxi'
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:26:10: error: expected ‘;’ at end of member declaration
   Status() noexcept : state_(nullptr) { }
          ^
/usr/local/include/leveldb/status.h:26:12: error: ‘noexcept’ does not name a type
   Status() noexcept : state_(nullptr) { }
            ^
/usr/local/include/leveldb/status.h:26:12: note: C++11 ‘noexcept’ only available with -std=c++11 or -std=gnu++11
/usr/local/include/leveldb/status.h:32:16: error: expected ‘,’ or ‘...’ before ‘&&’ token
   Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
                ^
/usr/local/include/leveldb/status.h:32:22: error: invalid constructor; you probably meant ‘leveldb::Status (const leveldb::Status&)’
   Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
                      ^
/usr/local/include/leveldb/status.h:32:22: error: expected ‘;’ at end of member declaration
/usr/local/include/leveldb/status.h:32:24: error: ‘noexcept’ does not name a type
   Status(Status&& rhs) noexcept : state_(rhs.state_) { rhs.state_ = nullptr; }
                        ^
/usr/local/include/leveldb/status.h:32:24: note: C++11 ‘noexcept’ only available with -std=c++11 or -std=gnu++11
/usr/local/include/leveldb/status.h:33:27: error: expected ‘,’ or ‘...’ before ‘&&’ token
   Status& operator=(Status&& rhs) noexcept;
                           ^
/usr/local/include/leveldb/status.h:33:33: error: expected ‘;’ at end of member declaration
   Status& operator=(Status&& rhs) noexcept;
                                 ^
/usr/local/include/leveldb/status.h:33:35: error: ‘noexcept’ does not name a type
   Status& operator=(Status&& rhs) noexcept;
                                   ^
/usr/local/include/leveldb/status.h:33:35: note: C++11 ‘noexcept’ only available with -std=c++11 or -std=gnu++11
/usr/local/include/leveldb/status.h: In member function ‘bool leveldb::Status::ok() const’:
/usr/local/include/leveldb/status.h:56:39: error: ‘nullptr’ was not declared in this scope
   bool ok() const { return (state_ == nullptr); }
                                       ^
/usr/local/include/leveldb/status.h: In member function ‘leveldb::Status::Code leveldb::Status::code() const’:
/usr/local/include/leveldb/status.h:95:23: error: ‘nullptr’ was not declared in this scope
     return (state_ == nullptr) ? kOk : static_cast<Code>(state_[4]);
                       ^
/usr/local/include/leveldb/status.h: In copy constructor ‘leveldb::Status::Status(const leveldb::Status&)’:
/usr/local/include/leveldb/status.h:103:27: error: ‘nullptr’ was not declared in this scope
   state_ = (rhs.state_ == nullptr) ? nullptr : CopyState(rhs.state_);
                           ^
/usr/local/include/leveldb/status.h: In member function ‘leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)’:
/usr/local/include/leveldb/status.h:110:29: error: ‘nullptr’ was not declared in this scope
     state_ = (rhs.state_ == nullptr) ? nullptr : CopyState(rhs.state_);
                             ^
/usr/local/include/leveldb/status.h: At global scope:
/usr/local/include/leveldb/status.h:114:40: error: expected ‘,’ or ‘...’ before ‘&&’ token
 inline Status& Status::operator=(Status&& rhs) noexcept {
                                        ^
/usr/local/include/leveldb/status.h:114:48: error: expected initializer before ‘noexcept’
 inline Status& Status::operator=(Status&& rhs) noexcept {
                                                ^
In file included from /usr/local/include/leveldb/db.h:11:0,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/iterator.h:80:9: error: expected nested-name-specifier before ‘CleanupFunction’
   using CleanupFunction = void (*)(void* arg1, void* arg2);
         ^
/usr/local/include/leveldb/iterator.h:81:24: error: ‘CleanupFunction’ has not been declared
   void RegisterCleanup(CleanupFunction function, void* arg1, void* arg2);
                        ^
/usr/local/include/leveldb/iterator.h:88:5: error: ‘CleanupFunction’ does not name a type
     CleanupFunction function;
     ^
/usr/local/include/leveldb/iterator.h: In member function ‘bool leveldb::Iterator::CleanupNode::IsEmpty() const’:
/usr/local/include/leveldb/iterator.h:94:35: error: ‘function’ was not declared in this scope
     bool IsEmpty() const { return function == nullptr; }
                                   ^
/usr/local/include/leveldb/iterator.h:94:47: error: ‘nullptr’ was not declared in this scope
     bool IsEmpty() const { return function == nullptr; }
                                               ^
/usr/local/include/leveldb/iterator.h: In member function ‘void leveldb::Iterator::CleanupNode::Run()’:
/usr/local/include/leveldb/iterator.h:96:49: error: ‘function’ was not declared in this scope
     void Run() { assert(function != nullptr); (*function)(arg1, arg2); }
                                                 ^
In file included from /usr/local/include/leveldb/db.h:12:0,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/options.h: In constructor ‘leveldb::ReadOptions::ReadOptions()’:
/usr/local/include/leveldb/options.h:183:18: error: ‘nullptr’ was not declared in this scope
         snapshot(nullptr) {
                  ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_f_6plyvel_7_plyvel_db_get(__pyx_obj_6plyvel_7_plyvel_DB*, PyObject*, PyObject*, leveldb::ReadOptions)’:
plyvel/_plyvel.cpp:2469:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = __pyx_v_db->_db->Get(__pyx_v_read_options, __pyx_v_key_slice, (&__pyx_v_value));
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘int __pyx_pf_6plyvel_7_plyvel_2DB___init__(__pyx_obj_6plyvel_7_plyvel_DB*, PyObject*, PyBoolObject*, PyBoolObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, int, PyObject*, PyObject*)’:
plyvel/_plyvel.cpp:3935:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = leveldb::DB::Open(__pyx_v_self->options, __pyx_v_fsname, (&__pyx_v_self->_db));
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_pf_6plyvel_7_plyvel_2DB_10put(__pyx_obj_6plyvel_7_plyvel_DB*, PyObject*, PyObject*, PyBoolObject*)’:
plyvel/_plyvel.cpp:5187:22: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
           __pyx_v_st = __pyx_v_self->_db->Put(__pyx_v_write_options, __pyx_v_key_slice, leveldb::Slice(((const char *)__pyx_v_value_buffer.buf), __pyx_v_value_buffer.len));
                      ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_pf_6plyvel_7_plyvel_2DB_12delete(__pyx_obj_6plyvel_7_plyvel_DB*, PyObject*, PyBoolObject*)’:
plyvel/_plyvel.cpp:5417:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = __pyx_v_self->_db->Delete(__pyx_v_write_options, __pyx_v_key_slice);
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_pf_6plyvel_7_plyvel_repair_db(PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, int, PyObject*, PyObject*)’:
plyvel/_plyvel.cpp:9357:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = leveldb::RepairDB(__pyx_v_fsname, __pyx_v_options);
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_pf_6plyvel_7_plyvel_2destroy_db(PyObject*, PyObject*)’:
plyvel/_plyvel.cpp:9487:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = leveldb::DestroyDB(__pyx_v_fsname, __pyx_v_options);
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
plyvel/_plyvel.cpp: In function ‘PyObject* __pyx_pf_6plyvel_7_plyvel_10WriteBatch_10write(__pyx_obj_6plyvel_7_plyvel_WriteBatch*)’:
plyvel/_plyvel.cpp:10450:20: error: ambiguous overload for ‘operator=’ (operand types are ‘leveldb::Status’ and ‘leveldb::Status’)
         __pyx_v_st = __pyx_v_self->db->_db->Write(__pyx_v_self->write_options, __pyx_v_self->_write_batch);
                    ^
In file included from /usr/local/include/leveldb/iterator.h:20:0,
                 from /usr/local/include/leveldb/db.h:11,
                 from plyvel/_plyvel.cpp:547:
/usr/local/include/leveldb/status.h:105:16: note: candidate: leveldb::Status& leveldb::Status::operator=(const leveldb::Status&)
 inline Status& Status::operator=(const Status& rhs) {
                ^
/usr/local/include/leveldb/status.h:33:11: note: candidate: leveldb::Status& leveldb::Status::operator=(leveldb::Status)
   Status& operator=(Status&& rhs) noexcept;
           ^
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Add support for Bech32 segwit address format (BIP 0173)

There is currently no support for BECH32 addresses as specified by BIP 0173 (https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki),

Example:

> tx
Transaction(bb77f4a58e0c23a21f7a261255064acf69d939971f770b42d1f0d9839bdb469b)
> tx.txid
'f91d0a8a78462bc59398f2c5d7a84fcff491c26ba54c4833478b202796c8aafd'
> tx.hash
'bb77f4a58e0c23a21f7a261255064acf69d939971f770b42d1f0d9839bdb469b'
> tx.outputs[0].addresses
[]
# result should be bc1q34aq5drpuwy3wgl9lhup9892qp6svr8ldzyy7c
# https://blockchain.info/tx/f91d0a8a78462bc59398f2c5d7a84fcff491c26ba54c4833478b202796c8aafd?show_adv=true

This shows how it should work:

# Reference implementation:
# https://raw.githubusercontent.com/sipa/bech32/master/ref/python/segwit_addr.py
> import segwit_addr
> segwit_addr.encode(hrp="bc", witver=0, witprog=bytearray(tx.outputs[0]._script.hex[2:]))
'bc1q34aq5drpuwy3wgl9lhup9892qp6svr8ldzyy7c'

exporting data into a JSON or CSV file.

Hi,
I have been trying to export data extracted by these parser libraries, however I am receiving so many different errors. Below is my code:

import sys
import pandas as pd
from blockchain_parser.blockchain import Blockchain
from datetime import datetime

blockchain = Blockchain(sys.argv[1])
for block in blockchain.get_unordered_blocks():
for transaction in block.transactions:
for output in transaction.outputs:
if block.header.timestamp > datetime(2008, 1, 1):

            df = (block.header.timestamp, output.value)
            df_export = df.groupby([df[block.header.timestamp], output.value]).sum().reset_index()
			df_export = df_export.rename(coloums={block.header.timestamp : 'Time of transaction', output.value : 'Amount in Satoshi'})
			df_export.to_json('C:/btcdata.json', orient='records')

In the code above, I was trying to export data into a JSON file but I received the following errors:

File "jsontimestamp.py", line 15
df_export = df_export.rename(coloums={block.header.timestamp : 'Time of transaction', output.value : 'Amount in Satoshi'})
^
TabError: inconsistent use of tabs and spaces in indentation

I have checked the indentation and it doesn't seem to be a problem. Can you please help??

Thanks
Gurps

Locating an input's original transaction

I have been trying to find the transaction an input was originally part of, but have been unsuccessful. It should be as simple as comparing transaction_hash with every prior transaction's hash, but I've come up dry. Has this been tried? (And if so, could I see the code?) Here's what I wrote:

import sys
sys.path.append('..')
from blockchain_parser.blockchain import Blockchain

blockchain = Blockchain(sys.argv[1])

transaction_found = False
for block in blockchain.get_ordered_blocks(sys.argv[1] + '/index', start=10000, end=0, cache="cache_money"):	
	#Don't want a coinbase transaction
	if(block.n_transactions == 1):
		continue

	if not transaction_found:
		tx_hash = block.transactions[1].inputs[0].transaction_hash
		print("tx_hash: " + str(tx_hash))
		transaction_found = True

	#Check transactions to find a match
	for tx in block.transactions:
		if tx.hash == tx_hash:
			print("We found the transaction! Good work team!")
			break
		else:
			print(tx.hash)

c

Support Testnet addresses when using get_block_transactions()

After parse block from Testnet blockchain_parser.block.get_block_transactions() returns transactions with addresses started from "1", but for Testnet they should be started from "m" or "n". List of address prefixes

Add tests for ordered and unordered blocks parsing

#28 and #33 hightlight the need to add tests that simulate a Bitcoind datadir with an index to test that everything works as expected.

Run parser simultaneously with running bitcoin node

I want to write bitcoin cash wallet for educational purpose.

I need to know about wallet balance and statistics (like spends during month), but:

according to https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_2):_Data_Storage the block cache is guaranteed to be flushed once an hour.
according to https://github.com/google/leveldb/blob/master/doc/index.md database may only be opened by one process at a time.

It seems:
a) i can't get information about last transactions
b) i can't run parser simultaneously with running bitcoin node

How can i get information about last transactions?

Ascii values of hex characters

With some version of Python the following checks return always False, resulting in few addresses being decoded:

(in output.py)

if len(hex_data) == 65 and hex_data[0] == 4:
if len(hex_data) == 33 and hex_data[0] in [2, 3]:

I would suggest doing this:

if len(hex_data) == 65 and ord(hex_data[0]) == 4:
if len(hex_data) == 33 and ord(hex_data[0]) in [2, 3]:

Limited altcoins support

Add possibility to change the bitcoin specific constants so that alts like BCH, LTC and others can be (at least partially) supported.

Is there a method to extract OP_RETURN data from blocks?

testnet

it dose not support testnet And witness_v0_keyhash ,witness_v0_scripthash?

Typographical error

python-bitcoin-blockchain-parser/blockchain_parser/blockchain.py

Line 144 in 5069a1d

# we are ready to make a decesion on whether or not the block

~~decesion~~ decision

how do you get input address?

output.addresses works great but input doesn't have an addresses property?

AssertionError when parsing recent blocks

The parser is working properly with old blk files, but with recent blk files throws this error:

File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/block.py", line 81, in transactions
    self._transactions = list(get_block_transactions(self.hex))
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/block.py", line 29, in get_block_transactions
    transaction = Transaction.from_hex(transaction_data[offset:])
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/transaction.py", line 60, in from_hex
    return cls(hex)
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/transaction.py", line 39, in __init__
    input = Input.from_hex(raw_hex[offset:])
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/input.py", line 33, in from_hex
    return cls(hex_)
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/input.py", line 25, in __init__
    self._script_length, varint_length = decode_varint(raw_hex[36:])
  File "/usr/local/lib/python3.5/dist-packages/blockchain_parser/utils.py", line 50, in decode_varint
    assert(len(data) > 0)
AssertionError

... this is a log when use blk01347.dat file

Add support for getting ordered blocks

I have been using this library in one of my projects and it is working perfectly but I have ran into a really big roadblock: I need the blocks that your library returns to be ordered as my code requires this to work. Could you please implement this when you get a chance of even tell me how to implment it myself. Thanks so much for building this! :)

Improve orphan blocks handling

When iterating over ordered blocks and encountering a situation with two blocks at the same height, we read them from disk as well as their children to know which has 6 confs+. We could avoid reading these blocks by hashing the header information held in the index entries of these blocks since we already have those in memory.

Why self.blockIndexes returns an empty list?

in blockchain.py:

def __getBlockIndexes(self, index):
    """There is no method of leveldb to close the db (and release the lock).
    This creates problem during concurrent operations.
    This function also provides caching of indexes.
    """
    if self.indexPath != index:
        db = plyvel.DB(index, compression=None)
		print("%s" % (str(db))) # added by me and it returns <plyvel.DB with name 'something' at somewhere>
        self.blockIndexes = [DBBlockIndex(format_hash(k[1:]), v)
                             for k, v in db.iterator() if k[0] == ord('b')]
        db.close()
        self.blockIndexes.sort(key=lambda x: x.height)
        self.indexPath = index
    return self.blockIndexes

returned self.blockIndexes is an empty list. Wondering why? If you still maintaining this code pls reply!

exception while parsing first 100 bitcoin blk files

Is this something I should just catch and ignore? I'm doing aggregate statistics on the whole ledger, so I don't need absolute accuracy for every single transaction.

  File "/home/205740/.local/lib/python3.5/site-packages/blockchain_parser/output.py", line 60, in addresses
    if self.type == "pubkey":
  File "/home/205740/.local/lib/python3.5/site-packages/blockchain_parser/output.py", line 101, in type
    if self.is_pubkey():
  File "/home/205740/.local/lib/python3.5/site-packages/blockchain_parser/output.py", line 84, in is_pubkey
    return self.script.is_pubkey()
  File "/home/205740/.local/lib/python3.5/site-packages/blockchain_parser/script.py", line 103, in is_pubkey
    return len(self.operations) == 2 \
  File "/home/205740/.local/lib/python3.5/site-packages/blockchain_parser/script.py", line 74, in operations
    self._operations = list(self.script)
  File "/home/205740/.local/lib/python3.5/site-packages/bitcoin/core/script.py", line 622, in __iter__
    for (opcode, data, sop_idx) in self.raw_iter():
  File "/home/205740/.local/lib/python3.5/site-packages/bitcoin/core/script.py", line 607, in raw_iter
    raise CScriptTruncatedPushDataError('%s: truncated data' % pushdata_type, data)
bitcoin.core.script.CScriptTruncatedPushDataError: PUSHDATA(1): truncated data

Transaction time

Is there a way to access transaction time? I only see the following methods on transaction:

['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 '_hash',
 '_locktime',
 '_txid',
 '_version',
 'from_hex',
 'hash',
 'hex',
 'inputs',
 'is_coinbase',
 'is_segwit',
 'locktime',
 'n_inputs',
 'n_outputs',
 'outputs',
 'size',
 'txid',
 'uses_bip69',
 'uses_replace_by_fee',
 'version']

And the following on block:

['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 '_hash',
 '_header',
 '_n_transactions',
 '_transactions',
 'from_hex',
 'hash',
 'header',
 'height',
 'hex',
 'n_transactions',
 'size',
 'transactions']

locktime on transaction appears to be 0 in the first few I've tried to access.

setup.py fails

Hello,

I am trying to install this library. After installing all listed dependencies I run setup.py, only to have it fail with this errror:

zip_safe flag not set; analyzing archive contents...
blockchain_parser.tests.test_transaction: module references file
warning: no files found matching 'plyvel/.pxi'
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
plyvel/_plyvel.cpp: In function ‘int __pyx_f_6plyvel_7_plyvel_parse_options(leveldb::Options, bool, bool, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, PyObject*, int, PyObject*, PyObject*)’:
plyvel/_plyvel.cpp:3341:22: error: ‘struct leveldb::Options’ has no member named ‘max_file_size’
__pyx_v_options->max_file_size = __pyx_t_4;
^
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Before the error, output from setup.py is:
setup_before_error.txt

things about get_ordered_blocks

When I running ordered-blocks.py on Mac terminal, it always comes out as follows:
Traceback (most recent call last):
File "plyvel/_plyvel.pyx", line 247, in plyvel._plyvel.DB.init
File "plyvel/_plyvel.pyx", line 88, in plyvel._plyvel.raise_for_status
plyvel._plyvel.IOError: b'IO error: Users/fanfangege/Desktop/blocks/index/LOCK: No such file or directory'

And I my settings was that:
blockchain = Blockchain(os.path.expanduser('/Users/fanfangege/Desktop/blocks'))
for block in blockchain.get_ordered_blocks( os.path.expanduser('Users/fanfangege/Desktop/blocks/index'), end=1000):
print("height=%d block=%s" % (block.height, block.hash))

How can we find the from address coins?

We cannot find the logic involving with "from address coins" in your parsing tool. How can we find the from address coins? Is it in the previous transaction hash' s output file?

Orphan blocks

jonathanbier/forkmonitor#1 (comment)

Output Addresses always none

Firstly thanks for your source, But i have the problem when get transaction.outputs and give me Always none

pypi python package no upgrade?

why not upgrade python package on pypi?

Testnet. Not getting info (-1) for some blocks.

Hi.

Thanks for this amazing software.

Unfortunately I couldn't go beyond block 1414414 cause after that block the program returns many -1:
DBBlockIndex(000000001098e10889c0ffd39c5d39cc5a54f4f456adda5b52e017072f918a16, height=1414416, file_no=-1, file_pos=-1)

Why is that?

Convoluted logic

python-bitcoin-blockchain-parser/blockchain_parser/blockchain.py

Lines 147 to 150 in 5069a1d

 if first_block.hash in chain: 

 return True 

 else: 

 return False

can be simplified to: return first_block.hash in chain

Add support for UTXO parsing

See https://github.com/eklitzke/utxodump for an example implementation.

This could allow querying of balances of live addresses, quite exciting!

Block.height always None

Is the height currently not parsed? It's always None for me and looking at the source it does not appear to be set anywhere, am I missing anything?

Thanks

alecalve / python-bitcoin-blockchain-parser Goto Github PK

python-bitcoin-blockchain-parser's People

Stargazers

Watchers

Forkers

python-bitcoin-blockchain-parser's Issues

Recommend Projects

Recommend Topics

Recommend Org