vkobel / ethdrain Goto Github PK
View Code? Open in Web Editor NEWPython script allowing to copy the Ethereum blockchain to an ElasticSearch instance in an efficient way by connecting to a local RPC node
License: MIT License
ethdrain's People
Contributors
Stargazers
Watchers
Forkers
toadkicker cheekybastard wqwrtaufxjw7uflcxzxz thomaskirst cyberfund prof-schacht ivorobyev prabushitha sunfinite snail-fuji jin10086 sambachaethdrain's Issues
Elasticsearch timeout - heap size too low
multiprocessing.pool.RemoteTraceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1198, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py", line 114, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 649, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 324, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.5/dist-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 388, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "./ethdrain.py", line 123, in setup_process
helpers.bulk(elasticsearch, out_actions)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 194, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 162, in streaming_bulk
for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 91, in _process_bulk_chunk
raise e
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 87, in _process_bulk_chunk
resp = client.bulk('\n'.join(bulk_actions) + '\n', **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/__init__.py", line 1167, in bulk
doc_type, '_bulk'), params=params, body=self._bulk_body(body))
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 318, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py", line 122, in perform_request
raise ConnectionTimeout('TIMEOUT', str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (rea
d timeout=10))
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./ethdrain.py", line 151, in <module>
POOL.map(setup_process, CHUNKS)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(None: None)
./ethdrain.py 0 3476300 1922.24s user 56.31s system 41% cpu 1:20:05.84 total
Proposal: Save last run info in elasticsearch
When ethdrain runs it could call to ES to find out the metadata regarding failed blocks and blocks that are already stored in the index, then build a list from there. If there is no index available, it can then assume it is a first run and proceed from there.
¯\_(ツ)_/¯
325595
325596
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/toadkicker/.pyenv/versions/3.6.1/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/Users/toadkicker/.pyenv/versions/3.6.1/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/Users/toadkicker/Projects/ethdrain/ethdrain.py", line 106, in setup_process
loop.run_until_complete(future)
File "/Users/toadkicker/.pyenv/versions/3.6.1/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
return future.result()
File "/Users/toadkicker/Projects/ethdrain/ethdrain.py", line 65, in run
await asyncio.gather(*tasks)
File "/Users/toadkicker/Projects/ethdrain/ethdrain.py", line 50, in sema_fetch
await fetch(url, session, blockNb, fn, actions)
File "/Users/toadkicker/Projects/ethdrain/ethdrain.py", line 45, in fetch
if data:
UnboundLocalError: local variable 'data' referenced before assignment
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/toadkicker/Projects/ethdrain/ethdrain.py", line 132, in <module>
p.map(setup_process, chks)
File "/Users/toadkicker/.pyenv/versions/3.6.1/lib/python3.6/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/toadkicker/.pyenv/versions/3.6.1/lib/python3.6/multiprocessing/pool.py", line 608, in get
raise self._value
UnboundLocalError: local variable 'data' referenced before assignment
Handle TimeoutError
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/ubuntu/ethdrain/ethdrain.py", line 109, in setup_process
loop.run_until_complete(future)
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/ubuntu/ethdrain/ethdrain.py", line 66, in run
await asyncio.gather(*tasks)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/ubuntu/ethdrain/ethdrain.py", line 50, in sema_fetch
await fetch(url, session, block, process_fn, actions)
File "/home/ubuntu/ethdrain/ethdrain.py", line 42, in fetch
async with session.post(url, data=make_request(block), headers=HTTP_HEADERS) as response:
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 626, in __aenter__
self._resp = yield from self._coro
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 295, in _request
break
File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 699, in __exit__
raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/ethdrain/ethdrain.py", line 142, in <module>
POOL.map(setup_process, CHUNKS)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
concurrent.futures._base.TimeoutError
Timeout error on elasticsearch bulk operation
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "./ethdrain.py", line 123, in setup_process
helpers.bulk(elasticsearch, out_actions)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 194, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 162, in streaming_bulk
for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 91, in _process_bulk_chunk
raise e
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/helpers/__init__.py", line 87, in _process_bulk_chunk
resp = client.bulk('\n'.join(bulk_actions) + '\n', **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/client/__init__.py", line 1167, in bulk
doc_type, '_bulk'), params=params, body=self._bulk_body(body))
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/transport.py", line 318, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/elasticsearch/connection/http_urllib3.py", line 122, in perform_request
raise ConnectionTimeout('TIMEOUT', str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read
timeout=10))
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./ethdrain.py", line 151, in <module>
POOL.map(setup_process, CHUNKS)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(None: None)
helpers.bulk line seems to timeout in some cases
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.