romanz / electrs Goto Github PK

An efficient re-implementation of Electrum Server in Rust

License: MIT License

Rust 89.34% Python 8.18% Shell 2.10% Dockerfile 0.38%

bitcoin rust indexer electrum electrum-server rocksdb

electrs's Introduction

Electrum Server in Rust

An efficient re-implementation of Electrum Server, inspired by ElectrumX, Electrum Personal Server and bitcoincore-indexd.

The motivation behind this project is to enable a user to self host an Electrum server, with required hardware resources not much beyond those of a full node. The server indexes the entire Bitcoin blockchain, and the resulting index enables fast queries for any given user wallet, allowing the user to keep real-time track of balances and transaction history using the Electrum wallet. Since it runs on the user's own machine, there is no need for the wallet to communicate with external Electrum servers, thus preserving the privacy of the user's addresses and balances.

BTC Prague 2024 dev/hack/day slides are here: https://bit.ly/electrs

Usage

Please prefer to use OUR usage guide!

External guides can be out-of-date and have various problems. At least double-check that the guide you're using is actively maintained. If you can't use our guide, please ask about what you don't understand or consider using automated deployments.

Note that this implementation of Electrum server is optimized for personal/small-scale (family/friends) usage. It's a bad idea to run it publicly as it'd expose you to DoS and maybe also other attacks. If you want to run a public server you may be interested in the Blockstream fork of electrs which is better optimized for public usage at the cost of consuming significantly more resources.

Installation from source
Pre-built binaries (No official binaries available but a beta repository is available for installation)
Configuration
Usage
Monitoring
Upgrading - contains information about important changes from older versions

Features

Supports Electrum protocol v1.4
Maintains an index over transaction inputs and outputs, allowing fast balance queries
Fast synchronization of the Bitcoin blockchain (~6.5 hours for ~504GB @ August 2023) using HDD storage.
Low index storage overhead (~10%), relying on a local full node for transaction retrieval
Efficient mempool tracker (allowing better fee estimation)
Low CPU & memory usage (after initial indexing)
txindex is not required for the Bitcoin node
Uses a single RocksDB database, for better consistency and crash recovery

Altcoins

Altcoins are not supported! Forks of Bitcoin codebase that relax the consensus rules (hard forks) are also not supported.

You may be able to find a fork of electrs that does support them, look around or make your own, just don't file issues/PRs here.

Index database

The database schema is described here.

Contributing

All contributions to this project are welcome. Please refer to the Contributing Guidelines for more details.

Logo

Our logo is generously provided by Dominik Průša under the MIT license. Based on the Electrum logo and the Rust language logo.

electrs's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger tempbottle greenaddress rcasatta sangaman araspitzu moshababo federicobond sha-0 blockstream dexfinexchange khoazany dagurval stadicus openoms gfloresechaiz romanornr echterago cryptobuks jb55 erasmospunk kixunil aussiehash metaver5o muun satoshi-n dyzz stevenroose kirillkovalenko valyakin michelkansou zengo-x samourai-wallet lukechilds dmp1ce cambricorp x1ddos joemphilips groestlcoin burguesia dr-orlovsky nessa-g22 rafaelagp browep peleion justinvforvendetta willcl-ark augustoresende buzzkillb tookennysupreme mohneeshjhapuresoftware abhisheknishantpuresoftware cryptapus darosior sfzxc metaspartan jamesob candlehater pox nolim1t npodonnell leohchen zoedberg vergecurrency d4nte k3tan172 weijun-sh dylanparker schildbach jpdarago szollo sohomang odolvlobo thebluematt matthewlilley flatcloud0b3 bitcoincore-dev code1110 janhn bigspider canndrew wyager bitweb-project cryptoquick jettoblack kcebul12 aarondewes praexor tdcoin-dev-team tcharding amirulmus lirre8 simonvrouwe pantamis notmandatory fortax gcharang majikaz keep-network dorucioclea

electrs's Issues

Experiment with universal compaction for bulk load

https://github.com/facebook/rocksdb/wiki/Universal-Compaction

Persist stderr & panic backtraces

At scripts/run.sh

Use JSONRPC `id` field for bitcoind communication

Following #24

Transaction notification

Hi, with this server, running on a EC2/VPS is possible to get notifications for new transactions on a public address?

What I want is to run this server to index and notify my node application about received payments on a generated address.

Thank you.

Use peer-to-peer protocol for block download

Following the idea from bitcoincore-indexd.

Move index queries functions into Index

So Query won't need ReadStore access (via App::read_store).

cmake is needed too

I suggest to update the usage to also install cmake since it's needed to build snappy-sys

expected u32, found u64

--> src/bulk.rs:164:21
|
164 | rlim.rlim_cur = limit; // set softs limit only.
| ^^^^^ expected u32, found u64

Assuming this is a 32/64bit error?

Compiling on 32bit.

error: Could not compile `chrono`.

./run.sh

Checking chrono v0.4.6
error: couldn't read "/home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.6/src/offset/local.rs": stream did not contain valid UTF-8
   --> /home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.6/src/offset/mod.rs:414:5
    |
414 | mod local;
    |     ^^^^^

error: aborting due to previous error

error: Could not compile `chrono`.
warning: build failed, waiting for other jobs to finish...
error: build failed

rustc --version
rustc 1.29.0 (aa3ca1994 2018-09-11)

ERROR - server failed: Error: invalid blockchain info

Caused by: invalid value: integer 198853069730, expected usize

Not sure why it's throwing this. Node is current.

Re-use `rust-bitcoin` Network enum

https://github.com/rust-bitcoin/rust-bitcoin/blob/bf9c749b0c3c63dc2154bacf4d8c1c61dfe1d734/src/network/constants.rs#L53

Retrieve and log bitcoind version

Should be 0.16 or above.

Compiling: error[E0308]: mismatched types

error[E0308]: mismatched types
--> /home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/sysconf-0.3.3/src/page/hugepage.rs:346:23
|
346 | if unsafe { lstat(path.as_ptr() as *const i8, &mut s) } < 0 {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
|
= note: expected type *const u8
found type *const i8

error: aborting due to previous error

For more information about this error, try rustc --explain E0308.
error: Could not compile sysconf.
warning: build failed, waiting for other jobs to finish...
error: build failed

Electrs hang when submitting an invalid transaction

As a test I submitted a transaction built with invalid signatures:

Some("{\"result\":null,\"error\":{\"code\":-25,\"message\":\"Missing inputs\"},\"id\":517379}")
2018-08-14T13:26:17.847+00:00 - WARN - connection failed: request failed "HTTP/1.1 500 Internal Server Error": ["Content-Type: application/json", "Date: Tue, 14 Aug 2018 13:26:17 GMT", "Content-Length: 76"] = Some("{\"result\":null,\"error\":{\"code\":-25,\"message\":\"Missing inputs\"},\"id\":517379}")
2018-08-14T13:26:20.847+00:00 - WARN - connection failed: request failed "HTTP/1.1 500 Internal Server Error": ["Content-Type: application/json", "Date: Tue, 14 Aug 2018 13:26:20 GMT", "Content-Length: 76"] = Some("{\"result\":null,\"error\":{\"code\":-25,\"message\":\"Missing inputs\"},\"id\":517379}")
2018-08-14T13:26:23.848+00:00 - WARN - connection failed: request failed "HTTP/1.1 500 Internal Server Error": ["Content-Type: application/json", "Date: Tue, 14 Aug 2018 13:26:23 GMT", "Content-Length: 76"] = Some("{\"result\":null,\"error\":{\"code\":-25,\"message\":\"Missing inputs\"},\"id\":517379}")
2018-08-14T13:26:26.849+00:00 - WARN - connection failed: request failed "HTTP/1.1 500 Internal Server Error": ["Content-Type: application/json", "Date: Tue, 14 Aug 2018 13:26:26 GMT", "Content-Length: 76"] = Some("{\"result\":null,\"error\":{\"code\":-25,\"message\":\"Missing inputs\"},\"id\":517379}")

Electrs should return directly the error to the client.

Compile error: > error: failed to run custom build command for `librocksdb-sys v5.14.2`

   Compiling librocksdb-sys v5.14.2
error: failed to run custom build command for `librocksdb-sys v5.14.2`
process didn't exit successfully: `/mnt/storage/.pin0de/src/github.com/romanz/electrs/target/release/build/librocksdb-sys-75503bf7a5321165/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=rocksdb/
cargo:rerun-if-changed=snappy/

--- stderr
rocksdb/include/rocksdb/c.h:48:9: warning: #pragma once in main file [-Wpragma-once-outside-header]
rocksdb/include/rocksdb/c.h:68:10: fatal error: 'stdarg.h' file not found
rocksdb/include/rocksdb/c.h:48:9: warning: #pragma once in main file [-Wpragma-once-outside-header], err: false
rocksdb/include/rocksdb/c.h:68:10: fatal error: 'stdarg.h' file not found, err: true
thread 'main' panicked at 'unable to generate rocksdb bindings: ()', libcore/result.rs:945:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:511
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:426
   6: rust_begin_unwind
             at libstd/panicking.rs:337
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:92
   8: core::result::unwrap_failed
   9: build_script_build::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  12: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  13: std::panicking::try
             at libstd/panicking.rs:289
  14: std::rt::lang_start_internal
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  15: main
  16: __libc_start_main

rustc --version
rustc 1.28.0 (9634041f0 2018-07-30)

fatal runtime error: memory allocation failed

On my Raspberry Pi 3 I'm receiving this error.

fatal runtime error: memory allocation failed
Aborted

sometimes it throws

2018-07-22T12:25:45.772+00:00 - ERROR - server failed: Error: RPC failed: {"id":3,"method":"getbestblockhash","params":[]}
Caused by: disconnected from daemon

but works re-running it.

Switch to signal-hook (chan crate is deprecated)

https://users.rust-lang.org/t/ann-chan-is-deprecated-use-crossbeam-channel-instead/19251
https://github.com/vorner/signal-hook

unknown method blockchain.scripthash.listunspent [String(" when trying to swipe a private key

If I try to swipe a private key into electrum the server stop responding and in the log of the server appears:

2018-07-21T17:17:08.388+02:00 - ERROR - [127.0.0.1:33256] connection handling failed: Error: unknown method blockchain.scripthash.listunspent [String("c09711449cd08e347cc70a32f6f4efacf5ef75c1d7d17dc53c84b9ade0af5143")]

2018-07-21T17:17:08.403+02:00 - INFO - [127.0.0.1:33256] shutting down connection
2018-07-21T17:17:08.404+02:00 - INFO - [127.0.0.1:33256] disconnected peer
2018-07-21T17:17:08.958+02:00 - INFO - [127.0.0.1:33308] connected peer
2018-07-21T17:17:09.466+02:00 - ERROR - [127.0.0.1:33308] connection handling failed: Error: unknown method blockchain.scripthash.listunspent [String("c09711449cd08e347cc70a32f6f4efacf5ef75c1d7d17dc53c84b9ade0af5143")]

2018-07-21T17:17:09.466+02:00 - INFO - [127.0.0.1:33308] shutting down connection
2018-07-21T17:17:19.072+02:00 - INFO - [127.0.0.1:33310] connected peer
2018-07-21T17:17:19.548+02:00 - ERROR - [127.0.0.1:33310] connection handling failed: Error: unknown method blockchain.scripthash.listunspent [String("c09711449cd08e347cc70a32f6f4efacf5ef75c1d7d17dc53c84b9ade0af5143")]

Use separate crate for Bitcoin JSONRPC handling

Use rust-bitcoin-rpc or rust-jsonrpc, instead of reimplementing Bitcoin JSONRPC.

Add more unittests

Any help will be appreciated :)

Script hash collisions

Just wanted to say that

SHA256(script)[:8]

is absolutely not secure or recommendable on BCH or BTC chains. 64 bits is already just slightly below the prefix length where there are collisions on BCH (currently 56). There may well be collisions already on BTC. ElectrumX uses 88 bit indexing and I'm already getting uncomfortable; I will be raising it to 96 or 104 bits soon. 104 bits should suffice GB blocks for quite a long time to come.

The index DB grows too much

Is there a way to control the amount of RAM usage? Would be helpful on low memory systems.

For the time being I just made a large swap for testing.

Cannot compile (Ubuntu 18.04) -> old Rust version

 cargo build --release
   Compiling electrs v0.1.0 (file:///home/kirill/electrs)
error[E0432]: unresolved import `std::ops::Bound`
 --> src/mempool.rs:7:5
  |
7 | use std::ops::Bound;
  |     ^^^^^^^^^^^^^^^ no `Bound` in `ops`

error: `impl Trait` in return position is experimental (see issue #34511)
  --> src/query.rs:44:26
   |
44 |     fn funding(&self) -> impl Iterator<Item = &FundingOutput> {
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: `impl Trait` in return position is experimental (see issue #34511)
  --> src/query.rs:48:27
   |
48 |     fn spending(&self) -> impl Iterator<Item = &SpendingInput> {
   |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: use of unstable library feature 'fs_read_write' (see issue #46588)
   --> src/bulk.rs:140:23
    |
140 |             let msg = fs::read(f).chain_err(|| format!("failed to read {:?}", f));
    |                       ^^^^^^^^

error: use of unstable library feature 'fs_read_write' (see issue #46588)
  --> src/daemon.rs:32:5
   |
32 |     fs::read(&path).chain_err(|| format!("failed to read cookie from {:?}", path))
   |     ^^^^^^^^

error: non-reference pattern used to match a reference (see issue #42640)
  --> src/query.rs:37:17
   |
37 | fn calc_balance((funding, spending): &(Vec<FundingOutput>, Vec<SpendingInput>)) -> u64 {
   |                 ^^^^^^^^^^^^^^^^^^^ help: consider using a reference: `&(funding, spending)`

error: non-reference pattern used to match a reference (see issue #42640)
   --> src/query.rs:382:13
    |
382 |         for (fee_rate, vsize) in self.tracker.read().unwrap().fee_histogram() {
    |             ^^^^^^^^^^^^^^^^^ help: consider using a reference: `&(fee_rate, vsize)`

error: aborting due to 7 previous errors

error: Could not compile `electrs`.

To learn more, run the command again with --verbose.

Allow skipping fast initial indexing to support low-memory devices

e.g. #38

Handle JSONRPC disconnections gracefully

Currently electrs panics on any disconnection from bitcoind.

Support SSL connections for Electrum RPC

Currently, electrs crashes if Electrum wallet tries to connect using SSL [1] (instead of TCP [2])

[1] electrum -1 --server=127.0.0.1:50001:s
[2] electrum -1 --server=127.0.0.1:50001:t

Document the limitations of current implementation

optimize for N>2 CPU cores
control RocksDB RAM usage
optimize for SSD-based index storage (e.g. use concurrent lookups for lower latency)
test on non-Linux platforms (macOS, BSD, Windows...)

See https://github.com/romanz/electrs/blob/master/TODO.md for more action items.

Document usage for low-memory systems (e.g. Raspberry Pi or Odroid HC1)

Following #38, #58 and #65

failed to read cookie from "/home/bitcoin/.bitcoin/.cookie"

I suppose the file is not present because I have the rpcuser and
rpcpassword variables set.

Is it possible to set them in alternative to the cookie file?

Document useful Prometheus queries for performance analysis

e.g. time spent by calling bitcoind (grouped by RPC).

ERROR - server failed: Error: incorrect magic 2a9315cb at 14742100

2018-07-09T14:46:37.258+00:00 - DEBUG - applying 531189 new headers from height 0
2018-07-09T14:46:38.829+00:00 - INFO - indexing 1310 blk*.dat files
2018-07-09T14:46:38.829+00:00 - DEBUG - 370713 blocks are already indexed
2018-07-09T15:27:34.207+00:00 - ERROR - server failed: Error: incorrect magic 2a9315cb at 14742100

Handle disconnections during reading RPC response

Following #55 (comment)

thread 'fetcher' panicked at 'failed sending blocks to be indexed: "SendError(..)"', libcore/result.rs:945:5

2018-07-22T07:46:48.808-04:00 - INFO - best=0000000000000000002951d3e7c55d648db084a352ece44c74e99a675d4cabd0 height=533089 @ 2018-07-22T11:45:58Z (84197 left to index)
thread 'fetcher' panicked at 'failed sending blocks to be indexed: "SendError(..)"', libcore/result.rs:945:5
note: Run with RUST_BACKTRACE=1 for a backtrace.
2018-07-22T08:07:59.964-04:00 - TRACE - closing DB
2018-07-22T08:08:00.694-04:00 - ERROR - server failed: Error: RPC failed: [{"id":84251,"method":"getblock","params":["0000000000000000020eb999832487d18f24cd859ea7db742a92b0567ab91ca1",false]},{"id":84251,"method":"getblock","params":["000000000000000000d6c24c1ace1e7b3da8bc06c29178a597a78e5515ea45ae",false]},{"id":84251,"method":"getblock","params":["000000000000000001eb58dc48854852661ae8e1b771e47c77f9babbd89bd225",false]},{"id":84251,"method":"getblock","params":["00000000000000000115debee56fc1537d19bc9d84b93d166414bf1317884f99",false]},{"id":84251,"method":"getblock","params":["000000000000000001ee5208fc1e2fddf65738bfc8a1f2088d3fa9108a08ff8f",false]},{"id":84251,"method":"getblock","params":["0000000000000000021086e98039c290247c509b80c0c1b3d0dd0f923e506e52",false]},{"id":84251,"method":"getblock","params":["000000000000000000eec3e5eaea812f975dfefed2ba1dd05676b58fe2036676",false]},{"id":84251,"method":"getblock","params":["000000000000000001533c7ffa5e7ad648d60080d8a36da6e83ada7602f85a07",false]},{"id":84251,"method":"getblock","params":["000000000000000001b54ed8a86715f7518b7494824a8d37fcd75cd8016b0355",false]},{"id":84251,"method":"getblock","params":["000000000000000000c1362e6be20bd1d1e2b2c703d4a14f6501f7bbdc74c29c",false]},{"id":84251,"method":"getblock","params":["000000000000000000bb786a53ee0b1c2c6cbed1154e6687ffed0b9098eda346",false]},{"id":84251,"method":"getblock","params":["00000000000000000191dc941e341f7dbc70c3a9b59f6e88aa978a7f63f0581c",false]},{"id":84251,"method":"getblock","params":["000000000000000002001441ed0a1395d68b6c8b74b692e1cba9c21e2d886f49",false]},{"id":84251,"method":"getblock","params":["0000000000000000010c0c045b527ba2e8248a9b1475c69d388599575e171aa7",false]},{"id":84251,"method":"getblock","params":["000000000000000002019ea4d197840ddc30b39777803274b8c4aa4031bcefab",false]},{"id":84251,"method":"getblock","params":["0000000000000000009ea93af4170cd712726ad8f4ddf715c4adeaf3374e3b58",false]},{"id":84251,"method":"getblock","params":["000000000000000000da6ecaeb65eee83244715c3c3e90d1ac6afe770e3a6554",false]},{"id":84251,"method":"getblock","params":["000000000000000001005d250ef9f43056ca34d17f10e556aaac7ac9324b78f5",false]},{"id":84251,"method":"getblock","params":["000000000000000001b29c4386d1b36c25c3408ad49cd85d2dd0163a0f800628",false]},{"id":84251,"method":"getblock","params":["000000000000000002164d6005ef38ea869b995609b83648262b59978b35dae2",false]},{"id":84251,"method":"getblock","params":["0000000000000000016e42152acd207519de23ac4f66ae3d2b2442f8e885573a",false]},{"id":84251,"method":"getblock","params":["0000000000000000020fd12cce1144736f8f49fa85b8cf96ecdd570195f4b8bd",false]},{"id":84251,"method":"getblock","params":["000000000000000001e18c1afd2358da6502724543eda891485707af6b947546",false]},{"id":84251,"method":"getblock","params":["0000000000000000003b7c45567de4c1c3eb361b8466d162f1388b0bb82401b9",false]},{"id":84251,"method":"getblock","params":["0000000000000000005994be75df44df4bdf0e5299322edf76e30f5bbdec05d4",false]},{"id":84251,"method":"getblock","params":["0000000000000000020e3f83a772e71b414c3d76f6664022caec0f17df6247d4",false]},{"id":84251,"method":"getblock","params":["0000000000000000020dceec1a49b119909dcc8651be5db7edf622267878d9a2",false]},{"id":84251,"method":"getblock","params":["0000000000000000014018334494759c9b5e6c0553ded85a469c54cc6889b8e7",false]},{"id":84251,"method":"getblock","params":["000000000000000000de068e372ff71fd9af03ee166c2b8daeb22af78c400afe",false]},{"id":84251,"method":"getblock","params":["00000000000000000153f24946d33ac8d5a1ea912f04148a28a483c4dc83e662",false]},{"id":84251,"method":"getblock","params":["0000000000000000015ce530dc65869e2f1f3a2f4548a1673f564a80ff8299b6",false]},{"id":84251,"method":"getblock","params":["00000000000000000257351c48730e0b4db45cc058960b233bccf0cfb9bbf339",false]},{"id":84251,"method":"getblock","params":["000000000000000000572995c8938b7df198bba4fedb0b8504e442a39d3601e3",false]},{"id":84251,"method":"getblock","params":["00000000000000000186d7c821f27c09464c7997a19036d31d4b022c97ab12ca",false]},{"id":84251,"method":"getblock","params":["0000000000000000017cf6276a890909982f507b7e8e646fa51223438762e210",false]},{"id":84251,"method":"getblock","params":["000000000000000000ae6c8d829f3ecc0e7dd4aceba568a7eb596f68aafebd69",false]},{"id":84251,"method":"getblock","params":["0000000000000000004c87dac4cd541d203e2f221d3c676f11c005ebd416f436",false]},{"id":84251,"method":"getblock","params":["000000000000000000c20f8a3ea20ccc71907994787ea1d5a6e6cd4f8aa65dfc",false]},{"id":84251,"method":"getblock","params":["000000000000000000bd8c9d1cec11cc80792d8a7e25993d711fe29099ec1e70",false]},{"id":84251,"method":"getblock","params":["00000000000000000182d0184efb1e329378a6eecfb93f8a3bebc9df7fc1b846",false]},{"id":84251,"method":"getblock","params":["000000000000000001dc8901d3fd152fbf7aa749588aca78183e47f98bfa0361",false]},{"id":84251,"method":"getblock","params":["000000000000000001ab457d3345447818373518407682ee993cb6a7c468b800",false]},{"id":84251,"method":"getblock","params":["00000000000000000134e0dabb631ce577b1d2bd3fd07529d1d8e92d6edefcc9",false]},{"id":84251,"method":"getblock","params":["000000000000000002721b0c026b24af0290ad12eb90dd055e12e4285e60b61f",false]},{"id":84251,"method":"getblock","params":["0000000000000000019ca83342fa0b318a30e5616b663dd03fb1d38722853767",false]},{"id":84251,"method":"getblock","params":["000000000000000000214f6f37ad06bdfeae69f846511fd84a5fe8e4136d3de3",false]},{"id":84251,"method":"getblock","params":["000000000000000001131227cba56dec47b6a69a043b87ab3acc6535e703fe7b",false]},{"id":84251,"method":"getblock","params":["000000000000000002079d661dc1ec13141d5004985974cd1400fcd2c8a81c54",false]},{"id":84251,"method":"getblock","params":["0000000000000000023c9ed52c4ed27e96cd1e5f53eca2d2c693e35091f87c18",false]},{"id":84251,"method":"getblock","params":["000000000000000002756f93f3e0246a36e0716415c2e38efb751967759ab060",false]},{"id":84251,"method":"getblock","params":["0000000000000000008220d0338f03a512dbab6c3801510ffd1c4e324398af01",false]},{"id":84251,"method":"getblock","params":["0000000000000000016866b1460672d7e862562cc37cfe42eca49822bdec54a4",false]},{"id":84251,"method":"getblock","params":["0000000000000000015d95e11dfce3c2c75261875025e804a98b35419698458e",false]},{"id":84251,"method":"getblock","params":["000000000000000000ecb7a9069f15e892fd8200fc7ed32015b684e1e959f880",false]},{"id":84251,"method":"getblock","params":["000000000000000001056cdcba2cc0eea79abdc694223085c615e0b7672d85c8",false]},{"id":84251,"method":"getblock","params":["000000000000000001e3db3cb5e7b66824e02614b935fcf91fde79e1a9a68039",false]},{"id":84251,"method":"getblock","params":["00000000000000000209cefa216ed76e3d2ffa15483edbc7785c668f10f518e1",false]},{"id":84251,"method":"getblock","params":["000000000000000000a3eda906f3482bca4fdc44c1455172c0f56b0b6489aa81",false]},{"id":84251,"method":"getblock","params":["0000000000000000005511b8312fa987764c05172eb7740aa8e48befc7caf11a",false]},{"id":84251,"method":"getblock","params":["000000000000000000a346ae6a7309e90eb9f079fe0b080ff58414d0d7a3411a",false]},{"id":84251,"method":"getblock","params":["000000000000000001b6a3249c66ce50610d8f14061372725044bb3d162b475d",false]},{"id":84251,"method":"getblock","params":["000000000000000000c162dcd38a0013badf14b5bfaab892faa86294392cb377",false]},{"id":84251,"method":"getblock","params":["000000000000000001bcee4fb91d457177b0c20581e0b63de90dd1b044f0eec2",false]},{"id":84251,"method":"getblock","params":["0000000000000000005e223fa18463ef66e0b8b1f6f7d7c141abecb9c06f6d9a",false]},{"id":84251,"method":"getblock","params":["0000000000000000014fa721a30cec4e7ce72a341f676fc9b22c8e41d6cdc0f9",false]},{"id":84251,"method":"getblock","params":["0000000000000000016d230dc2ffd8eb98ad14c5a5ad04245fa0cb42977d12ca",false]},{"id":84251,"method":"getblock","params":["000000000000000000b0e4d4babcb8f6799c0a7d33dea71f095e44eb4d5a6349",false]},{"id":84251,"method":"getblock","params":["00000000000000000207ad54f117034bb49d8c5128c63fe4289d29a2fb14ced2",false]},{"id":84251,"method":"getblock","params":["0000000000000000017666a32a27624bda261c6b3cb3b60c7924a75c4d6590ef",false]},{"id":84251,"method":"getblock","params":["0000000000000000023a7dfb642c52d65ec1c1bc61511d2771bf9aaf794b5a71",false]},{"id":84251,"method":"getblock","params":["00000000000000000026b72386cbbe789f530ed47f841c52ebafb8d364029475",false]},{"id":84251,"method":"getblock","params":["000000000000000002480d7f8865e433cf54b2439f97aa7a338ca792f55b784f",false]},{"id":84251,"method":"getblock","params":["000000000000000000351fac2df364687b51899896740583f50a70765e5926cd",false]},{"id":84251,"method":"getblock","params":["00000000000000000095237aea75f7c762c9b69ef23b0d0acfb3fc27699b760d",false]},{"id":84251,"method":"getblock","params":["00000000000000000095d70067c6bdcda55024fcf43b2b629ecc3214fbe8a681",false]},{"id":84251,"method":"getblock","params":["000000000000000000620897447da831fe6046bfc062f1f5ce9100c1f7ee851d",false]},{"id":84251,"method":"getblock","params":["0000000000000000027e2ed23fcca3fcbd895567c616a4e9190203be09a73a9d",false]},{"id":84251,"method":"getblock","params":["000000000000000001faa369d755bcbacb971f2807cec957e495ebce8e8b9bb5",false]},{"id":84251,"method":"getblock","params":["0000000000000000014f1e436f5daf530c6e6810713bf1270c64d0237ed2062d",false]},{"id":84251,"method":"getblock","params":["00000000000000000110e09d7f6fb5b4bb06cbed462f32b282f900639c8eaf33",false]},{"id":84251,"method":"getblock","params":["0000000000000000010e5ace3626873b43458d9055d184dca30f081ebd702424",false]},{"id":84251,"method":"getblock","params":["000000000000000002313a8e4eec1d7b597e41d22d5347bcbc3429b7155bd4f5",false]},{"id":84251,"method":"getblock","params":["000000000000000001b25b94dd4d628b9de046e26b53c9786be5b9a52683662f",false]},{"id":84251,"method":"getblock","params":["000000000000000001841b357586393707a0326a61622b43b18069d9b241c81a",false]},{"id":84251,"method":"getblock","params":["000000000000000001d01607ad2c9a5cc647f18dce1f064e2df9a6d1c12352f1",false]},{"id":84251,"method":"getblock","params":["0000000000000000025136ddd4cf47907473478b45482f78d6bca1761df967e0",false]},{"id":84251,"method":"getblock","params":["0000000000000000004c6bc2f9a89c414217421a31cee7fbac6ff1fe8dc38922",false]},{"id":84251,"method":"getblock","params":["000000000000000001bc619185c3bccf2c8894382c59ba4c76867b3215f53a72",false]},{"id":84251,"method":"getblock","params":["0000000000000000009f3c658f72ad7abc0d52e6ab0b13f33881f761009a2afa",false]},{"id":84251,"method":"getblock","params":["0000000000000000025485c6a58ecde13772bb9438b9693908736083c1028992",false]},{"id":84251,"method":"getblock","params":["000000000000000001dcab3c9961dcaff81e21851acf28c79bffc1e5b645e073",false]},{"id":84251,"method":"getblock","params":["0000000000000000017b911c332a70089f1260569905c2fd1dda46fc35272f58",false]},{"id":84251,"method":"getblock","params":["000000000000000001b844111d40c1582c131a1ee4ae105607e67684a0bb8bfd",false]},{"id":84251,"method":"getblock","params":["000000000000000001fe10a77d40a339651162a041445ea3c91197143ee41039",false]},{"id":84251,"method":"getblock","params":["0000000000000000010a2638c69028ea328bde155e3ad29d7b5c72036c8aae8f",false]},{"id":84251,"method":"getblock","params":["000000000000000001208f2bec0de18e983ae506d31ea8597fc6a92c440aa731",false]},{"id":84251,"method":"getblock","params":["0000000000000000015beb647799c06b348f914ce96ccae4066fc25f68314a3c",false]},{"id":84251,"method":"getblock","params":["00000000000000000007d205b68eb98437b8780324eb35d3d4fe479577711ded",false]},{"id":84251,"method":"getblock","params":["000000000000000001a213fb5ca2178a00a44ab2b8cdc8c8b081ea3827c81088",false]},{"id":84251,"method":"getblock","params":["000000000000000001cec9e0bb10cef09d48ce5ce2781e9d5b062f5e30cc79c7",false]}]
Caused by: disconnected from daemon

Use correct bucketing for various histograms

The default one is sub-optimal for bytes-based metrics.

Too many open files.

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "IO error: While opendir: /mnt/DATA/.btc.n0de/electrs/mainnet: Too many open files"', libcore/result.rs:945:5

Use P2P connection for new blocks' notifications

To decrease the latency between getting a new block and notifying the Electrum wallet.
(https://github.com/rust-bitcoin/rust-bitcoin/blob/master/src/network/listener.rs)

Travis build fails due to '/usr/bin/ld: final link failed: No space left on device'

https://travis-ci.com/romanz/electrs/builds/79488852#L513

Invalid UTF8 error on connection with Electrum client

Setup:
Electrum client: v3.2.2 from Windows 10
electrs server: running in Windows Linux Subsystem

server has synced up ok... but spits out multiple errors regarding invalid UTF8 when attempting to connect with an Electrum client:

2018-07-20T23:57:37.253+12:00 - INFO - RPC server running on 127.0.0.1:50001
2018-07-20T23:57:42.259+12:00 - TRACE - 438 transactions in mempool (+6)
2018-07-20T23:57:47.271+12:00 - TRACE - 454 transactions in mempool (+16)
2018-07-20T23:57:52.285+12:00 - TRACE - 472 transactions in mempool (+18)
2018-07-20T23:57:55.658+12:00 - INFO - [127.0.0.1:62716] connected peer
2018-07-20T23:57:55.667+12:00 - TRACE - RPC Done
2018-07-20T23:57:55.668+12:00 - INFO - [127.0.0.1:62716] shutting down connection
2018-07-20T23:57:55.669+12:00 - ERROR - [127.0.0.1:62716] receiver failed: invalid UTF8 "\u{16}\u{3}\u{1}\u{2}\u{0}\u{1}\u{0}\u{1}�\u{3}\u{3}PP���Y\u{6}\"o�(�)V\\��(���K_3�}<\u{12}k0O\u{0}\u{0}��0�,�2�.�/�+�1�-\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}��(�$�\u{14}�\n": FromUtf8Error { bytes: [22, 3, 1, 2, 0, 1, 0, 1, 252, 3, 3, 80, 80, 254, 183, 144, 89, 6, 34, 111, 225, 175, 40, 187, 41, 86, 92, 238, 229, 40, 140, 138, 177, 75, 95, 51, 187, 125, 60, 18, 107, 48, 79, 0, 0, 130, 192, 48, 192, 44, 192, 50, 192, 46, 192, 47, 192, 43, 192, 49, 192, 45, 0, 165, 0, 163, 0, 161, 0, 159, 0, 164, 0, 162, 0, 160, 0, 158, 192, 40, 192, 36, 192, 20, 192, 10], error: Utf8Error { valid_up_to: 8, error_len: Some(1) } }
2018-07-20T23:57:55.671+12:00 - INFO - [127.0.0.1:62716] disconnected peer
2018-07-20T23:57:55.679+12:00 - INFO - [127.0.0.1:62717] connected peer
2018-07-20T23:57:55.688+12:00 - TRACE - RPC Done
2018-07-20T23:57:55.689+12:00 - INFO - [127.0.0.1:62717] shutting down connection
2018-07-20T23:57:55.689+12:00 - ERROR - [127.0.0.1:62717] receiver failed: invalid UTF8 "\u{16}\u{3}\u{1}\u{2}\u{0}\u{1}\u{0}\u{1}�\u{3}\u{3}\u{16}{��Q��&�=\u{4}G\u{1}��\u{f}�^���rm��,B|\u{1a}�6\u{0}\u{0}\u{0}��0�,�2�.�/�+�1�-\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}��(�$�\u{14}�\n": FromUtf8Error { bytes: [22, 3, 1, 2, 0, 1, 0, 1, 252, 3, 3, 22, 123, 158, 186, 81, 170, 189, 38, 221, 61, 4, 71, 1, 255, 233, 15, 177, 94, 252, 213, 208, 114, 109, 146, 128, 44, 66, 124, 26, 128, 54, 0, 0, 0, 130, 192, 48, 192, 44, 192, 50, 192, 46, 192, 47, 192, 43, 192, 49, 192, 45, 0, 165, 0, 163, 0, 161, 0, 159, 0, 164, 0, 162, 0, 160, 0, 158, 192, 40, 192, 36, 192, 20, 192, 10], error: Utf8Error { valid_up_to: 8, error_len: Some(1) } }
2018-07-20T23:57:55.690+12:00 - INFO - [127.0.0.1:62717] disconnected peer
2018-07-20T23:57:57.310+12:00 - TRACE - 487 transactions in mempool (+15)

ERROR - server failed: Error: invalid blockhash

1b4adfd was merged before:

2018-07-16T14:01:47.411+02:00 - WARN - no mempool entry 05946c6eff46ffdfbba92f05bdb64553dcbd4fec70bd60d818b37e23d67816ce: missing fees section
2018-07-16T14:01:47.412+02:00 - WARN - no mempool entry 995e64664ca0aaeac8ba5c7a642ab103232cd4c0a2d7393ef7ee2c0f0981cd8a: missing fees section
2018-07-16T14:01:47.412+02:00 - WARN - no mempool entry 86eba5549c40831c68c3721dbdca21bc8add924c37ec892436f22cbc83a26c74: missing fees section
2018-07-16T14:01:47.412+02:00 - WARN - no mempool entry 02984b91d3926d84e9dd756e05b9f11e0ef4bfbfc55f03789f32f23b9d3bcfab: missing fees section
2018-07-16T14:01:47.412+02:00 - TRACE - no mempool entries found!
2018-07-16T14:01:52.413+02:00 - ERROR - server failed: Error: invalid blockhash
Caused by: non-string value: {"ancestorcount":1,"ancestorfees":672,"ancestorsize":167,"depends":[],"descendantcount":2,"descendantfees":15662,"descendantsize":1773,"fee":0.00000672,"height":532141,"modifiedfee":0.00000672,"size":167,"time":1531737113,"wtxid":"1604b7230b5316f886d7d6e46dc9dad3c779316ef0ade30115c973c36202af62"}
stack backtrace:
   0:     0x55f756231dec - backtrace::backtrace::trace::hd402417859783b40
   1:     0x55f756231442 - backtrace::capture::Backtrace::new_unresolved::hce1354e135c2f28e
   2:     0x55f756230d17 - error_chain::backtrace::imp::InternalBacktrace::new::habdd565038f6f85c
   3:     0x55f756231084 - <error_chain::State as core::default::Default>::default::hee976682d00eb04c
   4:     0x55f7561f54a8 - electrs::daemon::parse_hash::h93f9be35b39a7f95
   5:     0x55f7561f9fce - electrs::daemon::Daemon::getbestblockhash::h897efc5c99f6fa52
   6:     0x55f7561d43b6 - electrs::app::App::update::h9aa7658f54799234
   7:     0x55f756159cf2 - electrs::main::hb7893ebc8a38666e
   8:     0x55f75615efb2 - std::rt::lang_start::{{closure}}::h825937f5b6ee0646
   9:     0x55f756662d82 - std::rt::lang_start_internal::{{closure}}::h342eaeec3a520255
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::h8b090b6144a018fb
                        at libstd/panicking.rs:310
  10:     0x55f756680ce9 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:105
  11:     0x55f75666a275 - std::panicking::try::h11349d7f0dce3b3c
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::hfbb0f172d1ed4df2
                        at libstd/panic.rs:374
                         - std::rt::lang_start_internal::h6c73689767a9401e
                        at libstd/rt.rs:58
  12:     0x55f75615a4c3 - main
  13:     0x7f3d13d9082f - __libc_start_main
  14:     0x55f756157b58 - _start
  15:                0x0 - <unknown>

electrs can't be killed with control + c when it is disconnected from full node rpc

For example when elects is logging things like


2018-07-28T21:19:33.989+00:00 - WARN - connection failed: failed to read cookie from "/home/example/.bitcoin/.cookie"
2018-07-28T21:19:36.990+00:00 - WARN - failed to connect daemon at 127.0.0.1:8332: Connection refused (os error 111)
2018-07-28T21:19:39.990+00:00 - WARN - failed to connect daemon at 127.0.0.1:8332: Connection refused (os error 111)
2018-07-28T21:19:42.990+00:00 - WARN - failed to connect daemon at 127.0.0.1:8332: Connection refused (os error 111)
2018-07-28T21:19:45.991+00:00 - WARN - failed to connect daemon at 127.0.0.1:8332: Connection refused (os error 111)
2018-07-28T21:19:48.991+00:00 - WARN - failed to connect daemon at 127.0.0.1:8332: Connection refused (os error 111)
2018-07-28T21:19:51.992+00:00 - WARN - connection failed: request failed "HTTP/1.1 500 Internal Server Error"

To reproduce: connect electrs to full node, shut down the full node. Now electrs can't be killed with control + c

Bulk indexing should skip already indexed blocks

Otherwise, the DB will grow forever (#11) in case one of the blk*.dat files fails to parse (#13).

mainnet, master, electrs dies

mainnet, full node synchronized and electrs synchronized

After a while the server dies because of some probably related to #18

2018-07-15T21:56:23.362+00:00 - WARN - no mempool entry 2b85b3dfdfb0f95644a49a154c720135effcee26d8b5c658376f43d7fd8dd695: missing fees section
2018-07-15T21:56:23.362+00:00 - ERROR - server failed: Error: non-array response
stack backtrace:
   0:     0x5618e5e05dac - backtrace::backtrace::trace::h44f27ad6608f02da
   1:     0x5618e5e05402 - backtrace::capture::Backtrace::new_unresolved::hb555da2dec445946
   2:     0x5618e5e04cd7 - error_chain::backtrace::imp::InternalBacktrace::new::hc1c04021afee31b3
   3:     0x5618e5e05044 - <error_chain::State as core::default::Default>::default::he24c386d1d298d64
   4:     0x5618e5d4187e - electrs::daemon::Daemon::requests::h9854bd24136c8b73
   5:     0x5618e5d43eed - electrs::daemon::Daemon::gettransactions::h93b25cda33204a41
   6:     0x5618e5d82ce0 - electrs::mempool::Tracker::update::h9fe07ffc4b13e9f1
   7:     0x5618e5d6002b - electrs::query::Query::update_mempool::h5a3567190cf4c2bb
   8:     0x5618e5d31df6 - electrs::main::h5294e30fc465d8cf
   9:     0x5618e5d30782 - std::rt::lang_start::{{closure}}::hc74e7e3dcbd83f22
  10:     0x5618e61e24b2 - std::rt::lang_start_internal::{{closure}}::h72602870b4a34363
                        at libstd/rt.rs:59
                         - std::panicking::try::do_call::hc2917ab2c572e22d
                        at libstd/panicking.rs:310
  11:     0x5618e6200469 - __rust_maybe_catch_panic
                        at libpanic_unwind/lib.rs:105
  12:     0x5618e61e99a5 - std::panicking::try::h3777f1f99a655493
                        at libstd/panicking.rs:289
                         - std::panic::catch_unwind::hab6d1467193fc938
                        at libstd/panic.rs:374
                         - std::rt::lang_start_internal::h106bcdfb2e3beaf8
                        at libstd/rt.rs:58
  13:     0x5618e5d32553 - main
  14:     0x7f4d433072b0 - __libc_start_main
  15:     0x5618e5d2a619 - _start
  16:                0x0 - <unknown>

Parse blk*.dat files for initial bulk indexing

Should achieve ~100MB/s on a standard HDD.

Server panics upon Electrum client connection

Following #9 (comment), the server panics with:

thread 'reader' panicked at 'failed to read a request: \
 Custom { kind: InvalidData, error: StringError("stream did not contain valid UTF-8") }', \
  libcore/result.rs:945:5

2018-08-07T08:43:13.273+12:00 - ERROR - [127.0.0.1:63592] connection handling failed: Error: unknown method blockchain.scripthash.listunspent [String("033b.....07fb")]

The client then shows as offline (red dot) and the errors continue to be logged by the server until the client is shutdown.

Allow the user to reduce RAM usage

It seems that most of the memory is used by RocksDB caches.

(see #11 for more details)

Improve error message when connecting with SSL

Current server supports only TCP connections - and the error message is not helpful at all:

ERROR - [192.168.1.201:32910] receiver failed: invalid UTF8 "\u{16}\u{3}\u{1}\u{2}\u{0}\u{1}\u{0}\u{1}�\u{3}\u{3}\u{f}�1�����4:��1ċB\u{16}�6q�y��W?45�k�V\u{0}\u{0}��0�,�2�.�/�+�1�-\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}�\u{0}��(�$�\u{14}�\n": FromUtf8Error { bytes: [22, 3, 1, 2, 0, 1, 0, 1, 252, 3, 3, 15, 218, 49, 163, 186, 179, 241, 249, 52, 58, 245, 155, 49, 196, 139, 66, 22, 191, 54, 113, 136, 121, 226, 223, 87, 63, 52, 53, 169, 107, 216, 86, 0, 0, 130, 192, 48, 192, 44, 192, 50, 192, 46, 192, 47, 192, 43, 192, 49, 192, 45, 0, 165, 0, 163, 0, 161, 0, 159, 0, 164, 0, 162, 0, 160, 0, 158, 192, 40, 192, 36, 192, 20, 192, 10], error: Utf8Error { valid_up_to: 8, error_len: Some(1) } }

Following #32