Git Product home page Git Product logo

electrsd's Introduction

MIT license Crates

Electrsd

Utility to run a regtest electrs process connected to a given bitcoind instance, useful in integration testing environment.

let bitcoind = bitcoind::BitcoinD::new("/usr/local/bin/bitcoind").unwrap();
let electrsd = electrsd::ElectrsD::new("/usr/local/bin/electrs", bitcoind).unwrap();
let header = electrsd.client.block_headers_subscribe().unwrap();
assert_eq!(header.height, 0);

Automatic binaries download

In your project Cargo.toml, activate the following features

electrsd = { version= "0.23", features = ["bitcoind_23_1", "electrs_0_9_1"] }

Then use it:

let bitcoind_exe = bitcoind::downloaded_exe_path().expect("bitcoind version feature must be enabled");
let bitcoind = bitcoind::BitcoinD::new(bitcoind_exe).unwrap();
let electrs_exe = electrsd::downloaded_exe_path().expect("electrs version feature must be enabled");
let electrsd = electrsd::ElectrsD::new(electrs_exe, bitcoind).unwrap();

When the ELECTRSD_DOWNLOAD_ENDPOINT/BITCOIND_DOWNLOAD_ENDPOINT environment variables are set, electrsd/bitcoind will try to download the binaries from the given endpoints.

When you don't use the auto-download feature you have the following options:

  • have electrs executable in the PATH
  • provide the electrs executable via the ELECTRS_EXEC env var
if let Ok(exe_path) = electrsd::exe_path() {
  let electrsd = electrsd::electrsD::new(exe_path).unwrap();
}

Startup options could be configured via the Conf struct using electrsD::with_conf or electrsD::from_downloaded_with_conf.

Nix

For determinisim, in nix you cannot hit the internet within the build.rs. Moreover, some downstream crates cannot remove the auto-download feature from their dev-deps. In this case you can set the ELECTRSD_SKIP_DOWNLOAD env var and provide the electrs executable in the PATH (or skip the test execution).

Issues with traditional approach

I used integration testing based on external bash script launching needed external processes, there are many issues with this approach like:

Features

  • electrsd use a temporary directory as db dir
  • A free port is asked to the OS (a very low probability race condition is still possible)
  • The process is killed when the struct goes out of scope no matter how the test finishes
  • Automatically download electrs executable with enabled features. Since there are no official binaries, they are built using the manual workflow under this project. Supported version are:

Thanks to these features every #[test] could easily run isolated with its own environment

Deprecations

  • Starting from version 0.26 the env var ELECTRS_EXE is deprecated in favor of ELECTRS_EXEC.

Used by

electrsd's People

Contributors

afilini avatar leocomandini avatar llfourn avatar maxfangx avatar noib3 avatar notmandatory avatar oleonardolima avatar rajarshimaitra avatar rcasatta avatar realeinherjar avatar tcharding avatar tnull avatar ulrichard avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

electrsd's Issues

Download using http proxy

It would be really cool, if downloading the executables would use a proxy if one of the following environmant variables are set:

  • HTTP_PROXY
  • HTTPS_PROXY

Implements `ElectrsD::wait_tx` and `ElectrsD::wait_block`

It's frequent in test code to have bitcoind to generate blocks and wallets to send transactions and then tests must assert things only if electrs indexed the block or transaction, so code that busy wait on that is repeated in downstream test codes.

Features are not additive

The features are not additive, this means running cargo test --all-features doesn't work.

The problem is the electrs/esplora features used to define VERSION, perhaps there is another way?

#[cfg(feature = "electrs_0_8_10")]
const VERSION: &str = "v0.8.10";

#[cfg(feature = "esplora_a33e97e1")]
const VERSION: &str = "esplora_a33e97e1a1fc63fa9c20a116bb92579bbf43b254";

#[cfg(feature = "electrs_0_9_1")]
const VERSION: &str = "v0.9.1";

#[cfg(feature = "electrs_0_9_11")]
const VERSION: &str = "v0.9.11";

v0.13.0 build failure due to bitcoind incompatibility

Due to recent changes made in https://github.com/RCasatta/bitcoind/pulls?q=is%3Apr+is%3Aclosed the current master build is failing.

$ cargo build
   Compiling bitcoind v0.21.0
error: cannot find derive macro `Display` in this scope
  --> /home/raj/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.21.0/src/lib.rs:71:17
   |
71 | #[derive(Debug, Display)]
   |                 ^^^^^^^

error: could not compile `bitcoind` due to previous error

Because master still points to bitcoind v0.21.0 while having its Error std::error::Error.

Also because the v0.13.0 is not tagged yet, I think the electrsd v0.13.0 in cargo is actually building the current master so I am seeing the failure in downstream too.

Simple fix for people with the same issue : Use electrsd v0.12.0 in your Cargo.toml until v0.13.0 is tagged.

`test_electrsd` test fails with bitcoind version `23_0`

In the process of debugging a problem I discovered that the electrsd::test::test_electrsd test fails if the bitcoind_23_0 feature flag is used in place of bitcoind_22_0.

22_0 passes as expected:

cargo test --features bitcoind_22_0,esplora_a33e97e1,legacy                                                                                          (0.22.1)
   Compiling bitcoind v0.28.1
   Compiling electrsd v0.22.1 (/Users/fang/lexe/dev/electrsd)
    Finished test [unoptimized + debuginfo] target(s) in 10.58s
     Running unittests src/lib.rs (target/debug/deps/electrsd-cf3fbf42b92914db)

running 2 tests
test test::test_kill ... ok
test test::test_electrsd ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.48s

23_0 fails with a WouldBlock socket error:

$ cargo test --features bitcoind_23_0,esplora_a33e97e1,legacy
    Finished test [unoptimized + debuginfo] target(s) in 0.15s
     Running unittests src/lib.rs (target/debug/deps/electrsd-18608eb7c9663c80)

running 2 tests
test test::test_kill ... ok
test test::test_electrsd ... FAILED

failures:

---- test::test_electrsd stdout ----
thread 'test::test_electrsd' panicked at 'called `Result::unwrap()` on an `Err` value: JsonRpc(Transport(SocketError(Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" })))', src/lib.rs:398:60
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/result.rs:1805:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/result.rs:1098:23
   4: electrsd::test::test_electrsd
             at ./src/lib.rs:398:9
   5: electrsd::test::test_electrsd::{{closure}}
             at ./src/lib.rs:393:5
   6: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
   7: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    test::test_electrsd

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 31.27s

error: test failed, to rerun pass '--lib'

I am running this tests on electrsd version 0.22.1 on an M1 Mac, Rust version 1.66. I have seen the bitcoind_23_0,esplora_a33e97e1,legacy feature flag combination elsewhere so at this point I am not sure if it's a problem with my specific environment or if it applies to all users.

add --txid-limit flag and set to 0

@sandipndev found that electrs by default will throw an error if trying to sync an address that has more than 100 transactions against it. This error can be avoided by setting the --txid-limit flag to 0. I think it would be good to set for flag for this project. Some more details here:

bitcoindevkit/bdk#406

FTB on Windows

CI says:

error[E0433]: failed to resolve: could not find `unix` in `os`
 --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\electrsd-0.22.2\build.rs:4:14
  |
4 | use std::os::unix::fs::PermissionsExt;
  |              ^^^^ could not find `unix` in `os`

error[E0425]: cannot find value `OS` in this scope
  --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\electrsd-0.22.2\src/versions.rs:30:30
   |
30 |     format!("electrs_{}_{}", OS, VERSION)
   |                              ^^
  --> /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74\library\core\src\result.rs:507:5
   |
   = note: similarly named tuple variant `Ok` defined here
   |
help: a tuple variant with a similar name exists
   |
30 |     format!("electrs_{}_{}", Ok, VERSION)
   |                              ~~
help: consider importing this constant
  --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\electrsd-0.22.2\build.rs:1:1
   |
1  | use std::env::consts::OS;
   |

     Running `D:\a\rust-lightning\rust-lightning\lightning-transaction-sync\target\debug\build\bitcoind-b1632355beeb3a81\build-script-build`
error[E0599]: no function or associated item named `from_mode` found for struct `Permissions` in the current scope
  --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\electrsd-0.22.2\build.rs:67:35
   |
67 |             std::fs::Permissions::from_mode(0o755),
   |                                   ^^^^^^^^^ function or associated item not found in `Permissions`

Some errors have detailed explanations: E0425, E0433, E0599.
For more information about an error, try `rustc --explain E0425`.
error: could not compile `electrsd` due to 3 previous errors

Update bitcoind dependency to 0.21.0 to fix M1 builds

same M1 build failure that (rust-bitcoin/bitcoind#38) solved is causing electrsd not to build on M1, which is causing bitcoindevkit build failures on my machine

Compiling electrsd v0.12.0 (/Users/futurepaul/dev/rust/other-peoples-code/electrsd)
error[E0425]: cannot find function `download_filename` in this scope
  --> /Users/futurepaul/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.20.0/build.rs:49:29
   |
49 |     let download_filename = download_filename();
   |                             ^^^^^^^^^^^^^^^^^ not found in this scope

error[E0277]: the size for values of type `str` cannot be known at compilation time
   --> /Users/futurepaul/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.20.0/build.rs:62:9
    |
62  | /         println!(
63  | |             "filename:{} version:{} hash:{}",
64  | |             download_filename, VERSION, expected_hash
65  | |         );
    | |__________^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `str`
note: required by a bound in `ArgumentV1::<'a>::new`
   --> /Users/futurepaul/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:293:20
    |
293 |     pub fn new<'b, T>(x: &'b T, f: fn(&T, &mut Formatter<'_>) -> Result) -> ArgumentV1<'b> {
    |                    ^ required by this bound in `ArgumentV1::<'a>::new`
    = note: this error originates in the macro `$crate::format_args_nl` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the size for values of type `str` cannot be known at compilation time
   --> /Users/futurepaul/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.20.0/build.rs:67:19
    |
67  |           let url = format!(
    |  ___________________^
68  | |             "https://bitcoincore.org/bin/bitcoin-core-{}/{}",
69  | |             VERSION, download_filename
70  | |         );
    | |_________^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `str`
note: required by a bound in `ArgumentV1::<'a>::new`
   --> /Users/futurepaul/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:293:20
    |
293 |     pub fn new<'b, T>(x: &'b T, f: fn(&T, &mut Formatter<'_>) -> Result) -> ArgumentV1<'b> {
    |                    ^ required by this bound in `ArgumentV1::<'a>::new`
    = note: this error originates in the macro `$crate::__export::format_args` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0277, E0425.
For more information about an error, try `rustc --explain E0277`.

Allow using non-temp data dir

To create automated regtest backend for bdk-cli (bitcoindevkit/bdk-cli#76) we need to have the option to create a non-temp data dir, inside the bdk wallet directory, so the node states can be made persistent between runs..

This issue is opened to get generic approach on doing that.. I am thinking of making the Electrsd data_dir optional, which defaults to temp directory if provide with None.

Happy to open a PR on both eletrsd and bitcoind if this sounds reasonable..

failed to select a version for `secp256k1-sys`

bumping bitcoind to 0.27.0 cause this error

error: failed to select a version for `secp256k1-sys`.
    ... required by package `secp256k1 v0.22.0`
    ... which satisfies dependency `secp256k1 = "^0.22.0"` of package `bitcoin v0.28.0`
    ... which satisfies dependency `bitcoin = "^0.28"` of package `electrum-client v0.10.2 (/home/casatta/git/rust-electrum-client)`
    ... which satisfies path dependency `electrum-client` of package `electrsd v0.20.0 (/home/casatta/git/electrsd)`
versions that meet the requirements `^0.5.0` are: 0.5.1, 0.5.2, 0.5.0

the package `secp256k1-sys` links to the native library `rustsecp256k1_v0_5_0`, but it conflicts with a previous package which links to `rustsecp256k1_v0_5_0` as well:
package `secp256k1-sys v0.6.0`
    ... which satisfies dependency `secp256k1-sys = "^0.6.0"` of package `secp256k1 v0.24.0`
    ... which satisfies dependency `secp256k1 = "^0.24.0"` of package `bitcoin v0.29.1`
    ... which satisfies dependency `bitcoin = "^0.29.0"` of package `bitcoincore-rpc-json v0.16.0`
    ... which satisfies dependency `bitcoincore-rpc-json = "^0.16.0"` of package `bitcoincore-rpc v0.16.0`
    ... which satisfies dependency `bitcoincore-rpc = "^0.16.0"` of package `bitcoind v0.27.0`
    ... which satisfies dependency `bitcoind = "^0.27.0"` of package `electrsd v0.20.0 (/home/casatta/git/electrsd)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the links ='secp256k1-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `secp256k1-sys` which could resolve this conflict

add devenv or nix shell

Kinda hard to guess which versions this crate works with, would be great to provide a devenv.nix or nix shell that builds the specific versions that its expecting and use that rather than installing and downloading things separately (I couldn't get it to work on macos).

Once I get this to work, I'm happy to submit that

{ pkgs, lib, config, inputs, ... }:

let
  customBitcoind = pkgs.callPackage ./bitcoind.nix {};
in
{
  env.DATABASE_URL="postgres";

 packages = [
    pkgs.git
    pkgs.just
    pkgs.openssl
    pkgs.cargo-nextest
    pkgs.hurl
    pkgs.sqlx-cli
    pkgs.tailwindcss
    pkgs.cargo-watch
    pkgs.electrs
    customBitcoind
  ] ++ lib.optionals pkgs.stdenv.isDarwin (with pkgs.darwin.apple_sdk; [
    frameworks.CoreFoundation
    frameworks.Security
    frameworks.SystemConfiguration
  ]);

  enterShell = ''
    git --version
    cargo --version
    alias c="cargo"
    bitcoind --version  # Check Bitcoin Core version
    export BITCOIND_EXE=$(which bitcoind)
    echo $BITCOIND_EXE
    export ELECTRS_EXEC=$(which electrs)
    echo $ELECTRS_EXEC
  '';

  # https://devenv.sh/pre-commit-hooks/
  pre-commit.hooks = {
    clippy.enable = true;
    cargo-check.enable = true;
    rustfmt.enable = true;
  };

  # https://devenv.sh/languages/
  languages.nix.enable = true;
  languages.rust = {
    enable = true;
    channel = "stable";
  };

  services.postgres = {
    enable = true;
    # initialDatabases = [{ name = "keel"; }];
  };
}
{ lib
, stdenv
, fetchurl
, pkg-config
, autoreconfHook
, boost
, libevent
, miniupnpc
, zeromq
, zlib
, db48
, sqlite
, qrencode
, python3
, util-linux
, darwin
}:

stdenv.mkDerivation rec {
  pname = "bitcoind";
  version = "25.0";

  src = fetchurl {
    url = "https://bitcoincore.org/bin/bitcoin-core-${version}/bitcoin-${version}.tar.gz";
    sha256 = "sha256-XfZ89CyjuaDDjNr+xbu1F9pbWNJR8yyNKkdRH5vh68I=";
  };

  nativeBuildInputs = [ pkg-config autoreconfHook ];
  buildInputs = [
    boost
    libevent
    miniupnpc
    zeromq
    zlib
    db48
    sqlite
    qrencode
    python3
    util-linux
  ] ++ lib.optionals stdenv.isDarwin [ darwin.apple_sdk.frameworks.Foundation ];

  configureFlags = [
    "--with-boost-libdir=${boost.out}/lib"
    "--disable-bench"
    "--disable-tests"
    "--disable-gui-tests"
    "--disable-fuzz-binary"
  ];

  enableParallelBuilding = true;

  meta = with lib; {
    description = "Bitcoin Core daemon";
    longDescription = ''
      Bitcoin is a free open source peer-to-peer electronic cash system that is
      completely decentralized, without the need for a central server or trusted
      parties. Users hold the crypto keys to their own money and transact directly
      with each other, with the help of a P2P network to check for double-spending.
    '';
    homepage = "https://bitcoincore.org/";
    maintainers = with maintainers; [ roconnor ];
    license = licenses.mit;
    platforms = platforms.unix;
  };
}

Introduce and enforce MSRV

A dependency of zip recently bumped the rustc requirement to 1.6, leading to MSRV violations in downstream projects such as LDK (cf. lightningdevkit/rust-lightning#2055) and BDK:

error: package `base64ct v1.6.0` cannot be built because it requires rustc 1.60 or newer, while the currently active rustc version is 1.57.0

It would be great if this crate and bitcoind could pin an old version of zip and enforce a reasonable MSRV, e.g., 1.48, which would allow us to keep using them.

Happy to open corresponding PRs after a concept ACK.

Problem with docker

When I run the unit test inside a docker container, the test freezes and never finishes.
To try, checkout https://github.com/weareseba/electrsd/tree/docker and run docker.sh

ps lists an electrs process that is defunct:

$ ps aux | grep electrs
richi      57351  0.0  0.1 2307184 55488 pts/5   Sl+  10:18   0:00 docker run -it rust-electrsd cargo test --features trigger,bitcoind_22_0,electrs_0_8_10 -- --nocapture
root       57401  0.1  0.0 1181080 24224 pts/0   Ss+  10:18   0:00 /usr/local/rustup/toolchains/1.55.0-x86_64-unknown-linux-gnu/bin/cargo test --features trigger,bitcoind_22_0,electrs_0_8_10 -- --nocapture
root       59725  0.0  0.0  76704  3892 pts/0    Sl+  10:19   0:00 /electrsd/target/debug/deps/electrsd-14536ff35a3aeaf8 --nocapture
root       59727  0.3  0.2 1927748 81688 pts/0   SLl+ 10:19   0:01 /electrsd/target/debug/build/bitcoind-fb3f177d66da844f/out/bitcoin/bitcoin-22.0/bin/bitcoind -datadir=/tmp/.tmp8vtftO -rpcport=34709 -listen=0 -regtest -fallbackfee=0.0001
root       59757  0.0  0.0      0     0 pts/0    Z+   10:19   0:00 [electrs] <defunct>

After inserting some println!() statements, I figured out, that the loop

let client = loop {

never terminates.

If I modify the loop as in #32, the test terminates with an error instead of the infinite loop. That's much better, but I still don't know why electrsd didn't start.

Electrs doesn't seem to have problems with docker in general, as manually running the commands at the bottom of
https://github.com/romanz/electrs/blob/master/.github/workflows/rust.yml
runs just fine.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.