bitcoindevkit / bdk-cli Goto Github PK

The bdk-cli --help command in electrum mode is little weird.

$ ./target/debug/bdk-cli --help
BDK CLI 0.5.0
Alekos Filini <[email protected]>:Riccardo Casatta <[email protected]>:Steve Myers <[email protected]>
Top level options and command modes

USAGE:
    bdk-cli [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       
            Prints help information

    -V, --version    
            Prints version information


OPTIONS:
    -n, --network <NETWORK>    
            Sets the network [default: testnet]


SUBCOMMANDS:
    help      Prints this message or the help of the given subcommand(s)
    key       Key management sub-commands
    repl      Electrum options
    wallet    Electrum options

Both repl and wallet are marked as Electrum options.

Only happening in electrum feature.

Issue

Change from a wallet is not being picked up in get_balance when both change and deposit descriptors are used together.
When Change descriptor is used alone, funds are detected.

Reproduction Steps

• Account 0 Deposit Descriptor

wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/0/*)

• Account 0 Change Descriptor

wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)

• bdk-cli get_balance

Ignoring sync

bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/0/*)" -c "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance 
{
  "satoshi": 1253
}

bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance
{
  "satoshi": 1154933
}

failure is officially deprecated/unmaintained

The failure crate is officially end-of-life: it has been marked as deprecated
by the former maintainer, who has announced that there will be no updates or
maintenance work on it going forward.

The following are some suggested actively developed alternatives to switch to:

• anyhow
• eyre
• fehler
• snafu
• thiserror

See advisory page for additional details.

net2 crate has been deprecated; use socket2 instead

The net2 crate has been deprecated
and users are encouraged to considered socket2 instead.

See advisory page for additional details.

ansi_term is Unmaintained

The maintainer has adviced this crate is deprecated and will not
receive any maintenance.

The crate does not seem to have much dependencies and may or may not be ok to use as-is.

Last release seems to have been three years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• anstyle
• nu-ansi-term
• yansi

See advisory page for additional details.

The work needs to be done in two parts:

• Create non tmp datadir for regtest node data in ~/.bdk-bitcoin.
• Open up basic rpc calls via repl commands to do basic node operations.

Current master (4ede4a4ad0bbf1b9c10fda9bea33a34cfb9587ba) doesn't warn the user in case of wrong command, like:

RUST_LOG=info cargo run --release --example repl --features cli-utils,esplora,electrum -- --wallet single_change_deep --descriptor "wpkh(tpubD6NzVbkrYhZ4YmSHJMXPEvd6dnPgH55EwGBs2AJHyiBgnT7zJzQ1ywHDxxZveoApLeBeSFcjysjQ5PebSg4gsdrVdRxCLAgHK8jKUiydMrg/*)" sinc 

while for example a while back (ae16c8b) an error with suggestions was shown

error: The subcommand 'sinc' wasn't recognized
	Did you mean 'sign'?

probably due to structopt migration

Just for completeness. After a few minutes in the repl I tried exiting and ended up realizing that there were no commands for it and I had to ctrl+d out.

An exit subcommand would be a nice touch.

Currently we are only calling specific core RPC calls in the node commands.

We should move on to generic rpc calls that can handle any valid bitcoin-cli commands.

And we should only include composite commands (that combines multiple rpc calls) and have the in node command list.

Issue

A wallet using electrum node does not sync on first or second try and requires another command - in this case get_new_address to trigger a correct sync.

Reproduction Steps

• Clear bdk-bitcoin working directory
• Run sync and get_balance (n number of times)
• Get 0 balance
• Run get_new_address (once)
• Rerun sync and get_balance (once)
• Get correct balance

~/Code ❯ rm -ri ~/.bdk-bitcoin
                                                                                                                                                            
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance    
{
  "satoshi": 0
}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" sync       
{}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance
{
  "satoshi": 0
}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" sync          
{}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance 
{
  "satoshi": 0
}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_new_address
{
  "address": "tb1q883xtcfqlw0744rwx3c583lujhn8mpfnr35pvx"
}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" sync           
{}
~/Code ❯ bdk-cli wallet -d "wpkh([8099ce1e/84h/1h/0h]tpubDCBjCC5aZ6wXLtZMSJDkBYZ3AFuors2YzzBhD5ZqP3uPqbzzH5YjD2CA9HDhUYNhrqq67v4XAN93KSbSL4bwa5hEvidkFuj7ycWA7EYzp41/1/*)" get_balance    
{
  "satoshi": 1154933
}

Attempting to install the cli from Cargo returns the following error:

cargo install bdk-cli
    Updating crates.io index
  Installing bdk-cli v0.2.0
  Downloaded bitcoin v0.26.2
  Downloaded 1 crate (269.9 KB) in 12.42s
   Compiling proc-macro2 v1.0.27
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.72
   Compiling version_check v0.9.3
   Compiling libc v0.2.95
   ...
   Compiling bdk-cli v0.2.0
    Finished release [optimized] target(s) in 1m 00s
error: no binaries are available for install using the selected features

Not sure what is going on here. Do I need to provide a --features argument when downloading from Crates.io?

After migrating to clap 3.0 from structopt I am observing the vector of strings are not being parsed properly.. Even if we declare the arg as Vec<String> its only reading the first value, creating a vector of 1 item and when encountering the second value throwing an error..

Possible course of actions.

• Figure out whats the correct way to parse vector of strings in clap 3.0.
• If that's not possible for some reason figure out a work around.

Observations

• Vec parsing happens correctly for the top level command, which derives Parser.
• Vec parsing fails at internal subcommand options, like create_tx, compact_filters_options etc. These options cannot derive Parser as Subcommand is already derived for them, and causes conflicting implementation.

After the introduction of the --log_level cli parameter, we lost the ability to filter logs by package making it unusable for log levels like debug or more granular (because there is too much output from package like rustls and sled).

example

RUST_LOG=debug,sled=info,rustls=info cargo run ...

I think we should either:

• remove --log_level parameter
• if possible accept in --log_level the same format supported in RUST_LOG and even allow RUST_LOG to work in alternative (I missed the --log_levelupdate and I started trying to use RUST_LOG automatically which is the standard in the rust ecosystem and I was surprised it had no effect)

with the regtest-* features and bdk-cli repl mode various multisig and other smart contracting situations can be demonstrated with bdk-cli..

Examle: Show an workflow of a Liquid like federation using a corresponding descriptor and bdk cli.

These demos can be recorded in various forms.

• Tutorial on the website.
• A demo videos in the youtube channel.
• Add it as a integration tests as both code level demo as well as consistency of these workflows with BDK API.

The general approach for a trial can be something like this

• Use the https://min.sc/ compiler to create your own weird descriptor.
• Split the descriptor for different parties involved in the compiler
• Start multiple wallet repl terminals with each parties descriptor. They will default connect to a single regtest node so all parties will be in sync
• Play around making transactions for various contract situations..

With this in mind we can start making a list of such interesting smart contracting situations with bitcoin scripts and demonstrate them out through BDK. Below is a list of basic ideas I imagined. Please comment below with more interesting contract ideas.

• Basic demo of a single sig wallet. (An test version of this is included in #102 )
• Basic 2-of-3 multisig demo
• Multisig with time locks. Something like @sandipndev cooked here.
• A Liquid like federation demo
• An HTLC lighting contract demo. Playing out what exactly happens with with lighting state updates and penalty mechanisms.

Hello,
I ran the code mentioned in https://bitcoindevkit.org/bdk-cli/installation/

cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=esplora,compiler

is throwing an error as shown below

It works fine if I remove "esplora" and just run with
cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=compiler

bdk-cli currently uses rpc "user:password" authentication.

We can also allow it to connect to rpc via cookie file. Although looking for opinions on how useful it can be.

When I use the recommended cargo install for the binaries I get this error:

# rust 1.53.0 on Ubuntu
➜ cargo install bdk-cli --features electrum
error: no binaries are available for install using the selected features

I could easily build from source so it's not a problem for me right now but I just wanted to point it out!

Electrs enforce build with a specific toolchain, which is good practice for binary projects as explained in the commit
romanz/electrs@3c4cf72

we may want to consider to do the same

All features should be additive and none should be mutually exclusive. See Cargo book section on feature unification.

Current blockchains features (electrum, esplora, compact_filters, rpc) are mutually exclusive. The need to be made to be additive with a runtime flag to select which one the wallet should use.

In light of bitcoindevkit/bdk#535, I feel the cli code can be optimized a lot. We can remove the OnlineWalletSubcommand and OfflineWalletSubcommand, as there is now no distinction between online and offline wallets. All wallets are offline and we only need a blockchian when we want to sync and broadcast..

This needs to be done before other open PRs can be merged or else it might cause rework on them.

So opening up this issue for approach discussion. Whats the best way structure bdk-cli after the big bdk change? Or should the same structure be kept as it is and adopt with minimal changes?

We currently have multiple versions of base64 in our dependency tree.. Ideally we should use the one exposed by rust-bitcoin.

[[package]]
name = "bdk-cli"
version = "0.5.0"
dependencies = [
 "base64 0.11.0",
 "bdk",
 "bdk-macros",
 "bdk-reserves",
 "dirs-next",
 "electrsd",
 "env_logger",
 "log",
 "regex",
 "rustyline",
 "serde_json",
 "structopt",
 "zeroize",
]

[[package]]
name = "bitcoin"
version = "0.28.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05bba324e6baf655b882df672453dbbc527bc938cadd27750ae510aaccc3a66a"
dependencies = [
 "base64-compat",
 "bech32",
 "bitcoin_hashes 0.10.0",
 "secp256k1 0.22.1",
 "serde",
]

[[package]]
name = "ureq"
version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3131cd6cb18488da91da1d10ed31e966f453c06b65bf010d35638456976a3fd7"
dependencies = [
 "base64 0.13.0",
 "chunked_transfer",
 "log",
 "once_cell",
 "rustls 0.19.1",
 "serde",
 "serde_json",
 "socks",
 "url",
 "webpki 0.21.4",
 "webpki-roots 0.21.1",
]

From Cargo.lock at b3d81b4

Currently our code coverage tests are only including unit tests. Which provides almost no coverage in the handlers and utils module. This can be improved by adding the integration tests into CI codecov jobs.

After Proof of Reserves capability is merged in bitcoindevkit/bdk@b2ac4a0, its time we require some integration tests in the library. Not only for Proof of Reserves functionality, but in general wallet functions too.

This then can be added into CI tests to give us better reliability on the underlying bdk functions.

electrsd can be used similarly to the bdk test framework to simulate a electrum and bitcoin core backend. https://github.com/bitcoindevkit/bdk/blob/64e88f0e006c68315142d53dc35b633327dde4b5/src/testutils/blockchain_tests.rs#L19-L22

The current Github Actions codecov-action@v1 tool is being deprecated and we must upgrade our code_coverage.yml workflow to the new version codecov-action@v2 by early next year.

Would love to be able to tell the address index on the get_new_address method if a verbose flag was enabled!

Currently bdk-cli can only use the bdk/key-value-db feature for wallet data storage. I propose we make the bdk/default memory database the default for bdk-cli, and add two new optional and mutually exclusive features that enable use of the bdk/key-value-db or sqlite DBs which override using the memory DB. These features could be named key-value-db and sqlite.

The purpose of this requested change is to make it possible to test bdk using the bdk-cli tool with any of the three possible wallet data storage options.

This is a discussion issue for selecting default options and feature sets in bdk-cli.

Currently as per #104 the defaults are this

• Network: Testnet
• Database : Sqlite
• Backend: None

with the new regtest-* features I was thinking weather to make the default bdk-cli built with something like this

• Network : Regtest
• Database: Sqlite
• Backend: Auto deployed Bitcoin Core

I feel that completes the basic suit of "whole" functionality bdk-cli can provide, and that will make the app just run and play.. No config required..

The repl command currently displays Proxy Server options as its short help message.

It should probably say something like Enter read-eval-print loop workflow or something similar.

If the bdk-cli command is used with an existing database but given a new descriptor an error like this is returned:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ChecksumMismatch', /Users/td/.cargo/registry/src/github.com-1ecc6299db9ec823/bdk-cli-0.1.0/src/bdk_cli.rs:119:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

A more informative message should be given so the user knows they can fix the problem by using a different wallet name or removing the existing ~/.bdk-bitcoin directory.

Given a miniscript spending "policy" and an optional script type ("sh", "wsh", or "sh-wsh") with default "wsh", compile the policy into it's corresponding output descriptor.

This is a command we found would be handy to have while creating the "Spending Policy Demo" blog post.

Currently we have two different source of bdk in the library. Our own dependency itself and one that is used by bdk-reserves which is in the bdk-cli dependency tree.

So as we are updating bdk-cli to maintain with current version of bdk, bdk-reserve is stable at v0.17.0 so we are basically duplicating the entire bdk dependencies in our tree twice. And there has been a lot of changes in upstreams of bdk which are included in this open PR. https://github.com/bitcoindevkit/bdk/pull/593/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542

So this issue is to open discussion on how to handle handle this now and in future properly..

Is it possible to expose bdk through bdk-cli and then have bdk-reserve use it, so we ensure both are using the same version?? But then we might have circular dependencies as bdk-cli also depends on bdk-reserve..

cc @notmandatory @ulrichard

This issue is to do a proof of concept test with BDK to confirm BDK can create a watch-only wallet to monitor the LND UTXOs and generate a PSBT that can be signed by LND. This idea is based on a discussion I had with @nicolasburtey and @bodymindarts about doing a BDK based wallet microservice for Galoy, initially for batching payments.

Goal of this test:

• Setup regtest bitcoind with LND and BDK-CLI
• Create LND on-chain wallet and load with test coins
• Setup BDK-CLI wallet to watch LND on-chain wallet (xpub descriptor)
• Create batch spending PSBT with BDK-CLI
• Sign and spend BDK-CLI generated PSBT with LND on-chain wallet

Description

Implement (optional) BIP-0078 PayJoin support in the bdk-cli wallet using the rust bip78 project. PayJoin transactions provides a way to collaboratively create a transaction with a receiver in a way that prevents on-chain surveillance from using the common input ownership heuristic. Adding this feature to the bdk-cli project will demonstrate to other wallet builders using bdk how to add PayJoin support.

Expected Outcomes

• Integrate BIP-0078 functionality as an optional feature in bdk-cli using the rust bip78 crate
• Contribute upstream to the rust bip78 project if any bugs or other issues are found
• Document how to use new PayJoin functionality with bdk-cli and common services such as BTCPayServer

Resources

• bdk and bdk-cli projects
• rust-payjoin project and PayJoin
• The Rust Book
• PayJoin adoption

Skills Required

• Rust language development, or similar language with willingness to learn rust
• Be able to send and receive bitcoin transactions on testnet with bdk-cli and other wallets
• Basic understanding bitcoin transaction building

Mentor(s)
@notmandatory

Difficulty
Hard

Competency Test

• Install rust
• Clone and run bdk-cli, create single signature descriptor wallet, receive and send testnet bitcoin
• Demonstrate rust coding skill:
  • via simple rust program
  • OR by completing rustlings exercises: variables, functions, if, primitive types

This is to document the recent cargo-audit failures happening in CI.

$ cargo-audit audit
    Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
      Loaded 456 security advisories (from /home/raj/.cargo/advisory-db)
    Updating crates.io index
    Scanning Cargo.lock for vulnerabilities (280 crate dependencies)
Crate:         rocksdb
Version:       0.14.0
Title:         Out-of-bounds read when opening multiple column families with TTL
Date:          2022-05-11
ID:            RUSTSEC-2022-0046
URL:           https://rustsec.org/advisories/RUSTSEC-2022-0046
Solution:      Upgrade to >=0.19.0
Dependency tree: 
rocksdb 0.14.0
└── bdk 0.22.0
    ├── bdk-reserves 0.22.0
    │   └── bdk-cli 0.5.0
    └── bdk-cli 0.5.0

Crate:         ansi_term
Version:       0.12.1
Warning:       unmaintained
Title:         ansi_term is Unmaintained
Date:          2021-08-18
ID:            RUSTSEC-2021-0139
URL:           https://rustsec.org/advisories/RUSTSEC-2021-0139
Dependency tree: 
ansi_term 0.12.1
└── clap 2.34.0
    └── structopt 0.3.26
        └── bdk-cli 0.5.0

Crate:         stdweb
Version:       0.4.20
Warning:       unmaintained
Title:         stdweb is unmaintained
Date:          2020-05-04
ID:            RUSTSEC-2020-0056
URL:           https://rustsec.org/advisories/RUSTSEC-2020-0056
Dependency tree: 
stdweb 0.4.20
└── time 0.2.27
    ├── cookie_store 0.12.0
    │   └── ureq 1.5.5
    └── cookie 0.14.4
        ├── ureq 1.5.5
        └── cookie_store 0.12.0

error: 1 vulnerability found!
warning: 2 allowed warnings found

There is a vulnerability in rocksdb which was originally reported by @afilini here bitcoindevkit/bdk#724.

Depending on the outcome of experimentation with nakamoto for cbf, we might be able to get rid of rocksdb fully from our dep tree.

Till then I guess we have to live with this audit failure?

Or we can disable compact_filters temporarily in bdk-cli..

For users who are manually testing bdk with bdk-cli in regtest mode, add a new regtest-net cargo feature that will automatically start and stop a bitcoind and if needed an electrsd daemon with electrum or esplora support. The regtest net daemons will only be started if the --network regtest option is enabled.

The feature should work similarly to the bdk integration tests but store node data in the users ~/.bdk-bitcoin directory and not a random temp directory, and use standard regtest network ports.

Using this feature would look something like this:

# start regtest `bitcoind` and `electrs` and stop daemons when repl (or wallet command) finishes
cargo run --features regtest-net,electrum -- --network regtest repl --descriptor "wpkh(tpubEBr4i6yk5nf5DAaJpsi9N2pPYBeJ7fZ5Z9rmN4977iYLCGco1VyjB9tvvuvYtfZzjD5A8igzgw3HeWeeKFmanHYqksqZXYXGsw5zjnj7KM9/*)" 

By enabling the regtest-net feature when running or installing bdk-cli a localhost bitcoind and (if electrum or esplora features enabled) electrsd daemon would be started and stopped for each command. This will be most convenient when running multiple commands via the repl shell but still usable (but slow) for single wallet commands.

The bitcoind and electrsd crates provide most of what is needed to download, start, and stop the needed daemons. This will only work on linux and macos systems.

It may also be useful to add additional wallet commands to get a new address from bitcoind or trigger bitcoind to send regtest coins to the bdk-cli wallet.

Out-of-bounds write in nix::unistd::getgrouplist

On certain platforms, if a user has more than 16 groups, the
nix::unistd::getgrouplist function will call the libc getgrouplist
function with a length parameter greater than the size of the buffer it
provides, resulting in an out-of-bounds write and memory corruption.

The libc getgrouplist function takes an in/out parameter ngroups
specifying the size of the group buffer. When the buffer is too small to
hold all of the reqested user's group memberships, some libc
implementations, including glibc and Solaris libc, will modify ngroups
to indicate the actual number of groups for the user, in addition to
returning an error. The version of nix::unistd::getgrouplist in nix
0.16.0 and up will resize the buffer to twice its size, but will not
read or modify the ngroups variable. Thus, if the user has more than
twice as many groups as the initial buffer size of 8, the next call to
getgrouplist will then write past the end of the buffer.

The issue would require editing /etc/groups to exploit, which is usually
only editable by the root user.

See advisory page for additional details.

As discussed in #65 (comment) we might want to change the default network value to regtest instead of testnet.

Rationale: bdk-cli is mostly used in testing and demo purpose as an easy way to interact with BDK without write separate wallet codes. In such environment most of the testing I have found myself and others doing are on regtest. So it makes sense to make regtest the default option. And user can switch to testnet with -n flag. And mainnet is not allowed in bdk-cli anyway.

Update: I also feel having Sqlite as default db would move us towards same direction.. Sqlite makes it really easy to real time observe database updates in terminal as devs test out different scenarios..

Required changes: Mostly documentation updates to reflected the new cli usage. Code change is simple.

The help printout works well on v0.3.0:

But is broken on v0.3.1:

Currently bdk-cli only uses sled database. But BDK now supports sqlite too. This can be extended to bdk-cli using a sqlite feature flag.

Having Sqlite DB in wallet database can be useful for lib users in many situations.

I am trying to install bdk-cli on ubuntu 20.04. I run the following command:

$ cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=esplora,compiler

And I get this error.

error[E0412]: cannot find type `EsploraOpts` in this scope
   --> src/lib.rs:433:23
    |
433 |     pub esplora_opts: EsploraOpts,
    |                       ^^^^^^^^^^^ not found in this scope

error[E0283]: type annotations needed
    --> src/lib.rs:432:17
     |
432  |     #[structopt(flatten)]
     |                 ^^^^^^^ cannot infer type
     |
     = note: cannot satisfy `_: StructOpt`
note: required by `from_clap`
    --> /home/cypher/.asdf/installs/rust/1.55.0/registry/src/github.com-1ecc6299db9ec823/structopt-0.3.25/src/lib.rs:1129:5
     |
1129 |     fn from_clap(matches: &clap::ArgMatches<'_>) -> Self;
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some errors have detailed explanations: E0283, E0412.
For more information about an error, try `rustc --explain E0283`.
error: could not compile `bdk-cli` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `bdk-cli v0.3.1-dev (https://github.com/bitcoindevkit/bdk-cli#daf0f920)`, intermediate artifacts can be found at `/tmp/cargo-installRyN0c0`

Caused by:
  build failed

Any help would be appreciated. I am a newbie to the Rust ecosystem.

Describe the bug
I tried to install the binaries on my machine by running the following command:

cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=electrum,compiler

Then, I encountered the following error message:

Updating git repository `https://github.com/bitcoindevkit/bdk-cli`
  Installing bdk-cli v0.5.0 (https://github.com/bitcoindevkit/bdk-cli#80260e1d)
    Updating crates.io index
  Downloaded instant v0.1.12
  Downloaded getrandom v0.1.16
  Downloaded itoa v1.0.2
  Downloaded lock_api v0.4.7
  Downloaded once_cell v1.12.0
  Downloaded base64 v0.10.1
  Downloaded proc-macro-error-attr v1.0.4
  Downloaded structopt v0.3.26
  Downloaded ansi_term v0.12.1
  Downloaded bech32 v0.8.1
  Downloaded cc v1.0.73
  Downloaded fs2 v0.4.3
  Downloaded dirs-sys-next v0.1.2
  Downloaded dirs-next v2.0.0
  Downloaded memchr v2.5.0
  Downloaded parking_lot v0.11.2
  Downloaded quote v1.0.20
  Downloaded scopeguard v1.1.0
  Downloaded rand_core v0.5.1
  Downloaded rand_chacha v0.2.2
  Downloaded smallvec v1.8.0
  Downloaded serde_derive v1.0.137
  Downloaded rand v0.7.3
  Downloaded unicode-segmentation v1.9.0
  Downloaded rand_core v0.4.2
  Downloaded unicode-ident v1.0.1
  Downloaded textwrap v0.11.0
  Downloaded strsim v0.8.0
  Downloaded zeroize v1.3.0
  Downloaded nix v0.22.3
  Downloaded log v0.4.17
  Downloaded proc-macro-error v1.0.4
  Downloaded pin-project-lite v0.2.9
  Downloaded sct v0.6.1
  Downloaded ryu v1.0.10
  Downloaded serde_json v1.0.81
  Downloaded utf8parse v0.2.0
  Downloaded socks v0.3.4
  Downloaded crc32fast v1.3.2
  Downloaded memoffset v0.6.5
  Downloaded spin v0.5.2
  Downloaded version_check v0.9.4
  Downloaded untrusted v0.7.1
  Downloaded termcolor v1.1.3
  Downloaded webpki v0.21.4
  Downloaded clap v2.34.0
  Downloaded aho-corasick v0.7.18
  Downloaded proc-macro2 v1.0.40
  Downloaded env_logger v0.7.1
  Downloaded syn v1.0.98
  Downloaded base64-compat v1.0.0
  Downloaded webpki-roots v0.19.0
  Downloaded miniscript v6.1.0
  Downloaded parking_lot_core v0.8.5
  Downloaded autocfg v1.1.0
  Downloaded crossbeam-utils v0.8.9
  Downloaded unicode-normalization v0.1.9
  Downloaded crossbeam-epoch v0.9.9
  Downloaded unicode-width v0.1.9
  Downloaded atty v0.2.14
  Downloaded byteorder v1.4.3
  Downloaded lazy_static v1.4.0
  Downloaded structopt-derive v0.4.18
  Downloaded serde v1.0.137
  Downloaded ppv-lite86 v0.2.16
  Downloaded nibble_vec v0.1.0
  Downloaded humantime v1.3.0
  Downloaded heck v0.3.3
  Downloaded fd-lock v3.0.2
  Downloaded endian-type v0.1.2
  Downloaded cfg-if v1.0.0
  Downloaded bitflags v1.3.2
  Downloaded smallvec v0.6.14
  Downloaded regex v1.5.6
  Downloaded radix_trie v0.2.1
  Downloaded rustyline v9.0.0
  Downloaded base64 v0.11.0
  Downloaded bitcoin_hashes v0.9.7
  Downloaded electrum-client v0.8.0
  Downloaded secp256k1 v0.20.3
  Downloaded vec_map v0.8.2
  Downloaded fxhash v0.2.1
  Downloaded bdk-macros v0.6.0
  Downloaded sled v0.34.7
  Downloaded secp256k1-sys v0.4.2
  Downloaded quick-error v1.2.3
  Downloaded rustls v0.16.0
  Downloaded libc v0.2.126
  Downloaded bitcoin v0.27.1
  Downloaded tokio v1.14.1
  Downloaded regex-syntax v0.6.26
  Downloaded bip39 v1.0.1
  Downloaded bitcoin_hashes v0.10.0
  Downloaded maybe-uninit v2.0.0
  Downloaded bdk v0.18.0
  Downloaded ring v0.16.20
  Downloaded 96 crates (12.3 MB) in 0.95s (largest was `ring` at 5.1 MB)
   Compiling libc v0.2.126
   Compiling cfg-if v1.0.0
   Compiling proc-macro2 v1.0.40
   Compiling unicode-ident v1.0.1
   Compiling quote v1.0.20
   Compiling autocfg v1.1.0
error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

error: could not compile `quote` due to previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `proc-macro2` due to previous error
error: could not compile `libc` due to previous error
error: failed to compile `bdk-cli v0.5.0 (https://github.com/bitcoindevkit/bdk-cli#80260e1d)`, intermediate artifacts can be found at `/tmp/cargo-installebWrRC`

To Reproduce

cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=electrum,compiler

or

cargo install bdk-cli --features electrum

Expected behavior
I expected a clean installation and compiling, and then being able to run bdk-cli.

Build environment

• BDK tag/commit: v0.5.0, 80260e1
• OS+version: ubuntu 20.04.01
• Rust/Cargo version: rustc 1.61.0 (fe5b13d68 2022-05-18)
• Rust/Cargo target: x86_64-unknown-linux-gnu

Additional context
I tried first to run this installation command, with Fulcrum as a feature an then received an error message:

~$ cargo install --git https://github.com/bitcoindevkit/bdk-cli --features=fulcrum,compiler
    Updating git repository `https://github.com/bitcoindevkit/bdk-cli`
  Installing bdk-cli v0.5.0 (https://github.com/bitcoindevkit/bdk-cli#80260e1d)
error: failed to compile `bdk-cli v0.5.0 (https://github.com/bitcoindevkit/bdk-cli#80260e1d)`, intermediate artifacts can be found at `/tmp/cargo-installQZEW56`

Caused by:
  Package `bdk-cli v0.5.0 (/home/ubuntu/.cargo/git/checkouts/bdk-cli-7ad4595d2026f0f8/80260e1)` does not have the feature `fulcrum`

After some researchs, I have found that fulcrum was not part of the features:

At most one blockchain feature can be enabled, available blockchain client features are: electrum, esplora-ureq (blocking), esplora-reqwest (async), compact_filters and rpc.

When trying to broadcast a PSBT such as

cHNidP8BAHEBAAAAAQU2MK4mbnsx/zjbmKwHGUAMVT1zXRFTPBArkMACRZjxAQAAAAD9////AhAnAAAAAAAAFgAU/52lZ+YvMOqGVPodX71HvvjjvhP7FQEAAAAAABYAFKqBwKFeT43famR0JJGaAhoMZKrXAAAAAAABAN4BAAAAAAEBArVvNyXe4TvWObt2vPpIv4IJPeFG1hRK8RtTgzVx3MEAAAAAAP3///8CECcAAAAAAAAWABT/naVn5i8w6oZU+h1fvUe++OO+E+ZAAQAAAAAAFgAUvv5IaH57cUVjoZ35bLxinO1BwkwCRzBEAiABTG7xfKRyZF2ezFCByBLSMGTA2FXYpMN881YXQZB/TgIgbU/OqbuWbe4KhWnjKeglVbnkko70H6glFe/zoo069fUBIQIeb6icGFSB9miQOCV9IMVmJdbEkXG8Hi86LaEyJRa5swAAAAABAR/mQAEAAAAAABYAFL7+SGh+e3FFY6Gd+Wy8YpztQcJMIgICZNLhaTjDm6k30vM/U2+d1TKQ02pk3MFxixFp4EQN1G5IMEUCIQCYRpRlO8YiUXZHLRYmS7fxhgCCDtYRVJ7Tb1rVOKqsHAIgcAeg6Dh8l2N9cixRUyCKwe0jWC9Xc7lNMrxJnDnlmN8BAQMEAQAAACIGAmTS4Wk4w5upN9LzP1NvndUykNNqZNzBcYsRaeBEDdRuGCrNFF1UAACAAQAAgAAAAIABAAAAAQAAAAAAIgIDRcB4bLvY45WvIPqto3nRP4nAQm1FHeIBWcqo2UiIHYoYKs0UXVQAAIABAACAAAAAgAEAAAACAAAAAA==

the transaction gets (rightfully) rejected by the network

[2022-05-26T13:05:29Z ERROR bdk_cli] Electrum(Protocol(String("sendrawtransaction RPC error: {\"code\":-26,\"message\":\"non-mandatory-script-verify-flag (Witness program hash mismatch)\"}")))

due to not being finalized.

To avoid user confusion I think the user should be either instructed to finalize the PSBT or have this be done automatically (which would be in line with what other wallets are doing).

Anywhere where there are only a limited enum set of options, such as with the network option, we should show in the help what those options are. This is done with the possible_values structopt feature, see example in Compile command:

Compile {
        /// Sets the spending policy to compile
        #[structopt(name = "POLICY", required = true, index = 1)]
        policy: String,
        /// Sets the script type used to embed the compiled policy
        #[structopt(name = "TYPE", short = "t", long = "type", default_value = "wsh", possible_values = &["sh","wsh", "sh-wsh"])]
        script_type: String,
    },

Out-of-bounds read when opening multiple column families with TTL

Affected versions of this crate called the RocksDB C API
rocksdb_open_column_families_with_ttl() with a pointer to a single integer
TTL value, but one TTL value for each column family is expected.

This is only relevant when using
rocksdb::DBWithThreadMode::open_cf_descriptors_with_ttl() with multiple
column families.

This bug has been fixed in v0.19.0.

See advisory page for additional details.

With recent merge of #105. We have OPRETURN output creation in bdk-cli.. This can be covered in the integration tests in tests/integration.rs using the regtsts-* backend.

I guess we should change the instructions on bitcoindevkit.org to install the repl from the other repo - or even better as soon as we publish bdk-cli to crates.io we could just switch to using that by default instead of git.

Originally posted by @afilini in bitcoindevkit/bdk#248 (comment)

Currently the "wallet" subcommands "create_tx", "sign", and "finalize_psbt" returned a PSBT in serialized format. Add a new wallet flag with long name "verbose" and short name "v" that when enabled with cause these sub-sub-commands to also return the PSBT in deserialized JSON format, in addition to the existing serialized form currently returned.

This is a good first issue for @sandipndev