bitcoin-core / docs Goto Github PK
View Code? Open in Web Editor NEWExternal documentation related to Bitcoin Core
License: MIT License
External documentation related to Bitcoin Core
License: MIT License
Trying to do a gitian build I followed the instructions here to create a virtualbox VM and setup gitian where I get to the part Setting up the Gitian image which fails with this error:
gitianuser@debian:~/gitian-builder$ bin/make-base-vm --lxc --arch amd64 --suite bionic
[sudo] password for gitianuser:
E: No such script: /usr/share/debootstrap/scripts/bionic
# history
1 echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
2 echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
3 echo '#!/bin/sh -e' > /etc/rc.local
4 echo 'brctl addbr br0' >> /etc/rc.local
5 echo 'ip addr add 10.0.3.1/24 broadcast 10.0.3.255 dev br0' >> /etc/rc.local
6 echo 'ip link set br0 up' >> /etc/rc.local
7 echo 'firewall-cmd --zone=trusted --add-interface=br0' >> /etc/rc.local
8 echo 'exit 0' >> /etc/rc.local
9 chmod +x /etc/rc.local
10 echo 'export USE_LXC=1' >> /home/gitianuser/.profile
11 echo 'export GITIAN_HOST_IP=10.0.3.1' >> /home/gitianuser/.profile
12 echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/gitianuser/.profile
13 reboot
$ history
1 mkdir .ssh
2 vi .ssh/authorized_keys
3 sudo apt-get install git ruby apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make ubuntu-archive-keyring curl firewalld
4 sudo -s
5 wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
6 echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e vm-builder_0.12.4+bzr494.orig.tar.gz" | sha256sum -c
7 tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
8 cd vm-builder-0.12.4+bzr494
9 sudo python setup.py install
10 cd ..
11 git clone https://github.com/devrandom/gitian-builder.git
12 git clone https://github.com/bitcoin/bitcoin
13 git clone https://github.com/bitcoin-core/gitian.sigs.git
14 git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
15 cd gitian-builder
16 bin/make-base-vm --lxc --arch amd64 --suite bionic # For releases after and including 0.17.0
17 bin/make-base-vm --lxc --arch amd64 --suite trusty # For releases before 0.17.0
$ ls /usr/share/debootstrap/scripts/{bionic,trusty}
ls: cannot access /usr/share/debootstrap/scripts/bionic: No such file or directory
/usr/share/debootstrap/scripts/trusty
./gitian-build.sh --setup
throws this. Not sure if it's a warning or an error.
Hi everyone
I'm trying to follow this guide about setting Fedora up for gitian building:
but I keep running into problems with the proxy, and get errors like Unable to connect to 172.17.0.1:3142:
when running the bin/make-base-vm
command.
I have reported the issue with gitian here:
but was thinking perhaps someone here could help with a solution?
I have tried different versions of Fedora and both installing docker through dnf install
and the latest packages from docker.com, but with no luck.
Any ideas or hints appreciated.
Cheers
The VirtualBox instructions say the hostname should be "gitian", but that causes ./gitian-build.sh --setup
to throw a bunch of unable to resolve host gitian
errors.
.
Instructions here only refer to 10.11.
There's instructions for 10.14 here
https://github.com/bitcoin/bitcoin/tree/0.20/contrib/macdeploy#sdk-extraction
For 10.15
https://github.com/bitcoin/bitcoin/tree/master/contrib/macdeploy#sdk-extraction
When setting up gitian on Ubuntu 20.04 according to this guide there's a dependency on a package called ubuntu-archive-keyring
that seems to be gone.
$ sudo apt install ubuntu-archive-keyring
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package ubuntu-archive-keyring is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'ubuntu-archive-keyring' has no installation candidate
fatal: destination path 'gitian.sigs' already exists and is not an empty directory.
fatal: destination path 'bitcoin-detached-sigs' already exists and is not an empty directory.
fatal: destination path 'gitian-builder' already exists and is not an empty directory.
This contradicts the clone instructions here.
(I'm filing a bunch of issues as a brain dump)
bitcoin/contrib/gitian-build.sh --help
Usage: gitian-build.sh [-c|u|v|b|s|B|o|h|j|m|] signer version
Run this script from the directory containing the bitcoin, gitian-builder, gitian.sigs, and bitcoin-detached-sigs.
Arguments:
...
The instructions so far didn't mention cloning bitcoin-detached-sigs
.
Also the instructions should point out the --help
command exists.
At this point you have two options, you can either use the automated script (found in https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.sh, only works in Debian/Ubuntu) or you could manually do everything by following this guide.
Lies! :-) You always need to do some things manually, or choose to produce an incomplete set of binaries. This should be clarified.
Generating report
f0f58a3c4f262cd5d8ba91774f0c0ad643ca83c14a6a4991918f8a02c87fa483 bitcoin-0.16.0-aarch64-linux-gnu-debug.tar.gz
...
b801f1eeeb3ad4aeb7f1b58b91b8a382fe138399223430d8de4b790a4452918d bitcoin-linux-0.16-res.yml
Done.
./bin/gsign:52:in `<main>': invalid option: --detach-sign (OptionParser::InvalidOption)
I got the same error last summer. The correct syntax on Debian seems to be gsign -p true
The repo can probably be wiped at some point.
.
The instructions here won't work if ifconfig
is not present:
Solution is to tell user to sudo apt-get install net-tools
or change the instructions to use the ip
command.
That still doesn't work for me. I had to sudo -s
and then source /etc/rc.local
.
(also, there's some inconsistency between this user being called debian
vs gitianuser
, but that creates a clear error message. However gitian-builder/var/lxc.config
assumes debian
)
I followed this documentation, ran into issues as the VirtualBox instructions recommend debian 8, fixed those issues updating the VM to debian 10 and now fail building bitcoin/bitcoin.
I'm slightly confused by the whole stack with an LXC VM inside a VirtualBox VM and might be doing it wrong as I try to skip the "sign your results" part as I just try to get results for myself for now but here's the log:
$ ./gitian-build.py --detach-sign --no-commit -b satoshi 0.20.0
Cannot build for MacOS, SDK does not exist. Will build for other OSes
v0.20.0
HEAD is now at a62f0ed64 doc: Manpages update pre-final
Already up to date.
Building Dependencies
--2020-08-31 21:06:18-- https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/mtrojnar/osslsigncode/tar.gz/2.0 [following]
--2020-08-31 21:06:19-- https://codeload.github.com/mtrojnar/osslsigncode/tar.gz/2.0
Resolving codeload.github.com (codeload.github.com)... 140.82.114.10
Connecting to codeload.github.com (codeload.github.com)|140.82.114.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘inputs/osslsigncode-2.0.tar.gz’
inputs/osslsigncode-2.0.tar.gz [ <=> ] 56.59K 186KB/s in 0.3s
2020-08-31 21:06:20 (186 KB/s) - ‘inputs/osslsigncode-2.0.tar.gz’ saved [57951]
inputs/osslsigncode-2.0.tar.gz: OK
make: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make: Leaving directory '/home/gitianuser/bitcoin/depends'
Compiling 0.20.0 Linux
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
sudo: unable to resolve host gitian: Resource temporarily unavailable
Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED:
rsyslog*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1466 kB disk space will be freed.
(Reading database ... 18583 files and directories currently installed.)
Removing rsyslog (8.32.0-1ubuntu4) ...
invoke-rc.d: could not determine current runlevel
* Stopping enhanced syslogd rsyslogd
...done.
(Reading database ... 18525 files and directories currently installed.)
Purging configuration files for rsyslog (8.32.0-1ubuntu4) ...
dpkg: warning: while removing rsyslog, directory '/etc/apparmor.d/local' not empty so not removed
Processing triggers for systemd (237-3ubuntu10) ...
Adding 'local diversion of /sbin/initctl to /sbin/initctl.distrib'
Adding 'local diversion of /usr/bin/ischroot to /usr/bin/ischroot.distrib'
Adding 'local diversion of /usr/sbin/policy-rc.d to /usr/sbin/policy-rc.d.distrib'
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Traceback (most recent call last):
6: from bin/gbuild:329:in `<main>'
5: from bin/gbuild:329:in `each'
4: from bin/gbuild:331:in `block in <main>'
3: from bin/gbuild:331:in `each'
2: from bin/gbuild:336:in `block (2 levels) in <main>'
1: from bin/gbuild:117:in `build_one_configuration'
bin/gbuild:23:in `system!': failed to run on-target -u root apt-get update >> var/install.log 2>&1 (RuntimeError)
Traceback (most recent call last):
File "./gitian-build.py", line 262, in <module>
main()
File "./gitian-build.py", line 250, in main
build()
File "./gitian-build.py", line 63, in build
subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'bitcoin='+args.commit, '--url', 'bitcoin='+args.url, '../bitcoin/contrib/gitian-descriptors/gitian-linux.yml'])
File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/gbuild', '-j', '2', '-m', '2000', '--commit', 'bitcoin=v0.20.0', '--url', 'bitcoin=https://github.com/bitcoin/bitcoin', '../bitcoin/contrib/gitian-descriptors/gitian-linux.yml']' returned non-zero exit status 1.
If the VM clock is off by too much, you may get error messages, see IRC log.
cc @instagibbs
The following commands copy assert files to the host machine (which has your PGP key)
git clone [email protected]:bitcoin-core/gitian.sigs.git
git remote add you [email protected]:you/gitian.sigs.git
export NAME=you
export VERSION=0.21.1rc1
mkdir -p $VERSION-osx-unsigned/$NAME $VERSION-win-unsigned/$NAME $VERSION-linux/$NAME
scp -r gitian:gitian.sigs/$VERSION-osx-unsigned/$NAME $VERSION-osx-unsigned
scp -r gitian:gitian.sigs/$VERSION-win-unsigned/$NAME $VERSION-win-unsigned
scp -r gitian:gitian.sigs/$VERSION-linux/$NAME $VERSION-linux
(would be nice to have a single command for this, maybe with some grep magic to find the relevant ../...$NAME...$VERSION paths)
gpg --output $VERSION-linux/$NAME/bitcoin-core-linux-0.21-build.assert.sig --detach-sign $VERSION-linux/sjors/bitcoin-core-linux-0.21-build.assert
gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-core-osx-0.21-build.assert.sig --detach-sign $VERSION-osx-unsigned/$NAME/bitcoin-core-osx-0.21-build.assert
gpg --output $VERSION-win-unsigned/$NAME/bitcoin-core-win-0.21-build.assert.sig --detach-sign $VERSION-win-unsigned/$NAME/bitcoin-core-win-0.21-build.assert
git checkout -b $VERSION
git add $VERSION-*/$NAME
git commit -S -a -m "Add $NAME $VERSION non-code signed signatures"
git push --set-upstream $NAME $VERSION
./gitian-build.sh --detach-sign -s sjors $VERSION`
mkdir -p $VERSION-osx-signed/$NAME $VERSION-win-signed/$NAME
scp -r gitian:gitian.sigs/$VERSION-osx-signed/$NAME $VERSION-osx-signed
scp -r gitian:gitian.sigs/$VERSION-win-signed/$NAME $VERSION-win-signed
gpg --output $VERSION-osx-signed/$NAME/bitcoin-dmg-signer-build.assert.sig --detach-sign $VERSION-osx-signed/$NAME/bitcoin-dmg-signer-build.assert
gpg --output $VERSION-win-signed/$NAME/bitcoin-win-signer-build.assert.sig --detach-sign $VERSION-win-signed/$NAME/bitcoin-win-signer-build.assert
git checkout -b $VERSION-signed
git add $VERSION-*-signed/$NAME
git commit -S -a -m "Add $NAME $VERSION code signed signatures"
git push --set-upstream $NAME $VERSION-signed
Current docs say:
If you are using the automated script, then run it with the --setup command. Afterwards, run it with the --build command (example: contrib/gitian-build.sh -b signer 0.15.0). Otherwise ignore this.
This leaves out details that confuse new developers (it confused me last summer, and still does).
It doesn't mention that this command should be run from ~, so it should be bitcoin/contrib/gitian-build.sh
. It doesn't explain if you need to checkout the right tag of the bitcoin repository (afaik you shouldn't).
It doesn't explain how to use the --setup
command, in particular whether arguments like singer
are required (--help
isn't clear on that either).
I am getting below error while running this command to build wallet for my coin:
./gitian-build.py --setup mcoin 0.17.0rc1
OptiPlex-390:~/mcoin/contrib$ ./gitian-build.py --setup mcoin 0.17.0rc1
Cannot build for MacOS, SDK does not exist. Will build for other OSes
v0.17.0rc1
Cloning into 'gitian.sigs.ltc'...
remote: Repository not found.
fatal: repository 'https://github.com/mcoin-project/gitian.sigs.ltc.git/' not found
Traceback (most recent call last):
File "./gitian-build.py", line 220, in
main()
File "./gitian-build.py", line 203, in main
setup()
File "./gitian-build.py", line 26, in setup
subprocess.check_call(['git', 'clone', 'https://github.com/mcoin-project/gitian.sigs.ltc.git'])
File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/mcoin-project/gitian.sigs.ltc.git']' returned non-zero exit status 128.
People setting up via quick skimming of the images will do this wrong(me)
Not sure if this is documented anywhere, but I had an issue with Debian 10 in that the targets seemed to not have routes. I found that Debian 10 (after an upgrade) doesn't appear to populate /etc/default/lxc-net
by default, which should need the following:
USE_LXC_BRIDGE="true"
LXC_BRIDGE="br0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE=""
LXC_DOMAIN=""
Maybe this is useful if/when the docs are updated to the latest version of Debian for the VM.
E.g. assuming a Debian VM.
See here for ingredients to distill simplified OSX instructions from.
Commands for creating the virtual interface on both distributions uses the name br0
:
echo 'brctl addbr br0' >> /etc/rc.local
and
echo 'ifconfig br0 10.0.3.2/24 up' >> /etc/rc.local
But gitian-build.sh
script looks instead for lxcbr0
, not br0
.
Missing apt-cacher-ng
and ubuntu-archive-keyring
on Debian.
I have a fairly popular gist on reducing bitcoind memory usage here: https://gist.github.com/laanwj/efe29c7661ce9b6620a7
Might make sense to move it here instead of maintaining it as a gist so that other people can contribute to it.
It's clear one needs to run the --setup
command first (but see #9) and --build
after that.
But then the next paragraph says:
Follow the instructions in release-process.md in the bitcoin repository under 'Fetch and create inputs' to install sources which require manual intervention
This is confusing. Do I need to do this before, during or after --setup
and --build
? What is meant by "sources which require manual intervention" (I think it's something related to OSX and Windows signed builds, but we can't assume a first time Gitian user knows that).
Does the automated script do this or doesn't it?
I have not been able to create the codesigned binaries.
When I go according to this doc
https://github.com/bitcoin-core/docs/blob/master/gitian-building.md
There is no mention of how to create the signed versions.
You may have other .assert files as well (e.g. signed ones), in which case you should sign them too
In this document, there is something written too, but I am not 100% how it relates to the document in this repo
https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#next-steps
There is this code block
pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
popd
The first command starts to download the bitcoin repo again (and maybe build?), so I cancelled it by ctrl-c; the second command now writes this error
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign:70:in `<main>': result/bitcoin-dmg-signer-res.yml does not exist (RuntimeError)
And indeed in result, there is no bitcoin-dmg-signer-res.yml; in result, I see just:
bitcoin-linux-0.17-res.yml
bitcoin-osx-0.17-res.yml
bitcoin-windows-0.17-res.yml
Do I really need to build the OSX version again with the first command from the release-process code block? Or is this unrelated?
The instructions for setting up Virtual Box tell the user to give it enough CPU and RAM, but that's not utilized unless they also call gitian-build.sh
with -j
and -m
. New gitian builders are likely to miss it and be needlessly frustrated by how long the build takes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.