bitcoin-core / docs Goto Github PK

View Code? Open in Web Editor NEW

35.0 35.0 67.0 1.28 MB

External documentation related to Bitcoin Core

License: MIT License

docs's People

Contributors

Stargazers

Watchers

docs's Issues

bin/make-base-vm failing on Fedora

Hi everyone

I'm trying to follow this guide about setting Fedora up for gitian building:

https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-setup-gitian-fedora.md

but I keep running into problems with the proxy, and get errors like Unable to connect to 172.17.0.1:3142: when running the bin/make-base-vm command.

I have reported the issue with gitian here:

devrandom/gitian-builder#227

but was thinking perhaps someone here could help with a solution?

I have tried different versions of Fedora and both installing docker through dnf install and the latest packages from docker.com, but with no luck.

Any ideas or hints appreciated.

Cheers

gitian: add ntp instructions

If the VM clock is off by too much, you may get error messages, see IRC log.

cc @instagibbs

gitian: "base-trusty-amd64 already exists, please remove it first"

./gitian-build.sh --setup throws this. Not sure if it's a warning or an error.

gitian: invalid option: --detach-sign

Generating report
f0f58a3c4f262cd5d8ba91774f0c0ad643ca83c14a6a4991918f8a02c87fa483  bitcoin-0.16.0-aarch64-linux-gnu-debug.tar.gz
...
b801f1eeeb3ad4aeb7f1b58b91b8a382fe138399223430d8de4b790a4452918d  bitcoin-linux-0.16-res.yml
Done.
./bin/gsign:52:in `<main>': invalid option: --detach-sign (OptionParser::InvalidOption)

I got the same error last summer. The correct syntax on Debian seems to be gsign -p true

gitian: more specific build, (detached) sign and PR instructions

The following commands copy assert files to the host machine (which has your PGP key)

git clone [email protected]:bitcoin-core/gitian.sigs.git
git remote add you [email protected]:you/gitian.sigs.git

export NAME=you
export VERSION=0.21.1rc1
mkdir -p  $VERSION-osx-unsigned/$NAME  $VERSION-win-unsigned/$NAME $VERSION-linux/$NAME
scp -r gitian:gitian.sigs/$VERSION-osx-unsigned/$NAME $VERSION-osx-unsigned
scp -r gitian:gitian.sigs/$VERSION-win-unsigned/$NAME $VERSION-win-unsigned
scp -r gitian:gitian.sigs/$VERSION-linux/$NAME $VERSION-linux

(would be nice to have a single command for this, maybe with some grep magic to find the relevant ../...$NAME...$VERSION paths)

sign from the host machine

gpg --output $VERSION-linux/$NAME/bitcoin-core-linux-0.21-build.assert.sig --detach-sign $VERSION-linux/sjors/bitcoin-core-linux-0.21-build.assert 
gpg --output $VERSION-osx-unsigned/$NAME/bitcoin-core-osx-0.21-build.assert.sig --detach-sign $VERSION-osx-unsigned/$NAME/bitcoin-core-osx-0.21-build.assert 
gpg --output $VERSION-win-unsigned/$NAME/bitcoin-core-win-0.21-build.assert.sig --detach-sign $VERSION-win-unsigned/$NAME/bitcoin-core-win-0.21-build.assert

if code-signed binaries are not yet available: make a PR without code-signed binaries (optional)

git checkout -b $VERSION
git add $VERSION-*/$NAME
git commit -S -a -m "Add $NAME $VERSION non-code signed signatures"
git push --set-upstream $NAME $VERSION

once code-signed binaries are available: [...]

./gitian-build.sh --detach-sign -s sjors $VERSION`

mkdir -p $VERSION-osx-signed/$NAME $VERSION-win-signed/$NAME
scp -r gitian:gitian.sigs/$VERSION-osx-signed/$NAME $VERSION-osx-signed
scp -r gitian:gitian.sigs/$VERSION-win-signed/$NAME $VERSION-win-signed

gpg --output $VERSION-osx-signed/$NAME/bitcoin-dmg-signer-build.assert.sig --detach-sign $VERSION-osx-signed/$NAME/bitcoin-dmg-signer-build.assert 
gpg --output $VERSION-win-signed/$NAME/bitcoin-win-signer-build.assert.sig --detach-sign $VERSION-win-signed/$NAME/bitcoin-win-signer-build.assert

git checkout -b $VERSION-signed
git add $VERSION-*-signed/$NAME
git commit -S -a -m "Add $NAME $VERSION code signed signatures"
git push --set-upstream $NAME $VERSION-signed

gitian: unclear after which step you need to "Follow the instructions in release-process.md"

It's clear one needs to run the --setup command first (but see #9) and --build after that.

But then the next paragraph says:

Follow the instructions in release-process.md in the bitcoin repository under 'Fetch and create inputs' to install sources which require manual intervention

This is confusing. Do I need to do this before, during or after --setup and --build? What is meant by "sources which require manual intervention" (I think it's something related to OSX and Windows signed builds, but we can't assume a first time Gitian user knows that).

Does the automated script do this or doesn't it?

gitian: explicitly enumerate manual steps prerequisites for gitian-build-sh

At this point you have two options, you can either use the automated script (found in https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.sh, only works in Debian/Ubuntu) or you could manually do everything by following this guide.

Lies! :-) You always need to do some things manually, or choose to produce an incomplete set of binaries. This should be clarified.

MacOS build will be skipped, unless you (insert link to instructions, see #13)

gitian: "run it with the --setup command" needs example

Current docs say:

If you are using the automated script, then run it with the --setup command. Afterwards, run it with the --build command (example: contrib/gitian-build.sh -b signer 0.15.0). Otherwise ignore this.

This leaves out details that confuse new developers (it confused me last summer, and still does).

It doesn't mention that this command should be run from ~, so it should be bitcoin/contrib/gitian-build.sh. It doesn't explain if you need to checkout the right tag of the bitcoin repository (afaik you shouldn't).

It doesn't explain how to use the --setup command, in particular whether arguments like singer are required (--help isn't clear on that either).

How to create the codesigned binaries?

I have not been able to create the codesigned binaries.

When I go according to this doc

https://github.com/bitcoin-core/docs/blob/master/gitian-building.md

There is no mention of how to create the signed versions.

You may have other .assert files as well (e.g. signed ones), in which case you should sign them too

In this document, there is something written too, but I am not 100% how it relates to the document in this repo

https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#next-steps

There is this code block

pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
popd

The first command starts to download the bitcoin repo again (and maybe build?), so I cancelled it by ctrl-c; the second command now writes this error

./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign:70:in `<main>': result/bitcoin-dmg-signer-res.yml does not exist (RuntimeError)

And indeed in result, there is no bitcoin-dmg-signer-res.yml; in result, I see just:

bitcoin-linux-0.17-res.yml
bitcoin-osx-0.17-res.yml
bitcoin-windows-0.17-res.yml

Do I really need to build the OSX version again with the first command from the release-process code block? Or is this unrelated?

gitian-build.sh --help says bitcoin-detached-sigs dir should exist

(I'm filing a bunch of issues as a brain dump)

bitcoin/contrib/gitian-build.sh --help

Usage: gitian-build.sh [-c|u|v|b|s|B|o|h|j|m|] signer version

Run this script from the directory containing the bitcoin, gitian-builder, gitian.sigs, and bitcoin-detached-sigs.

Arguments:
...

The instructions so far didn't mention cloning bitcoin-detached-sigs.

Also the instructions should point out the --help command exists.

gitian: CalledProcessError

I am getting below error while running this command to build wallet for my coin:
./gitian-build.py --setup mcoin 0.17.0rc1

OptiPlex-390:~/mcoin/contrib$ ./gitian-build.py --setup mcoin 0.17.0rc1
Cannot build for MacOS, SDK does not exist. Will build for other OSes
v0.17.0rc1
Cloning into 'gitian.sigs.ltc'...
remote: Repository not found.
fatal: repository 'https://github.com/mcoin-project/gitian.sigs.ltc.git/' not found
Traceback (most recent call last):
File "./gitian-build.py", line 220, in
main()
File "./gitian-build.py", line 203, in main
setup()
File "./gitian-build.py", line 26, in setup
subprocess.check_call(['git', 'clone', 'https://github.com/mcoin-project/gitian.sigs.ltc.git'])
File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://github.com/mcoin-project/gitian.sigs.ltc.git']' returned non-zero exit status 128.

gitian: point out -j -m are useful and should be <= than VM settings

The instructions for setting up Virtual Box tell the user to give it enough CPU and RAM, but that's not utilized unless they also call gitian-build.sh with -j and -m. New gitian builders are likely to miss it and be needlessly frustrated by how long the build takes.

Package 'ubuntu-archive-keyring' has no installation candidate

When setting up gitian on Ubuntu 20.04 according to this guide there's a dependency on a package called ubuntu-archive-keyring that seems to be gone.

$ sudo apt install ubuntu-archive-keyring
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package ubuntu-archive-keyring is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'ubuntu-archive-keyring' has no installation candidate

Delete after gitian was removed?

The repo can probably be wiped at some point.

gitian: Debian 9 doesn't ship with ifconfig

The instructions here won't work if ifconfig is not present:

https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-setup-gitian-debian.md#setting-up-debian-for-gitian-building

Solution is to tell user to sudo apt-get install net-tools or change the instructions to use the ip command.

That still doesn't work for me. I had to sudo -s and then source /etc/rc.local.

(also, there's some inconsistency between this user being called debian vs gitianuser, but that creates a clear error message. However gitian-builder/var/lxc.config assumes debian)

Interface name different from documentation

Commands for creating the virtual interface on both distributions uses the name br0:

echo 'brctl addbr br0' >> /etc/rc.local

and

echo 'ifconfig br0 10.0.3.2/24 up' >> /etc/rc.local

But gitian-build.sh script looks instead for lxcbr0, not br0.

MacOSX extraction needs to be updated for 10.14 and 10.15

Instructions here only refer to 10.11.

There's instructions for 10.14 here
https://github.com/bitcoin/bitcoin/tree/0.20/contrib/macdeploy#sdk-extraction

For 10.15
https://github.com/bitcoin/bitcoin/tree/master/contrib/macdeploy#sdk-extraction

Move "reducing bitcoind memory usage" here?

I have a fairly popular gist on reducing bitcoind memory usage here: https://gist.github.com/laanwj/efe29c7661ce9b6620a7

Might make sense to move it here instead of maintaining it as a gist so that other people can contribute to it.

gitian: Debian 10 Virtualbox Host: lxc-net not populated

Not sure if this is documented anywhere, but I had an issue with Debian 10 in that the targets seemed to not have routes. I found that Debian 10 (after an upgrade) doesn't appear to populate /etc/default/lxc-net by default, which should need the following:

USE_LXC_BRIDGE="true"
LXC_BRIDGE="br0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE=""
LXC_DOMAIN=""

Maybe this is useful if/when the docs are updated to the latest version of Debian for the VM.

gitian: missing apt packages in gitian-build.sh setup

Missing apt-cacher-ng and ubuntu-archive-keyring on Debian.

VM setup has `debian` as username rather than `gitianuser` in images

People setting up via quick skimming of the images will do this wrong(me)

gitian: simplified OSX instructions

E.g. assuming a Debian VM.

See here for ingredients to distill simplified OSX instructions from.

gitian: hostname should be "gitian" instead of "debian"?

The VirtualBox instructions say the hostname should be "gitian", but that causes ./gitian-build.sh --setup to throw a bunch of unable to resolve host gitian errors.

E: No such script: /usr/share/debootstrap/scripts/bionic

Trying to do a gitian build I followed the instructions here to create a virtualbox VM and setup gitian where I get to the part Setting up the Gitian image which fails with this error:

gitianuser@debian:~/gitian-builder$ bin/make-base-vm --lxc --arch amd64 --suite bionic
[sudo] password for gitianuser: 
E: No such script: /usr/share/debootstrap/scripts/bionic

Additional Debug Information

# history
    1  echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
    2  echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-execute" >> /etc/sudoers.d/gitian-lxc
    3  echo '#!/bin/sh -e' > /etc/rc.local
    4  echo 'brctl addbr br0' >> /etc/rc.local
    5  echo 'ip addr add 10.0.3.1/24 broadcast 10.0.3.255 dev br0' >> /etc/rc.local
    6  echo 'ip link set br0 up' >> /etc/rc.local
    7  echo 'firewall-cmd --zone=trusted --add-interface=br0' >> /etc/rc.local
    8  echo 'exit 0' >> /etc/rc.local
    9  chmod +x /etc/rc.local
   10  echo 'export USE_LXC=1' >> /home/gitianuser/.profile
   11  echo 'export GITIAN_HOST_IP=10.0.3.1' >> /home/gitianuser/.profile
   12  echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/gitianuser/.profile
   13  reboot
$ history
    1  mkdir .ssh
    2  vi .ssh/authorized_keys
    3  sudo apt-get install git ruby apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils make ubuntu-archive-keyring curl firewalld
    4  sudo -s
    5  wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
    6  echo "76cbf8c52c391160b2641e7120dbade5afded713afaa6032f733a261f13e6a8e  vm-builder_0.12.4+bzr494.orig.tar.gz" | sha256sum -c
    7  tar -zxvf vm-builder_0.12.4+bzr494.orig.tar.gz
    8  cd vm-builder-0.12.4+bzr494
    9  sudo python setup.py install
   10  cd ..
   11  git clone https://github.com/devrandom/gitian-builder.git
   12  git clone https://github.com/bitcoin/bitcoin
   13  git clone https://github.com/bitcoin-core/gitian.sigs.git
   14  git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
   15  cd gitian-builder
   16  bin/make-base-vm --lxc --arch amd64 --suite bionic # For releases after and including 0.17.0
   17  bin/make-base-vm --lxc --arch amd64 --suite trusty # For releases before 0.17.0
$ ls /usr/share/debootstrap/scripts/{bionic,trusty}
ls: cannot access /usr/share/debootstrap/scripts/bionic: No such file or directory
/usr/share/debootstrap/scripts/trusty

.

gitian: gitian-build.sh --setup complains if gitian.sigs etc already exist

fatal: destination path 'gitian.sigs' already exists and is not an empty directory.
fatal: destination path 'bitcoin-detached-sigs' already exists and is not an empty directory.
fatal: destination path 'gitian-builder' already exists and is not an empty directory.

This contradicts the clone instructions here.

bin/gbuild:23:in `system!': failed to run on-target -u root apt-get update >> var/install.log 2>&1 (RuntimeError)

I followed this documentation, ran into issues as the VirtualBox instructions recommend debian 8, fixed those issues updating the VM to debian 10 and now fail building bitcoin/bitcoin.

I'm slightly confused by the whole stack with an LXC VM inside a VirtualBox VM and might be doing it wrong as I try to skip the "sign your results" part as I just try to get results for myself for now but here's the log:

$ ./gitian-build.py --detach-sign --no-commit -b satoshi 0.20.0
Cannot build for MacOS, SDK does not exist. Will build for other OSes
v0.20.0
HEAD is now at a62f0ed64 doc: Manpages update pre-final
Already up to date.

Building Dependencies

--2020-08-31 21:06:18--  https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/mtrojnar/osslsigncode/tar.gz/2.0 [following]
--2020-08-31 21:06:19--  https://codeload.github.com/mtrojnar/osslsigncode/tar.gz/2.0
Resolving codeload.github.com (codeload.github.com)... 140.82.114.10
Connecting to codeload.github.com (codeload.github.com)|140.82.114.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘inputs/osslsigncode-2.0.tar.gz’

inputs/osslsigncode-2.0.tar.gz                           [  <=>                                                                                                                 ]  56.59K   186KB/s    in 0.3s    

2020-08-31 21:06:20 (186 KB/s) - ‘inputs/osslsigncode-2.0.tar.gz’ saved [57951]

inputs/osslsigncode-2.0.tar.gz: OK
make: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make[1]: Entering directory '/home/gitianuser/bitcoin/depends'
make[1]: Leaving directory '/home/gitianuser/bitcoin/depends'
make: Leaving directory '/home/gitianuser/bitcoin/depends'

Compiling 0.20.0 Linux
--- Building for bionic amd64 ---
Stopping target if it is up
Making a new image copy
sudo: unable to resolve host gitian: Resource temporarily unavailable
Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED:
  rsyslog*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1466 kB disk space will be freed.
(Reading database ... 18583 files and directories currently installed.)
Removing rsyslog (8.32.0-1ubuntu4) ...
invoke-rc.d: could not determine current runlevel
 * Stopping enhanced syslogd rsyslogd
   ...done.
(Reading database ... 18525 files and directories currently installed.)
Purging configuration files for rsyslog (8.32.0-1ubuntu4) ...
dpkg: warning: while removing rsyslog, directory '/etc/apparmor.d/local' not empty so not removed
Processing triggers for systemd (237-3ubuntu10) ...
Adding 'local diversion of /sbin/initctl to /sbin/initctl.distrib'
Adding 'local diversion of /usr/bin/ischroot to /usr/bin/ischroot.distrib'
Adding 'local diversion of /usr/sbin/policy-rc.d to /usr/sbin/policy-rc.d.distrib'
Starting target
Checking if target is up
Preparing build environment
Updating apt-get repository (log in var/install.log)
Traceback (most recent call last):
	6: from bin/gbuild:329:in `<main>'
	5: from bin/gbuild:329:in `each'
	4: from bin/gbuild:331:in `block in <main>'
	3: from bin/gbuild:331:in `each'
	2: from bin/gbuild:336:in `block (2 levels) in <main>'
	1: from bin/gbuild:117:in `build_one_configuration'
bin/gbuild:23:in `system!': failed to run on-target -u root apt-get update >> var/install.log 2>&1 (RuntimeError)
Traceback (most recent call last):
  File "./gitian-build.py", line 262, in <module>
    main()
  File "./gitian-build.py", line 250, in main
    build()
  File "./gitian-build.py", line 63, in build
    subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'bitcoin='+args.commit, '--url', 'bitcoin='+args.url, '../bitcoin/contrib/gitian-descriptors/gitian-linux.yml'])
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bin/gbuild', '-j', '2', '-m', '2000', '--commit', 'bitcoin=v0.20.0', '--url', 'bitcoin=https://github.com/bitcoin/bitcoin', '../bitcoin/contrib/gitian-descriptors/gitian-linux.yml']' returned non-zero exit status 1.

bitcoin-core / docs Goto Github PK

docs's People

Contributors

Stargazers

Watchers

Forkers

docs's Issues

Additional Debug Information

Recommend Projects

Recommend Topics

Recommend Org