Git Product home page Git Product logo

eth-nodes's Introduction

Ethereum Node Packaging

This project aims to simplify the packaging of various Ethereum nodes for Debian-based systems, providing a streamlined approach to generate Debian packages for multiple Ethereum clients. It is actively under development, and you can check the roadmap for upcoming features.

Table of Contents

Installation

Debian 12 (bookworm)

  1. Add the repository signing key:

    sudo curl -fsSL https://packages.eth-pkg.com/keys/ethpkg-archive-keyring.asc -o /usr/share/keyrings/ethpkg-archive-keyring.asc
  2. Add the repository to sources.list:

    sudo echo "deb [arch=amd64 signed-by=/usr/share/keyrings/ethpkg-archive-keyring.asc] http://packages.eth-pkg.com/bookworm-main bookworm main" | sudo tee -a /etc/apt/sources.list.d/ethpkg.list
  3. Update package lists:

    sudo apt update

Ubuntu 24.04 LTS (noble numbat)

  1. Add the repository signing key:

    sudo curl -fsSL https://packages.eth-pkg.com/keys/ethpkg-archive-keyring.asc -o /usr/share/keyrings/ethpkg-archive-keyring.asc
  2. Add the repository to sources.list:

    sudo echo "deb [arch=amd64 signed-by=/usr/share/keyrings/ethpkg-archive-keyring.asc] http://packages.eth-pkg.com/noble-main noble main" | sudo tee -a /etc/apt/sources.list.d/ethpkg.list
  3. Update package lists:

    sudo apt update

Install Clients

Once the repository is added, you can install the clients using apt. Note that some clients might require additional runtime dependencies.

besu
  1. Install Java 21:

    sudo apt -y install wget curl
    wget https://download.oracle.com/java/21/archive/jdk-21.0.2_linux-x64_bin.deb
    sudo apt install ./jdk-21.0.2_linux-x64_bin.deb
  2. Set up Java environment:

    cat <<'EOF' | sudo tee /etc/profile.d/jdk.sh
    export JAVA_HOME=/usr/lib/jvm/jdk-21/
    export PATH=\$PATH:\$JAVA_HOME/bin
    EOF
    
    source /etc/profile.d/jdk.sh
    sudo ln -s /usr/lib/jvm/jdk-21-oracle-x64 /usr/lib/jvm/jdk-21
    java -version
  3. Install besu:

    sudo apt install eth-node-besu
  4. Verify installation:

    besu --data-path <YOUR_DATA_DIR>
erigon
  1. Install erigon:

    sudo apt install eth-node-erigon
  2. Verify installation:

    erigon
geth
  1. Install geth:

    sudo apt install eth-node-geth
  2. Verify installation:

    geth
lodestar
  1. Install Node.js:

    curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
    sudo apt install -y nodejs
  2. Install lodestar:

    sudo apt install eth-node-lodestar
  3. Verify installation:

    lodestar
nethermind
  1. Install .NET runtime:

    wget https://packages.microsoft.com/config/debian/12/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
    sudo dpkg -i packages-microsoft-prod.deb
    rm packages-microsoft-prod.deb
    sudo apt update
    sudo apt install -y aspnetcore-runtime-7.0
  2. Install nethermind:

    sudo apt install eth-node-nethermind
  3. Verify installation:

    nethermind -dd <YOUR_DATA_DIR>
lighthouse
  1. Install lighthouse:

    sudo apt install eth-node-lighthouse
  2. Verify installation:

    lighthouse
nimbus-eth2
  1. Install nimbus-eth2:

    sudo apt install eth-node-nimbus-eth2
  2. Verify installation:

    nimbus_beacon_node
prysm
  1. Install prysm:

    sudo apt install eth-node-prysm
  2. Verify installation:

    beacon-chain
reth
  1. Install reth:

    sudo apt install eth-node-reth
  2. Verify installation:

    reth
teku
  1. Install Java 21:

    sudo apt -y install wget curl
    wget https://download.oracle.com/java/21/archive/jdk-21.0.2_linux-x64_bin.deb
    sudo apt install ./jdk-21.0.2_linux-x64_bin.deb
  2. Set up Java environment:

    cat <<'EOF' | sudo tee /etc/profile.d/jdk.sh
    export JAVA_HOME=/usr/lib/jvm/jdk-21/
    export PATH=\$PATH:\$JAVA_HOME/bin
    EOF
    
    source /etc/profile.d/jdk.sh
    sudo ln -s /usr/lib/jvm/jdk-21-oracle-x64 /usr/lib/jvm/jdk-21
    java -version
  3. Install teku:

    sudo apt install eth-node-teku
  4. Verify installation:

    teku

Building Packages

Prerequisites

To begin building packages, you need to install pkg-builder. Refer to the pkg-builder README for installation instructions.

Building and Verifying Packages

  1. Navigate to the directory you want to package:

    cd debian-12/amd64/eth-node-nimbus-eth2/24.3.0-1
  2. Create an environment:

    pkg-builder env create
  3. Build the package:

    pkg-builder package
  4. Verify the build:

    pkg-builder verify --no-package true

Verifying

There are several methods to verify builds:

  1. Verify by rebuilding on your own machine.
  2. Verify by building using GitHub Actions.
  3. Verify that the hashes of distributed packages through apt match those provided in pkg-builder-verify.toml.

Verifying by Rebuilding on Your Machine or Cloud Instance

For detailed instructions, please refer to the section on how to build packages.

Note that verification cannot be performed inside a Docker image due to the current lack of support for stacking kernel namespaces with sbuild.

Verifying Builds through GitHub Actions

This method offers weak verifiability because GitHub Actions runners use uniform machines, which increases the likelihood of matching hashes. However, hashes might differ on locally built packages due to non-uniformity of machines. Running this verification is still recommended as it guarantees reproducibility on GitHub Actions and is easy to perform. (note: The built packages are built against multiple machines to check hashes.)

To verify through GitHub Actions:

  1. Fork the repository.
  2. Select a release to verify (e.g., releases/bookworm/amd64/eth-node-erigon/2.60.0-1).
  3. Create a branch named verify/bookworm/amd64/eth-node-erigon/2.60.0-1 (replace releases with verify).
  4. Push the branch to GitHub and create a PR.
  5. Wait for the action runner to complete.

Note: You cannot create any branch starting with verify/* on this repository to avoid dummy PRs.

Verifying Package Hashes with pkg-builder-verify.toml

Packages distributed through apt or downloadable from GitHub releases can be verified to ensure their hashes match the ones in pkg-builder-verify.toml. Follow these steps:

  1. Download the package:

    mkdir /tmp/tempdir && cd /tmp/tempdir
    sudo apt download <package_name>
  2. Check the SHA1 sum:

    sha1sum <package_name>*.deb
  3. Verify the hash:

    cat releases/bookworm/amd64/<package_name>/pkg-builder-verify.toml

Example for Teku:

sudo apt download eth-node-teku
sha1sum eth-node-teku_24.4.0-1_amd64.deb
cat releases/bookworm/amd64/eth-node-teku/24.4.0-1/pkg-builder-verify.toml

How It Works

This process utilizes debcrafter and pkg-builder to create reproducible environments. debcrafter aids in generating reproducible Debian directories from detailed specification files (.sss and .sps). While debcrafter already supports reproducible builds, it doesn't pin environment dependencies, which led to the development of pkg-builder. Pkg-builder employs debcrafter to establish minimal environments for building packages in line with Debian's best practices. This includes tools like sbuild, piuparts, lintian, and autopkgtest to ensure fully functional packages.

pkg-builder enhances debcrafter by adding package pinning, checking against hashes, and supporting multiple programming languages such as C, .NET, Java, Rust, Go, and Nim. It also supports various Linux backends for packaging, including Noble Numbat, Jammy, and Debian 12. Additionally, pkg-builder ensures that all tools work together in a uniform manner, addressing the challenge of using these tools consistently and correctly.

debcrafter also makes it possible to create modular and dependent Debian packages, simplifying the handling of package relationships. Without debcrafter, the eth-node project would not be possible.

One significant challenge in Debian packaging is the requirement for a separate git repository per package, which can hinder support for numerous applications. While Debian packaging promotes reproducible builds, this requirement poses some limitations. Furthermore, not everyone uses git repositories, making it difficult to track which packages are updated. Having a monorepo simplifies this by providing a clear view of what is shipped and what is in development. Through debcrafter, it is very easy to have one organization’s interconnected packages in one repository and manage different versions, updates, and relationships together without having to rehost the source code of the original packages. Since the source code is just one input to the package process, tracking it in the packaging repository host makes little sense. Simple patching allows for including only the patched files with their patches in the repository, saving cognitive overhead and git space.

eth-nodes's People

Contributors

eenagy avatar omahs avatar taxmeifyoucan avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

eth-nodes's Issues

Setup minimal config for eth-node-nimbus

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Note: this is both the execution and consensus client configuration

Setup minimal config for eth-node-besu

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Setup minimal config for eth-node-nethermind

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Update Readme

Readme is outdated, needs to be updated with the current supports.

  • How to run the project
  • Dependencies of project
  • How the project works
  • FAQ
  • Contribution

Prysm libs are linked against unused libraries

dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/validator debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/beacon-chain were not linked against libdl.so.2 (they use none of the library's symbols)
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/validator debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/beacon-chain were not linked against libresolv.so.2 (they use none of the library's symbols)
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/validator debian/eth-node-prysm/usr/lib/eth-node-prysm/bin/beacon-chain were not linked against libpthread.so.0 (they use none of the library's symbols)

Add reproducible build flags to all clients

Currently running the packaging process, some clients still result in the wrong hash, even though the dependencies are the same and the build environment is reproducible.

Completion criteria:
Check all clients, and run the verify command to see the hash. Please note that lodestar might not be reproducible.

Setup minimal config for eth-node-lighthouse

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Chroot and dependencies in different repositories

There are two questions here:

  • ca-certificates and external repos is a chicken and egg problem
  • external repositories are non-standard, also apt fails to install if adding as such

The main problem here is that the chroot environment requires ca-certificates present to add additional repositories as the repositories are external.

The second problem is that additional repositories are non-uniform, so adding them is not an easy feat.

Setup minimal config for eth-node-teku

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Setup minimal config for eth-node-geth

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Inhibit nodes to shut down until state is sync

Keeping a node online

Your node doesn't have to be online all the time, but you should keep it online as much as possible to keep it in sync with the network. You can shut it down to restart it, but keep in mind that:

    Shutting down can take a few minutes if the recent state is still being written on disk.
    Forced shut downs can damage the database requiring you to resync the entire node.
    Your client will go out of sync with the network and will need to resync when you restart it. While the node can begin syncing from were it was last shutdown, the process can take time depending on how long it has been offline.

Fix eth-node-teku debian12 version 23.10.0-1 test


CKZG4844UtilsTest > testFlattenBlobsWithUnexpectedSizeThrows() FAILED
    org.opentest4j.AssertionFailedError at CKZG4844UtilsTest.java:42

CKZG4844UtilsTest > testFlattenBlobsWithBigBoySizeThrows() FAILED
    org.opentest4j.AssertionFailedError at CKZG4844UtilsTest.java:53

21 tests completed, 2 failed

> Task :infrastructure:kzg:test FAILED

> Task :teku:test
Exception in thread "nioEventLoopGroup-4-1" java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException: Task tech.pegasys.teku.infrastructure.events.AsyncEventDeliverer$QueueReader@1279c4ab rejected from java.util.concurrent.ThreadPoolExecutor@42b81fe8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:722)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:623)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:31)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.propagateResult(SafeFuture.java:141)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$exceptionallyCompose$34(SafeFuture.java:423)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at org.ethereum.beacon.discovery.network.NettyDiscoveryServerImpl.lambda$startServer$1(NettyDiscoveryServerImpl.java:101)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:578)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:600)
	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:579)
	at io.netty.handler.logging.LoggingHandler.bind(LoggingHandler.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:602)
	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:579)
	at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:260)
	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.util.concurrent.RejectedExecutionException: Task tech.pegasys.teku.infrastructure.events.AsyncEventDeliverer$QueueReader@1279c4ab rejected from java.util.concurrent.ThreadPoolExecutor@42b81fe8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at tech.pegasys.teku.infrastructure.events.AsyncEventDeliverer.subscribe(AsyncEventDeliverer.java:60)
	at tech.pegasys.teku.infrastructure.events.EventChannel.subscribeMultithreaded(EventChannel.java:167)
	at tech.pegasys.teku.infrastructure.events.EventChannels.subscribeMultithreaded(EventChannels.java:102)
	at tech.pegasys.teku.infrastructure.events.EventChannels.subscribe(EventChannels.java:84)
	at tech.pegasys.teku.networking.eth2.ActiveEth2P2PNetwork.startup(ActiveEth2P2PNetwork.java:128)
	at tech.pegasys.teku.networking.eth2.ActiveEth2P2PNetwork.lambda$start$0(ActiveEth2P2PNetwork.java:122)
at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
... 56 more

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ':infrastructure:kzg:test'.

There were failing tests. See the report at: file:///<>/infrastructure/kzg/build/reports/tests/test/index.html

Err system.d service if node doesn't have hardware requirements

Based on official docs

There is enough disk space considering the chosen network and sync mode.
Memory and CPU is not halted by other programs.
Operating system is updated to the latest version.
System has the correct time and date.
Your router and firewall accept connections on listening ports. By default Ethereum clients use a listener (TCP) port and a discovery (UDP) port, both on 30303 by default

Setup minimal config for eth-node-erigon

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Setup minimal config for eth-node-lodestar

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Research: Chroot and reproducibility when it comes to debian

The chroot environment is reproducible, but it points against Debian, which means it is not fully reproducible. It will create different environments at different points in time, which might be a problem.

The same issue might arise using apt install when the latest packages are fetched.

Node.js reproducibility

Cand node.js projects built with reproducibility in mind? Locking packages does allow for the same dependency, but the build process or hashes of built modules still could be different.

Setup minimal config for eth-node-prysm

Acceptance criteria:

  • test should be running that node starts, and no error
  • test upgrade/remove/install
  • system.d service is running without error
  • dpkg-configure reconfigures config

Fix eth-node-nethermind debian12 version 1.21.1-1 test

Failed!  - Failed:     1, Passed:   859, Skipped:    11, Total:   871, Duration: 2 m 50 s - Nethermind.Blockchain.Test.dll (net7.0)
  Nethermind.AuRa.Test -> /<<PKGBUILDDIR>>/src/Nethermind/Nethermind.AuRa.Test/bin/Release/net7.0/Nethermind.AuRa.Test.dll
Test run for /<<PKGBUILDDIR>>/src/Nethermind/Nethermind.AuRa.Test/bin/Release/net7.0/Nethermind.AuRa.Test.dll (.NETCoreApp,Version=v7.0)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.