firedancer-io / firedancer Goto Github PK

CI is currently broken because test_compact_u16 times out (>300s)

https://github.com/firedancer-io/firedancer/actions/runs/3797111361/jobs/6457780671#step:5:383

Problem

Tango is incompatible with other architectures like armv8, RISC-V, Power9.

Most of Firedancer's assumptions about the target platform are compatible with the above architectures.

However, Tango currently depends on the fact that in x86 writes from one CPU don't get reordered when observed from another CPU.
Thus, the code only uses compiler fences to prevent writes from being reordered at compile-time.

Common RISC architectures do not provide such guarantees.

Suggested Fix

• add write memory barriers when writing into critical sections
• add data dependency barriers before and after reading sequence numbers from critical sections

We are currently sourcing GCC from bootlin and LLVM from LLVM official binary releases.

We should compile our own compilers and host these tarballs on our own buckets instead.

Once #31 is done, add option to our plugin to reassemble shreds back to blocks.

Problem

Various performance-optimized parts of the code use static inline functions defined in header files.

Rust's bindgen fails to generate FFI bindings for them.

Currently, this requires reimplementing these functions in "unsafe" Rust.
Any changes to C code will also require changing these Rust bindings which is a big maintenance ask.

Proposed Solution

Add the following C macro

#if defined(FD_NO_INLINE) && FD_NO_INLINE
#define FD_INLINE
#else
#define FD_INLINE static inline
#endif

And change the declarations of all inline functions part of the external API to use this macro.

This change makes those functions available to Rust code as regular function calls while having no effect on C applications.

Note that this is a stopgap solution and doesn't entirely fix the underlying problem -- The Rust code compiled with FD_NO_INLINE incurs a performance penalty as it'll have to call down to C.

• Implement QUIC TLS handshake, key exchange

• Implement the the following TLS 1.3 cipher suites for use in QUIC

  • TLS_AES_128_GCM_SHA256 (0x1301, TLS 1.3)
  • TLS_CHACHA20_POLY1305_SHA256 (0x1303, TLS 1.3)

Tracking issue for Rust support

• Add Solana submodule a1df833
• Add Bazel source modules and cargo-raze f4522d7
• Create script to fix aliases google/cargo-raze#474
• Create build definitions for Solana crates of interest

Implement a C tile serving the ping/pong services of the gossip protocol (trivial)

For outgoing pings, maintain a map of peers that have responded with pongs (IP spoofing detection)

Depends on #153

Problem

Our code reviews and patch sets land in Gerrit, but don't get automatically mirrored to GitHub.

Suggested fix

Integrate Gerrit with GitHub to push anytime a new commit gets added to main.

Add Wireshark module implementing shred parsing

Add Wireshark module implementing gossip protocol parsing

Add asan, ubsan, msan, etc. as Bazel build profiles to tests

Problem

The current ffi/rust crates only contain bindings to Firedancer's C libraries but not the C library objects themselves.

This requires users will have to build Firedancer manually outside of their existing Rust-Cargo toolchain and then instruct their top-level crates to source these libraries from .a files.

This is not only annoying but also dangerous if the Rust crate and the manually compiled version use different revisions of the Firedancer code (no internal stability guarantees).

Suggested Fix

We have an existing Makefile setup that's widely compatible (basically any host that matches the target architecture, has GNU Make and a C/C++17 toolchain).

So, get our build.rs script to invoke Make and build Firedancer for the user.

This also requires vendoring the Firedancer C sources into the firedancer-sys crate, but that shouldn't be an issue.

NIST published a test suite for a number of their cryptographic algorithm standards. We should integrate those as unit tests

• Add code generator script for test vectors (https://git.firedancer.io/c/firedancer/+/285)
• Integrate code generator with build system (https://git.firedancer.io/c/firedancer/+/320)
• Add tests for SHA-256 (https://git.firedancer.io/c/firedancer/+/287)
• Add tests for SHA-512 (https://git.firedancer.io/c/firedancer/+/286)

After all transaction ingest-related modules are implemented and tested, milestone 1.2 concludes when we have docs for the community on how to run a Solana Labs validator with a Firedancer transaction ingest stage.

Add a non-optimized implementation of the Solana binary Merkle tree commitment scheme.

Implement both the 20-byte (used in shreds) and 32-byte (used in entries) variants.

Feature Request

Firedancer will rely heavily on fuzzing to ensure code correctness beyond unit tests.

The build system should support compilation with fuzz-instrumentation and linkage with libFuzzer/AFL/etc.

Ideally, we can also integrate Firedancer with the famous OSS-Fuzz to maintain corpora over time and periodically re-run tests, but that's a separate issue.

Suggested Fix

• Pull in Bazel's rules_fuzzing https://github.com/bazelbuild/rules_fuzzing
• Add an example test to illustrate how to write fuzz targets (e.g. pcap parser)
• Documentation

Problem

We currently have build files for both Make and Bazel.

Suggested fix

Once the team is familiar and happy with Bazel, declare a switch over and remove Makefiles.

Create a test suite that checks compatibility between the Solana Labs QUIC modules and Firedancer

libbpf depends on libelf via elfutils.

Should be vendored via Bazel instead of being a system dependency

Summary

Add an application that accepts and processes Turbine traffic into block data.

At a high-level, this application should act as a "spy" an existing stream of Turbine traffic.
Implement all steps of a "follower" validator short of any stateful processing in the runtime.

Most results of this work can then be repurposed into production validator code.
Unused bits can serve as integration tests.

Steps

1. Network interface for hooking into a stream of block data
   Getting a stream of Turbine block data requires basic plumbing with the P2P network.

   • Use an external packet mirroring setup to splice Turbine traffic arriving at a real Solana validator to another Firedancer test box

     • Pro: Relatively easy to maintain
     • Con: Complicated initial setup
     • Con: Does not work with local test validators
     • Note: The Jito team has offered us free access to their product for this, but should be easy to set up on local infra too

2. Shred Parser

   • Parser supporting legacy and merkle-variant shreds

3. Initial PoH sync
   The spy is not aware of the network-wide clock when it first starts.

   • Algorithm to train clock on incoming shred data when first starting up
   • Note: Once gossip is implemented, get initial clock from gossip instead

4. Leader Schedule
   Accepting Turbine traffic requires knowledge of the leader schedule.

   • Define text-oriented file format for reading a leader schedule, implement parser (requires Base58)
   • Plumbing to periodically export this leader schedule file from a trusted node
   • Plumbing to periodically load this leader schedule file
   • Note: Once snapshots & database are implemented, use trust-on-first-use approach to acquire initial leader schedule -- then trustlessly derive subsequent schedules

5. Shred sigverify stage

   • Implement high-performance/parallel sigverify stage for shreds (Unclear whether the existing verify stage should be extended or whether a second one should be implemented)
   • Add SigVerify cache for Merkle shreds

6. Deshred & Block parser
   Shred data will have to be defragmented and converted to entries

   • Add shred reassemble buffer
     Shreds arrive out-of-order, so need scratch space to reassemble them.
     Might require thread-safe depending on plumbing as shreds might come in from multiple tiles
   • Add shred-to-batch defragmentation
   • Add transaction parser
   • Add entry (block) parser (#27)
   • Add entry buffer
     Processing each entry can take up to ~100ms, so need to allocate shm for block data and descriptors

7. PoH hash chain
   Basic verifier whether blocks adhere to PoH constraints

   • Impl basic PoH algorithm
   • Impl entry mixin (#27)

8. Data dumping
   Add command-line interface to dump data

   • Dump shreds to .ar file
   • Dump entries to .ar file
   • Print blocks to log
   • Print txn sigs to log

9. Fork tracking
   The network partitions occasionally, Firedancer needs to be able to keep record of available forks

   • Add local PoH clock
   • Add shm buffer that tracks multiple forks within window

Next Goals

• Erasure coding recovery
• Shred Repair API

Implement a high-performance single-threaded fixed-capacity deque

https://about.codecov.io

see Bazel README how Bazel generates cov reports

Problem

The workspace / tile system uses various Linux-only APIs. Because of this, Firedancer fails to build on macOS.
There are lots of macOS developers who might want to use Firedancer APIs, or even run a macOS validator (although that is not recommended)

Suggested fix

However, the Darwin kernel offers replacements for most of these APIs (e.g. CPU pinning and huge pages).
They are just largely undocumented and annoying to work with.

Our current README jumps straight into technical configuration of the host.

We should probably add a brief summary of this project to Firedancer & development status & not-prod-ready disclaimer

• add entry header data structure
• add entry tx list data structure
• derive PoH mixins from entries

Hi, not sure if I am missing a configuration step before building with bazel but it seems

"//src/util/pod",

is missing in this build file be able to build the entire project from source bazel build //src/...

Happy to open a PR not sure what the preferred approach is.

Replace libnuma from numactl for the following reasons:

• Unresolved issues compiling numactl with Bazel rules_foreign_cc autotools on some hosts due to non-hermeticity
• Bazel rules_foreign_cc autotools is slow (>20s configure times, often takes longer than the rest of the build)
• libnuma is LGPLv2 -- While technically not an issue for linking, could become annoying
• Less dependencies is better in our development approach

Bazel currently supports a linux_x86_64_gcc and linux_x86_64_llvm target.

The GCC targets should be extended to support x86-64-v2, x86-64-v3, and x86-64-v4 equivalent targets.

https://www.phoronix.com/news/GCC-11-x86-64-Feature-Levels

In the block packing / pre-execution stage of the block production lifecycle, the block producer determines the fee reward of each tx.
This requires interpreting the "compute budget program" instructions to calculate CU limits and max fee rewards.

Problem

//src/net/xdp currently depends on libbpf APIs that were deprecated in v0.8.0 and removed in v1 releases.

Suggested Fix

Should be resolved so we can upgrade to the latest libbpf version (as of now v1.1.0 which was released last year)

• Import SHA-256 from OpenSSL
• Add SIMD-accelerated backend for single-block hashing
• Add SIMD-accelerated backend for multi-message parallel hashing

Add API for AVX2/AVX512-based parallel multiblock SHA-256 implementation

https://github.com/firedancer-io/specs should be merged into the main Firedancer tree eventually.

The Makefile setup defines a "brutality" C flags setting that enables a wide variety of checks.

https://github.com/firedancer-io/firedancer/blob/main/config/with-brutality.mk

This should be ported to bazelrc too

Firedancer primarily allocates memory in shared memory workspaces (fd_wksp.h).
These are mapped into memory as part of large shared memory objects.

ASan currently fails to detect most forms of out-of-bounds memory accesses because it considers an entire mmap()ed shared memory object as valid.

To fix, use the ASan poison API (#39) to mark workspaces invalid by default.
Then, selectively unpoison when a user joins or creates objects in workspace memory regions.

Add reference implementation of the Proof-of-History hashchain

Problem

Hex dumping an arbitrary-sized blob is a fairly common operation.

But at the moment, we only have a helper macro for printf-style formatting of 16-byte chunks: FD_LOG_HEX16_FMT.

Suggested Fix

Introduce an API for writing hexdump / xxd style dumps to fd_util log.

Our current SigVerify implementation (derived from OpenSSL) is more permissive than the current mainnet ed25519-dalek implementation.

Our build config for Bazel currently sets some GCC-only flags, causing clang to error out.

We should add most of these flags at toolchain-level configuration instead of the fd_cc_library wrapper rule.

This project is missing a code style guide

Should be a basic list of rules in a markdown doc

Add a Rust crate and a C package providing a basic IPC facility via shared memory.

This is mainly going to be used to provide connectivity for milestone 1.2 block packing.

Import ASan interface from https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/sanitizer/asan_interface.h

Add C structs for various data structures used in the QUIC protocol

Problem

GitHub CI runners are underpowered

Suggested Fix

See if we can run GitHub Actions runners on-prem on high-spec bare-metal nodes

Closes #168