ethereum / c-kzg-4844 Goto Github PK

Minimal 4844 version of c-kzg

License: Apache License 2.0

C# 3.84% Makefile 1.81% C 26.46% Dockerfile 0.03% JavaScript 0.69% Python 1.41% C++ 2.91% TypeScript 3.03% Java 6.12% Rust 7.21% Go 3.64% Nim 42.85%

c-kzg-4844's Introduction

C-KZG-4844

A minimal implementation of the Polynomial Commitments API for EIP-4844, written in C.

Bindings

While the core implementation is in C, bindings are available for various high-level languages, providing convenient wrappers around C functions. These bindings are intended to be used by Ethereum clients, to avoid re-implementation of crucial cryptographic functions.

Language	Link
C#	README
Go	README
Java	README
Nim	README
Node.js	README
Python	README
Rust	README

Interface functions

The C-KZG-4844 library provides implementations of the public KZG functions that are defined in the Polynomial Commitments specification. The aim is to align these functions as closely as possible with the specification.

blob_to_kzg_commitment
compute_kzg_proof
compute_blob_kzg_proof
verify_kzg_proof
verify_blob_kzg_proof
verify_blob_kzg_proof_batch

This library also provides functions for loading and freeing the trusted setup, which are not defined in the specification. These functions are intended to be executed once during the initialization process. As the name suggests, the trusted setup file is considered to be trustworthy.

load_trusted_setup
load_trusted_setup_file
free_trusted_setup

Remarks

Tests

All the bindings are tested against the KZG reference tests, which are defined in the consensus-spec-tests. Additionally, a suite of unit tests for internal C functions is located here.

Parallelization

The interface functions in C-KZG-4844 are single-threaded for simplicity, as implementing multi-threading across multiple platforms can be complex. While performance is important, these functions are already quite fast and efficient. For instance, verify_blob_kzg_proof is expected to finish in under 3ms on most systems.

Batched verification

When processing multiple blobs, verify_blob_kzg_proof_batch is more efficient than calling verify_blob_kzg_proof individually. In CI tests, verifying 64 blobs in batch is 53% faster per blob than verifying them individually. For a single blob, verify_blob_kzg_proof_batch calls verify_blob_kzg_proof, and the overhead is negligible.

Benchmarks

C-KZG-4844 does not include C benchmarks; however, some bindings (Go, Java, and Rust) have their own benchmarks. Including benchmarks in the bindings offers a more realistic performance estimate, as C-KZG-4844 is not expected to be used outside the bindings.

Security audit

The source code of C-KZG-4844 was audited by Sigma Prime in June 2023. You can find the audit report in the doc/audit/ directory.

Why C?

The primary reason for choosing C is that blst, the BLS12-381 signature library we wanted to use, is mostly written in C. Rust was a viable alternative, but it has some disadvantages. The C toolchain is ubiquitous, and it would be somewhat awkward for all the bindings to depend on another toolchain, such as Rust. Compared to Rust, C offers a lighter memory and binary footprint. Furthermore, C serves as the de facto language for FFI, so we could not have completely avoided using C anyway.

c-kzg-4844's People

Contributors

Stargazers

Watchers

c-kzg-4844's Issues

Go bindings

So I think I finally figured out how to provide Go bindings. I've made a proof of concept here:

https://github.com/jtraglia/cgo-kzg-4844

You can import this package and use the exported C-KZG-4844 functions.

Is this something we want in this repository?

Stack overflow in Rust benchmarks on Windows

https://github.com/ethereum/c-kzg-4844/actions/runs/5813951682/job/15762618420?pr=330

It's a bit jumbled, but the benchmarks on Windows are overflowing the stack somehow:

Benchmarking verify_blob_kzg_proof
Benchmarking verify_blob_kzg_proof: Warming up for 3.0000 s
Benchmarking verify_blob_kzg_proof: Collecting 100 samples in estimated 5.0302 s (1300 iterations)
Benchmarking verify_blob_kzg_proof: Analyzing
Benchmarking verify_blob_kzg_proof_batch/1
Benchmarking verify_blob_kzg_proof_batch/1: Warming up for 3.0000 s
verify_blob_kzg_proof   time:   [3.8203 ms 3.8244 ms 3.8285 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild


thread 'main' has overflowed its stack
error: bench failed, to rerun pass `--bench kzg_benches`

Caused by:
  process didn't exit successfully: `D:\a\c-kzg-4844\c-kzg-4844\bindings\rust\target\release\deps\kzg_benches-887efed028bd7e9a.exe --bench` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

NULL checks before free() unnecessary

free(NULL) is a no-op, so we can simplify code that currently looks like if (ptr != NULL) free(ptr); to free(ptr);

See e.g. https://stackoverflow.com/questions/1938735/does-freeptr-where-ptr-is-null-corrupt-memory

File descriptor leak in the nodejs bindings

Nice find. Maybe we suffer from a similar file descriptor leak in the nodejs bindings: https://github.com/ethereum/c-kzg-4844/blob/main/bindings/node.js/src/kzg.cxx#L142

Should we fix it as part of this PR or open a new one?

Originally posted by @asn-d6 in #241 (comment)

Can I just compile as a lib in C?

just make makes .a lib file with includes. but those includes does not have

blob_to_kzg_commitment
compute_kzg_proof
compute_blob_kzg_proof
verify_kzg_proof
verify_blob_kzg_proof
verify_blob_kzg_proof_batch

Nodejs does not support browsers

Node.js uses the 'fs' and 'path' modules that are not supported in the browser, can it be changed to pass the data of the file?

Make sure that `g1_lincomb()` handles input zero points correctly

Right now g1_lincomb() will attempt to convert all points to affine in batch using blst_p1s_to_affine(). However blst_p1s_to_affine() might have undefined behavior if one of the input points is zero (and IIRC returns zero for the result).

This can cause issues in our batch verification routine by zeroing out the pairing, so we should make sure we are protected against it.

Perhaps one way to do it is to check if one of the inputs are zero, and if it is, pass the entire computation through the basic non-pippenger MSM path. Another approach would be to try to weed out the bad points from the list of points but this might be more troublesome to implement.

Add function documentation comments

Some of the functions (mostly towards the bottom of the source file) are lacking documentation (doxygen) comments. I think it would be a good idea for someone to add these. Especially for the public/exported functions, but ideally all functions. I would do this myself, but I don't feel qualified to write these comments.

Problems with cross-platform docker images

It seems that amd64 image of the C# bindings crash when being run on an M1 Mac via docker.

$ docker run --rm -it jtraglia/ckzg-csharp-tests
Unable to find image 'jtraglia/ckzg-csharp-tests:latest' locally
latest: Pulling from jtraglia/ckzg-csharp-tests
14726c8f7834: Pull complete
6c3981608c2b: Pull complete
a2e354eccee4: Pull complete
cbca4ad4689c: Pull complete
31e2a952779c: Pull complete
6ecbf1fb37e1: Pull complete
6d8c48eeb4a8: Pull complete
2e9754864261: Pull complete
3e544255b783: Pull complete
d48c03db7ca1: Pull complete
4f4fb700ef54: Pull complete
e48d511b25b5: Pull complete
Digest: sha256:3f3de9187639a237ad04a831444637a9491817402c72b19cd357bda8f42f44b2
Status: Downloaded newer image for jtraglia/ckzg-csharp-tests:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
cd ../../blst && ./build.sh -D__BLST_PORTABLE__
+ clang -O2 -fno-builtin -fPIC -Wall -Wextra -Werror -D__BLST_PORTABLE__ -c ./src/server.c
+ clang -O2 -fno-builtin -fPIC -Wall -Wextra -Werror -D__BLST_PORTABLE__ -c ./build/assembly.S
+ llvm-ar rc libblst.a assembly.o server.o
clang -fPIC -O2 -Wall -Wextra -shared -DFIELD_ELEMENTS_PER_BLOB=4096 -I../../src -I../../blst/bindings -o Ckzg.Bindings/runtimes/linux-x64/native/ckzg.so ckzg.c ../../src/c_kzg_4844.c ../../blst/libblst.a
dotnet build
MSBuild version 17.7.1+971bf70db for .NET
make: *** [Makefile:64: ckzg-dotnet] Segmentation fault

But the Go version of this does work, which implies it's C# specific:

$ docker run --rm -it jtraglia/ckzg-go-tests
Unable to find image 'jtraglia/ckzg-go-tests:latest' locally
latest: Pulling from jtraglia/ckzg-go-tests
785ef8b9b236: Already exists
5a6dad8f55ae: Already exists
bd36c7bfe5f4: Already exists
9e59e6b803ed: Already exists
9e40ac7456b5: Already exists
8e11a6c7a2e6: Already exists
98e1ff1d4522: Pull complete
0c6b6951a232: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:30f635d1dd9a0ee9cf8d401e13b89e9c98c0c2a2b06904b4345d178867bb105d
Status: Downloaded newer image for jtraglia/ckzg-go-tests:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
go: downloading github.com/supranational/blst v0.3.11
go: downloading github.com/stretchr/testify v1.8.1
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
PASS
ok  	github.com/ethereum/c-kzg-4844/bindings/go	3.931s

Special thanks to @parithosh for pointing this out.

Verify that blst_uint64_from_fr() works with uninit input

This is a discussion from #196:

Just out of curiosity, what happened in f4c2749 ?

That was a finding from -fsantitize=memory before it crashed with this error:

$ make sanitize_memory
Running memory sanitizer
FATAL: Code 0xaaab4cc9c1e0 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.

I felt like because of the error, it was showing an incomplete list of findings. I plan to investigate why this is happening more later, but I suspect it's because blst isn't built with -fsanitize=memory. It seemed wrong to fix one instance when there are other instances of the same thing elsewhere, for example the original + one more:

c-kzg-4844/src/c_kzg_4844.c

Lines 228 to 232 in da83e45

 static bool fr_is_one(const fr_t *p) { 

 uint64_t a[4]; 

 blst_uint64_from_fr(a, p); 

 return a[0] == 1 && a[1] == 0 && a[2] == 0 && a[3] == 0; 

 }

c-kzg-4844/src/c_kzg_4844.c

Lines 243 to 248 in da83e45

 static bool fr_equal(const fr_t *aa, const fr_t *bb) { 

 uint64_t a[4], b[4]; 

 blst_uint64_from_fr(a, aa); 

 blst_uint64_from_fr(b, bb); 

 return a[0] == b[0] && a[1] == b[1] && a[2] == b[2] && a[3] == b[3]; 

 }

I may make this change again later. I don't think they are crucial though.

Originally posted by @jtraglia in #196 (comment)

Missing `verifyKzgProof` from NodeJS bindings

In the spec for EIP-4844, the point evaluation precompile has the below line
assert verify_kzg_proof(commitment, z, y, kzg_proof) but there is no method matching this interface exposed in the node-js bindings for this library. It looks like it does exist in the underlying C code for c-kzg so could we have it added to the bindings as well so we can finish implementing the point evaluation precompile?

Intermittent CI failure with build-ckzg-dotnet

Every so often, the build-ckzg-dotnet CI action will fail with this error:

System.Exception: Unable to read beyond the end of the stream.

He's an instance of this error:

https://github.com/ethereum/c-kzg-4844/actions/runs/4485648662/jobs/7887434752?pr=238

Here's the full output:

  cd bindings/csharp && dotnet test -c release --no-restore
  shell: /usr/bin/bash -e {0}
  env:
    binding_build_number_based_version: 0.1.2.108
/home/runner/.nuget/packages/microsoft.build.tasks.git/1.1.1/build/Microsoft.Build.Tasks.Git.targets(25,5): warning : Could not find file '/home/runner/work/c-kzg-4844/c-kzg-4844/blst/.git'. The source code won't be available via Source Link. [/home/runner/work/c-kzg-4844/c-kzg-4844/bindings/csharp/Ckzg.Bindings/Ckzg.Bindings.csproj]
  Ckzg.Bindings -> /home/runner/work/c-kzg-4844/c-kzg-4844/bindings/csharp/Ckzg.Bindings/bin/Release/net6.0/Ckzg.Bindings.dll
  Ckzg.Test -> /home/runner/work/c-kzg-4844/c-kzg-4844/bindings/csharp/Ckzg.Test/bin/Release/net7.0/Ckzg.Test.dll
Test run for /home/runner/work/c-kzg-4844/c-kzg-4844/bindings/csharp/Ckzg.Test/bin/Release/net7.0/Ckzg.Test.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.5.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Some ops seem exceedingly slow on Apple M2

I wrote an integration today for Geth for both the C and the Go libs. ethereum/go-ethereum#27155

Interestingly, blob to commitmets and proof generations with this lib seem to be 5-6x slower than the pure Go one, which seems be mean that something goes very wrong in compiling the C code: ethereum/go-ethereum#27155 (comment)

Any idea on how to investigate this?

free_trusted_setup should also free `s`

Is there a reason we are not also freeing the pointer s in free_trusted_setup() and the bindings need to do it themselves?

Intermittent segfault in rust bindings

There is a problem in the rust bindings (or c-kzg) that is causing an intermittent segfault. The following is output from running the rust bindings tests in a loop. If you can find them, there are two segfault errors in there.

jtraglia at xanax in ~/Projects/ethereum/c-kzg-4844/bindings/rust (main) 
$ git rev-parse HEAD
50bf358c6d7641328520c2cb4a17ca6566fd9ffb

jtraglia at xanax in ~/Projects/ethereum/c-kzg-4844/bindings/rust (main) 
$ while true; do cargo test --all --release --tests; done
   Compiling autocfg v1.1.0
   Compiling cfg-if v1.0.0
   Compiling libc v0.2.137
   Compiling proc-macro2 v1.0.49
   Compiling quote v1.0.23
   Compiling unicode-ident v1.0.6
   Compiling syn v1.0.107
   Compiling crossbeam-utils v0.8.14
   Compiling serde_derive v1.0.150
   Compiling serde v1.0.150
   Compiling scopeguard v1.1.0
   Compiling serde_json v1.0.89
   Compiling either v1.8.0
   Compiling rayon-core v1.10.1
   Compiling itoa v1.0.4
   Compiling os_str_bytes v6.4.1
   Compiling hashbrown v0.12.3
   Compiling plotters-backend v0.3.4
   Compiling half v1.8.2
   Compiling ryu v1.0.11
   Compiling ciborium-io v0.2.0
   Compiling cast v0.3.0
   Compiling bitflags v1.3.2
   Compiling regex-syntax v0.6.28
   Compiling itertools v0.10.5
   Compiling textwrap v0.16.0
   Compiling clap_lex v0.2.4
   Compiling plotters-svg v0.3.3
   Compiling memoffset v0.7.1
   Compiling crossbeam-epoch v0.9.13
   Compiling indexmap v1.9.2
   Compiling num-traits v0.2.15
   Compiling ciborium-ll v0.2.0
   Compiling same-file v1.0.6
   Compiling ppv-lite86 v0.2.17
   Compiling oorandom v11.1.3
   Compiling walkdir v2.3.2
   Compiling c-kzg v0.1.0 (/Users/jtraglia/Projects/ethereum/c-kzg-4844/bindings/rust)
   Compiling lazy_static v1.4.0
   Compiling anes v0.1.6
   Compiling hex v0.4.3
   Compiling crossbeam-channel v0.5.6
   Compiling regex v1.7.0
   Compiling num_cpus v1.15.0
   Compiling getrandom v0.2.8
   Compiling atty v0.2.14
   Compiling rand_core v0.6.4
   Compiling clap v3.2.23
   Compiling criterion-plot v0.5.0
   Compiling rand_chacha v0.3.1
   Compiling crossbeam-deque v0.8.2
   Compiling rand v0.8.5
   Compiling rayon v1.6.1
   Compiling plotters v0.3.4
   Compiling ciborium v0.2.0
   Compiling tinytemplate v1.2.1
   Compiling criterion v0.4.0
    Finished release [optimized] target(s) in 9.45s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.03s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.42s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.03s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_end_to_end ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.41s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.39s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.45s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.40s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.95s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.61s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.40s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.78s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.69s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.53s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.61s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.50s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.02s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.01s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.85s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.39s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.40s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.53s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.69s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.37s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_end_to_end ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.39s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.62s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.61s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `/Users/jtraglia/Projects/ethereum/c-kzg-4844/bindings/rust/target/release/deps/c_kzg-28ad36221918cd1f` (signal: 11, SIGSEGV: invalid memory reference)
    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.77s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.53s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.94s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.19s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.11s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.86s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.52s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.27s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.40s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.62s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_end_to_end ... ok
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.41s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.86s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.61s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.44s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.94s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `/Users/jtraglia/Projects/ethereum/c-kzg-4844/bindings/rust/target/release/deps/c_kzg-28ad36221918cd1f` (signal: 11, SIGSEGV: invalid memory reference)
    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

running 3 tests
test tests::test_verify_kzg_proof ... ok
test tests::test_compute_agg_proof ... ok
test tests::test_end_to_end ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.53s

    Finished release [optimized] target(s) in 0.03s
     Running unittests src/lib.rs (target/release/deps/c_kzg-28ad36221918cd1f)

Expose `compute_kzg_proof` in interface

I suggest adding compute_kzg_proof to the interface as it is required in order to compute valid inputs to the point evaluation precompile.

It will also be useful for testing purposes (easier to generate test cases for point evaluation precompile).

Make c# bindings more defensive

This ticket tracks the changes that need to happen to the C# bindings to make them more defensive and validate their inputs: #146 (comment)

/cc @flcl42

Make the interfaces all bytes-only

Currently there's a mix of byte arrays and internal types (like g1_t) in the interface. It should all be byte arrays. The conversion functions (like bytes_to_g1) should be removable from the interface.

Move away from patching sha256.h?

The current approach to making the sha256 functions visible is to patch the header file. This works fine for bindings built via some script/make process, where patching is just a step amongst others to be executed.

In the case of Nim however, c bindings are integrated with the regular compilation (basically, the c code is just "inlined"), so there's no good place to insert a patching step.

For the time being I have forked this repo and added a change which removes the patch and instead includes sha256.h. Can similarly remove the patch here?

Update nodejs readme with new interface

The nodejs bindings readme is pretty out of date. It still references the old interface.

check for non-canonical field element representations

Considering the conversation here ethereum/consensus-specs#3057 and ongoing discussion, just want to record that the C library may need to detect and fail on non-canonical inputs (i.e., greater than the modulus).

Consider using a char buffer instead of FILE

Using a char buffer would be IMO a more compatible version of the current api.

Chars have a standard representation across platforms, while FILEs are a linux api that windows supports to some degree. The windows version of it are handles (HFILE). I see other languages are lenient on dealing with files/os paths but in general these don't exactly map since windows paths are encoded using wide chars. Under the current api we need to deal with these nuances at the ffi boundary. Using a char buffer instead we would delegate opening the file to the language implementation, and simply pass the contents to the c code

Rust benchmarks are broken

┌─[tautvydas][kagamin][±][main ✓][~/Git/c-kzg-4844/bindings/rust]
└─▪ cargo bench
   Compiling c-kzg v0.1.0 (/mnt/home/Git/c-kzg-4844/bindings/rust)
error[E0433]: failed to resolve: use of undeclared type `KZGSettings`
  --> benches/kzg_benches.rs:31:33
   |
31 |     let kzg_settings = Arc::new(KZGSettings::load_trusted_setup_file(trusted_setup_file).unwrap());
   |                                 ^^^^^^^^^^^
   |                                 |
   |                                 use of undeclared type `KZGSettings`
   |                                 help: a struct with a similar name exists: `KzgSettings`

error[E0433]: failed to resolve: use of undeclared type `KZGCommitment`
  --> benches/kzg_benches.rs:39:13
   |
39 |             KZGCommitment::blob_to_kzg_commitment(*blob, &kzg_settings)
   |             ^^^^^^^^^^^^^
   |             |
   |             use of undeclared type `KZGCommitment`
   |             help: a struct with a similar name exists: `KzgCommitment`

error[E0433]: failed to resolve: use of undeclared type `KZGProof`
  --> benches/kzg_benches.rs:48:13
   |
48 |             KZGProof::compute_blob_kzg_proof(*blob, *commitment, &kzg_settings)
   |             ^^^^^^^^
   |             |
   |             use of undeclared type `KZGProof`
   |             help: a struct with a similar name exists: `KzgProof`

error[E0433]: failed to resolve: use of undeclared type `KZGCommitment`
  --> benches/kzg_benches.rs:58:19
   |
58 |         b.iter(|| KZGCommitment::blob_to_kzg_commitment(*blobs.first().unwrap(), &kzg_settings))
   |                   ^^^^^^^^^^^^^
   |                   |
   |                   use of undeclared type `KZGCommitment`
   |                   help: a struct with a similar name exists: `KzgCommitment`

error[E0433]: failed to resolve: use of undeclared type `KZGProof`
  --> benches/kzg_benches.rs:63:13
   |
63 |             KZGProof::compute_kzg_proof(
   |             ^^^^^^^^
   |             |
   |             use of undeclared type `KZGProof`
   |             help: a struct with a similar name exists: `KzgProof`

error[E0433]: failed to resolve: use of undeclared type `KZGProof`
  --> benches/kzg_benches.rs:73:13
   |
73 |             KZGProof::compute_blob_kzg_proof(
   |             ^^^^^^^^
   |             |
   |             use of undeclared type `KZGProof`
   |             help: a struct with a similar name exists: `KzgProof`

error[E0433]: failed to resolve: use of undeclared type `KZGProof`
  --> benches/kzg_benches.rs:83:13
   |
83 |             KZGProof::verify_blob_kzg_proof(
   |             ^^^^^^^^
   |             |
   |             use of undeclared type `KZGProof`
   |             help: a struct with a similar name exists: `KzgProof`

error[E0433]: failed to resolve: use of undeclared type `KZGProof`
  --> benches/kzg_benches.rs:96:17
   |
96 |                 KZGProof::verify_blob_kzg_proof_batch(
   |                 ^^^^^^^^
   |                 |
   |                 use of undeclared type `KZGProof`
   |                 help: a struct with a similar name exists: `KzgProof`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `c-kzg` due to 8 previous errors
warning: build failed, waiting for other jobs to finish...

Bytes length checks in Node.js bindings

So when looking at this again, I noticed a lack of length checks. I think there should be length checks for these types:

c-kzg-4844/bindings/node.js/kzg.ts

Lines 8 to 12 in 3ec2e9c

 export type Bytes32 = Uint8Array; // 32 bytes 

 export type Bytes48 = Uint8Array; // 48 bytes 

 export type KZGProof = Uint8Array; // 48 bytes 

 export type KZGCommitment = Uint8Array; // 48 bytes 

 export type Blob = Uint8Array; // 4096 * 32 bytes

This should be updated after #92 is merged.

Release Rust bindings on crates.io

We'd like to use the c-kzg bindings in revm. Currently, we would have to import them as a git dependency on this repository, but then we will not be able to release the revm crates on crates.io ourselves, since git dependencies are not allowed: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories.

Fold in nim bindings

They currently live in https://github.com/henridf/nim-kzg-4844

This seems like a good idea so that we have everything in one place and they are also included in the auditing scope.

Some things that also need to happen:

The bindings should implement the various required length checks (e.g. see the python bindings)
The bindings should be able to parse the yaml test vectors (see https://github.com/ethereum/c-kzg-4844/blob/main/bindings/python/tests.py)

We can perhaps add support for parsing the yaml after merging, just to get something in first.

/cc @henridf

Cleanup node bindings

The node bindings (https://github.com/ethereum/c-kzg-4844/tree/main/bindings/node.js) seem to contain quite a few files and it makes it hard to review the actual code. Can we tidy them up somehow?

/cc @dgcoffman

Double check handling of trusted setup by bindings

Two things to check here:

Are bindings freeing the trusted setup correctly? Are they calling free_trusted_setup() and also freeing the pointer (see #172).
Are we correctly protecting the bindings from double-freeing the trusted setup?

Add library prefix to all ckzg symbols

Currently the c-kzg library uses #define constants for the mainnet/minimal paramaters like FIELD_ELEMENTS_PER_BLOB:

c-kzg-4844/src/c_kzg_4844.h

Lines 39 to 43 in cfdd9e5

 /* 

  * This value represents the number of field elements in a blob. It must be 

  * supplied externally. It is expected to be 4096 for mainnet and 4 for minimal. 

  */ 

 #ifndef FIELD_ELEMENTS_PER_BLOB

This makes it very hard (but not impossible*) to link two copies of the library to enable switching between mainnet and minimal configurations at runtime.

*My attempt at doing this is documented here: sigp/lighthouse#4276. I had to use objcopy to rename c-kzg's symbols so that they wouldn't conflict. Needless to say this is not a sensible approach to take.

To do this better I was thinking the library could accept another #define constant which it prepends to all symbol names. Then the downstream consumer of the library could easily avoid collisions, with names like ckzg_blob_to_kzg_commitment and ckzgmin_blob_to_kzg_commitment instead of just blob_to_kzg_commitment. The ckzg prefix could be the default if no prefix is specified.

load_trusted_setup should take setup contents instead of setup file

Current load_trusted_setup implementation takes a file as input

c-kzg-4844/src/c_kzg_4844.c

Line 773 in 9611bfd

C_KZG_RET load_trusted_setup(KZGSettings *out, FILE *in) {

To support browser light-clients we'll compile c-kzg to WASM, which would require no interaction with the file system. It would also make DX more flexible on how client implementations want to handle the trusted setup loading.

This was discussed offline in a chat and agreed to adopt a simple serialization format for the trusted setup. Just concat the point count and the serialized points in the following order.

G1Count | G2Count | [G1] | [G2]

lincomb / Pippenger uses 256 instead of 255 bit for curve order

In this call:

c-kzg-4844/src/c_kzg_4844.c

Line 832 in 97c46a8

blst_p1s_mult_pippenger(out, points_arg, len, scalars_arg, 256, scratch);

The 256 corresponds to the scalar bit width:

size_t blst_p1s_mult_pippenger_scratch_sizeof(size_t npoints);
void blst_p1s_mult_pippenger(blst_p1 *ret, const blst_p1_affine *const points[],
                             size_t npoints, const byte *const scalars[],
                             size_t nbits, limb_t *scratch);
void blst_p1s_tile_pippenger(blst_p1 *ret, const blst_p1_affine *const points[],
                             size_t npoints, const byte *const scalars[],
                             size_t nbits, limb_t *scratch,
                             size_t bit0, size_t window);

But the curve order is 255-bit hence 256 is doing extra work.
I don't think the cost is just 1/256 and even if it was, over thousands of points, that's a lot more computations to do, especially given that it's likely one of the biggest bottleneck.

I think we need a test for it and optionally a bench.
The function is used in g1_lincomb:

c-kzg-4844/src/c_kzg_4844.c

Lines 796 to 842 in c6fa137

 static C_KZG_RET g1_lincomb(g1_t *out, const g1_t *p, const fr_t *coeffs, const uint64_t len) { 

 C_KZG_RET ret = C_KZG_MALLOC; 

 void *scratch = NULL; 

 blst_p1_affine *p_affine = NULL; 

 blst_scalar *scalars = NULL; 

 // Tunable parameter: must be at least 2 since Blst fails for 0 or 1 

 if (len < 8) { 

 // Direct approach 

 g1_t tmp; 

 *out = G1_IDENTITY; 

 for (uint64_t i = 0; i < len; i++) { 

 g1_mul(&tmp, &p[i], &coeffs[i]); 

 blst_p1_add_or_double(out, out, &tmp); 

 } 

 } else { 

 // Blst's implementation of the Pippenger method 

 scratch = malloc(blst_p1s_mult_pippenger_scratch_sizeof(len)); 

 if (scratch == NULL) goto out; 

 p_affine = malloc(len * sizeof(blst_p1_affine)); 

 if (p_affine == NULL) goto out; 

 scalars = malloc(len * sizeof(blst_scalar)); 

 if (scalars == NULL) goto out; 

 // Transform the points to affine representation 

 const blst_p1 *p_arg[2] = {p, NULL}; 

 blst_p1s_to_affine(p_affine, p_arg, len); 

 // Transform the field elements to 256-bit scalars 

 for (int i = 0; i < len; i++) { 

 blst_scalar_from_fr(&scalars[i], &coeffs[i]); 

 } 

 // Call the Pippenger implementation 

 const byte *scalars_arg[2] = {(byte *)scalars, NULL}; 

 const blst_p1_affine *points_arg[2] = {p_affine, NULL}; 

 blst_p1s_mult_pippenger(out, points_arg, len, scalars_arg, 256, scratch); 

 } 

 ret = C_KZG_OK; 

 out: 

 free(scratch); 

 free(p_affine); 

 free(scalars); 

 return ret; 

 }

which is used in the most "raw" manner in poly_to_kzg_commitment

c-kzg-4844/src/c_kzg_4844.c

Lines 946 to 957 in c6fa137

 /** 

  * Compute a KZG commitment from a polynomial. 

  * 

  * @param[out] out The resulting commitment 

  * @param[in] p The polynomial to commit to 

  * @param[in] s The settings struct containing the commitment key (i.e. the trusted setup) 

  * @retval C_KZG_OK Commitment computation successful 

  * @retval C_KZG_MALLOC Memory allocation failed 

  */ 

 static C_KZG_RET poly_to_kzg_commitment(g1_t *out, const Polynomial *p, const KZGSettings *s) { 

 return g1_lincomb(out, s->g1_values, (const fr_t *)(&p->evals), FIELD_ELEMENTS_PER_BLOB); 

 }

Remove "-Bsymbolic" compilation requirement from python bindings

Currently the python bindings do not compile on MacOS. It seems like the "-Bsymbolic" compiler option is not available, and without it it's reporting lots of missing python symbols.

Tagging @hwwhww

npm package error

Is this repo the src for npm package?

Error log:
http://pastie.org/p/25xkd6j82tGM4x8gjQ4bMl

Cant get it installed on Ubuntu18, but I think this one flag it complains about -fPIC might solve the issue. on gcc11 seems to work fine.

Segfault in computeAggregateKzgProof with invalid blob

This may only pertain to the Java bindings for this function.

If a zero-length blob is passed to computeAggregateKzgProof it will cause a segmentation fault. The problem is that this is allowed. I know that a blob should be 32 bytes, but the method allows arbitrary byte[] types without length constraints.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000103059330, pid=53565, tid=5891
#
# JRE version: OpenJDK Runtime Environment Homebrew (11.0.16.1) (build 11.0.16.1+0)
# Java VM: OpenJDK 64-Bit Server VM Homebrew (11.0.16.1+0, mixed mode, tiered, compressed oops, g1 gc, bsd-aarch64)
# Problematic frame:
# C  [libckzg4844jni.dylib+0x21330]  blst_scalar_from_lendian+0x8
#

Excerpt of the stack trace:

C  [libckzg4844jni.dylib+0x21330]  blst_scalar_from_lendian+0x8
C  [libckzg4844jni.dylib+0x39a8]  Java_ethereum_ckzg4844_CKZG4844JNI_computeAggregateKzgProof+0x68
j  ethereum.ckzg4844.CKZG4844JNI.computeAggregateKzgProof([BJ)[B+0
j  tech.pegasys.teku.kzg.ckzg4844.CKZG4844.computeAggregateKzgProof(Ljava/util/List;)Ltech/pegasys/teku/kzg/KZGProof;+13
j  tech.pegasys.teku.kzg.ckzg4844.CKZG4844PropertyTest.computeAggregateKzgProofThrowsExpected(Ljava/util/List;)V+11
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x32ffa8]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x2e0
V  [libjvm.dylib+0x5d60c0]  invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x76c
V  [libjvm.dylib+0x5d58f8]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x114
V  [libjvm.dylib+0x3c2350]  JVM_InvokeMethod+0x210

compute_aggregate_kzg_proof calls poly_from_blob:

c-kzg-4844/src/c_kzg_4844.c

Lines 1202 to 1205 in 130b629

 C_KZG_RET ret; 

 for (size_t i = 0; i < n; i++) { 

 ret = poly_from_blob(polys[i], blobs[i]); 

 if (ret != C_KZG_OK) {

poly_from_blob calls bytes_to_bls_field:

c-kzg-4844/src/c_kzg_4844.c

Lines 934 to 938 in 130b629

 static C_KZG_RET poly_from_blob(Polynomial p, const Blob blob) { 

 C_KZG_RET ret; 

 for (size_t i = 0; i < FIELD_ELEMENTS_PER_BLOB; i++) { 

 ret = bytes_to_bls_field(&p[i], &blob[i * BYTES_PER_FIELD_ELEMENT]); 

 if (ret != C_KZG_OK) return ret;

bytes_to_bls_field calls blst_scalar_from_lendian:

c-kzg-4844/src/c_kzg_4844.c

Lines 849 to 855 in 130b629

 C_KZG_RET bytes_to_bls_field(BLSFieldElement *out, const uint8_t bytes[32]) { 

 blst_scalar tmp; 

 blst_scalar_from_lendian(&tmp, bytes); 

 if (!blst_scalar_fr_check(&tmp)) return C_KZG_BADARGS; 

 blst_fr_from_scalar(out, &tmp); 

 return C_KZG_OK; 

 }

Associated with:

Consensys/teku#6596

Include subgroup checks when deserialising points

Checking the code here : https://github.com/ethereum/c-kzg-4844/blob/main/src/c_kzg_4844.c#L498 It seems that we are not checking whether points are in the subgroup and are just checking whether points are in the group.

Simplify `verify_kzg_proof()` return convention

Right now verify_kzg_proof() (and verify_blob_kzg_proof() and verify_kzg_proof_batch()) will return C_KZG_RET to indicate if the computation was finished properly, and then it will store the result of the verification in the argument ok.

This means that their callers should check both the retval and ok to handle verification correctly. Given the crucial importance of getting verification correctly, it might be a good idea to simplify this two-layer checking. The worst case scenario is that a user of the library thinks that C_KZG_OK means that verification went well.

My suggestion atm would be to merge ok into retval and make it such that verify_kzg_proof() returning C_KZG_OK denotes both a correct computation and also a correct verification. Then we could add another element to the enum called C_KZG_BAD_VERIFICATION if the caller wants to handle that case in a special way. This way the verifier just needs to check retval and I assume that for most clients it's gonna look like:

if retval == C_KZG_OK:
   accept_tx
else:
  reject_tx

but perhaps for others it could look like:

if retval == C_KZG_OK:
  accept_tx
else if retval == C_KZG_MALLOC:
  die()
else:
  reject_tx

It still seems simpler than the status quo.

Link readme in C# nuget package

When publishing the C# package, nuget prints a warning about a missing readme:

Pushing Ckzg.Bindings.0.1.2.51.nupkg to 'https://www.nuget.org/api/v2/package'...
  PUT https://www.nuget.org/api/v2/package/
warn : Readme missing. Go to https://aka.ms/nuget-include-readme learn How to include a readme file within the package.
  Created https://www.nuget.org/api/v2/package/ 1174ms
Your package was pushed.

I think it would be good to add this, seems pretty simple:

https://aka.ms/nuget-include-readme

Crashes with SIGILL on Linux/amd64 i7 CPU

Caught SIGILL in blst_cgo_init [...]

Benchmarks are broken

Benchmarks for other bindings are likely broken as well. It would be great if they were included in CI.

┌─[tautvydas][kagamin][±][main ✓][~/Git/c-kzg-4844/bindings/rust]
└─▪ cargo bench
   Compiling c-kzg v0.1.0 (/mnt/home/Git/c-kzg-4844/bindings/rust)
error[E0308]: mismatched types
   --> benches/kzg_benches.rs:56:61
    |
56  |                         .verify_aggregate_kzg_proof(&blobs, &kzg_commitments, &kzg_settings)
    |                          --------------------------         ^^^^^^^^^^^^^^^^ expected slice, found struct `Vec`
    |                          |
    |                          arguments to this function are incorrect
    |
    = note: expected reference `&[Bytes48]`
               found reference `&Vec<c_kzg::KZGCommitment>`
note: associated function defined here
   --> /mnt/home/Git/c-kzg-4844/bindings/rust/src/lib.rs:205:12
    |
205 |     pub fn verify_aggregate_kzg_proof(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0308`.
error: could not compile `c-kzg` due to previous error

Broken CI in rust-windows because of rust-1.7.0

See #311 (comment)

Run integration tests with ASan

We should tweak our Makefile to run our unittests with ASan at some point, to detect various C memory issues that can happen.

Align trusted setups with the spec

Align with: https://github.com/ethereum/consensus-specs/tree/dev/presets

Java/node bindings: Insufficient validation of inputs to `verify_aggregate_kzg_proof()`

The c-kzg public method verify_aggregate_kzg_proof() expects that its blobs and expected_kzg_commitments arguments have length of size n.

This is an undocumented requirement but still real because of the array accessing loop in

c-kzg-4844/src/c_kzg_4844.c

Lines 1290 to 1294 in c72ea8e

 for (size_t i = 0; i < n; i++) { 

 ret = poly_from_blob(&polys[i], &blobs[i]); 

 if (ret != C_KZG_OK) goto out; 

 ret = poly_to_kzg_commitment(&commitments[i], &polys[i], s); 

 if (ret != C_KZG_OK) goto out;

However, this is not respected by all bindings:

The Java bindings do not check that there are count commitments present
The nodejs bindings assume that there are as many commitments as blobs and then freely copy memory into the short commitments[blob_index] array

I suggest we document this requirement in the core c-kzg library so that bindings know what to expect. We also need to fix the above two cases. We might also need some unittests that check this behavior.

Update java ckzg header to new interface

This file is pretty out-of-date:

https://github.com/ethereum/c-kzg-4844/blob/main/bindings/java/c_kzg_4844_jni.h

It says that it's a machine generated file though, not quite sure how to generate it.

c-kzg-4844/bindings/java/c_kzg_4844_jni.h

Lines 1 to 3 in edb1274

 /* DO NOT EDIT THIS FILE - it is machine generated */ 

 #include <jni.h> 

 /* Header for class ethereum_ckzg4844_CKZG4844JNI */

I'm surprised there are no complaints anywhere. Where is this used?

Polish nodejs bindings

We merged the nodejs bindings in #147 but there is still polishing work to be done.
Please look at #147 for unresolved issues.

Most importantly:

the VerifyBlobKzgProofBatch() function is quite messy. How can we simplify it? (See #147 for some ideas)
verification checks are split between the C code and the typescript code. How can we unify them?

/cc @dapplion @dgcoffman

load_trusted_setup_file() cannot load the minimal trusted setup

Test hard-wired to 4096, no check for FIELD_ELEMENTS_PER_BLOB not a power of 2.

The current tests run against the (mainnet) trusted setup, which has a value of FIELD_ELEMENTS_PER_BLOB of 4096 and a correspondingly sized trusted setup.

However, the library actually takes FIELD_ELEMENTS_PER_BLOB as an external input (defined in the Makefile as a compile-flag). This suggest that the library should also work for a different value of FIELD_ELEMENTS_PER_BLOB. In particular, the way at least parts of the code are written suggest that the library should also work for FIELD_ELEMENTS_PER_BLOB not a power of 2.

With the test setup as is, it is not possible to check that (if it is even true in the first place).

A possible fix would be for the tests to create their own mock (un-)trusted setup of the correct size. (Such a mock trusted setup would basically choose a known tau rather than powers-of-tau ceremony-- note that this would not be secure in production).

Consider simplifying `g1_mul()`

g1_mul() is being too smart in an attempt to optimize for FFTs which we don't really use:

c-kzg-4844/src/c_kzg_4844.c

Lines 362 to 378 in ecc668b

 static void g1_mul(g1_t *out, const g1_t *a, const fr_t *b) { 

 blst_scalar s; 

 blst_scalar_from_fr(&s, b); 

 // Count the number of bytes to be multiplied. 

 int i = sizeof(blst_scalar); 

 while (i && !s.b[i - 1]) 

 --i; 

 if (i == 0) { 

 *out = G1_IDENTITY; 

 } else if (i == 1 && s.b[0] == 1) { 

 *out = *a; 

 } else { 

 // Count the number of bits to be multiplied. 

 blst_p1_mult(out, a, s.b, 8 * i - 7 + log_2_byte(s.b[i - 1])); 

 } 

 }

I think we can probably make it look exactly like g2_mul() without much loss in performance (maybe the current impl is faster when doing 1*P which happens in batch verification, but IMO that's not sufficient gain to warrant this extra complexity).

Be intentional about zero inputs in `fr_batch_inv()`

fr_batch_inv() is used in c-kzg to implement the bls_modular_inverse() of polynomial-commitments.md.

The documentation of the latter says return 0 for x == 0, but fr_batch_inv() would not have this behavior. In particular, due to using the batch inversion algorithm, if an input is zero it would also zero the entire output array and not just that index.

This should not be exploitable with the current protocol (since we are only inverting known values or RO outputs) but we should figure out how to handle this in a less suprirsing manner either way IMO.

Two approaches:

Return C_KZG_BADARGS from fr_batch_inv() if an input is zero (not the spec's behavior, but at least it's not a surprising behavior if we document it)
Skip zeroes during the batch inversion like kev's gnark library is doing. This would satisfy the spec behavior IIUC, but it's slightly more complicated to implement. This is also what arkworks is doing fwiw.

(while we are at it, we should also see if we can ditch the prod array and only do it with a single field element like gnark is doing. or i suspect we might be doing an extra field mul in the second loop)
(while we are at it, we should also document that fr_batch_inv does not handle len==0)

Optimise polynomial evaluation

Copying an idea of @asn-d6's from chat:

you can avoid a needless non-batched bls_modular_inverse in evaluate_polynomial_in_evaluation_form() by computing and caching inverse_width at boot time. then you wouldn't need:

fr_from_uint64(&tmp, FIELD_ELEMENTS_PER_BLOB);
fr_div(out, out, &tmp);

and you could replace it with a multiplication by inverse_width instead

	static bool fr_is_one(const fr_t *p) {
	uint64_t a[4];
	blst_uint64_from_fr(a, p);
	return a[0] == 1 && a[1] == 0 && a[2] == 0 && a[3] == 0;
	}

	static bool fr_equal(const fr_t aa, const fr_t bb) {
	uint64_t a[4], b[4];
	blst_uint64_from_fr(a, aa);
	blst_uint64_from_fr(b, bb);
	return a[0] == b[0] && a[1] == b[1] && a[2] == b[2] && a[3] == b[3];
	}

	export type Bytes32 = Uint8Array; // 32 bytes
	export type Bytes48 = Uint8Array; // 48 bytes
	export type KZGProof = Uint8Array; // 48 bytes
	export type KZGCommitment = Uint8Array; // 48 bytes
	export type Blob = Uint8Array; // 4096 * 32 bytes

	/*
	* This value represents the number of field elements in a blob. It must be
	* supplied externally. It is expected to be 4096 for mainnet and 4 for minimal.
	*/
	#ifndef FIELD_ELEMENTS_PER_BLOB

	static C_KZG_RET g1_lincomb(g1_t out, const g1_t p, const fr_t *coeffs, const uint64_t len) {
	C_KZG_RET ret = C_KZG_MALLOC;
	void *scratch = NULL;
	blst_p1_affine *p_affine = NULL;
	blst_scalar *scalars = NULL;

	// Tunable parameter: must be at least 2 since Blst fails for 0 or 1
	if (len < 8) {
	// Direct approach
	g1_t tmp;
	*out = G1_IDENTITY;
	for (uint64_t i = 0; i < len; i++) {
	g1_mul(&tmp, &p[i], &coeffs[i]);
	blst_p1_add_or_double(out, out, &tmp);
	}
	} else {
	// Blst's implementation of the Pippenger method
	scratch = malloc(blst_p1s_mult_pippenger_scratch_sizeof(len));
	if (scratch == NULL) goto out;
	p_affine = malloc(len * sizeof(blst_p1_affine));
	if (p_affine == NULL) goto out;
	scalars = malloc(len * sizeof(blst_scalar));
	if (scalars == NULL) goto out;

	// Transform the points to affine representation
	const blst_p1 *p_arg[2] = {p, NULL};
	blst_p1s_to_affine(p_affine, p_arg, len);

	// Transform the field elements to 256-bit scalars
	for (int i = 0; i < len; i++) {
	blst_scalar_from_fr(&scalars[i], &coeffs[i]);
	}

	// Call the Pippenger implementation
	const byte scalars_arg[2] = {(byte )scalars, NULL};
	const blst_p1_affine *points_arg[2] = {p_affine, NULL};
	blst_p1s_mult_pippenger(out, points_arg, len, scalars_arg, 256, scratch);
	}

	ret = C_KZG_OK;

	out:
	free(scratch);
	free(p_affine);
	free(scalars);
	return ret;
	}

	/**
	* Compute a KZG commitment from a polynomial.
	*
	* @param[out] out The resulting commitment
	* @param[in] p The polynomial to commit to
	* @param[in] s The settings struct containing the commitment key (i.e. the trusted setup)
	* @retval C_KZG_OK Commitment computation successful
	* @retval C_KZG_MALLOC Memory allocation failed
	*/
	static C_KZG_RET poly_to_kzg_commitment(g1_t out, const Polynomial p, const KZGSettings *s) {
	return g1_lincomb(out, s->g1_values, (const fr_t *)(&p->evals), FIELD_ELEMENTS_PER_BLOB);
	}

	C_KZG_RET ret;
	for (size_t i = 0; i < n; i++) {
	ret = poly_from_blob(polys[i], blobs[i]);
	if (ret != C_KZG_OK) {

	static C_KZG_RET poly_from_blob(Polynomial p, const Blob blob) {
	C_KZG_RET ret;
	for (size_t i = 0; i < FIELD_ELEMENTS_PER_BLOB; i++) {
	ret = bytes_to_bls_field(&p[i], &blob[i * BYTES_PER_FIELD_ELEMENT]);
	if (ret != C_KZG_OK) return ret;

	C_KZG_RET bytes_to_bls_field(BLSFieldElement *out, const uint8_t bytes[32]) {
	blst_scalar tmp;
	blst_scalar_from_lendian(&tmp, bytes);
	if (!blst_scalar_fr_check(&tmp)) return C_KZG_BADARGS;
	blst_fr_from_scalar(out, &tmp);
	return C_KZG_OK;
	}

	for (size_t i = 0; i < n; i++) {
	ret = poly_from_blob(&polys[i], &blobs[i]);
	if (ret != C_KZG_OK) goto out;
	ret = poly_to_kzg_commitment(&commitments[i], &polys[i], s);
	if (ret != C_KZG_OK) goto out;

	/* DO NOT EDIT THIS FILE - it is machine generated */
	#include <jni.h>
	/* Header for class ethereum_ckzg4844_CKZG4844JNI */

	static void g1_mul(g1_t out, const g1_t a, const fr_t *b) {
	blst_scalar s;
	blst_scalar_from_fr(&s, b);

	// Count the number of bytes to be multiplied.
	int i = sizeof(blst_scalar);
	while (i && !s.b[i - 1])
	--i;
	if (i == 0) {
	*out = G1_IDENTITY;
	} else if (i == 1 && s.b[0] == 1) {
	out = a;
	} else {
	// Count the number of bits to be multiplied.
	blst_p1_mult(out, a, s.b, 8 * i - 7 + log_2_byte(s.b[i - 1]));
	}
	}

ethereum / c-kzg-4844 Goto Github PK

c-kzg-4844's Introduction

C-KZG-4844

Bindings

Interface functions

Remarks

Tests

Parallelization

Batched verification

Benchmarks

Security audit

Why C?

c-kzg-4844's People

Contributors

Stargazers

Watchers

Forkers

c-kzg-4844's Issues

Recommend Projects

Recommend Topics

Recommend Org