layr-labs / eigenda Goto Github PK
View Code? Open in Web Editor NEWSecure, high-throughput, and decentralized Data Availability
Home Page: https://www.eigenlayer.xyz/
License: Other
Secure, high-throughput, and decentralized Data Availability
Home Page: https://www.eigenlayer.xyz/
License: Other
When setting up the disperser client, I used the following config:
disp := clients.NewDisperserClient(&clients.Config{
Hostname: "disperser-holesky.eigenda.xyz",
Port: "443",
Timeout: 10 * time.Second,
UseSecureGrpcFlag: true,
}, signer)
Then, I called DisperseBlob:
blobStatus1, key1, err := disp.DisperseBlob(ctx, paddedData, []uint8{})
And I had no problem because the DisperseBlob method adds the dial options to the grpc call:
dialOptions := c.getDialOptions()
conn, err := grpc.Dial(addr, dialOptions...)
But then, when I wanted to run GetBlobStatus:
reply1, err = disp.GetBlobStatus(context.Background(), key1)
I get:
panic: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"
Because the grpc call is insecure by default in the GetBlobStatus:
conn, err := grpc.Dial(
addr,
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
To fix this, GetBlobStatus should use the dialOptions provided by the config in the disperserClient.
Set up a disperserClient with the following config:
disp := clients.NewDisperserClient(&clients.Config{
Hostname: "disperser-holesky.eigenda.xyz",
Port: "443",
Timeout: 10 * time.Second,
UseSecureGrpcFlag: true,
}, signer)
Call GetBlobStatus:
reply1, err = disp.GetBlobStatus(context.Background(), key1)
High: It blocks me from completing my task.
Running mainnet node with all env vars set from operators-setup-repo
and NODE_HOSTNAME is set to public IP but the dataapi.eigenda.xyz
port-check query is showing our private IP and but correct disperal/retieval port.
NODE_CLIENT_IP_HEADER=x-real-ip
NODE_HOSTNAME=<our-public-ip>
NODE_PUBLIC_IP_PROVIDER=seeip
Using nomad orchestrator and docker.
I can exec into the docker container and curl ifconfig.me
returns correct public IP.
I can curl our public IP / disperser port and the connection opens fine.
Why can't I just hardcode my public IP? We have tons of static public IPs routed for every server in all our datacenters and I have to rely on software to incorrectly guess my IP?
Please allow env var for static public IP
We can't run our node because of it wont receive disperser traffic..
eigenda v0.7.3
debian-bookworm
nomad with docker
High: It blocks me from completing my task.
failed to opt-in EigenDA Node Network for operator ID:xxxxxxxxxx, operator address: XXXXXXXXXXX, error: EstimateGasPriceAndLimitAndSendTx: failed to send txn (RegisterOperatorWithCoordinatorWithChurn): INTERNAL_ERROR: blobs limit in txpool is full
How to solve this problem
A runtime panic due to division by zero occurs in the InplaceFFT
function of the kzg
package when the input slice vals
is empty.
func (fs *FFTSettings) InplaceFFT(vals []bls.Fr, out []bls.Fr, inv bool) error {
n := uint64(len(vals))
...
if !bls.IsPowerOfTwo(n) {
return fmt.Errorf("got %d values but not a power of two", n)
}
...
// Potential division by zero
stride := fs.MaxWidth / n
...
}
func IsPowerOfTwo(v uint64) bool {
return v&(v-1) == 0
}
Testing IsPowerOfTwo
I found that it's incorrectly implemented. 0
is a power of 2, but shouldn't be.
func TestIsPowerOfTwo(t *testing.T) {
testCases := []struct {
input uint64
expected bool
}{
{0, false}, // 0 is not a power of 2
{1, true}, // 2^0
{2, true}, // 2^1
{3, false}, // Not a power of 2
{4, true}, // 2^2
}
for _, tc := range testCases {
t.Run(fmt.Sprintf("Input%d", tc.input), func(t *testing.T) {
result := IsPowerOfTwo(tc.input)
if result != tc.expected {
t.Fatalf("IsPowerOfTwo(%d) = %v; want %v", tc.input, result, tc.expected)
}
})
}
}
--- FAIL: TestIsPowerOfTwo (0.00s)
--- FAIL: TestIsPowerOfTwo/Input0 (0.00s)
/Users/estensen/Developer/eigenda/pkg/kzg/bn254/globals_test.go:49: IsPowerOfTwo(0) = true; want false
The reason is that v-1
will underflow to 18446744073709551615
Then 0000...0000 & 1111...1111 == 0
The correct implementation of IsPowerOfTwo
func IsPowerOfTwo(v uint64) bool {
return v != 0 && v&(v-1) == 0
}
While there might be validation other places so vals []bls.Fr
is never an empty slice I think there should be multiple layers of validation.
cefcf3e (latest main)
go version go1.21.4 darwin/amd64
$ cd pkg/encoding/encoder
$ go test -fuzz=FuzzOnlySystematic
...
--- FAIL: FuzzOnlySystematic (0.38s)
--- FAIL: FuzzOnlySystematic (0.00s)
testing.go:1504: panic: runtime error: integer divide by zero
goroutine 193 [running]:
runtime/debug.Stack()
/usr/local/Cellar/go/1.21.4/libexec/src/runtime/debug/stack.go:24 +0x9b
testing.tRunner.func1()
/usr/local/Cellar/go/1.21.4/libexec/src/testing/testing.go:1504 +0x1ee
panic({0x136cf00?, 0x15e65f0?})
/usr/local/Cellar/go/1.21.4/libexec/src/runtime/panic.go:914 +0x21f
github.com/Layr-Labs/eigenda/pkg/kzg.(*FFTSettings).InplaceFFT(0xc000331780, {0xc0069afd00, 0x0, 0x1}, {0x162a8c0, 0x0, 0x0}, 0x1)
/Users/estensen/Developer/eigenda/pkg/kzg/fft_fr.go:90 +0x51d
github.com/Layr-Labs/eigenda/pkg/encoding/encoder.(*Encoder).GetInterpolationPolyCoeff(0xc00032b740, {0xc0069afd00, 0x0, 0x1}, 0x0)
/Users/estensen/Developer/eigenda/pkg/encoding/encoder/interpolation.go:71 +0x128
github.com/Layr-Labs/eigenda/pkg/encoding/encoder.(*Encoder).MakeFrames(0xc00032b740, {0xc0069afd00?, 0x15fbc20?, 0x1})
/Users/estensen/Developer/eigenda/pkg/encoding/encoder/encode.go:94 +0x2b9
github.com/Layr-Labs/eigenda/pkg/encoding/encoder.(*Encoder).Encode(0xc00032b740, {0x162a8c0?, 0x0, 0x0})
/Users/estensen/Developer/eigenda/pkg/encoding/encoder/encode.go:52 +0x2e7
github.com/Layr-Labs/eigenda/pkg/encoding/encoder.(*Encoder).EncodeBytes(0x0?, {0x162a8c0?, 0x1?, 0x0?})
/Users/estensen/Developer/eigenda/pkg/encoding/encoder/encode.go:20 +0x58
github.com/Layr-Labs/eigenda/pkg/encoding/encoder_test.FuzzOnlySystematic.func1(0xc0069b81a0, {0x162a8c0, 0x0, 0x0})
/Users/estensen/Developer/eigenda/pkg/encoding/encoder/encoder_fuzz_test.go:22 +0xd4
reflect.Value.call({0x1361500?, 0x13f9bc8?, 0x13?}, {0x13b1ba6, 0x4}, {0xc00695b290, 0x2, 0x2?})
/usr/local/Cellar/go/1.21.4/libexec/src/reflect/value.go:596 +0xce7
reflect.Value.Call({0x1361500?, 0x13f9bc8?, 0x15bdfa0?}, {0xc00695b290?, 0x13b1100?, 0x113292d?})
/usr/local/Cellar/go/1.21.4/libexec/src/reflect/value.go:380 +0xb9
testing.(*F).Fuzz.func1.1(0x1006ca0?)
/usr/local/Cellar/go/1.21.4/libexec/src/testing/fuzz.go:335 +0x347
testing.tRunner(0xc0069b81a0, 0xc006959200)
/usr/local/Cellar/go/1.21.4/libexec/src/testing/testing.go:1595 +0xff
created by testing.(*F).Fuzz.func1 in goroutine 6
/usr/local/Cellar/go/1.21.4/libexec/src/testing/fuzz.go:322 +0x597
...
None
Is there support for restakers by email?
When I run eigenDA locally on my machine sometimes I see errors appear in terminal
ERROR[02-28|12:42:20.255|github.com/Layr-Labs/eigenda/retriever/metrics.go:69] Prometheus server failed
n tcp :9100: bind: address already in use" caller=metrics.go:69
ERROR[02-28|12:42:35.512|cmd/main.go:68] could not start node error
="failed to register the operator: failed to request churn approval: rpc error: code = Unavailable desc = connection error: desc =
\"transport: authentication handshake failed: tls: first record does not look like a TLS handshake\"" caller=main.go:68
2024/02/28 12:42:35 application failed: failed to register the operator: failed to request churn approval: rpc error: code
= Unava ilable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not
look like a TLS han dshake"
ERROR[02-28|13:30:26.504|github.com/Layr-Labs/eigenda/core/eth/tx.go:180] Failed to register operator
execution reverted: BLSRegistryCoordinatorWithIndices._registerOperatorWithCoordinatorAndNoOverfilledQuorums: quorum is overfilled " caller=tx.go:180
ERROR[02-28|13:30:26.532|cmd/main.go:68] could not start node error
EigenDA: opstack-poc repository
golang: 1.21 version
OS: MacOS Ventura 13.4
Machine: Macbook pro m1
Follow guide: https://docs.eigenlayer.xyz/eigenda/rollup-guides/op-stack/deploying-op-stack-+-eigenda-locally
Error appears in terminal when eigenDA is running locally
Low: It annoys or frustrates me.
When I run the end-to-end test, I cannot create an S3 bucket. The specific error returned is:
"An error occurred (502) when calling the CreateBucket operation (reached max retries: 2): Bad Gateway."
I am sure that Localstack is starting up normally.
eigenad branch: opstack-poc
os: macos
go version: go1.21.6
cd inabox; make run-e2e
Medium: It is a significant difficulty but I can work around it.
How should we address the problem of data explosion as time progresses? So far, I have not seen any discussions on this matter. My short-term solution has been to continuously add more disk storage, but this approach keeps driving up costs. Are there other viable alternatives?
AWS uses checksum for uploading files, but another checksum can be added and stored as metadata to be absolutely sure that the data that was supposed to be stored is stored. It's best practice for migrations of data and while you're not doing that here it might be good to have some extra protection.
Checksums can be specified on s3.PutObjectInput
Either use SHA256 as checksum or CRC32C. Since blobs are keyed by it can be used, but it's a bit less performant than CRC32C.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
https://aws.amazon.com/getting-started/hands-on/amazon-s3-with-additional-checksums/?ref=docs_gateway/amazons3/checking-object-integrity.html
When using the disperse function in Testnet, times until confirmation are quite high. It is takes up to 8-9 minutes. For example, in three samples, we got 8 minutes, 1:30 minutes, and then 9 minutes. This makes EigenDA not suitable for our application. We want to check if this is an expected, or just some temporal issue with the testnet
EigenDA - Testnet
You can run this script to take a sample of the time until confirmation. It needs jq
and grpcurl
, and to be run in the EigenDA repo after make build.
https://gist.github.com/MauroToscano/b6a15942e38715c5039503583a98bcbc
High: It blocks me from completing my task.
Please add github on eigenlayer website https://www.eigenlayer.xyz/
Running the unit tests locally fails
...
2023/11/16 16:00:45 Localstack started successfully! URL: http://0.0.0.0:4568
2023/11/16 16:00:45 Current Working Directory: /Users/estensen/Developer/eigenda/inabox
Creating S3 bucket
Creating S3 bucket
Creating S3 bucket
Creating S3 bucket
Creating S3 bucket
Creating S3 bucket
Creating S3 bucket
Could not connect to docker: reached retry deadline: exit status 127:
Stopping Dockertest resources
Expiring docker resource
Purging docker resource
panic: failed to deploy AWS resources
goroutine 1 [running]:
github.com/Layr-Labs/eigenda/disperser/apiserver_test.setup(0xc00039fe48?)
/Users/estensen/Developer/eigenda/disperser/apiserver/server_test.go:291 +0x258
github.com/Layr-Labs/eigenda/disperser/apiserver_test.TestMain(0x9fbb5a2deb6cb00b?)
/Users/estensen/Developer/eigenda/disperser/apiserver/server_test.go:45 +0x27
main.main()
_testmain.go:61 +0x308
FAIL github.com/Layr-Labs/eigenda/disperser/apiserver 16.282s
...
I also think that the docs should mention that you need to run Docker to run the unit tests.
main branch
go version go1.21.4 darwin/amd64
make unit-test
High: It blocks me from completing my task.
when launching a goerli testnet retriever from scratch, i got this error
ERROR[12-14|04:12:16.803|github.com/Layr-Labs/eigenda/indexer/indexer.go:140] Error pulling new headers err="413 Request Entity Too Large: <html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" caller=indexer.go:140
this is because indexer trying batch get headers from block number 0 to latest block number
https://github.com/Layr-Labs/eigenda/blob/master/indexer/eth/header_service.go#L45
there is no pagination logic, and the batch size is too large
https://github.com/Layr-Labs/eigenda/blob/master/indexer/eth/header_service.go#L43
should check the count, if count > X, then let count = X
No response
No response
eigenda/contracts/src/libraries/EigenDARollupUtils.sol
Lines 32 to 36 in 3c66c99
If this function is only exposed as external in a library, then any contract that uses it will be forced to deploy a linked library instead of having the flexibility to internalize it in their own contract's bytecode.
I spun up eigenDA and opstack locally according to the guide in docs:
https://docs.eigenlayer.xyz/eigenda/rollup-guides/op-stack/deploying-op-stack-+-eigenda-locally
After running "make devnet-up" command I saw the following error.
Even though in the step where it says "You can check docker logs of node and batcher to see relevant events related to EigenDA?"
Are there any suggestions to resolve the issue?
I try to follow the README in the inabox folder, but the make run-e2e
command fails. I expect it to succeed.
{"time":"2024-05-22T18:26:29.120717+01:00","level":"DEBUG","source":{"function":"github.com/Layr-Labs/eigenda/indexer.(*indexer).HandleAccumulator","file":"/Users/wawrzek/Ethereum/EigenLayer/eigenda/indexer/indexer.go","line":233},"msg":"Handling event","component":"Indexer","event":{"Type":"operator_socket_update","Payload":{"OperatorId":[251,57,10,100,18,45,179,149,127,178,32,195,196,45,95,113,233,122,176,201,149,218,78,30,92,195,38,22,2,218,197,39],"Socket":"localhost:32016;32017","Raw":{"address":"0xa82ff9afd8f496c3d6ac40e2a0f282e47488cfc9","topics":["0xec2963ab21c1e50e1e582aa542af2e4bf7bf38e6e1403c27b42e1c5d6e621eaa","0xfb390a64122db3957fb220c3c42d5f71e97ab0c995da4e1e5cc3261602dac527"],"data":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000156c6f63616c686f73743a33323031363b33323031370000000000000000000000","blockNumber":"0x5e","transactionHash":"0x9e445a8cb471364916e82681ee53ead76c04eb2110bd93e1468afc57d54f5d55","transactionIndex":"0x0","blockHash":"0x86a81ff82f5a182ed245e4fed8e52513e1c811093fb13ecd12160a92703a1d11","logIndex":"0x1","removed":false}}}}
{"time":"2024-05-22T18:26:29.120747+01:00","level":"DEBUG","source":{"function":"github.com/Layr-Labs/eigenda/indexer.(*indexer).HandleAccumulator","file":"/Users/wawrzek/Ethereum/EigenLayer/eigenda/indexer/indexer.go","line":233},"msg":"Handling event","component":"Indexer","event":{"Type":"operator_socket_update","Payload":{"OperatorId":[102,22,56,252,189,227,235,170,8,74,106,36,248,167,138,5,103,42,64,85,139,118,242,242,13,242,13,228,6,23,117,236],"Socket":"localhost:32006;32007","Raw":{"address":"0xa82ff9afd8f496c3d6ac40e2a0f282e47488cfc9","topics":["0xec2963ab21c1e50e1e582aa542af2e4bf7bf38e6e1403c27b42e1c5d6e621eaa","0x661638fcbde3ebaa084a6a24f8a78a05672a40558b76f2f20df20de4061775ec"],"data":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000156c6f63616c686f73743a33323030363b33323030370000000000000000000000","blockNumber":"0x5f","transactionHash":"0x002e0b87bba7366d531fdb147840c8735322980f0fe77fb4caa3b03f3d9e0d9a","transactionIndex":"0x0","blockHash":"0x293bc302c4e1faa4d0231e2240599c09aa184d18d4b3578466a5c0ad72c266e3","logIndex":"0x1","removed":false}}}}
{"time":"2024-05-22T18:26:29.120758+01:00","level":"DEBUG","source":{"function":"github.com/Layr-Labs/eigenda/indexer.(*indexer).HandleAccumulator","file":"/Users/wawrzek/Ethereum/EigenLayer/eigenda/indexer/indexer.go","line":233},"msg":"Handling event","component":"Indexer","event":{"Type":"operator_socket_update","Payload":{"OperatorId":[62,183,213,223,97,196,142,194,113,141,140,138,213,35,4,239,252,151,10,233,47,25,19,142,3,45,174,7,183,192,214,41],"Socket":"localhost:32011;32012","Raw":{"address":"0xa82ff9afd8f496c3d6ac40e2a0f282e47488cfc9","topics":["0xec2963ab21c1e50e1e582aa542af2e4bf7bf38e6e1403c27b42e1c5d6e621eaa","0x3eb7d5df61c48ec2718d8c8ad52304effc970ae92f19138e032dae07b7c0d629"],"data":"0x000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000156c6f63616c686f73743a33323031313b33323031320000000000000000000000","blockNumber":"0x5f","transactionHash":"0x68c69cbb716d41d826446e443602ded7e8d07c75561d00c28138380170fafcdc","transactionIndex":"0x1","blockHash":"0x293bc302c4e1faa4d0231e2240599c09aa184d18d4b3578466a5c0ad72c266e3","logIndex":"0xa","removed":false}}}}
------------------------------
• [FAILED] [10.001 seconds]
Inabox Integration [It] test end to end scenario
/Users/wawrzek/Ethereum/EigenLayer/eigenda/inabox/tests/integration_test.go:29
[FAILED] Expected
<*status.Error | 0x14000080480>:
rpc error: code = DeadlineExceeded desc = context deadline exceeded
{
s: {
s: {
state: {
NoUnkeyedLiterals: {},
DoNotCompare: [],
DoNotCopy: [],
atomicMessageInfo: nil,
},
sizeCache: 0,
unknownFields: nil,
Code: 4,
Message: "context deadline exceeded",
Details: nil,
},
},
}
to be nil
In [It] at: /Users/wawrzek/Ethereum/EigenLayer/eigenda/inabox/tests/integration_test.go:54 @ 05/22/24 18:26:39.115
------------------------------
Stopping binaries
Stopping anvil
Stopping graph node
Stopping Dockertest resources
Expiring docker resource
Purging docker resource
Summarizing 1 Failure:
[FAIL] Inabox Integration [It] test end to end scenario
/Users/wawrzek/Ethereum/EigenLayer/eigenda/inabox/tests/integration_test.go:54
Ran 1 of 1 Specs in 409.497 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestInaboxIntegration (409.50s)
=== RUN TestRatelimit
ratelimit_test.go:172: Manual test for now
--- SKIP: TestRatelimit (0.00s)
FAIL
FAIL github.com/Layr-Labs/eigenda/inabox/tests 409.768s
FAIL
make: *** [run-e2e] Error 1
EigenDA: master (commit cb32efd)
GoLang: go version go1.22.2 darwin/arm64
Os: MacOS (I see the same problem on Linux)
make build && cd inabox && make run-e2e
Medium: It is a significant difficulty but I can work around it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.