Problem to be solved

app package unit tests sometimes flap due to bind: address already in use.

Proposed solution

Not sure how to ensure/lock an available port. I can only think of somehow catching that error and skipping the test in that case.

Problem to solve

I cannot access or download the latest docker images after #134 was merged.

Proposed Solution

• charon/.github/workflows/release.yml wasn't updated (still points to old url)
• I can only get access and download the old stuff (docker.pkg.github.com)
• Maybe revert until we can fix things

Problem to solve

We have all the workflow component, but they are not integrated.

Proposed solution

Since we only have simnet at this point, integrate a simnet core workflow with beacon mock and validator mock.

Problem to solve

We are going to open source this repo soon, but it is still in early phase of development, and this isn't clear from the main readme.

Proposed Solution

• Add a "Project Status" section to the main readme.
• Explain that it is still in early phase of development
• Add a list of high level TODO checkmarks that contain all the components in the core workflow architecture.

Problem to solve

As the next step in the charon evolution, we want to run a simnet but with a real lighthouse validator client instead of a validator mock.

Proposed solution

• Refactor simnet keys file to EIP 2335 format.
• Add a flag to disable simnet mock validator
• Add lighthouse to charon-docker-compose (node 1)
• Do something with non-attestion endpoints that the validator will be querying...

Problem to solve

We have a required PR templates in the docs. But it needs to be created from scratch in each PR and the result in verified.

Proposed solution

Add a PR template that is automatically suggested when creating new PRs.
Add a github action that verifies the PR matches the template. Preventing merging if PR incorrect.

Problem to solve

We currently verify partial signatures submitted by the VC in the validatorapi. This doesn't prevent invalid signatures being added by our peers.

Proposed solution

• Decouple verification by providing a VerifyFunc to parsigDB.
• type VerifyFunc func(ctx context.Context, duty core.Duty, pubkey core.PubKey, parSig core.ParSignedData) error
• Pass in verifyFunc into NewParSigDB, call this function before storing.
• Extract verification logic from validatorapi and move to eth2util/signing, providing a Veirfy(...) error funciton.
• For DKG, we implement a custom Varify function that can do lockhash and depositdata verifucation based on custom duty types.

Problem to solve

We are adding support for DutyRandao, but the aggsigdb component doesn't exist yet.

Proposed solution

• Create an AggSigDB component, see DutyDB for reference.
• In-memory DB is fine for now.
• Store all duty types forever for now, we can add trimming later.
• See #219 for how the AggSignedData is encoded for DutyRandao (it is just a eth2p0.BLSSignature). This might need to be implemented if #219 isn't done yet.
• Integrate it with SigAgg component (not Fetcher yet)

Problem to solve

The issues that we raised(#168) on coinbase/kryptology are fixed in their latest release of v1.8.0. We should revert back to original kryptology package from our temporary one that is github.com/dB2510/kryptology.

Proposed solution

• Migrate back to github.com/coinbase/kryptology

Problem to be solved

We have a basic Jaeger config, but we haven't integrated it properly or tested it simnet yet.

Proposed solution

• Add deterministic tracing root so simnet traces are correlated.
• Add support for jaeger service name overrides, so simnet has different colored traces per node.
• Add abstraction layer to wrap core workflow component functions.
• Trace all components

Problem to solve

The leadercast package was implemented against an old Consensus interface. The latest architecture defines a different core workflow interface.

Proposed solution

Wait for #163 and #146 , then based on that final types and interfaces, move and refactor the leadercast package to be compatible.

Problem to solve

Charon needs to schedule duties as part of the core workflow, we therefore need the component is called the Scheduler.

Proposed solution

As per architecture doc, create a package scheduler

At the start of each epoch

• It should resolve the validator_index for all DVs in the cluster
• It should calculate the duties for active validators
• It should then call fetcher when the duty should be performed

Problem to solve

Encountered the following error running docker compose:

charon-docker-compose-node0-1       | fatal error: concurrent map writes
charon-docker-compose-node0-1       |
charon-docker-compose-node0-1       | goroutine 12 [running]:
charon-docker-compose-node0-1       | runtime.throw({0x12f0ae1, 0xc000dbbe70})
charon-docker-compose-node0-1       | 	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/panic.go:1198 +0x71 fp=0xc000dbbe28 sp=0xc000dbbdf8 pc=0x437051
charon-docker-compose-node0-1       | runtime.mapassign_faststr(0x102fcc0, 0xc000c75c80, {0xc000252360, 0x27})
charon-docker-compose-node0-1       | 	/opt/homebrew/Cellar/go/1.17.6/libexec/src/runtime/map_faststr.go:294 +0x38b fp=0xc000dbbe90 sp=0xc000dbbe28 pc=0x4131ab
charon-docker-compose-node0-1       | github.com/obolnetwork/charon/p2p.newPingLogger.func1({0x15a99d0, 0xc000d8e9c0}, {0xc000252360, 0x27}, {0x0, 0x0})
charon-docker-compose-node0-1       | 	/Users/corver/repos/charon/p2p/ping.go:98 +0xf3 fp=0xc000dbbf20 sp=0xc000dbbe90 pc=0xe5c373
charon-docker-compose-node0-1       | github.com/obolnetwork/charon/p2p.pingPeer({0x15a99d0, 0xc000d8e9c0}, 0xc0001a8e20, {0xc000252360, 0x27}, 0xc000d9a160, 0x0)
charon-docker-compose-node0-1       | 	/Users/corver/repos/charon/p2p/ping.go:61 +0x15d fp=0xc000dbbf98 sp=0xc000dbbf20 pc=0xe5c01d
charon-docker-compose-node0-1       | github.com/obolnetwork/charon/p2p.NewPingService.func1·dwrap·1()

Proposed Solution

Add a mutex to the stateful ping logger in charon/p2p/ping.go:82#newPingLogger

Problem to solve

The core workflow types (DutyArg, DutyArgSet, etc) can be simplified.

Also, I think we should refactor DV identifier from VIdx PubKey. This was the original design, and since we now have explicit DutyArg/FetchArg type, we do not need to pull query args to the top level of the type system. Another reason why DVs should be identified by the pubkey rather than then VIdx, is that when downstream components need to lookup TSS from the manifest, doing so via the VIdx require additional shared state, while PubKey doesn't.

Proposed solution

If we assume the project structure in #155:

• Refactor VIdx -> PubKey
• Keep Duty
• Rename DutyArg -> FetchArg (and FetchArgSet)
• Rename DutyData -> UnsignedData (and UnsignedDataSet)
• Rename SignedDutyData -> ParSignedData (and ParSignedDataSet)
• Introduce -> AggSignedData (and AggSignedDataSet)
• Rename SigDB -> ParSigDB
• Rename SigEx -> ParSigEx
• Rename AggDB -> AggSigDB

Problem to solve

We are adding support for DutyRandao, but the validatorapi service doesn't support it yet.

Proposed solution

• When the VC requests a new unsigned block, it provides the randao reveal in the request
• See https://ethereum.github.io/beacon-APIs/#/ValidatorRequiredApi/produceBlockV2
• Add support for above endpoint to validatorapi/router.go and validatorapi.go
• For now, just extract the randao reveal, verify the signature, store it parsigdb and return an error.
• Define (De|En)codeRandaoParSignedData, using eth2p0.BLSSignature as the type.
• Get the sigRoot for verification by inferring epoch from the provided slot
• Store the randao reveal in the parsigdb.

Problem to solve

We want to integrate simnet core workflow into charon run command, but it requires private shares for the validatormock.

Proposed solution

• We need to add private shares for the validatormock.
• Suggest simple json array with hex encoded bytes called simnetkeys.json in datadir.
• gen-simnet command should generate above file
• Create a ticket to support this in charon-docker-compose

Problem to solve

We do not have a dutyDB yet.

Proposed solution

Wait for #163 and #146 , then based on that final types and interfaces, implement a dutyDB package with the following features:

• Implement an in-memory database.
• Only support on DutyAttester.
• Support idempotent inserts (duplicate primary key, identical data inserts is a noop)
• Error on slashable duplicate inserts (duplicate primary key, mismatching data inserts is a error)
• Add unit tests
• Add query interfaces (for DutyAttester)

Problem to be solved

Need a Bulldozer bot auto-merging PRs when they certain conditions.

Proposed solution

Ref: https://github.com/palantir/bulldozer

Problem to solve

When running the simnet, the following errors occur sporadically:

Simulate duty error: context deadline exceeded

Proposed solution

This is probably due to the nextSlot inline function in charon/app/simulate.go#newDutySimulator that is nondeterministic. I expect it is weird timing shenanigans that is causing it.

Since we are going to deprecate that duty simulator and replace it with real core workflow. This isn't a high priority to fix though.

Problem to solve

We do not have a ParSigDB yet.

Proposed solution

Wait for #163 and #146 , then based on that final types and interfaces, implement a ParSigDB package with the following features:

• Implement an in-memory database.
• Only support on DutyAttester.
• Support idempotent inserts (duplicate primary key, identical data inserts is a noop)
• Add unit tests
• Add query interfaces (for DutyAttester)

Problem to solve

We have a validatorAPI router, but it just does http wranging, we still need to implement the validatorAPI core workflow component that queries the DutyDB and stores results in the ParSigDB.

Proposed solution

Implement a validatorapi component as a handler to the router as per architecture spec.

Problem to solve

When running a simnet cluster via the run_cluster.sh script, the log output of all the nodes is merged, making it hard to distinguish which node is doing what.

Proposed solution

• Maybe look at using https://github.com/remi/teamocil to run the nodes in their own panes.
• Generate teamocil config to run the nodes in a 4-panel
• If teamocil/tmux not installed, print that the output is merged and that teamocil could be installed for better UX.
• If teamocol/tmux is installed, run it.

Problem to be solved

ENRs are long complex things that are calculated from p2pkeys and runtime IP addresses, this makes them hard to configure.

Configuring bootnode ENRs in docker-compose or k8s platforms are especially tricky since IPs are dynamic. The DNS name of a bootnode is however often known within a system (docker-compose/k8s), so this allows nodes to query the bootnode via DNS/http to resolve its ENR.

This approach has been very handy in docker-compose and k8s to resolve bootnode ENRs

Proposed solution

Adds support for querying/resolving bootnode ENRs via http/curl

Problem to be solved

The core package uses []byte as the type for signatures. This doesn't make the encoding of signatures explicit and will results in bugs sooner or later.

Proposed solution

• Introduce a new type core.Signature that extends []byte and is the binary form of a BLSSIgnature.
• Add tblsconv.SigToCore and tblsconv.SigFromCore for converting to/from our crypto library
• Also add core.SigFromETH2(eth2p0.Signature) core.Signature and a method to core.Signature{}.ToEth2() eth2p0.Signature to convert to/from the eth2 library.
• Replace the Signature field with this type in AggSignedData and ParSignedData.

Problem to solve

We want to add support for DutyProposer, but the validatorapi doesn't support it yet.

Proposed solution

• The /eth/v2/validator/blocks/{slot} endpoint should already have been added by #218 , if not , add it.
• After storing the randao reveal in the ParSigDB (added by #218)
• Query the DutyDB.AwaitBeaconBlock and return the result to the VC

When starting a node with stale manifest ENRs AND with a valid node ENR, discv5 doesn't resolve the correct IPs and pinging and duty simulation keeps on failing.

Figure out how discv5 works and to resolve this.

Might possibly need to only provide bootnodes that one can connect to? But since it is UDP, you cannot create a connection... 🤷

Problem to solve

We are adding support for DutyProposer, but the sigagg component doesn't support it yet.

Proposed solution

• See #225 for decoding core.ParSignedData into eth2spec.VersionedSignedBeaconBlock
• Define (De|En)codeBlockAggSignedData, using eth2spec.VersionedSignedBeaconBlock as the type.
• Support DutyProposer in getAggSignedData and getSignedRoot

Problem to solve

Charon nodes need to exchange partial signatures after receiving them from the validator clients. See architecture doc.

Proposed Solution

Wait for #163 and #146 , then based on that final types and interfaces, create the SigEx libp2p protocol and component (see architecture doc for interface).

• see leadercast.Transport interface and implementation for reference
• Can skip the MemTransport that leadercast implements in v1.
• Keep error handling very simple.
• write unit tests.

Problem to solve

We discovered two issues with the kryptology library and opened issues and a PR, but this is still unresolved.

Proposed solution

• Migrate to @dB2510 PR branch in the mean time to resolve the issues.
• Push the guys from kryptology to address the issues.

Problem to be solved

Charon only support console logging and the level cannot be controlled.

Proposed solution

Add log-level and log-format flags. Add support for json and logfmt formats.

Problem to be solved

The aid charon cluster p2p node discovery, we need discv5 bootnodes. But we do not have an easy way to run them.

Proposed solution

Add a new command bootnode that runs a p2p discv5 bootnode. Ensure it also serves it ENR via http.

Problem to be solved

We do not understand how Attestation Aggregation duties are to be supported in middleware architecture.

From the spec:

def get_slot_signature(state: BeaconState, slot: Slot, privkey: int) -> BLSSignature:
    domain = get_domain(state, DOMAIN_SELECTION_PROOF, compute_epoch_at_slot(slot))
    signing_root = compute_signing_root(slot, domain)
    return bls.Sign(privkey, signing_root)
    
def is_aggregator(state: BeaconState, slot: Slot, index: CommitteeIndex, slot_signature: BLSSignature) -> bool:
    committee = get_beacon_committee(state, slot, index)
    modulo = max(1, len(committee) // TARGET_AGGREGATORS_PER_COMMITTEE)
    return bytes_to_uint64(hash(slot_signature)[0:8]) % modulo == 0

That shows that slot_signature is required to calculate if a VC is an aggregator.

Not sure how/where we can get a slot signature, since the VC only submits a slot signature when submitting the result of attestions after it has calculated that it has the duty

Our downstream VCs will calculate "partial slot signatures" which will be different, so they will produce a slot signatures in the wrong (and random) slots.

Proposed solution

Figure something out

Problem to be solved

Charon needs to be backwards/forwards compatible with older/newer versions of itself. One of the points of interaction is parsigex p2p protocol. At this point, it just sends json messages. Json messages are hard to keep compatible.

Proposed solution

• Use protobufs for parsigex transport messages.
• Setup test framework for different versions of the protobufs.

Problem to solve

The "Project Structure" docs placeholder is empty, it needs to be populated.

Proposed Solution

Document what each folder in the repo contains and the reasoning behind it.

Problem to solve

The app run function is pretty complex since it not only wire states but also explicitly starts and stops all processes.

Proposed solution

Simplify by decoupling starting and stopping of processes by implementing a life cycle manager.

It should provide the following features:

• Processes register start and stop hooks (functions)
• Start hooks can either be called synchronously or asynchronously.
• Start hooks can use the application context (hard shutdown) or background context (graceful shutdown).
• Stop hooks are synchronous and use a shutdown context with 10s timeout.
• Ordering of start and stop hooks.
• Any error from start hooks immediately triggers graceful shutdown.
• Closing application context triggers graceful shutdown.
• Any error from stop hooks immediately triggers hard shutdown.

Problem to solve

The p2p flag names are bit long and convoluted, can't we simplify and clarify them?

Proposed solution

Remove reference to udp and tcp in all cases, except where necessary (for the addresses)

--p2p-allowlist string           
--p2p-denylist string            
--p2p-tcp-address strings        
--p2p-udp-address string         
--p2p-bootmanifest           
--p2p-bootnodes strings      
--p2p-peerdb string       

Problem to solve

The architecture spec defines a SigAgg component, but it doesn't exist yet.

Proposed solution

Implement it according to the spec.

Problem to be solved

We have a github action release.yml that is triggered when a commit is tagged. It generates a static changelog. But we also have a tool go run testutil/genchangelog/main.go that auto generates a changelog between two tags. But this is still run manually.

Proposed solution

Run genchangelog as part of release.yml and push its output as the release changlog.

Problem to solve

We have an acceptable architecture doc specifying the fetcher component, but we do not have an implementation.

Proposed solution

Follow spec

Problem to solve

Networking in a charon cluster is tricky. We need dedicated documentation aimed at explaining the networking architecture, requirements, and how to configure and debug it.

Proposed solution

• Add a markdown doc to docs/networking.md
• Explain what networking interfaces are used and what they do
• Explain how to setup a cluster.
• This document will need a lot of work over time, so just get something up as a start.

Problem to solve

We want to add support for DutyProposer, but the validatorapi doesn't support it yet.

Proposed solution

• Add the /eth/v1/beacon/blocks endpoint to the router and validatorapi component via the eth2client.BeaconBlockSubmitter interface.
• Lookup the DV pubkey by querying the DutyDB.AwaitBeaconBlock
• Verify the partial signature (unless #217 has been done)
• Add (En|De)codeBlockParSignedData to/from eth2spec.VersionedSignedBeaconBlock
• Encode the data and store it in parsigdb

Problem to be Solved

We use gpl code and currently have an apache license on this repo. This is not compliant.

Proposed Solution

Lets change our code to GPLv3

Why?

Its compliant with copyleft licensing

Why not something else?

AGPL is stricter I understand
LGPL is for libraries rather than code you run. You run charon you don't use it as a library.

Problem to solve

We want to add support for DutyProposer. But fetcher doesn't support it yet.

Proposed solution

• Add AggSigDB lookup registration.
• See #221 for how FetchArg is encoded.
• Add a fetchProposerData method that does the following
• Fetch RandaoReveal from AggSigDB and decode it.
• Fetch unsigned block from beacon node (by adding BeaconBlockProposalProvider to eth2Provider)
• Add (De|En)codeProposerUnsignedData to/from eth2spec.VersionedBeaconBlock
• Encode the block and send to consensus.

Problem to solve

charon-docker-compose supports running the simnet with real validator clients, lighthouse and teku. We need a unit/integration test to ensure that compatibility remains.

Proposed solution

• Write a unit test in a new package: /testutil/endtoend
• Copy app/simnet_test.go
• Add a flag to start the teku binary (charon will provide flags)
• Note we use teku since it doesn't have hardcoded networks and therefore support quick repetitive slots/epochs.

Problem to solve

We want to add support for DutyProposer, but Broadcaster doesn't support it yet.

Proposed solution

• Add support to Broadcast method
• Use DecodeBlockAggSignedData
• Call eth2client.BeaconBlockSubmitter

Problem to be solved

Charon currently auto-creates a p2p key (and enr) if no p2p key is found. The UX of this isn't great, since charon will still error on startup when a p2p key was generated since the manifest cannot match that p2p key.

Proposed solution

Add an explicit gen-p2pkey command that creates a p2p key and outputs the enr.
Do not auto generate p2pkey on startup, rather error.

Problem to solve

We are adding support for DutyProposer, but the scheduler doesn't support it yet.

Proposed solution

• Create (En|De)codeProposerFetchArg to and from eth2v1.ProposerDuty
• Add support for DutyRandao to resolveDuties

Problem to solve

We want to add support for DutyProposer, but the DutyDB doesn't support it yet.

Proposed solution

• See #222 for how UnsignedData is encoded and decode it
• Store it in a new map with slot as key and tuple of pubkey and block as value
• Add AwaitBeaconBlock method (follow AwaitAttestation as example)

Problem to solve

We are adding support for DutyRandao, but the sigagg component doesn't support it yet.

Proposed solution

• See #218 for decoding core.ParSignedData into eth2p0.BLSSignature
• Define (De|En)codeRandaoAggSignedData, using eth2p0.BLSSignature as the type.
• Support DutyRandao in getAggSignedData and getSignedRoot

Problem to solve

The unit test github action's cache is not working. It spends on average 1min downloading dependencies each time.

Here the cache is empty.
Here is the cache is not found or created

Proposed solution

Up to you