Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update reports to the Lido smart contracts.

There are two modules in the oracle:

• Accounting
• Ejector

Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck validators and processes user withdrawal requests. Also Accounting module makes decision to turn on/off the bunker.

Flow

The oracle work is delineated by time periods called frames. Oracles finalize a report in each frame. The default Accounting Oracle frame length on mainnet is 225 epochs, which is 24 hours (it could be changed by DAO). The frame includes these stages:

• Waiting - oracle starts as daemon and wakes up every 12 seconds (by default) in order to find the last finalized slot (ref slot). If ref slot missed, Oracle tries to find previous non-missed slot.
• Data collection: oracles monitor the state of both the execution and consensus layers and collect the data;
• Hash consensus: oracles analyze the data, compile the report and submit its hash to the HashConsensus smart contract;
• Core update report: once the quorum of hashes is reached, meaning required number of Oracles submitted the same hash, one of the oracles chosen in turn submits the actual report to the AccountingOracle contract, which triggers the core protocol state update, including the token rebase, finalization of withdrawal requests, and deciding whether to go in the bunker mode.
• Extra data report: an additional report carrying additional information. This part is required. All node operators rewards are distributed in this phase.

Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user withdrawals.

Flow

• Finds out how much ETH is needed to cover withdrawals.
• Predicts mean Lido income into Withdrawal and Execution Rewards Vaults.
• Figures out when the next validator will be withdrawn.
• Encode Lido validators to eject int bytes and send transaction.

Only Oracle:

• vCPUs - 2
• Memory - 8 GB

Oracle + KAPI:

• vCPU - 4
• Memory - 16 GB

To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on historical blocks. This requires an archive execution node. Oracle needs two weeks of archived data.

Also, to calculate some metrics for bunker mode Oracle needs archive consensus node.

This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in database.

Lido Keys API repository.

Oracle daemon could be run using a docker container. Images is available on Docker Hub. Pull the image using the following command:

docker pull lidofinance/oracle:{tag}

Where {tag} is a version of the image. You can find the latest version in the releases OR
You can build it locally using the following command (make sure build it from latest release):

docker build -t lidofinance/oracle .

Full variables list could be found here.

1. Use .env.example file content to create your own .env file. Set required values. It will be enough to run the oracle in check mode.
2. Check that your environment is ready to run the oracle using the following command:

   docker run -ti --env-file .env --rm lidofinance/oracle:{tag} check

   If everything is ok, you will see that all required checks are passed and your environment is ready to run the oracle.

1. By default, the oracle runs in dry mode. It means that it will not send any transactions to the Ethereum network. To run Oracle in production mode, set MEMBER_PRIV_KEY or MEMBER_PRIV_KEY_FILE environment variable:

   MEMBER_PRIV_KEY={value}
   

   Where {value} is a private key of the Oracle member account or:

   MEMBER_PRIV_KEY_FILE={path}
   

   Where {path} is a path to the private key of the Oracle member account.

2. Run the container using the following command:

   docker run --env-file .env lidofinance/oracle:{tag} {type}

   Replace {tag} with the image version and {type} with one of the two types of oracles: accounting or ejector.

Note: of course, you can pass env variables without using .env file. For example, you can run the container using the following command:

docker run --env EXECUTION_CLIENT_URI={value} --env CONSENSUS_CLIENT_URI={value} --env KEYS_API_URI={value} --env LIDO_LOCATOR_ADDRESS={value} lidofinance/oracle:{tag} {type}

Oracle could be executed once in "manual" mode. To do this setup DAEMON variable to 'False'.

Note: Use -it option to run manual mode in Docker container in interactive mode. Example docker run -ti --env-file .env --rm lidofinance/oracle:{tag} {type}

In this mode Oracle will build report as usual (if contracts are reportable) and before submitting transactions Oracle will ask for manual input to send transaction.

In manual mode all sleeps are disabled and ALLOW_REPORTING_IN_BUNKER_MODE is True.

LIDO_LOCATOR_ADDRESS=0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb ALLOW_REPORTING_IN_BUNKER_MODE=False

A few basic alerts, which can be configured in the Prometheus Alertmanager.

groups:
  - name: oracle-alerts
    rules:
      - alert: AccountBalance
        expr: lido_oracle_account_balance / 10^18 < 1
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Dangerously low account balance"
          description: "Account balance is less than 1 ETH. Address: {.labels.address}: {.value} ETH"
      - alert: OutdatedData
        expr: (lido_oracle_genesis_time + ignoring (state) lido_oracle_slot_number{state="head"} * 12) < time() - 300
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Outdated Consensus Layer HEAD slot"
          description: "Processed by Oracle HEAD slot {.value} too old"

Note: all metrics are prefixed with lido_oracle_ by default.

The oracle exposes the following basic metrics:

Special metrics for accounting oracle:

Special metrics for ejector oracle:

Special metrics for CSM oracle:

Python version: 3.12

1. Setup poetry
2. Install dependencies

poetry install

Required variables for accounting and ejector modules

export EXECUTION_CLIENT_URI=...
export CONSENSUS_CLIENT_URI=...
export KEYS_API_URI=...
export LIDO_LOCATOR_ADDRESS=...

Required variables for CSM module

export EXECUTION_CLIENT_URI=...
export CONSENSUS_CLIENT_URI=...
export KEYS_API_URI=...
export LIDO_LOCATOR_ADDRESS=...
export CSM_ORACLE_ADDRESS=...
export CSM_MODULE_ADDRESS=...
export MAX_CYCLE_LIFETIME_IN_SECONDS=60000  # Reasonable high value to make sure the oracle has enough time to process the whole frame.

Run oracle module

poetry run python -m src.main {module}

Where {module} is one of:

• accounting
• ejector
• csm
• check

Testing guide

poetry run pytest .

Used the following tools:

• black
• pylint
• mypy See the configuration for details for each linter.

Make sure that your code is formatted correctly and passes all checks:

black tests
pylint src tests
mypy src

2023 Lido [email protected]

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License, or any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

To create new release:

1. Merge all changes to the master branch
2. Navigate to Repo => Actions
3. Run action "Prepare release" action against master branch
4. When action execution is finished, navigate to Repo => Pull requests
5. Find pull request named "chore(release): X.X.X" review and merge it with "Rebase and merge" (or "Squash and merge")
6. After merge release action will be triggered automatically
7. Navigate to Repo => Actions and see last actions logs for further details