NOTE: README IS WIP

The XRP ledger produces a large amount of immutable historical data which currently must be maintained across a network of Full History nodes or it will be forever lost.

Note that some optimisation of history sharing and storage has already been achieved through History Sharding. However this proposal will make the case that the robustness and searchability of network history would be vastly improved by a new canonical mega-ledger format: Catalogue.

An XRPL Catalogue is a file. Each Cataglogue contains the complete data of 1048575 sequential validated ledgers. The file format is organised such that the ledger data is both well compressed and fast to search without an external index, or preparation time.

The file format is a binary format. It begins with a header followed by several fixed length indicies, finally followed by several variable length sections.

Each section is explained in detail after this table.

Grammatically:

    Catalogue ::= Header StateIndex LedgerIndex TransactionIndex AccountIndex Transaction Account Ledger State

The header is 48 bytes containing the following fields.

Grammatically:

    Header ::= MagicNumber(uint32) Version(uint32) FileOffset(uint64){4}

An array of 1048576 offets relative to LedgerOffset allowing the direct indexing of a particular ledger header given a ledger sequence number. The array index is the ledger sequence number less the FirstLedgerIndex.

    LedgerIndex ::= FileOffset(uint64){1048576}

An array of 128 offsets relative to StateOffset allowing the direct indexing of a particular StateEntry.

    StateIndex ::= FileOffset(uint64){128}

An array of 16777216 offsets relative to TransactionOffset. The index of this array is the first 3 bytes of the transaction hash. The offset is the first TransactionEntry that begins with those bytes or NOT_FOUND if no such transaction exists across the whole span of 1048576 ledgers.

    TransactionIndex ::= FileOffset(uint64){16777216}

An array of 16777216 offsets relative to the AccountOffset. The index of this array is the first 3 bytes of the Account ID under query. The offset is the beginning of the set of transactions that affected any account with this prefix within the 1048576 ledgers. The end of that set is the next non-NOT_FOUND entry in the array or the beginning of the next section if there are no further non-NOT_FOUND entries in the AccountIndex array.

    AccountIndex ::= FileOffset(uint64){16777216}

An ordered list of transactions with their metadata that occured across the whole 1048576 ledgers.

    TransactionEntry ::= Len(uint32) TransactionHash(uint256) TxnBlob(varlen) Meta(varlen)

    TransactionSection ::= TransactionEntry*

A list of AccountEntries sorted first by account ID then by transaction hash of transactions that affected those aforesaid accounts. Transactions are referenced via an offset relative to the Transaction Section (TransactionOffset). Transactions may be referenced more than once.

    AccountEntry ::= Len(uint32) AccountID(uint160) FileOffset(uint64)

    AccountSection ::= AccountEntry*

Ledger headers are stored uncompressed in this section.

Each Ledger header is placed into canonical XRPL serialized binary format then uint32 length prefixed.

    LedgerEntry ::= Len(uint32) SLedgerHeader(varlen)

These LedgerEntries are concatenated in sequential order to make up the section:

    LedgerSection ::= LedgerEntry{1048576}

Each consecutive set of 8192 ledgers, in canonical XRPL serialized binary format, are solidly compressed using 7z compression into an archive where the filename is the ledger number, as below:

    7ZipArchive ::= 
    7z compressed file structure
    [
        /
          |- 67256320           ** example **
          |- 67256321
          |- 67256322
          .
          .
          .
          |- 67264511
     ]

The 7ZipArchive is prefixed with a uint32 length. This is called a StateEntry:

    StateEntry ::= Len(uint32) 7ZipArchive(varlen)

Each of 128 State Entries are then sequentially written from the start of the State section to the end of the file.

    StateSection ::= StateEntry{128}