I'd like to revive Desub. Given the various things that have changed in the last couple of years, I'm proposing to restructure and rework the library with the following rough goals:
- To expose an as-simple-as-possible interface for decoding arbitrary blocks/state. With the right configuration it should be possible to tie this in with pulling from a DB or from RPCs to index chain state/blocks in some arbitrary way.
- To be able to reuse the extrinsic/state decoding logic in eg Subxt (without pulling in a load of unnecessary stuff).
- To lean on the modern libraries we have nowadays like
scale_value and scale_decode.
- Excellent errors, examples, and a CLI utility to help one to iteratively construct/add type mappings for a chain (but we'll provide default mappings from PJS which should get things going).
Some more specific details:
Decoding arbitrary bytes
This is the core thing that needs doing, regardless of metadata version.
To be general over how we decode types, TypeDecoder is a trait that can be implemented on anything capable of decoding bytes into outputs.
It might look something like:
trait TypeDecoder {
/// Something that identifies a type. in V14/V15 metadata
/// this might be a u32, and in earlier versions it might be
/// some struct with chain and type name (maybe spec too).
type TypeId;
/// Error type we'll return if decoding fails.
type Error;
/// Given a type ID and a cursor, attempt to decode the type into a
/// `scale_value::Value`, consuming from the cursor. The Value will
/// contain the `TypeId` as context, so that we have information on the
/// type that the Value came from.
fn decode_type<V: scale_decode::Visitor>(
&self,
type_id: &Self::TypeId,
bytes: &mut &[u8],
visitor: V
) -> Result<V::Value, Self::Error>;
}This could be implemented directly on a scale_info::PortableRegistry. For older versions of metadata, we'll need a set of manual type mappings and we'll implement it on that.
Hopefully decode_type can take a scale_decode::Visitor, and will call that as needed. This would allow us to do things like skip over bytes for more decode flexibility, or decode to scale_value::Values or whatever, and generally leverage the Visitor stuff better. In V14 and V15 this would all "just work", but we'd need to implement a visitor based decoder to work with legacy type mappings.
Decoding extrinsics
We can use the type decoder above, as well as something capable of getting the relevant type IDs, to decode extrinsics
/// Provide the type information needed to decode extrinsics. This allows
/// us to write `decode_Extrinsic_v4` once for all metadata versions
trait ExtrinsicTypes {
/// Expected to be compatible with a `TypeDecoder`, so that we can
/// use the type IDs to actually decode some things.
type TypeId;
// Signature type
fn signature_type(&self) -> Self::TypeId;
// Address type
fn address_type(&self) -> Self::TypeId;
// Info on how to decode each of the signed extra types.
fn signed_extra_types(&self) -> impl Iterator<Item = (&str, Self::TypeId)>;
// Names of each argument and the type IDs to help decode them.
fn argument_types(&self, pallet_id: u8, call_id: u8) -> impl Iterator<Item = (&str, Self::TypeId)>;
}
// Now, combining the above with a compatible type decoder should let us decode extrinsics.
fn decode_extrinsic<D, E, Id>(extrinsic_types: &E decoder: &D, bytes: &mut &[u8]) -> Result<ExtrinsicDetails<Id>, Error<D::Error>>
where
D: TypeDecoder<TypeId = Id>,
E: ExtrinsicTypes<TypeId = Id>
{
// We should be able to write this logic once, and then reuse it for any V4 extrinsic
}Potentially decode_extrinsic could take a visitor too to define how the args are decoded, or it just stores byte offsets for things after decoding everything with an IgnoreVisitor and allows the user to then decode each signed extension etc into Values or concrete types or whatever (one might want to decode eg CheckMortality into a concrete type, but take the call data into Values or whatever.
Decoding storage
I imagine that we can follow the same pattern as for decoding extrinsics; a trait to fetch whatever information we need and hand it back in some generic way, and then a decode_storage type call which can decode into some StorageDetails struct.
General structure
We might end up with the following general crate/export structure:
// The core decode logic.
use desub::{
// The core traits:
TypeDecoder,
ExtrinsicTypes,
StorageTypes,
// The core functionality we expose is quite simple; start with eg:
decode_extrinsic,
decode_storage_entry,
Error,
// Define the type mapping stuff that legacy metadatas need to be
// able to decode types, and impl the above traits on the relevant
// decoding and old metadatas.
#[cfg(feature = "legacy")]
legacy::{
LegacyTypeDecoder
// legacy impls for the above live here too; if using this in
// Subxt we should be able to not include the legacy stuff
}
#[cfg(feature = "current")]
current::{
// Probably just impls of the traits for V14/V15 metadatas here.
}
};After this initial restructuring is done, we might also end up wanting to add:
- A
desub CLI utility tool which can:
- Scan a node to find where all of the spec version change (or runtime update) blocks are and which metadata version is in use for each spec version (binary chopping to locate the blocks with the diffs shouldnt take toooo long). Could dump this all to a file, and dump the metadatas too. Would be very easy to then scan and decode all blocks (assuming correct type mappings).
- Check whether an RPC connection is an archive node (maybe by querying for block 1 hash and then seeing whether we can get state there)
- Sample and attempt to decode blocks in each different spec version pre-V14 to help a user to construct and validate type mappings and spot errors. (maybe allow to scan in one spec version at a time to help the building of mappings)
We can raise separate issues to implement these bits later.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent