Add support for condition and context about deku HOT 18 CLOSED

Anyway, if we give a lot of useful attributes to a user to make the requirement of writing custom reader/writer be a very very rare case, then the static anlysis/the change of writing wrong code are all totally unimportant.
And if the rare case happens, it's not a big deal to write code carefully.

from deku.

@constfold Thanks for the ticket! ie. Context: Would you be able to provide an example of how it could be written?

1. Condition

I read the edit history, and yes you're right. You can pass previously read fields to the "reader" function. I think a cond attribute may be useful, as it could create the "wrapper" code for you for example. #deku[skip, cond = "a == 0x01"]. Writing a reader function is the way to do it currently as you discovered. The documentation could be expanded to clarify that you can pass previously read fields to the function.

2. Context

Currently the way it's structured, a child parser would not have the result of a parent parser.

Here is a hacky work around... I don't really like how arbitrary code can be written in attributes like that as it can lead to hacky solutions like the following (also contains a copy... maybe there's a way around that?) Let me know if you find a better solution :)

use deku::prelude::*;

#[derive(Debug, Clone, DekuRead)]
struct Data {
    a: u8,
    #[deku(reader = "Data::field_b_reader(rest)")]
    b: Option<u8>
}

impl Data {
    fn field_b_reader(
        //header_flag: u8, // ideally, we could get a reference to header.flag here
        rest: &BitSlice<Msb0, u8>,
    ) -> Result<(&BitSlice<Msb0, u8>, Option<u8>), DekuError> {

        Ok((rest, None))
    }
}

#[derive(Debug, DekuRead)]
struct Header {
    flag: u8,
    #[deku(map = "|data: Vec<Data>| -> Result<_, DekuError> {
        let data = data.clone().iter_mut().map(|v| {
            let mut v = v.clone();
            if (flag == 0xAB) {
                v.b = Some(1);
            }
            v
        }).collect::<Vec<Data>>();
        Ok(data)
    }", count = "1")]
    data: Vec<Data>
}

fn main() {
    let data = vec![0xABu8, 0x02];
    let (_rest, val) = Header::from_bytes((data.as_ref(), 0)).unwrap();
    println!("{:?}", val);
}

from deku.

@sharksforarms Thanks for your reply. If i do not understand it wrong, your example wont read any bytes from data but just skip that field and give it a hardcode value? It's not what i want to do exactly.
The solution i found is

1. using static(ugly but work)
2. make one of (input_is_le, bit_size, count) to represent the field we want. example:

use deku::prelude::*;

#[derive(DekuRead, PartialEq, Debug)]
struct FooPart(u8);

#[derive(PartialEq, Debug)]
struct Foo {
    a: FooPart,
    b: Option<u8>,
}

impl DekuRead for Foo {
    fn read(rest: &BitSlice<Msb0, u8>, input_is_le: bool, bit_size: Option<usize>, count: Option<usize>) -> Result<(&BitSlice<Msb0, u8>, Self), DekuError>
        where Self: Sized
    {
        // We separate those fields which we dont need to care to a new type
        let (rest, a) = FooPart::read(rest, input_is_le, None, None)?;
        let bar_flag = bit_size.unwrap() == 1;
        let (rest, b) = if bar_flag {
            u8::read(rest, input_is_le, None, None).map(|(r, v)|(r, Some(v)))?
        } else {
            (rest, None)
        };

        Ok((rest, Self { a, b }))
    }
}

#[derive(DekuRead)]
struct Bar {
    flag: u8,
    count: u8,
    #[deku(reader = "read(rest, input_is_le, count, flag)")]
    b: Vec<Foo>,
}

fn read(rest: &BitSlice<Msb0, u8>, input_is_le: bool, count: u8, a: u8) -> Result<(&BitSlice<Msb0, u8>, Vec<Foo>), DekuError> {
    // Using `bit_size` as flag is very tricky, but it often works even with more complex conditions.
    // And when we want get it(say bit_size), we can just use few bits of it. e.g:
    // 0 0 0 0 0 0 0 0 0 0   0 0 0 0 0 0 0 0 ... 0 0 0 0
    // <-bits means flag->   <- other bits means size ->
    // We can't use count to do this way because `Vec` will consume it.
    let flag = if a == 0x00 { 1 } else { 0 };
    let (rest, value) = Vec::<Foo>::read(rest, input_is_le, Some(flag), Some(count as usize))?;
    Ok((rest, value))
}


fn main() {
    let data = vec![0x00_u8, 0x01, 0x00, 0x01];
    let (_, bar) = Bar::from_bytes((&data[..], 0)).unwrap();

    assert_eq!(bar.b.len(), 1);
    assert_eq!(bar.b.first().unwrap(), &Foo{a: FooPart(0), b: Some(1)});

    let data = vec![0x01_u8, 0x01, 0x00];
    let (_, bar) = Bar::from_bytes((&data[..], 0)).unwrap();

    assert_eq!(bar.b.len(), 1);
    assert_eq!(bar.b.first().unwrap(), &Foo{a: FooPart(0), b: None});

}

But it's tricky and ugly too.

from deku.

and just one question: Seems like the existence of count is only for satisfying Vec? Maybe we could replace it with a Any for pass context? although it's a breaking change.
I mentioned BinRead has a args attribute, is it a solution?

from deku.

Ahh, ok. You'd like to conditionally read Foo.b depending on the value of Bar.flag, I see.

The use of Any could be interesting... it would be a breaking change, but that's OK, the library is still in development :)

from deku.

@sharksforarms In fact, i want to replace all of the (endian, bitsize, count, etc.) to a Vec<&Any>(because they all is context IMO)(maybe variadic generic can do also, but we dont have it yet), so that we can make this crate more powerful.
It allow us write codes like String::read(src, StrType::NullEnd) or StrType::Length(10)(we can do so in another crate called Scroll, but it implement by generic, makes writing derive macro diffcult).

from deku.

Anyway the feature of context is a bit heavy. Maybe i need to write some more detailed explanations or just make a PR for it.

from deku.

Very interesting! I'm in favor of the idea/feature.

from deku.

@constfold Before I get too far into the weeds, is this kinda what you were thinking? May need better benchmarks to measure the performance hit... every context will add some dynamic dispatching?

(pseudo-code-ish)

// lib.rs

trait AsAny {
    pub fn as_any(&self) -> &dyn Any {
        self
    }
}

pub struct DekuBitSize(pub usize);
pub struct DekuCount(pub usize);
pub struct DekuInputIsLe(pub bool);

impl AsAny for DekuBitSize {};
impl AsAny for DekuCount {};
impl AsAny for DekuInputIsLe {};

// ...
// impl DekuRead for $typ
fn read<'a>(
    input: &'a BitSlice<Msb0, u8>,
    context: Vec<&dyn Any>,
) -> Result<(&'a BitSlice<Msb0, u8>, Self), DekuError> {
   // ...
   
   let mut bit_size = None;
   let mut count = None;
   let mut input_is_le = None;
    for ctx in context {
        bit_size = ctx.downcast_ref::<DekuBitSize>().map(|v| v.0);
        count = ctx.downcast_ref::<DekuCount>().map(|v| v.0);
        input_is_le = ctx.downcast_ref::<DekuInputIsLe>().map(|v| v.0);
    }
   // ...
}


// deku-derive/deku_read.rs

let mut contexts = vec![];

if let Some(bit_size) = f.bits {
    contexts.push(quote!{DekuBitSize(#bit_size).as_any()})
}
quote! { 
  // ...
  #type::read(input, vec![$(#contexts),*])
}

Here's the branch I'm working off of, transferring the arguments input_is_le, bit_size and count to Vec<&Any>
https://github.com/sharksforarms/deku/compare/dynamic-context-test#diff-b4aea3e418ccdb71239b96952d9cddb6R268

from deku.

@sharksforarms Thanks for your work! I am too busy to give this idea deepper thinking those days, so it is just somewhat a draft.
And yea, we need to add a dynamic dispatching for every context(maybe add some attribute checking to eliminate it when we get fathur). But i dont think this will make a big performance effect.
In my mind, the context wokrs like a stack, the caller pushes its context and the callee consumes(or just look) the context. Then we give them a convention to make user write their own reader/writer easier.
In fact, i dont like this design even though this is the only implemention i can image, because it destory all static analysis.

from deku.

For example, A Vec only need to consume a count(or not consume if the top is not a Count type). then only primary types(or those fields the user added attribute manully) need to care Endian and BitSize. thus what we need is not a Vec but a &mut Vec(this will reduce the number of allocation).
And i dont want to use Vec<&Any> directly, but a newtype for it. It allows us to add more thing on it.

from deku.

And sorry if I look like a PM :P (At first, i would like to do it myself)

from deku.

Yeah I'm not fond of the Vec<&Any> either, but it is interesting as a concept..

As a simple solution like BinRead does, I think something like this would be possible:

#[derive(DekuRead)]
#[deku(import(flagA, flagB))]
struct Data {
    #[deku(reader = "Data::reader(rest, flagA, flagB, ...)")]
    data: u8
}

impl Data {
    fn reader(rest: ..., flagA: u8, flagB: u8, ...) {}
}

#[derive(DekuRead)]
struct Header {
    flagA: u8,
    flagB: u8,
    
    #[deku(export(flagA, flagB))]
    data: Data
}

from deku.

Is it helpful when Data hidden behind a Vec or something?

from deku.

Is it helpful when Data hidden behind a Vec or something?

Hm, maybe not as it's another layer of indirection...

impl DekuRead for Vec<T>

Back to the drawing board!

Edit: I see what you mean with the stack concept. I can see this being error-prone though..

from deku.

I see what you mean with the stack concept.
I can see this being error-prone though..

yeah, when you gonna use Any, everything is error-prone. but cause a lot of codes are generated by macro, it's under-control(uhh.. maybe)
By the way, i found the current Vec implemention will panic when count is not setted. i think is somehow the same thing.
When a struct need a context but not be given, the best way is give a compilation error. But what can we do? We are juat a derive macro.

from deku.

By the way, i found the current Vec implemention will panic when count is not setted.

Yes it will panic, Vec currently requires a count. This could be checked at compile time.

i think is somehow the same thing.

Yes it would be, count would be a "context"

from deku.

Condition: #65
Context: #61

from deku.