Git Product home page Git Product logo

serde-bencode's People

Contributors

5225225 avatar adamhammes avatar casey avatar chpio avatar josecelano avatar letfunny avatar madadam avatar nrempel avatar thequux avatar toby avatar topologicallyspeaking avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

serde-bencode's Issues

RawValue as a type for deserializing

What do you think of a RawValue type like in serde_json?

Use case

I want to use it to deserialize a Torrent struct without having to declare all the fields. I only care for the Infohash so I only want the name of the torrent bdecoded and the raw encoded Infohash. I think this can be an enhancement over declaring all the complex structs and then encoding again.

Problems

At the top of my head we have two problems. First of all is that bencoding lacks the structure necessary to just crop the string without context. If we encode a struct with 3 integers: a:1, b:2 and c:3; we get "d1:ai1e1:bi2e1:ci3ee". If I wanted to leave the "c" field raw I cannot just crop the string like "1:ci3ee" because that is not valid bencoding.

Quick solution

Off the top of my head, we could just create a new type that decoded the data to any format without caring what it is and then encodes it to a bytestring.

#[derive(Debug)]
pub struct RawValue(ByteBuf);

impl<'de> Deserialize<'de> for RawValue {
    fn deserialize<D>(deserializer: D) -> Result<RawValue, D::Error>
        where D: Deserializer<'de>
    {
        struct RawValueVisitor;
        impl<'de> Visitor<'de> for RawValueVisitor {
            type Value = RawValue;

            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                write!(formatter, "any valid bencoded value")
            }
            fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E>
            where
                E: serde::de::Error,
            {
                let bytes = to_bytes(&value);
                match bytes {
                    Ok(b) => Ok(RawValue(ByteBuf::from(b))),
                    Err(_) => panic!("Unknown error during serialization and deserialization of RawValue"),
                }
            }
        }

        let value = deserializer.deserialize_any(RawValueVisitor)?;
        Ok(value)
    }
}

I have only implemented the visitor for i64 and I would have to implement all the others in the same manner.

I also tried to have deserialize into a generic type without caring what it is (only that is Serializable):

let value: Box<dyn Serialize> = Box::new(Deserialize::deserialize(deserializer)?);

But it does not work and I cannot figure out how can I use generics to get it working because the trait cannot be made into an object.

I don't know enough about serde and rust to figure out a better solution so, what do you think?

OOM when parsing byte string with extremely long (incorrect) length

use serde_bencode::value::Value;
use serde_bencode::from_bytes;

fn main() {
    let data = b"123456789123:1";
    let _: Result<Value, _> = from_bytes(data);
}

reproduced with git version (553adb4) and latest crates.io (0.2.3).

The issue is the preallocation in

serde-bencode/src/de.rs

Lines 211 to 222 in 553adb4

fn parse_bytes(&mut self, len_char: u8) -> Result<Vec<u8>> {
let len = self.parse_bytes_len(len_char)?;
let mut buf = vec![0u8; len];
let actual_len = self
.reader
.read(buf.as_mut_slice())
.map_err(Error::IoError)?;
if len != actual_len {
return Err(Error::EndOfStream);
}
Ok(buf)
}

specifically line 213

Probably best to just remove that preallocation. I'll make a PR to do that in a bit.

Deserialization for enums with `#[serde(flatten)]` is not working

I think these two failing tests could be the same problem because they both use #[serde(flatten)] for an enum field.

  • cargo test ser_de_flattened_enum -- --nocapture
  • cargo test ser_de_flattened_adjacently_tagged_enum -- --nocapture

Failing test 1

$ cargo test ser_de_flattened_enum -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests src/lib.rs (target/debug/deps/torrust_serde_bencode-61b7eea2ea5618bf)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests.rs (target/debug/deps/tests-b55451bec73c8180)

running 1 test
bytes: "d12:message_type8:Responsee"
bytes: "d12:message_type8:Responsee"
thread 'ser_de_flattened_enum' panicked at tests/tests.rs:34:41:
called `Result::unwrap()` on an `Err` value: InvalidType("Invalid Type: byte array (expected: `string or map`)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test ser_de_flattened_enum ... FAILED

failures:

failures:
    ser_de_flattened_enum

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test tests`

Failing test 2

$ cargo test ser_de_flattened_adjacently_tagged_enum -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests src/lib.rs (target/debug/deps/torrust_serde_bencode-61b7eea2ea5618bf)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/tests.rs (target/debug/deps/tests-b55451bec73c8180)

running 1 test
bytes: "d7:contentd5:tokeni456ee2:idi123e4:type7:Requeste"
thread 'ser_de_flattened_adjacently_tagged_enum' panicked at tests/tests.rs:34:41:
called `Result::unwrap()` on an `Err` value: InvalidType("Invalid Type: byte array (expected: `string or map`)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test ser_de_flattened_adjacently_tagged_enum ... FAILED

failures:

failures:
    ser_de_flattened_adjacently_tagged_enum

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out; finished in 0.00s

error: test failed, to rerun pass `--test tests`

Related commits

How to unwrap pieces?

Sorry, I am still a new Rust user. I realize it's a wrapper, but I can't seem to convert pieces to an Vec<String>!!!

pieces: b"J\u{a0}\u{c3}\u{fb}\u{ce}q&\u{8c}N\u{fb}V\u{fe}L\u{b1}\u{f4}\"&\u{bc}YY&\u{c5}\u{f5}\u{90}\u{f8}\u{8d}\u{c5}`\u{90}]-,\u{1d}J\u{82}\u{db}9O\u{ae}\u{98}\u{c4}Sa\u{a1}\u{85}\u{8c}7\u{cf}\u{df}w\u{bb}qlH\u{fa}6\u{8f}6\u{5}\u{af}MB\u{89}\u{c7}i\u{94}\u{ee}\u{95}\u{b0}0+v\u{ca}\r\u{f2}\u{a3}Q\u{a1}\n\u{fc}\u{84}\u{ea}\u{c8}-?8>l\u{1b}\u{b9}\u{d5}\u{a0}\u{c1}\u{8b}\\\u{db}\u{c1}\u{b7})\u{af}\u{db}&_\u{87}\u{a7}\u{f6}\u{4}y\u{16}\u{c3}\u{2}\u{98}G\u{9c}\u{ae}\u{3}\u{c9}\u{dc}\u{ec}\u{cb}\u{fa}(W\u{f4}\u{fb}\u{eb}M>\u{9d}}\u{84}~K\u{94}\u{c6}\u{b4}\u{18}\u{f4}\u{fa}\u{83}"

This is my current self.info.pieces

Tuple structs cannot be deserialized

It seems you can not deserialize a tuple struct like this struct Node(String, i64) from a nested list in encoded format. There are already two tests for this behavior.

   #[test]
    fn deserialization() {
        // todo: you cannot deserialize to the same struct used in serialization.
        // It does not work with a tuple struct `struct Node(String, i64)`
        // instead of a tuple `(String, i64)`.

        #[allow(dead_code)]
        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Torrent {
            info: Info,
            #[serde(default)]
            nodes: Option<Vec<(String, i64)>>,
        }

        #[allow(dead_code)]
        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Info {
            #[serde(default)]
            pub length: Option<i64>,

            #[serde(default)]
            pub name: String,

            #[serde(rename = "piece length")]
            pub piece_length: i64,

            #[serde(default)]
            pub pieces: ByteBuf,
        }

        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Node(String, i64);

        // cspell:disable-next-line
        let b = "d4:infod6:lengthi8e4:name11:minimal.txt12:piece lengthi1e6:pieces1:pe5:nodesll15:188.163.121.224i56711eel14:162.250.131.26i13386eeee";

        let r: Torrent = from_str(b).unwrap();

        assert_eq!(
            r,
            Torrent {
                info: Info {
                    name: "minimal.txt".to_string(),
                    pieces: ByteBuf::from(vec![b'p']),
                    piece_length: 1,
                    length: Some(8),
                },
                nodes: Some(vec![
                    ("188.163.121.224".to_string(), 56711),
                    ("162.250.131.26".to_string(), 13386),
                ]),
            }
        );
    }

Although you can serialize them:

    #[test]
    fn serialization() {
        #[allow(dead_code)]
        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Torrent {
            info: Info,

            #[serde(default)]
            nodes: Option<Vec<Node>>,
        }

        #[allow(dead_code)]
        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Info {
            #[serde(default)]
            pub length: Option<i64>,

            #[serde(default)]
            pub name: String,

            #[serde(rename = "piece length")]
            pub piece_length: i64,

            #[serde(default)]
            pub pieces: ByteBuf,
        }

        #[derive(PartialEq, Debug, Serialize, Deserialize)]
        struct Node(String, i64);

        let torrent = Torrent {
            info: Info {
                name: "minimal.txt".to_string(),
                pieces: ByteBuf::from(vec![b'p']),
                piece_length: 1,
                length: Some(8),
            },
            nodes: Some(vec![
                Node("188.163.121.224".to_string(), 56711),
                Node("162.250.131.26".to_string(), 13386),
            ]),
        };

        // cspell:disable-next-line
        assert_eq!(to_string(&torrent).unwrap(), "d4:infod6:lengthi8e4:name11:minimal.txt12:piece lengthi1e6:pieces1:pe5:nodesll15:188.163.121.224i56711eel14:162.250.131.26i13386eeee");
    }

Both options:

  • nodes: Option<Vec<Node>>, where Node is struct Node(String, i64)
  • nodes: Option<Vec<(String, i64)>>,

produce the same bencoded, but you can only deserialize the second one. The first one gives you only one item in the parent list (one node).

I am still determining if that's a bug. As bencoded format is ordered, there should be no problem assigning the values to the tuple struct fields (0 and 1). The first value should be assigned to the first element in the tuple.

Anyway, ti seems the problem is not even in this package but when the series values are mapped to the final struct (not sure).

Encoding Vec<u8> as bytes

I would like to use this library to handle BT DHT messages. I represent some fields (like TransactionId, NodeId) as Vec but I see that it encoded as arrays of integers, not as byte strings as expected. I don't sure that it is a wrong behavior, but I need a way to fix it.

Improve testing coverage

Current coverage report:

Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
de.rs                             190                60    68.42%          36                 9    75.00%         237                43    81.86%           0                 0         -
error.rs                           31                31     0.00%          12                12     0.00%          51                51     0.00%           0                 0         -
ser.rs                            118                43    63.56%          57                13    77.19%         277                59    78.70%           0                 0         -
ser/string.rs                      30                28     6.67%          30                28     6.67%         125               119     4.80%           0                 0         -
value.rs                          174                13    92.53%         112                 5    95.54%         502                19    96.22%           0                 0         -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                             543               175    67.77%         247                67    72.87%        1192               291    75.59%           0                 0         -

Failure to deserialize into adjacently tagged enum

Minimal reproducible test-case:

#[test]
fn ser_de_adjacently_tagged_enum() {
    #[derive(Serialize, Deserialize, Debug, Eq, PartialEq)]
    #[serde(tag = "t", content = "c")]
    enum Mock {
        A,
        B,
    }

    test_ser_de_eq(Mock::A);
    test_ser_de_eq(Mock::B);
}

Fixed in #23

Invalid type. Expected `Str`

Hi,

I just started using serde and serde_bencode, and I get this result from serde_bencode: Syntax(Custom("Invalid type. Expected Str"), 0)

My code:

serde_types.in.rs:

#[derive(Debug)]
#[derive(Eq)]
#[derive(PartialEq)]
#[derive(Serialize, Deserialize)]
pub enum QueryType {
    #[serde(rename="gp")]
    GetPeers,
    #[serde(rename="fn")]
    FindNode,
}

#[derive(Debug)]
#[derive(Eq)]
#[derive(PartialEq)]
#[derive(Serialize, Deserialize)]
pub struct RoutePacket {
    #[serde(rename="q")]
    query_type: Option<QueryType>,
    #[serde(rename="txid")]
    transaction_id: Vec<u8>,
    #[serde(rename="tar")]
    target_address: Option<String>,
    #[serde(rename="n")]
    nodes: Option<String>,
}

route_packet.rs:

extern crate serde;
extern crate serde_bencode;

include!(concat!(env!("OUT_DIR"), "/serde_types.rs"));

impl RoutePacket {
    pub fn new_from_slice(v: &[u8]) -> Result<RoutePacket, serde_bencode::error::Error> {
        serde_bencode::from_slice(v)
    }

    pub fn to_vec(&self) -> Result<Vec<u8>, serde_bencode::error::Error> {
        serde_bencode::to_vec(self)
    }
}

#[cfg(test)]
mod tests {
    use bencode;
    use super::*;
    use rustc_serialize::Decodable;

    #[test]
    fn test_fn() {
        let s = "d1:q2:fn3:tar16:abcdefghhijklmno4:txid5:12345e".as_bytes();
        let m = RoutePacket {
            query_type: Some(QueryType::FindNode),
            target_address: Some("abcdefghhijklmno".to_owned()),
            nodes: None,
            transaction_id: b"12345".to_vec()
        };

        let s_decoded = RoutePacket::new_from_slice(s).unwrap();
        let m_encoded = m.to_vec();

        assert_eq!(s_decoded, m);
        assert_eq!(m_encoded.unwrap(), s);
    }

    #[test]
    fn test_n() {
        let s = "d1:n80:cdefghijklmnopqrstuvwxyzabcdefghi1234567qponmlkjihgzyxwvutsrstuvwxyzabcde23456784:txid5:12345e".as_bytes();
        let m = RoutePacket {
            query_type: None,
            target_address: None,
            nodes: Some("cdefghijklmnopqrstuvwxyzabcdefghi1234567qponmlkjihgzyxwvutsrstuvwxyzabcde2345678".to_owned()),
            transaction_id: b"12345".to_vec(),
        };

        let s_decoded = RoutePacket::new_from_slice(s).unwrap();
        let m_encoded = m.to_vec();

        assert_eq!(s_decoded, m);
        assert_eq!(m_encoded.unwrap(), s);
    }
}

Am I doing anything wrong?

not support complex struct?

My cargo.toml

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.10.0"
serde_bencode = { path = "serde-bencode" }

Version of rustc

rustc 1.35.0 (3c235d560 2019-05-20)

And my code

use serde::{Deserialize, Serialize};

use serde_bencode::de;
use serde_bencode::ser;

use std::str;

#[derive(Debug, Deserialize, Serialize)]
struct Node(String, i64);

fn main() {

    let nodes: Vec<Node> = vec![
        Node("aaa".to_owned(), 5665),
        Node("bbb".to_owned(), 5665),
        Node("ccc".to_owned(), 5665),
    ];

    let buffer: Vec<u8> = match ser::to_bytes(&nodes) {
        Ok(s) => s,
        Err(e) => panic!(e),
    };

    println!("{}", str::from_utf8(&buffer).unwrap());


    match de::from_bytes::<Vec<Node>>(&buffer) {
        Ok(t) => {
            // println!("{}", t.len());
            println!("{:?}", t);
        }
        Err(e) => println!("ERROR: {:?}", e),
    };
}

And just got

ll3:aaai5665eel3:bbbi5665eel3:ccci5665eee
[Node("aaa", 5665)]

By the way, I am just a rust noob. Maybe I just made some mistake?

Write better documentation

We should be using better (some!) source code annotations to generate documentation for rustdoc/docs.rs.

Update to work with Serde 0.9.X

There are a number of breaking changes when you move from Serde 0.8 -> 0.9. The Serializer trait has changed the most but there are some changes to enum deserialization as well. serde-bencode needs to be reworked to be compatible.

Error round-tripping complex struct

I ran into a problem deserializing a complex struct:

fn ser_de_field_vec_tuple() {
    #[derive(Deserialize, Serialize, Eq, PartialEq, Debug)]
    struct Foo {
        bar: Vec<(u16,)>,
    }

    let foo = Foo {
        bar: vec![(1,), (3,)],
    };

    test_ser_de_eq(foo);
}

The error message is InvalidType("Invalid Type: sequence (expected: field identifier)")'

I wasn't able reduce the test case by removing the struct, the vec, or the tuple, so it seems to be some weird combination of the three.

Can't serialize struct with Encoder

I am trying to re-serialize the Info struct provided in the example code, but I am getting a stracktrace error. Basically I added a #[derive(Serialize)] to the Info struct, and then I am using the following snippet in main() to attempt to serialize...

let mut encoder = Encoder::new();
torrent.info.serialize(&mut encoder);
let info_ser: Vec<u8> = encoder.into();
let info_str: String = String::from_utf8(info_ser).unwrap();

the error being thrown is:
"thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', .../serde_bencode-0.1.1/src/encoder.rs:64"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.