toby / serde-bencode Goto Github PK
View Code? Open in Web Editor NEWSerde backed Bencode encoding/decoding library for Rust.
License: MIT License
Serde backed Bencode encoding/decoding library for Rust.
License: MIT License
I think these two failing tests could be the same problem because they both use #[serde(flatten)]
for an enum
field.
cargo test ser_de_flattened_enum -- --nocapture
cargo test ser_de_flattened_adjacently_tagged_enum -- --nocapture
$ cargo test ser_de_flattened_enum -- --nocapture
Finished test [unoptimized + debuginfo] target(s) in 0.00s
Running unittests src/lib.rs (target/debug/deps/torrust_serde_bencode-61b7eea2ea5618bf)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running tests/tests.rs (target/debug/deps/tests-b55451bec73c8180)
running 1 test
bytes: "d12:message_type8:Responsee"
bytes: "d12:message_type8:Responsee"
thread 'ser_de_flattened_enum' panicked at tests/tests.rs:34:41:
called `Result::unwrap()` on an `Err` value: InvalidType("Invalid Type: byte array (expected: `string or map`)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test ser_de_flattened_enum ... FAILED
failures:
failures:
ser_de_flattened_enum
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out; finished in 0.00s
error: test failed, to rerun pass `--test tests`
$ cargo test ser_de_flattened_adjacently_tagged_enum -- --nocapture
Finished test [unoptimized + debuginfo] target(s) in 0.00s
Running unittests src/lib.rs (target/debug/deps/torrust_serde_bencode-61b7eea2ea5618bf)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running tests/tests.rs (target/debug/deps/tests-b55451bec73c8180)
running 1 test
bytes: "d7:contentd5:tokeni456ee2:idi123e4:type7:Requeste"
thread 'ser_de_flattened_adjacently_tagged_enum' panicked at tests/tests.rs:34:41:
called `Result::unwrap()` on an `Err` value: InvalidType("Invalid Type: byte array (expected: `string or map`)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test ser_de_flattened_adjacently_tagged_enum ... FAILED
failures:
failures:
ser_de_flattened_adjacently_tagged_enum
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out; finished in 0.00s
error: test failed, to rerun pass `--test tests`
My cargo.toml
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.10.0"
serde_bencode = { path = "serde-bencode" }
Version of rustc
rustc 1.35.0 (3c235d560 2019-05-20)
And my code
use serde::{Deserialize, Serialize};
use serde_bencode::de;
use serde_bencode::ser;
use std::str;
#[derive(Debug, Deserialize, Serialize)]
struct Node(String, i64);
fn main() {
let nodes: Vec<Node> = vec![
Node("aaa".to_owned(), 5665),
Node("bbb".to_owned(), 5665),
Node("ccc".to_owned(), 5665),
];
let buffer: Vec<u8> = match ser::to_bytes(&nodes) {
Ok(s) => s,
Err(e) => panic!(e),
};
println!("{}", str::from_utf8(&buffer).unwrap());
match de::from_bytes::<Vec<Node>>(&buffer) {
Ok(t) => {
// println!("{}", t.len());
println!("{:?}", t);
}
Err(e) => println!("ERROR: {:?}", e),
};
}
And just got
ll3:aaai5665eel3:bbbi5665eel3:ccci5665eee
[Node("aaa", 5665)]
By the way, I am just a rust noob. Maybe I just made some mistake?
Currently I can deserialize HashMap with key as an enum as example bellow:
#[derive(PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
enum Extension {
#[serde(rename = "ut_metadata")]
Metadata,
}
serde_bencode::from_str::<std::collections::HashMap<Extension, u8>>("d11:ut_metadatai1ee").unwrap();
But I cannot serialize a HashMap to a string:
serde_bencode::to_string(&std::collections::HashMap::from([(Extension::Metadata, 1)])).unwrap();
If I modifiy the serialize_newtype_struct() to allow serialize unit variant as a string the above code work. I don't know if it will break anything else so please consider.
Since we can use serde 1.0 now, can you explain how to use Vec rather then bytebuf now? I am still getting errors -_-
We should be using better (some!) source code annotations to generate documentation for rustdoc/docs.rs.
I ran into a problem deserializing a complex struct:
fn ser_de_field_vec_tuple() {
#[derive(Deserialize, Serialize, Eq, PartialEq, Debug)]
struct Foo {
bar: Vec<(u16,)>,
}
let foo = Foo {
bar: vec![(1,), (3,)],
};
test_ser_de_eq(foo);
}
The error message is InvalidType("Invalid Type: sequence (expected:
field identifier)")'
I wasn't able reduce the test case by removing the struct, the vec, or the tuple, so it seems to be some weird combination of the three.
From: rust-lang/rust#49934
Looks like Rust 1.26 breaks our build.
I am trying to re-serialize the Info
struct provided in the example code, but I am getting a stracktrace error. Basically I added a #[derive(Serialize)] to the Info
struct, and then I am using the following snippet in main() to attempt to serialize...
let mut encoder = Encoder::new();
torrent.info.serialize(&mut encoder);
let info_ser: Vec<u8> = encoder.into();
let info_str: String = String::from_utf8(info_ser).unwrap();
the error being thrown is:
"thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', .../serde_bencode-0.1.1/src/encoder.rs:64"
I'm trying to move away from bencode to serde-bencode due to the rustc-serialize deprecation. (kpcyrd/cjdns-rs#2)
Could you please consider upgrading to 1.0 :) ?
I would like to use this library to handle BT DHT messages. I represent some fields (like TransactionId, NodeId) as Vec but I see that it encoded as arrays of integers, not as byte strings as expected. I don't sure that it is a wrong behavior, but I need a way to fix it.
use serde_bencode::value::Value;
use serde_bencode::from_bytes;
fn main() {
let data = b"123456789123:1";
let _: Result<Value, _> = from_bytes(data);
}
reproduced with git version (553adb4) and latest crates.io (0.2.3).
The issue is the preallocation in
Lines 211 to 222 in 553adb4
specifically line 213
Probably best to just remove that preallocation. I'll make a PR to do that in a bit.
It seems you can not deserialize a tuple struct like this struct Node(String, i64)
from a nested list in encoded format. There are already two tests for this behavior.
#[test]
fn deserialization() {
// todo: you cannot deserialize to the same struct used in serialization.
// It does not work with a tuple struct `struct Node(String, i64)`
// instead of a tuple `(String, i64)`.
#[allow(dead_code)]
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Torrent {
info: Info,
#[serde(default)]
nodes: Option<Vec<(String, i64)>>,
}
#[allow(dead_code)]
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Info {
#[serde(default)]
pub length: Option<i64>,
#[serde(default)]
pub name: String,
#[serde(rename = "piece length")]
pub piece_length: i64,
#[serde(default)]
pub pieces: ByteBuf,
}
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Node(String, i64);
// cspell:disable-next-line
let b = "d4:infod6:lengthi8e4:name11:minimal.txt12:piece lengthi1e6:pieces1:pe5:nodesll15:188.163.121.224i56711eel14:162.250.131.26i13386eeee";
let r: Torrent = from_str(b).unwrap();
assert_eq!(
r,
Torrent {
info: Info {
name: "minimal.txt".to_string(),
pieces: ByteBuf::from(vec![b'p']),
piece_length: 1,
length: Some(8),
},
nodes: Some(vec![
("188.163.121.224".to_string(), 56711),
("162.250.131.26".to_string(), 13386),
]),
}
);
}
Although you can serialize them:
#[test]
fn serialization() {
#[allow(dead_code)]
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Torrent {
info: Info,
#[serde(default)]
nodes: Option<Vec<Node>>,
}
#[allow(dead_code)]
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Info {
#[serde(default)]
pub length: Option<i64>,
#[serde(default)]
pub name: String,
#[serde(rename = "piece length")]
pub piece_length: i64,
#[serde(default)]
pub pieces: ByteBuf,
}
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Node(String, i64);
let torrent = Torrent {
info: Info {
name: "minimal.txt".to_string(),
pieces: ByteBuf::from(vec![b'p']),
piece_length: 1,
length: Some(8),
},
nodes: Some(vec![
Node("188.163.121.224".to_string(), 56711),
Node("162.250.131.26".to_string(), 13386),
]),
};
// cspell:disable-next-line
assert_eq!(to_string(&torrent).unwrap(), "d4:infod6:lengthi8e4:name11:minimal.txt12:piece lengthi1e6:pieces1:pe5:nodesll15:188.163.121.224i56711eel14:162.250.131.26i13386eeee");
}
Both options:
nodes: Option<Vec<Node>>,
where Node
is struct Node(String, i64)
nodes: Option<Vec<(String, i64)>>,
produce the same bencoded, but you can only deserialize the second one. The first one gives you only one item in the parent list (one node).
I am still determining if that's a bug. As bencoded format is ordered, there should be no problem assigning the values to the tuple struct fields (0 and 1). The first value should be assigned to the first element in the tuple.
Anyway, ti seems the problem is not even in this package but when the series values are mapped to the final struct (not sure).
I'm going to publish a new release so users can apply this patch 0c6c144
It seems nested lists do not work correctly.
I've created a PR to reproduce the problem.
I was trying to write the PR myself but the code is quite confusing for me sorry.
The idea is that instead of a generic error:
Invalid Type: byte array (expected:
a sequence
)
A serializing library should give you the position, in this case we should know what field we are trying to parse:
Invalid Type for field "pieces": byte array (expected:
a sequence
)
What do you think of a RawValue type like in serde_json?
I want to use it to deserialize a Torrent struct without having to declare all the fields. I only care for the Infohash so I only want the name of the torrent bdecoded and the raw encoded Infohash. I think this can be an enhancement over declaring all the complex structs and then encoding again.
At the top of my head we have two problems. First of all is that bencoding lacks the structure necessary to just crop the string without context. If we encode a struct with 3 integers: a:1, b:2 and c:3; we get "d1:ai1e1:bi2e1:ci3ee". If I wanted to leave the "c" field raw I cannot just crop the string like "1:ci3ee" because that is not valid bencoding.
Off the top of my head, we could just create a new type that decoded the data to any format without caring what it is and then encodes it to a bytestring.
#[derive(Debug)]
pub struct RawValue(ByteBuf);
impl<'de> Deserialize<'de> for RawValue {
fn deserialize<D>(deserializer: D) -> Result<RawValue, D::Error>
where D: Deserializer<'de>
{
struct RawValueVisitor;
impl<'de> Visitor<'de> for RawValueVisitor {
type Value = RawValue;
fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
write!(formatter, "any valid bencoded value")
}
fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E>
where
E: serde::de::Error,
{
let bytes = to_bytes(&value);
match bytes {
Ok(b) => Ok(RawValue(ByteBuf::from(b))),
Err(_) => panic!("Unknown error during serialization and deserialization of RawValue"),
}
}
}
let value = deserializer.deserialize_any(RawValueVisitor)?;
Ok(value)
}
}
I have only implemented the visitor for i64 and I would have to implement all the others in the same manner.
I also tried to have deserialize into a generic type without caring what it is (only that is Serializable):
let value: Box<dyn Serialize> = Box::new(Deserialize::deserialize(deserializer)?);
But it does not work and I cannot figure out how can I use generics to get it working because the trait cannot be made into an object.
I don't know enough about serde and rust to figure out a better solution so, what do you think?
Minimal reproducible test-case:
#[test]
fn ser_de_adjacently_tagged_enum() {
#[derive(Serialize, Deserialize, Debug, Eq, PartialEq)]
#[serde(tag = "t", content = "c")]
enum Mock {
A,
B,
}
test_ser_de_eq(Mock::A);
test_ser_de_eq(Mock::B);
}
Fixed in #23
Sorry, I am still a new Rust user. I realize it's a wrapper, but I can't seem to convert pieces to an Vec<String>!!!
pieces: b"J\u{a0}\u{c3}\u{fb}\u{ce}q&\u{8c}N\u{fb}V\u{fe}L\u{b1}\u{f4}\"&\u{bc}YY&\u{c5}\u{f5}\u{90}\u{f8}\u{8d}\u{c5}`\u{90}]-,\u{1d}J\u{82}\u{db}9O\u{ae}\u{98}\u{c4}Sa\u{a1}\u{85}\u{8c}7\u{cf}\u{df}w\u{bb}qlH\u{fa}6\u{8f}6\u{5}\u{af}MB\u{89}\u{c7}i\u{94}\u{ee}\u{95}\u{b0}0+v\u{ca}\r\u{f2}\u{a3}Q\u{a1}\n\u{fc}\u{84}\u{ea}\u{c8}-?8>l\u{1b}\u{b9}\u{d5}\u{a0}\u{c1}\u{8b}\\\u{db}\u{c1}\u{b7})\u{af}\u{db}&_\u{87}\u{a7}\u{f6}\u{4}y\u{16}\u{c3}\u{2}\u{98}G\u{9c}\u{ae}\u{3}\u{c9}\u{dc}\u{ec}\u{cb}\u{fa}(W\u{f4}\u{fb}\u{eb}M>\u{9d}}\u{84}~K\u{94}\u{c6}\u{b4}\u{18}\u{f4}\u{fa}\u{83}"
This is my current self.info.pieces
Hi,
I just started using serde and serde_bencode, and I get this result from serde_bencode: Syntax(Custom("Invalid type. Expected
Str"), 0)
My code:
serde_types.in.rs:
#[derive(Debug)]
#[derive(Eq)]
#[derive(PartialEq)]
#[derive(Serialize, Deserialize)]
pub enum QueryType {
#[serde(rename="gp")]
GetPeers,
#[serde(rename="fn")]
FindNode,
}
#[derive(Debug)]
#[derive(Eq)]
#[derive(PartialEq)]
#[derive(Serialize, Deserialize)]
pub struct RoutePacket {
#[serde(rename="q")]
query_type: Option<QueryType>,
#[serde(rename="txid")]
transaction_id: Vec<u8>,
#[serde(rename="tar")]
target_address: Option<String>,
#[serde(rename="n")]
nodes: Option<String>,
}
route_packet.rs:
extern crate serde;
extern crate serde_bencode;
include!(concat!(env!("OUT_DIR"), "/serde_types.rs"));
impl RoutePacket {
pub fn new_from_slice(v: &[u8]) -> Result<RoutePacket, serde_bencode::error::Error> {
serde_bencode::from_slice(v)
}
pub fn to_vec(&self) -> Result<Vec<u8>, serde_bencode::error::Error> {
serde_bencode::to_vec(self)
}
}
#[cfg(test)]
mod tests {
use bencode;
use super::*;
use rustc_serialize::Decodable;
#[test]
fn test_fn() {
let s = "d1:q2:fn3:tar16:abcdefghhijklmno4:txid5:12345e".as_bytes();
let m = RoutePacket {
query_type: Some(QueryType::FindNode),
target_address: Some("abcdefghhijklmno".to_owned()),
nodes: None,
transaction_id: b"12345".to_vec()
};
let s_decoded = RoutePacket::new_from_slice(s).unwrap();
let m_encoded = m.to_vec();
assert_eq!(s_decoded, m);
assert_eq!(m_encoded.unwrap(), s);
}
#[test]
fn test_n() {
let s = "d1:n80:cdefghijklmnopqrstuvwxyzabcdefghi1234567qponmlkjihgzyxwvutsrstuvwxyzabcde23456784:txid5:12345e".as_bytes();
let m = RoutePacket {
query_type: None,
target_address: None,
nodes: Some("cdefghijklmnopqrstuvwxyzabcdefghi1234567qponmlkjihgzyxwvutsrstuvwxyzabcde2345678".to_owned()),
transaction_id: b"12345".to_vec(),
};
let s_decoded = RoutePacket::new_from_slice(s).unwrap();
let m_encoded = m.to_vec();
assert_eq!(s_decoded, m);
assert_eq!(m_encoded.unwrap(), s);
}
}
Am I doing anything wrong?
Currently points to https://github.com/toby/serde-bencode/blob/master/src/bencode_enum.rs
There are a number of breaking changes when you move from Serde 0.8 -> 0.9. The Serializer
trait has changed the most but there are some changes to enum deserialization as well. serde-bencode
needs to be reworked to be compatible.
Current coverage report:
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
de.rs 190 60 68.42% 36 9 75.00% 237 43 81.86% 0 0 -
error.rs 31 31 0.00% 12 12 0.00% 51 51 0.00% 0 0 -
ser.rs 118 43 63.56% 57 13 77.19% 277 59 78.70% 0 0 -
ser/string.rs 30 28 6.67% 30 28 6.67% 125 119 4.80% 0 0 -
value.rs 174 13 92.53% 112 5 95.54% 502 19 96.22% 0 0 -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 543 175 67.77% 247 67 72.87% 1192 291 75.59% 0 0 -
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.