p3ki / bendy Goto Github PK
View Code? Open in Web Editor NEWA rust library for encoding and decoding bencode with enforced cannonicalization rules.
License: BSD 3-Clause "New" or "Revised" License
A rust library for encoding and decoding bencode with enforced cannonicalization rules.
License: BSD 3-Clause "New" or "Revised" License
Currently I have the following code in which I have two possible enumerations for an object, first I have to parse the bytes to figure out what object to initialize for the data and after that I have to parse the fields. I need the ability to clone either Object
or DictDecoder
for this:
impl FromBencode for Info {
const EXPECTED_RECURSION_DEPTH: usize = 999;
fn decode_bencode_object(
object: bendy::decoding::Object,
) -> Result<Self, bendy::decoding::Error>
where
Self: Sized,
{
let dict = object.try_into_dictionary()?;
let mut info = Info::default();
// figure out what mode the info type is but gets consumed so borrow checker complains after this point
info.mode = info.try_mode(dict).unwrap();
match info.mode {
// ...
}
}
Hello,
I discovered bendy
a few days ago, and noticed that it uses failure
for error handling. Unfortunately, since commit
135e2a3, failure is deprecated. The notice says that users may like to use either anyhow
or thiserror
.
Is there any plan in switching to another error handling library?
What? Add a flag to enable/disable the parsing of non-canonical input.
How? Make the decoder parameterized on the state tracker.
I'm trying to parse torrent files, I've tried a dozen and with any of them i get the NestingTooDeep error, here's my code:
use bendy::decoding::FromBencode;
use bendy::encoding::AsString;
use std::fs::File;
use std::io::prelude::Read;
fn main() {
let mut fc = Vec::new(); // file content
let mut file = File::open("test.torrent").expect("Unable to read file.");
file.read_to_end(&mut fc).expect("Unable to read");
let example = AsString::from_bencode(&fc).expect("Unable to bendecode");
println!("{:#?}", example.0);
}
Would it be possible to push a new release of bendy on crates.io?
I'm preparing to release a crate that depends on serde support. Currently I'm depending on bendy via github, but released crates may only have crates.io dependencies.
The name of an unexpected field is generally available to client code as a &[u8]
; it would be much more convenient to be able to report that directly than to have to convert it to a string (for example).
This would likely be a breaking change, thus necessitating a v0.3.0
Should we provide more convenience functions to unpack directly into common container types -- if possible --?
Potential candidates:
Bytes
to Vec<u8>
, String
or str
Integer
to i8
, i16
, ...List
to Vec
or [_: ExpectedLength]
Dict
to Map
, BTreeMap
Hi folks,
I use Bendy in my project for decoding a bencode and it worked very well for me so far. But data on the Internet is not well-formed in all cases.
So, I have some problems parsing bencode data which I'm getting from a server because this data is not well-formed (keys are not sorted). I can't tell the server to give me the response I want, because the server doesn't belong to me.
My question: Is there any way to relax the enforced correctness rules to be able to parse the response?
Thanks,
Eugene
The port of additional test cases (#20) to fix issue #7 revealed a new question. The original test suite contained some cases marked as "illegal" which contain more tokens than actually required to restore the expected data type. Example: Decoding i12345e5:hello
into an integer.
How should FromBencode
handle the case? Potential solutions I can think of:
from_bencode
with stricter handlingTrying to use serde rename in a filed causes an UnexpectedToken error
#[derive(Debug, Serialize, Deserialize)]
pub struct MetaInfo {
#[serde(rename = "creation date")]
creation_date: Option<u64>,
}
let t = fs::read("temp/manjaro.torrent").unwrap();
let torrent: MetaInfo = bendy::serde::from_bytes(&t).unwrap();
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Decode(Error { context: None, error: UnexpectedToken("List", "Num") })', src/main.rs:70:58
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
This makes it impossible to deserialize certain fields
I just noticed, that your MapAccess
and SeqAccess
implementors does not implement size_hint
, but it seems that should be pretty easy to implement.
I have the following FromBencode
implementation which very surprisingly fails with an error that should be only relevant to encoding.
impl FromBencode for ScrapeResponse {
const EXPECTED_RECURSION_DEPTH: usize = 5;
fn decode_bencode_object(
object: bendy::decoding::Object,
) -> Result<Self, bendy::decoding::Error>
where
Self: Sized,
{
// &bytes = b"d5:filesd20:\x1b\x84\xcb\xd2TZf\x8a\xd85\x04!\xd4\x1b\x88\x0f?d\xcc\xf4d8:completei17e10:incompletei2e10:downloadedi1539eeee"
let mut dict = object.try_into_dictionary()?;
let mut result = Vec::new();
while let Some(files) = dict.next_pair()? {
let mut files = files.1.try_into_dictionary()?;
while let Some(file) = files.next_pair()? {
let mut decoder = file.1.try_into_dictionary()?;
let mut status: Status = Default::default();
while let Some(pair) = decoder.next_pair()? {
match pair {
(b"complete", _) => {
status.seeders = u32::decode_bencode_object(pair.1)?;
}
(b"incomplete", _) => {
status.leechers = u32::decode_bencode_object(pair.1)?;
}
(b"downloaded", _) => {
let err = u32::decode_bencode_object(pair.1);
dbg!(err.unwrap());
}
_ => {}
}
}
result.push((file.0.to_vec(), status));
}
}
Ok(Self { files: result })
}
}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { context: None, error: StructureError(UnsortedKeys) }
Per the documentation in encoding.rs
, MAX_DEPTH of a variable-sized object should be set to 0 and then explicitly padded when creating the decoder.
This will fix #37 properly, but is a breaking change.
Based on the latest update of failure
the crate exposes failure_derive
through a feature flag. This should allow us to remove failure_derive
as direct dependency.
Input streams with large blocks of binary data should be decodable without copying their contents.
This is probably best done by making FromBencode take 'ser
(the lifetime of the data being deserialized) as a parameter, thus we could have impls for &'ser [u8]
, Cow<'ser, [u8]>
, and (possibly) &'ser str
and Cow<'ser, str>
I've been using serde-bencode, but I ran into some issues, so I'm reconsidering my choice of library.
Bendy looks great, but I'm making heavy use of serde derived serialization and deserialization, which I think Bendy doesn't do.
Is this a feature which bendy might eventually support?
BEP 9 (the protocol for resolving magnet URIs) defines a data packet that consists of a bencoded dictionary followed by arbitrary data (actually a slice of a torrent's info dict), and I would like to parse such a packet using bendy. The best way I've figured out to do this so far is to create a Decoder
, call next_object()
, retrieve the DictDecoder
, iterate through & massage the dict's fields, call DictDecoder::into_raw()
after handling all fields, get the length of the raw dict, and use that length as an offset to slice the original payload. This is rather unwieldy, and I believe this could be simplified by just giving Decoder
a remaining(self) -> &'ser [u8]
method for retrieving all remaining bytes of the payload after doing next_object()
+ FromBencode::decode_bencode_object()
.
We should port the test cases from https://github.com/jamesleonis/bencode-cljc/blob/75c973b55e4905c378d5335806774ec22d30affc/test/bencode_cljc/core_test.cljc
If cfg(doctest) stabilized (or cfg(test)
got fixed), we can use doc-comment for including code in README
in doc-test, then we can remove tests/readme.rs
.
The AsString
wrapper should derive all of the basic traits (Default
, Eq
, Hash
, ...) to prevent any kind of artificial restriction. E.g. without these derives it is not possible to parse a bencode dictionary into sth. like a HashMap<AsString<Vec<u8>>, _>
.
Easiest to illustrate with an example, attempting to encode an empty Vec<Value>
use bendy::{value::Value,encoding::ToBencode};
fn main() {
let _ = Vec::<Value>::new().to_bencode();
}
Results in a compile error:
error[E0080]: could not evaluate constant
--> $path/bendy-0.3.0/src/encoding/to_bencode.rs:114:38
|
114 | const MAX_DEPTH: usize = ContentT::MAX_DEPTH + 1;
| ^^^^^^^^^^^^^^^^^^^^^^^ attempt to add with overflow
Which I suspect is due to Value::MAX_DEPTH
being the max_usize. A saturating add would be ideal in this case, but using a slightly lower MAX_DEPTH
with room for a bunch parent containers may work, too.
Hi there,
Thanks for the handy library!
I'm trying to deserialize an ip
block from the Bittorrent DHT according to BEP 5.
I can use the following to deserialize the ip
block into a byte vector:
#[derive(Debug, Deserialize)]
struct KRPCMessage {
#[serde(rename = "t")]
transaction_id: String,
#[serde(rename = "y")]
type_code: String,
#[serde(rename = "v")]
version: Option<String>,
#[serde(rename = "r")]
response: HashMap<String, ByteBuf>,
#[serde(with = "serde_bytes")]
ip: Vec<u8>,
}
from_bytes::<KRPCMessage>(&data).unwrap();
This works and gives me:
KRPCMessage { transaction_id: "aa", type_code: "r", version: None, response: {"id": [50, 245, 78, 105, 115, 81, 255, 74, 236, 41, 205, 186, 171, 242, 251, 227, 70, 124, 194, 103]}, ip: [70, 66, 178, 80, 26, 225] }
But what I would like to do is deserialize the ip block into a std::net::SocketAddr
struct. I've tried various approaches including using the "new type" pattern over SocketAddr
but I can't seem to figure out how to get it to work:
fn deserialize<'de, D>(deserializer: D) -> Result<SocketAddr, D::Error>
where
D: Deserializer<'de>,
{
Ok(SocketAddr::new(
IpAddr::V4(Ipv4Addr::new(67, 215, 246, 10)),
6881,
))
}
#[derive(Debug, Deserialize)]
struct KRPCMessage {
#[serde(rename = "t")]
transaction_id: String,
#[serde(rename = "y")]
type_code: String,
#[serde(rename = "v")]
version: Option<String>,
#[serde(rename = "r")]
response: HashMap<String, ByteBuf>,
#[serde(with = "self")]
ip: SocketAddr,
}
from_bytes::<KRPCMessage>(&data).unwrap();
I always get the error:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Decode(Error { context: None, error: MalformedContent(Utf8Error { valid_up_to: 2, error_len: Some(1) }) })'
It seems that it always try to decode the block as a utf-8 string.
Can anyone guide me on how to accomplish this? Please let me know if there any obvious errors—I'm new to Rust.
To use AsString
as key with the default HashMap
encodable implementation we need to implement AsRef<[u8]>
. If we have AsRef<[u8]>
we should also add From<&[u8]>
.
I'm trying to implement FromBencode for an enum but I'm at a loose, decode_bencode_object doesn't take &self as a parameter so I don't have anything to match against to choose an enum variant
pub enum Info {
SingleFile {
length: u64,
md5sum: Option<String>,
name: String,
piece_length: u64,
pieces: Vec<u8>,
private: Option<u8>,
},
MultiFile {
files: Vec<File>,
name: String,
pieces: Vec<u8>,
piece_length: u64,
private: Option<u8>,
},
}
impl FromBencode for MetaInfo {
const EXPECTED_RECURSION_DEPTH: usize = 3;
fn decode_bencode_object(object: Object) -> Result<Self, DecodingError>
where
Self: Sized,
{
// I can't match self because the trait signature doesn't include it
}
}
I was able to easily implement ToBencode by matching self
impl ToBencode for Info {
const MAX_DEPTH: usize = 1;
fn encode(&self, encoder: SingleItemEncoder) -> Result<(), EncodingError> {
match self {
Info::SingleFile {
name,
length,
md5sum,
piece_length,
pieces,
private,
} => encoder.emit_unsorted_dict(|e| {
e.emit_pair(b"name", name)?;
e.emit_pair(b"length", length)?;
if let Some(sum) = md5sum {
e.emit_pair(b"md5sum", sum)?;
}
e.emit_pair(b"piece length", piece_length)?;
e.emit_pair(b"pieces", AsString(pieces))?;
if let Some(p) = private {
e.emit_pair(b"private", p)?;
}
Ok(())
})?,
Info::MultiFile {
name,
files,
piece_length,
pieces,
private,
} => encoder.emit_unsorted_dict(|e| {
e.emit_pair(b"name", name)?;
e.emit_pair(b"files", files)?;
e.emit_pair(b"piece length", piece_length)?;
e.emit_pair(b"pieces", AsString(pieces))?;
if let Some(p) = private {
e.emit_pair(b"private", p)?;
}
Ok(())
})?,
}
Ok(())
}
}
Hopefully, I'm missing something obvious but I haven't found any example of this
Many times Vec<T>
is used as a set datastructure, where nothing cares about the order of elements within the list. To allow such lists to easily be emitted canonically, we should have methods on the various Enocder types (including SingleItemEncoder) that buffer the encoded objects and output them sorted by either Bencode order (i.e, ordered ASCIIbetically by their encoded form) or by any existing Ord
impl (if one exists)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.