Git Product home page Git Product logo

rust-jsonm's Introduction

rust-jsonm

Build Status

jsonm implementation port for Rust.

Original library written in JS is here: https://github.com/lennartcl/jsonm

jsonm is a fast and safe way to compress JSON messages using memoization. jsonm makes messages up to several orders of magnitude smaller by getting rid of repeated names and values.

jsonm packs

[
    { "firstName": "Francis", "lastName": "Doe" },
    { "firstName": "Anna", "lastName": "Smith" },
    { "firstName": "Agent", "lastName": "Smith", "isAlias": true },
    { "firstName": "Anna", "lastName": "Francis" }
]

into

[ 0,
    ["firstName", "lastName", "Francis", "Doe"],
    [3, 4, "Anna", "Smith"],
    [3, 4, "isAlias", "Agent", 8, true],
    [3, 4, 7, 5]
]

Notice how it eliminates all common substrings like "firstName" using memoization! jsonm keeps a dictionary to compress future messages even further. Send the message above a second time, and it becomes:

[0,[3,4,5,6],[3,4,7,8],[3,4,9,10,8,11],[3,4,7,5],1]

And

[
    { "firstName": "Bryan", "lastName": "Fuller" },
    { "firstName": "Anna", "lastName": "Adams" },
    { "firstName": "Tim", "lastName": "Peterson" },
    { "firstName": "Francis", "lastName": "Peterson" }
]

becomes

[0,[3,4,"Bryan","Fuller"],[3,4,7,"Adams"],[3,4,"Tim","Peterson"],[3,4,5,16]]

By avoiding repetition, jsonm can for example help decrease the size of messages sent from a web server to the client. It effectively leaves out all information the client already knows about.

Installation

[dependencies]
"jsonm" = "0.1"

Examples

#[macro_use]
extern crate serde_json;
extern crate jsonm;

use jsonm::packer::{PackOptions, Packer};
use jsonm::unpacker::Unpacker;

let mut packer = Packer::new();
let options = PackOptions::new();
let packed = packer.pack(&json!({ "bar": 1, "foo": "1" }), &options).unwrap(); // packed is ["bar", "foo", "1", "~1", 0]

let mut unpacker = Unpacker::new();
let unpacked: Value = unpacker.unpack(&packed).unwrap(); // unpacked is Object({ "bar": 1, "foo": "1" })

rust-jsonm's People

Contributors

andrewnester avatar anester-aws avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

fxwiegand wabinab

rust-jsonm's Issues

Bug trying to unpack empty json.

There is a bug if one tries to unpack an empty json. Example:

let mut packer = Packer::new();
let options = PackOptions::new();
let packed = packer.pack(&json!({}), &options).unwrap();

let mut unpacker = Unpacker::new();
let unpacked: Result<Value, UnpackerError> = unpacker.unpack(&packed);

will result in:

thread 'main' panicked at /workspace/.cargo/registry/src/index.crates.io-6f17d22bba15001f/jsonm-0.2.0/src/unpacker.rs:162:39:
index out of bounds: the len is 0 but the index is 0

The value can be packed, and the output is:

Array [
    Number(0),
]

Unfortunately, that packed value cannot be unpacked.

For the meantime, you'd need to check whether the pack value equals that above, then manually return json!({}).

Nested optional property issue

Hi there!

Just trying out your crate and get an issue which seems to be down to optional nested properties, where some of the data appears in the wrong place when unpacking. Details and a sample test below...

image

[dependencies]
jsonm = "0.2.0"
#[cfg(test)]
mod tests {
    use jsonm::packer::{PackOptions, Packer};
    use jsonm::unpacker::Unpacker;
    use serde::{Deserialize, Serialize};
    use std::collections::BTreeMap;

    #[derive(Debug, Deserialize, Serialize)]
    struct Person {
        name: String,
        age: u8,
        address: String,
    }

    impl Person {
        fn new(name: &str, age: u8, address: &str) -> Person {
            Person {
                name: name.to_string(),
                age,
                address: address.to_string(),
            }
        }
    }

    #[derive(Debug, Deserialize, Serialize)]
    struct Record {
        person: Person,
        tag: Option<String>,
    }

    impl Record {
        fn new(person: Person) -> Record {
            Record { person, tag: None }
        }
    }

    #[test]
    // fails as "tag":null becomes "tag": "name 2" through packing to string and then unpacking
    fn test() {
        let records: BTreeMap<u32, Record> = BTreeMap::from([
            (1, Record::new(Person::new("name", 18, "address 1"))),
            (2, Record::new(Person::new("name 2", 60, "address 2"))),
            (3, Record::new(Person::new("name 3", 32, "address 3"))),
            (4, Record::new(Person::new("name 4", 9, "address 2"))),
        ]);
        let serialised = serde_json::to_string(&records).unwrap();

        let packed = pack(records);
        let unpacked: BTreeMap<u32, Record> = unpack(packed);

        assert_eq!(serialised, serde_json::to_string(&unpacked).unwrap())
    }

    fn pack<T: Serialize>(value: T) -> String {
        let mut packer = Packer::new();
        let options = PackOptions::new();
        let value = serde_json::value::to_value(value).unwrap();
        println!("pack input: {value}");
        let packed = packer.pack(&value, &options).unwrap();
        println!("packed: {packed}");
        packed.to_string()
    }

    fn unpack<T: for<'de> Deserialize<'de>>(value: String) -> T {
        println!("unpack input: {value}");
        let packed = serde_json::from_str(&value).unwrap();
        println!("unpacked value: {packed}");
        let mut unpacker = Unpacker::new();
        let unpacked = unpacker.unpack(&packed).unwrap();
        println!("unpacked: {unpacked}");
        let unpacked = serde_json::from_value(unpacked).unwrap();
        unpacked
    }
}

error "index out of bounds: the len is 0 but the index is 0"

Hi, I'm trying to use the library to unpack the attached data and I am getting the error thread 'tokio-runtime-worker' panicked at 'index out of bounds: the len is 0 but the index is 0', /Users/sammyliu/.cargo/registry/src/github.com-1ecc6299db9ec823/jsonm-0.1.4/src/unpacker.rs:161:27. The same data unpacks fine with the js library though. Any idea?

packed-data.json.zip

Maximum dict size leads to unexpected behaviour

Hey there,

first of all thanks for building this crate.

When applying jsonm to our data we found an issue related to the maximum dict size causing mismatches of dict entries.
For some reason the dict_index will be set to MIN_DICT_INDEX overwriting existing entries when max_dict_size is exceeded (see L309).

As this leads to non-deterministic results the execution should fail and an error be returned.

But what I'm wondering is, why even a maximum dict size needs to be defined. I did not find a reason for not just increasing the index being only limited by the maximum u64 value.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.