bolerio / mjson Goto Github PK
View Code? Open in Web Editor NEWLean JSON Library for Java, with a compact, elegant API.
License: Apache License 2.0
Lean JSON Library for Java, with a compact, elegant API.
License: Apache License 2.0
There is the occasional parsing bug that manifests itself. Also, our parser is not written for super high performance so we may decide to replace it some day. For that, we need tests that JSON documents & values are parsed correctly into the corresponding Json elements.
The unit tests should focus especially on corner cases with lots of ill-formed JSON documents. Well-formed document with no syntax errors are important as well, but new bugs are likely to be discovered with how the parser deals with bad syntax.
Also, it would be nice, once we have error test cases to improve the error reporting of the parser itself..
Using mjson to parse untrusted JSON String may be vulnerable to denial of service (DOS) attacks. If the parser is running on user supplied input, an attacker may supply content that causes the parser to crash by stackoverflow.
Exception in thread "main" java.lang.StackOverflowError
at mjson.Json$Reader.next(Json.java:2630)
at mjson.Json$Reader.skipWhiteSpace(Json.java:2669)
at mjson.Json$Reader.read(Json.java:2709)
at mjson.Json$Reader.readObjectKey(Json.java:2751)
at mjson.Json$Reader.readObject(Json.java:2765)
at mjson.Json$Reader.read(Json.java:2718)
at mjson.Json$Reader.readObjectKey(Json.java:2751)
at mjson.Json$Reader.readObject(Json.java:2765)
at mjson.Json$Reader.read(Json.java:2718)
at mjson.Json$Reader.readObjectKey(Json.java:2751)
at mjson.Json$Reader.readObject(Json.java:2765)
at mjson.Json$Reader.read(Json.java:2718)
at mjson.Json$Reader.readObjectKey(Json.java:2751)
at mjson.Json$Reader.readObject(Json.java:2765)
at mjson.Json$Reader.read(Json.java:2718)
at mjson.Json$Reader.readObjectKey(Json.java:2751)
at mjson.Json$Reader.readObject(Json.java:2765)
at mjson.Json$Reader.read(Json.java:2718)
<dependency>
<groupId>org.sharegov</groupId>
<artifactId>mjson</artifactId>
<version>1.4.1</version>
</dependency>
import mjson.Json;
public class PoC {
public final static int TOO_DEEP_NESTING = 9999;
public final static String TOO_DEEP_Object = _nestedDoc(TOO_DEEP_NESTING, "{ ", "} ", "0");
public static String _nestedDoc(int nesting, String open, String close, String content) {
StringBuilder sb = new StringBuilder(nesting * (open.length() + close.length()));
for (int i = 0; i < nesting; ++i) {
sb.append(open);
if ((i & 31) == 0) {
sb.append("\n");
}
}
sb.append("\n").append(content).append("\n");
for (int i = 0; i < nesting; ++i) {
sb.append(close);
if ((i & 31) == 0) {
sb.append("\n");
}
}
return sb.toString();
}
public static void main(String[] args) {
String jsonString = TOO_DEEP_Object;
Json.read(jsonString);
}
}
Refer to the solution of jackson-databind: Add the depth variable to record the current parsing depth. If the parsing depth exceeds a certain threshold, an exception is thrown. (FasterXML/jackson-databind@fcfc499)
Refer to the GSON solution: Change the recursive processing on deeply nested arrays or JSON objects to stack+iteration processing.((google/gson@2d01d6a20f39881c692977564c1ea591d9f39027))
The API is simple and well-defined, but our pluggable factory allows for different implementations of all JSON types. It would be nice to have a comprehensive test harness so alternative implementations can be tests. One way to organize the test suite is by JSON type. So have a class containing tests for boolean, another for strings, another for objects, another for arrays, another for numbers and one for nulls. Because of the unified interface for all JSON types, it should be easy to just go over the methods and make sure they all work.
I have several cases where I have the json element in a hierarchy (eg after an event has occurred) but I need to know its key in the parent json. Basically I need to recurse the path back to the top level. This is remarkably hard to do efficiently! Ive considered several possibilities:
Any ideas?
Indeed, the currently existing test-case actually runs into infinite loop with them in (testSortedArrayMerge).
Possible fix (at least the testcase is fixed by that):
unserializable@63d1773?diff=unified
Hey , is it possible to pretty print Json as a string ? For now I found only 2 toString methods(default one and one with maxCharacters) and non of them is able to pretty print Json object
This should be in a related to mJson project (e.g. mJsonPlus that could contains all extra goodies while keeping mJson itself small). The idea is to create Java annotations that validate input and output of Json methods:
@JsonValidate("/resources/schemas/person.json")
public void doSomethingWithPerson(Json person)
{
....// here we know the person arguments is a valid person
}
such validating annotations would be placed only at entry points...of course, to avoid validating every single time something as passed as an argument. Alternatively, we might have some meta information kept in a global map that says that such and such objects has been validated and follows a schema.
As stated in the javadoc of the interface Json.Factory the method make(Object) allows me to plug-in my own mapping of java.util.Date to Json. But how to achive this without reimplementing the whole method?
Futhermore I would like to know if it is possible to globally map the other way around from a Json string to java.util.Date. I would have expected something like Json.asDate() or Json.asType(Class type) for the conversion to Java instances.
Not sure if a great idea, but an 'overlay' or 'overwrite' method would behave sort of like 'with', except recursively it will perform a merge (actually maybe 'merge' is a better name for such a method then): if A is overlaid over B, all properties in A that are not in B will be copied to B and also all properties in A that are also in B will replace B's values. But because this is done recursively, we may have nested properties deep in B's structure be preserved instead of just disappearing.
So this is essentially a recursive "with". While "with" overwrite blindly, a recursive with does a deep merging.
I've only seen the need for this popup very rarely though.
This example:
{
"type":"something",
"value1":"Blabla",
"value2":"years"
"value3":"wtf"
}
is missing a coma after the 3d value, and instead of an error being thrown it parses as:
{"value2":"wtf","value1":"Blabla","type":"something"}
Hi,
I'm trying to convert JSON from/to String as follows:
String myJSONString = "{\"user\":\"myuser\",\"pw\":\"mypass\",\"log\":true}";
System.out.println(myJSONString);
Json json = Json.read(myJSONString);
System.out.println(json.toString());
That is, I expect that myJSONString.equals(json.toString()) returns true, but it does not.
The values that System.out.println() show are:
{"user":"myuser","pw":"mypass","log":true}
{"log":true,"pw":"mypass","user":"myuser"}
For myJSONString and json.toString(), respectively. Therefore, json.toString() inverts the string. Is this a bug? I want to load/store the same JSON string from/to disk several times but due to this behaviour I'm not able to do that.
PS: The problem is that next time I load the MyJSONString from this it cannot be validated using the JSON schema.
ResourceLoader resourceLoader ;
Json.Schema schema = Json.schema(resourceLoader.getResource("classpath:" + jsonSchemaFile).getURI());
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at mjson.Json$DefaultSchema.<init>(Json.java:1009) ~[mjson-1.3.jar:na]
at mjson.Json.schema(Json.java:1035) ~[mjson-1.3.jar:na]
... 48 common frames omitted
Caused by: java.lang.NullPointerException: null
at mjson.Json.resolveRef(Json.java:462) ~[mjson-1.3.jar:na]
at mjson.Json.expandReferences(Json.java:508) ~[mjson-1.3.jar:na]
at mjson.Json.expandReferences(Json.java:529) ~[mjson-1.3.jar:na]
at mjson.Json.expandReferences(Json.java:519) ~[mjson-1.3.jar:na]
at mjson.Json$DefaultSchema.<init>(Json.java:1006) ~[mjson-1.3.jar:na]
When you do:
Json.object().at("count", 10)
will automatically figure that there is not property named "count" in the object and add one with the value 10. The 2nd argument here is intended to be the default for missing properties. It seemed sensible for that accessor 'at' method to mutate the object because from a declarative perspective, a default value is a default value, always the same no matter the user. But I find myself worrying over modifying existing objects, especially if they are being saved back into a database where you want to avoid storing default to keep the data sparse and less storage consuming. The solution is call 'dup()' so that the original DB entity is not modified. But that's not very pleasant either. On the other hand, I've very rarely come across a use case where I really wanted 'at' the insert the specified default. So, breaking backward compatibility unfortunately, the behavior should be change for 'at' not the modify the object.
Parsing currently assumes we have the full document and it's a one short operation. But for processing streams of Json documents a user of mjson needs to do extra work to separate the start and end of documents which amounts almost to a full parsing as well. Instead, we should have a Stream<Json> readStream(source)
method that does that automatically.
The current release of MJSON (1.4.1) includes junit as a dependency in it's pom. This is surprising since it promises to have no dependencies. I see that it's actually been fixed in master but no new version has been released since 2017.
Using mjson to parse untrusted JSON String may be vulnerable to denial of service (DOS) attacks. If the parser is running on user supplied input, an attacker may supply content that causes the parser to crash by stackoverflow.
Exception in thread "main" java.lang.StackOverflowError
at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:165)
at mjson.Json.factory(Json.java:1192)
at mjson.Json.array(Json.java:1291)
at mjson.Json$Reader.readArray(Json.java:2787)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
at mjson.Json$Reader.read(Json.java:2715)
at mjson.Json$Reader.readArray(Json.java:2788)
<dependency>
<groupId>org.sharegov</groupId>
<artifactId>mjson</artifactId>
<version>1.4.1</version>
</dependency>
import mjson.Json;
public class PoC {
public final static int TOO_DEEP_NESTING = 9999;
public final static String TOO_DEEP_DOC = _nestedDoc(TOO_DEEP_NESTING, "[ ", "] ", "0");
public static String _nestedDoc(int nesting, String open, String close, String content) {
StringBuilder sb = new StringBuilder(nesting * (open.length() + close.length()));
for (int i = 0; i < nesting; ++i) {
sb.append(open);
if ((i & 31) == 0) {
sb.append("\n");
}
}
sb.append("\n").append(content).append("\n");
for (int i = 0; i < nesting; ++i) {
sb.append(close);
if ((i & 31) == 0) {
sb.append("\n");
}
}
return sb.toString();
}
public static void main(String[] args) {
String jsonString = TOO_DEEP_DOC;
Json.read(jsonString);
}
}
Refer to the solution of jackson-databind: Add the depth variable to record the current parsing depth. If the parsing depth exceeds a certain threshold, an exception is thrown. (FasterXML/jackson-databind@fcfc499)
Refer to the GSON solution: Change the recursive processing on deeply nested arrays or JSON objects to stack+iteration processing.((google/gson@2d01d6a20f39881c692977564c1ea591d9f39027))
Since we moved the test cases to junit (from testng), the suite of JSON schema tests doesn't run anymore. This needs fixing before 1.4.
v1.4.0 binary seems to be compiled with Java 7 (Unsupported major.minor version 51.0)
Is there a binary version for Java 1.6?
Thanks!
Seems pretty advance and nice:
https://code.google.com/p/json-path/
Since Spring framework is adopting it, probably it's going to gain industry acceptance, and there's nothing else like it.
You are creating a topnull
object in Json$NullJson class.
static NullJson topnull = new NullJson();
This instance is static and therefore, will never be garbage collected.
Every time some json data containing null fields is processed, the enclosing
field of this object is growing.
You can reproduce the problem running this code:
package mjson;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import java.util.stream.IntStream;
public class Main {
public static void main(String[] args) throws FileNotFoundException, InterruptedException {
String content = new Scanner(new File("resources/file.json")).useDelimiter("\\Z").next();
Scanner in = new Scanner(System.in);
while(true){
in.nextLine();
IntStream.range(0, 100000).forEach(i -> {
Json.read(content);
});
System.out.println("SIZE: "+ Json.NullJson.topnull.enclosing.asList().size());
}
}
}
Used file.json:
{
"whatever1": null,
"whatever2": null,
"whatever3": null,
"whatever4": null,
"whatever5": null,
"whatever6": null,
"whatever7": null,
"whatever8": null,
"whatever9": null
}
On the screenshot below, you can see what happens on the heap.
I sequentially run multiple Json.read()
calls and then trigger GC.
After just few runs, heap size is already above 0.5GB.
Quick fix:
With @KamilKrol we introduced a quick fix, by removing the topnull
field and creating new NullJson
instance every time DefaultFactory#nil()
is called.
This seems to be solving the leak problem, but the proper solution probably requires investigating why NullJson#enclosing field is not empty after processing is finished.
I integrated mjson inside a JSF web application and it requires the Json instances to be serializable:
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: mjson.Json$ObjectJson
The issue may be solved by adding "implements Serializable" to the objects ArrayJson, BooleanJson, NullJson, NumberJson, ObjectJson and StringJson. What do you think?
How about adding a feature for pretty output?
Hi,
do you plan to ad date support?
Regards
Massi
The interface will of course fatten a bit, but that's also the point of having a single class. Instead of writing
object.at(propertyname).asString()
we could write
object.asString(propertyname)
or for arrays:
object.asString(10)
and like this for all asXXX methods.
Or drop the "as" here since we are not "at the value" yet to be say that we look at it "as something". So maybe just like this:
object.string(propname)
object.bool(properyname)
that's even shorter, but the problem is new method names and less consistency with the rest of the API. Is saving a few characters worth the loss of consistency? Don't think so..
Hi,
Found I can get a huge increase in speed by caching in the escaper. In my case I am outputting periodic updates to the same data, so its especially useful. the escape method was using 5% CPU, now its below 1%. I know you dont want any dependencies but I used Google guava cache. Code is here if you want to use it:
escapeHash = CacheBuilder.newBuilder()
.maximumSize(1000)
.build(
new CacheLoader<CharSequence, String>() {
public String load(CharSequence plainText) throws Exception {
StringBuilder escapedString = new StringBuilder(plainText.length() + 20);
try {
escapeJsonString(plainText, escapedString);
} catch (IOException e) {
throw new RuntimeException(e);
}
return escapedString.toString();
}
});
}
public String escapeJsonString(CharSequence plainText) {
return escapeHash.getUnchecked(plainText);
}
The current default implementation of Json.toString
will output an ellipsis (...
) when the traversal is about to output the same object (same Java reference). This is to avoid circularity issues and because the aim of toString
was for display purposes. We might break backward compatibility (and maybe this can be made optional), but this decision is a bit counter-intuitive. One expected toString to at least produce a valid JSON string. Because we set out to support graphs in this API and not just trees, we have to handle circular structures. But we can do this in a separate method, e.g. displayString
or some such. Alternatively, we could offer a hook to be provided by the user to deal with serializing the same object, but in a different context.
How would you implement firing an event whenever data in the Json object changed? I have a use case where I want to trigger external processing when certain values are added/updated, or deleted.
For a predictable standard string serialization, we should use RFC 7159
The following code produces a NullPointerException:
Json json = Json.object();
String value = json.at("notAvailable").asString();
What is the advantage of json.at("...") returning null instead of a json object? I would have expected the java null value from the asString() method.
The method Json.read() for example will never return the actual Java null. Could this also be done for the method json.at("...")?
Nice work on mJson, I like the concept. Been trying it out and I found I added these pretty quick for big trees. Typing at("person").at(".address").at("city")... gets quite tedious :-)
/**
* Get the Json node at "person.address.city"
* @param path
* @return
/
public Json atPath(String path){
String[] paths = path.split(".");
return atPath(paths);
}
/*
* Get the json node at {"person","address","city"}
* Convenient when using CONSTANTS, atPath(PERSON,ADDRESS,CITY)
* @param path
* @return
*/
public Json atPath(String ... path ){
Json json=null;
for(String k:path){
if(json==null)json=at(k);
json=json.at(k);
if(json==null) return null;
}
return json;
}
This seems like a new effort, but important for REST services to have something like this: http://jsonpatch.com/, so we could support it.
Persistable json objects usually carry some kind of technical identifier with them. Is there some way to compare two json objects without technical properties?
For example:
{"id":"1","type":"car","model":"Ferrari","color":"red"}
{"id":"2","type":"car","model":"Ferrari","color":"red"}
Both objects represent the same car (functional equality) but they are not equal/identical because of their different IDs. If there would be a way to exclude/ignore certain properties during the equals operation would be a great feature. Maybe the API could look something like car1.functionalEquals(car2,"id"); whereas the last argument is a variable parameter list. Or maybe it would be better to mark technical properties within the interal data structure of mjson itself for beeing able to handle nested/more complex objects. This would require some API for marking properties as beeing "excluded" from the functional equals comparison.
public Json delAt(String property)
{
Json el = object.remove(property);
removeParent(el, this);
return this;
}
when there is no value for the property being deleted, the removeParent
call leads to an NPE
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.