ngs-doo / dsl-json Goto Github PK

I have a field that depends on another field's value to define what type it is, something like this:

@CompiledJson
public class MyMessage {
    public String valueOne;
    public String valueTwo;
    
    public Thing thing;
    public ThingType type;
}

public enum ThingType { Abc, Xyz }

public interface Thing {}
public class AbcThing implements Thing {}
public class XyzThing implements Thing {}

I'm just wondering how the best way to go about this is. I'm not really certain of how I can use JsonConverter for this - because the field is dependent on another field being read first.

The following JSON breaks the parser with
SEVERE: IOException parsing entry: Unexpected end of JSON input

{
	"@timestamp": "2016-09-05T12:44:46.000Z",
	"url": {
		"analyzed": "/-/-/call-1111",
		"path": "/-/-/call-1111",
		"query": {
			"was": "sägewerk gmbh"
		}
	},
	"event": "view",
	"origin": "detail",
	"session": ["-9035111172716044880", "6141211111163737100", "-6891111112685400843", "-275111111849250600"],
	"@version": "1"
}

(The was field contains user input which includes HTML-valid Umlaute as ä,ü,ö, or ß.)

I have some "Boolean" attributes in my model class. The generated code contains multiple references to "com.dslplatform.BoolConverter" but it should be referencing to "com.dslplatform.json.BoolConverter".

Hi，i got an error while compiling code,but i don't know where it come from.

Warning:Supported source version 'RELEASE_6' from annotation processor 'com.dslplatform.json.CompiledJsonProcessor' less than -source '1.8'

Add specific Scala types in separate jar

Problem:
Consider multi module project, consists of 'app' and 'lib1' modules, where 'app' module depends on 'lib1'. Each module has POJO classes which annotated with 'CompiledJson'. When we compile 'app' module, the 'dsl_json_Annotation_Processor_External_Serialization' is generated by annotation processor at the top level. But the same file also generated for 'lib1' module. As result we are unable to use generated code from 'lib1' module because, there is class collision.

Solution:
I propose to generate converters for each POJO class, instead of single fat 'dsl_json_Annotation_Processor_External_Serialization' class.
For example if we have 'User' class inside 'com.example.app' package, then generate 'User_dslJsonConverter' class inside same package and register it into 'META-INF/services/com.dslplatform.json.CompiledJson'

100% repro with the 2 simple following files:

build.gradle

plugins {
    id "net.ltgt.apt" version "0.6"
}

apply plugin: 'java'

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
    mavenCentral()
}

sourceSets {
  main {
    java {
      srcDir 'src'
    }
  }
}

dependencies {
    // DSL-json
    compile group: 'com.dslplatform', name: 'dsl-json', version: '1.1.1'
    apt group: 'com.dslplatform', name: 'dsl-json-processor', version: '1.2'

    // Test
    testCompile group: 'junit', name: 'junit', version: '4.12'
}


test {
    testLogging {
        exceptionFormat = 'full'
        // showStandardStreams = true
    }
}

in src/thepack/TheObj.java

package thepack;

import com.dslplatform.json.CompiledJson;

@CompiledJson
public class TheObj {

  public String _id;

}

then: gradle build

:compileJava
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:44: error: cannot find symbol
            if (self.id != null) {
                    ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:47: error: cannot find symbol
                sw.writeString(self.id);
                                   ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:55: error: cannot find symbol
            if (self.id != null) {
                    ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:57: error: cannot find symbol
                sw.writeString(self.id);
                                   ^
  symbol:   variable id
  location: variable self of type TheObj
~/dsljsonbug/build/generated/source/apt/main/dsl_json/json/ExternalSerialization.java:138: error: cannot find symbol
        instance.id = _id_;
                ^
  symbol:   variable id
  location: variable instance of type TheObj
5 errors
:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 6.618 secs

Versions:

> java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

> mono --version
Mono JIT compiler version 4.2.3 (Stable 4.2.3.4/832de4b Wed Mar 30 13:57:48 PDT 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
  TLS:           normal
  SIGSEGV:       altstack
  Notification:  kqueue
  Architecture:  amd64
  Disabled:      none
  Misc:          softdebug
  LLVM:          supported, not enabled.
  GC:            sgen

Using enums with the mandatory attribute seems to cause invalid Java to be generated. I tested with a small sample below:
(dsl-json-processor version 1.4.5)

MyObject.java:

import com.dslplatform.json.CompiledJson;
import com.dslplatform.json.JsonAttribute;

@CompiledJson
public class MyObject {
    @JsonAttribute(mandatory = true)
    public MyEnum value;
}

MyEnum.java:

public enum MyEnum {
    Hello,
    World
}

ExternalSerialization.java error lines:

	static void __serializeJsonObjectMinimal(final MyObject self, com.dslplatform.json.JsonWriter sw, boolean hasWrittenProperty) {
		
		
		else {
			hasWrittenProperty = true;
			sw.writeAscii("\"value\":null", 12);
		}
	}

The inserted "else" here is causing a syntax error during compilation.

Support all relevant Jackson annotations

Using the instructions outlined in this SO post:
http://stackoverflow.com/questions/37022606/dsl-json-compiledjson-bean-instantiation-error

I was able to build my project on a Windows machine by manually downloading dsl-compiler.exe. However, this compiler is a Windows-specific binary. How would I go about building my project on a non-Windows system? Thanks.

Hi, I tried to parse the json like the following one by the newest version 1.7.1.

{"1":"457899683877105664","2":2084,"3":{"safe":0.11111111,"blessed":0.11111111,"easter":0.11111111,"hope":0.11111111},"4":"I hope everyone has a safe and blessed Easter :) 💗"}

My Code is as below:

public class TopicHandler {
    private static final DslJson.Settings settings = Settings.withRuntime().allowArrayFormat(true).includeServiceLoader();
    private static final DslJson<Object> json = new DslJson<Object>(settings);
    .....
    public void handle(RoutingContext routingContext) {
        Map<String, Object> mapContent2 = json.deserialize(Map.class, str.getBytes(), str.getBytes().length);
        System.out.println(mapContent2.get("4").toString());
    }
}

It shows me a strange character after the emoji.

I hope everyone has a safe and blessed Easter :) 💗

Do you know why? It seems every emoji will be with a strange character after deserialization

The following JSON breaks the parser:

{
	"@timestamp": "2016-09-05T12:44:46.000Z",
	"url": {
		"analyzed": "/-/-/call-1111",
		"path": "/-/-/call-1111",
		"query": []
	},
	"event": "view",
	"origin": "detail",
	"session": ["-9035111172716044880", "6141211111163737100", "-6891111112685400843", "-275111111849250600"],
	"@version": "1"
}

(Query is a nested JSON object, but when it's empty it is written out as an empty array.)
It would be nice to have an option like ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT when deserializing JSON.

From what I can gather, it seems like this happens after JsonStreamReader reads from the stream into the buffer - in-between the name and value of the json property. The buffer doesn't contain the name anymore, so it can't be extracted properly (specifically here).

For @CompiledJson classes, would it be possible to register generated JsonReader/Writers with the interface they implement as well? The behavior I am observing is that only the annotated class is registered (DslJson#registerReader). Maybe adding a list of superclasses to the @CompiledJson annotation would be the easiest route?

Currently, I have an if statement for each expected interface and then manually pass the corresponding implementation class.

I just ran into this issue. It seems that the type signature is included even if EXCLUDE is used for getter/setter methods.

TestObject:

@CompiledJson
public class TestObject {
    private Thing value;

    public TestObject() {
    }

    @JsonAttribute(typeSignature = CompiledJson.TypeSignature.EXCLUDE)
    public Thing getValue() {
        return value;
    }

    @JsonAttribute(typeSignature = CompiledJson.TypeSignature.EXCLUDE)
    public void setValue(Thing value) {
        this.value = value;
    }
}

Main:

public class Main {
    public static void main(String arg[]) {
        TestObject x = new TestObject();
        x.setValue(new BlueThing());
        System.out.println(serialize(x));
    }
}

Output:
{"value":{"$type":"test_env.thing.BlueThing","abc":"aaa"}}

https://github.com/ngs-doo/dsl-json/blob/master/library/src/main/java/com/dslplatform/json/JsonWriter.java#L359

Instead of just specifying byte where the error occurred, try to show line number/column number also when possible.
It's possible to do so when input is byte array or the stream is still on the first buffer (which should be most of the time)

This image will be referenced in the README.md:

Reduce how often Double.parseDouble must be called

Most of the performance comes from fast converters. Reflection based databinding should be supported to avoid the need for custom forks due to .net dependency in databinding

We already have support for Java8 time and Joda time, but it would be awesome also to add support for ThreeTen backport project https://github.com/ThreeTen/threetenbp

I tried the new version 1.7.2. The problem about json with emoji has gone.

However, after handling some JSON strings, it continued raising the following error even if the JSON is a very short string.

Unable to process input JSON. Maximum string buffer limit exceeded: 134217728

I print the json when it raise error and it is as simple as {"458165560803868672":{"1":7722,"2":"RT @SivuNgcaba: \"Things we buy to cover up what's inside, coz they made us hate ourselves and love their wealth.\""}}

I increased the string buffer limitation to 1GB, the whole system will raise heap size is over even if I set the -Xmx=3000m.

Do you have any clue what happened?

I tried 1.7.1 and it is the same.

I've been using the JsonWriter(byte[]) constructor with a ThreadLocal. I just noticed all of the JsonWriter constructors are package private now. What is the best practice?

The error message states:
error: Specified type not supported. If you wish to ignore this property, use one of the supported JsonIgnore annotations (such as Jackson @JsonIgnore or DSL-JSON @JsonAnnotation(ignore = true) on field
but the DSL-JSON annotation type is JsonAttribute, not JsonAnnotation.

I'm trying to parse JSON like:

{
  "success": 1,
  "return": {
    "343152": {
      "pair": "abcd",
      "type": "xyz",
      "start_amount": 13.345,
      "amount": 12.345,
      "rate": 485,
      "timestamp_created": 1342448420,
      "status": 0
    }
  }
}

But it fails with:

Exception in thread "main" java.io.IOException: Unexpected end of JSON input
	at com.dslplatform.json.JsonReader.read(JsonReader.java:292)
	at com.dslplatform.json.JsonReader.getNextToken(JsonReader.java:684)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:122)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:31)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:28)
	at com.dslplatform.json.DslJson.deserialize(DslJson.java:1263)
	at com.example.Test.main(Test.java:16)

I've tried to simplify the JSON to find what exactly causes the problem to:

{
  "a": 1,
  "b": {
    "c": {
      "d": "e"
    }
  }
}

but now it fails with:

Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.lang.String.<init>(String.java:196)
	at com.dslplatform.json.DslJson$SimpleStringCache.createAndPut(DslJson.java:539)
	at com.dslplatform.json.DslJson$SimpleStringCache.get(DslJson.java:530)
	at com.dslplatform.json.JsonReader.readKey(JsonReader.java:929)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:120)
	at com.dslplatform.json.ObjectConverter.deserializeObject(ObjectConverter.java:87)
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:125)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:31)
	at com.dslplatform.json.ObjectConverter$4.read(ObjectConverter.java:28)
	at com.dslplatform.json.DslJson.deserialize(DslJson.java:1263)
	at com.example.Test.main(Test.java:16)

My Java code is :

String json = "<ONE OF THE ABOVE SNIPPETS>";
DslJson<Object> dsl = new DslJson<>();
final Map deserialize = dsl.deserialize(Map.class, json.getBytes(StandardCharsets.UTF_8), 16);
System.err.println("des\n" + deserialize);

I'm deserializing an array of objects as follows:

// This is cached, only one instance created
DslJson<Object> dslJson = new DslJson<>(Settings.withRuntime().includeServiceLoader());
JsonReader reader = dslJson.newReader();

// I run this part in a loop thousands of times per second
HttpResponse<InputStream> httpResponse = request.asBinary();
reader.process(httpResponse.getBody());
if (reader.getNextToken() == '[') {
    do {
        // Read beginning of object
        reader.getNextToken(); // {

        // Read id
        reader.getNextToken(); // "
        reader.fillName();     // id
        reader.getNextToken(); // "
        String id = reader.readString(); // foo
        reader.getNextToken(); // ,

        // Read price
        reader.getNextToken(); // "
        reader.fillName();     // price
        reader.getNextToken(); // "
        double price = NumberConverter.deserializeDouble(reader);

        // Read ending of object
        reader.getNextToken(); // }

        // Update values
        update(id, price);
    } while (reader.getNextToken() == ',');
}

It's working fine, but I'm not sure this is the fastest way of doing this as I barely know DSL-JSON api.

How can I increase the performance?

I would like to add dsl-json support to https://github.com/soabase/soabase-halva/blob/master/halva/src/main/java/io/soabase/halva/caseclass/README.md . halva generates immutable classes that are paired with builder classes. Jackson support for the builder pattern can be implemented with @JsonDeserialize.

Example: https://github.com/soabase/soabase-halva/blob/master/examples/example-generated/JsonTestCase.java

Is there an equivalent way to specify a custom builder object with dsl-json? Default compilation results in:

CompiledJson requires accessible public no argument constructor, therefore 'com.dslplatform.maven.Example.ExampleCase' requires public no argument constructor

The folder structure for test classes is not sub-folder like - currently it is single folder named 'com.dslplatform.json' instead of standard-correct sub-folder structure (com/dslplatform/json), which causes problems with certain IDEs.

There seems to be a new issue with JsonAttribute in the latest version.

public interface TestInt {
    long getValue();
    void setValue(long v);
}

@CompiledJson
public class TestObject implements TestInt {
    private long value;

    @Override
    @JsonAttribute(name = "my_data")
    public long getValue() {
        return value;
    }

    @Override
    @JsonAttribute(name = "my_data")
    public void setValue(long v) {
        this.value = v;
    }
}

Output is: {"value":123}, but should be {"my_data":123}

I just ran into an issue while handling some map data from another library - nested lists can't be serialized.

My test code:

DslJson dsl = new DslJson();
dsl.serializeMap(
        Collections.singletonMap("x",
                Collections.singletonList(Collections.singletonList("Hello"))
        ),
        dsl.newWriter()
);

I think the issue is around here

Add support for binding instance to input. This can reduce GC by reusing instances during low level processing

1. There are few places where usage of raw generic type leads to unchecked warnings, e.g:

public MyConfiguration implements Configuration {
	void configure(DslJson json) {
               json.registerReader(...)
        }
}

since DslJson is generic, calling methods on json object produces warning. As solution we should replace raw generic types by unbounded wildcard parmeterized type (e.g DslJson<?>) or where possible by concrete parmeterized type (e.g DslJson<T>).

2. Replace usage of generic arrays with generic list. Since java does not allow create generic arrays, we have to cast object array to generic array which always produce unchecked warnings. eg

public static <D> ObjectFormatDescription<D, D> create(
			final Class<D> manifest,
			final InstanceFactory<D> newInstance,
			final JsonWriter.WriteObject[] encoders,
			final DecodePropertyInfo<JsonReader.BindObject>[] decoders,
			final DslJson json,
			final boolean skipOnUnknown) { ... }

ObjectFormatDescription<MyClass, MyClass> description = ObjectFormatDescription.create(
				MyClass.class,
				MyClass::new,
				new JsonWriter.WriteObject[0],
				new DecodePropertyInfo[] {
					Settings.<MyClass, Integer>createDecoder(...),
					Settings.<MyClass, String>createDecoder(...)
				},
				dslJson, false);

here we should use final List<DecodePropertyInfo<JsonReader.BindObject<D>>> decoders and instead of new DecodePropertyInfo[] {..} use Arrays.asList(...), this is clean, type safe and warning free way.

3. Use upper bound wildcard in some cases, e.g

public Settings<TContext> resolveReader(ConverterFactory<JsonReader.ReadObject> reader) { ... }

class MyReader implements JsonReader.ReadObject { ... }
class MyConverterFactory implements ConverterFactory<MyReader> { ... }

json.resolveReader(new MyConverterFactory()); // error

to resolve the error above, we should change signature to resolveReader(ConverterFactory<? extends JsonReader.ReadObject> reader)

This is not complete list, perhaps there are other similar issues.

https://docs.gradle.org/4.8.1/userguide/java_plugin.html#sec:incremental_annotation_processing

After PR #61 it seems that our annotation processor almost fits 'isolating' category. But we use Filer#createResource, which currently is limitation by Gradle. We need to consider use runtime probing for type converters instad of writing to META-INF/services, as you suggested in here.

JsonReader.parseString() contains a bug related to the variable _tmpLen, which is at first used to keep track of how many bytes are still ahead in the buffer, and later to reflect the size of the temporary buffer. But it isnt't set to the size of the temporary buffer when it changes its role, so whenever a backslash or UTF-8 multibyte character was encountered, the temporary buffer will often unnecessarily double its size, which will soon lead to OutOfMemoryError.

	if (i == _tmp.length) {
		_tmp = chars = Arrays.copyOf(chars, chars.length * 2);
	}
	_tmpLen = _tmp.length; // This line needs to be inserted!
	currentIndex = ci;
	int soFar = --currentIndex - startIndex;

This will allow use reflection data binding on platform which currently does not support Java8 (e.g. Android prior API 24).

I propose to move appropriate classes from 'dsl-json-java8' module to core module 'dsl-json' or we can create new one 'dsl-json-reflection'.

It seems relatively easy to do. We just need to replace usage of 'java.lang.function.*' with our custom interfaces. And keep ImmutableAnalyzer class in 'dsl-json-java8' module, since parameter names are not supported prior Java8.

I'm evaluating dsl-json and considering migrating from jackson. The performance I saw on a few benchmarks seemed very impressive and I'd love to play with it a bit more. However, one of my core needs is to deal with graph ser/deser.

Does the lib provide ways to handle circular dependency out-of-the-box? I couldn't find anything on docs or examples. If so, how does it work? If not, is there any way to solve that?

Instead of using virtual dispatch, reader should check for stream field while processing input

I'm getting this error when trying to build an Android (Kotlin) project with dsl-json.

error: Failed saving compiled json serialization files

This is my build.gradle:

dependencies {
  ...
  kapt "com.dslplatform:dsl-json-processor:1.7.3"
  implementation "com.dslplatform:dsl-json-java8:1.7.3"
}

These are the entities I'm trying to deserialize:

@CompiledJson
data class CityJson(
        var id:         Long        = 0,
        var name:       String      = "",
        var country:    String      = "",
        var coord:      CoordJson   = CoordJson(0.0, 0.0)
)

@CompiledJson
data class CoordJson(
        var lat: Double = 0.0,
        var lon: Double = 0.0
)

and this is how I'm trying to do use dsl-json:

val input = ctx.resources.openRawResource(R.raw.city_list)
val dslJson = DslJson<Any>(Settings.withRuntime<Any>().includeServiceLoader())
val reader =  dslJson.newReader().process(input)
var city = reader.next(CityJson::class.java)

while (city != null) {
  ...
  city = reader.next(CityJson::class.java)
}

Link to debug log (with gradle --stacktrace option and dsl-json.loglevel=DEBUG):
https://gist.github.com/dvhp/d1be8a8bebad9f0539b4a50b16374cd8

I figured that the current version of Gradle/Android uses jack and is incompatible with the android-apt plugin. How do i use dsl-json in current android. The provided sample is also "outdated"

It should be possible to pass in stream which would be written to directly instead of buffering the response and then flushing it

Would it be possible to add a setting to avoid the serialization of a field if it is null? Or to conditionally apply the @ignore annotation.

Currently I am serializing a POJO that has nullable fields with a requirement to only serialize non-null values. In Jackson it could be achieved with a @JsonInclude(JsonInclude.Include.NON_NULL) annotation.

Run all tests for PRs

We can use Jsr305 as base

Create custom NonNullApi annotation, which makes all parameters and return values non-nullable by default, e.g.

@Target(ElementType.PACKAGE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Nonnull
@TypeQualifierDefault({ElementType.METHOD, ElementType.PARAMETER})
public @interface NonNullApi {
}

then annotate package with it:

// file package-info.java
@NonNullApi
package com.dslplatform.json;

then annotate only nullable parameters or return values

import javax.annotation.Nullable;

public class DslJson<TContext> {
    @Nullable
    public JsonWriter.WriteObject<?> tryFindWriter(final Type manifest) {
    }
}

http://json-b.net/getting-started.html

I'm currently using dsl-json in an API server. We have several types of response messages with different parameters, all of which implement the Response interface. I believe type information is included automatically because we're using an abstract type, but I'm looking for some way to disable that.

@CompiledJson
public class ApiOutput {
    public Response data;
    public Integer status_code;
    ....
}

@CompiledJson
public class ResultResponse implements Response {
    public String result = "success";
}

The output JSON of serialization:

{
    "data": {
        "$type": "responses.ResultResponse",
        "result": "success"
    },
    "status_code": 200,
    ...
}

Is there some way to disable the $type field during serialization? Maybe some annotation parameter?

Jackson will default to using LinkedHashMap to represent JSON where there is no type information given, or the type information is given as Object.class.

Is there any support for this type of behaviour with dsl-json?

I am looking to deserialize json where a field may have varying types. For example, in my model class I have an instance variable declared as Map<String, Object> fieldName. The object value may be represented in the JSON as a string, int, object or array etc. Does dsl-json support a method of generically deserializing this into some sort of structure so it can be accessed from the instance variable.

Here is an example of some JSON that I hope to deserialize into a Map<String, Object>

 "fieldName": {
            "alwaysString": {
                "fieldInsideObject": 5
            },
            "anotherString": 40
        }

Thanks

DslJson dslJson = new DslJson<>();
JsonWriter writer2 = dslJson.newWriter();
List o = new ArrayList();
o.add("Hello Word!");
o.add(1);
o.add(false);

dslJson.serialize(writer2, o);
System.out.println(writer2.toString());

how could this happen ?

In the readme it says that this library supports external annotations like Jackson's @JsonProperty. Can't find any example of an annotation being used to change the name used for deserialization/serialization declared outside of the model class itself.

In Jackson I'd just create a new mixin class and register it with my object mapper. Is their similar support for dsl-json?

Thanks

Currently the dsl-platform.com website is down, meaning I cannot download the compiler tool as required. It would be good if the executable is cached in case of server downtime.

Currently if a json missing some attribute, the 'null' will be assign to POJO property. But this no make sense if property is annotated as nonnull. We should throw exception instead.