Excessive memory consumption in parse() about json-simple HOT 8 CLOSED

You don't have to convert the stream into a String to pass it to JSONParser.
JSONParser accepts a Reader as its parameter.

Could you comment out the last line and then test it again?

return (JSONObject) jsonParser.parse(sb.toString());

And could you help to print out the result of sb.toString() and post it here 
before
passing it to JSONParser? That will be very helpful to analyze this issue.

Thanks.

That didn't help, unfortunately. I also can't log the StringBuilder contents, 
because
adb just cuts off strings which are too long to log, but I've attached the JSON 
data
served by our web service.

It's not a problem with that specific document though. I've seen this happening 
all
over the place.

Thanks a lot for the data.
It's strange that from my initial testing (using a very simple method), nothing 
seems
to be special.(Local JDK is 1.6. What is the version of the JDK/JRE in your 
environment?)

Here's the code:
{{{{
public static void main(String[] args) throws Exception{
        long t = System.currentTimeMillis();
        long m = Runtime.getRuntime().freeMemory();
    Object obj = JSONValue.parse(new FileReader("reviews.json"));
        long delta = System.currentTimeMillis() - t;
        long deltaM = m - Runtime.getRuntime().freeMemory();
        System.out.println(delta + "ms");
        System.out.println(deltaM/1024 + "K");
    System.out.println(obj);
}
}}}}

Here's the result:
79ms
537K
(The resulting JSON text)

I also used YourKit 7.5 to do memory profiling and got the following data:
Overall objects:
All live objects: 13,025 Shallow size: 1,098,824 bytes Retained size: 1,098,824 
bytes

Dominant objects:
char[] 2,902 objects, shallow size: 429,656
java.lang.String 3,132 objects, shallow size: 75,168

Thanks.

I just replaced JSON simple with the reference JSON implementation from json.org
(which is shipped with Android), and it doesn't suffer from this problem.

There must be something leaking (and massively so -- considering it turns a 40k 
JSON
string into 12 megabytes), in JSONParser.parse().

However, to not waste your time:
It may well be that this is in fact something related to Apache Harmony, the
open-source Java implementation that powers Android. I have been running into
problems on Android several times now where Harmony turned out to be the source 
of
the problem (it's scary sometimes how buggy it can be at its core 
infrastructure).

I guess the best thing to do would be:
Write a simple test case which parses a JSON file, and profile it both using 
the Sun
JVM and the Harmony JVM.

The Android profiler unfortunately isn't detailed enough to tell me exactly 
which
method/object is actually consuming so much memory, but it shows me 10-12MB 
worth of
character data being allocated during a parse(), so it must be something 
related to
reading/buffering character data.

Thanks a lot for your information.
I'll test it in Harmony JVM.

One solution would be to just use the lexer. I've posted some code to Fang that 
extends the lexer and uses it 
similar to the XMLStreamReader in Java 1.6, effectivly saving lots of RAM when 
processing large amounts of 
JSON. Let me know and I'll post it again.

Hi,

The following patch should fix the OutOfMemory problem. At least, it works for 
my JVM.


--- a/src/org/json/simple/parser/Yylex.java
+++ b/src/org/json/simple/parser/Yylex.java
@@ -577,7 +577,7 @@ int getPosition(){
           }
         case 25: break;
         case 4:
-          { sb.delete(0, sb.length());yybegin(STRING_BEGIN);
+          {  sb = null; sb = new StringBuffer(); yybegin(STRING_BEGIN);
           }
         case 26: break;
         case 16:

Glen

Hi glen.tw,

I've applied it to the code. Thanks a lot for your contribution.
Unfortunately I cannot validate it on Android Phone because I don't have one. 
Can any of you help validate it and let me know the result?

Thanks,
Yidong.