Thaw fails when nippy comes from AOT jar about nippy HOT 21 CLOSED

Hey Matt, what's the urgency on this?

from nippy.

I'm not sure how @mbossenbroek feels, it's blocking some work that I'm doing, so I'd of course love to see it addressed. That being said, this seems to me to be a pretty unique set of intersecting problems. I don't know of anyone else that I've seen on the internet talking about it.

from nippy.

Entirely up to @aconbere :)

This was found while using Prismatic's graph library with PigPen. I think we have a few workarounds though (disabling the use of records, not using AOT)

from nippy.

Where is the GitHub repo for [com.netflix.pigpen/pigpen "0.2.13"]?

from nippy.

https://github.com/Netflix/PigPen

It's up to 0.2.15 at the moment, but the last two releases were only minor changes. Everything related to this should be the same.

from nippy.

Could you get me some context? Has anything changed recently with PigPen that prompted this error? This is a new error?

from nippy.

I don't think that any recent changes would have triggered this. My guess is that no one has ever used pigpen with records before (or didn't tell me that it didn't work). I would speculate that this has always been a problem & we never noticed.

from nippy.

I would speculate that this has always been a problem & we never noticed.

Okay, makes sense - thanks.

Any idea why this happens if nippy is AOT'ed into another jar?

And could you clarify what you mean here by Nippy being "AOT'ed into another jar"? I'm looking over the PigPen source but am not familiar at all. Are you depending on Nippy internally? Where?

from nippy.

pigpen-core depends on nippy here: https://github.com/Netflix/PigPen/blob/master/pigpen-core/build.gradle#L7

When switching on AOT, clojuresque seems to take all dependencies and AOT them into the pigpen jar:

jar tf ~/.m2/repository/com/netflix/pigpen/pigpen/0.2.13/pigpen-0.2.13.jar | grep nippy
taoensso/nippy$loading__4958__auto__.class
taoensso/nippy$fn__4174.class
taoensso/nippy$when_debug_mode.class
taoensso/nippy$fn__4179.class
taoensso/nippy$fn__4182$G__4178__4185.class
taoensso/nippy$fn__4182$G__4177__4189.class
taoensso/nippy$fn__4182.class
taoensso/nippy$write_id.class
...

It has the effect of making pigpen an uberjar of sorts.

from nippy.

clojuresque is this?

Can you provide steps for how to recompile a local PigPen copy using a (local) debug build of Nippy?

from nippy.

Yep - it's the clojure support for gradle builds.

Rebuilding should be this:

$ ./gradlew clean build install

Just change the build.gradle mentioned above to

    compile 'com.taoensso:nippy:2.8.1-SNAPSHOT'

or whatever the name is. Let me know if that works for you.

from nippy.

Sorry, am struggling to get the build working so that I can run lein test against the local Gradle build. Will try come back to this on Monday.

The more info you can get me, the more likely I'll be able to help; just juggling some high-priority work atm. Not at all familiar with PigPen or Gradle.

Nippy's record de/serialization is pretty simple:

• During freeze, the record's class name is serialized.
• During thaw, a call is made to (Class/forName <serialized-class-name>).

The actual content serialization is just done as a PersistentMap.

For thawing to work, the thawing JVM process needs to recognize the serialized class name. I suspect that it's Class/forName that's failing, but would like a debug build to confirm.

from nippy.

When switching on AOT

How are you switching on/off AOT with Gradle?

from nippy.

No worries - it can be confusing :)

Here's what I just ran:

$ git clone [email protected]:ptaoussanis/nippy.git
$ vim project.clj
##    ^ Change the version to 2.8.0-SNAPSHOT
$ lein install
$ cd ..
$ git clone [email protected]:Netflix/PigPen.git pigpen2
$ cd pigpen2
$ vim pigpen-core/build.gradle
##    ^ Change the nippy version to 2.8.0-SNAPSHOT
$ ./gradlew clean build install
$ cd ..
$ git clone [email protected]:mbossenbroek/pigpen-graph-test.git
$ cd pigpen-graph-test
$ vim project.clj
##    ^ Change the pigpen version to 0.2.16-SNAPSHOT
$ lein test

The AOT flag for pigpen builds is here: https://github.com/Netflix/PigPen/blob/master/build.gradle#L42

Just remove that line to disable AOT.

Let me know if you need more info.

from nippy.

Were you able to find anything with this?

Thanks,
Matt

from nippy.

Hi Matt, Anders. Sorry about that, have been struggling to find any time to look into this again. Realistically, might not have the time for the next week or two at least (?). Currently travelling + am on some high-priority work.

I'll keep this issue open and let you know if any unexpected opportunities come up. Would suggest one of your workarounds in the meantime if that's acceptable (disabling the use of records, not using AOT).

Sorry I couldn't be of more immediate assistance.

from nippy.

Thanks for the update. If I get some time, I'll try digging into the compiled jar to see what's different.

from nippy.

Sure. I'd start with the info here and maybe try a debug build that'll let you know where along the thaw process the error is popping up.

That was my intention before I ran into issues with the Gradle build.

from nippy.

@ptaoussanis what's the process of producing a debug build? I see the when-debug-mode macro, but not any information on flipping the project into debug mode. (Which I can just change the defn of the macro, but I wasn't sure if there is a "right" way)

from nippy.

@ptaoussanis Anders found that my instructions above were missing a step. You also need to add mavenLocal() to pigpen's build.gradle file:

 allprojects {
     repositories {
+        mavenLocal()
         mavenCentral() // maven { url: 'http://jcenter.bintray.com' }
         maven { url 'http://clojars.org/repo' }
     }

from nippy.

Sorry for the delay replying guys, just concluded my last trip yesterday so schedule's back to normal from today.

what's the process of producing a debug build? I see the when-debug-mode macro, but not any information on flipping the project into debug mode. (Which I can just change the defn of the macro, but I wasn't sure if there is a "right" way)

@aconbere There's a custom debug build up on Clojars at [com.taoensso/nippy "2.8.0-debug1"]. If you could run that for me and get me the output, that should give some good basic starting info and we can go from there?

Cheers! :-)

from nippy.