Hello! I have no idea what this pipe is, but with it, criu fails. If you

sudo rm /var/lib/sss/pipes/nss

it, it will start work... a bit of magic for me. SHoudl be fixed, or documented or IDK....

Experimenting with CRIU for a relatively complex application (in my case Open Liberty https://github.com/OpenLiberty/open-liberty) I have my doubts on the hook for CheckpointRestore.

My main doubt is in the requirement to serialize on the dump operation and deserialize on the restore operation. If on a dump the complete process and all of its state is saved in the image dump then why do we also need to save the state of the hooks with serialization separately? Would we not already have the hooks objects available to us once we restore such that we can call the restore side straight away when the process is resumed?

For Open Liberty I likely will need to introduce my own hook SPI/API [1] to give us the flexibility to have subsystems within Liberty to participate in the prepare/restore operations anyways, but my current thinking is to not have the hooks get serialized at all. The general flow is

1. Gather up all the hook implementations into a snapshot
2. Invoke prepare on all the hooks
3. Perform the CRIU dump (I currently modified the JavaCriuJar to terminate the process on dump)
4. Use criu command to restore, I would like to avoid invoking the JVM to restore to avoid overhead of firing up a Java process
5. The process picks up from when it was frozen, the thread that invoked the dump picks right up and has a reference to the snapshot of hooks used during the prepare phase that thread then immediately calls restore on the snapshot of hooks that it called prepare on.
6. Each hook will have its own state that is persisted with the dumped image. On restore, from the hook POV, it thinks it is the same object and will be able to restore anything it was responsible for in prepare.

My reason for opening this issue is to discuss the strategy for how the hooks work in JavaCriuJar to make sure I am not overlooking something in my strategy that I described above for Open Liberty.

If my understanding is correct we could decide to push a similar strategy down into the JavaCriuJar library. The difficulty I will have with that is that I need the ability to separate the CheckPointRestore/Hook API from the implementation such that the implementation can be installed on top while lower levels of Open Liberty can participate and implement the restore/prepare hooks without requiring the actual implementation of CheckPointRestore be available at runtime. One solution to that is to separate the JavaCriuJar into to JARs: 1) for API 2) for implementation. On the other hand I am also fine with Open Liberty having its own hook API and only using JavaCriuJar for invoking the criu lib calls to perform the dump.

[1] https://github.com/tjwatson/open-liberty/blob/criu/dev/com.ibm.ws.kernel.boot.core/src/io/openliberty/checkpoint/spi/SnapshotHook.java

Currently save the world only takes a single argument to specify the directory to save the dumped image. I imagine we will need more flexibility in configuring the calls to save the world. For example, my usage scenario will want to terminate the process as soon as the dump is made. This will ensure the process does not make modifications post dump that could inhibit a successful restore of the process later.

I would suggest adding a simple boolean option for this one option (leaveRunning). But perhaps something more advanced is needed to hold the various CRIU options.

Even with #26 , the autotools are unabel to crosscompileproeprly. They cannot find -lcriu
As direct crosscompilation works (see #27). But maybe already that is wrong. See:

$ ./configure --build x86_64-pc-linux-gnu --host aarch64-linux-gnu  CC=aarch64-linux-gnu-gcc LDFLAGS="-static" CFLAGS="--sysroot=/var/lib/mock/fedora-35-aarch64/root/"
checking build system type... x86_64-pc-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking target system type... aarch64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for aarch64-linux-gnu-strip... aarch64-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to create a pax tar archive... gnutar
checking whether to enable maintainer-specific portions of Makefiles... yes
checking for aarch64-linux-gnu-gcc... aarch64-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether aarch64-linux-gnu-gcc accepts -g... yes
checking for aarch64-linux-gnu-gcc option to accept ISO C89... none needed
checking whether aarch64-linux-gnu-gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of aarch64-linux-gnu-gcc... none
checking for a JDK home directory... not specified
checking for /usr/lib/jvm/java-openjdk... found
checking for the JDK version... 11
checking for aarch64-linux-gnu-rmdir... no
checking for rmdir... /usr/bin/rmdir
configure: WARNING: using cross tools not prefixed with host triplet
checking if /usr/bin/rmdir supports --ignore-fail-on-non-empty... yes
checking for aarch64-linux-gnu-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for CRIU... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating autotools.pom.properties
config.status: creating MANIFEST.MF
config.status: executing depfiles commands
$  make
 cd . && /bin/sh /home/jvanek/git/jigawatts-ja/missing automake-1.16 --foreign Makefile
 cd . && /bin/sh ./config.status Makefile 
config.status: creating Makefile
/usr/lib/jvm/java-openjdk/bin/javac -g  \
      -d /home/jvanek/git/jigawatts-ja/build -h /home/jvanek/git/jigawatts-ja/build/include \
      -sourcepath /home/jvanek/git/jigawatts-ja/src/main/java \
      @/home/jvanek/git/jigawatts-ja/build/source-files.txt ;
mkdir -p /home/jvanek/git/jigawatts-ja/stamps
touch /home/jvanek/git/jigawatts-ja/stamps/classes.stamp
cd /home/jvanek/git/jigawatts-ja/build && \
  /usr/bin/mkdir -p META-INF/maven/com.redhat/jigawatts && \
  cp /home/jvanek/git/jigawatts-ja/pom.xml META-INF/maven/com.redhat/jigawatts && \
  cp /home/jvanek/git/jigawatts-ja/autotools.pom.properties META-INF/maven/com.redhat/jigawatts/pom.properties && \
  /usr/lib/jvm/java-openjdk/bin/jar -cvfm jigawatts.jar /home/jvanek/git/jigawatts-ja/MANIFEST.MF ./com META-INF
added manifest
adding: com/(in = 0) (out= 0)(stored 0%)
adding: com/redhat/(in = 0) (out= 0)(stored 0%)
adding: com/redhat/jigawatts/(in = 0) (out= 0)(stored 0%)
adding: com/redhat/jigawatts/Jigawatts.class(in = 5250) (out= 2452)(deflated 53%)
adding: com/redhat/jigawatts/Hook.class(in = 370) (out= 249)(deflated 32%)
adding: com/redhat/jigawatts/LibraryLoader.class(in = 8169) (out= 4031)(deflated 50%)
ignoring entry META-INF/
adding: META-INF/maven/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.redhat/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.redhat/jigawatts/(in = 0) (out= 0)(stored 0%)
adding: META-INF/maven/com.redhat/jigawatts/pom.properties(in = 61) (out= 62)(deflated -1%)
adding: META-INF/maven/com.redhat/jigawatts/pom.xml(in = 7231) (out= 1987)(deflated 72%)
aarch64-linux-gnu-gcc -static /home/jvanek/git/jigawatts-ja/build/com_redhat_jigawatts_Jigawatts.o -shared -o /home/jvanek/git/jigawatts-ja/build/libJigawatts.so -lcriu  --sysroot=/var/lib/mock/fedora-35-aarch64/root/
/usr/bin/aarch64-linux-gnu-ld: cannot find -lcriu
collect2: error: ld returned 1 exit status
make: *** [Makefile:689: /home/jvanek/git/jigawatts-ja/build/libJigawatts.so] Error 1

The autotools are just better for native build
and maven is better for java build

My idea is to make both unverses to coopearte and reduce duplicated code.

• The maven usecases is simple

  • the exec plugin will call configure and make exact target to compile the binary (and cross compiled binaries in future) instead of direct gcc call
  • the maven will build java, and use autotooled natives

• auto tools use case

  • autotools build bianry
  • from autotools maven is called in profile, which do not build binary
  • as autotools saves the binary, the binary is correctly included in jar

Downstream will need to patch a bit to get rid of bundled binary and avoid cross compilation.

do I miss something? Ok to try?

Both pom and configure reffers final jar as jigawatt.jar Maybe it shoudl be jigawatts.jar? This is not an issue, just verifying. Project is jigawatts, library is libJigawatts.so....

I have the following installed:

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic

adoptopenjdk-8-openj9xl/bionic,now 8u282-b08.openj9-0.24.0-3 amd64 [installed]

criu/bionic,now 3.6-2 amd64 [installed]

The classes and native library compile fine. In order to find the libcriu.so library I had to add /usr/lib/x86_64-linux-gnu to the LD_LIBRARY_PATH. Not sure why the library was not automatically available from the install of criu to Java. After that the tests still fail but now with this error:

[ERROR] /usr/lib/jvm/adoptopenjdk-8-openj9xl-amd64/jre/bin/java: symbol lookup error: /tmp/libCheckpointRestore.so: undefined symbol: criu_init_opts

What version of criu must be installed for this to work? (warning, I am very rusty on developing and building JNI libraries so this could just be a simple error on my part.