Failed to load java libraries about dask-yarn HOT 6 CLOSED

Java tracebacks are terrible - the important bit is hidden in the middle:

Suppressed: java.lang.UnsatisfiedLinkError: /tmp/libcom_anaconda_skein_shaded_netty_tcnative_linux_x86_64218435490097249322.so: /tmp/libcom_anaconda_skein_shaded_netty_tcnative_linux_x86_64218435490097249322.so: failed to map segment from shared object: Operation not permitted

This indicates that /tmp does not have exec privileges on your machine, which prevents the native libraries from being loaded properly. I've seen this before on worker nodes, but never on an edge node. Skein already deals with this when running in a container (we use the container local directory, which is always cleaned up), but we can't make this assumption when running on an edge node (as the driver does).

To get around this, setting the environment variableSKEIN_DRIVER_JAVA_OPTIONS="-Dio.netty.native.workdir=SOME_TEMPORARY_DIRECTORY" (where SOME_TEMPORARY_DIRECTORY is a directory that already exists that can be used as temp storage) should work, but it doesn't seem to in my test setup. I'll either debug this tonight, or on Monday.

from dask-yarn.

So export SKEIN_DRIVER_JAVA_OPTIONS="-Djava.io.tmpdir=SOME_TEMPORARY_DIRECTORY" does work. It's not clear to me why the difference matters (internally we set io.netty.native.workdir when patching on the worker nodes), but this should get you going.

Alternatively you can set this programmatically:

import skein
import dask_yarn

client = skein.Client(java_options="-Djava.io.tmpdir=SOME_TEMPORARY_DIRECTORY")
cluster = dask_yarn.YarnCluster(skein_client=client, ...)

from dask-yarn.

That works for me. I was able to get it to work without this java option once and another user was using it successfully. I don't know why it is intermittent. The permissions on /tmp are listed as drwxrwxrwt. 1300 root root 5128192 Feb 15 20:31 , which is not exec but spark jobs use this directory to launch from the edge nodes.

from dask-yarn.

I can't comment on why things worked intermittently (perhaps admin changing permissions?), but the reason things don't work for skein is that it has native library dependencies which need to be decompressed into a temporary directory. Spark may not have this need, or admin may have configured spark to avoid this issue (I'm not sure if this is an issue for spark).

from dask-yarn.

I am talking to the admins, but I think I found another application that ran into this intermittent /tmp permission issue. Happy to close.

from dask-yarn.

If you figure out why it's intermittent, I'd be curious to know. If this is something we can easily work around I wouldn't be opposed to a fix.

from dask-yarn.