DistributedException caused by java.lang.NullPointerException about sparkling-water HOT 6 CLOSED

@rahuljain788 from which location are you starting sparkling-shell?

from sparkling-water.

@mdymczyk am running it from the shell using command bin/sparkling-shell --num-executors 3 --executor-memory 2g --master yarn-client

Its is successfully creating sql and sparContext but here it's throwing error

Using this example to test H20 : https://github.com/h2oai/sparkling-water/blob/c454f1f8b2e29c04b49df5bc629dbadce45bc89a/examples/README.md

Maybe am getting this error because am trying to run it in yarn-client mode

from sparkling-water.

@rahuljain788 when you're running it on yarn then the data file has to be accessible to all executors (either on a distributed FS like HDFS or copied to all the executors). You're getting an NPE because you're trying to pass a local path to the parser and that file is accessible only from your driver. Passing the absolute path might work if your Hadoop cluster is running on your laptop.

from sparkling-water.

@mdymczyk Thank you for your reply.

The path i am passing is indeed HDFS directory path and not my driver path.

from sparkling-water.

@rahuljain788 so it's different than val dataFile = "examples/smalldata/allyears2k_headers.csv.gz"? Because that's not a proper HDFS URI, this will look for files on your local drive.

Please post the exact failing scenario.

from sparkling-water.

you are right this is the local path and i was assuming it to be HDFS.

Thank you

from sparkling-water.