[BUG] - Path error about starlake HOT 15 CLOSED

Hi @cedrickab , I just tried to launch starlake.cmd and it works fine on my laptop.

Can you tell me where do you launch this command from ? You should do it in a cmd not a powershell terminal.

Please add jq in your path as well if you plan to fetch the latest version automatically or else set COMET_VERSION to the version you want. The list of starlake's version are available here : https://central.sonatype.com/artifact/ai.starlake/starlake-spark3_2.12/0.7.2.2.

If you're not able to download again, please add more details:

• terminal you use
• output logs

Best regards

from starlake.

Hello @hayssams
Thank you for the tips, I thought that was about the whitespace,but It still doesn't work :( . It might be a permission issue too, I'll try to run the project on another computer and let you know by then.

from starlake.

Hi @cedrickab, I tried to run starlake with spark 3.4 on my windows laptop and didn't encounter any issue. The last call seems to be different from the one I get with a fresh spark.

CALL {{HOME}}\test-starlake\bin\spark\bin\spark-submit.cmd --driver-java-options "" --class ai.starlake.job.Main {{HOME}}\test-starlake\bin\spark\jars\starlake-spark3_2.12-0.7.2.2-assembly.jar

I don't have -JAR and I was wondering why you get this ?

from starlake.

Re @hayssams @tiboun ,
I did indeed probably modify it when I was trying to solve my initial path problem. But it works now with the option removed and the java_home set before the bin directory.
thanks for your help

from starlake.

hello @tiboun , that's what i did (install jq and use the cmd terminal) but i don't know which path cause this error .

here is my ouput logs :

Current state

• hadoop: OK
• hadoop dll: OK
• hadoop winutils: OK
• spark: OK
• starlake: OK
• spark bq: OK

Launching starlake.

• HADOOP_HOME={{HOME}}\starlake\bin\hadoop
• JAVA_HOME=C:\Program Files\Java\jdk-11\bin
• COMET_ROOT={{HOME}}\starlake
• COMET_ENV=FS
• COMET_FS=file://
• COMET_MAIN=ai.starlake.job.Main
• COMET_VALIDATE_ON_LOAD=false
• SPARK_DRIVER_MEMORY=4G
  Make sure your java home path does not contain space
  The system cannot find the specified path

from starlake.

Hello @cedrickab
Looks like you have a space in your path to the jdk and you forgot to enclose it in quotes.
You may enclose your path by quotes or more simply use the short pathname
To view the short pathnames of your folder type « dir /x » and use « C:\PROGRA~1 » instead of « C:\Program Files » in your path.
Hope this help

from starlake.

Are you getting the exact same error message after removing white spaces ?

from starlake.

yes same error (except JAVA_HOME=C:\PROGRA~1\Java\jdk-11\bin) , but I tested this project on a linux environment and it worked, I specifically encounter this problem only on windows env

from starlake.

Java home should not be set to the bin directory but to the one above it.
See the link below for more details https://docs.oracle.com/cd/E19182-01/821-0917/inst_jdk_javahome_t/index.html#:~:text=To%20set%20JAVA_HOME%2C%20do%20the,Program%20Files%5CJava%5Cjdk1.

from starlake.

Thanks for the docs , I assumed that the java_home corresponded to all the java commands (path on system variables), without the bin directory I get the following error:
Error: Unrecognized option: -JAR

from starlake.

from starlake.

hello,
you will find all the logs here as requested. Seems like the error is caused by the last command spark-submit.cmd
note.txt

from starlake.

Looks like you are using the just released spark3.4 version which break the compatibility of the CLI.
If you use the default Spark version (3.3.1 line 140 of the cmd) does it solve the issue ?
In the meantime we are working on upgrading the windows script for spark 3.4

from starlake.

@cedrickab any chance this was added on your side ?

from starlake.

Great thanks
We're closing the issue in that case

from starlake.