[BUG] - Databricks absolute path must be written differently about starlake HOT 6 CLOSED

Thank you for reporting this @ametwalli1
When the hostname is not present, the correct syntax for xdfs compatible filesystem should be xdfs:///.
Could try using dbfs:///mnt/datasets_area/dbfs:/mnt/pending_area instead and let me know how this work for you.

from starlake.

Thank you very much @hayssams, using dbfs:///mnt/pending_area as SL_AREA_PENDING is working fine !

However, I have an other trouble here.

When doing the import with this setting, starlake pushes all the source files without filtering by domain. This was done by default in the DatasetArea.scala path function that would add the /$domain variable at the end of the path. But it doesn't do it anymore.

So in my situation if I wanted to create the domain directories in my pending area, I thought using dbfs:///mnt/pending_area/$domain would work in my cluster environment. It didn't change anything ($domain was ignored), so I tried dbfs:///mnt/pending_area/{domain} instead and it created a {domain} directory.

Do you know how can I use variables in my cluster environment ?

from starlake.

Could you try {{domain}} instead and let me know

from starlake.

dbfs:///mnt/pending_area/{{domain}} creates a {{domain}} directory also.
The only way to use variables is done by using /$domain but I guess that you have some regex security check and it is ignored.

The better way to solved this would be to add the /$domain at the end of the area in DatasetArea.scala line 50.
Or maybe there is another way to introduce variables.

from starlake.

Can We setup a call 5:30 PM?

from starlake.

Sure ! Here is my email : [email protected]

from starlake.