Description

Se discussion #664

Description

This ticket lists all validity checks that should be made on the project before executing it.

• Check that connection references are valid and reference existing connections
• Make sure that disposition and merge options are compatible
• Make sure that every expected input files are defined depending on the task launched. Eg: an error is thrown in Attribute (Should never happen: empty list of attributes) but is related to the function call ai.starlake.schema.model.Attribute.sparkType(Attribute.scala:249)) and doesn't help the user to understand what is its mistakes and how to resolve it.

Description

This feature concerns data loading jobs with a huge number of files to ingest (1000+) with GROUPED mode (true). In such case, a major part of time is passed on moving files between zones (landing to pending, ingesting to archive). This is due to the fact that files are moved sequentially ( simple foreach). In some cases, 30 min are spent on moving files whereas the actual ingestion process takes around 5 min.

Expected behavior

Parallelize file handling when possible.

Possible implementation

using scala palallel collections
Example in the ingest() method on IngestionWorkflow
ingestingPath.par.foreach { // back up file in archive}
instead of
ingestingPath..foreach { // back up file in archive}

Complexity estimation

T-Shirt Size: S

Description

When I set a custom audit sink name using COMET_AUDIT_SINK_NAME, the audit table is missing.
I am running load jobs on Google BigQuery

Expected behavior

The job should insert in the two audit tables audit and rejected

Current Behavior

audit table is no longer created/populated. Check the attached screenshot for a more visual illustration of the current behaviour

Steps to reproduce

Steps to reproduce the behavior:

1. Run a Load into a BigQuery table
2. Make sure you have both tables (audit and rejected) with data in them
3. Set the variable COMET_AUDIT_SINK_NAME to custom_audit
4. Make sure you have created the custom_audit dataset in you BigQuery project
5. Run the same Load as in (1)
6. See that the audit table is missing the custom_audit dataset

Context

GCP - BigQuery -

Life cycle

Possible implementation

TBD

Complexity estimation

TBD

Description

Regression since 0.2.7 - we can't use DML (DELETE, MERGE, ...) statements in autotaskjob because of the added parentheses at this line

This change was added in 9a7b0f2.

Expected behavior

Excepted same behavior as 0.2.7

Description

Use Spark Job Scheduler to parallelize ingestion of multiple different schemas

Solution proposition

A clear and concise description of what you want to happen.
Use Spark Pool and execute conurrently jobs on file arrival.

Require the rewrite of https://github.com/starlake-ai/starlake/blob/master/src/main/scala/ai/starlake/workflow/IngestionWorkflow.scala#L216

Complexity estimation

L.

Description

I was looking at my audit table and something was wrong on the countAccepted column on my json files :

Expected behavior

On my last row I have 5 row processed, 5 accepted and 3 rejected. When looking at my data I actually just have 2 accepted.
So countAccepted has a problem

Possible implementation

When looking in the source code I think I found the problem. In ai/starlake/job/ingest/JsonIngestionJob.scala, line 164 you return (rejectedDS, toValidate) instead of (rejectedDS, validationResult). I don't understand all your code architecture but it seems to be the problem.

Thank you in advance.

Description

Many Database including Snowflake and other relational databases require the table schema to be created upfront.
Even when addressing BigQuery, some users would like the schema to be created upfront too even though it is not required.

This feature would add a new command that would create the required DDL. We call this command infer-ddl.
Databases usually evolve in time. This command should also address the issue of altering the table schema if it already exist in the database.

// Use mapping specifict BigQuery and generate create/alter table statements
starlake.sh infer-ddl --datawarehouse bigquery --connection bq

// Use mapping default mapping and generate create/alter table statements
starlake.sh infer-ddl --connection bq
starlake.sh infer-ddl --datawarehouse sql --connection bq --output ./sql

// use default mapping and generate and apply create/alter table statements
starlake.sh infer-ddl --datawarehouse sql --connection bq --apply

// use default mapping and generate and apply create/alter table statements using the provided moustache template
starlake.sh infer-ddl --datawarehouse sql --template my template.mustache --connection bq --apply

Solution proposition

We augment the types.yml file to include a mapping section with case specific to each data warehouse when necessary :

type:
  name: int
  pimitiveType: long
  mapping:
    bigquery: LONG # Specific to BigQuery
    synapse: INTEGER # Specific to Synapse
    sql: NUMERIC # used by default when no specific defined for the target data warehouse

From the schema we generate the final DDL statements and apply them gif requested using the provided connection

Complexity estimation

T-Shirt Size: M

Description

On startup, starlake.sh check the last version on maven repository and parse the result using "jq"

This fails if jq is not already installed.

Steps to reproduce

Steps to reproduce the behavior:

1. Launch a fresh WSL on Windows
2. Launch starlake.sh for the first time

Complexity estimation

XS

Description

Hello, in our process we use xlsx files to generate our schemas, and then from these schemas we generate SQL+ extraction scripts.
We are facing two limiting cases in this process (sometimes both at the same time):

• the source names are not really descriptive and we would like to have a meaningful name for our table once ingested into BQ.
• the source names are too long for Excel sheet name (limited to 31 characters).

Solution proposition

In order to keep the original source name to generate the correct SQL extraction scripts from the schemas and also change the table name in BQ during the ingestion process. I propose to add two fields in the xlsx files:

• rename_source: optional value to replace the name and thus bypass the excel sheet name limit.
• rename_target: optional value to replace the name in the ingestion process.

Considered alternatives

I went through the code base and documentation but couldn't find any information to solve my problem. I'm also not sure if its technically possible in the current implementation, so feel free to correct me or suggest a better solution and I'll be happy to take it.

Description

Ingestion job is tested but not all details are currently tested such as the result of audit output.

This issue is related to #498.

Description

I try to distribute my ingestion areas on several containers in an azure data lake mounted on a dbfs path on the databricks platform (ex: /mnt/starlake).
For that I specified in environment variable SL_AREA_PENDING=dbfs://mnt/pending_area for example for the pending area.
Problem, when you specify an absolute path on databricks with the filesystem included in it, you have to write it with a single '/'.

%fs ls dbfs://mnt/starlake gives us this error:
IllegalArgumentException: Hostname not allowed in dbfs uri. Please use 'dbfs:/' instead of 'dbfs://' in uri: dbfs://mnt/starlake

While %fs ls dbfs:/mnt/starlake works correctly.

However I can't change the absolute path of SL_AREA_PENDING=dbfs:/mnt/pending_area because according to your code, in DatasetArea.scala line 49, a path is recognized as absolute only if it contains "://".
So if I include the variable this way, starlake will recognize this path as relative and concatenate me SL_DATASETS with SL_AREA_PENDING : "dbfs:/mnt/datasets_area/dbfs:/mnt/pending_area"
This will cause an error at runtime.

So it might be useful to change this condition to contains(":/") or just create a specific condition for databricks.

Description

I'm trying to run starlake on Databricks, and it works fine when using import and watch command lines.
However, I'm unable to use infer-schema awhich is pretty useful. Bottstrap command has some problems too but it is not really important for the moment.
For now I'm using manually edited yaml domain files but I would like to get them quicker.

Expected behavior

Current Behavior

I get this error in my spark logs :

23/05/04 08:37:24 ERROR IngestionWorkflow: java.io.FileNotFoundException: dbfs:/mnt/sample-data/sales/customers-2018-01-01.psv (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at scala.io.Source$.fromFile(Source.scala:94)
at scala.io.Source$.fromFile(Source.scala:79)
at scala.io.Source$.fromFile(Source.scala:57)
at ai.starlake.job.infer.InferSchemaJob.$anonfun$infer$1(InferSchemaJob.scala:217)

I looked at the code and I didn't find any visible error so it may come from how you manage dbfs.
I tried to use {{incoming_path}} a variable that i setted in my env.comet.yml but it didn't change anything.
My path is correct and I don't have any authorization issues so I'm asking here.

Steps to reproduce

Steps to reproduce the behavior:

1. Create a job with your assembly jar file, (jackson yaml jar in already included in my cluster), and put the arguments as shown in my screenshot.
2. Mount an azure blob storage or adls gen2 with the actual jar files. I mounted two different containers but the error occurs even when I put an incoming directory in my starlake root project.
3. Execute the job.
4. When failing go to the spark/log and look at the Log4J outputs

Description

Settings and their corresponding env variables are not yet documented.
We will add documentation inside the reference-*.conf files and generate the HTML page based on the comments.
Documentation should start with a # and document the top level properties

Complexity estimation

T-Shirt Size: M

Description

The bug #516 has not been unit tested. This should be done

Complexity estimation

T-Shirt Size: XS

Description

Where is documentation missing? What could be added or simplified?
What's the type of documentation requested? (scaladoc, technical on readthedocs, user guide, ...)
Please provide guidelines, package or classes if appropriate.

Complexity estimation

T-Shirt Size: XS, S, M, L. If you would recommend XL or more, we advise you to discuss with the community how to split this issue.
Don't forget to tag this issue as a 'good first issue' if you think it is appropriate!

Description

the error occur when the cmd command should run spark-submit

Current Behavior

Launch starlake.cmd for the first time

See error complete

Le chemin d’accès spécifié est introuvable.

Complexity estimation

S