Factor out all common spark/hadoop properties about transmogrifai HOT 8 OPEN

Hey @gerashegalov
I'm guessing this issue is still concern since helloworld/gradle/spark.gradle and templates/simple/spark.gradle are duplicates of gradle/spark.gradle but are still being tracked by git.

In order to keep a single spark.gradle file, can we simply replace the spark.gradle paths in build.gradle to reference spark.gradle as ../gradle/spark.gradle ?

PS. I'm fairly new to the project. Pardon me if I'm missing something. 😅

from transmogrifai.

Hi @py-ranoid, thanks for looking into this issue. It makes sense, however if possible we should strive to use absolute paths built from project properties (to avoid dealing with relative path attacks with symlinks etc).

from transmogrifai.

How about keeping keeping only spark.gradle in the repository but copying it to helloworld/gradle/ and templates/simple/ during installation ?

from transmogrifai.

@tovbinm @gerashegalov Could you suggest a solution?

1. Removing helloworld/gradle/spark.gradle and templates/simple/spark.gradle and referring to gradle/spark.gradle using relative paths
2. Keeping only spark.gradle but copying it to helloworld/gradle/ and templates/simple/ during installation

from transmogrifai.

since helloworld is a source-controlled directory rather than installed than 1 seems better, (and I think you should be able to construct an absolute path.

from transmogrifai.

@gerashegalov In that case, can I replace
apply from: 'gradle/spark.gradle
with
apply from: "${rootProject.projectDir}/../gradle/spark.gradle"
in helloworld/build.gradle ?
Would this still be vulnerable to a relative path attack ?

Also, I noticed that the following are duplicates too.

1. helloworld/gradle/scalastyle-config.xml and gradle/scalastyle-config.xml.
2. helloworld/gradle/wrapper/* and gradle/wrapper/*

Would you suggest factoring these out as well ?

from transmogrifai.

@gerashegalov @tovbinm Thoughts?

from transmogrifai.

Hi @py-ranoid I suggest you try it out and don't hesitate to submit a PR. We can discuss it more concretely on the PR. It does not have to be perfect, just something to iterate on. The preference is to avoid '..'

from transmogrifai.