EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.
Library Documentation https://tharwaninitin.github.io/etlflow/site/docs
You can use this library in different ways mentioned below.
- Core Module:
Using this module you can use features of Step API into your project. - Spark Module (spark steps):
Using this addon module along with core you can use Apache Spark steps into your project. - GCP Module (spark steps):
Using this addon module along with core you can use GCP steps into your project. - Server Module:
Using this module along with job you can use additional features of Server API into your project.
| Module Name | Scala 2.12 | Scala 2.13 | Scala 3.1 |
|---|---|---|---|
| Core | ✅ | ✅ | ✅ |
| Server | ✅ | ✅ | ❌ |
| Module Name | Scala 2.12 | Scala 2.13 | Scala 3.1 |
|---|---|---|---|
| Json | ✅ | ✅ | ✅ |
| Module Name | Scala 2.12 | Scala 2.13 | Scala 3.1 |
|---|---|---|---|
| Db | ✅ | ✅ | ✅ |
| Http | ✅ | ✅ | ✅ |
| Cloud | ✅ | ✅ | ✅ |
| ✅ | ✅ | ✅ | |
| Aws | ✅ | ✅ | ✅ |
| Gcp | ✅ | ✅ | ✅ |
| Redis | ✅ | ✅ | ❌ |
| Spark | ✅ | ✅ | ❌ |
This project is compiled with scala versions 2.12.15, 2.13.7, 3.1.0 Available via maven central. Add the latest release as a dependency to your project
SBT
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-server" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-cloud" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "x.x.x"
Maven
<dependency>
<groupId>com.github.tharwaninitin</groupId>
<artifactId>etlflow-core_2.12</artifactId>
<version>x.x.x</version>
</dependency>
Please feel free to add issues to report any bugs or to propose new features.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent