The project just got its own article at Towards Data Science Medium blog! ✨

This project gives you an Apache Spark cluster in standalone mode with a JupyterLab interface built on top of Docker. Learn Apache Spark through its Scala, Python (PySpark) and R (SparkR) API by running the Jupyter notebooks with examples on how to read, process and write data.

curl -LO https://raw.githubusercontent.com/andre-marcos-perez/spark-standalone-cluster-on-docker/master/docker-compose.yml
docker-compose up

• Quick Start
• Tech Stack
• Docker Hub Metrics
• Contributing
• Contributors

• Install Docker and Docker Compose, check infra supported versions

1. Download the source code or clone the repository;
2. Edit the docker compose file with your favorite tech stack version, check apps supported versions;
3. Build the cluster;

docker-compose up

4. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
5. Stop the cluster by typing ctrl+c.

Note: Local build is currently only supported on Linux OS distributions.

1. Download the source code or clone the repository;
2. Move to the build directory;

cd build

3. Edit the build.yml file with your favorite tech stack version;
4. Match those version on the docker compose file;
5. Build the images;

chmod +x build.sh ; ./build.sh

6. Build the cluster;

docker-compose up

7. Run Apache Spark code using the provided Jupyter notebooks with Scala, PySpark and SparkR examples;
8. Stop the cluster by typing ctrl+c.

• Infrastructure

• Jupyter Kernels

• Applications

Apache Spark R API (SparkR) is only supported on version 2.4.4. Full list can be found here.

Image	Size	Downloads
JupyterLab
Spark Master
Spark Worker

We'd love some help. To contribute, please read this file.

Staring us on GitHub is also an awesome way to show your support ⭐

• André Perez - dekoperez - [email protected]