Work in Progress
Tools to install:
- Opensearch
- Kubeflow + MLflow
- MongoDB
- COS
- Superset
- PostgreSQL
- Kafka
- Spark History Server
Example notebooks/programs:
- Access OpenSearch
- Access PostgreSQL
- Access MongoDB
- Spark example with Minio access
- Produce + Consume Kafka
- Superset dashboard from MongoDB, PostgreSQL, Opensearch
- Spark example with push metrics
Other:
- Add list of URLs or way to get URL + user + password for each tool
Create a VM with minimum 8 vCPU, 64GB RAM and 150GB disk.
Go to the folder "env" and run commands in the script files:
Log into the Kubeflow Dashboard using credentails: admin / admin
Run enable spark service account in the admin namespace - enable-spark-ckf.sh
Add PodDefaults for Notebook creation using script: spark-pod-default.sh
Log into the Kubeflow Dashboard using credentails: admin / admin
Add PodDefaults for Notebook creation using script: postgres-pod-default.sh
Connect to the VM using sshuttle and expose networks
sshuttle -r [email protected] 10.0.0.0/8
- Go to Kubeflow Dashboard
- Select Notebook tab
- Create new notebook
- Custom notebook -> Custom image:
docker.io/edeusebio85/charmed-spark-jupyter@sha256:722d027ce6a371d063c8945c808274351ffc49f5368cdf6697394078f46fc29d
- Advanced options -> Configurations: select all from drop-down
- Custom notebook -> Custom image:
- Go to notebook and create new notebook in Jupyter UI.
- Run example spark notebook
- Go to Kubeflow Dashboard
- Select Notebook tab
- Create new notebook
- Use default scipy notebook image
- Advanced options -> Configurations: Allow access to PostgreSQL
- Go to notebook and create new notebook in Jupyter UI.
- Run example postgresql notebook