Cloud native risk analytics application stack leveraging

• Risk Calc Engine built using Akka Cluster as detailed here
• Cloud Native Object Store in Apache Hadoop Ozone
• Risk Aggregation and Analytics Engine using Apache Spark on Ozone
• Analytics Notebooks using Apache Zeppelin

all running on K8s equipped with Prometheus for metrics and monitoring.

see scripts/setup/install-utils-mac.sh

1. Build the following project distributions and Docker images from source:

   a. Apache Hadoop Ozone 0.4 [steps]

   [OPTIONAL]

   b. Apache Spark 3.0 [steps] | [prebuilt image - amolthacker/spark:3.0.0]

   c. Apache Zeppelin 0.9 [steps] | [prebuilt image - amolthacker/zeppelin:0.9.0-snapshot]

   d. spark-ozone image [steps] | [amolthacker/spark-ozone:3.0.0]

2. Install the built hadoop-ozone pom and jars in local maven repo

   $ ./scripts/common/setup-ozlib-mvn.sh
   

3. Build the compute-engine code and Docker image

   $ ./rebuild-app.sh
   

4. Spin up a Minikube cluster (GKE soon) and deploy Hadoop Ozone and the compute-engine app in K8s

   $ ./redeploy-minikube.sh
   

   The script automates the following steps:

   a. Spins up minikube K8s cluster

   b. Deploys Helm

   c. Deploys Prometheus and Istio

   d. Deploys Hadoop Ozone and initializes Ozone S3 bucket valuations

   e. Deploys the compute-engine app

   f. Deploys Zeppelin

   As the script executes, one should see the following:

   

5. Once the K8s infrastructure is up, one can forward service ports for access through local browser

   $ sudo kubefwd svc -n default
   

   

   One can now access the services as follows:

   Service	URL
   Ozone Manager	http://om-public:9874
   Ozone S3 Bucket Explorer	http://s3g-public:9878/valuations?browser=true
   Zeppelin	http://zeppelin-server/#/

   and K8s Dashboard

   $ minikube dashboard
   

   

6. One can submit risk compute jobs and have the comput-engine calculate risk metrics and persist them in Ozone

   $ ./scripts/submit-compute-batch
   Usage: ./scripts/common/submit-compute-batch 
       -t, --k8s-type    <mkube|gke|eks|aks>   : K8s Type
       -n, --num-jobs    <num_jobs>            : No.of Valuation Jobs
       -s, --compute-svc <compute_svc_address> : Compute Service DNS/PIP
       -h, --help                              : Help
   

   eg:

   $ ./scripts/common/submit-compute-batch -t mkube -n 10 -s 192.168.99.100:30309
   

   

7. Launch Zeppelin in the browser and import notebook notebook/hwxpe.json

8. Run the paragraphs for analytics on the computed risk metrics using Spark on Ozone

   

• Thanks to Anu Engineer, Marton Elek, Bharat Viswanadham for guidance

• Apache Hadoop Ozone

• Apache Spark on Kubernetes

• Apache Zeppelin on Kubernetes