Convert iPython/Jupyter notebook to first-class spark-submit .py file about pipeline HOT 4 CLOSED

ideal flow:

1. commit .ipynb file to github

2. generate .html from .ipynb preserving output images (lots of examples using nbconvert)

3. generate .py from .ipynb - and make sure .py is runnable as a PySpark app (with SparkContext/SparkConf/etc setup)

from pipeline.

here is a good link related to job scheduling: http://docs.qubole.com/en/latest/rest-api/scheduler_api/index.html

from pipeline.

https://github.com/fluxcapacitor/source.ml/blob/master/jupyterhub.ml/notebooks/zz_old/Spark/NotebookJobs/PySpark-Notebook-Job.ipynb

https://github.com/fluxcapacitor/source.ml/blob/master/scheduler.ml/airflow/dags/update_sparksubmit_pyspark.py

https://github.com/fluxcapacitor/source.ml/blob/master/scheduler.ml/airflow/dags/pyspark_pi.py

from pipeline.

complete.

from pipeline.