goldshtn / spark-workshop Goto Github PK

Simply run pyspark --packages com.databricks:spark-csv_2.11:1.4.0 to have it detect and download the dependency automatically. Also update the slides to mention Spark Packages and this specific spark-csv package.

The docs for spark-csv are quite good as well. Should explain that the SQLContext.read method has support for pluggable format, and maybe also mention that there's roundtripping support with SQLContext.write.

Ideally a simple bash script that install dependencies, downloads and extracts Spark, Zeppelin, the course data files, and sets up everything so that workshop attendees can start being productive right away. If necessary, can assume Ubuntu 14.04 as the base image.

NOTE: Some Python labs depend on external modules that need to be brought in through easy_install.

topFive = sorted(enumerate(similarities.collect()), key=lambda (k, v): -v)[0:5]
Near this particula line it shows pyspark 5063 error

There's no reason to suffer through all the aggregations when DataFrames are becoming more and more prevalent. At least point out that the DataFrame version is available, and have attendees experiment with it -- even if it's before we had a chance to teach SparkSQL.

For each lab, add side-by-side Scala instructions. For the first couple of labs, attendees should be using spark-shell directly, and there should be at least one example of taking a stand-alone .scala file and submitting it. The rest of the labs can use Zeppelin.

• Lab 0
• Lab 1
• Lab 2
• Lab 3
• Lab 4
• Lab 5
• Lab 6
• Lab 7

Rework the course structure so that it puts an emphasis on the DataFrame / DataSet API first, and covers RDD as an implementation detail.

The SparkSQL examples can be adjusted to also cover the DataFrame API without writing SQL statements. The same applies to UDF. For example:

compensation = udf(lambda delay: 0 if delay < 15 else delay * 10)
df.select(..., compensation(df['ArrDelay'])).groupBy("carrier")...

Add an intro that highlights key concepts in ML and a few comments on some algorithms that Spark has to offer. Then we can do labs on:

• TF-IDF example -- plagiarism detection
• Binary classification -- census data, predict >$50K income using Naive Bayes or Decision Tree (Forest)
• Clustering -- TBD
• Collaborative filtering -- MovieLens dataset, adapted from Databricks training