I am running these scripts on Debian Stretch.

Path issues

I had to import os, then add this:

    projRootPath = os.path.dirname(os.path.realpath(__file__)) + "/../"
    lines = sc.textFile(projRootPath + "in/word_count.text")

Otherwise I am unable to run the script from different points in the file system.

Python version

It does not work on python 2. To make it work I had to use this environment variable:

export PYSPARK_PYTHON=python3

This information could be added to a README file.

I can provide a PR in case.

These tutorials could be improved with some tips/examples/workarounds on how to integrate python code from data scientists that are used to libraries like pandas and numpy. There is no example on how to include 3rd party libraries (from pip, conda etc.) into PySpark (a Java based environment) and how to make the 2 different APIs work together.

An Ansible script to provision a Vagrantfile could be provided, so a dedicated Vagrant VM (Virtual Machine) could be bootstrapped for local development.

I can submit a PR if needed.

Issue:

Thendash characters in word_count.txt cause an error when following the "Run your first Spark Job" tutorial. There are only two occurences of this character here: "from 1913–74." and here: "near–bankruptcy".

To Recreate:

using spark-2.3.2-bin-hadoop2.7 on Ubuntu18, pyspark/python 2.7, Installed following instructions from lecture 5, go to directory where you cloned python-spark-tutorial and run the following from lecture 6:

spark-submit ./rdd/WordCount.py

The execution halts about halfway through the frequency counter with the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position4: ordinal not in range(128)

Spoiler, it's the dash. I'm not sure whether or not the utf16 dash was intentional, so I'm posting.

Work-Around:

I changed the two ndash characters to "from 1913-74." and "near-bankruptcy", which solved the issue for me. Related stackoverflow thread where someone else ran into a similar problem with python2.7 and used the same solution.