In our second assignment we'll dive deeper into the Map Reduce paradigm and introduce a new framework for parallel computation: Hadoop MapReduce. This assignment has historically been one of the most challenging for students because of the learning curve associated with this techonology and the programming paradigm. This semester we used the live sessions to give you a structured introduction to the framework and Hadoop syntax before you go on to use it to perform NaiveBayes. If you have not yet completed the wk2Demo and wk3Demonotebooks you should do that now before the homework. We strongly encourage you to start early and to support each other on Slack.

Submit your work by pushing your completed notebook and supplemental files to your assigned student repository on Git. The file structure should look like this:

S18-1-<user>
    --Assignments
        --HW1
        --HW2
           |__hw2_workbook.ipynb
           |__README.md
           |__data
           |__EnronEDA
           |__NaiveBayes
        --HW3
           |__ etc ...
     --HelpfulResources
     etc..

• Do the readings! Many of the tasks and conceptual questions in this assignment are directly explained in the texts (se notes in the workbooks).
• If you are struggling with Hadoop, take a moment to read our Debugging Tips.
• As always we ask that students don't share code because we want your submissions to accurately reflect your own coding style and understanding. However you will likely find it very tempting to share code for debugging purposes. Help each other debug without sharing code. Here's the right way to ask for help from your peers or TAs:
  1. State exactly where in the HW you are working.
  2. List the precise error message you are getting (this should be just a line or two... not the whole Hadoop mess!)
  3. Describe where you've already looked for solutions (this will help us avoid repeating things like "read the Debugging file" or "look in the Hadoop UI task logs")
  4. Describe what you've already tried (this will help us narrow down the issue and give you better suggestions)
  5. Respond to other people's questions even if you aren't sure you know the solution! (an "I wonder if it could be..." or "I saw something that could be similar which I solved by..." goes a long way in a class that has everone working at 150%).
  6. Use Slack's thread feature