Apache Spark 2 Data Processing and Real-Time Analytics

Apache Spark is an in-memory, cluster-based data processing system that provides a wide range of functionalities such as big data processing, analytics, machine learning, and more. With this Learning Path, you can take your knowledge of Apache Spark to the next level by learning how to expand Spark's functionality and building your own data flow and machine learning programs on this platform.

You will work with the different modules in Apache Spark, such as interactive querying with Spark SQL, using DataFrames and datasets, implementing streaming analytics with Spark Streaming, and applying machine learning and deep learning techniques on Spark using MLlib and various external tools.

By the end of this elaborately designed Learning Path, you will have all the knowledge you need to master Apache Spark, and build your own big data processing and analytics pipeline quickly and without any hassle.

Apache Spark 2 Data Processing and Real-Time Analytics by Romeo Kienzler, Md. Rezaul Karim, Sridhar Alla, Siamak Amirghodsi, Meenakshi Rajendran, Broderick Hall and Shuen Mei

What you will learn

• Get to grips with all the features of Apache Spark 2.x
• Perform highly optimized real-time big data processing
• Use ML and DL techniques with Spark MLlib and third-party tools
• Analyze structured and unstructured data using SparkSQL and GraphX
• Understand tuning, debugging, and monitoring of big data applications
• Build scalable and fault-tolerant streaming applications
• Develop scalable recommendation engines

Hardware Requirements

For an optimal student experience, we recommend the following hardware configuration:

• Processor: Intel Core i5 or equivalent
• Memory: 4 GB RAM
• Storage: 35 GB available space

Software Requirements

You'll also need the following software installed in advance:

• OS: Windows 7 SP1 64-bit, Windows 8.1 64-bit or Windows 10 64-bit
• Browser: Google Chrome, Latest Version
• VirtualBox 5.1.22 or greater
• Hortonworks HDP Sandbox V2.6 or above
• Spark 2.0.0 (or higher)
• Hadoop 2.7 (or higher)
• Java (JDK and JRE) 1.7+/1.8+
• Scala 2.11.x (or higher)
• Python 2.7+/3.4+
• R 3.1+ and Rstudio 1.0.143 (or higher)
• Eclipse Mars or Luna
• Maven Eclipse plugin (2.9 or higher)
• Maven compiler plugin for Eclipse (2.3.2 or higher)
• Maven assembly plugin for Eclipse (2.4.1 or higher)