by Frank Kane | Released September 2016 | ISBN: 9781787129849

Get to grips with the fundamentals of Apache Spark for real-time Big Data processingAbout This VideoUnderstand the fundamentals of Scala and the Apache Spark ecosystemHandle large streams of data with Spark Streaming and perform Machine Learning in real time with Spark MLlibComprehensive tutorial packed with practical examples to help you develop real-world Big Data applications with Spark with ScalaIn DetailWith the rise in popularity of the term 'Big Data', there is an increasing need to process large amounts of data in real-time, with maximum efficiency. This has led to Apache Spark gaining popularity in the Big Data market very quickly. The Spark ecosystem allows you to process large streams of data in real-time. As Spark is built on Scala, knowledge of both has become vital for data scientists and data analysts today. This comprehensive 7 hour course will empower you to build efficient Spark applications to fulfill your Big Data needs.You will start with quickly understanding the basics of Scala and proceed to set up the development environment for Apache Spark and Scala for Big Data processing. You will understand the different modules of Spark like Spark SQL, Spark Streaming and GraphX, along with when and how to use them. While doing so, you will build practical, real-world Spark applications in Scala and see how you can deploy them on the cloud. You will also learn how to perform machine learning in real time using Spark's MLlib module. Finally, you will learn how to run Spark on Hadoop clusters along with best practices and troubleshooting techniques.With over 20 carefully selected examples and abundant explanation to explain even the most difficult concepts, this course will ensure your success in taming your Big Data challenges using Spark with Scala. Show and hide more Publisher resources Download Example Code

1. Chapter 1 : Getting Started
   • Introduction and Getting Set Up 00:14:30
   • [Activity] Create a Histogram of Real Movie Ratings with Spark! 00:12:58
2. Chapter 2 : Scala Crash Course
   • [Activity] Scala Basics, Part 1 00:12:52
   • [Exercise] Scala Basics, Part 2 00:09:41
   • [Exercise] Flow Control in Scala 00:07:18
   • [Exercise] Functions in Scala 00:08:47
   • [Exercise] Data Structures in Scala 00:16:38
3. Chapter 3 : Spark Basics and Simple Examples
   • Introduction to Spark 00:08:40
   • The Resilient Distributed Dataset 00:11:04
   • Ratings Histogram Walkthrough 00:07:33
   • Spark Internals 00:04:42
   • Key/Value RDDs and the Average Friends by Age example 00:12:21
   • [Activity] Running the Average Friends by Age Example 00:07:58
   • Filtering RDDs and the Minimum Temperature by Location Example 00:06:43
   • [Activity] Running the Minimum Temperature Example and Modifying It for Maximum Temperature 00:10:11
   • [Activity] Counting Word Occurrences Using flatmap() 00:08:59
   • [Activity] Improving the Word Count Script with Regular Expressions 00:06:42
   • [Activity] Sorting the Word Count Results 00:08:11
   • [Exercise] Finding the Total Amount Spent by Customer 00:03:38
   • [Exercise] Check your Results, and Sort Them by Total Amount Spent 00:04:26
   • Check Your Results and Implementation against Mine 00:03:26
4. Chapter 4 : Advanced Examples of Spark Programs
   • [Activity] Find the Most Popular Movie 00:04:30
   • [Activity] Use Broadcast Variables to Display Movie Names 00:08:53
   • [Activity] Find the Most Popular Superhero in a Social Graph 00:14:10
   • Superhero Degrees of Separation – Introducing Breadth-First Search 00:06:53
   • Superhero Degrees of Separation – Accumulators and Implementing BFS in Spark 00:05:54
   • Superhero Degrees of Separation – Review the Code, and Run It! 00:10:42
   • Item-Based Collaborative Filtering in Spark, cache(), and persist() 00:08:17
   • [Activity] Running the Similar Movies Script using Spark's Cluster Manager 00:14:13
   • [Exercise] Improve the Quality of Similar Movies 00:02:42
5. Chapter 5 : Running Spark on a Cluster
   • [Activity] Using spark-submit to Run Spark Driver Scripts 00:06:59
   • [Activity] Packaging Driver Scripts with SBT 00:14:07
   • Introducing Amazon Elastic MapReduce 00:07:12
   • Creating Similar Movies from One Million Ratings on EMR 00:12:47
   • Partitioning 00:05:07
   • Best Practices for Running on a Cluster 00:05:31
   • Troubleshooting and Managing Dependencies 00:09:08
6. Chapter 6 : SparkSQL, DataFrames, and DataSets
   • Introduction to SparkSQL 00:07:08
   • [Activity] Using SparkSQL 00:07:01
   • [Activity] Using DataFrames and DataSets 00:06:38
   • [Activity] Using DataSetsInstead of RDDs 00:07:24
7. Chapter 7 : Machine Learning with MLLib
   • Introducing MLLib 00:07:38
   • [Activity] Using MLLib to Produce Movie Recommendations 00:07:23
   • [Activity] Linear Regression with MLLib 00:11:37
   • [Activity] Using DataFrames with MLLib 00:10:05
8. Chapter 8 : Intro to Spark Streaming
   • Spark Streaming Overview 00:09:54
   • [Activity] Set Up a Twitter Developer Account, and Stream Tweets 00:12:13
   • Structured Streaming 00:04:01
9. Chapter 9 : Intro to GraphX
   • GraphX, Pregel, and breadth-first search with Pregel. 00:10:39
   • [Activity] Superhero Degrees of Separation using GraphX 00:08:59
10. Chapter 10 : You Made It! Where to Go from Here?
    • Learning More, and Career Tips 00:04:15
Show and hide more