by Tianhui Michael Li | Released November 2016 | ISBN: 9781491974438

Spark is one of today's most popular distributed computation engines for processing and analyzing big data. This course provides data engineers, data scientist and data analysts interested in exploring the technology of data streaming with practical experience in using Spark. You'll learn about the Spark Structured Streaming API, the powerful Catalyst query optimizer, the Tungsten execution engine, and more in this hands-on course where you'll build small several applications that leverage all the aspects of Spark 2.0. While not a requirement, the course works best for those with some Scala experience.Understand the main features of Spark and its advantages over existing systemsLearn the basics of parallelism, streaming computation, and Spark streamingExplore the distinctions between Spark Structured Streaming and legacy DStream APIsUnderstand how to write to and use the Spark Structured Streaming APILearn about the new Catalyst query optimizer and the Tungsten execution engineDiscover how Scala and Spark Structured Streaming simplify distributed streaming tasksGain hands-on experience building applications using Spark 2.0Michael Li is the founder of The Data Incubator, which provides big data corporate training and a selective eight-week fellowship for PhDs transitioning into industry. Previously, he worked as a data scientist, software engineer, and researcher at Foursquare, Google, Andreessen Horowitz, J.P. Morgan, and NASA. He is a regular contributor to VentureBeat, The Next Web, and Harvard Business Review. Michael earned his Ph.D. at Princeton and was a Marshall Scholar in Cambridge. Show and hide more Publisher resources Download Example Code

1. Overview
   • Overview 00:02:06
2. Spark Datasets and Structured Streaming
   • Spark Overview 00:02:11
   • Spark Wordcount Using RDD Example 00:05:01
   • Spark Wordcount Using Scala Example 00:02:37
   • Spark and Datasets 00:01:56
   • Spark Wordcount Using Datasets Example 00:03:06
   • Joining Data Using Spark Datasets 00:03:32
   • Structured Streaming Overview 00:03:18
   • Spark Structured Streaming Wordcount Example 00:03:20
3. Spark Structured Streaming
   • Spark Structured Streaming 00:00:46
   • Netcat Socket Structured Streaming Example 00:02:27
   • Socket Structured Streaming Example 00:02:55
   • Spark Structured Streaming Parsing Data 00:02:56
   • Constructing Columns in Structured Streaming 00:02:47
   • Selecting and Filtering Columns Using Structured Streaming 00:02:07
   • GroupBy and Aggregation in Structured Streaming 00:03:33
   • Joining Structured Stream with Datasets 00:03:39
   • SQL Queries in Spark Structured Streaming 00:02:19
4. DStream Comparison
   • Comparing Structured Streaming with DStream 00:03:39
   • Custom Receivers in Spark DStream 00:02:18
   • Iterative Wordcount Using Spark DStream 00:03:30
   • Cumulative Wordcount using Spark DStream 00:06:31
   • Benefits of Spark Tungsten 00:04:43
   • Tungsten Performance Benefit Demonstration 00:02:58
   • Benefits of Spark Catalyst 00:03:18
   • Viewing Query Plans in Spark Shell 00:01:36
   • Visualizing Query Stages in Spark UI Viewer 00:00:51
   • Viewing Spark Catalyst-Optimized Physical Plans 00:02:56
5. Standalone Spark Streaming Applications
   • Writing Standalone Spark Streaming Applications 00:01:03
   • Two Environments for Running Spark 00:01:57
   • Spark Streaming Standalone Code - Meetup Events Example 00:07:37
   • Scala Build Tool (SBT) and Spark 00:06:01
   • Compiling and Building a Standalone Spark Application 00:04:29
   • Spark Twitter Streaming Example 00:03:54
Show and hide more



9781491974445.mastering.spark.for.OR.part1.rar

9781491974445.mastering.spark.for.OR.part2.rar