Last updated 4/2019MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHzLanguage: English | Size: 4.98 GB | Duration: 9h 52m
Perform analytics on real- data by discovering techniques to test and parallelize Spark jobs & solve common problems What you'll learn Implement high-velocity streaming and data processing use cases while working with streaming API. Dive into MLlib– the machine learning functional library in Spark with highly scalable algorithms. Create machine learning pipelines to combine multiple algorithms in a single workflow. Create highly concurrent Spark programs by leveraging immutability. Re-design your jobs to use reduceByKey instead of groupBy. Create robust processing pipelines by testing Apache Spark jobs. Solve repeated problems by leveraging the GraphX API. Solve long-running computation problems by leveraging lazy evaluation in Spark. Avoid memory leaks by understanding the internal memory management of Apache Spark. Troubleshoot real- pipelines written in Spark Streaming, APIs for joins - DataFrames or DataSets. Requirements Basic Knowledge of Spark Programming and the fundamentals of Apache Spark along with some basic understanding of real- data processing is necessary. Some familiarity with Scala would be helpful. Description If you face challenges on how to analyze real- data, create real-world streaming processing in Spark, and face some common pitfalls in your Spark code and are looking for a solution to get you out of the development problems providing you with some best practices so that you can code better, efficiently and faster for analyzing a large amount of data, then this learning series is perfect for you!With this well thought out Learning Path, you will first b by learning the fundamentals of Apache Spark which includes Resilient Distributed Datasets (RDD), HDFS, YARN, create effective Spark application and execute it on Hadoop cluster & much more. Then you will learn to analyze data using machine learning techniques and graphs. Moving further you will focus o some amazing tips & tricks to improve particular aspects of programming & administration in Apache Spark & also speed up your Spark jobs by reducing shuffles. Finally, you will learn some quick & simple solutions to troubleshoot development issues and debugging techniques with Apache Spark.Contents and OverviewThis training program includes 4 complete courses, carefully chosen to give you the most comprehensive training possible.The first course, Apache Spark Fundamentals you will b learning about the Apache Spark programming fundamentals such as Resilient Distributed Datasets (RDD) and See which operations can be used to perform a transformation or action operation on the RDD. We'll show you how to load and save data from various data sources as a different type of files, No-SQL and RDBMS databases, etc.. We’ll also explain Spark advanced programming concepts such as managing Key-Value pairs, accumulators, etc. Finally, you'll discover how to create an effective Spark application and execute it on the Hadoop cluster to the data and gain insights to make informed business decisions. By the end of this video, you will be well-versed with all the fundamentals of Apache Spark and implementing them in Spark.The second course, Advanced Analytics, and Real- Data Processing in Apache Spark you will learn how to implement the high-velocity streaming operation for data processing in order to perform efficient analytics on your real- data. You’ll analyze data using machine learning techniques and graphs. You’ll learn about Spark Streaming and create real-world streaming processing that addresses all the problems that need to be solved. You’ll solve problems using Machine Learning techniques and find out about all the tools available in the MLlibtoolkit. You’ll find out how to leverage Graphs to solve real-world problems. At the end of this video, you’ll also see some useful Machine Learning algorithms with the help of Spark MLlib and will integrate Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you’ll learn more about the GraphX API. By the end, you’ll be well-versed in the aspects of real- analytics and implement them with Apache Spark.The third course, Apache Spark: Tips, Tricks, & Techniques you'll learn to implement some practical and proven techniques to improve particular aspects of programming and administration in Apache Spark. You will explore 7 sections that will address different aspects of Spark via 5 specific techniques with clear instructions on how to carry out different Apache Spark tasks with hands-on experience. The techniques are demonstrated using practical examples and best practices. By the end of this course, you will have learned some exciting tips, best practices, and techniques with Apache Spark. You will be able to perform tasks and get the best data out of your databases much faster and with ease.The fourth course, Troubleshooting Apache Spark will give you new possibilities and you'll cover many aspects of Apache Spark; some you may know and some you probably never knew existed. If you take a lot of learning and perfog tasks on Spark, you are unable to leverage Apache Spark's full capabilities and features, and face a roadblock in your development journey. You'll face issues and will be unable to optimize your development process due to common problems and bugs; you'll be looking for techniques which can save you from falling into any pitfalls and common errors during development. With this course, you'll learn to implement some practical and proven techniques to improve particular aspects of Apache Spark with proper research. You need to understand the common problems and issues Spark developers face, collate them, and build simple solutions for these problems. One way to understand common issues is to look out for Stack Overflow queries. This course is a high-quality troubleshooting course, highlighting issues faced by developers in different stages of their application development and providing them with simple and practical solutions to these issues. It supplies solutions to some problems and challenges faced by developers; however, this course also focuses on discovering new possibilities with Apache Spark. By the end of this course, you will have solved your Spark problems without any hassle.About the Authors:Nishant Garg has over 16 years of software architecture and development experience in various technologies, such as Java Enterprise Edition, SOA, Spring, Hadoop, Hive, Flume, Sqoop, Oozie, Spark, YARN, Impala, Kafka, Storm, Solr/Lucene, NoSQL databases (such as HBase, Cassandra, and MongoDB), and MPP databases (such as Greenplum). He received his MS in software systems from the Birla Institute of Technology and Science, Pilani, India, and is currently working as a senior technical architect for the Big Data R&D Labs with Impetus Infotech Pvt. Ltd. Previously, Nishant has enjoyed working with some of the most recognizable names in IT services and financial industries, employing full software life cycle methodologies such as Agile and SCRUM. Nishant has also undertaken many speaking engagements on big data technologies and is also the author of Learning Apache Kafka & HBase Essentials, Packt Publishing.Tomasz Lelek is a Software Eeer and Co-Founder of InitLearn. He mostly does programming in Java and Scala. He dedicates his and effort to get better at everything. He is currently diving into Big Data technologies. Tomasz is very passionate about everything associated with software development. He has been a speaker at a few conferences in Poland-Confitura and JDD, and at the Krakow Scala User Group. He has also conducted a live coding session at Geecon Conference. He was also a speaker at an international event in Dhaka. He is very enthusiastic and loves to share his knowledge. Keywords: Data processing, data modeling, data analysis, data analytics, graphical processing, data frame operations, R algorithm. Overview Section 1: Apache Spark Fundamentals Lecture 1 Course Overview Lecture 2 Spark Introduction Lecture 3 Spark Components Lecture 4 Getting Started Lecture 5 Introduction to Hadoop Lecture 6 Hadoop Processes and Components Lecture 7 HDFS and YARN Lecture 8 Map Reduce Lecture 9 Introduction to Scala Lecture 10 Scala Programming Fundamentals Lecture 11 Objects in Scala Lecture 12 Collections Lecture 13 Spark Execution Lecture 14 Understanding RDD Lecture 15 RDD Operations Lecture 16 Loading and Saving Data in Spark Lecture 17 Managing Key-Value Pairs Lecture 18 Accumulators Lecture 19 Writing a Spark Application Section 2: Advanced Analytics and Real- Data Processing in Apache Spark Lecture 20 The Course Overview Lecture 21 Introducing Spark Streaming Lecture 22 Streaming Context Lecture 23 Processing Streaming Data Lecture 24 Use Cases Lecture 25 Spark Streaming Word Count Hands-On Lecture 26 Spark Streaming – Understanding Master URL Lecture 27 Integrating Spark Streaming with Apache Kafka Lecture 28 mapWithState Operation Lecture 29 Transform and Window Operation Lecture 30 Join and Output Operations Lecture 31 Output Operations –Saving Results to Kafka Sink Lecture 32 Handling in High Velocity Streams Lecture 33 Connecting External Systems That Works in At Least Once Guarantee – Deduplicaion Lecture 34 Building Streaming Application –Handling Events That Are Not in Order Lecture 35 Filtering Bots from Stream of Page View Events Lecture 36 Introducing Machine Learning with Spark Lecture 37 Feature Extraction and Transformation Lecture 38 Transfog Text into Vector of Numbers – ML Bag-of-Words Technique Lecture 39 Logistic Regression Lecture 40 Model Evaluation Lecture 41 Clustering Lecture 42 Gaussian Mixture Model Lecture 43 Principal Component Analysis and Distributing the Singular Value Decomposition Lecture 44 Collaborative Filtering – Building Recommendation Ee Lecture 45 Introducing Spark GraphX–How to Represent a Graph? Lecture 46 Limitations of Graph-Parallel System – Why Spark GraphX? Lecture 47 Importing GraphX Lecture 48 Create a Graph Using GraphX and Property Graph Lecture 49 List of Operators Lecture 50 Perform Graph Operations Using GraphX Lecture 51 Triplet View Lecture 52 Perform Subgraph Operations Lecture 53 Neighbourhood Aggregations – Collecting Neighbours Lecture 54 Counting Degree of Vertex Lecture 55 Caching and Uncaching Lecture 56 GraphBuilder Lecture 57 Vertex and Edge RDD Lecture 58 Structural Operators – Connected Components Lecture 59 Introduction to SparkR and How It’s Used? Lecture 60 Setting Up from RStudio Lecture 61 Creating Spark DataFrames from Data Sources Lecture 62 SparkDataFrames Operations – Grouping, Aggregation Lecture 63 Run a Given Function on a Large Dataset Using dapply or dapplyCollect Lecture 64 Running Large Dataset by Input Column(s) and Using gapply or gapplyCollect Lecture 65 Run Local R Functions Distributed Using spark.lapply Lecture 66 Running SQL Queries from SparkR Lecture 67 PageRank Using Spark GraphX Lecture 68 Sending Real- NotificationWhen User Want to Buy a Product on E-Commerce site Section 3: Apache Spark: Tips, Tricks, & Techniques Lecture 69 The Course Overview Lecture 70 Using Spark Transformations to Defer Computations to a Later Lecture 71 Avoiding Transformations Lecture 72 Using reduce and reduceByKey to Calculate Results Lecture 73 Perfog Actions That Trigger Computations Lecture 74 Reusing the Same RDD for Different Actions Lecture 75 Delve into Spark RDDs Parent/Child Chain Lecture 76 Using RDD in an Immutable Way Lecture 77 Using DataFrame Operations to Transform It Lecture 78 Immutability in the Highly Concurrent Environment Lecture 79 Using Dataset API in an Immutable Way Lecture 80 Detecting a Shuffle in a Processing Lecture 81 Testing Operations That Cause Shuffle in Apache Spark Lecture 82 Chag Design of Jobs with Wide Dependencies Lecture 83 Using keyBy Operations to Reduce Shuffle Lecture 84 Using Custom Partitioner to Reduce Shuffle Lecture 85 Saving Data in Plain Text Lecture 86 Leveraging JSON as a Data Format Lecture 87 Tabular Formats – CSV Lecture 88 Using Avro with Spark Lecture 89 Columnar Formats – Parquet Lecture 90 Available Transformations on Key/Value Pairs Lecture 91 Using aggregateByKey Instead of groupBy Lecture 92 Actions on Key/Value Pairs Lecture 93 Available Partitioners on Key/Value Data Lecture 94 Implementing Custom Partitioner Lecture 95 Separating Logic from Spark Ee – Unit Testing Lecture 96 Integration Testing Using SparkSession Lecture 97 Mocking Data Sources Using Partial Functions Lecture 98 Using ScalaCheck for Property-Based Testing Lecture 99 Testing in Different Versions of Spark Lecture 100 Creating Graph from Datasource Lecture 101 Using Vertex API Lecture 102 Using Edge API Lecture 103 Calculate Degree of Vertex Lecture 104 Calculate Page Rank Section 4: Troubleshooting Apache Spark Lecture 105 The Course Overview Lecture 106 Eager Computations: Lazy Evaluation Lecture 107 Caching Values: In-Memory Persistence Lecture 108 Unexpected API Behavior: Picking the Proper RDD API Lecture 109 Wide Dependencies: Using Narrow Dependencies Lecture 110 Making Computations Parallel: Using Partitions Lecture 111 Defining Robust Custom Functions: Understanding User-Defined Functions Lecture 112 Logical Plans Hiding the Truth: Examining the Physical Plans Lecture 113 Slow Interpreted Lambdas: Code Generation Spark Optimization Lecture 114 Avoid Wrong Join Strats: Using a Join Type Based on Data Volume Lecture 115 Slow Joins: Choosing an Execution Plan for Join Lecture 116 Distributed Joins Problem: DataFrame API Lecture 117 TypeSafe Joins Problem: The Newest DataSet API Lecture 118 Minimizing Object Creation: Reusing Existing Objects Lecture 119 Iterating Transformations – The mapPartitions Method Lecture 120 Slow Spark Application Start: Reducing Setup Overhead Lecture 121 Perfog Unnecessary Recomputation: Reusing RDDs Lecture 122 Repeating the Same Code in Stream Pipeline: Using Sources and Sinks Lecture 123 Long Latency of Jobs: Understanding Batch Internals Lecture 124 Fault Tolerance: Using Data Checkpointing Lecture 125 Maintaining Batch and Streaming: Using Structured Streaming Pros This course is for data scientists, big data technology developers and analysts, Apache Spark developers, who want to learn the fundamentals of Apache Spark & improve their Apache Spark skills with amazing tricks & techniques. HomePage: gfxtra__Apache_Spa.part1.rar.html gfxtra__Apache_Spa.part2.rar.html gfxtra__Apache_Spa.part3.rar.html
TO MAC USERS: If RAR password doesn't work, use this archive program:
RAR Expander 0.8.5 Beta 4 and extract password protected files without error.
TO WIN USERS: If RAR password doesn't work, use this archive program:
Latest Winrar and extract password protected files without error.