by Stéphane Maarek | Released May 2018 | ISBN: 9781789342604

Tutorial: Learn the Apache Kafka ecosystem, core concepts, operations, Kafka API, and build your own producers and consumersAbout This VideoLearn the Apache Kafka ecosystem architectureLearn the Kafka core concepts: topics, partitions, brokers, replicas, producers, consumersLaunch your own Kafka cluster in no time using Docker – Windows / MacOS X / LinuxLearn and practice using the Kafka command line toolsLearn and code consumers and producers using the Java APILearn advanced Kafka topic configurationsPointers for integration between Kafka and other frameworks (Spark, NiFi, Akka, Scala).In DetailApache Kafka has become the leading data-streaming enterprise technology. Kafka is used in production by over 2000 companies like Netflix, Airbnb, Uber and LinkedIn. To learn Kafka easily, step-by-step, you have come to the right place! Apache Kafka and its ecosystem: In this section, we will learn about the Apache Kafka ecosystem, and see how some target architectures may look. This high-level section helps you to set context around Kafka! Apache Kafka Core concepts: In this section, we will learn about all the fundamental concepts of Kafka like topics, partitions, replication, brokers, producers, consumers, consumer groups, Zookeeper, delivery semantics, and more! Docker and Kafka Setup: In this section, we will learn how to install Docker on your machine and get started with Apache Kafka, in the simplest way possible. Apache Kafka Hands-on Practice: In this section, we will gain some practical experience by learning how the various command lines tool work, as well as how to use the Kafka Topics UI, and create your very first producer and consumer in Java. Code Examples - Libraries Integrations: In this section, we will learn about some more advanced code examples, and understand where to find the libraries to integrate with frameworks such as Spark, Spark Streaming, Akka Streams, Scala, Actors, Apache NiFi. Advanced Topic Configuration: In this section, we will understand the main configurations for your topics, learn about log compaction, and understand exactly what your partitions are made of! Show and hide more Publisher Resources Download Example Code

1. Chapter 1 : Kafka Introduction
   • Welcome! 00:01:11
   • Apache Kafka in 5 minutes 00:05:20
   • Course Objectives 00:01:47
2. Chapter 2 : Kafka Theory
   • Kafka Theory Overview 00:00:29
   • Topics, Partitions and Offsets 00:06:48
   • Brokers and Topics 00:03:20
   • Topic Replication 00:04:12
   • Producers and Message Keys 00:05:25
   • Consumers & Consumer Groups 00:04:37
   • Consumer Offsets & Delivery Semantics 00:04:40
   • Kafka Broker Discovery 00:02:23
   • Zookeeper 00:02:42
   • Kafka Guarantees 00:01:09
   • Theory Roundup 00:01:59
3. Chapter 3 : Starting Kafka
   • Important: Starting Kafka 00:00:24
   • Mac OS X - Download and Setup Kafka in PATH 00:04:45
   • Mac OS X - Using brew 00:01:30
   • Mac OS X - Start Zookeeper and Kafka 00:05:38
   • Linux - Download and Setup Kafka in PATH 00:05:34
   • Linux - Start Zookeeper and Kafka 00:04:56
   • Windows - Download Kafka and PATH Setup 00:08:13
   • Windows - Start Zookeeper & Kafka 00:04:59
4. Chapter 4 : CLI (Command Line Interface) 101
   • CLI Introduction 00:00:23
   • Kafka Topics CLI 00:07:32
   • Kafka Console Producer CLI 00:07:09
   • Kafka Console Consumer CLI 00:03:21
   • Kafka Consumers in Group 00:06:29
   • Kafka Consumer Groups CLI 00:04:13
   • Resetting Offsets 00:04:08
   • Kafka Tools UI 00:03:48
5. Chapter 5 : Kafka Java Programming 101
   • Intro to Kafka Programming 00:00:25
   • Creating Kafka Project 00:06:25
   • Java Producer 00:10:49
   • Java Producer Callbacks 00:06:52
   • Java Producer with Keys 00:06:08
   • Java Consumer 00:11:50
   • Java Consumer inside Consumer Group 00:06:29
   • Java Consumer with Threads 00:11:51
   • Java Consumer Seek and Assign 00:04:23
   • Client Bi-Directional Compatibility 00:01:17
6. Chapter 6 : Kafka Real World Project
   • Real World Project Overview 00:00:45
7. Chapter 7 : Kafka Twitter Producer & Advanced Configurations
   • Producer and Advanced Configurations Overview 00:00:28
   • Twitter Setup 00:04:44
   • Producer Part 1 - Writing Twitter Client 00:10:14
   • Producer Part 2 - Writing the Kafka Producer 00:07:46
   • Producer Configurations Introduction 00:00:49
   • acks & min.insync.replicas 00:06:16
   • retries & max.in.flight.requests.per.connection 00:02:11
   • Idempotent Producer 00:04:03
   • Producer Part 3 - Safe Producer 00:04:50
   • Producer Compression 00:04:44
   • Producer Batching 00:04:52
   • Producer Part 4 - High Throughput Producer 00:03:07
   • Producer Default Partitions and Key Hashing 00:02:24
   • [Advanced] max.block.ms and buffer.memory 00:02:31
   • Refactoring the Project 00:03:04
8. Chapter 8 : Kafka ElasticSearch Consumer & Advanced Configurations
   • Consumer and Advanced Configuration Overview 00:00:27
   • Setting up ElasticSearch in the Cloud 00:03:13
   • ElasticSearch 101 00:05:48
   • Consumer Part 1 - Setup Project 00:08:22
   • Consumer Part 2 - Write the Consumer & Send to ElasticSearch 00:06:21
   • Delivery Semantics for Consumers 00:04:12
   • Consumer Part 3 – Idempotence 00:07:45
   • Consumer Poll Behaviour 00:05:43
   • Consumer Offset Commit Strategies 00:02:44
   • Consumer Part 4 - Manual Commit of Offsets 00:04:39
   • Consumer Part 5 - Performance Improvement using Batching 00:05:40
   • Consumer Offsets Reset Behaviour 00:02:07
   • Consumer Part 6 - Replaying Data 00:01:58
   • Consumer Internal Threads 00:03:02
9. Chapter 9 : Kafka Ecosystem & Real-World Architectures
   • Kafka in the Real World 00:00:47
10. Chapter 10 : Kafka Extended APIs for Developers
    • Kafka Connect Introduction 00:04:47
    • Kafka Connect Twitter Hands-On 00:08:07
    • Kafka Streams Introduction 00:03:21
    • Kafka Streams Hands-On 00:11:58
    • Kafka Schema Registry Introduction 00:04:42
11. Chapter 11 : Real World Insights and Case Studies (Big Data / Fast Data)
    • Choosing Partition Count & Replication Factor 00:08:36
    • Case Study – MovieFlix 00:05:09
    • Case Study – GetTaxi 00:04:18
    • Case Study – MySocialMedia 00:05:32
    • Case Study – MyBank 00:03:42
    • Case Study - Big Data Ingestion 00:01:49
    • Case Study - Logging and Metrics Aggregation 00:01:59
12. Chapter 12 : Kafka in the Enterprise for Admins
    • Kafka Cluster Setup High Level Architecture Overview 00:02:43
    • Kafka Monitoring & Operations 00:02:59
    • Kafka Security 00:05:02
    • Kafka Multi Cluster & MirrorMaker 00:06:48
13. Chapter 13 : Advanced Topics Configurations
    • Changing a Topic Configuration 00:04:43
    • Segment and Indexes 00:05:11
    • Log Cleanup Policies 00:02:30
    • Log Cleanup Delete 00:02:07
    • Log Compaction Theory 00:06:04
    • Log Compaction Practice 00:07:02
    • min.insync.replicas reminder 00:03:04
    • Unclean Leader Election 00:01:17
14. Chapter 14 : Annexes
    • What are annexes? 00:00:34
15. Chapter 15 : Annex - Starting Kafka Differently
    • Annex 1 – Overview 00:00:34
    • Starting Kafka with the Confluent CLI 00:03:28
    • Starting a multi broker Kafka Cluster using Binaries 00:06:32
    • Start Kafka Development environment using Docker 00:04:20
    • Starting a multi broker Kafka Cluster using Docker 00:02:55
    • Kafka Advertised Host Setting 00:03:25
    • Starting Kafka on a Remote Machine 00:05:43
Show and hide more