by | Released March 2015 | ISBN: 9781491923306

Currently one of the hottest projects across the Hadoop ecosystem, Apache Kafka is a distributed, real-time data system that functions in a manner similar to a pub/sub messaging service, but with better throughput, built-in partitioning, replication, and fault tolerance. In this video course, host Gwen Shapira from Cloudera shows developers and administrators how to integrate Kafka into a data processing pipeline.You'll start with Kafka basics, walk through code examples of Kafka producers and consumers, and then learn how to integrate Kafka with Hadoop. By the end of this course, you'll be ready to use this service for large-scale log collection and stream processing.Learn Kafka's use cases and the problems that it solvesUnderstand the basics, including logs, partitions, replicas, consumers, and producersSet up a Kafka cluster, starting with a single node before adding moreWrite producers and consumers, using old and new APIsUse the Flume log aggregation framework to integrate Kafka with HadoopConfigure Kafka for availability and consistency, and learn how to troubleshoot various issuesBecome familiar with the entire Kafka ecosystemGwen Shapira is a software engineer at Cloudera with 15 years of experience working with customers to design scalable data architectures. Working as a data warehouse DBA, ETL developer, and a senior consultant, Gwen specializes in building scalable data processing pipelines and integrating existing data systems with Hadoop. She's a committer to Apache Sqoop and an active contributor to Apache Kafka. Show and hide more Publisher resources View/Submit Errata

1. The Case for Kafka 00:11:23
2. The Basics 00:09:10
3. Setting up a Kafka Cluster 00:15:30
4. Writing a Kafka Producer 00:14:33
5. Writing a Kafka Consumer 00:16:34
6. Using Kafka from Python 00:08:03
7. Troubleshooting Kafka 00:29:29
8. Integrating Kafka and Hadoop with Flafka 00:26:06
9. Kafka Availability and Consistency 00:22:38
10. Kafka Ecosystem 00:13:13
11. Future of Kafka 00:08:53
Show and hide more