by David Talby, Claudiu Branzan | Released July 2016 | ISBN: 9781491964293

A text mining system must go way beyond indexing and search to appear truly intelligent. First, it should understand language beyond keyword matching. For example, it should be able to distinguish the critical difference between “Jane has the flu” and “Jane had the flu when she was 9.” Second, it should be capable of making likely inferences even if they're not explicitly written. For example, inferring that Jane may have the flu if she has had a fever, headache, fatigue, and runny nose for three days. And third, it should do its work as part of a robust, scalable, efficient, and easy to extend system. This course teaches software engineers and data scientists how to build intelligent natural language understanding (NLU) based text mining systems at scale using Java, Scala, and Spark for distributed processing. Learn the meaning of natural language understanding (NLU) and its use in text mining Discover how to build a natural language processing (NLP) pipeline within a big data framework Recognize the differences between NLP pipelines and other approaches to semantic text mining Learn about standard UIMA annotators, custom annotators, and machine learned annotators Discover how different types of annotators are composed into a text processing pipeline Use machine learning to generate annotators and apply them within a data pipeline See pipeline architectures that incorporate Kafka, Spark, SparkSQL, Cassandra, and ElasticSearchDavid Talby (PhD , Computer Science, Hebrew University) and Claudio Branzan (Masters, Industrial Intelligent Systems, Polytechnic University of Timișoara) work for big data analytics firm Atigeo. David is CTO and Claudio runs the Modeling and Predictive Analytics team. David and Claudio co-presented on text mining and natural language understanding at O'Reilly's Strata+Hadoop World London 2016 conference. Show and hide more Publisher resources Download Example Code

1. Part 1: Introduction
   • Welcome to the Course 00:01:39
   • Natural Language Understanding in Examples 00:10:09
2. Part 2: NLP Pipelines
   • Building an NLP Pipeline 00:15:49
3. Part 3 - Annotators
   • Commonly Used Annotators 00:08:47
   • Detecting Positive, Negative & Speculative Polarity 00:12:09
   • Machine Learned Annotators 00:12:16
4. Part 4: Custom Annotators
   • NLP Pipelines are Domain Specific 00:06:55
   • Unified Medical Language System (UMLS) 00:03:33
   • Coding Custom Annotators 00:07:17
5. Part 5: Machine Learned Annotators
   • Training & Using Machine Learned Annotators 00:09:45
6. Part 6: Ontology Enrichment
   • The Need for Learned and Updated Ontologies 00:09:39
   • Learning New Medical Concepts and Relationships 00:19:37
7. Part 7: Architecture
   • An End-to-End Reference Architecture 00:04:19
   • Spark, SparkSQL, Cassandra Workflow 00:03:16
   • ElasticSearch & SparkSQL 00:06:52
8. Part 8: Parting Advice
   • Language is Source and Domain-Specific 00:09:32
Show and hide more