Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

Description

Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

In this course you won't be a consumer anymore, i'll teach you how you can build your own scraping tool ( spider ) using Scrapy.

You will learn:

What makes this course different from the others, and why you should enroll ?

• First, this is the most updated course. You will be using Python 3.7, Scrapy 1.6 and Splash 3.0

• You will have an in-depth step by step guide on how to become a professional web scraper.

• You will learn how to use Splash & Selenium to scrape JavaScript websites and I can assure you, you won't find any tutorials out there that teaches how to really use Splash like I'll be doing in this course.

• You will learn how to host spiders in Heroku as well as Splash(Exclusive).

• You will learn how to create a custom script so spiders can run periodically without any intervention from you.

• 30 days money back guarantee by Udemy

So whether you are a data analyst who wants to add web scraping to his tool set or someone else who wants to learn how to extract unstructured data from unstructured HTML web pages and then store back that data in a structured way to apply some data analysis on it then you are welcome to join this course.

**STUDENTSTHOUGHTSABOUTTHISCOURSE **

"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! ". By Hiran Kumar

"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed". By Raylyson Estanista

"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses. ". By Richard Blackmon

"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever).". By I S

"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." By Robert Nolte

"That course is very good and explanation is crystal clear! The instructor is very supportive in case of questions. Highly recommended." By Shubina Ekaterina

"I like the course. Clear explanations and good comunication with Ahmed. All topics is interesting and full of information. I improved my skils in Scrapy. Author update course content by new videos. It's a big bonus) Explained more advance topics I never see in other courses. Thank you, Ahmed. Waiting for new videos)". By Ruslan Romanenko

• Anyone who wants to scrape data from any website
• Anyone who wants to learn Scrapy
• Anyone who wants to automate the task of copying contents from websites
• Anyone who wants to learn how to scrape Javascript websites using Scrapy-Splash & Selenium

Course content

• Introduction
  • Intro to Web Scraping & Scrapy
  • Setting up Scrapy using Anaconda
  • Udemy 101 (Please don't skip*)
  • Asking questions
• Scrapy Fundamentals
  • Scrapy fundamentals PART 1
  • Scrapy fundamentals PART 2
  • Scrapy fundamentals PART 3
  • Scrapy fundamentals PART 4
  • Scrapy fundamentals PART 5
• XPath expressions & CSS Selectors
  • XPath & CSS Selectors
  • CSS Selectors fundamentals
  • CSS selectors in theory
  • XPath fundamentals
  • Navigating using XPath(Going UP)
  • Navigating using XPath(Going DOWN)
  • XPath in theory
• Project 1 Spiders from A to Z
  • Worldometers PART 1
  • Worldometers PART 2
  • Worldometers PART 3
  • Worldometers PART 4
  • Project source code
  • Exercise
• Building Datasets
  • Bulding datesets
• Project 2 Dealing with Multiple pages
  • IMPORTANT NOTE
  • Setting up the project
  • Building the spider
  • Dealing with pagination
  • Spoofing request headers
  • Project source code
  • Exercise
• Debugging spiders
  • Debugging spiders PART 1
  • Debugging spiders PART 2
• Let's take a break !
  • The "whys" & "whens" of web scraping
  • Web scraping challenges
• Project 3 Build Crawlers using Scrapy
  • Crawl spider structure
  • The Rule object
  • Following links in pagination
  • Spoofing request headers
  • Project source code
  • Exercise
• Splash crash course
  • What dilemma splash came to solve
  • Setting up Splash
  • Introduction to Splash
  • Working with elements
  • Spoofing request headers
• Project 4 Scraping JavaScript websites using Splash
  • Splash incognito mode
  • Using Splash with Scrapy
  • Parsing (BAD HTML MARKUP)
  • Project source code
  • Exercise
• Project 5 Scraping JavaScript websites using Selenium
  • Selenium basics
  • ElementNotInteractable Exception
  • Selenium with Scrapy
  • Selenium Middleware PART 1 (NEW)
  • Selenium Middleware PART 2 (NEW)
  • Project source code
• Working with Pipelines
  • Pipelines
  • Storing data in MongoDB
  • Storing data in SQLite3
  • Project source code
• Avoid Getting Banned (OLD update)
  • *IMPORTANT*
  • Technics Used by Websites Administrators to Prevent Web Scraping
  • Web Crawling/Scraping Best Practices
  • Custom Middleware (User Agent Rotator Middleware)
• Scraping APIs(REST API) - Infinite Scroll Pagination (OLD update)
  • *IMPORTANT*
  • Introduction
  • REST API
  • Working With JSON Objects
  • The Airbnb JSON Object
  • Hidden XHR
  • Airbnb Spider
  • IMPORTANT NOTE
  • Infinite Scroll Pagination
  • Spider Arguments
  • Airbnb code UPDATE (Request Cookies) **NEW
  • Another way to scrape Airbnb restaurant detail page
• Hosting spiders for free - Exclusive - (OLD update)
  • *IMPORTANT*
  • Deploy spiders to ScrapingHub cloud
  • Deploy spiders locally
  • Deploy spiders to Heroku
  • The MLab add-on
  • Execute spiders periodically
  • Deploy Splash to Heroku
  • Project source code
  • Project source code
  • Challenge for those who are adventurous
• Scrapy POST requests (OLD update)
  • Login to websites using FormRequest
  • XML Http Post Requests
  • XML Http Post requests assignment
  • Project source code
  • Code UPDATE XHR repeated data (Assignment)
• The Media Pipeline (OLD update)
  • *IMPORTANT*
  • Media Pipelines
  • The Images Pipeline
  • Extending The Images Pipeline (Store images with custom names)
  • Files Pipeline (Article)
  • Challenge (Files Pipeline)
  • Project source code
• Paid and Free proxies with Scrapy/Splash (OLD update)
  • *IMPORTANT*
  • Using Crawlera with Scrapy
  • Using Crawlera with Splash
  • Using Heroku as a Proxy (FREE)
  • Using FREE Proxies with the CrawlSpider
  • Challenge
  • Project source code
• BONUS
  • Files Pipeline
  • Crawlera GIFT
  • Bonus Lecture