How to Create a Web Scraper/Crawler using Python and Scrapy
How to Create a Web Scraper/Crawler using Python and Scrapy
with babis|6 years experience|C-C++
164227total views
Intro: Web Scraper/Crawler using Python and Scrapy
Nov 5, 2018 | 3:37 PM1:19
Session 1: Setting up the Environment
Nov 5, 2018 | 3:37 PM7:1
Session 2: Scrapy Installation
Nov 5, 2018 | 3:38 PM7:11
Session 3: Our first Scrapy project
Nov 5, 2018 | 3:41 PM9:9
Session 4: Extracting website data
Nov 5, 2018 | 3:41 PM9:32
Session 5: Scrapy shell
Nov 5, 2018 | 3:41 PM8:2
Session 6: About web crawling / scrapping
Nov 5, 2018 | 3:41 PM5:24
Session 7: Creating a Scrapy spider
Nov 5, 2018 | 3:41 PM24:20
Session 8: Wiki Scraper Intro
Nov 5, 2018 | 3:42 PM10:7
Sessions 9.1: Scraping Wikipedia Part 1
Nov 5, 2018 | 3:42 PM20:
Sessions 9.2: Scraping Wikipedia Part 2
Nov 5, 2018 | 3:42 PM13:54
Session 10: Scraping Multiple pages
Nov 5, 2018 | 3:42 PM18:33
Session 11: Scrapy Items
Nov 5, 2018 | 3:43 PM19:16
Session 12: Scrapy selection using CSS
Nov 5, 2018 | 3:43 PM22:57
How to Create a Web Scraper/Crawler using Python and Scrapy
- English
- Programming
Python- Project length: 2h 57m
This tutorial will cover the widely used Python framework - Scrapy. You will learn how to use this great tool to create your own web scrapers/crawlers. We are going to create a couple different spiders including a simple Wikipedia Scraper.
Overview
In this tutorial, we are going to learn about the widely used Python framework - Scrapy. Using our new knowledge in Scrapy we are going to create a few 'spiders' including a Wikipedia scraper. I will guide you through each step of the process including debugging common issues when working with Scrapy and web scrapers/crawlers in general.
What are the requirements?
- Basic Python knowledge is required as we are not going to go over how Python works in this tutorial
- HTML and CSS knowledge would be very beneficial to understand how the spiders work
What is the target audience?
- Everyone interested in creating a web scraper/crawler
- Everyone who wants to polish their python skills
Project Outline
Session 1: Setting up the Environment
- In this video, we are going to perform a full install of our editor VsCode in a Linux environment (Lubuntu ).
Session 2: Scrapy Installation
- This video will guide you through Scrapy installation with a demonstration in our Linux environment.
Session 3: Our first Scrapy project
- Introduction to Scrapy. How to create your first Scrapy project.
Session 4: Extracting website data
- In this video, you will scrap your first website data using a Scrapy spider
Session 5: Scrapy shell
- Here, you will learn how to use the very powerful Scrapy shell
Session 6: About web crawling / scrapping
A short video with a little discussion about the legal aspect of web crawling/scraping
Session 7: Creating a Scrapy spider
- In this video we are going to create our first fully function spider, that will scrap quotes from multiple web pages.
Session 8: Wiki Scraper Intro
- In this short video, you will learn about the main project of this tutorial. The Wikipedia Scraper
Sessions 9.1: Scraping Wikipedia Part 1
- In this session we are going to start scraping data from Wikipedia using our new Wikipedia Spider
Sessions 9.2: Scraping Wikipedia Part 2
- In this session we are going to solve a few problems that came up in the previous session and also test our new spider
Session 10: Scraping Multiple pages
- We are going to take our app one step further and scrap multiple Wikipedia pages at the same time
Session 11: Scrapy Items
- We will polish our program by changing the way we store our items using Scrapy Items
Session 12: Scrapy selection using CSS
- In this final part of the project, we are going to create another spider that instead of XPath, it is going to use CSS to scrap our web page elements