Python Web Scraping Tutorial: Step-by-Step Guide

Python web scraping lets you automate data collection and skip tedious manual copying. Thanks to a massive developer community, you have access to powerful libraries you can use for free to extract data from target web pages.

We’ll show you how to build a basic web scraper to extract book titles and prices from a sample bookstore. We will also explore advanced tools you can use as your projects scale.

What Is Web Scraping in Python?

Web scraping is using automation to request content from a server and parse the HTML to extract data. Python is one of the best options due to its readable syntax and powerful data processing and data science libraries that help clean the extracted information.

There are many use cases for web scraping, some of which include tracking ecommerce prices, building datasets, collecting research data, and monitoring search engine rankings.

Modern websites generally fall into two categories: static sites that load HTML immediately, and dynamic sites that rely on JavaScript to render content in the browser. Basic web scraping tools only read static HTML, which means you’ll often need browser automation tools like Selenium or Playwright to scrape dynamically rendered sites.

How to Scrape a Website With Python: Step-by-Step

Let’s start with the basics and build a script to scrape Books to Scrape, a practice online bookstore. We will extract the title, price, and link for every book on the front page.

Step 1. Install Python and Required Libraries

Install Python 3.10 or newer from python.org; its installer includes pip. Then open your terminal to install the needed libraries to handle network connections and parse HTML. The Requests package manages HTTP requests, and BeautifulSoup structures the returned HTML.

pip install requests beautifulsoup4

Step 2. Send an HTTP Request

Your script needs to request the target page from the server. Pass the URL to the Requests library to get a response object containing the HTTP status code and the HTML document. Store the raw bytes from response.content, not response.text: this site’s headers declare no charset, so Requests would decode the page as ISO-8859-1 and turn every £ into £.

import requests

url = "https://books.toscrape.com/"
response = requests.get(url, timeout=10)
print(response.status_code)  # 200
html_content = response.content

Step 3. Parse HTML With BeautifulSoup

Pass the HTML document to BeautifulSoup to parse it into a navigable structure. You can then use basic CSS selectors to pinpoint the exact HTML elements containing the book titles and prices.

If you’re not sure which selectors to use, right-click a book title in your browser and click Inspect. Each book sits in an article element with the class product_pod, and its price is in a p element with the class price_color.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, "html.parser")
books = soup.select("article.product_pod")

Step 4. Extract and Save the Data

Loop through these HTML elements to extract the title, price, and link as structured data. Once extracted, you can append these variables to a list and later write them to a standard CSV file for viewing in a spreadsheet.

import csv

all_books = []
for book in books:
    title = book.h3.a["title"]
    price = book.select_one("p.price_color").text
    link = url + book.h3.a["href"]
    all_books.append([title, price, link])

with open("books.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["Title", "Price", "Link"])
    writer.writerows(all_books)

Step 5. Handle Pagination and Errors

To handle pagination, locate the next button in the HTML and instruct your script to follow links to the next page until no more pages exist.

Also, it’s recommended to implement basic error handling to catch connection timeouts and prevent the script from crashing while extracting data during a large job.

Here’s the full script, which collects all 1,000 books from the site’s 50 pages:

import requests
from bs4 import BeautifulSoup
import csv
import time
import random

def scrape_bookstore():
    base_url = "https://books.toscrape.com/catalogue/"
    current_page = "page-1.html"
    all_books = []
    
    while current_page:
        try:
            response = requests.get(base_url + current_page, timeout=10)
            response.raise_for_status()
        except requests.exceptions.RequestException as e:
            print(f"Request failed on {current_page}: {e}")
            break
            
        soup = BeautifulSoup(response.content, "html.parser")
        books = soup.select("article.product_pod")
        
        for book in books:
            title = book.h3.a["title"]
            price = book.select_one("p.price_color").text
            link = base_url + book.h3.a["href"]
            all_books.append([title, price, link])
            
        next_button = soup.select_one("li.next a")
        if next_button:
            current_page = next_button["href"]
            time.sleep(random.uniform(1, 3))
        else:
            current_page = None
            
    with open("books.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.writer(file)
        writer.writerow(["Title", "Price", "Link"])
        writer.writerows(all_books)

    print(f"Saved {len(all_books)} books to books.csv")

scrape_bookstore()

Running it prints this line, and books.csv opens in any spreadsheet app:

Saved 1000 books to books.csv

Best Python Libraries for Web Scraping

The Python ecosystem offers powerful libraries tailored to different web scraping challenges.

For example, Requests handles basic HTTP calls, and HTTPX supports asynchronous requests to fetch multiple pages concurrently. There’s plenty of other libraries for HTTP and request management as well, so you can always find the one that suits your use case best.

BeautifulSoup is great at parsing standard HTML, while lxml is a much faster parser for both HTML and XML. If speed matters, BeautifulSoup can also use lxml as its parser.

For large-scale extraction, Scrapy offers a complete framework with built-in request scheduling and item pipelines.

For dynamically rendered sites, Playwright and Selenium automate real web browsers to execute JavaScript before extracting the data.

Python Web Scraping Challenges and Best Practices

Best practices around web scraping cover how to handle unstable connections, rate limits, and IP blocks, and how to collect data responsibly.

You need to implement retry logic to manage timeouts and dropped connections. For dynamically rendered websites, standard parsers won’t work in most cases, so you’ll need to work with browser automation tools to access the JavaScript data.

Validate the data before you save it, too. Check that each record has a title, a price, and a link, so a change in the page layout doesn’t fill your CSV with empty or broken rows.

While you’re still developing the scraper, save a copy of each page you download and work from that copy instead. Otherwise, every test run requests the same page again, which is slow for you and needless traffic for the site.

Don’t overwhelm target servers with thousands of requests per second; add rate limiting to your scripts. If you send too many requests in a very short window, you’ll almost definitely trigger IP bans, so adding randomized delays between your HTTP calls will help your web scrapers remain operational.

For more advanced web scraping tasks, you may consider using rotating proxy networks to distribute the load across multiple IP addresses and reduce the likelihood of being blocked. With anyIP, all proxy settings are included in the username, and Requests accepts the proxy URL through its proxies argument:

import requests

proxy = "http://user_XXXX,type_residential,country_US:PASSWORD@portal.anyip.io:1080"
proxies = {"http": proxy, "https": proxy}

response = requests.get("https://books.toscrape.com/", proxies=proxies, timeout=10)

Without a session flag, anyIP will route each request through a different IP address. To stick to a single IP address for a multi-step task, add session_ to the username and a unique name of your choice, such as session_books1. Sticky sessions will last up to 7 days, depending on IP availability. If your network blocks port 1080, you can use port 443 instead.

Before starting to scrape data from websites, check the robots.txt file and Terms of Service. These will help ensure that your web scraping efforts remain within legal and ethical boundaries.

Conclusion

You now have a functioning Python web scraper. It uses Requests to fetch each page, BeautifulSoup to parse the HTML, handles pagination, and saves every book to a CSV file. For websites that render their content through JavaScript, use Playwright or Selenium instead. As your jobs grow, add retries and rate limiting, and use proxies to distribute the load.