Level 5 · In the Wild

Web Scraping 🕸️

When there is no API, the data is still there, in the page. Here is how to get it, and the rules that keep you out of trouble.

The rules, before the code

⚖️ Read this section properly

Scraping is legal in many places and not in others, and the deciding factors are usually the site's terms of service, what you do with the data, and whether you caused harm. This is not legal advice. What follows is the professional standard of behaviour.

  1. Look for an API first. Most sites worth scraping have one. It will be faster, more reliable and explicitly permitted.
  2. Read /robots.txt. It tells you which paths the operator does not want automated access to. It is not legally binding everywhere, and ignoring it is bad faith.
  3. Read the terms of service. Many explicitly prohibit automated collection.
  4. Rate limit yourself. One request every second or two. You are a guest on someone else's hardware, and they pay for it.
  5. Identify yourself in the User-Agent, with a way to contact you.
  6. Never scrape personal data without a lawful basis. GDPR and similar laws apply to you even when the data is publicly visible.
  7. Cache aggressively. Fetch once, parse many times while developing.
import urllib.robotparser

rules = urllib.robotparser.RobotFileParser()
rules.set_url("https://example.com/robots.txt")
rules.read()

if rules.can_fetch("my-scraper", "https://example.com/products"):
    print("allowed")
else:
    print("robots.txt says no. Stop.")

Parsing HTML

from bs4 import BeautifulSoup      # pip install beautifulsoup4

html = """
<html><body>
  <h1 class="title">Stan's Previously Owned Vessels</h1>
  <ul id="ships">
    <li class="ship" data-id="1"><span class="name">Sea Monkey</span>
        <span class="price">£4,000</span></li>
    <li class="ship" data-id="2"><span class="name">Flying Dutchman</span>
        <span class="price">£12,500</span></li>
  </ul>
  <a href="/page/2">Next</a>
</body></html>
"""

soup = BeautifulSoup(html, "html.parser")

print(soup.h1.text)
print(soup.find("span", class_="name").text)

for ship in soup.find_all("li", class_="ship"):
    name = ship.find("span", class_="name").text
    price = ship.find("span", class_="price").text
    print(f"{ship['data-id']}  {name:16} {price}")

print(soup.find("a")["href"])

That example needs beautifulsoup4 installed, so it has no run button. The standard library does include an HTML parser, though, and for a simple job it is enough: 

from html.parser import HTMLParser


class LinkFinder(HTMLParser):
    """Collect every href in a document. No packages required."""

    def __init__(self):
        super().__init__()
        self.links = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            for name, value in attrs:
                if name == "href":
                    self.links.append(value)


finder = LinkFinder()
finder.feed("""
<p>See <a href="/python/">the course</a> and
<a href="https://rustyschool.com">the school</a>.</p>
""")
print(finder.links)
['/python/', 'https://rustyschool.com']

Selectors, the concise way

from bs4 import BeautifulSoup

soup = BeautifulSoup("<div class='card'><h2>Title</h2><p>Body</p></div>", "html.parser")

print(soup.select_one("div.card h2").text)
print([tag.name for tag in soup.select("div.card > *")])
print(soup.select("p")[0].get_text(strip=True))
SelectorMatches
divevery div
.cardanything with class card
#shipsthe element with id ships
div.card pa p anywhere inside a div.card
div > pa p that is a direct child
a[href^='/page']links whose href starts with /page

A complete, polite scraper

import time
from pathlib import Path

import requests
from bs4 import BeautifulSoup

CACHE = Path("scrape-cache")
HEADERS = {"User-Agent": "python-school-example (contact: you@example.com)"}


def get_page(url: str, session: requests.Session, delay: float = 1.5) -> str:
    """Fetch a page, using a local cache so development costs the site nothing."""
    CACHE.mkdir(exist_ok=True)
    key = CACHE / (url.replace("/", "_").replace(":", "") + ".html")

    if key.exists():
        return key.read_text(encoding="utf-8")

    response = session.get(url, timeout=10)
    response.raise_for_status()
    key.write_text(response.text, encoding="utf-8")
    time.sleep(delay)          # only sleep when we really hit the network
    return response.text


def parse_ships(html: str) -> list[dict]:
    """Pull the ship records out of one page."""
    soup = BeautifulSoup(html, "html.parser")
    ships = []
    for item in soup.select("li.ship"):
        name = item.select_one(".name")
        price = item.select_one(".price")
        if not name or not price:
            continue          # the page changed; skip rather than crash
        ships.append({
            "id": item.get("data-id"),
            "name": name.get_text(strip=True),
            "price": price.get_text(strip=True),
        })
    return ships


def scrape_all(start_url: str, max_pages: int = 5) -> list[dict]:
    """Follow 'Next' links, politely, up to a hard limit."""
    results = []
    url = start_url

    with requests.Session() as session:
        session.headers.update(HEADERS)
        for _ in range(max_pages):
            html = get_page(url, session)
            results.extend(parse_ships(html))

            soup = BeautifulSoup(html, "html.parser")
            next_link = soup.select_one("a.next")
            if not next_link:
                break
            url = requests.compat.urljoin(url, next_link["href"])

    return results
PARANOIA[Medium: Success]

Note the hard limit on pages and the skip-rather-than-crash when a field is missing. Both exist because scrapers run unattended against pages that change without warning.

A scraper with no page limit that follows links is a program that can crawl an entire site, or loop forever between two pages that link to each other. Put a ceiling on anything that follows links it did not choose.

When the content is not in the HTML

Fetch a modern site and find an almost empty page: the content is loaded by JavaScript after the page arrives, and requests does not run JavaScript. Three options, in order of preference:

  1. Find the underlying API. Open your browser's developer tools, Network tab, and reload. The page is almost certainly fetching JSON from an endpoint you can call directly. This is faster and more stable than parsing HTML, and it is what experienced scrapers do first.
  2. Look for embedded data. Many pages ship their data inside a <script type="application/json"> tag.
  3. Drive a real browser with Playwright or Selenium. Powerful, slow, fragile, and a much heavier commitment.

Tables, the easy case

import pandas as pd      # pip install pandas lxml

tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_programming_languages")
print(len(tables), "tables found")
print(tables[0].head())

If the data you want is already in an HTML <table>, one pandas call turns every table on the page into a dataframe. It is worth checking for this before writing any parsing code at all.

Exercise 1

Extract structured data

Parse this fragment with the standard library only and produce a list of dictionaries.

<div class="book"><span class="t">Dune</span><span class="y">1965</span></div>
<div class="book"><span class="t">Neuromancer</span><span class="y">1984</span></div>
Reveal solution
from html.parser import HTMLParser


class BookParser(HTMLParser):
    """Collect book titles and years without any third-party packages."""

    def __init__(self):
        super().__init__()
        self.books = []
        self.current = {}
        self.field = None

    def handle_starttag(self, tag, attrs):
        classes = dict(attrs).get("class", "")
        if tag == "div" and "book" in classes:
            self.current = {}
        elif tag == "span" and classes in ("t", "y"):
            self.field = "title" if classes == "t" else "year"

    def handle_data(self, data):
        if self.field:
            self.current[self.field] = data.strip()
            self.field = None

    def handle_endtag(self, tag):
        if tag == "div" and self.current:
            self.books.append(self.current)
            self.current = {}


parser = BookParser()
parser.feed("""
<div class="book"><span class="t">Dune</span><span class="y">1965</span></div>
<div class="book"><span class="t">Neuromancer</span><span class="y">1984</span></div>
""")

for book in parser.books:
    print(f"{book['title']:12} ({book['year']})")
Dune         (1965)
Neuromancer  (1984)

This is a state machine: it remembers what it is currently inside. That is why BeautifulSoup exists, and why it is worth the install for anything beyond a simple document.

Exercise 2

Would you scrape it?

For each, decide: scrape, use an API, or do not touch it.

  1. Product prices from a shop that has a public API.
  2. Your own posts from a forum you belong to.
  3. Email addresses from a member directory.
  4. Public weather data from a government site.
  5. Every article from a news site, to train a model.
Reveal solution
  1. Use the API. Faster, allowed, and it will not break when they change their CSS.
  2. Fine, and check whether there is an export feature first. Your own data is the easiest case there is.
  3. Do not. Harvesting personal contact details is a data protection problem in most jurisdictions and a spam problem everywhere.
  4. Usually fine, and often explicitly encouraged. Check for a bulk download; government sites frequently publish the whole dataset.
  5. Careful. Copyright applies to the articles, terms of service usually forbid it, and this is the subject of active litigation. At minimum, read the terms and prefer licensed datasets.
Exercise 3

Make a scraper survive a redesign

Your scraper breaks every time the site changes. List four things that make one more durable.

Reveal solution
  • Select on meaning, not on layout. [data-product-id] or a semantic class survives a redesign; div > div > div:nth-child(3) does not.
  • Fail loudly on structure, quietly on content. If zero items parse, raise: the page has changed. If one item is missing a field, log it and continue.
  • Cache raw HTML. When parsing breaks you can then debug against the exact page that failed, without hitting the site again.
  • Write a test with a saved page. A stored HTML fixture plus an expected result turns 'it broke' into a failing test that shows you where.

And the honest one: scrapers rot. Anything built on someone else's HTML is borrowed time, which is the strongest argument for checking one more time whether an API exists.

+100 XP
← PreviousLesson 42: The Web: HTTP and APIs