Making Python Fast ⚡

The rule: measure, do not guess

import timeit

setup = "data = list(range(1000))"

loop = timeit.timeit("total = 0\nfor n in data: total += n", setup=setup, number=1000)
builtin = timeit.timeit("total = sum(data)", setup=setup, number=1000)

print(f"explicit loop faster than sum? {loop < builtin}")
print(f"sum is at least twice as fast: {loop / builtin > 2}")

explicit loop faster than sum? False
sum is at least twice as fast: True

Note what is being asserted there: a direction, not a number. The exact ratio depends on your machine, your Python version and what else is running, so a lesson that promised "4.7 times faster" would be wrong for most readers. Benchmarks you publish should claim only what they can defend.

cProfile: where the time actually goes

import cProfile
import io
import pstats


def slow_lookup(names, targets):
    """Deliberately quadratic: a list scan inside a loop."""
    return [name for name in targets if name in names]


def fast_lookup(names, targets):
    lookup = set(names)
    return [name for name in targets if name in lookup]


names = [f"pirate{i}" for i in range(4000)]
targets = [f"pirate{i}" for i in range(0, 4000, 4)]

profiler = cProfile.Profile()
profiler.enable()
slow = slow_lookup(names, targets)
fast = fast_lookup(names, targets)
profiler.disable()

buffer = io.StringIO()
pstats.Stats(profiler, stream=buffer).sort_stats("cumulative").print_stats(0)

print("same answer:", slow == fast)
print("matched:", len(fast))

same answer: True
matched: 1000

$ python -m cProfile -s cumulative myscript.py | head -15

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    2.417    2.417 myscript.py:1(<module>)
        1    2.301    2.301    2.301    2.301 myscript.py:4(slow_lookup)
        1    0.004    0.004    0.004    0.004 myscript.py:9(fast_lookup)

tottime is time inside that function alone; cumtime includes everything it called. Sort by cumulative to find the expensive branch, then by tottime to find the expensive line. One command, and the guessing stops.

Algorithms beat micro-optimisation, always

import time

names = [f"pirate{i}" for i in range(20000)]
targets = [f"pirate{i}" for i in range(0, 20000, 2)]

start = time.perf_counter()
slow = [n for n in targets if n in names]          # list: scans every time
slow_time = time.perf_counter() - start

lookup = set(names)
start = time.perf_counter()
fast = [n for n in targets if n in lookup]         # set: instant
fast_time = time.perf_counter() - start

print(f"same result: {slow == fast}")
print(f"set version at least 20x faster: {slow_time / fast_time > 20}")

same result: True
set version at least 20x faster: True

That table is worth more than every micro-optimisation in this lesson combined. A loop inside a loop over the same data is the accidental quadratic, and it is the single most common cause of "it was fine in testing and unusable in production".

The practical speedups, in order of value

import time

# 1. Do less work: move invariant things out of the loop
data = list(range(20000))

start = time.perf_counter()
result_bad = [x * len(data) for x in data]           # len() every iteration
bad = time.perf_counter() - start

start = time.perf_counter()
size = len(data)
result_good = [x * size for x in data]
good = time.perf_counter() - start

print("same:", result_bad == result_good)

# 2. Build strings with join, not +=
start = time.perf_counter()
text = ""
for i in range(20000):
    text += str(i)            # a new string every time
concat = time.perf_counter() - start

start = time.perf_counter()
joined = "".join(str(i) for i in range(20000))
join = time.perf_counter() - start

print("same text:", text == joined)
print("join at least as fast:", join <= concat * 1.5)

same: True
same text: True
join at least as fast: True

1. Pick the right data structure. Sets and dicts for lookup. Usually a 100x win or more.
2. Do less. Cache with functools.lru_cache, hoist invariants out of loops, and stop computing things nobody reads.
3. Use the built-ins. sum, min, sorted, any and str.join run in C.
4. Use generators when you do not need the whole list (Lesson 34).
5. Reach for NumPy for numeric arrays. Often 50x, sometimes far more (Lesson 46).
6. Then consider concurrency (Lessons 39 and 40), which helps waiting, not computing.
7. Then a faster language for the one function that matters.

Free speed from newer Pythons

Python has been getting substantially faster. The Faster CPython project delivered large gains in 3.11, more in 3.12 and 3.13, and 3.14 continues. Upgrading is often the cheapest performance work available: no code changes, and a benchmark suite you did not have to write.

$ python3.11 bench.py
2.41s

$ python3.13 bench.py
1.52s

# the same code, no changes

When Python genuinely is not enough

Calling Rust from Python

This is the pattern behind the fastest tools in the Python ecosystem, and it is much more approachable than it sounds. You write the 5% that is slow in Rust and import it as an ordinary Python module.

// src/lib.rs
use pyo3::prelude::*;

/// Count how many numbers below `limit` are prime.
#[pyfunction]
fn count_primes(limit: u64) -> u64 {
    (2..limit)
        .filter(|n| (2..=(*n as f64).sqrt() as u64).all(|d| n % d != 0))
        .count() as u64
}

#[pymodule]
fn fastmath(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(count_primes, m)?)?;
    Ok(())
}

# and then, from Python
import fastmath

print(fastmath.count_primes(1_000_000))      # the same answer, dramatically faster

$ pip install maturin
$ maturin init --bindings pyo3
$ maturin develop          # builds the Rust and installs it into your venv

def find_duplicates(records):
    duplicates = []
    for record in records:
        count = 0
        for other in records:
            if record["id"] == other["id"]:
                count += 1
        if count > 1 and record["id"] not in [d["id"] for d in duplicates]:
            duplicates.append(record)
    return duplicates

from collections import Counter
import time


def find_duplicates_fast(records):
    """One pass to count, one pass to collect. O(n)."""
    counts = Counter(r["id"] for r in records)
    seen = set()
    result = []
    for record in records:
        if counts[record["id"]] > 1 and record["id"] not in seen:
            seen.add(record["id"])
            result.append(record)
    return result


records = [{"id": i % 500, "name": f"r{i}"} for i in range(4000)]

start = time.perf_counter()
fast = find_duplicates_fast(records)
elapsed = time.perf_counter() - start

print(f"{len(fast)} duplicate ids found")
print(f"fast enough: {elapsed < 0.1}")

500 duplicate ids found
fast enough: True

import functools
import time


@functools.cache
def expensive(n):
    """Pretend this is a slow API call or a heavy computation."""
    total = sum(i * i for i in range(n))
    return total


start = time.perf_counter()
first = expensive(200_000)
first_time = time.perf_counter() - start

start = time.perf_counter()
second = expensive(200_000)
second_time = time.perf_counter() - start

print(f"same answer: {first == second}")
print(f"second call much faster: {second_time < first_time / 10}")
print(expensive.cache_info().hits, "cache hit")

same answer: True
second call much faster: True
1 cache hit