Testing 🧪

You are already testing

Every time you run your program and look at the output, you are testing. The problem is that you do it by hand, only for the thing you just changed, and you stop doing it when you are tired. Automated tests are the same checks, written down once, run in a second, forever.

The simplest possible test

def add(a, b):
    return a + b


assert add(2, 3) == 5
assert add(-1, 1) == 0
assert add(0, 0) == 0

print("all assertions passed")

all assertions passed

assert does nothing when the condition is true and raises AssertionError when it is false. That is genuinely a test suite. It is just one that stops at the first failure and tells you very little.

pytest: the one everybody uses

Put this in test_maths.py:

def add(a, b):
    return a + b


def test_add_positive():
    assert add(2, 3) == 5


def test_add_negative():
    assert add(-1, -1) == -2


def test_add_zero():
    assert add(5, 0) == 5

Then run pytest:

$ pip install pytest
$ pytest -v

test_maths.py::test_add_positive PASSED                    [ 33%]
test_maths.py::test_add_negative PASSED                    [ 66%]
test_maths.py::test_add_zero PASSED                        [100%]

============= 3 passed in 0.01s =============

The rules are minimal: files named test_*.py, functions named test_*, plain assert. No classes, no special methods, no boilerplate. That is why pytest won.

What a failure looks like

$ pytest

    def test_add_positive():
>       assert add(2, 3) == 6
E       assert 5 == 6
E        +  where 5 = add(2, 3)

test_maths.py:6: AssertionError
============= 1 failed, 2 passed in 0.02s =============

pytest rewrites your assertions so the failure shows both sides and what produced them. This is the feature that makes it pleasant rather than a chore.

unittest, which needs no install

import unittest


def add(a, b):
    return a + b


class TestAdd(unittest.TestCase):
    def test_positive(self):
        self.assertEqual(add(2, 3), 5)

    def test_negative(self):
        self.assertEqual(add(-1, -1), -2)

    def test_raises_on_text(self):
        with self.assertRaises(TypeError):
            add(1, "two")


import io

suite = unittest.TestLoader().loadTestsFromTestCase(TestAdd)
runner = unittest.TextTestRunner(stream=io.StringIO())     # keep the report quiet
result = runner.run(suite)

print("tests run:", result.testsRun)
print("failures:", len(result.failures))
print("errors:", len(result.errors))

tests run: 3
failures: 0
errors: 0

unittest is in the standard library, so it needs nothing installed. It is more verbose (classes, self.assertEqual instead of assert) and it is what you will find in older codebases. Learn pytest, recognise unittest.

What to actually test

def apply_discount(price, percent):
    """Reduce price by percent. Percent must be 0-100."""
    if not 0 <= percent <= 100:
        raise ValueError(f"percent must be 0-100, got {percent}")
    return round(price * (1 - percent / 100), 2)


# the normal case
assert apply_discount(100, 10) == 90.0

# the boundaries, where bugs live
assert apply_discount(100, 0) == 100.0
assert apply_discount(100, 100) == 0.0

# the awkward reality
assert apply_discount(0, 50) == 0.0
assert apply_discount(19.99, 33) == 13.39

# and that it refuses bad input
for bad in [-1, 101]:
    try:
        apply_discount(100, bad)
        raise SystemExit("should have raised!")
    except ValueError:
        pass

print("every case passed")

every case passed

The pattern to internalise: normal case, boundaries, and failure. Most bugs live at the edges, at zero, at one, at empty, at the last element, and at the input nobody expected. A test that only covers the happy path is a test that will pass while your program is broken.

Parametrising: many cases, one test

import pytest


def is_even(n):
    return n % 2 == 0


@pytest.mark.parametrize("number,expected", [
    (2, True),
    (3, False),
    (0, True),
    (-4, True),
    (-3, False),
])
def test_is_even(number, expected):
    assert is_even(number) is expected

That is five separate tests, each reported individually, from one function. When one fails you are told exactly which input broke it.

Fixtures: shared setup

import pytest


@pytest.fixture
def crew():
    """A fresh crew list for each test that asks for one."""
    return ["Guybrush", "Elaine", "Otis"]


def test_crew_size(crew):
    assert len(crew) == 3


def test_adding_does_not_leak(crew):
    crew.append("Meathook")
    assert len(crew) == 4


def test_still_three(crew):
    # the fixture ran again, so this list is untouched
    assert len(crew) == 3

A test that depends on another test having run first is a broken test. Fixtures give each one a clean slate. pytest also ships tmp_path, which hands you a fresh temporary directory, so file-touching code can be tested without leaving a mess.

Test-driven development, briefly

Write the test first, watch it fail, then make it pass. Red, green, refactor.

# 1. RED: the test, before any implementation
def test_initials():
    assert initials("guybrush ulysses threepwood") == "G.U.T."


# 2. GREEN: the simplest thing that passes
def initials(full_name):
    return ".".join(part[0].upper() for part in full_name.split()) + "."


test_initials()
print("green")

# 3. REFACTOR: now handle the edge case the test made you think about
def initials(full_name):
    """Return dotted initials, or an empty string for empty input."""
    parts = full_name.split()
    if not parts:
        return ""
    return ".".join(part[0].upper() for part in parts) + "."


assert initials("") == ""
assert initials("elaine marley") == "E.M."
print("still green, and now it survives an empty name")

green
still green, and now it survives an empty name

The real benefit of writing the test first is not discipline, it is design: you are forced to decide what the function is called, what it takes and what it returns before you can hide those decisions inside an implementation.

How much testing is enough?

• Test the logic, not the language. You do not need a test proving that + adds.
• Test anything you got wrong once. Every bug you fix deserves a test that would have caught it. This is the highest-value test you will ever write.
• Test the edges. Empty, zero, one, negative, huge, missing, wrong type.
• Do not chase 100% coverage. Coverage tells you what was executed, not what was checked. It is a smoke detector, not a safety certificate.
• Fast tests get run. A suite that takes ten minutes gets skipped, and a skipped test protects nothing.

def grade(score):
    if score > 90:
        return "A"
    elif score > 80:
        return "B"
    elif score > 70:
        return "C"
    return "F"

def grade(score):
    """Convert a percentage to a letter grade. Boundaries are inclusive."""
    if score >= 90:
        return "A"
    if score >= 80:
        return "B"
    if score >= 70:
        return "C"
    return "F"


# normal
assert grade(95) == "A"
assert grade(85) == "B"
assert grade(50) == "F"

# boundaries: this is where the bug was
assert grade(90) == "A"
assert grade(80) == "B"
assert grade(70) == "C"
assert grade(69) == "F"

# extremes
assert grade(0) == "F"
assert grade(100) == "A"

print("all 9 assertions passed")

all 9 assertions passed

# The tests, written first. These are decisions, not checks.
def check(word_frequencies):
    assert word_frequencies("") == {}
    assert word_frequencies("hi") == {"hi": 1}
    assert word_frequencies("hi hi") == {"hi": 2}
    assert word_frequencies("Hi hi") == {"hi": 2}          # case insensitive
    assert word_frequencies("hi, hi!") == {"hi": 2}        # punctuation ignored
    return "all passed"


# Now the implementation has no choice but to satisfy them.
from collections import Counter


def word_frequencies(text):
    """Count words, ignoring case and surrounding punctuation."""
    words = [w.strip(".,!?;:\"'") .lower() for w in text.split()]
    return dict(Counter(w for w in words if w))


print(check(word_frequencies))

all passed

import datetime


def test_everything():
    result = process_orders(load_orders("/Users/chris/data/orders.csv"))
    assert result
    assert result["date"] == datetime.date.today()