pytest β The Full Picture
You've seen pytest used in earlier phases. Now master it β the discovery rules, assertion introspection, marks, xfail, the full test execution model, and everything that makes test suites fast and trustworthy.
pytest uses test discovery β it automatically finds your tests without any registration or configuration. Understanding the rules means you never have a test that silently doesn't run.
| Rule | Example |
|---|---|
Files matching test_*.py or *_test.py | test_tasks.py, auth_test.py |
Functions starting with test_ | def test_create_task(): |
Methods starting with test_ in classes starting with Test | class TestAuth: def test_login(): |
| Collects from current directory recursively unless told otherwise | pytest tests/ or just pytest |
tasks_test_helper.py or your function is check_task(), pytest ignores it entirely β no error, no warning. Always follow the naming conventions or explicitly configure python_files in pyproject.toml.pytest rewrites your assert statements at collection time. When an assertion fails, it shows you exactly what both sides evaluated to β no need for assertEqual, assertIn, or any special methods.
# Different method for every comparison self.assertEqual(result, 42) self.assertIn("key", my_dict) self.assertIsNone(value) self.assertGreater(x, 0) self.assertRaises(ValueError, func)
# One keyword for everything assert result == 42 assert "key" in my_dict assert value is None assert x > 0 with pytest.raises(ValueError): func()
FAILED tests/test_tasks.py::test_priority_label AssertionError: assert 'medium' == 'high' Left side: task.priority_label β 'medium' Right side: 'high' Where: task = Task(title='Buy milk', priority=2)
import pytest # ββ @pytest.mark.skip β always skip ββ @pytest.mark.skip(reason="Feature not yet implemented") def test_export_to_pdf(): pass # ββ @pytest.mark.skipif β skip conditionally ββ import sys @pytest.mark.skipif(sys.platform == "win32", reason="Unix only") def test_symlinks(): pass # ββ @pytest.mark.xfail β expected to fail ββ # Different from skip β the test RUNS and is expected to fail # If it fails: XFAIL (expected) β green # If it passes: XPASS (unexpected pass) β warning @pytest.mark.xfail(reason="Known bug #142 β see GitHub") def test_unicode_task_titles(): task = create_task("ε¦δΉ Python") assert task.title == "ε¦δΉ Python" # ββ Custom marks β group and filter tests ββ @pytest.mark.slow # mark as slow (>1 second) @pytest.mark.integration # mark as needing external services def test_scrape_live_site(): pass # Run only fast tests: pytest -m "not slow" # Run only unit tests: pytest -m "not integration" # Run specific mark: pytest -m "slow"
[tool.pytest.ini_options] with markers = ["slow: marks tests as slow", "integration: requires external services"]. Without registration, pytest warns about unknown marks. With it, pytest --markers lists them with descriptions.# Run all tests pytest # Run with verbose output β show each test name pytest -v # Stop at first failure pytest -x # Stop after N failures pytest --maxfail=3 # Run only tests matching a name pattern pytest -k "auth" # runs test_auth_login, test_auth_logout, etc. pytest -k "create and task" # must match both words # Run a specific file or test pytest tests/test_tasks.py pytest tests/test_tasks.py::TestCreate::test_success # Show print() output (normally captured) pytest -s # Show slowest 10 tests pytest --durations=10 # Run previously failed tests first pytest --lf # last-failed pytest --ff # failed-first (then rest) # Run tests in parallel (pip install pytest-xdist first) pytest -n auto # uses all CPU cores
Fixtures & conftest.py
Fixtures are pytest's dependency injection system β they set up and tear down the things your tests need, automatically, with the right scope. Master fixtures and your test suite becomes genuinely maintainable.
Without fixtures, every test sets up its own data, creates its own objects, and cleans up after itself. This leads to hundreds of lines of repeated setup code and tests that fail for the wrong reason (bad setup, not bad code).
Fixtures extract setup into reusable, named functions that pytest automatically calls and injects. They can also clean up after themselves using yield.
import pytest from myapp.models import Task, TaskManager # ββ Basic fixture β setup only ββ @pytest.fixture def sample_task(): return Task(title="Write tests", priority=3) # ββ Fixture with teardown β use yield ββ @pytest.fixture def task_manager(tmp_path): # tmp_path is a built-in pytest fixture β gives a temp directory db_file = tmp_path / "test_tasks.json" manager = TaskManager(filepath=str(db_file)) yield manager # everything above = setup db_file.unlink(missing_ok=True) # everything below = teardown (always runs) # ββ Fixture using another fixture ββ @pytest.fixture def manager_with_tasks(task_manager): task_manager.add("Task A", priority=1) task_manager.add("Task B", priority=5) task_manager.add("Task C", priority=3, due="2026-01-01") return task_manager # ββ Tests request fixtures by name ββ def test_task_creation(sample_task): assert sample_task.title == "Write tests" assert sample_task.priority == 3 assert sample_task.done == False def test_pending_count(manager_with_tasks): pending = list(manager_with_tasks.pending()) assert len(pending) == 3 def test_sorted_by_priority(manager_with_tasks): pending = list(manager_with_tasks.pending()) assert pending[0].priority == 5 # highest first
By default, fixtures are created and destroyed for each test. This is safe but slow for expensive setup. Scope lets you control lifetime.
| Scope | Created Once Per⦠| Use For |
|---|---|---|
function (default) | Each test function | Mutable state β each test gets a clean copy |
class | Each test class | Shared setup across methods in one class |
module | Each test file | Expensive setup shared across one file |
session | Entire test run | Database connections, heavy imports, API clients |
@pytest.fixture(scope="session") def anthropic_client(): """Created once for the whole test run β expensive import.""" import anthropic return anthropic.Anthropic() # No teardown needed β client has no cleanup @pytest.fixture(scope="session") def test_database(): """In-memory DB β created once, shared across all tests.""" from sqlmodel import create_engine, SQLModel engine = create_engine("sqlite://") SQLModel.metadata.create_all(engine) yield engine SQLModel.metadata.drop_all(engine) @pytest.fixture # function scope β fresh session per test def db_session(test_database): from sqlmodel import Session with Session(test_database) as session: yield session session.rollback() # undo every change after each test
conftest.py is a special file pytest loads automatically. Fixtures defined there are available to every test in the same directory and all subdirectories β no import needed.
tmp_path (temporary directory, Path object), tmp_path_factory (session-scoped temp dir), capsys (capture stdout/stderr), monkeypatch (patch environment variables, functions, attributes safely), caplog (capture log output). These are available in every pytest run without any imports.Mocking & Patching
Tests should be fast, isolated, and not cost money. Mocking replaces real external services β databases, APIs, file systems, time β with controlled fakes. Master this and no code is untestable.
Without mocking, testing code that calls an external API means: making real network calls (slow, costs money, flaky), needing credentials in CI, and test results depending on external service uptime. Mocking eliminates all of this.
from unittest.mock import Mock, MagicMock, patch, call # ββ Mock β a fake object that records calls ββ mock = Mock() mock.some_method("arg1", key="val") # Inspect what happened print(mock.some_method.call_count) # 1 mock.some_method.assert_called_once_with("arg1", key="val") mock.some_method.assert_called_with("arg1", key="val") # Configure return values mock.get_user.return_value = {"id": 1, "name": "Jude"} result = mock.get_user(1) assert result["name"] == "Jude" # Configure to raise an exception mock.dangerous_call.side_effect = ConnectionError("Network down") with pytest.raises(ConnectionError): mock.dangerous_call()
patch temporarily replaces a real object in your code with a mock during the test. The key rule: patch where the name is used, not where it's defined.
from unittest.mock import patch, MagicMock import pytest # The code under test (in weather.py): # import requests # def get_weather(city): return requests.get(...).json() # ββ patch as decorator ββ @patch("weather.requests.get") # patch in the module that USES it def test_get_weather_success(mock_get): # Configure what the mock returns mock_response = MagicMock() mock_response.json.return_value = { "current": {"temperature_2m": 28.5, "wind_speed_10m": 12.0} } mock_get.return_value = mock_response from weather import get_weather result = get_weather("Bossier City") assert result["temp_c"] == 28.5 mock_get.assert_called_once() # verify it was called # ββ patch as context manager ββ def test_get_weather_timeout(): with patch("weather.requests.get") as mock_get: mock_get.side_effect = TimeoutError() from weather import get_weather with pytest.raises(TimeoutError): get_weather("Bossier City") # ββ patch as fixture (pytest style β cleaner) ββ @pytest.fixture def mock_requests(monkeypatch): mock_response = MagicMock() mock_response.json.return_value = {"data": "test"} mock_response.status_code = 200 monkeypatch.setattr("requests.get", MagicMock(return_value=mock_response)) return mock_response
from unittest.mock import patch from datetime import datetime # ββ Freeze time β test time-sensitive code reliably ββ @patch("myapp.tasks.datetime") def test_task_created_today(mock_dt): mock_dt.now.return_value = datetime(2026, 6, 15) task = create_task("Test") assert task.created_date == "2026-06-15" # ββ Patch environment variables ββ def test_api_key_missing(monkeypatch): monkeypatch.delenv("OPENAI_API_KEY", raising=False) with pytest.raises(RuntimeError, match="OPENAI_API_KEY"): from myapp.config import get_settings get_settings() def test_with_test_api_key(monkeypatch): monkeypatch.setenv("OPENAI_API_KEY", "test-key-123") from myapp.config import get_settings settings = get_settings() assert settings.openai_api_key == "test-key-123"
tasks.py imports requests and calls requests.get(). You want to mock the HTTP call. Which patch target is correct?Parametrize & Coverage
Test many inputs with one test function using parametrize. Measure what percentage of your code is actually exercised with coverage β and understand what the number does and doesn't tell you.
import pytest from myapp.models import Task # ββ Basic parametrize β one argument ββ @pytest.mark.parametrize("priority", [1, 2, 3, 4, 5]) def test_valid_priorities(priority): task = Task(title="Test", priority=priority) assert task.priority == priority # ββ Parametrize β multiple arguments ββ @pytest.mark.parametrize("title, priority, expect_error", [ ("Valid title", 3, False), # normal case ("", 3, True), # empty title ("Valid", 0, True), # priority too low ("Valid", 6, True), # priority too high (" ", 3, True), # whitespace only title ("A" * 201, 3, True), # title too long ]) def test_task_validation(title, priority, expect_error): if expect_error: with pytest.raises(ValueError): Task(title=title, priority=priority) else: task = Task(title=title, priority=priority) assert task.priority == priority # ββ Parametrize with pytest.param β add IDs and marks ββ @pytest.mark.parametrize("email, valid", [ pytest.param("jude@example.com", True, id="valid_email"), pytest.param("not-an-email", False, id="missing_at"), pytest.param("@nodomain", False, id="missing_local"), pytest.param("a@b.c", True, id="minimal_valid"), pytest.param("", False, id="empty"), ]) def test_email_validation(email, valid): from myapp.utils import is_valid_email assert is_valid_email(email) == valid
pip install pytest-cov # Run tests with coverage report pytest --cov=myapp --cov-report=term-missing # Output: ---------- coverage: platform win32 ---------- Name Stmts Miss Cover Missing ----------------------------------------------------- myapp/__init__.py 0 0 100% myapp/models.py 45 3 93% 78-80 myapp/services.py 62 8 87% 34, 55-60, 89 myapp/auth.py 38 12 68% 45-52, 67-70 ----------------------------------------------------- TOTAL 145 23 84% # Generate HTML report (browsable) pytest --cov=myapp --cov-report=html # Open htmlcov/index.html β click each file to see line-by-line coverage # Fail if coverage drops below threshold pytest --cov=myapp --cov-fail-under=80
100% coverage doesn't mean 100% correct
Coverage measures which lines were executed during tests, not whether they were tested with the right inputs. You can achieve 100% line coverage with one test that calls every function with valid inputs and never tests error cases, edge cases, or invalid data.
Coverage is a floor, not a ceiling. Use it to find untested code (coverage = 0% for a function means no tests reach it). Don't use it as a quality metric β a test that calls a function without asserting anything counts as covered.
A practical target: 80-90% for application code is realistic and meaningful. Chasing 100% usually means writing trivial tests for trivial code.
mypy β Static Type Checking
mypy reads your type hints and catches type errors before your code ever runs. It's like a compiler for Python β finds the bugs that tests miss because tests only check what you thought to test.
from typing import Optional def get_task_title(task_id: int) -> str: task = db.get(task_id) # returns Optional[Task] return task.title # mypy ERROR: task could be None! # Tests might not catch this if db.get() always returns a value # Fix: def get_task_title(task_id: int) -> Optional[str]: task = db.get(task_id) return task.title if task else None # ββ mypy catches wrong argument types ββ def set_priority(task_id: int, priority: int) -> None: pass set_priority("42", 3) # mypy ERROR: Argument 1 has type "str", expected "int" # ββ mypy catches unreachable code ββ def process(items: list[str]) -> str: if not items: return "" return items[0] return "unreachable" # mypy: Statement is unreachable
pip install mypy # Basic check mypy myapp/ # Strict mode β catches more issues, requires full annotations mypy --strict myapp/ # Check a single file mypy myapp/models.py
[tool.mypy] python_version = "3.12" strict = true # enables all strict checks warn_return_any = true warn_unused_ignores = true # Ignore third-party libraries with no stubs [[tool.mypy.overrides]] module = ["anthropic.*", "bs4.*"] ignore_missing_imports = true
mypy --ignore-missing-imports myapp/models.py on the most critical file. Fix those errors. Then expand to more files over time. Adding # type: ignore to a line suppresses that specific error when you can't fix it immediately β use it sparingly as a TODO marker.Ruff β The Modern Python Linter
Ruff replaces flake8, pylint, isort, and pyupgrade in a single tool. It's written in Rust β 10-100x faster than the tools it replaces β and is the current industry standard.
| Old Tool | What It Did | Ruff Replacement |
|---|---|---|
| flake8 | Style and logic checks (PEP 8) | ruff check |
| isort | Sort import statements | ruff check --fix (I rules) |
| pyupgrade | Modernize syntax to newer Python | ruff check --fix (UP rules) |
| pylint | Deep code analysis, conventions | Ruff (partial β mypy for types) |
| black | Code formatting | ruff format |
pip install ruff # Check for issues ruff check . # Fix auto-fixable issues (import sorting, unused imports, etc.) ruff check --fix . # Format code (like black) ruff format . # Check + format in one command ruff check --fix . && ruff format . # Check a single file ruff check myapp/models.py # Show all available rules ruff rule --all
[tool.ruff] line-length = 100 target-version = "py312" [tool.ruff.lint] select = [ "E", # pycodestyle errors "F", # pyflakes (undefined names, unused imports) "I", # isort (import sorting) "UP", # pyupgrade (use f-strings, newer syntax) "B", # bugbear (common bugs and design issues) "SIM", # simplify (simplifiable conditions) "N", # pep8-naming conventions ] ignore = [ "E501", # line too long β handled by formatter ] [tool.ruff.lint.per-file-ignores] "tests/*" = ["S101"] # allow assert in tests
# F401 β imported but unused import os, sys, json # ruff: 'os' imported but unused # F841 β assigned but never used result = expensive_call() # ruff: local variable 'result' is assigned but never used # UP β use f-strings instead of format() name = "Jude" "Hello, {}".format(name) # ruff UP032: use f-string f"Hello, {name}" # correct # B β common bugs def append_to(item, lst=[]): # ruff B006: mutable default argument! lst.append(item) return lst # SIM β simplifiable conditions if x == True: # ruff SIM210: use 'if x:' pass if not x == None: # ruff SIM201: use 'if x is not None:' pass # E711 β comparison to None if x == None: # ruff E711: use 'is None' pass
Pre-commit Hooks β Automatic Quality Gates
Pre-commit hooks run automatically before every git commit β checking formatting, linting, type errors, and secrets. Bad code never reaches your repository.
The pre-commit framework manages a set of hooks that run before each commit. If any hook fails, the commit is blocked until you fix the issue. This enforces standards automatically β no "I forgot to run Ruff" excuses.
- Install the framework:
pip install pre-commit - Create
.pre-commit-config.yamlin your project root - Install the hooks into your local git repo:
pre-commit install - Now every
git commitruns the hooks first. Failures block the commit. - Run manually on all files at any time:
pre-commit run --all-files
repos: # ββ Ruff: lint and format ββ - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.4.0 hooks: - id: ruff args: [--fix] # auto-fix what can be auto-fixed - id: ruff-format # format like black # ββ Basic file hygiene ββ - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.6.0 hooks: - id: trailing-whitespace # remove trailing spaces - id: end-of-file-fixer # ensure files end with newline - id: check-yaml # validate YAML syntax - id: check-toml # validate TOML syntax - id: check-json # validate JSON syntax - id: check-merge-conflict # catch unresolved merge conflicts - id: debug-statements # catch leftover breakpoint()/pdb.set_trace() - id: detect-private-key # !! catch accidentally committed secrets # ββ mypy: type checking ββ - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.10.0 hooks: - id: mypy additional_dependencies: [pydantic, sqlmodel]
# Skip ALL hooks (emergency commit β use sparingly) git commit --no-verify -m "Emergency fix" # Skip a specific hook SKIP=mypy git commit -m "WIP β types to fix later" # Run just one hook manually pre-commit run ruff pre-commit run mypy --all-files # Update all hooks to latest versions pre-commit autoupdate
Profiling & Performance
Never optimize without measuring first. Profiling shows you exactly which function is slow and how many times it's called β the 80/20 rule almost always applies.
Donald Knuth: "Premature optimization is the root of all evil." The practical version: write clear code first, measure where it's actually slow, then optimize only that part. Profiling is how you find the actual bottleneck β not where you think it is.
import cProfile, pstats, io # ββ Profile a function ββ def profile(func, *args, **kwargs): pr = cProfile.Profile() pr.enable() result = func(*args, **kwargs) pr.disable() s = io.StringIO() stats = pstats.Stats(pr, stream=s).sort_stats("cumulative") stats.print_stats(20) # top 20 functions by cumulative time print(s.getvalue()) return result profile(build_index, documents) # ββ Command line profiling ββ # python -m cProfile -s cumulative myscript.py # ββ Profile output explanation ββ ncalls tottime percall cumtime percall filename:lineno(function) 1 0.001 0.001 4.523 4.523 main.py:12(build_index) 500 0.012 0.000 4.510 0.009 rag.py:28(get_embedding) 500 3.891 0.008 4.498 0.009 openai/resources.py:... 1 0.000 0.000 0.012 0.012 rag.py:45(chunk_text) # ncalls = how many times called # tottime = time IN this function only # cumtime = total time including called functions β sort by this
pip install line_profiler
from line_profiler import LineProfiler def slow_function(items: list) -> list: results = [] for item in items: processed = expensive_transform(item) # β is this the slow part? if passes_filter(processed): # β or this? results.append(processed) return results profiler = LineProfiler() profiler.add_function(slow_function) profiler.run("slow_function(data)") profiler.print_stats() # Output shows time per line: Line # Hits Time Per Hit % Time Line Contents 4 1000 234 0.2 2% processed = expensive_transform(item) 5 1000 9834 9.8 98% if passes_filter(processed): β HERE # passes_filter is the bottleneck β optimize that, not the transform
import timeit # Compare two approaches list_approach = timeit.timeit( "[x**2 for x in range(1000)]", number=10000 ) map_approach = timeit.timeit( "list(map(lambda x: x**2, range(1000)))", number=10000 ) print(f"List comp: {list_approach:.3f}s") print(f"map(): {map_approach:.3f}s") # In Jupyter / IPython: use %timeit magic # %timeit [x**2 for x in range(1000)]
Documentation & README Writing
Code that can't be understood can't be used, maintained, or trusted. Learn docstrings, type-driven self-documentation, and how to write a README that makes people want to use your project.
# ββ Level 1: One-liner β for obvious functions ββ def is_overdue(due: date) -> bool: """Return True if the due date is before today.""" return due < date.today() # ββ Level 2: Multi-line β for non-obvious functions ββ def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]: """Split text into overlapping chunks for RAG indexing. Chunks overlap to preserve context at boundaries β a sentence split across two chunks will appear in both. Args: text: The input text to split. chunk_size: Approximate number of words per chunk. overlap: Number of words to repeat between adjacent chunks. Returns: List of text chunks. Empty list if text is empty. Example: >>> chunks = chunk_text("one two three", chunk_size=2, overlap=1) >>> chunks ['one two', 'two three'] """ if not text.strip(): return [] # ... implementation # ββ Level 3: Class docstring β describe purpose and usage ββ class TaskManager: """Manages a persistent collection of tasks. Tasks are stored as JSON at the given filepath. All operations are immediately persisted to disk. Thread-safety is not guaranteed. Attributes: filepath: Path to the JSON storage file. Example: >>> mgr = TaskManager("tasks.json") >>> task = mgr.add("Write tests", priority=3) >>> list(mgr.pending()) [Task(title='Write tests', priority=3, done=False)] """
# Project Name One sentence. What does this do and who is it for? ## Features - Feature 1 β described from the user's perspective - Feature 2 - Feature 3 ## Quick Start ```bash git clone https://github.com/you/project cd project python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.example .env # fill in your keys python main.py ``` ## Usage Show a real, copy-pasteable example. Not "use --help". ```bash python cli.py add "Write tests" --priority 4 python cli.py list python cli.py ai ask "what's due this week?" ``` ## Configuration | Variable | Required | Description | |---|---|---| | OPENAI_API_KEY | Yes | Your OpenAI API key | | DATABASE_URL | No | Defaults to SQLite | ## Development ```bash pip install -r requirements-dev.txt pre-commit install pytest ``` ## License MIT
Harden the Command Center
Take everything built across Phases 1β6 and make it genuinely production-quality. Tests, types, lint pipeline, pre-commit hooks, profiling, documentation β the full professional treatment.
You have a working project. This phase's project isn't building new features β it's doing the work that separates code you're proud to show from code you hope no one reads.
β pytest suite β 90%+ coverage on core business logic (models.py, services.py). Tests organised into classes. At least 3 uses of
@pytest.mark.parametrize.
β Fixtures in conftest.py β all shared setup extracted. No duplicated setup code across test files.
β All external calls mocked β HTTP requests, Anthropic API, file writes in unit tests. Integration tests clearly separated.
β mypy passes β
mypy myapp/ --ignore-missing-imports returns zero errors.
β Ruff passes β
ruff check . returns zero issues.
β pre-commit installed β hooks run on every commit. Team members can onboard with
pre-commit install.
β Profiled β run cProfile on the slowest operation. Document findings in a comment: what's slow, what you did or would do about it.
β All public functions have docstrings β one-liner minimum for simple functions, full Args/Returns for complex ones.
β README passes the 30-second test β quick start, real usage examples, config table.
β CI-ready β all checks can be run with a single command:
pytest && mypy myapp/ && ruff check .
After you've completed the checklist, run these commands and aim for these results:
| Command | Target | Meaning |
|---|---|---|
pytest --cov=myapp | > 85% coverage | Most code is exercised by tests |
mypy myapp/ | 0 errors | No type inconsistencies |
ruff check . | 0 issues | Code follows style and avoids common bugs |
pre-commit run --all-files | All passed | Every automated check green |
pytest --durations=5 | Suite < 10s | Tests run fast enough to run constantly |
After the baseline is solid:
- Add
pytest-xdistand run your suite in parallel (pytest -n auto) β measure the speedup - Add mutation testing with
mutmutβ it deliberately breaks your code to see if your tests catch it. A test suite that doesn't catch mutations isn't really testing - Write a
CONTRIBUTING.mdexplaining the development workflow, how to run tests, and the pre-commit setup - Profile the RAG indexing pipeline and identify the actual bottleneck. Can you cache embeddings to avoid recomputing them?