Chapter 20 of 24 | Python

Advanced language topics

Advanced Python is less about collecting syntax tricks and more about understanding protocols, execution, and abstraction costs. Learn the simpler model first, then use advanced features when they clarify a real requirement.

Iterators and generators

def read_nonempty_lines(path):
    with open(path, encoding="utf-8") as file:
        for line in file:
            cleaned = line.strip()
            if cleaned:
                yield cleaned

for line in read_nonempty_lines("notes.txt"):
    print(line)

A generator function pauses at yield and resumes when the next value is requested. It processes data lazily and can handle streams larger than memory, but it is usually a one-pass iterator. Generator expressions provide a compact lazy form. Resource ownership should remain clear: keep file iteration inside the context manager so the file is closed when iteration ends or the generator is closed.

Decorators

from functools import wraps
from time import perf_counter

def timed(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        start = perf_counter()
        try:
            return function(*args, **kwargs)
        finally:
            print(f"{function.__name__}: {perf_counter() - start:.6f}s")
    return wrapper

@timed
def calculate():
    return sum(range(1000))

A decorator receives a function and returns a replacement, often a wrapper. functools.wraps preserves useful metadata. Decorators can implement logging, caching, registration, and access policies, but hidden control flow can make debugging harder. Keep their effects documented and test exceptions as well as successful returns.

Descriptors and the data model

Python syntax often delegates behavior to special methods: iteration uses __iter__/__next__, context managers use __enter__/__exit__, and attribute access can involve descriptors. A descriptor implements methods such as __get__ or __set__ and is stored on a class to control attribute behavior. Properties are a convenient descriptor-based tool. Custom descriptors are appropriate for reusable validation or managed attributes, but can obscure simple state.

Async programming

import asyncio

async def fetch_label(delay, label):
    await asyncio.sleep(delay)
    return label

async def main():
    labels = await asyncio.gather(
        fetch_label(0.1, "A"),
        fetch_label(0.1, "B"),
    )
    print(labels)

asyncio.run(main())

async def defines a coroutine function. Calling it creates a coroutine; it does not run to completion until awaited or scheduled. Async I/O is useful when many operations spend time waiting, not as automatic CPU parallelism. Blocking calls inside the event loop can stall every task. Use structured concurrency facilities available in the Python version you target, and ensure cancellation and exceptions are handled deliberately.

Metaclasses and advanced typing

Metaclasses customize class creation; most application code should prefer decorators, class decorators, descriptors, or explicit registration functions. Type variables, generics, protocols, overloads, and gradual typing can improve large APIs. Static type checkers approximate Python behavior, so annotations should express intent without being mistaken for runtime validation.

Beginner-to-master progression

Exercises

  1. Write a generator that yields records from a large input one at a time.
  2. Write a decorator that retries a function only for a specific exception, with a bounded retry count.
  3. Compare sequential waiting with asyncio.gather for simulated I/O.
  4. Inspect a built-in descriptor such as property and explain what attribute access invokes.

Coroutine lifecycle and structured concurrency

A coroutine object is a suspended computation with state. Creating it does not execute its body; awaiting it drives it forward. Tasks schedule coroutines on an event loop and expose results or exceptions. If a parent task exits while children continue, their failures and resources can become detached. Structured concurrency keeps child tasks within a lexical lifetime so cancellation and errors are collected before the scope exits.

Python 3.11 introduced asyncio.TaskGroup for structured task management. A task group cancels related tasks when an unhandled child failure occurs and reports failures after the group exits. This changes error shape and timing compared with manually created tasks, so understand cancellation propagation and exception groups when using it.

Descriptors and attribute lookup

Attribute access is governed by lookup rules involving data descriptors, instance dictionaries, non-data descriptors, and class attributes. A descriptor with __set__ or __delete__ is a data descriptor and normally takes precedence over an instance attribute. Functions are non-data descriptors that bind an instance to produce a method. Properties use the descriptor protocol to implement managed attributes.

This machinery powers ORM fields, validation frameworks, and class methods. It can also make code behavior invisible when overused. Before writing a descriptor, check whether a property, composition, or explicit method is easier to understand.

Typing beyond basic annotations

Python's type system is gradual and static tools approximate a dynamic language. Generics preserve relationships between types; protocols describe structural behavior; overloads describe several valid call shapes. Variance determines whether a generic type can safely substitute a more specific or general type. Mutable containers are usually invariant because a value safe to read may not be safe to write through a wider type.

Annotations do not validate external data at runtime, and introspection can evaluate forward references in ways that execute code. Treat type information from untrusted sources cautiously. Use a runtime validation library or explicit parsing for external payloads.

Metaprogramming costs

Metaclasses, dynamic class creation, monkey patching, and import hooks can build frameworks, but they make control flow and static analysis harder. Prefer explicit registration, decorators, or ordinary composition when possible. If metaprogramming is necessary, keep the mechanism localized and expose a simple public interface.

Mastery checkpoint

  1. Use a task group for concurrent work and document how one child failure affects siblings.
  2. Trace attribute lookup for an instance attribute, a property, and a bound method.
  3. Write a protocol and a generic helper, then verify both with a static type checker.
  4. Replace an unnecessary metaprogramming feature with an explicit design and compare maintainability.
0% read