Memory and resource management

Resources include files, sockets, locks, temporary directories, and database transactions. Memory may be reclaimed automatically, but external resources still need deliberate cleanup.

Use context managers

with open("notes.txt", "w", encoding="utf-8") as file:
    file.write("Learning Python\n")
# the file is closed when the block exits

The with statement asks an object to enter a context and guarantees its exit method is called as control leaves the block, including when an exception occurs. For files, this closes the file. The same pattern works with locks, temporary resources, and database transactions.

Exceptions and deterministic cleanup

from contextlib import contextmanager

@contextmanager
def temporary_message(message):
    print("start:", message)
    try:
        yield
    finally:
        print("finish:", message)

with temporary_message("task"):
    print("working")

A context manager can be implemented with __enter__/__exit__ or with contextlib. Cleanup belongs in a finally path so it runs on normal return and exceptions. Avoid using __del__ as the primary resource-cleanup mechanism; finalization timing and shutdown behavior can be difficult to control.

Memory ownership and garbage collection

Most Python programs let the runtime manage object memory. Remove unnecessary references and avoid retaining large objects in caches or global collections longer than needed. A memory leak in a garbage-collected program often means objects remain reachable through an unintended reference, not that deallocation is missing.

Use weakref where an observer should not keep an object alive, but understand that not every type supports weak references. A cache needs an explicit size or expiration policy; garbage collection alone does not know which still-referenced values the application no longer wants.

Transactional resource patterns

For an operation that can fail midway, use a context manager that commits on success and rolls back on failure, as database transaction contexts often do. Keep cleanup behavior explicit and avoid swallowing exceptions unless the context manager intentionally handles them. A context manager should document whether it suppresses an exception.

Beginner-to-master progression

• Beginner: use with open(...) rather than remembering to close files manually.
• Practitioner: create context managers for paired setup and cleanup, and test exceptional exits.
• Advanced: reason about ownership graphs, finalization cycles, memory profiles, weak references, and the distinction between memory lifetime and external-resource lifetime.

Exercises

1. Read a file with a context manager and handle a missing-file exception.
2. Write a context manager that temporarily changes a setting and always restores it.
3. Use a profiler or tracemalloc to investigate retained allocations in a small program.

Context-manager details

The context manager's exit method receives information about an exception leaving the block and can choose whether to suppress it. Suppression should be deliberate and documented. A generator-based context manager must yield exactly once; cleanup belongs in finally. For multiple resources, nested with statements or a single with statement acquire resources in order and release them in reverse order.

from contextlib import ExitStack

paths = ["one.txt", "two.txt"]
with ExitStack() as stack:
    files = [
        stack.enter_context(open(path, encoding="utf-8"))
        for path in paths
    ]
    for file in files:
        print(file.readline().rstrip())

ExitStack is useful when the number of resources is dynamic. If opening a later file fails, already-open files are still closed as the stack exits.

Async resources and cancellation

Asynchronous resources may require async with and __aenter__/__aexit__. Cancellation can occur at an await, so cleanup must be structured around cancellation-safe context managers. Do not catch cancellation and silently continue; clean up, then generally allow cancellation to propagate unless the API explicitly owns recovery.

Resource budgets and ownership

Applications should bound file descriptors, open connections, memory buffers, task counts, and cache sizes. Automatic cleanup does not prevent unbounded resource creation. A pool or semaphore can limit concurrent use, while a timeout prevents indefinitely waiting on an unavailable resource.

Python's garbage collector manages object memory, not application-level resource policy. An object still reachable in a cache remains alive even if the application has no practical use for it. Add cache eviction, expiration, or explicit shutdown when needed.

Mastery checkpoint

1. Use ExitStack to manage a dynamic list of resources and test failure during acquisition.
2. Implement a context manager that suppresses only one documented exception type.
3. Design a resource budget for concurrent connections and explain how cancellation releases permits.