Chapter 17 of 24 | Python

File input and output

Files let programs preserve and exchange information. Use pathlib.Path for paths and a context manager so files close even if an operation fails.

from pathlib import Path

path = Path("notes.txt")
path.write_text("First line\nSecond line\n", encoding="utf-8")
text = path.read_text(encoding="utf-8")
print(text)

Text files should specify an encoding. UTF-8 is a common choice. A relative path is interpreted from the process's current working directory, which may differ from the script's folder. Make this assumption clear or construct paths relative to a known directory.

Process large files incrementally

from pathlib import Path

with Path("events.log").open(encoding="utf-8") as file:
    for line_number, line in enumerate(file, start=1):
        if "ERROR" in line:
            print(line_number, line.rstrip())

Iterating over a file reads it line by line and avoids loading the entire contents into memory. For very long lines or binary data, choose a chunking strategy. Catch expected errors such as FileNotFoundError and PermissionError at a level that can provide useful context.

Structured formats: JSON and CSV

import json
from pathlib import Path

settings = {"theme": "dark", "retries": 3}
Path("settings.json").write_text(
    json.dumps(settings, indent=2),
    encoding="utf-8",
)

Use the json module for JSON rather than manually joining strings. JSON decoding can fail, and successfully decoded data still needs validation: a field may be missing or have the wrong type. Use the csv module for CSV because quoted fields can contain commas and newlines.

Atomic updates and untrusted input

Writing directly over an important file can leave partial data if the process stops. A safer pattern writes a temporary file in the same directory, flushes it as required, then replaces the target. Exact durability guarantees depend on the operating system and filesystem. Treat file contents as untrusted input: validate sizes and fields before using them or allocating based on them.

Beginner-to-master progression

Exercises

  1. Count lines in a large file without calling read_text().
  2. Load JSON, validate required fields, and report malformed input clearly.
  3. Read CSV with quoted commas using csv.DictReader.
  4. Write a versioned settings file through a temporary replacement and document the platform guarantees you rely on.

Buffering, durability, and atomic replacement

File objects buffer writes, so a successful call to write does not necessarily mean bytes are durable on physical storage. Closing flushes Python's buffers and may report errors, but strong durability can require operating-system calls such as syncing file data and directory entries. Requirements differ between a convenience script and a database-like system.

For a safer replacement, write a new complete file in the same directory and then use Path.replace after successful serialization. Replacement is commonly atomic within a filesystem, but guarantees vary by platform; cross-filesystem moves are different. Preserve permissions and metadata deliberately when they matter, and handle interruption and cleanup of temporary files.

Bytes, streams, and decoding errors

Text mode decodes bytes into strings and may translate newline conventions. Binary mode returns bytes and leaves encoding decisions to your code. Select an error policy for invalid text: reject it, replace invalid sequences for display, or preserve bytes. Silent replacement is unsuitable when exact data integrity matters.

from pathlib import Path

raw = Path("payload.bin").read_bytes()
try:
    text = raw.decode("utf-8")
except UnicodeDecodeError as error:
    print(f"Invalid UTF-8 at byte {error.start}")

Filesystem security and path races

Paths can be influenced by users and may contain traversal components, symlinks, or unexpected file types. Resolve and constrain paths according to the application's trust boundary, but recognize that a check followed by a later open can race with filesystem changes. Security-sensitive file operations may require directory handles and platform-specific APIs to avoid time-of-check/time-of-use problems.

Never assume a file's contents are safe because its extension or location looks familiar. Enforce size limits before loading large data, validate decoded structures, and avoid unsafe object deserialization.

Mastery checkpoint

  1. Write data to a temporary file and replace a target only after complete serialization succeeds.
  2. Document whether your file operation requires atomicity, durability, or both.
  3. Compare text and binary modes for a file with non-ASCII text and newline differences.
  4. Threat-model a user-supplied path and identify traversal, symlink, and race conditions.
0% read