Numerical computing: precision, error, and reliable results

Understand how computers represent real numbers, why rounding and cancellation occur, and how numerical algorithms are chosen, tested, and applied responsibly.

What numerical computing does

Numerical computing uses algorithms to approximate mathematical quantities with finite computational resources. It underpins simulation, graphics, weather prediction, engineering, statistics, and machine learning.

A computer represents only a finite set of numbers, while real numbers include infinitely many values. Numerical algorithms therefore introduce approximation. The goal is not always an exact answer; it is a useful result with error understood and controlled well enough for the task.

• Model error: the mathematical model does not fully represent the real system.
• Data error: measurements are noisy or limited.
• Representation error: values are rounded to finite precision.
• Algorithmic error: the method approximates a solution or accumulates rounding.

These error sources are different. A highly accurate computation of an inaccurate model may still produce a misleading prediction.

Representing real numbers

Floating-point numbers represent a value approximately as a sign, significand, and exponent. This is similar to scientific notation, where a number is written as a limited number of significant digits multiplied by a power of a base.

The spacing between representable values changes with magnitude. Near one, nearby floating-point numbers are close together; at very large magnitudes, the gap grows. A value may be rounded to the closest representable number, and some values such as decimal 0.1 do not have a finite binary representation.

Why 0.1 + 0.2 may not equal 0.3

Many programming languages use binary floating-point for ordinary decimal literals. The values 0.1 and 0.2 are approximations in binary, and their sum may be slightly different from the stored approximation of 0.3. A displayed result may round the value for readability, hiding the small difference.

0.1 + 0.2  // may be represented as 0.30000000000000004

For approximate measurements, compare with a tolerance appropriate to scale and units. For money or exact decimal rules, use an appropriate decimal or integer representation rather than assuming binary floating-point represents every decimal exactly.

Overflow, underflow, and special values

Overflow occurs when a result exceeds the representable range; underflow occurs when a nonzero value is too small to represent normally. IEEE 754 floating-point also defines special values such as positive and negative infinity, NaN (not a number), and signed zero. Language behavior and hardware support should be checked for the target system.

Rounding and accumulated error

Each floating-point operation may round its exact mathematical result to a representable value. A single rounding error may be tiny, but repeated operations can accumulate or interact with later calculations. The final error depends on the algorithm and data, not only on the number of decimal digits printed.

Absolute error is the magnitude of the difference from a reference value. Relative error scales that difference by the magnitude of the reference. Absolute error can be more informative near zero; relative error can be more meaningful across values with different scales.

Rounding policy matters in applications. Financial rules may require a particular decimal scale and tie-breaking convention. Scientific results may need uncertainty intervals. Always specify units and precision when reporting a result.

Cancellation

Catastrophic cancellation occurs when subtracting two nearly equal approximate values removes leading significant digits and leaves a result dominated by rounding error. Algebraically equivalent formulas can have very different numerical behavior.

For example, computing the difference between two nearby large numbers can lose precision. A numerically stable reformulation may avoid subtracting the nearly equal quantities directly. The right method depends on the mathematical problem and expected input range.

Conditioning and stability

Conditioning describes how sensitive a mathematical problem is to small changes in its input. A well-conditioned problem changes its output modestly when inputs change slightly; an ill-conditioned problem can amplify tiny input differences.

Stability describes whether an algorithm controls the additional error it introduces while solving a problem. A stable algorithm cannot make an ill-conditioned problem insensitive, but it can avoid making the situation worse than necessary.

These ideas distinguish problem difficulty from algorithm quality. If a weather model is highly sensitive to initial measurements, the problem may be ill-conditioned in that regime. A stable numerical method can still be the best available approach, while the result should reflect input uncertainty.

Forward and backward error

Forward error measures how far a computed answer is from the exact answer to the original problem. Backward error asks how much the input would need to change for the computed answer to be exact. A method with small backward error is often considered numerically stable, especially when the original problem is well-conditioned.

In practice, exact reference answers may not be available. Compare against higher precision or a trusted independent method, use known bounds, and check whether residuals satisfy expected properties.

Numerical algorithms

Numerical methods solve problems such as roots of equations, systems of linear equations, integration, and differential equations. Methods trade off accuracy, speed, memory, and convergence assumptions.

• Bisection: repeatedly halves an interval that brackets a root; reliable when the function changes sign and is continuous.
• Newton's method: uses a local derivative to approach a root; can converge quickly but depends on a good starting point and derivative behavior.
• Numerical integration: approximates an area using weighted samples; accuracy depends on smoothness and sampling.
• Iterative solvers: improve an approximate solution step by step and stop when an error measure meets a tolerance.

An algorithm should define a stopping rule. “Repeat until accurate” is incomplete: specify a tolerance, maximum work, and behavior if convergence fails. A small residual is useful evidence but does not always guarantee a small solution error for an ill-conditioned problem.

Choose tolerances with scale in mind

A fixed absolute tolerance may be too strict for very large values and too loose for values near zero. Relative and absolute tolerances are often combined. Tolerances should reflect input precision, the application's needs, and expected numerical conditioning rather than being chosen merely to make a test pass.

Worked example: sum a sequence

Suppose a program adds many measured decimal values. A left-to-right sum rounds after each addition. If values vary greatly in magnitude, a small value may be lost when added to a much larger running total.

total = 0
for value in measurements:
    total = total + value

For a sequence containing a large value, a small value, and the negative of the large value, the small contribution may disappear depending on precision and order. Reordering additions can change the result because floating-point addition is not perfectly associative.

One improvement is compensated summation, which tracks some low-order information lost during addition. Pairwise summation adds values in a tree-like structure and can reduce error for long arrays. Neither method makes every result exact; choose based on required accuracy and data.

Testing should include mixed magnitudes, positive and negative values, empty input, and values near the expected range. Compare with a higher-precision reference when possible, and report the tolerance used.

Linear algebra and scientific computing

Linear algebra represents vectors, matrices, and transformations. It is central to graphics, simulation, statistics, and machine learning. Solving a system Ax = b means finding a vector x whose matrix product with A approximates b.

Direct methods such as Gaussian elimination are useful for many systems; iterative methods can be better for very large sparse systems. Matrix structure, conditioning, sparsity, and the desired error tolerance determine which method is appropriate.

A matrix can be singular or nearly singular, making the solution nonexistent, non-unique, or highly sensitive to small perturbations. A program should detect solver failures and communicate them rather than returning a success-shaped value without checking the result.

Units and dimensional analysis

Numbers need units to have physical meaning. Adding meters to seconds is invalid even though both may be stored as floating-point values. Dimensional analysis checks that equations and transformations preserve compatible units and can catch model or implementation errors before numerical precision becomes relevant.

Validate and communicate numerical results

Numerical tests should check properties, not only exact output strings. A computed solution can be checked by substituting it back into the original equation and measuring the residual. Conservation laws, monotonicity, bounds, and symmetry can provide additional checks.

• Test zero, very small, very large, and mixed-magnitude inputs.
• Check expected units, range, and sign.
• Use absolute and relative tolerances with a documented rationale.
• Compare against an independent or higher-precision method.
• Test failure to converge and invalid input explicitly.

Do not print more digits than the computation or measurements justify. Report uncertainty and assumptions, particularly when results influence health, safety, finance, or policy. Numerical precision is not the same as real-world accuracy.

A compact glossary

Floating point

A finite representation of real values using a significand and exponent.

Roundoff error

Error introduced when an exact value is rounded to a representable number.

Conditioning

Sensitivity of a mathematical problem's answer to input changes.

Stability

How well an algorithm controls errors introduced during computation.

Cancellation

Loss of significant digits when subtracting nearly equal values.

Residual

The discrepancy remaining when a computed solution is substituted into a problem.

References and further reading

1. David Goldberg. “What Every Computer Scientist Should Know About Floating-Point Arithmetic.” ACM Computing Surveys, 1991. Article DOI.
2. Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms, 2nd edition. SIAM, 2002. Publisher information.
3. Numerical Recipes. Numerical computation resources. Reference site.
4. University of California, Berkeley. CS 267: Applications of Parallel Computers, numerical computing materials. Course archive.

Try it yourself

1. Why can a decimal fraction such as 0.1 be inexact in binary floating-point?
2. Explain the difference between absolute and relative error.
3. Why can changing the order of a floating-point sum change its result?
4. Describe one test property for a numerical solution to Ax = b.
5. Why does printing 20 decimal places not prove a result is accurate to 20 places?

Remember: numerical answers are computed with finite precision; select stable methods, measure error, and report results honestly.