Data science: ask questions and learn from data

Explore the full process of turning data into useful evidence: frame questions, collect and prepare information, analyze patterns, visualize results, and communicate uncertainty responsibly.

What data science does

Data science combines computing, statistics, domain knowledge, and communication to answer questions using data. It includes more than building predictive models. Cleaning records, checking definitions, designing measurements, and explaining limitations are often the most important parts of a project.

A data set is not a neutral description of reality. It reflects what someone decided to measure, when and how measurements were collected, whose experiences are represented, and which records were missing or excluded. Interpretation requires understanding those choices.

• Descriptive: What happened in the data?
• Diagnostic: What patterns might help explain what happened?
• Predictive: What could happen for new observations?
• Prescriptive: What action should be considered given goals and constraints?

These question types call for different evidence. A dashboard can describe an increase; it does not by itself prove what caused it. A model can predict an outcome without identifying an effective intervention.

Start with a useful question

Good analysis begins with a clear question tied to a decision. “Look at the sales data” is broad. “Did the new store schedule reduce average customer wait time without increasing missed breaks?” identifies a comparison and competing outcomes.

Define the unit of analysis: a customer, a transaction, a day, or a store. Define the outcome and the time window. Clarify what population the result should describe, and what action someone might take after seeing it.

Operational definitions prevent confusion

An operational definition specifies how a concept is measured. “Wait time” might mean time from joining a queue to first contact, or time until the full service is complete. Different definitions answer different questions.

Write down assumptions and success criteria before looking for a pleasing result. This reduces the risk of changing definitions after seeing the data and reporting only the comparison that supports an existing expectation.

Collect, inspect, and clean data

Data can come from databases, surveys, sensors, application logs, public records, or controlled experiments. Check whether collection was authorized, whether people were informed, and whether the data is appropriate for the question. More records do not automatically improve an analysis if they are inaccurate or systematically incomplete.

Inspect before analysis

1. Check the number of records and fields.
2. Inspect types, ranges, units, and example values.
3. Look for missing values, duplicates, and impossible values.
4. Confirm time zones, category definitions, and identifiers.
5. Record where data came from and what transformations were applied.

Cleaning means making documented decisions about inconsistencies, not silently deleting inconvenient records. A missing age, a zero value, and a value recorded as “unknown” may have different meanings. Track exclusions so another person can understand how the analysis was produced.

Sampling and representativeness

A sample is a subset of a larger population. If survey responses come mostly from people who had a strong experience, the sample may not reflect all users. A large biased sample can produce a very precise estimate of the wrong population.

Compare who is included with the population the analysis is meant to describe. Consider time periods, selection rules, non-response, and groups that may be missing. If the sample is narrow, report the limitation rather than generalizing beyond the evidence.

Explore with summaries and visualizations

Exploratory data analysis uses summaries and plots to understand distributions, spot errors, and generate questions. Begin with simple views: counts for categories, ranges and quantiles for numeric values, and trends over time where order matters.

The mean can be pulled by extreme values; the median is often more representative of a skewed distribution. A spread measure such as range or interquartile range helps explain how much observations vary. A single average can hide meaningful differences between groups.

Visualize to answer a question

Question	Possible visual	Check carefully
How do values vary?	Histogram or box plot	Bin widths, outliers, and sample size
How do two numeric variables relate?	Scatter plot	Clustering, nonlinear patterns, and hidden groups
How does a measure change over time?	Time-series line chart	Time interval, missing periods, and axis scale
How do categories compare?	Sorted bar chart	Clear labels, baseline, and uncertainty

Use labels, units, and honest scales. A truncated axis can exaggerate small differences; an overly crowded chart can conceal a pattern. Color should not be the only way information is distinguished, and charts should remain interpretable for people with color-vision differences.

Reason with statistics

Statistics help describe uncertainty and distinguish patterns from random variation. A point estimate such as an average is calculated from a sample; it is not necessarily the exact population value. A confidence interval expresses uncertainty under a particular statistical procedure and its assumptions.

A p-value is calculated under a specified null model and summarizes how incompatible the observed data are with that model. It is not the probability that a hypothesis is true, nor does a small value establish that an effect is important. Study design, effect size, uncertainty, and prior evidence matter.

Correlation and confounding

Two variables are correlated when they vary together. Correlation alone does not establish that one causes the other. A third variable may influence both, or the apparent association may result from selection or chance.

For example, ice-cream sales and swimming incidents may rise during warm months. This does not imply that ice cream causes swimming incidents; temperature and seasonal activity may affect both. A causal explanation requires a design and assumptions that address alternative explanations.

Report effect sizes and uncertainty, not only whether a result crossed a threshold. A tiny effect can be statistically detectable in a large sample but irrelevant to a real decision. Conversely, an important effect may remain uncertain when data is limited.

Worked example: analyze customer wait times

Suppose a team wants to know whether a revised staffing schedule changed wait times. First define the measure: minutes from entering a queue to first service. Define which locations and dates are included, and check that clock systems use the same time zone.

1. Inspect timestamps for missing values, negative durations, and duplicate visits.
2. Summarize median and upper-quantile waits by location and day.
3. Plot daily values to see whether weekends or holidays differ.
4. Compare periods before and after the schedule change, noting other changes.
5. Report sample size, uncertainty, and known limitations.

If wait times fell after the schedule change, the result is evidence of an association, not automatically proof that the schedule caused the decrease. Seasonality, different customer volume, or a new queue system could also explain it. A stronger study might compare similar locations with and without the change, if assignment and assumptions support that comparison.

The decision also includes trade-offs: shorter customer waits may come with increased staffing costs or reduced breaks. Analyze relevant outcomes together rather than optimizing one measure while ignoring others.

Experiments, causality, and A/B tests

A controlled experiment assigns participants or units to different conditions and measures outcomes. Random assignment can help balance known and unknown factors on average, making causal comparisons more credible when the experiment is well designed and conducted.

An A/B test compares two versions, such as two page layouts. Before starting, define the primary metric, minimum effect worth detecting, sample size approach, duration, and safety checks. Repeatedly checking results and stopping at the first favorable moment can inflate false-positive risk.

Experiments can harm or exclude people if consent, privacy, fairness, or safety is ignored. Randomization may be inappropriate for some services or decisions. A controlled trial is one source of causal evidence, not a universal requirement or permission to expose users to risk.

When experimentation is not possible, observational methods can help, but their conclusions depend on assumptions about confounding and selection. Describe what was adjusted for and what may still explain the observed relationship.

Communicate findings clearly

A useful data report states the question, population, data source, method, main result, uncertainty, limitations, and decision relevance. Separate what the data shows from what the team infers. Include enough detail for a reader to understand how the conclusion was reached.

Use plain language and visuals suited to the audience. Explain practical size: “the median decreased by two minutes” is often more informative than a model coefficient without context. Avoid false precision and avoid implying that a relationship is causal unless the design supports that claim.

Reproducibility helps others verify results. Keep analysis code, data definitions, environment information, and transformation steps where access and privacy rules allow. Record any manual decisions such as exclusions, category mappings, and changes made after exploratory review.

Privacy and responsible analysis

Data can identify people even when direct identifiers are removed, especially when records are combined. Collect the minimum information needed, restrict access, set retention limits, and consider whether a result could expose or stigmatize a group.

Analyze who is missing and who bears errors. An average improvement across all users can hide a decline for a smaller group. Subgroup analysis needs enough data and careful interpretation; small samples can create unstable estimates and privacy concerns.

Store analysis data securely and avoid copying sensitive records into notebooks, screenshots, logs, or shared documents without an approved reason. Responsible data science includes asking whether analysis should be performed at all and whether the resulting decision is fair and contestable.

A compact glossary

Population

The full group a question aims to describe or understand.

Sample

The subset of observations actually collected and analyzed.

Confounder

A factor that can influence both a measured exposure and outcome.

Effect size

A measure of the magnitude of an observed difference or relationship.

Data provenance

Information about where data came from and how it was transformed.

Reproducibility

The ability to repeat an analysis using its documented data and methods.

References and further reading

1. Hadley Wickham and Garrett Grolemund. R for Data Science, 2nd edition. Free online book.
2. OpenIntro. OpenIntro Statistics. Free textbook.
3. U.S. Census Bureau. Data Visualization and Statistical Literacy resources. Learning resources.
4. UK Government Analysis Function. Quality, methodology and analysis guidance. Guidance library.

Try it yourself

1. Turn “Are our users happier?” into a measurable question with a defined population and time period.
2. Why can a large sample still be unrepresentative?
3. Sketch a chart for comparing wait times across days and explain how you would display outliers.
4. Give a confounder that might affect both study time and exam results.
5. Write a short report sentence that distinguishes an observed association from a causal claim.

Remember: data supports decisions only when the question, measurement, uncertainty, and limits are made clear.