Tidy Data

Abstract

📜 Abstract

A huge amount of effort is spent cleaning data to get it ready for analysis, but there has been little research on how to make data cleaning as easy and effective as possible. This paper tackles a small, but important, component of data cleaning: data tidying. Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table. This framework makes it easy to tidy messy datasets because only a small set of tools are needed to deal with a wide range of un-tidy datasets. This structure also makes it easier to develop tidy tools for data analysis, tools that both input and output tidy datasets. The advantages of a consistent data structure and matching tools are demonstrated with a case study free from mundane data manipulation chores.

Description

✨ Summary

Hadley Wickham defines tidy data as a standard mapping between a dataset’s semantics and its physical structure. A tidy dataset follows three principles: each variable forms a column, each observation forms a row, and each type of observational unit forms a table. The paper distinguishes variables, observations, and observational-unit types, emphasizing that these concepts depend on the analytical purpose and experimental design. It also discusses the importance of preserving meaningful missing values, including the distinction between missing measurements and structural missingness.

The paper identifies five common forms of messy data: values stored in column headers; multiple variables combined in one column; variables distributed across both rows and columns; multiple observational-unit types stored in one table; and one observational-unit type distributed across multiple tables. These problems can generally be addressed through a small collection of operations: melting or stacking, splitting compound fields, casting or unstacking, normalization, joins, and combining files while retaining their source identifiers.

Wickham extends the data-structure principles to tidy tools: tools that accept tidy data and return tidy data. Consistent inputs and outputs make operations composable across manipulation, visualization, and modeling workflows. The case study uses Mexican mortality data to show how filtering, aggregation, joining, transformation, robust modeling, and visualization can be chained without repeated ad hoc reshaping. It identifies unusual hourly patterns for causes of death associated mainly with murder, drowning, and transportation incidents.

The paper also states limitations. Tidiness is not presented as a final or universal representation: multidimensional array-based formats may be preferable for some biomedical and matrix-oriented workloads, and many other data-cleaning tasks—such as parsing, encoding correction, typo matching, experimental-design validation, and imputation—remain outside the framework.

The paper’s principles became an explicit design foundation for the R tidyr ecosystem, whose documentation presents tidy data as a standard structure for analysis and reshaping. (tidyr.tidyverse.org) Later research has directly extended or built on the framework, including a tidy data structure designed specifically for temporal data and methods for exploring and assessing missing-data patterns using tidy principles. (arxiv.org) Research-data guidance also applies the same three rules to analysis-ready scientific datasets, indicating adoption beyond R-specific workflows. (datamanagement.hms.harvard.edu)