What is Data Cleaning?

Data cleaning is the systematic detection and correction (or documented removal) of errors and low-quality records in a dataset before analysis: duplicates, invalid values, fraudulent or inattentive responses, broken formats, and inconsistencies. Its purpose is honest signal: analysis of contaminated data produces confident conclusions about contamination. Its danger is equally real: cleaning is where a dataset can be quietly shaped, exclusion by exclusion, toward a preferred answer, which is why the discipline is as much about process (pre-set rules, logged decisions, reproducibility) as about the screens themselves.

The Standard Screens

Duplicates and identity problems. The same respondent twice (retakes, panel overlap), multiple submissions from one device, and, increasingly, bots and survey farms: caught by identifiers, timestamps, fingerprinting, and the attention checks below. Recruitment quality upstream is the cheapest cleaning there is, one reason vetted panels with built-in quality controls beat open links.

Speeders and straight-liners. Completion far below the plausible minimum (a fraction of median duration is the working benchmark) and flat-lined grid answers signal attention checked out; both are standard, defensible exclusions when the thresholds were set in advance, and both are symptoms shorter instruments prevent.

Failed attention and consistency checks. Instructed-response items ("select 'somewhat agree'"), trap questions, and internal contradictions: powerful screens, used sparingly and fairly (one honest lapse shouldn't void ten minutes of good-faith answers; pre-decide how many failures exclude).

Invalid and impossible values. Ages of 250, negative durations, categories that don't exist: fix from source where the truth is recoverable, remove where it isn't, and trace the root cause, since impossible values usually indicate instrument or pipeline bugs that corrupted plausible values too.

Missing data, read before treated. Item-level gaps and drop-off aren't just holes to fill: where people abandon and what they skip is behavioural data (the attrition pattern), and whether missingness is random or systematic decides what any imputation can honestly do. Open-text and extreme numeric values get their own investigation rather than reflex deletion.

The Process Discipline

1. Write the rules before fielding.
Exclusion criteria, thresholds, and handling policies decided pre-data (ideally in the study plan) cannot be accused of chasing a result; the same rules invented afterwards, with results visible, always can be. This is pre-registration's logic applied to hygiene.

2. Never edit the raw file.
Cleaning produces a new dataset from an untouched original, via scripts or logged steps: the raw data is the evidence, the cleaned data is the exhibit, and the transformation between them must be reproducible.

3. Log every decision.
What was excluded, why, under which rule, and how many records each rule caught: the cleaning log (part of the study's audit trail) turns "we removed bad data" from an assertion into an inspectable claim.

4. Report the cleaning with the results.
Exclusion counts and criteria belong in the readout: they're sample facts, they affect interpretation, and their absence is the first thing a sceptical reviewer notices. Sensitivity checks (do conclusions survive with the borderline cases included?) close the loop.

The Takeaway

Data cleaning is quality control with a conflict of interest: essential for signal, dangerous as a place to shape results, resolved by discipline: rules before data, raw files preserved, every exclusion logged, everything reported. Clean hard, clean transparently, and let the log prove the difference between hygiene and laundering, because both look identical in the final chart.