L4StatisticsCore

Outliers and Cleaning Data

Q: Why is the standard deviation rule circular?

A large outlier inflates, which widens the boundary x 2, potentially far enough that the outlier no longer falls outside it.

An $\textbf{outlier}$ is a value far from the rest of the data. Finding one is arithmetic; deciding what to do with it is not. A recording error should be removed — but a genuine extreme value may be the most important observation you have.

30 min Data Presentation

The big picture

This is the point in the statistics course where judgement is examined directly rather than calculation. Deleting inconvenient data is the oldest way to make a result look better than it is, so the discipline is to justify any removal with a reason external to the number itself — an impossible value, a known instrument fault, a duplicated entry. "It spoiled my correlation" is not a reason. The habit matters beyond the exam: the difference between cleaning data and manipulating it is exactly whether the justification would survive someone else reading it.

What you'll be able to do

Identify outliers using the $1.5\times\text{IQR}$ rule and the standard deviation rule
Explain the effect of an outlier on the mean, median and standard deviation
Decide whether to remove or retain an outlier, with justification
Recognise other data quality problems and describe appropriate cleaning

Two definitions, and using the one you are given

The $\textbf{quartile rule}$ flags any value more than $1.5\times\text{IQR}$ below $Q_{1}$ or above $Q_{3}$ . It pairs with the median and IQR and is unaffected by the outlier it is testing for, which makes it the more robust choice.

The $\textbf{standard deviation rule}$ flags values more than a stated number of standard deviations from the mean — usually two or three. It is quicker when you already have $\bar{x}$ and $\sigma$ , but it has a circularity: a large outlier inflates $\sigma$ , which widens the boundary, which can hide the outlier that caused it.

Exam questions specify which rule to use. Apply the one given, show the boundary calculation, and compare each candidate value against it explicitly.

x < Q_{1}-1.5\,\text{IQR} \;\text{or}\; x > Q_{3}+1.5\,\text{IQR}

Quartile rule.

\left|x - \bar{x}\right| > 2\sigma

Standard deviation rule, with the multiplier the question states.

Example 1A data set has

\bar{x} = 50

and

\sigma = 6

. Using the two-standard-deviation rule, determine whether 65 is an outlier.

1Boundaries:

50 \pm 2(6)

, so

38

and

62

265 lies above 62.

AnswerYes, 65 is an outlier under this rule.

Why this works

The two rules disagree more often than students expect. A value can be an outlier by one and not the other, which is why the question always tells you which to apply — and why quoting the rule you used is part of the answer.

What an outlier does to each statistic

The $\textbf{mean}$ moves, sometimes a great deal, because every value contributes to it. The $\textbf{median}$ barely moves, because only position matters and one extra extreme value shifts the middle by at most one place.

The $\textbf{standard deviation}$ is affected most of all. It squares deviations, so a value far from the mean contributes disproportionately — a single outlier can inflate $\sigma$ noticeably. The $\textbf{IQR}$ is untouched, since it discards the outer quarters entirely.

That pattern is why median and IQR are described as $\textbf{resistant}$ statistics, and why they are the right summary for data known to contain extremes.

Example 1For

10, 12, 13, 14, 15

the mean is 12.8. Recalculate the mean and median after 90 is added to the set.

1New mean:

\dfrac{10+12+13+14+15+90}{6} = \dfrac{154}{6} = 25.7

2Original median: middle of five values is 13.

3New median: six values, so the mean of the 3rd and 4th:

\tfrac{13+14}{2} = 13.5

AnswerMean jumps from 12.8 to 25.7; median moves only from 13 to 13.5.

Tip — If a question asks which average is "more appropriate" for data with an outlier, the answer is the median — but the mark is for saying $\textit{why}$ : the mean is distorted by the extreme value.

Remove or retain?

Remove an outlier when there is evidence it is $\textbf{not a valid observation}$ : a physically impossible value, a known equipment failure, an obvious transcription slip such as a decimal point in the wrong place, or a duplicate record.

Retain it when it appears to be a genuine measurement. Unusual is not the same as wrong, and genuine extremes are often the most informative points in a data set — the unusually hot day, the exceptional performance, the fault you are trying to detect.

Whichever you do, $\textbf{state it and justify it}$ . An analysis that silently drops values cannot be checked by anyone else, and in an exam the justification is where the marks are.

Example 1A set of recorded human heights includes a value of 3.2 m. Should it be removed? Justify your answer.

1The tallest recorded human height is well under 3 m, so 3.2 m is physically impossible.

2It is most likely a recording error — perhaps 1.32 m or 2.3 m mistyped.

3The justification comes from outside the data, not from it being inconvenient.

AnswerRemove it, and say why: the value is physically impossible and therefore a recording error, not a genuine observation.

Example 2A set of daily rainfall figures includes one very high value on a day with a recorded storm. Should it be removed?

1The value is extreme but there is corroborating evidence it is genuine.

2Removing it would understate rainfall variability, which is often the point of the study.

AnswerRetain it. It is a real observation, and extremes matter for this variable.

Why this works

A useful discipline is to report the analysis both ways when a removal is debatable — with and without the value. If the conclusion holds either way it is robust; if it does not, that fact is itself worth knowing.

Other cleaning problems

Outliers are the examined case, but real cleaning covers more. $\textbf{Missing values}$ are excluded with the reduced sample size stated, never replaced by zero. $\textbf{Duplicate records}$ are removed, since they double-weight one observation.

$\textbf{Inconsistent units}$ must be converted before combining — a column mixing metres and centimetres will produce nonsense silently. $\textbf{Impossible values}$ , such as a negative age or a percentage above 100, indicate errors regardless of how far they sit from the mean.

The general principle throughout: document every change so the analysis can be reproduced by someone else.

Example 1A data set of exam percentages contains the values

-5

and

130

. Comment.

1Percentages on this scale must lie between 0 and 100.

2Both are impossible, so both are errors regardless of the outlier rules.

3Investigate and correct if the true values can be recovered; otherwise exclude and record the exclusion.

AnswerBoth are invalid values rather than extreme observations, and should be excluded with the exclusion documented.

Tip — A value can be an error without being an outlier. A misrecorded 45 among values from 40 to 50 fails no outlier test — which is why cleaning also means checking that values are possible.

Think like an examiner

Show the fence or boundary calculation, then compare the value to it explicitly.
Use the outlier rule the question specifies, not the one you find easier.
Any removal must carry a justification about validity, never about convenience.
When asked which average to use, name the statistic $\textit{and}$ the reason it is preferable here.

Common misconceptions

Removing an outlier because it worsens a correlation.

Removal needs evidence the value is invalid. Inconvenience is not evidence.

Using

Q_{3} + 1.5Q_{3}

instead of

Q_{3}+1.5\,\text{IQR}

The

1.5

multiplies the interquartile range, not the quartile itself.

Claiming an outlier has little effect on the standard deviation.

It has a large effect, because deviations are squared before averaging.

Assuming every extreme value is an error.

Genuine extremes exist and are often the most informative observations.

Outliers

Q_{1}-1.5\,\text{IQR}

Lower fence.

Q_{3}+1.5\,\text{IQR}

Upper fence.

\bar{x} \pm 2\sigma

Standard deviation rule.

\text{median, IQR: resistant}

Barely moved by extremes.

Stretch yourself

A set of eight reaction times (seconds) is: $0.21, 0.23, 0.24, 0.25, 0.26, 0.28, 0.31, 1.90$ . Use the quartile rule to test the largest value, then advise whether to remove it, considering two possible explanations.

Hint — Find the quartiles from the ordered list first. Then think about what could physically produce a 1.90 s reaction time.

Reveal solution

SolutionWith

n = 8

Q_{1}

is the mean of the 2nd and 3rd values,

\tfrac{0.23+0.24}{2} = 0.235

;

Q_{3}

is the mean of the 6th and 7th,

\tfrac{0.28+0.31}{2} = 0.295

. So IQR

= 0.06

and the upper fence is

0.295 + 1.5(0.06) = 0.385

. Since

1.90 > 0.385

, it is an outlier. Whether to remove it depends on the explanation. If the participant was distracted or the timing equipment mis-triggered, it is not a valid measurement of reaction time and should be excluded with that reason recorded. But if the trial ran correctly, 1.90 s is a real reaction time — perhaps from fatigue or a lapse of attention — and removing it would understate genuine variability, which for reaction-time studies is often exactly what is being investigated. The decision cannot be made from the number alone, which is the point.

\blacksquare

Questions students ask

Should outliers always be removed?

No. Remove only when there is evidence the value is invalid — impossible, mis-recorded, duplicated, or from a known equipment failure. Genuine extremes are retained.

Which outlier rule should I use?

The one the question specifies. If none is given, the quartile rule is the safer default because it is not distorted by the outlier it is testing.

Why is the standard deviation rule circular?

A large outlier inflates

\sigma

, which widens the boundary

\bar{x} \pm 2\sigma

, potentially far enough that the outlier no longer falls outside it.

What if I am unsure whether to remove a value?

Report the analysis with and without it. If the conclusion is unchanged it is robust; if it changes, that sensitivity is important information.

Key takeaways

Outliers are identified by the quartile rule or a standard deviation rule — use the one given.
They move the mean and inflate the standard deviation, but barely affect median and IQR.
Remove only with evidence of invalidity; retain genuine extremes.
Always state and justify any removal so the analysis can be checked.
Cleaning also covers missing values, duplicates, inconsistent units and impossible values.

How this fits the course

Build on