Outliers II

DSCI 200

Katie Burak, Gabriela V. Cohen Freue

Last modified – 29 October 2026

\[ \DeclareMathOperator*{\argmin}{argmin} \DeclareMathOperator*{\argmax}{argmax} \DeclareMathOperator*{\minimize}{minimize} \DeclareMathOperator*{\maximize}{maximize} \DeclareMathOperator*{\find}{find} \DeclareMathOperator{\st}{subject\,\,to} \newcommand{\E}{E} \newcommand{\Expect}[1]{\E\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[2]{\mathrm{Cov}\left[#1,\ #2\right]} \newcommand{\given}{\ \vert\ } \newcommand{\X}{\mathbf{X}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\P}{\mathcal{P}} \newcommand{\R}{\mathbb{R}} \newcommand{\norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\snorm}[1]{\lVert #1 \rVert} \newcommand{\tr}[1]{\mbox{tr}(#1)} \newcommand{\brt}{\widehat{\beta}^R_{s}} \newcommand{\brl}{\widehat{\beta}^R_{\lambda}} \newcommand{\bls}{\widehat{\beta}_{ols}} \newcommand{\blt}{\widehat{\beta}^L_{s}} \newcommand{\bll}{\widehat{\beta}^L_{\lambda}} \newcommand{\U}{\mathbf{U}} \newcommand{\D}{\mathbf{D}} \newcommand{\V}{\mathbf{V}} \]

Attribution



In memory of my mentor and friend

Professor Ruben Zamar (1949-2023)

who contributed greatly to Robust Statistics and beyond …



Review

  • An outlier is an observation that deviates from the bulk of the data, an atypical observation.

  • As with missing values, outliers require careful diagnosis and appropriate handling before analysis.

  • We defined both casewise and cellwise outliers

- We used simulations to examine 2 univariate estimators: the mean and the median

Robustness vs Efficiency \(\rho(r)\) Estimator
square Mean
absolute Median
 
  

differenciable?		  

M-estimator

Different \(M\)-estimators of location have been proposed:

The tuning constant (c) controls the robustness–efficiency trade-off.

Computing M-estimators

To compute a robust M-estimators of location, we typically also need an estimate of the spread (\(\sigma\), aka scale) of the distribution.

  • The SD is efficient but not robust to outliers

  • The MAD (median absolute deviation) is highly robust:

\[1.4826 \, \text{median}(|x_i - \text{median}|)\]

  • We can define an M-estimator for the scale (M-scale)

We can also estimate location and scale jointly via M-estimation

In R

  • mad(x) computes the MAD

  • locScaleM(x) in RobStatTM package computes Huber and bisquare M-estimators:

    • jointly for location and scale
    • for various (asymptotic) efficiency values

Simulation design

Clean data

Generate a random sample of size 100:

  • from a \(\mathcal{N}(0,1)\)

Contaminated data

Generate a random sample of size 100

  • 90% from \(\mathcal{N}(0,1)\) and 10% from \(\mathcal{N}(0,10)\)

Estimators

Compute:

  • the mean

  • the median

  • two bisquare-M estimators, at 85% and 95% efficiency, respectively

Replication and summary

  • Repeat 1000 times

  • Approximate the variance of the estimators

Without and with contamination

library(RobStatTM)
set.seed(200)

rcont_norm <- function(n, eps = 0.10, mu0 = 0, sigma0 = 1, sigma1 = 10) {
  x <- rnorm(n, mean = mu0, sd = sigma0)
  
  k <- ceiling(eps * n)              
  if (k > 0) {
    idx <- sample(1:n, k)
    x[idx] <- rnorm(k, mean = mu0, sd = sigma1)
  }
  x
}

#no contamination

eps = 0
sim <- replicate(1000, {
    x <- rcont_norm(100, eps = eps)
    c(
      Mean     = mean(x),
      Median   = median(x),
      Bisquare_3.44 = locScaleM(x, psi="bisquare", eff=0.85)$mu,
      Bisquare_4.68 = locScaleM(x, psi="bisquare", eff=0.95)$mu
    )
  })

  
var0  <- apply(t(sim), 2, var) 

# with contamination
eps = 0.20
sim <- replicate(1000, {
    x <- rcont_norm(100, eps = eps)
    c(
      Mean     = mean(x),
      Median   = median(x),
      Bisquare_3.44 = locScaleM(x, psi="bisquare", eff=0.85)$mu,
      Bisquare_4.68 = locScaleM(x, psi="bisquare", eff=0.95)$mu
    )
  })
  
var20 <- apply(t(sim), 2, var) 

variance_table <- cbind(
  "Variance" = var0,
  "RE" = var0["Mean"] / var0,
  "Variance" = var20,
  "RE" = var20["Mean"] / var20
) |> round(3)
Sampling variance and relative efficiency of location estimators
No contamination
20% contamination
Estimator Variance RE Variance RE
Mean 0.010 1.000 0.208 1.000
Median 0.016 0.648 0.023 9.138
Bisquare_3.44 0.012 0.880 0.015 13.583
Bisquare_4.68 0.011 0.970 0.016 13.245

Relative efficiency (RE): ratio of the variances

Detecting Outliers

How do we know if the data contains outliers?


The 3-\(\sigma\) rule

  • Classical rule: observations far from the mean are considered outliers

  • What is “far”? We can measure

\[z_i = \frac{x_i-\bar x}{s}\]

  • Flag as outlier if: \(|z_i| > 3\)

iClicker 1: interpretation of the rule


With the 3-\(\sigma\) rule:

\[|z_i| = \left|\frac{x_i-\bar x}{s}\right| > 3 \]

an observation is flagged as an outlier if

  • A. The observation is more than 3 standard deviations from the mean

  • B. The observation is in the largest 3% of the data

  • C. The observation is larger than 3 times the mean

  • D. The observation is more than 3 units from the median

iClicker 2: use the rule


Which points are flagged as outliers using the 3-\(\sigma\) rule?

x <- c(-2,-1,0,1,2,13,20)

mean(x); sd(x)
[1] 4.714286
[1] 8.40068
z  <- (x-mean(x))/sd(x)
z
[1] -0.7992550 -0.6802170 -0.5611790 -0.4421411 -0.3231031  0.9863147  1.8195805

  • A. Both \(x=13\) and \(x=20\) are flagged

  • B. Only \(x=20\) is flagged

  • C. All points are flagged

  • D. None of points are flagged

Masking


  • The mean and SD are sensitive to outliers

  • Extreme observations pull the mean toward them and inflate the standard deviation, reducing their standardized distance

  • Robust alternative: use median and MAD (or other robust loc-scale estimators)

Robust z-score:

\[z_i^{rob} = \frac{x_i-\text{median}}{\text{MAD}}\]

Rule: \(|z_i^{rob}| > 3\)

iClicker 3: use the robust rule


Which points are flagged as outliers using the 3-\(\sigma\) rule?

x <- c(-2,-1,0,1,2,13,20)

median(x); mad(x)
[1] 1
[1] 2.9652
z  <- (x-median(x))/mad(x)
z
[1] -1.0117361 -0.6744908 -0.3372454  0.0000000  0.3372454  4.0469446  6.4076622

  • A. Both \(x=13\) and \(x=20\) are flagged

  • B. Only \(x=20\) is flagged

  • C. All points are flagged

  • D. None of points are flagged

Key Takeaways


  • Classical estimators (mean, SD) are highly sensitive to outliers

  • Robust estimators reduce the influence of extreme observations

  • M-estimators provide a flexible compromise between efficiency and robustness

  • The tuning constant controls this trade-off

  • Robust methods improve both estimation and outlier detection