Outliers III

DSCI 200

Katie Burak, Gabriela V. Cohen Freue

Last modified – 03 November 2026

\[ \DeclareMathOperator*{\argmin}{argmin} \DeclareMathOperator*{\argmax}{argmax} \DeclareMathOperator*{\minimize}{minimize} \DeclareMathOperator*{\maximize}{maximize} \DeclareMathOperator*{\find}{find} \DeclareMathOperator{\st}{subject\,\,to} \newcommand{\E}{E} \newcommand{\Expect}[1]{\E\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[2]{\mathrm{Cov}\left[#1,\ #2\right]} \newcommand{\given}{\ \vert\ } \newcommand{\X}{\mathbf{X}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\P}{\mathcal{P}} \newcommand{\R}{\mathbb{R}} \newcommand{\norm}[1]{\left\lVert #1 \right\rVert} \newcommand{\snorm}[1]{\lVert #1 \rVert} \newcommand{\tr}[1]{\mbox{tr}(#1)} \newcommand{\brt}{\widehat{\beta}^R_{s}} \newcommand{\brl}{\widehat{\beta}^R_{\lambda}} \newcommand{\bls}{\widehat{\beta}_{ols}} \newcommand{\blt}{\widehat{\beta}^L_{s}} \newcommand{\bll}{\widehat{\beta}^L_{\lambda}} \newcommand{\U}{\mathbf{U}} \newcommand{\D}{\mathbf{D}} \newcommand{\V}{\mathbf{V}} \]

Attribution

• Some examples and references are from Challenges of cellwise outliers, by Raymaekers and Rousseeuw

• Robust Statistics, by Maronna, Martin, Yohai, Salibian-Barrera

In memory of my mentor and friend

Professor Ruben Zamar (1949-2023)

who contributed greatly to Robust Statistics and beyond …

Review

• An outlier is an observation that deviates from the bulk of the data, an atypical observation.

• We defined both casewise and cellwise outliers in a dataset, but focused on univariate outliers.

• We examined by simulations three univariate estimators of location: the mean, the median, and the M-estimators.

• We learned about the trade-off between robustness and efficiency.

• We also learned about masking and robust outlier detection based on the median and MAD.

From one variable to an entire dataset

Three core tasks:

• Estimating multivariate parameters (e.g., location and covariance)
• Modeling relationships (e.g., regression)
• Discovering structure (e.g., unsupervised learning)

Goal for today

1. How do we detect outliers when we have several variables?
2. What happens when outliers affect regression? Can M-estimators help?
3. Identify different type of outliers: rowwise, cellwise and outlies in regression
4. Why do cellwise outliers need different tools?

From one variable to many variables

For one variable, we describe:

• center: mean, median
• spread: variance, SD, MAD

For several variable, we describe:

• multivariate center: the typical combination of values across variables.

• multivariate scatter: how observations spread around the center, accounting for relationships between variables.

The center locates the data cloud, while scatter describes its shape and orientation.

Robust multivariate estimators

We want to estimate the center and spread of the data without being overly influenced by outliers.

Two possible strategies:

• 1. M-estimators: give less weight to observations that are far from the main cloud of data, reducing their influence on the estimated center and scatter.
• 2. Minimum covariace determinant (MCD): select the 50% of observations (or other) that form the most compact multivariate cloud. Estimate the center and spread using that group.

Important

Unlike coordinate-wise estimates and pairwise correlations, these methods consider all variables jointly to estimate the center and scatter.

X <- glass_df[, c("V29", "V30", "V125", "V175", "V180")]

# Classical estimates
center_mean <- colMeans(X)
center_median <- sapply(X, median)
cov_classical <- cov(X)
cor_classical <- cor(X)

# Robust estimates (MCD)
mcd <- robustbase::covMcd(X)
center_MCD <- mcd$center
cov_MCD <- mcd$cov
cor_MCD <- mcd$cor

# Multivariate MM-estimator
mest <- covRob(as.matrix(X), type = "MM", corr = TRUE)

center_MMest <- mest$center
cov_MMest <- mest$cov
cor_MMest <- mest$cor

# Compare estimated centers
round(rbind(
  Mean = center_mean,
  Median = center_median,
  MM = center_MMest,
  MCD = center_MCD
), 2)

         V29   V30  V125    V175   V180
Mean   13.49 12.62 94.40 2013.37 511.22
Median 13.60 12.32 84.88 1754.15 466.60
MM     12.76 11.87 82.35 1729.24 450.51
MCD    12.46 11.63 79.33 1724.55 447.41

• V125–V175: the apparent positive relationship largely disappears with robust estimation.

• V175–V180: The correlation remains very strong across all methods.

Unusual observations can strongly affect estimated correlations when using classical estimators.

Rowwise outliers

How can we identify observations far from the main cloud?

Distance measure

For univariate detection: a robust z-score

\[z_i^{\text{rob}} = \frac{x_i-\text{median}}{\text{MAD}}\]

For multivariate detection: the Mahalanobis distance

• We need a distance measure that accounts for the variability of each variable and the relationships between variables.

• Accounting for how the data vary, being far from the center does not necessarily mean being unusual.

• Using robust estimates of center and scatter can help control masking.

Important

An observation can be unusual jointly even if none of its individual values is extreme.

Detecting rowwise outliers

• We can use the MCD or MM- estimates to calculate robust Mahalanobis distances (MD) of each data point.

• The threshold is determined based on how this distance is expected to behave when the data follow a Normal distribution.

# MCD estimator
mcd <- covMcd(X)
md <- mahalanobis(X, mcd$center, mcd$cov)
cutoff <- qchisq(0.975, ncol(X))

md_df <- data.frame(
  obs = seq_along(md),
  rob_MD = md,
  flagged = md > cutoff)

ggplot(md_df, aes(x = obs, y = rob_MD)) +
  geom_point(aes(shape = flagged, color = flagged), size = 2) +
  scale_shape_manual(values = c(16, 17)) +
  scale_color_manual(values = c("black", "blue")) +
  geom_hline(yintercept = cutoff, linetype = "dashed")

Data points (rows) with a squared MD exceeding the cutoff are flagged as rowwise outliers

Cellwise outliers

What if only a few values per observation are unusual in a row?

Cellwise outliers

• In high-dimensional data, contamination often occurs cell by cell, not row by row.
• Every row may contain at least one bad entry
• Removing or downweighting entire rows with rowwise methods may discard useful information.

Wrapping extends the idea of robust estimation to individual cells.

Robust correlation by wrapping

In the spirit of Huber’s estimation, wrapping have been proposed to compute robust correlations (fast):

• 1. robustly standardize each variable
• 2. transform extreme values with a bounded function
• 3. compute ordinary correlations on the transformed data

So we keep the speed and structure of classical correlation, but gain robustness.

from Raymaekers, J., and Rousseeuw, P.J, (Technometrics 2021)

Detecting cellwise outliers

DDC: Detect Deviating Cells

DDC combines robust correlation and regression to flag outlying cells.

• Robust correlations identify related variables.
• Robust regressions estimate relationships between pairs of variables.
• These relationships are used to predict individual cell values.
• Cells that deviate substantially from their predictions are flagged.
• Missing and flagged values can be replaced by predicted values.

DDC identifies unusual cells without discarding entire rows.

Applying DDC to the Glass data

# Run the fast DDC algorithm
fastDDCglass <- DDC(X, list(fastDDC = TRUE, silent = TRUE))

# Create the cell map
ggpfastDDC <- cellMap(
  R = fastDDCglass$stdResid,
  indcells = fastDDCglass$indcells,
  indrows = fastDDCglass$indrows,
  nrowsinblock = 5,
  mTitle = "",
  columnangle = 90
)

No rowblocklabels were given, so they are constructed automatically.

• stdResid: standardized differences between observed and predicted values.
• indcells: cells flagged as unusual.
• indrows: rows flagged as unusual.

The cell map

Let’s visualize the suspicious entries in the Glass data.

Yellow: regular cells. Red: unusually high values. Blue: unusually low values.

Outliers in regression

From multivariate outliers to regression

three_panel_reg

Outliers can strongly influence regression estimates

Least squares regression

\[ \begin{equation*} \frac{1}{n} \ \sum_{i=1}^n (\underbrace{y_i - {\color{blue}{a}} - {\color{blue}{b}} x_i }_{\text{Residual}})^2 \end{equation*} \]

• LS finds \({\color{blue}{a}}\) and \({\color{blue}{b}}\) that minimizes the sum of the squared residuals.

• Points with large residuals contribute heavily to the sum of squared residuals, pulling the LS fit toward them.

Attribution: slides 23-25 are based on lecture notes from Prof. Salibian-Barrera

Least squares:

Minimizes the sum of squared residuals.

Robust regression:

Limits the influence of large residuals.

Robust regression (for rowwise outliers)

M-estimators

Replace square with a robust function \(\rho(r)\) to limit the influence of outliers.

when \(\sigma\) is unknown, we can use MM-estimation, which first estimates the scale robustly.

\[ \begin{equation*} \frac{1}{n} \ \sum_{i=1}^n \rho \left(\frac{y_i - {\color{blue}{a}} - {\color{blue}{b}} x_i }{\sigma}\right) \end{equation*} \]

Least Trimmed Squares (LTS)

Minimizes the sum of the 50% smallest squared residuals (or other proportion).

set.seed(123)

reg_ls  <- lm(V175 ~ V125, data = dat)
reg_mm <- lmrob(V175 ~ V125, data = dat)
reg_lts <- ltsReg(V175 ~ V125, data = dat, alpha = 0.5)

Attribution: based on lecture notes from Prof. Salibian-Barrera

Outliers detection in regression

# Add observation numbers
dat <- dat |>
  mutate(id = row_number())

# Compute fitted values and standardized residuals
diag_dat <- bind_rows(
  dat |>
    mutate(
      Method = "Least squares",
      fitted = fitted(reg_ls),
      std_resid = resid(reg_ls) / sigma(reg_ls)
    ),
  dat |>
    mutate(
      Method = "MM-estimation",
      fitted = fitted(reg_mm),
      std_resid = resid(reg_mm) / reg_mm$scale
    ),
  dat |>
    mutate(
      Method = "LTS",
      fitted = reg_lts$raw.coefficients[1] +
               reg_lts$raw.coefficients[2] * V125,
      std_resid = (V175 - fitted) / reg_lts$raw.scale
    )
) |>
  mutate(
    Method = factor(Method,
      levels = c("Least squares", "MM-estimation", "LTS")),
    flag = abs(std_resid) > 2.5
  )

# Select observations to label
lab_dat <- diag_dat |>
  filter(flag) |>
  group_by(Method) |>
  slice_max(abs(std_resid), n = 3, with_ties = FALSE) |>
  ungroup()

Least squares can mask outliers because they pull the fitted line toward themselves, reducing their residuals.

Regression outliers: residuals and leverage

• Vertical residual: an observation far vertically from the fitted line.
• High-leverage observation has an unusual value of the predictor, good or bad, depending on whether they follow the main data pattern.

two_panel_reg

Key Takeaways

• In both univariate and multivariate cases, classical estimators can be sensitive to outliers.

• Robust estimators reduce the influence of extreme observations, while aiming to maintain high efficiency under Normality.

• Rowwise contamination

  • Methods such as M-estimators and MCD estimate robust multivariate center and scatter

  • These estimates can be used to compute robust Mahalanobis distances to detect outliers

• Cellwise contamination

  • Methods such as DDC and wrapping detect corrupted cells
  • They rely on relationships among variables and are used to compute robust multivariate estimates and flag outliers

• Robust methods improve both estimation and outlier detection

• Regression:

  • M-estimators extend the robust-loss idea to residuals
  • LTS uses all data points to select a subset of points with smallest residuals
  • Outliers can be identified using residual plots from robust regression fits.
  • Large residuals and high leverage are different.