Missing data and imputation

Published

August 6, 2026

Introduction

The aim of this practical is to build understanding of missing data, i.e., how different missingness mechanisms affect your estimates, why naive fixes like mean imputation and listwise deletion don’t solve the problem, and how proper (multiple) imputation does.

In Part 1, we manually generate missingness under three mechanisms in an income dataset, and show why mean imputation fails. In Part 2, we move to a new dataset and work through a sequence of increasingly better imputation approaches, ending with multiple imputation using the mice package, and compare how each one affects a regression estimate.

library(MASS)
library(tidyverse)
library(mice)
library(ggmice)
library(ISLR)

Make sure to load MASS before tidyverse, otherwise the function MASS::select() will overwrite dplyr::select().

Part 1: Missingness mechanisms

Before we start, set a seed for reproducibility, we use 45.

set.seed(45)

Income dataset

In this part, we will manually create missingness to get an idea how different missingness mechanisms affect the outcomes of analyses. To this end, we start with a complete dataset, income.rds which can be downloaded here. Load the data in R after downloading as follows:

income <- readRDS("data/income.rds")

Generating missing values

The general expression of the missing data model is \(Pr(R = 0|Y_{\text{obs}}, Y_{\text{mis}}, \psi)\). Let’s go through this formula step by step.

  • \(R\) is a matrix with response indicators, which contains the locations of the missing values. For each person and each variable, \(R_{i,j}\) indicates whether person \(i\) has a missing value (\(R_{i,j} = 0\)) or an observed value (\(R_{i,j} = 1\)) on variable \(j\). In our case, we have \(R_{\text{age}}\), \(R_{\text{gender}}\) and \(R_{\text{income}}\).

  • \(Y_{\text{obs}}\) denotes the observed data.

  • \(Y_{\text{mis}}\) denotes the missing data. Note, we don’t have these data, but they might influence whether or not a person has missings on a variable.

  • \(\psi\) denotes the parameters for the missing data models. These parameters relate the observed and missing values to the response indicator.

Now we have a binary response indicator, and some additional variables, we can build a classification model to predict whether or not someone has a missing value on some variable. Normally, we would use the data to evaluate whether the missingness is related to someone’s values on other variables. However, we can also reverse the process, and impose a missingness structure on the observed data that depends on the variables.

In this practical, we will create missingness in the variable income, and we will do that using a logistic regression model. The logistic regression model predicts the probability of a missing value, given a set of indicators.

Note that a logistic regression model for the missingness can be defined as \[ \Pr(R_{\text{income}} = 0 | Y_{\text{obs}}, Y_{\text{mis}}, \psi) = \frac{\exp\{\psi_0 + \psi_1 * \text{age} + \psi_2 * (\text{gender}=\text{male}) + \psi_3 * \text{income}\}}{1 + \exp\{\psi_0 + \psi_1 * \text{age} + \psi_2 * (\text{gender} = \text{male}) + \psi_3 * \text{income}\}}. \] In this equation, \(\psi_0\) is a parameter for the baseline probability of missingness in \(\text{income}\), \(\psi_1\) and \(\psi_2\) relate the observed data to the missingness in \(\text{income}\), and \(\psi_3\) relates unobserved information to the missingness in \(\text{income}\).

If we would translate this equation into R-code, we would have

prob_mis <- function(psi0, psi1, psi2, psi3, age, gender, income) {
  exp(psi0 + psi1 * age + psi2 * (gender == "Male") + psi3 * income) /
    (1 + exp(psi0 + psi1 * age + psi2 * (gender == "Male") + psi3 * income))
}

Not Data-Dependent (NDD)

The data are considered to be NDD if \(Pr(R = 0|Y_{obs}, Y_{mis}, \psi) = Pr(R=0|\psi)\), indicating that the missingness probability is unrelated to the data \(Y\) and only depends on some parameters \(\psi\).

1. Use the function above to add a variable with the missingness probabilities misprob_ndd to the income data, in which you specify all missingness parameters to be 0 (i.e., \(\psi_0 = \psi_1 = \psi_2 = \psi_3\) = 0), such that the probability of having a missing value equals \(0.5\) for everyone.

Tip: use the mutate() function to create the new variable.

income <-
  income |>
  mutate(misprob_ndd = prob_mis(
    psi0   = 0,
    psi1   = 0,
    psi2   = 0,
    psi3   = 0,
    age    = age,
    gender = gender,
    income = income
  ))

Using these probabilities, we can randomly draw the observations that will have a missing value on \(\text{income}\) from a binomial distribution.

2. Use rbinom() to create a missingness indicator R_ndd using the misprob_ndd probabilities, and add this indicator to the income data.

NB: The function prob_mis gives the probability of having a missing, while the missingness indicator must give zero if income is not observed, so use 1 - rbinom() to create a proper missingness indicator.

income <- 
  income |>
  mutate(R_ndd = 1 - rbinom(n(), 1, misprob_ndd))
3. Use the function ifelse() to add a new variable income_ndd to the income data, that equals NA if R_ndd equals 0, and the observed income value otherwise.
income <- 
  income |>
  mutate(income_ndd = ifelse(R_ndd == 0, NA, income))
4. Verify that approximately \(50\%\) of the observations have a missing value on income_ndd.
table(is.na(income$income_ndd))

FALSE  TRUE 
  263   237 

Seen Data-Dependent (SDD)

So far, there was a baseline probability to have missing values, but this was the same for all observations (hence the term, Not Data-Dependent). Data are considered to be Seen Data-Dependent if the probability of having a missing value depends on the observed variables. Formally, we can define this as \(Pr(R = 0|Y_{obs}, Y_{mis}, \psi) = Pr(R=0|Y_{obs},\psi)\).

In this example, we let the missingness in \(\text{income}\) depend on the observed values of \(\text{age}\) and \(\text{gender}\), in such a way that older people have a higher probability of having a missing, and that males have a higher probability of having a missing than females.

5. Use the prob_mis() function to create misprob_sdd with psi0 = -3, psi1 = 0.05, psi2 = 0.5 and psi3 = 0. Then, as before, use rbinom() to create a missingness indicator R_sdd, and ifelse() to create income_sdd (NA where R_sdd equals 0).
income <-
  income |>
  mutate(
    misprob_sdd = prob_mis(
      psi0   = -3,
      psi1   = 0.05,
      psi2   = 0.5,
      psi3   = 0,
      age    = age,
      gender = gender,
      income = income
    ),
    R_sdd = 1 - rbinom(n(), 1, misprob_sdd),
    income_sdd = ifelse(R_sdd == 0, NA, income)
  )
6. Create a plot that shows how the probability of having a missing value (on the y-axis) depends on \(\text{age}\) (on the x-axis) and \(\text{gender}\) (as color aesthetic).
income |> 
  ggplot(aes(x = age, y = misprob_sdd, colour = gender)) +
  geom_line() +
  geom_point() +
  theme_minimal() +
  labs(
    x = "Age", 
    y = "Missingness probability", 
    colour = "Gender",
    title = "Seen data dependent missingness"
  )

Unseen Data-Dependent (UDD)

We now take the missingness one step further again, by making it dependent on unobserved data. In practice, it is often the case that those who have a higher income are less willing to share their income, which is an example of unseen data-dependent missingness. Namely, we cannot observe whether those who have a missing on income have a higher income, because this is exactly the information that we miss. The missingness could also depend on variables that we have not observed, for example because people who are self-employed might be less willing to answer survey questions about their income than people who have an employer. In both situations, we call the missingness Unseen Data-Dependent. Formally, this is defined as \(Pr(R = 0|Y_{obs}, Y_{mis}, \psi) = Pr(R=0|Y_{obs},Y_{mis},\psi)\).

Because we have no unobserved variables in this example, we let the missingness in \(\text{income}\) depend on the income values themselves, such that higher incomes have a higher probability of being missing.

7. Use the prob_mis() function to create misprob_udd with psi0 = -7, psi1 = 0, psi2 = 0 and psi3 = 0.003. Then, as before, use rbinom() to create a missingness indicator R_udd, and ifelse() to create income_udd (NA where R_udd equals 0).
income <-
  income |>
  mutate(
    misprob_udd = prob_mis(
      psi0   = -7,
      psi1   = 0,
      psi2   = 0,
      psi3   = 0.003,
      age    = age,
      gender = gender,
      income = income
    ),
    R_udd = 1 - rbinom(n(), 1, misprob_udd),
    income_udd = ifelse(R_udd == 0, NA, income)
  )
8. Create a plot that shows how the probability of having a missing value misprob_udd (on the y-axis) depends on income (on the x-axis).

Note: In practice, we would never be able to do this, because we would not have the actual incomes for those who refuse to provide their income values.

income |> 
  ggplot(aes(x = income, y = misprob_udd)) +
  geom_line() +
  geom_point() +
  theme_minimal() +
  labs(
    x = "Income", 
    y = "Missingness probability",
    title = "Unseen data dependent missingness"
  )

Visualizing missing data

9. Create a boxplot of the observed values and missing values of income under each of the three missing data models. What do you notice?
# NB: this would be the "tidyverse" way, 
# but you are of course free to do this 
# separately for each mechanism.

income |> 
  pivot_longer(
    cols = c(R_ndd, R_sdd, R_udd),
    names_to = "mechanism",
    values_to = "observed",
    names_prefix = "R_"
  ) |>
  mutate(observed = factor(ifelse(observed == 0, "missing", "observed"))) |> 
  ggplot(aes(x = str_to_upper(mechanism), y = income, fill = observed)) +
  geom_boxplot() + 
  theme_minimal() + 
  scale_fill_manual(values = c("seagreen", "lightblue")) +
  labs(
    x = "Missingness mechanism", 
    y = "Income value",
    colour = ""
  )
Ignoring unknown labels:
• colour : ""

# These boxplots display the distributions of $Y_1$ for 
# the observed and the missing group under the three 
# different missing data mechanisms. We see that under 
# NDD, the distributions are approximately similar, while 
# they become progressively more distinct under SDD and 
# UDD.

Mean imputation

We will now show why mean imputation is, in general, a bad idea.

10. Use mutate() in combination with ifelse() and mean(x, na.rm = TRUE) to create imp_income_ndd, imp_income_sdd and imp_income_udd, in which missings on income_ndd, income_sdd and income_udd are replaced with the means of the respective variables. Then compare the means and standard deviations of the complete, observed income variable with each of the imputed variables. What differences do you see?
# with across() we can apply a mutate over several columns
# of course, it's also fine to do this one-by-one!
# (in fact that would be the same number of lines and 
# perhaps even more legible)
income <- 
  income |>
  mutate(across(
    .cols  = c(income_ndd, income_sdd, income_udd),
    .fns   = function(x) replace_na(x, mean(x, na.rm = TRUE)),
    .names = "imp_{.col}"
  ))

# again, there are many ways to do this
# we'll stay "tidyversy"
income |> 
  pivot_longer(
    cols = c(income, starts_with("imp_")), 
    names_to = "mechanism", 
    values_to = "value"
  ) |> 
  group_by(mechanism) |> 
  summarize(mean = mean(value), sd = sd(value))
# A tibble: 4 × 3
  mechanism       mean    sd
  <chr>          <dbl> <dbl>
1 imp_income_ndd 2291.  155.
2 imp_income_sdd 2297.  180.
3 imp_income_udd 2250.  165.
4 income         2306.  224.
11. Create a plot in which you visualize the density or histogram of observed income, imp_income_ndd, imp_income_sdd, imp_income_udd. What do you see? Do you think it is a good idea to do mean imputation?
income |> 
  pivot_longer(
    cols = c(income, starts_with("imp_")), 
    names_to = "mechanism", 
    values_to = "value"
  ) |> 
  ggplot(aes(x = value, fill = mechanism)) +
  geom_histogram() +
  facet_wrap(vars(mechanism)) +
  theme_minimal() +
  scale_fill_viridis_d()
`stat_bin()` using `bins = 30`. Pick better value `binwidth`.

# The mean imputation creates an enormous
# "peak" in the middle that is not there 
# in the real data.

Mean imputation clearly distorts the distribution. In Part 2, we work through better alternatives, ending with multiple imputation.

Part 2: Ad hoc and multiple imputation with mice

Before we start, set a seed for reproducibility, we use 123, and also use options(scipen = 999) to suppress scientific notations, making it easier to compare and interpret results later in the session.

set.seed(123)
options(scipen = 999)

Data processing

For this part of the practical, you can directly copy and paste the code as given.

From the ISLR package, we will use the College dataset. Check out help(College) to find out more about this dataset. We will only use the following four variables:

  • Outstate: Out-of-state tuition
  • PhD: Pct. of faculty with Ph.D.’s
  • Terminal: Pct. of faculty with terminal degree
  • Expend: Instructional expenditure per student

Hence, we create a new data frame containing only these four variables:

data_complete <- 
  College |> 
  select(Outstate, PhD, Terminal, Expend) 

summary(data_complete)
    Outstate          PhD            Terminal         Expend     
 Min.   : 2340   Min.   :  8.00   Min.   : 24.0   Min.   : 3186  
 1st Qu.: 7320   1st Qu.: 62.00   1st Qu.: 71.0   1st Qu.: 6751  
 Median : 9990   Median : 75.00   Median : 82.0   Median : 8377  
 Mean   :10441   Mean   : 72.66   Mean   : 79.7   Mean   : 9660  
 3rd Qu.:12925   3rd Qu.: 85.00   3rd Qu.: 92.0   3rd Qu.:10830  
 Max.   :21700   Max.   :103.00   Max.   :100.0   Max.   :56233  

During the practical, we focus on how different imputation approaches affect the linear relationship between Outstate and Expend. Therefore, we first inspect how this relationship looks like on the complete dataset.

lm_complete <- lm(Expend ~ Outstate, 
                  data = data_complete)

summary(lm_complete)

Call:
lm(formula = Expend ~ Outstate, data = data_complete)

Residuals:
   Min     1Q Median     3Q    Max 
 -6131  -2022   -637   1027  39273 

Coefficients:
             Estimate Std. Error t value            Pr(>|t|)    
(Intercept) 542.86969  385.92915   1.407                0.16    
Outstate      0.87325    0.03449  25.315 <0.0000000000000002 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3866 on 775 degrees of freedom
Multiple R-squared:  0.4526,    Adjusted R-squared:  0.4519 
F-statistic: 640.9 on 1 and 775 DF,  p-value: < 0.00000000000000022

We now generate missingness in the variables Expend and Terminal under a Seen Data-Dependent (SDD) mechanism, the same mechanism you generated by hand in Part 1, assuming that the missingness in these variables are dependent on the values of Outstate and PhD respectively.

logistic <- function(x) (exp(x) / (1 + exp(x))) # logistic function

N <- nrow(data_complete) # number of observations

misprob_out <- logistic(scale(data_complete$Outstate)) # predictor Outstate
misprob_phd <- logistic(scale(data_complete$PhD)) # predictor PhD

data_missing <- 
  data_complete |> 
  mutate(
    R_out = 1 - rbinom(N, 1, misprob_out), # reponse indicator outstate
    R_phd = 1 - rbinom(N, 1, misprob_phd), # response indicator PhD
    m_Expend = ifelse(R_out == 0, NA, Expend),    # Missing in expend
    m_Terminal = ifelse(R_phd == 0, NA, Terminal) # and in terminal
  ) |> 
  select(Outstate, m_Expend, m_Terminal) # Select variables that we will use

Inspection of missingness

12. Inspect the missingness pattern of the dataset using the function plot_pattern() from the ggmice package.
plot_pattern(data_missing)

Ad hoc imputation methods

Listwise deletion

13. Create the object lm_listwise by fitting a regression model with m_Expend as the outcome, predicted by Outstate, using the data containing missing values. Then inspect it with summary(). How many observations are excluded? Is there a significant effect of Outstate?

Hint: By default, lm() removes the observations that have missings on one of the variables in the model.

lm_listwise <- lm(m_Expend ~ Outstate,
                  data = data_missing)

summary(lm_listwise)

Call:
lm(formula = m_Expend ~ Outstate, data = data_missing)

Residuals:
    Min      1Q  Median      3Q     Max 
-4986.2 -1490.9  -398.1   736.4 30053.4 

Coefficients:
              Estimate Std. Error t value             Pr(>|t|)    
(Intercept) 2006.74472  396.63965   5.059          0.000000641 ***
Outstate       0.70211    0.04153  16.904 < 0.0000000000000002 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2758 on 402 degrees of freedom
  (373 observations deleted due to missingness)
Multiple R-squared:  0.4155,    Adjusted R-squared:  0.414 
F-statistic: 285.8 on 1 and 402 DF,  p-value: < 0.00000000000000022
# The coefficient is a bit smaller, the standard error somewhat larger, so we
# introduced some bias, and reduced the power. Both could have been expected,
# due to how we generated missings and since we lost 373 cases.

Regression imputation

14. Use mice() to impute data_missing with regression imputation (method = "norm.predict"), using a single imputation (m = 1) and a single iteration (maxit = 1), name the output imp_regression, then extract the complete data with complete() and confirm there are no more missing values.

NB: m = 1 implies that only a single imputed data set is created, whereas maxit = 1 implies that mice() does not iterate to converge to a distribution.

imp_regression <- mice(
  data   = data_missing,
  method = "norm.predict",
  m      = 1,
  maxit  = 1
)

 iter imp variable
  1   1  m_Expend  m_Terminal
complete(imp_regression) |>
  is.na() |>
  sum()
[1] 0

The with(data, expr) function allows to fit lm() and glm() models with imputed data sets (mids-objects, to be precise), without requiring any further data handling. Check how the function works by running ?with.mids().

15. Use with() to fit lm(m_Expend ~ Outstate) on imp_regression, assign it to lm_regression, and inspect the results. What happens to the regression coefficient and the standard error compared to listwise deletion?
lm_regression <- with(imp_regression, lm(m_Expend ~ Outstate))
summary(lm_regression)
# A tibble: 2 × 6
  term        estimate std.error statistic   p.value  nobs
  <chr>          <dbl>     <dbl>     <dbl>     <dbl> <int>
1 (Intercept) 1989.     199.          9.97 4.05e- 22   777
2 Outstate       0.703    0.0178     39.5  1.44e-187   777
# The regression coefficient is almost identical to the coefficient from the
# model with listwise deletion, which is not weird, because we used this
# relationship to impute the missing data. The standard error is substantially
# smaller though, because we treat the imputed values as the true values, and
# given that all are on the regression line, we are overly certain about the
# estimated regression coefficient.

# From an inferential statistics viewpoint, this approach would only be valid 
# if we have definitive proof that the unobserved values would exactly conform 
# to the observed data. If this assumption does not hold, `norm.predict` 
# creates too little variation in our dataset and we can not trust the 
# resulting standard errors and p-values.
16. Use ggmice() to create a scatterplot of m_Expend (y) versus Outstate (x) for the regression-imputed data. What do you notice?

Hint: ggmice() works much the same way as ggplot(), but accepts mids objects. Check ?ggmice for help.

imp_regression |> 
  ggmice(aes(x = Outstate, y = m_Expend)) +
  geom_point()

# Almost all imputed values are on a (almost) straight regression line (which 
# is how we specified the imputation model). The variation around this line
# comes from the other variables that are also used in calculating the
# predictions.

Multiple imputation

We skip straight to the proper solution: multiple imputation, which, unlike the ad hoc methods above, accounts for the uncertainty in both the imputed values themselves and in the regression coefficients used to generate them.

17. Impute the missing data by calling mice() with default settings, assign the result to imp_multiple, and inspect the output object. What information does it give you?
imp_multiple <- mice(data_missing)

 iter imp variable
  1   1  m_Expend  m_Terminal
  1   2  m_Expend  m_Terminal
  1   3  m_Expend  m_Terminal
  1   4  m_Expend  m_Terminal
  1   5  m_Expend  m_Terminal
  2   1  m_Expend  m_Terminal
  2   2  m_Expend  m_Terminal
  2   3  m_Expend  m_Terminal
  2   4  m_Expend  m_Terminal
  2   5  m_Expend  m_Terminal
  3   1  m_Expend  m_Terminal
  3   2  m_Expend  m_Terminal
  3   3  m_Expend  m_Terminal
  3   4  m_Expend  m_Terminal
  3   5  m_Expend  m_Terminal
  4   1  m_Expend  m_Terminal
  4   2  m_Expend  m_Terminal
  4   3  m_Expend  m_Terminal
  4   4  m_Expend  m_Terminal
  4   5  m_Expend  m_Terminal
  5   1  m_Expend  m_Terminal
  5   2  m_Expend  m_Terminal
  5   3  m_Expend  m_Terminal
  5   4  m_Expend  m_Terminal
  5   5  m_Expend  m_Terminal
# The imputations are now done. As you can see, the algorithm ran for 5 
# iterations (the default: `maxit = 5`) and presented us 5 imputations (the
# default: `m = 5`) for each missing datum.

imp_multiple
Class: mids
Number of multiple imputations:  5 
Imputation methods:
  Outstate   m_Expend m_Terminal 
        ""      "pmm"      "pmm" 
PredictorMatrix:
           Outstate m_Expend m_Terminal
Outstate          0        1          1
m_Expend          1        0          1
m_Terminal        1        1          0
# We have created m = 5 imputed data sets, using the method predictive mean 
# matching. Moreover, all variables are used in the imputation models, as
# shown by the variables in the columns that are used as predictors for the
# variable in each row.

# Moreover, the imp_multiple object contains the data with missings, the
# imputations, etc.
# To obtain a full overview of the information stored in the object 
# `imp_multiple`, we can use the `attributes()` function.
18. Perform the linear regression on the imputed data using with() and assign it to lm_multiple. What do you notice when you inspect this object?
lm_multiple <- with(imp_multiple, lm(m_Expend ~ Outstate))

# The `lm_multiple` object contains the regression summaries for each of the 5
# datasets. The new object `lm_multiple` is actually of class `mira`, which 
# stands for Multiply Imputed Repeated Analyses.
19. Use pool() to combine the regression analyses into pool_multiple. Inspect pool_multiple and summary(pool_multiple). What information do these objects provide?
pool_multiple <- pool(lm_multiple)
summary(pool_multiple)
         term    estimate    std.error statistic       df        p.value
1 (Intercept) 631.8857390 529.91732090  1.192423 10.07890 0.260412582554
2    Outstate   0.8563247   0.06139237 13.948389  6.44349 0.000004703746
# These results show the regression coefficient and corresponding standard 
# error obtained after pooling the estimates from the five regression models
# applied on the five imputed datasets.

# The complete pooled object gives the relevant pooled regression coefficients 
# and parameters, as well as the fraction of information about the coefficients
# missing due to nonresponse (`fmi`) and the proportion of the variation
# attributable to the missing data (`lambda`). The pooled fit object is of 
# class `mipo`, which stands for multiply imputed pooled object.

Unlike regression imputation, multiple imputation carries the uncertainty from both the imputed values and the estimated coefficients through into the pooled standard error, which is exactly what the lambda and fmi figures above quantify.

20. Now compare the results from the different methods applied (complete data, listwise deletion, regression imputation, multiple imputation). What is your conclusion?
data.frame("method" = c("complete data", 
                        "listwise deletion", 
                        "regression imputation",
                        "multiple imputation"),
           "coefficient" = c(summary(lm_complete)$coefficients[2,1],
                             summary(lm_listwise)$coefficients[2,1],
                             summary(lm_regression)$estimate[2],
                             summary(pool_multiple)$estimate[2]),
           "standard error" = c(summary(lm_complete)$coefficients[2,2],
                                summary(lm_listwise)$coefficients[2,2],
                                summary(lm_regression)$std.error[2],
                                summary(pool_multiple)$std.error[2]))
                 method coefficient standard.error
1         complete data   0.8732488     0.03449491
2     listwise deletion   0.7021111     0.04153479
3 regression imputation   0.7034919     0.01782889
4   multiple imputation   0.8563247     0.06139237
# When comparing the results from the different approaches, it can be 
# concluded that in case of a dataset where the missingness follows the 
# mechanism Seen Data Dependent, the estimate obtained after applying multiple
# imputation is closest to the original estimate obtained from the complete
# dataset. In addition, it can be concluded that the standard error obtained 
# using multiple imputation is somewhat larger than the standard error obtain 
# from the complete dataset. This is due to the fact that when the standard 
# errors are pooled over the different imputations, not only the average is 
# taken, but the variance over the differences is also incorporated (check 
# the pooling rules for more detail), i.e. the standard error also incorporates
# uncertainty due to the missing values. In contrast, the results obtained 
# using listwise deletion and regression imputation both show a biased 
# estimate, while regression imputation also shows a standard error that is 
# too low.