Clustering: hierarchical, k-means, and model-based (mclust)

Published

October 6, 2026

Introduction

In this practical, we apply three clustering approaches. In Part 1, we use hierarchical and k-means clustering on synthetic bivariate data, to build intuition about how distance metrics, linkage, and the number of clusters affect the result, and about the instability of k-means across runs. In Part 2, we move to a real dataset of Swiss banknote measurements and use model-based clustering (mclust) to fit and compare probabilistic cluster models, including choosing the number of clusters and comparing model fit via BIC.

library(MASS)
library(tidyverse)
library(patchwork)
library(ggdendro)
library(mclust)

Make sure to load MASS before tidyverse, otherwise the function MASS::select() will overwrite dplyr::select().

Part 1: Hierarchical and k-means clustering

Before we start, set a seed for reproducibility, we use 123, and also use options(scipen = 999) to suppress scientific notations, making it easier to compare and interpret results later in the session.

set.seed(123)
options(scipen = 999)

Data processing

1. The data can be generated by running the code below. Try to understand what is happening as you run each line of the code below.
# randomly generate bivariate normal data
set.seed(123)
sigma      <- matrix(c(1, .5, .5, 1), 2, 2)
sim_matrix <- mvrnorm(n = 100, mu = c(5, 5), Sigma = sigma)
colnames(sim_matrix) <- c("x1", "x2")

# change to a data frame (tibble) and add a cluster label column
sim_df <- 
  sim_matrix |> 
  as_tibble() |>
  mutate(class = sample(c("A", "B", "C"), size = 100, replace = TRUE))

# Move the clusters to generate separation
sim_df_small <- 
  sim_df |>
  mutate(x2 = case_when(class == "A" ~ x2 + .5,
                        class == "B" ~ x2 - .5,
                        class == "C" ~ x2 + .5),
         x1 = case_when(class == "A" ~ x1 - .5,
                        class == "B" ~ x1 - 0,
                        class == "C" ~ x1 + .5))
sim_df_large <- 
  sim_df |>
  mutate(x2 = case_when(class == "A" ~ x2 + 2.5,
                        class == "B" ~ x2 - 2.5,
                        class == "C" ~ x2 + 2.5),
         x1 = case_when(class == "A" ~ x1 - 2.5,
                        class == "B" ~ x1 - 0,
                        class == "C" ~ x1 + 2.5))
2. Prepare two unsupervised datasets by removing the class feature.
df_s <- sim_df_small |> select(-class)
df_l <- sim_df_large |> select(-class)
3. For each of these datasets, create a scatterplot. Combine the two plots into a single frame (look up the “patchwork” package to see how to do this!) What is the difference between the two datasets?
# patchwork defines the "+" operator to combine entire ggplots!
df_s |> ggplot(aes(x = x1, y = x2)) + geom_point() + ggtitle("Small") +
df_l |> ggplot(aes(x = x1, y = x2)) + geom_point() + ggtitle("Large")

# df_s has a lot of class overlap, df_l has very little overlap

Hierarchical clustering

4. Run a hierarchical clustering on these datasets and display the result as dendrograms. Use euclidian distances and the complete agglomeration method. Make sure the two plots have the same y-scale. What is the difference between the dendrograms?

Hint: functions you will need are hclust, ggdendrogram, and ylim.

dist_s <- dist(df_s, method = "euclidian")
dist_l <- dist(df_l, method = "euclidian")

res_s <- hclust(dist_s, method = "complete")
res_l <- hclust(dist_l, method = "complete")

ggdendrogram(res_s, labels = FALSE) + ggtitle("Small") + ylim(0, 10) +
ggdendrogram(res_l, labels = FALSE) + ggtitle("Large") + ylim(0, 10)
Scale for y is already present.
Adding another scale for y, which will replace the existing scale.
Scale for y is already present.
Adding another scale for y, which will replace the existing scale.

# The dataset with large differences segments into 3 classes much higher up.
# Interestingly, the microstructure (lower splits) is almost exactly the same,
# because within the three clusters there is no difference between the datasets.
5. For the dataset with small differences, also run a complete agglomeration hierarchical cluster with manhattan distance.
dist_s2 <- dist(df_s, method = "manhattan")
res_s2  <- hclust(dist_s2, method = "complete")
6. Use the cutree() function to obtain the cluster assignments for three clusters and compare the cluster assignments to the 3-cluster euclidian solution. Do this comparison by creating two scatter plots with cluster assignment mapped to the colour aesthetic. Which difference do you see?
clus_1 <- as_factor(cutree(res_s, 3))
clus_2 <- as_factor(cutree(res_s2, 3))

p1 <- df_s |> 
  ggplot(aes(x = x1, y = x2, colour = clus_1)) + 
  geom_point() + 
  ggtitle("Euclidian") + 
  theme(legend.position = "n") +
  coord_fixed()

p2 <- df_s |> 
  ggplot(aes(x = x1, y = x2, colour = clus_2)) + 
  geom_point() + 
  ggtitle("Manhattan") + 
  theme(legend.position = "n") +
  coord_fixed()

p1 + p2

# The manhattan distance clustering prefers more rectangular classes, whereas
# the euclidian distance clustering prefers circular classes. The difference is
# most prominent in the very center of the plot and for the top right cluster.

K-means clustering

7. Create k-means clustering with 2, 3, 4, and 6 clusters on the large difference data. Again, create coloured scatter plots for these clustering results.
# Let's set the seed for reproducibility
set.seed(45)
# we can do it in a single pipeline and with faceting 
# (of course there are many ways to do this, though)
df_l |> 
  mutate(
    class_2 = as_factor(kmeans(df_l, 2)$cluster),
    class_3 = as_factor(kmeans(df_l, 3)$cluster),
    class_4 = as_factor(kmeans(df_l, 4)$cluster),
    class_6 = as_factor(kmeans(df_l, 6)$cluster)
  ) |> 
  pivot_longer(cols = c(class_2, class_3, class_4, class_6), 
               names_to = "class_num", values_to = "cluster") |> 
  ggplot(aes(x = x1, y = x2, colour = cluster)) +
  geom_point() +
  scale_colour_brewer(type = "qual") + # use easy to distinguish scale
  facet_wrap(~class_num)

8. Do the same thing again a few times. Do you see the same results every time? where do you see differences?
# Let's set the seed for reproducibility
set.seed(46)
df_l |> 
  mutate(
    class_2 = as_factor(kmeans(df_l, 2)$cluster),
    class_3 = as_factor(kmeans(df_l, 3)$cluster),
    class_4 = as_factor(kmeans(df_l, 4)$cluster),
    class_6 = as_factor(kmeans(df_l, 6)$cluster)
  ) |> 
  pivot_longer(cols = c(class_2, class_3, class_4, class_6), 
               names_to = "class_num", values_to = "cluster") |> 
  ggplot(aes(x = x1, y = x2, colour = cluster)) +
  geom_point() +
  scale_colour_brewer(type = "qual") + # use easy to distinguish scale
  facet_wrap(~class_num)

# There is label switching in all plots. There is a different result altogether
# in the class_4 solution in this case.
9. OPTIONAL (if time permits): Find a way online to perform bootstrap stability assessment for the 3 and 6-cluster solutions.
# For this we decided to use the function clusterboot from the fpc package.
# NB: this package needs to be installed first!
# install.packages("fpc")
library(fpc)
Warning: package 'fpc' was built under R version 4.3.3
# You can read the documentation ?clusterboot to figure out how to use this 
# function. This can take a while but is something you will need to do a lot in
# the real world!
boot_3 <- clusterboot(df_l, B = 1000, clustermethod = kmeansCBI, k = 3, 
                      count = FALSE)
boot_6 <- clusterboot(df_l, B = 1000, clustermethod = kmeansCBI, k = 6, 
                      count = FALSE)

# the average stability is much lower for 6 means than for 3 means:
boot_3$bootmean
[1] 0.9844165 0.9730219 0.9739694
boot_6$bootmean
[1] 0.7844248 0.5383414 0.7593547 0.7230086 0.6897734 0.7091287

Hierarchical and k-means clustering both partition observations into hard, non-overlapping groups based on distance. In Part 2, we take a different, probabilistic approach: model-based clustering, which fits a mixture of distributions to the data and lets us evaluate fit statistically (via BIC) rather than only visually.

Part 2: Model-based clustering using mclust

We apply model-based clustering on a data set of bank note measurements. The data is built into the mclust package and can be loaded as a tibble:

df <- as_tibble(banknote)

Data exploration

10. Read the help file of the banknote data set to understand what it’s all about.
?banknote
11. Create a scatter plot of the left (x-axis) and right (y-axis) measurements on the data set. Map the Status column to colour. Jitter the points to avoid overplotting. Are the classes easy to distinguish based on these features?
df |> 
  ggplot(aes(x = Left, y = Right, colour = Status)) +
  geom_point(position = position_jitter())

# the classes are not easy to distinguish: there is considerable overlap.
12. From now on, we will assume that we don’t have the labels. Remove the Status column from the data set.
df <- df |> select(-Status)
13. Create density plots for all columns in the data set. Which single feature is likely to be best for clustering?
# big patchwork of density plots
df |> ggplot(aes(x = Length))   + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Left))     + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Right))    + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Bottom))   + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Top))      + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Diagonal)) + geom_density() + theme_minimal()

# the Diagonal feature looks good! Look at the two bumps in its density plot.

# Colourful alternative:
library(ggridges)
df |> 
  mutate(across(everything(), scale)) |> 
  pivot_longer(everything(), names_to = "feature", values_to = "value") |> 
  ggplot(aes(x = value, y = feature, fill = feature)) + 
  geom_density_ridges() +
  scale_fill_viridis_d(guide = "none") +
  theme_minimal()
Picking joint bandwidth of 0.31

Univariate model-based clustering

14. Use Mclust to perform model-based clustering with 2 clusters on the feature you chose. Assume equal variances. Name the model object fit_E_2. What are the means and variances of the clusters?
fit_E_2 <- Mclust(df$Diagonal, G = 2, modelNames = "E")
# means:
fit_E_2$parameters$mean
       1        2 
139.4464 141.5221 
# variances:
fit_E_2$parameters$variance$sigmasq
[1] 0.244004
15. (Optional) Use the formula from the slides and the model’s log-likelihood (fit_E_2$loglik) to compute the BIC for this model. Compare it to the BIC stored in the model object (fit_E_2$bic). Explain how many parameters (m) you used and which parameters these are.
# This model has 4 parameters: 1 class probability pi, 2 means, and 1 variance
- 2 * fit_E_2$loglik + log(nrow(df)) * 4 
[1] 569.4667
fit_E_2$bic
[1] -569.4667
# note: the BIC from the Mclust package is (wrongly) multiplied by -1!
16. Plot the model-implied density using the plot() function. Afterwards, add rug marks of the original data to the plot using the rug() function from the base graphics system.
# plot the density using base R plots
plot(fit_E_2, "density", xlab = "Diagonal measurement")
# add the observations using rug marks
rug(df$Diagonal)

17. Use Mclust to perform model-based clustering with 2 clusters on this feature again, but now assume unequal variances. Name the model object fit_V_2. What are the means and variances of the clusters? Plot the density again and note the differences.
fit_V_2 <- Mclust(df$Diagonal, G = 2, modelNames = "V")
fit_V_2$parameters$mean
       1        2 
139.4973 141.5604 
fit_V_2$parameters$variance$sigmasq
[1] 0.3589844 0.1500838
plot(fit_V_2, "density", xlab = "Diagonal measurement")
rug(df$Diagonal)

# The left cluster has larger variance now, because the data is more spread out
18. (Optional) How many parameters does this model have? Name them.
# 5 parameters:
# 1 class probability (pi)
# 2 means
# 2 variances
19. According to the deviance, which model fits better?
-2*fit_E_2$loglik
[1] 548.2735
-2*fit_V_2$loglik
[1] 537.0199
# The V model fits better (lower deviance)
20. According to the BIC, which model is better?
# remember, the package shows negative BIC
-fit_E_2$bic
[1] 569.4667
-fit_V_2$bic
[1] 563.5115
# the V model fits better (lower BIC), but it's closer because of the extra parameter!

Multivariate model-based clustering

We will now use all available information in the data set to cluster the observations.

21. Use Mclust with all 6 features to perform clustering. Allow all model types (shapes), and from 1 to 9 potential clusters. What is the optimal model based on the BIC?
fit <- Mclust(df)
summary(fit)
---------------------------------------------------- 
Gaussian finite mixture model fitted by EM algorithm 
---------------------------------------------------- 

Mclust VVE (ellipsoidal, equal orientation) model with 3 components: 

 log-likelihood   n df       BIC      ICL
      -663.3814 200 53 -1607.574 -1607.71

Clustering table:
 1  2  3 
18 98 84 
# The optimal model is a VVE model with 3 components
22. How many mean parameters does this model have?
fit$parameters$mean
               [,1]       [,2]      [,3]
Length   215.023017 214.971360 214.78091
Left     130.499684 129.929686 130.26435
Right    130.304813 129.701143 130.17988
Bottom     8.775842   8.301115  10.85719
Top       11.173812  10.162379  11.10798
Diagonal 138.731602 141.541673 139.62387
# 3 clusters * 6 variables = 18 mean parameters
23. OPTIONAL (if time permits): Run a 2-component VVV model on this data. Create a matrix of bivariate contour (“density”) plots using the plot() function. Which features provide good component separation? Which do not?
VVV2 <- Mclust(df, 2, "VVV")
plot(VVV2, "density")

# Diagonal-top, and diagonal-bottom separate the classes very well!
24. OPTIONAL (if time permits): Create a scatter plot just like the first scatter plot in this tutorial, but map the estimated class assignments to the colour aesthetic.

Map the uncertainty (part of the fitted model list) to the size aesthetic, such that larger points indicate more uncertain class assignments. Jitter the points to avoid overplotting. What do you notice about the uncertainty?

(This uses the VVV2 model from the previous optional exercise. Skip both together if you’re short on time.)

df |> 
  ggplot(aes(
    x      = Left, 
    y      = Right,
    colour = as_factor(VVV2$classification), 
    size   = VVV2$uncertainty 
  )) +
  geom_point(position = position_jitter()) +
  scale_size(range = c(1, 3)) +
  labs(colour = "Cluster", size = "Uncertainty")

# only 3 points are slightly uncertain, and they are not 
# around the border of the classes in these two dimensions. 
# Apparently, the other dimensions give enough information 
# about those points near the border!