library(MASS)
library(tidyverse)
library(patchwork)
library(ggdendro)
library(mclust)Clustering: hierarchical, k-means, and model-based (mclust)
October 6, 2026
Introduction
In this practical, we apply three clustering approaches. In Part 1, we use hierarchical and k-means clustering on synthetic bivariate data, to build intuition about how distance metrics, linkage, and the number of clusters affect the result, and about the instability of k-means across runs. In Part 2, we move to a real dataset of Swiss banknote measurements and use model-based clustering (mclust) to fit and compare probabilistic cluster models, including choosing the number of clusters and comparing model fit via BIC.
Make sure to load MASS before tidyverse, otherwise the function MASS::select() will overwrite dplyr::select().
Part 1: Hierarchical and k-means clustering
Before we start, set a seed for reproducibility, we use 123, and also use options(scipen = 999) to suppress scientific notations, making it easier to compare and interpret results later in the session.
Data processing
# randomly generate bivariate normal data
set.seed(123)
sigma <- matrix(c(1, .5, .5, 1), 2, 2)
sim_matrix <- mvrnorm(n = 100, mu = c(5, 5), Sigma = sigma)
colnames(sim_matrix) <- c("x1", "x2")
# change to a data frame (tibble) and add a cluster label column
sim_df <-
sim_matrix |>
as_tibble() |>
mutate(class = sample(c("A", "B", "C"), size = 100, replace = TRUE))
# Move the clusters to generate separation
sim_df_small <-
sim_df |>
mutate(x2 = case_when(class == "A" ~ x2 + .5,
class == "B" ~ x2 - .5,
class == "C" ~ x2 + .5),
x1 = case_when(class == "A" ~ x1 - .5,
class == "B" ~ x1 - 0,
class == "C" ~ x1 + .5))
sim_df_large <-
sim_df |>
mutate(x2 = case_when(class == "A" ~ x2 + 2.5,
class == "B" ~ x2 - 2.5,
class == "C" ~ x2 + 2.5),
x1 = case_when(class == "A" ~ x1 - 2.5,
class == "B" ~ x1 - 0,
class == "C" ~ x1 + 2.5))Hierarchical clustering
Hint: functions you will need are hclust, ggdendrogram, and ylim.
dist_s <- dist(df_s, method = "euclidian")
dist_l <- dist(df_l, method = "euclidian")
res_s <- hclust(dist_s, method = "complete")
res_l <- hclust(dist_l, method = "complete")
ggdendrogram(res_s, labels = FALSE) + ggtitle("Small") + ylim(0, 10) +
ggdendrogram(res_l, labels = FALSE) + ggtitle("Large") + ylim(0, 10)Scale for y is already present.
Adding another scale for y, which will replace the existing scale.
Scale for y is already present.
Adding another scale for y, which will replace the existing scale.
cutree() function to obtain the cluster assignments for three clusters and compare the cluster assignments to the 3-cluster euclidian solution. Do this comparison by creating two scatter plots with cluster assignment mapped to the colour aesthetic. Which difference do you see?
clus_1 <- as_factor(cutree(res_s, 3))
clus_2 <- as_factor(cutree(res_s2, 3))
p1 <- df_s |>
ggplot(aes(x = x1, y = x2, colour = clus_1)) +
geom_point() +
ggtitle("Euclidian") +
theme(legend.position = "n") +
coord_fixed()
p2 <- df_s |>
ggplot(aes(x = x1, y = x2, colour = clus_2)) +
geom_point() +
ggtitle("Manhattan") +
theme(legend.position = "n") +
coord_fixed()
p1 + p2K-means clustering
# Let's set the seed for reproducibility
set.seed(45)
# we can do it in a single pipeline and with faceting
# (of course there are many ways to do this, though)
df_l |>
mutate(
class_2 = as_factor(kmeans(df_l, 2)$cluster),
class_3 = as_factor(kmeans(df_l, 3)$cluster),
class_4 = as_factor(kmeans(df_l, 4)$cluster),
class_6 = as_factor(kmeans(df_l, 6)$cluster)
) |>
pivot_longer(cols = c(class_2, class_3, class_4, class_6),
names_to = "class_num", values_to = "cluster") |>
ggplot(aes(x = x1, y = x2, colour = cluster)) +
geom_point() +
scale_colour_brewer(type = "qual") + # use easy to distinguish scale
facet_wrap(~class_num)# Let's set the seed for reproducibility
set.seed(46)
df_l |>
mutate(
class_2 = as_factor(kmeans(df_l, 2)$cluster),
class_3 = as_factor(kmeans(df_l, 3)$cluster),
class_4 = as_factor(kmeans(df_l, 4)$cluster),
class_6 = as_factor(kmeans(df_l, 6)$cluster)
) |>
pivot_longer(cols = c(class_2, class_3, class_4, class_6),
names_to = "class_num", values_to = "cluster") |>
ggplot(aes(x = x1, y = x2, colour = cluster)) +
geom_point() +
scale_colour_brewer(type = "qual") + # use easy to distinguish scale
facet_wrap(~class_num)# For this we decided to use the function clusterboot from the fpc package.
# NB: this package needs to be installed first!
# install.packages("fpc")
library(fpc)Warning: package 'fpc' was built under R version 4.3.3
# You can read the documentation ?clusterboot to figure out how to use this
# function. This can take a while but is something you will need to do a lot in
# the real world!
boot_3 <- clusterboot(df_l, B = 1000, clustermethod = kmeansCBI, k = 3,
count = FALSE)
boot_6 <- clusterboot(df_l, B = 1000, clustermethod = kmeansCBI, k = 6,
count = FALSE)
# the average stability is much lower for 6 means than for 3 means:
boot_3$bootmean[1] 0.9844165 0.9730219 0.9739694
[1] 0.7844248 0.5383414 0.7593547 0.7230086 0.6897734 0.7091287
Hierarchical and k-means clustering both partition observations into hard, non-overlapping groups based on distance. In Part 2, we take a different, probabilistic approach: model-based clustering, which fits a mixture of distributions to the data and lets us evaluate fit statistically (via BIC) rather than only visually.
Part 2: Model-based clustering using mclust
We apply model-based clustering on a data set of bank note measurements. The data is built into the mclust package and can be loaded as a tibble:
Data exploration
Status column to colour. Jitter the points to avoid overplotting. Are the classes easy to distinguish based on these features?
Status column from the data set.
# big patchwork of density plots
df |> ggplot(aes(x = Length)) + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Left)) + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Right)) + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Bottom)) + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Top)) + geom_density() + theme_minimal() +
df |> ggplot(aes(x = Diagonal)) + geom_density() + theme_minimal()# the Diagonal feature looks good! Look at the two bumps in its density plot.
# Colourful alternative:
library(ggridges)
df |>
mutate(across(everything(), scale)) |>
pivot_longer(everything(), names_to = "feature", values_to = "value") |>
ggplot(aes(x = value, y = feature, fill = feature)) +
geom_density_ridges() +
scale_fill_viridis_d(guide = "none") +
theme_minimal()Picking joint bandwidth of 0.31
Univariate model-based clustering
Mclust to perform model-based clustering with 2 clusters on the feature you chose. Assume equal variances. Name the model object fit_E_2. What are the means and variances of the clusters?
fit_E_2$loglik) to compute the BIC for this model. Compare it to the BIC stored in the model object (fit_E_2$bic). Explain how many parameters (m) you used and which parameters these are.
plot() function. Afterwards, add rug marks of the original data to the plot using the rug() function from the base graphics system.
Mclust to perform model-based clustering with 2 clusters on this feature again, but now assume unequal variances. Name the model object fit_V_2. What are the means and variances of the clusters? Plot the density again and note the differences.
1 2
139.4973 141.5604
[1] 0.3589844 0.1500838
Multivariate model-based clustering
We will now use all available information in the data set to cluster the observations.
----------------------------------------------------
Gaussian finite mixture model fitted by EM algorithm
----------------------------------------------------
Mclust VVE (ellipsoidal, equal orientation) model with 3 components:
log-likelihood n df BIC ICL
-663.3814 200 53 -1607.574 -1607.71
Clustering table:
1 2 3
18 98 84
[,1] [,2] [,3]
Length 215.023017 214.971360 214.78091
Left 130.499684 129.929686 130.26435
Right 130.304813 129.701143 130.17988
Bottom 8.775842 8.301115 10.85719
Top 11.173812 10.162379 11.10798
Diagonal 138.731602 141.541673 139.62387
plot() function. Which features provide good component separation? Which do not?
Map the uncertainty (part of the fitted model list) to the size aesthetic, such that larger points indicate more uncertain class assignments. Jitter the points to avoid overplotting. What do you notice about the uncertainty?
(This uses the VVV2 model from the previous optional exercise. Skip both together if you’re short on time.)