Supervised learning: deep learning 2 / RNNs and Attention¶

In the previous practical we built feed-forward and convolutional networks for MNIST digit classification. Here we look at two more architectures: recurrent neural networks (RNNs) and attention. We use the same dataset so we can compare all four approaches directly.

RNNs read sequences one step at a time. An image isn't obviously a sequence, but we can make it one: treat each 28×28 digit as a sequence of 28 rows, each row a 28-pixel vector. A SimpleRNN or LSTM reads the image row by row, top to bottom, and builds up a hidden representation before classifying it, differently from a CNN's local, translation-invariant filters.

We'll then add attention on top, which learns how much weight to give each row's hidden state. That gives us something a CNN's filters don't hand us directly: a per-row importance score we can overlay on the digit and inspect visually.

Setup: installing TensorFlow¶

TensorFlow does not yet publish packages for very new Python versions (e.g. 3.13/3.14) — pip install tensorflow will fail with "No matching distribution found" on those. Use a dedicated conda environment pinned to Python 3.11 (any version in the 3.9–3.11 range works):

conda create -n daml python=3.11 -y
conda activate daml
pip install numpy matplotlib tensorflow
conda install jupyter -y
jupyter notebook
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Let's set the seed value and use the same number as below to reproduce the same results.

In [2]:
tf.random.set_seed(45)
np.random.seed(45)

Data preparation¶

As in the previous practical, we load MNIST and normalize pixel values to (0, 1). We won't add a channel dimension this time.

In [3]:
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train / 255
x_test = x_test / 255
In [4]:
def plot_img(img, cmap="gray_r", **kwargs):
    plt.imshow(img, cmap=cmap, **kwargs)
    plt.axis("off")
    plt.show()
In [5]:
def plot_history(history, title):
    acc = history.history["accuracy"]
    val_acc = history.history["val_accuracy"]
    loss = history.history["loss"]
    val_loss = history.history["val_loss"]
    x = range(1, len(acc) + 1)

    plt.figure(figsize=(12, 4))
    plt.subplot(1, 2, 1)
    plt.plot(x, acc, "b", label="Training accuracy")
    plt.plot(x, val_acc, "r", label="Validation accuracy")
    plt.title(f"{title} — accuracy")
    plt.xlabel("epoch")
    plt.legend()

    plt.subplot(1, 2, 2)
    plt.plot(x, loss, "b", label="Training loss")
    plt.plot(x, val_loss, "r", label="Validation loss")
    plt.title(f"{title} — loss")
    plt.xlabel("epoch")
    plt.legend()
    plt.tight_layout()
    plt.show()

1. Each MNIST image already has shape (28, 28), no reshaping needed. Explain why this shape already works as RNN input, and how it differs from what the CNN in the previous practical required.¶

Recall a Keras RNN layer expects input of shape (timesteps, features_per_timestep).

In [6]:
# (28, 28) is read directly as 28 timesteps (the rows, top to bottom),
# each with 28 features (the pixel values in that row), exactly the
# shape an RNN layer expects, so x_train / x_test can be fed in as-is.
#
# The CNN in the previous practical needed an extra channel dimension,
# (28, 28, 1), because Conv2D expects (height, width, channels), which is a
# spatial 2D structure rather than a sequence of rows.

Baseline: feed-forward network¶

A quick reference point, as in the previous practical.

2. Build and train a feed-forward network: flatten the input, a Dense(64, activation="relu") hidden layer, then a Dense(10, activation="softmax") output layer. Compile with loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"], and train for 5 epochs with validation_split=0.2. Plot the training history with plot_history().¶

In [7]:
ffnn = keras.Sequential([
    keras.Input(shape=(28, 28)),
    layers.Flatten(),
    layers.Dense(64, activation="relu"),
    layers.Dense(10, activation="softmax")
])

ffnn.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

history_ffnn = ffnn.fit(x_train, y_train, epochs=5, validation_split=0.2, verbose=2)
plot_history(history_ffnn, "Feed-forward baseline")
loss, acc = ffnn.evaluate(x_test, y_test, verbose=2)
print(f"Test accuracy: {acc:.4f}")
Epoch 1/5
1500/1500 - 1s - 723us/step - accuracy: 0.9082 - loss: 0.3249 - val_accuracy: 0.9482 - val_loss: 0.1819
Epoch 2/5
1500/1500 - 1s - 518us/step - accuracy: 0.9545 - loss: 0.1599 - val_accuracy: 0.9588 - val_loss: 0.1398
Epoch 3/5
1500/1500 - 1s - 561us/step - accuracy: 0.9662 - loss: 0.1178 - val_accuracy: 0.9641 - val_loss: 0.1220
Epoch 4/5
1500/1500 - 1s - 509us/step - accuracy: 0.9734 - loss: 0.0930 - val_accuracy: 0.9657 - val_loss: 0.1145
Epoch 5/5
1500/1500 - 1s - 508us/step - accuracy: 0.9784 - loss: 0.0754 - val_accuracy: 0.9673 - val_loss: 0.1099
No description has been provided for this image
313/313 - 0s - 454us/step - accuracy: 0.9698 - loss: 0.1014
Test accuracy: 0.9698

Recurrent neural networks¶

3. Build a model with a SimpleRNN(64) layer taking the (28, 28) images directly as input, followed by a Dense(10, activation="softmax") output layer. Compile the same way, train for 5 epochs, and plot the training history with plot_history().¶

In [8]:
rnn = keras.Sequential([
    keras.Input(shape=(28, 28)),
    layers.SimpleRNN(64),
    layers.Dense(10, activation="softmax")
])

rnn.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

history_rnn = rnn.fit(x_train, y_train, epochs=5, validation_split=0.2, verbose=2)
plot_history(history_rnn, "SimpleRNN")
loss, acc = rnn.evaluate(x_test, y_test, verbose=2)
print(f"Test accuracy: {acc:.4f}")
Epoch 1/5
1500/1500 - 3s - 2ms/step - accuracy: 0.8285 - loss: 0.5703 - val_accuracy: 0.9077 - val_loss: 0.3183
Epoch 2/5
1500/1500 - 2s - 2ms/step - accuracy: 0.9158 - loss: 0.2950 - val_accuracy: 0.9197 - val_loss: 0.2808
Epoch 3/5
1500/1500 - 3s - 2ms/step - accuracy: 0.9279 - loss: 0.2500 - val_accuracy: 0.9315 - val_loss: 0.2418
Epoch 4/5
1500/1500 - 2s - 1ms/step - accuracy: 0.9354 - loss: 0.2267 - val_accuracy: 0.9359 - val_loss: 0.2228
Epoch 5/5
1500/1500 - 2s - 1ms/step - accuracy: 0.9412 - loss: 0.2058 - val_accuracy: 0.9443 - val_loss: 0.1939
No description has been provided for this image
313/313 - 0s - 857us/step - accuracy: 0.9399 - loss: 0.2151
Test accuracy: 0.9399

A plain SimpleRNN struggles to carry information over long sequences. Its hidden state gets overwritten at every step. LSTM layers add gates that let the network learn what to keep and what to forget, which usually works better, especially over 28 timesteps.

4. Rebuild the same model, but replace SimpleRNN(64) with LSTM(64). Train it the same way, plot its training history, and compare its test accuracy to the SimpleRNN and feed-forward baselines.¶

In [9]:
lstm = keras.Sequential([
    keras.Input(shape=(28, 28)),
    layers.LSTM(64),
    layers.Dense(10, activation="softmax")
])

lstm.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

history_lstm = lstm.fit(x_train, y_train, epochs=5, validation_split=0.2, verbose=2)
plot_history(history_lstm, "LSTM")
loss, acc = lstm.evaluate(x_test, y_test, verbose=2)
print(f"Test accuracy: {acc:.4f}")

# LSTM typically improves over SimpleRNN here, though both usually
# trail the CNN from the previous practical. CNN's local filters
# are a better fit for images than reading rows in sequence.
Epoch 1/5
1500/1500 - 6s - 4ms/step - accuracy: 0.8441 - loss: 0.4896 - val_accuracy: 0.9456 - val_loss: 0.1809
Epoch 2/5
1500/1500 - 7s - 4ms/step - accuracy: 0.9526 - loss: 0.1571 - val_accuracy: 0.9668 - val_loss: 0.1090
Epoch 3/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9668 - loss: 0.1092 - val_accuracy: 0.9697 - val_loss: 0.0952
Epoch 4/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9738 - loss: 0.0859 - val_accuracy: 0.9767 - val_loss: 0.0781
Epoch 5/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9790 - loss: 0.0718 - val_accuracy: 0.9774 - val_loss: 0.0782
No description has been provided for this image
313/313 - 1s - 2ms/step - accuracy: 0.9781 - loss: 0.0772
Test accuracy: 0.9781

5. Row-scan vs column-scan: does the direction the RNN reads the image matter? Build the same LSTM(64) model as Q4, but feed it the transposed images (columns read top-to-bottom instead of rows) using np.transpose(x, axes=(0, 2, 1)). Train and evaluate it (plot its training history too), then compare its test accuracy to the row-scanning LSTM from Q4.¶

In [10]:
x_train_cols = np.transpose(x_train, axes=(0, 2, 1))
x_test_cols = np.transpose(x_test, axes=(0, 2, 1))

lstm_cols = keras.Sequential([
    keras.Input(shape=(28, 28)),
    layers.LSTM(64),
    layers.Dense(10, activation="softmax")
])

lstm_cols.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

history_lstm_cols = lstm_cols.fit(x_train_cols, y_train, epochs=5, validation_split=0.2, verbose=2)
plot_history(history_lstm_cols, "LSTM (column-scan)")
loss, acc = lstm_cols.evaluate(x_test_cols, y_test, verbose=2)
print(f"Test accuracy (column-scan): {acc:.4f}")
print(f"Test accuracy (row-scan, from Q4): see history above")

# Accuracy is usually very close between the two directions. MNIST
# digits don't have a strong row-vs-column asymmetry, so the LSTM
# picks up similar information either way. The gap that does exist
# is more a matter of training variance than orientation mattering
# in principle; on datasets with genuine row/column structure
# (e.g. left-to-right vs top-to-bottom text) the difference can be
# much larger.
Epoch 1/5
1500/1500 - 7s - 4ms/step - accuracy: 0.8676 - loss: 0.4393 - val_accuracy: 0.9446 - val_loss: 0.1892
Epoch 2/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9542 - loss: 0.1564 - val_accuracy: 0.9652 - val_loss: 0.1187
Epoch 3/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9667 - loss: 0.1099 - val_accuracy: 0.9705 - val_loss: 0.1001
Epoch 4/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9735 - loss: 0.0867 - val_accuracy: 0.9741 - val_loss: 0.0921
Epoch 5/5
1500/1500 - 6s - 4ms/step - accuracy: 0.9781 - loss: 0.0728 - val_accuracy: 0.9755 - val_loss: 0.0816
No description has been provided for this image
313/313 - 0s - 1ms/step - accuracy: 0.9773 - loss: 0.0748
Test accuracy (column-scan): 0.9773
Test accuracy (row-scan, from Q4): see history above

Attention¶

An LSTM with return_sequences=True outputs a hidden state for every row, not just the last one. Attention learns a weight for each row's hidden state, i.e., how much that row should contribute to the final prediction, and combines them into a single weighted vector.

6. Below is a small custom attention layer implementing additive (Bahdanau-style) attention. Read through it and make sure you understand what each part computes.¶

In [11]:
class AttentionLayer(layers.Layer):
    def __init__(self, units, **kwargs):
        super().__init__(**kwargs)
        self.units = units

    def build(self, input_shape):
        # input_shape: (batch, timesteps, hidden_dim)
        self.W = self.add_weight(
            shape=(input_shape[-1], self.units),
            initializer="glorot_uniform",
            trainable=True,
            name="attention_W"
        )
        self.v = self.add_weight(
            shape=(self.units, 1),
            initializer="glorot_uniform",
            trainable=True,
            name="attention_v"
        )

    def call(self, hidden_states):
        # hidden_states: (batch, timesteps, hidden_dim)
        score = tf.nn.tanh(tf.matmul(hidden_states, self.W))    # (batch, timesteps, units)
        score = tf.matmul(score, self.v)                         # (batch, timesteps, 1)
        weights = tf.nn.softmax(score, axis=1)                   # (batch, timesteps, 1)
        context = tf.reduce_sum(weights * hidden_states, axis=1)   # (batch, hidden_dim)
        return context, weights

7. Build a model using LSTM(64, return_sequences=True) followed by the AttentionLayer above, then a Dense(10, activation="softmax") output layer on the resulting context vector. Since this model has two outputs (context and weights) but we only train on the classification output, use the functional API rather than Sequential. Compile and train it the same way as the other models, and plot its training history.¶

In [12]:
inputs = keras.Input(shape=(28, 28))
lstm_out = layers.LSTM(64, return_sequences=True)(inputs)
context, attention_weights = AttentionLayer(32)(lstm_out)
outputs = layers.Dense(10, activation="softmax")(context)

attn_model = keras.Model(inputs=inputs, outputs=outputs)
attn_model.compile(loss="sparse_categorical_crossentropy", optimizer="adam", metrics=["accuracy"])

history_attn = attn_model.fit(x_train, y_train, epochs=5, validation_split=0.2, verbose=2)
plot_history(history_attn, "LSTM + Attention")
loss, acc = attn_model.evaluate(x_test, y_test, verbose=2)
print(f"Test accuracy: {acc:.4f}")
Epoch 1/5
1500/1500 - 8s - 5ms/step - accuracy: 0.8469 - loss: 0.4939 - val_accuracy: 0.9475 - val_loss: 0.1796
Epoch 2/5
1500/1500 - 7s - 4ms/step - accuracy: 0.9563 - loss: 0.1525 - val_accuracy: 0.9657 - val_loss: 0.1199
Epoch 3/5
1500/1500 - 7s - 4ms/step - accuracy: 0.9680 - loss: 0.1092 - val_accuracy: 0.9710 - val_loss: 0.1020
Epoch 4/5
1500/1500 - 7s - 4ms/step - accuracy: 0.9735 - loss: 0.0886 - val_accuracy: 0.9761 - val_loss: 0.0852
Epoch 5/5
1500/1500 - 7s - 4ms/step - accuracy: 0.9771 - loss: 0.0758 - val_accuracy: 0.9752 - val_loss: 0.0850
No description has been provided for this image
313/313 - 1s - 2ms/step - accuracy: 0.9774 - loss: 0.0755
Test accuracy: 0.9774

8. Extract the attention weights for a few test digits using a second model that outputs attention_weights instead of the final prediction. For 3 examples, plot the original digit next to a horizontal bar chart of its 28 row-attention weights, so the bars line up with the image rows. Which rows does the model attend to most?¶

In [13]:
attn_weights_model = keras.Model(inputs=inputs, outputs=attention_weights)

examples = [0, 1, 2]
weights = attn_weights_model.predict(x_test[examples], verbose=2)   # (3, 28, 1)

fig, axes = plt.subplots(3, 2, figsize=(6, 12))
for i, idx in enumerate(examples):
    axes[i, 0].imshow(x_test[idx], cmap="gray_r")
    axes[i, 0].set_title(f"Digit: {y_test[idx]}")
    axes[i, 0].axis("off")

    row_weights = weights[i].flatten()
    axes[i, 1].barh(range(28), row_weights)
    axes[i, 1].invert_yaxis()          # row 0 at the top, matching the image
    axes[i, 1].set_ylabel("row index")
    axes[i, 1].set_xlabel("attention weight")
    axes[i, 1].set_title("Attention per row")

plt.tight_layout()
plt.show()

# The model tends to attend most to the rows through the middle of
# the digit, where the strokes that distinguish e.g. a 3 from an 8
# or a 4 from a 9 are usually found - the top and bottom rows, often
# blank or a single simple stroke, get less weight.
1/1 - 0s - 66ms/step
No description has been provided for this image

9. Attention on a misclassified example: does attention look any different when the model gets it wrong? Find a test digit the attention model misclassifies (compare attn_model.predict() to y_test), and plot its row-attention weights next to a correctly classified example of the same true digit.¶

In [14]:
preds = attn_model.predict(x_test, verbose=2).argmax(axis=1)
wrong_idx = np.where(preds != y_test)[0]

# pick a misclassified example, then find a correctly classified
# example with the same true label for a fair comparison
example_wrong = wrong_idx[0]
true_label = y_test[example_wrong]
correct_candidates = np.where((preds == y_test) & (y_test == true_label))[0]
example_correct = correct_candidates[0]

pair = [example_correct, example_wrong]
weights_pair = attn_weights_model.predict(x_test[pair], verbose=2)  # (2, 28, 1)

fig, axes = plt.subplots(2, 2, figsize=(6, 8))
titles = [
    f"Correct: true={y_test[example_correct]}, pred={preds[example_correct]}",
    f"Wrong: true={y_test[example_wrong]}, pred={preds[example_wrong]}"
]
for i, idx in enumerate(pair):
    axes[i, 0].imshow(x_test[idx], cmap="gray_r")
    axes[i, 0].set_title(titles[i])
    axes[i, 0].axis("off")

    row_weights = weights_pair[i].flatten()
    axes[i, 1].barh(range(28), row_weights)
    axes[i, 1].invert_yaxis()
    axes[i, 1].set_ylabel("row index")
    axes[i, 1].set_xlabel("attention weight")

plt.tight_layout()
plt.show()

# On misclassified examples, attention often spreads more evenly
# across rows rather than concentrating on a few. The model isn't
# finding a clear, distinctive stroke pattern to lock onto, which
# lines up with it being genuinely unsure. This isn't a guaranteed
# pattern on every misclassified example, but it's common enough
# to be a useful diagnostic signal.
313/313 - 1s - 2ms/step
1/1 - 0s - 63ms/step
No description has been provided for this image

10. OPTIONAL (if time permits): Compare all four models (feed-forward, SimpleRNN, LSTM, and LSTM+Attention) on test accuracy in a single table.¶

In [15]:
results = {
    "Feed-forward": ffnn.evaluate(x_test, y_test, verbose=2)[1],
    "SimpleRNN": rnn.evaluate(x_test, y_test, verbose=2)[1],
    "LSTM": lstm.evaluate(x_test, y_test, verbose=2)[1],
    "LSTM + Attention": attn_model.evaluate(x_test, y_test, verbose=2)[1],
}

for name, acc in results.items():
    print(f"{name:20s}: {acc:.4f}")

# Compare these to the CNN's test accuracy from the previous
# practical. Despite reading the same pixels, row-by-row RNNs
# typically don't reach a CNN's accuracy on images, since they
# throw away the 2D spatial structure that convolution exploits.
# Attention narrows that gap a little and, more usefully, gives
# an interpretable view (Q8) that a plain CNN filter doesn't.
313/313 - 0s - 345us/step - accuracy: 0.9698 - loss: 0.1014
313/313 - 0s - 617us/step - accuracy: 0.9399 - loss: 0.2151
313/313 - 0s - 1ms/step - accuracy: 0.9781 - loss: 0.0772
313/313 - 0s - 1ms/step - accuracy: 0.9774 - loss: 0.0755
Feed-forward        : 0.9698
SimpleRNN           : 0.9399
LSTM                : 0.9781
LSTM + Attention    : 0.9774