NLP: Representation for Text Classification, Deep Learning & Word Embedding¶

27 October 2026¶

This lab session has three parts:

  • Part 1: Text classification with TFiDF features and a neural network classifier
  • Part 2: Word embedding
  • Part 3: Embedding layer for multiclass classification

The task in this lab session is multiclass text classification, in which we will work with the 20 Newsgroups dataset from the sklearn library and apply deep learning models using the keras library.

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. It was originally collected by Ken Lang, and it has become a popular data set for experiments in text applications of machine learning techniques.

We will use the keras library, which you already know and that is a deep learning / neural networks API by François Chollet's team capable of running on top of TensorFlow, JAX or PyTorch.

Setup: where to run the code¶

This page shows the lab questions and code. To run the code, open an empty notebook and copy the code cells from this page into it (one cell at a time, in order). You can work in Google Colab (nothing to install on your laptop) or locally.

Option A: Google Colab (recommended)

  1. Go to https://colab.research.google.com and sign in with a Google account.
  2. Click New notebook (or File → New notebook).
  3. TensorFlow/Keras, scikit-learn, pandas and matplotlib are already installed in Colab. Copy the install cell below into the first cell and run it once to add gensim (needed for Part 2).
  4. If Colab asks you to restart the session after installing, do so and then continue.
  5. A GPU is not needed for this lab (the default CPU runtime is fine).
  6. Files you download (e.g. the word embeddings) are stored in the Colab session and are deleted when the session ends. Save your notebook to your Google Drive (File → Save a copy in Drive).

Option B: on your own computer

You need Python 3.10–3.12. With Anaconda, open a terminal (Anaconda Prompt on Windows) and run:

conda create -n daml python=3.12
conda activate daml
pip install tensorflow keras scikit-learn gensim pandas matplotlib jupyter
jupyter notebook

Then create a new notebook (New → Python 3) and copy the code from this page into it. Without Anaconda, the same pip install ... line works inside a virtual environment.

This lab was tested with Keras 3.10, TensorFlow 2.20, scikit-learn 1.9 and gensim 4.4. Note that Part 2 downloads ~400 MB of pre-trained word embeddings.

In [1]:
# Copy this into the first cell of your notebook and run it once
# (in Colab only gensim is missing; already-installed packages are skipped)
%pip install -q tensorflow keras scikit-learn gensim pandas matplotlib
Note: you may need to restart the kernel to use updated packages.

Today we will use the following libraries. Take care to have them installed!

In [2]:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
from sklearn.preprocessing import LabelEncoder

from keras.utils import pad_sequences
from tensorflow.keras.preprocessing.text import Tokenizer

from keras.models import Sequential
from keras import layers, utils

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
In [3]:
import keras
print(keras.__version__)
3.10.0

Part 1: Text classification with TFiDF features and a neural network classifier¶

1. Load the train and test subsets of the 20 Newsgroups data set from sklearn datasets. Remove the headers, footers and quotes from the news article when loading data sets. Use number 321 for random_state. In order to get faster execution times for this practical we will work on a partial data set with only 5 categories out of the 20 available in the data set: 'rec.sport.hockey', 'talk.politics.mideast', 'soc.religion.christian', 'comp.graphics', and 'sci.med'.

In [4]:
categories = ['rec.sport.hockey', 'talk.politics.mideast', 'soc.religion.christian', 'comp.graphics', 'sci.med']
In [5]:
twenty_train = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'),
                                  categories=categories, shuffle=True, random_state=321)
# type(twenty_train)
In [6]:
twenty_test = fetch_20newsgroups(subset='test', remove=('headers', 'footers', 'quotes'),
                                 categories=categories, shuffle=True, random_state=321)

2. Find out about the number of news articles in the train and test sets, and convert them into dataframes.

In [7]:
twenty_train.target_names
Out[7]:
['comp.graphics',
 'rec.sport.hockey',
 'sci.med',
 'soc.religion.christian',
 'talk.politics.mideast']
In [8]:
twenty_train.filenames.shape
Out[8]:
(2941,)
In [9]:
twenty_test.filenames.shape
Out[9]:
(1958,)
In [10]:
df_train = pd.DataFrame(list(zip(twenty_train.data, twenty_train.target)), columns=['text', 'label'])
df_train.head()
Out[10]:
text label
0 \nDr. cheghadr bA namakand!  They just wait un... 4
1 \n\n\n\n\n:) No...I was one of the lucky ones.... 2
2 \n\n[After a small refresh Hasan got on the tr... 4
3 Before getting excited and implying that I am ... 4
4 I have posted disp135.zip to alt.binaries.pict... 0
In [11]:
df_test = pd.DataFrame(list(zip(twenty_test.data, twenty_test.target)), columns=['text', 'label'])
df_test.head()
Out[11]:
text label
0 hi all, Ive applied for the class of 93 at qui... 2
1 :In article <enea1-270493135255@enea.apple.com... 2
2 \nI don't know the answer the to this one, alt... 0
3 \n\nWe here at IBM have the same problem with ... 0
4 \nI was at an Adobe seminar/conference/propaga... 0

Train a neural network with a document-term matrix¶

3. In order to feed predictive deep learning models with text data, first you need to turn the text into vectors of numerical values suitable for statistical analysis. Use the binary representation with TfidfVectorizer and create document-term matrices for test and train (name them X_train and X_test).

In [12]:
# A function for transforming train or test into tfidf features
def tfidf_features(txt, flag):
    if flag == "train":
        x = tfidf.fit_transform(txt)
    else:
        x = tfidf.transform(txt)
    x = x.astype('float32')
    return x

tfidf = TfidfVectorizer(binary=True)
X_train = tfidf_features(df_train.text.values, flag="train")
X_test = tfidf_features(df_test.text.values, flag="test")

# Here is the solution for the case when we wanted to also count and not use the function above :-)
# from sklearn.feature_extraction.text import CountVectorizer
# count_vect = CountVectorizer()
# X_train = count_vect.fit_transform(df_train.text.values)
# X_test = count_vect.transform(df_test.text.values)
In [13]:
X_train.nnz / float(X_train.shape[0])
Out[13]:
111.5678340700442

The extracted vectors are very sparse, with an average of 111 non-zero components by sample in a more than 37000-dimensional space (less than 0.3% non-zero features)

In [14]:
X_test.nnz / float(X_test.shape[0])
Out[14]:
113.83605720122574
In [15]:
# tfidf.vocabulary_

4. Use the LabelEncoder to create y_train and y_test from df_train.label.values and df_test.label.values, respectively.

In [16]:
# Converting the list of strings to the matrix of vectors (to be fed neural network models)
# Encode the list of newsgroups into categorical integer values
lb = LabelEncoder()
y = lb.fit_transform(df_train.label.values)
y_train = utils.to_categorical(y)
In [17]:
y_train
Out[17]:
array([[0., 0., 0., 0., 1.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 0., 1.],
       ...,
       [1., 0., 0., 0., 0.],
       [0., 0., 0., 0., 1.],
       [0., 0., 0., 0., 1.]], shape=(2941, 5))
In [18]:
y_train.shape
Out[18]:
(2941, 5)
In [19]:
y = lb.transform(df_test.label.values)
y_test = utils.to_categorical(y)

5. Use the sequential API in keras and create a one-hidden-layer neural network. So, the first layer will be the input layer with the number of features in your X_train, followed by a single hidden layer, and an output layer. Set the number of neurons in the hidden layer to 10, and activation function as relu. For the output layer you can use a softmax activation function.

The sequential API (https://www.tensorflow.org/guide/keras/sequential_model) allows you to create models layer by layer. It is limited in that it does not allow to create models that share layers or have multiple inputs or outputs.

The functional API (https://www.tensorflow.org/guide/keras/functional) allows you to create models that have a lot more flexibility as you can define models where layers connect to more than just the previous and next layers. In this way, you can connect layers to (literally) any other layer. As a result, creating complex networks such as Siamese neural networks and residual neural networks become possible.

In [20]:
model = Sequential()
input_dim = X_train.shape[1]  # Number of features
model.add(layers.Input(shape=(input_dim,)))
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(5, activation='softmax'))

6. The compile function defines the loss function, the optimizer and the evaluation metrics. Call this function for your neural network model with loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']. Check the summary of the model.

Task Output type Last-layer activation Loss function Metric(s)
Regression Numerical Linear meanSquaredError (MSE),
meanAbsoluteError (MAE)
Same as loss
Classification Binary Sigmoid binary_crossentropy Accuracy, precision, recall, sensitivity,
TPR, FPR, ROC, AUC
Classification Single label, Multiple classes Softmax categorical_crossentropy Accuracy, confusion matrix
Classification Multiple labels, Multiple classes Sigmoid binary_crossentropy Accuracy, precision, recall, sensitivity,
TPR, FPR, ROC, AUC
In [21]:
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 10)             │       371,490 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 5)              │            55 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 371,545 (1.42 MB)
 Trainable params: 371,545 (1.42 MB)
 Non-trainable params: 0 (0.00 B)

7. Time to train your model! Train your model for 20 epochs. What does batch_size represent?

In [22]:
history = model.fit(X_train, y_train, epochs=20, batch_size=512)
# model.save_weights("model.h5")
# print("Saved model to disk")
Epoch 1/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 2s 483ms/step - accuracy: 0.2363 - loss: 1.6092

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.3635 - loss: 1.6050 

6/6 ━━━━━━━━━━━━━━━━━━━━ 1s 16ms/step - accuracy: 0.3796 - loss: 1.6044
Epoch 2/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 33ms/step - accuracy: 0.8184 - loss: 1.5756

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.8175 - loss: 1.5693

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step - accuracy: 0.8185 - loss: 1.5666
Epoch 3/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 32ms/step - accuracy: 0.8672 - loss: 1.5339

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step - accuracy: 0.8721 - loss: 1.5243

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step - accuracy: 0.8722 - loss: 1.5232
Epoch 4/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step - accuracy: 0.8848 - loss: 1.4867

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.8954 - loss: 1.4819

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step - accuracy: 0.8978 - loss: 1.4795
Epoch 5/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 34ms/step - accuracy: 0.9199 - loss: 1.4461

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9151 - loss: 1.4395

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step - accuracy: 0.9157 - loss: 1.4367
Epoch 6/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step - accuracy: 0.9375 - loss: 1.3951

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9267 - loss: 1.3922

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9265 - loss: 1.3913
Epoch 7/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 27ms/step - accuracy: 0.9375 - loss: 1.3597

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.9341 - loss: 1.3503

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9337 - loss: 1.3490
Epoch 8/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9355 - loss: 1.3068

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9378 - loss: 1.3024

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9375 - loss: 1.3018
Epoch 9/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step - accuracy: 0.9492 - loss: 1.2614

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9418 - loss: 1.2592

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step - accuracy: 0.9416 - loss: 1.2577
Epoch 10/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step - accuracy: 0.9316 - loss: 1.2334

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9382 - loss: 1.2202

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9391 - loss: 1.2188
Epoch 11/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9473 - loss: 1.1819

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9496 - loss: 1.1759

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9498 - loss: 1.1747
Epoch 12/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9512 - loss: 1.1514

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step - accuracy: 0.9520 - loss: 1.1385

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step - accuracy: 0.9524 - loss: 1.1346
Epoch 13/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step - accuracy: 0.9648 - loss: 1.0966

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9579 - loss: 1.0911

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9577 - loss: 1.0900
Epoch 14/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9551 - loss: 1.0586

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9579 - loss: 1.0505

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9582 - loss: 1.0493
Epoch 15/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9609 - loss: 1.0198

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9611 - loss: 1.0109

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9616 - loss: 1.0096
Epoch 16/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 27ms/step - accuracy: 0.9551 - loss: 0.9712

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.9625 - loss: 0.9710

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9629 - loss: 0.9699
Epoch 17/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9727 - loss: 0.9405

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9697 - loss: 0.9308

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9695 - loss: 0.9300
Epoch 18/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9746 - loss: 0.9058

5/6 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step - accuracy: 0.9707 - loss: 0.8954

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step - accuracy: 0.9702 - loss: 0.8934
Epoch 19/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step - accuracy: 0.9707 - loss: 0.8777

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9701 - loss: 0.8636

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9701 - loss: 0.8621
Epoch 20/20
1/6 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9746 - loss: 0.8209

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - accuracy: 0.9732 - loss: 0.8189

6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step - accuracy: 0.9728 - loss: 0.8188

Note that if you rerun the fit() method, you will start off with the computed weights from the previous training. Make sure to call clear_session() before you start training the model again:


from keras.backend import clear_session
clear_session()


8. Plot the accuracy and loss of your trained model.

In [23]:
print(history.history.keys())
plt.plot(history.history['accuracy'])
#plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()

# summarize history for loss
plt.plot(history.history['loss'])
# plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()
dict_keys(['accuracy', 'loss'])
No description has been provided for this image
No description has been provided for this image
In [24]:
# Here we converted the code to a function so we can use it later as well
plt.style.use('ggplot')

def plot_history(history, val=0):
    acc = history.history['accuracy']
    if val == 1:
        val_acc = history.history['val_accuracy'] # we can add a validation set in our fit function with nn
    loss = history.history['loss']
    if val == 1:
        val_loss = history.history['val_loss']
    x = range(1, len(acc) + 1)

    plt.figure(figsize=(12, 5))
    plt.subplot(1, 2, 1)
    plt.plot(x, acc, 'b', label='Training accuracy')
    if val == 1:
        plt.plot(x, val_acc, 'r', label='Validation accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.title('Accuracy')
    plt.legend()
    plt.subplot(1, 2, 2)
    plt.plot(x, loss, 'b', label='Training loss')
    if val == 1:
        plt.plot(x, val_loss, 'r', label='Validation loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.title('Loss')
    plt.legend()
In [25]:
plot_history(history)
No description has been provided for this image

9. Check the accuracy of your trained model on the test set. Compare them with the accuracy on the training set.

In [26]:
loss, accuracy = model.evaluate(X_test,y_test)
print('Test set\n  Loss: {:0.3f}\n  Accuracy: {:0.3f}'.format(loss,accuracy))
 1/62 ━━━━━━━━━━━━━━━━━━━━ 5s 93ms/step - accuracy: 0.9375 - loss: 1.0106

62/62 ━━━━━━━━━━━━━━━━━━━━ 0s 791us/step - accuracy: 0.8750 - loss: 1.0061
Test set
  Loss: 1.012
  Accuracy: 0.871

You can already see that the model is overfitting: the accuracy on the training set (last epoch) is clearly higher than the accuracy on the test set. When training a model, you can use a separate testing and validation set. What you would usually do is take the model with the highest validation accuracy and then test the model with the testing set.

Part 2: Word embedding¶

For part 2, let's first make sure that we have the following libraries installed.

In [27]:
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

from sklearn.decomposition import PCA
from numpy import linalg as LA

We are going to use the gensim library. This library offers a variety of methods to read in pre-trained word embeddings as well as train your own. The website contains a lot of documentation, for example here: https://radimrehurek.com/gensim/auto_examples/index.html#documentation

If gensim isn't installed yet, you can use the following command:

In [28]:
#%pip install gensim
#%pip install --upgrade gensim
In [29]:
from gensim.test.utils import datapath

Reading in a pre-trained model¶

10. First we load in a pre-trained GloVe model. Note: this can take around five minutes. See https://github.com/RaRe-Technologies/gensim-data for an overview of the models you can try. For example word2vec-google-news-300: word2vec trained on Google news. 1662 MB. glove-twitter-200: trained on Twitter: 758 MB. We are going to start with glove-wiki-gigaword-300 (376.1MB). These embeddings are trained on Wikipedia (2014) and the Gigaword corpus, a large collection of newswire text.

In [30]:
import gensim.downloader as api
wv = api.load('glove-wiki-gigaword-300')

11. How many words does the vocabulary contain? Is 'utrecht' in the vocabulary? If so, print its word embedding.

In [31]:
len(wv)
Out[31]:
400000
In [32]:
'utrecht' in wv
Out[32]:
True
In [33]:
wv["utrecht"]
Out[33]:
array([ 1.7246e-01, -9.2534e-02,  1.6428e-01,  5.6065e-01,  2.8004e-01,
       -1.5202e-02,  2.7825e-01, -6.5038e-01,  6.6932e-03,  1.0561e-01,
        7.0731e-01,  3.0788e-02,  1.6741e-02, -5.1297e-02,  3.9779e-02,
        5.6926e-01, -2.2903e-01,  3.7331e-01,  3.4440e-01,  2.1143e-01,
        1.9787e-01, -2.5727e-01,  2.0160e-01, -2.5648e-02, -4.7996e-01,
       -1.4263e-02, -5.5923e-01,  1.0823e-01, -9.0213e-01,  1.9254e-01,
        4.6962e-01, -4.2952e-01,  8.1015e-01,  1.1222e+00,  2.8237e-01,
       -6.0666e-01,  2.8948e-01,  3.1003e-01, -4.3745e-01, -1.0235e-01,
        3.1119e-01, -3.2637e-02, -5.5157e-01, -5.0208e-01,  4.2994e-01,
        3.2319e-01,  4.6997e-01,  7.5685e-02, -3.1806e-02,  3.5623e-01,
        5.9764e-02,  2.4482e-01,  1.0263e-03,  4.6040e-01,  1.5376e-01,
       -2.7841e-01,  2.6469e-02, -1.0785e-01, -4.5553e-01,  2.3964e-01,
       -4.4707e-01,  1.1521e-01,  2.1919e-01,  4.9167e-01,  3.4367e-01,
       -1.2389e-01,  3.1069e-01,  3.7095e-01,  3.8991e-04, -4.6645e-01,
        8.0286e-02,  1.8450e-01,  1.7059e-01, -4.2067e-01, -4.6711e-01,
        6.6363e-01, -1.6728e-01, -1.2503e-01,  1.9912e-01, -8.2380e-02,
       -1.6341e-01, -3.6798e-01, -1.2205e-01, -5.8729e-02, -6.8966e-01,
       -6.4324e-01,  3.2243e-01,  1.3532e-01, -6.3114e-01,  3.8847e-01,
        1.9127e-02, -9.1993e-01, -5.7550e-02, -4.5361e-01,  3.2246e-01,
       -2.3290e-01,  3.6797e-01, -5.0768e-01, -6.1523e-01, -2.7449e-02,
        9.9997e-02, -8.7165e-01,  7.3179e-01,  2.0256e-01, -9.5692e-02,
       -1.4106e-02,  3.4639e-01,  6.0384e-01, -2.1368e-01,  4.5634e-01,
        3.0276e-01, -2.1606e-01, -5.1111e-02, -8.0779e-02,  5.8191e-02,
       -3.4681e-01,  1.4722e-01, -9.8759e-02, -2.3481e-01, -8.1633e-02,
       -5.1594e-01, -2.9282e-01,  1.0665e-01, -3.3598e-02,  3.9112e-02,
        6.1789e-02, -2.7619e-03,  1.8009e-01,  1.1321e-01,  1.2848e-01,
        5.7384e-01,  2.0626e-01, -1.1215e-02,  5.8909e-01,  8.3938e-01,
       -2.5339e-01,  4.7566e-01, -5.3437e-01,  7.6082e-02,  1.8165e-01,
        5.7592e-01,  1.6382e-01, -2.9846e-02, -2.5505e-01,  2.9101e-01,
       -2.4089e-01,  1.1434e-01,  3.5274e-01, -4.2624e-01,  2.4286e-02,
        4.4413e-02, -4.6485e-02,  1.9847e-01,  7.0653e-01, -1.7389e-02,
        9.8610e-01, -2.3929e-01,  8.0001e-01, -3.6342e-01,  1.1249e-01,
       -3.0034e-01,  9.2404e-02, -2.5635e-01, -1.9328e-01, -1.2457e-01,
        2.5057e-01, -4.5034e-01, -1.0178e-01,  2.4003e-01, -2.0324e-01,
        3.9938e-02, -1.2958e-01, -3.9431e-01, -2.4845e-01, -4.1129e-03,
       -3.4170e-03,  3.8176e-01,  2.1570e-01, -7.4349e-01, -3.9406e-03,
       -2.1713e-01, -3.9769e-01, -2.8074e-01, -1.7159e-01,  8.7543e-01,
       -4.8529e-01, -7.8966e-01, -1.9181e-01,  1.2654e-02,  2.7394e-01,
        4.7824e-01, -1.3372e-01,  6.9161e-01, -2.8090e-01,  1.7082e-01,
       -6.2932e-01, -7.0490e-01, -1.6360e-01,  3.1653e-01,  3.1181e-01,
       -2.5984e-01, -7.1512e-02, -3.5759e-01, -1.2873e-01, -1.0136e-01,
       -8.4309e-01,  1.2680e-01,  2.1268e-01, -2.8869e-01,  9.0773e-02,
       -2.5738e-01, -1.8303e-01, -1.2904e-01,  4.4565e-01, -5.8736e-01,
        7.2059e-01, -3.3435e-01,  1.5862e-01, -9.1833e-02, -5.7568e-01,
        3.3082e-01, -1.8949e-01, -4.1058e-01,  2.8824e-01, -3.7240e-01,
       -3.2524e-01, -5.9380e-01, -1.9471e-01, -2.3064e-01,  5.9878e-02,
       -4.2104e-03,  1.2105e-01,  4.9247e-01, -1.9579e-01, -1.0380e-01,
       -2.4607e-01,  7.2958e-01,  1.6426e-02, -8.1131e-02, -3.9744e-02,
       -2.0024e-01, -1.0811e-01,  5.1185e-01,  1.2625e-01, -1.0223e-01,
        4.4235e-01, -5.3241e-01, -8.0116e-02,  5.3696e-01, -6.7007e-01,
        1.0677e-01, -6.4972e-02,  2.8323e-01, -1.1779e-01, -2.0966e-01,
        1.6076e-01,  7.3603e-01, -4.5484e-01, -4.4733e-01, -1.2544e-02,
        3.2596e-01,  2.9868e-01,  7.2416e-01,  8.8062e-01,  6.8829e-01,
        2.2420e-01, -5.9788e-01, -3.2725e-01,  2.0964e-01, -5.4107e-01,
       -4.6910e-01, -7.1801e-01, -1.5257e-01,  1.1137e-01,  4.0723e-01,
        3.6233e-01,  3.3883e-01, -6.8084e-01, -1.4319e-01,  8.5794e-01,
       -4.4418e-01,  5.6938e-01, -1.9381e-01, -8.5069e-01, -2.1100e-01,
        1.1303e-01, -3.8039e-01,  6.2520e-01, -4.8425e-02,  4.5813e-02,
       -3.5546e-01,  1.8457e-01,  2.1444e-02, -2.9848e-01,  4.2873e-01,
       -1.7749e-01,  5.0280e-01, -1.7376e-01,  4.2369e-01, -1.5048e-03],
      dtype=float32)
In [34]:
wv["utrecht"].shape
Out[34]:
(300,)

Question: Explore the embeddings for a few other words. Can you find words that are not in the vocabulary?

(For example, think of uncommon words, misspellings, new words, etc.)

12. We can calculate the cosine similarity between two words to find their similarity. First find the similarity between 'student' and 'university' in the pre-trained embedding. Then, print the top 5 most similar words to 'car'.

In [35]:
wv.similarity('university', 'student')
Out[35]:
np.float32(0.5970513)

Optional: cosine similarity is the same as the dot product between the normalized word embeddings

In [36]:
wv_university_norm = wv['university']/ LA.norm(wv['university'], 2)
wv_student_norm = wv['student'] / LA.norm(wv['student'], 2)

wv_university_norm.dot(wv_student_norm)
Out[36]:
np.float32(0.59705126)
In [37]:
print(wv.most_similar(positive=['car'], topn=5))
[('cars', 0.7827163338661194), ('vehicle', 0.7655367851257324), ('truck', 0.7350621223449707), ('driver', 0.7114784717559814), ('driving', 0.6442224383354187)]

Optional: We can calculate the cosine similarity between a list of word pairs and correlate these with human ratings. One such dataset with human ratings is called WordSim353.

Goto https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/test/test_data/wordsim353.tsv to get a sense of the data.

Gensim already implements a method to evaluate a word embedding model using this data.

  • It calculates the cosine similarity between each word pair
  • It calculates both the Spearman and Pearson correlation coefficient between the cosine similarities and human judgements

See https://radimrehurek.com/gensim/models/keyedvectors.html for a description of the methods.

In [38]:
wv.evaluate_word_pairs(datapath('wordsim353.tsv'))
Out[38]:
(PearsonRResult(statistic=np.float64(0.6040760928273887), pvalue=np.float64(1.752304156284728e-36)),
 SignificanceResult(statistic=np.float64(0.6085349998820805), pvalue=np.float64(3.879629536780591e-37)),
 0.0)

13. Use the pre-trained vectors and apply analogies such as 'Man is to woman as king is to. ..?'

This can be converted into vector arithmetic:

king - ? = man - woman.

king - man + woman = ?
In [39]:
wv.most_similar(negative=['man'], positive=['king', 'woman'])
Out[39]:
[('queen', 0.6713277101516724),
 ('princess', 0.5432624220848083),
 ('throne', 0.5386105179786682),
 ('monarch', 0.5347574949264526),
 ('daughter', 0.4980250597000122),
 ('mother', 0.49564430117607117),
 ('elizabeth', 0.483265221118927),
 ('kingdom', 0.47747084498405457),
 ('prince', 0.4668239951133728),
 ('wife', 0.46473270654678345)]

france - paris + amsterdam = ?

In [40]:
wv.most_similar(negative=['paris'], positive=['france', 'amsterdam'])
Out[40]:
[('netherlands', 0.7304362654685974),
 ('dutch', 0.5829050540924072),
 ('belgium', 0.5607962012290955),
 ('holland', 0.5492808222770691),
 ('denmark', 0.5330449938774109),
 ('sweden', 0.4875030517578125),
 ('germany', 0.47103551030158997),
 ('utrecht', 0.46798408031463623),
 ('spain', 0.46100154519081116),
 ('rotterdam', 0.45599010586738586)]

Note that if we would just retrieve the most similar words to 'amsterdam' we would receive a different result.

In [41]:
print(wv.most_similar(positive=['amsterdam'], topn=5))
[('rotterdam', 0.6485880613327026), ('schiphol', 0.5740087032318115), ('utrecht', 0.5608800053596497), ('netherlands', 0.5472348928451538), ('frankfurt', 0.5457332134246826)]

cat is to cats as girl is to ?

girl - ? = cat - cats
girl - cat + cats = ?
In [42]:
wv.most_similar(negative=['cat'], positive=['cats', 'girl'])
Out[42]:
[('girls', 0.6908110976219177),
 ('boys', 0.6055022478103638),
 ('boy', 0.5850904583930969),
 ('teenage', 0.5735769867897034),
 ('teenagers', 0.5600976943969727),
 ('teenager', 0.553008496761322),
 ('teen', 0.5423318147659302),
 ('children', 0.5312656164169312),
 ('woman', 0.5233089327812195),
 ('babies', 0.516354501247406)]

Compare against a baseline. What if we would just have retrieved the most similar words to 'girl'?

In [43]:
print(wv.most_similar(positive=['girl'], topn=5))
[('boy', 0.8272891044616699), ('woman', 0.729641854763031), ('girls', 0.7227292060852051), ('teenager', 0.650977373123169), ('teenage', 0.6492719054222107)]

14. We can't visualize embeddings in their raw format, because of their high dimensionality. However, we can use dimensionality reduction techniques such as PCA to project them onto a 2D space. Use the function below on a selected list of words and visualize them.

In [44]:
def display_scatterplot(wv, words=None, sample=0):

    # first get the word vectors
    word_vectors = np.array([wv[w] for w in words])

    # transform the data using PCA
    wv_PCA = PCA().fit_transform(word_vectors)[:,:2]

    plt.figure(figsize=(10,10))

    plt.scatter(wv_PCA[:,0], wv_PCA[:,1],
                edgecolors='k', c='r')

    for word, (x,y) in zip(words, wv_PCA):
        plt.text(x+0.05, y+0.05, word)
In [45]:
display_scatterplot(wv,
                        ['dog', 'cat', 'dogs', 'cats', 'horse', 'tiger',
                         'university', 'lesson', 'student', 'students',
                         'netherlands', 'amsterdam', 'utrecht', 'belgium', 'spain', 'china',
                         'coffee', 'tea', 'pizza', 'sushi', 'sandwich',
                         'car', 'train', 'bike', 'bicycle', 'trains'])
No description has been provided for this image
In [46]:
display_scatterplot(wv,
                        ['he', 'she', 'sister',
                         'brother', 'man', 'woman',
                         'nurse', 'doctor',
                         'grandfather', 'grandmother',
                         'math', 'arts',
                         'daughter', 'son'])
No description has been provided for this image

Part 3: Embedding layer for multiclass classification¶

The embedding layer¶

Here we want to create a sequential model with an embedding layer as the input layer followed by dense layers. To do this first we need to apply the Tokenizer from keras and convert the text data into sequences which can be passed into the embedding layer.

15. Use the tokenizer from Keras with 20,000 words and create X_train and X_test sequences.

In [47]:
tokenizer = Tokenizer(num_words=20000)
tokenizer.fit_on_texts(df_train.text.values)
In [48]:
X_train = tokenizer.texts_to_sequences(df_train.text.values)
X_test = tokenizer.texts_to_sequences(df_test.text.values)
In [49]:
vocab_size = len(tokenizer.word_index) + 1  # Adding 1 because of reserved 0 index for sequence padding
vocab_size
Out[49]:
38111

Note that a document-term matrix uses vectors of word counts, and each vector has the same length (the size of the total corpus vocabulary). With keras tokenizer, the resulting vectors equal the length of each text, and the numbers don’t denote counts, but rather correspond to the word values from the dictionary tokenizer.word_index.

In [50]:
for word in ['the', 'all', 'happy', 'sad']:
    print('{}: {}'.format(word, tokenizer.word_index[word]))
the: 1
all: 35
happy: 1043
sad: 3422

16. Use the pad_sequences() function to pad each text sequence with zeros, so that each vector has the same length of 100 words.

In [51]:
maxlen = 100
X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)
In [52]:
print(X_train[0, :])
[  555 12221    23    75  1507   379    23    16 15253     3   621    63
  5371 10293    73  2745     5   246   686     2 10294     2  2340     4
  3927     6   332  4110  1290     3 12222   131    37     2     1    72
  7235    34   177    21     7  3928     4    33    16    35    86 12223
  3226     4    23   241   753     3 12222     3    47    38  7953    42
     4    93    17   364     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0]

Typically it does not matter whether you prepend or append zeros. The first values represent the index in the vocabulary, and the rest are zeros from sequence padding, since you have a short document.

17. Now it is time to create a neural network model using an embedding layer as input. Take the output of the embedding layer (embedding_dim = 50) and plug it into a Dense layer with 10 neurons, and the relu activation function. In order to do this, you have to add a Flatten layer in between that prepares the sequential input for the Dense layer. Note that in the Embedding layer, input_dim is the size of the vocabulary and output_dim is the size of the embedding vector. For the output layer, use 5 neurons with a softmax activation and compile with categorical_crossentropy. (In Keras 3 you no longer pass input_length to the Embedding layer; call model.build(input_shape=(None, maxlen)) before model.summary() to see the number of parameters.)

In [53]:
embedding_dim = 50

model = Sequential()
model.add(layers.Embedding(input_dim=vocab_size,
                           output_dim=embedding_dim))
model.add(layers.Flatten())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(5, activation='softmax'))
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.build(input_shape=(None, maxlen))
model.summary()
Model: "sequential_1"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ embedding (Embedding)           │ (None, 100, 50)        │     1,905,550 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 5000)           │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 10)             │        50,010 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 5)              │            55 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 1,955,615 (7.46 MB)
 Trainable params: 1,955,615 (7.46 MB)
 Non-trainable params: 0 (0.00 B)

You can now see that we have 1,905,550 new parameters to train. This number comes from vocab_size (38,111) times the embedding_dim (50). Note that vocab_size is based on tokenizer.word_index, which contains all words in the training texts; num_words=20000 only limits which words are kept by texts_to_sequences(), so in practice only the first 20,000 rows of the embedding matrix are used. These weights of the embedding layer are randomly initialized and then are adjusted through backpropagation during training. This model takes the words as they come in the order of the sentences as input vectors.

In [54]:
# Note: for simplicity we use the test set as validation data here;
# in practice you would use a separate validation set (e.g. validation_split=0.2)
history = model.fit(X_train, y_train,
                    epochs=10,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=64)
In [55]:
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
Training Accuracy: 0.9772
Testing Accuracy:  0.7109
In [56]:
plot_history(history, val=1)
No description has been provided for this image

(OPTIONAL) Pretrained word embeddings¶

18. **Pretrained word embeddings are the embeddings learned in one task that are used for solving another similar task. These embeddings are trained on large data sets, saved, and then used for solving other tasks. Here, we are going to use the GloVe embeddings which are precomputed word embeddings simply trained on a large corpus of text. For this purpose, we wrote the following function to apply on the pretrained word embeddings and use the corresponding word vectors for words in our vocabulary. Download one of the GloVe embeddings (e.g. glove.6B.50d.txt) and create the embedding matrix using the provided function. (Link to download: https://nlp.stanford.edu/projects/glove/)**

Tip: instead of downloading and unzipping the 822 MB file from Stanford, you can create the same glove.6B.50d.txt file from the 50-dimensional GloVe vectors in gensim (66 MB download). This also works in Colab (copy the cell below into your notebook):

In [57]:
import os
import gensim.downloader as api

if not os.path.exists('glove.6B.50d.txt'):
    glove50 = api.load('glove-wiki-gigaword-50')
    with open('glove.6B.50d.txt', 'w', encoding='utf8') as f:
        for word in glove50.index_to_key:
            f.write(word + ' ' + ' '.join(f'{x:.5f}' for x in glove50[word]) + '\n')
In [58]:
def create_embedding_matrix(filepath, word_index, embedding_dim):
    vocab_size = len(word_index) + 1  # Adding again 1 because of reserved 0 index
    embedding_matrix = np.zeros((vocab_size, embedding_dim))

    with open(filepath, encoding="utf8") as f:
        for line in f:
            word, *vector = line.split()
            if word in word_index:
                idx = word_index[word]
                embedding_matrix[idx] = np.array(
                    vector, dtype=np.float32)[:embedding_dim]

    return embedding_matrix
In [59]:
embedding_matrix = create_embedding_matrix('glove.6B.50d.txt',
                                           tokenizer.word_index, embedding_dim = 50)
In [60]:
nonzero_elements = np.count_nonzero(np.count_nonzero(embedding_matrix, axis=1))
nonzero_elements / vocab_size
Out[60]:
np.float64(0.7771771929364225)

19. Build your previous neural network model again, but this time with the initial weights from the pretrained word embeddings, and replace the Flatten layer with a GlobalMaxPool1D layer (which keeps, for each embedding dimension, the maximum value over all words in the text). Set the trainable argument False so that your embedding layer does not learn the word vectors anymore, and then again back to True. How does the performance change?

In [61]:
model = Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim,
                           weights=[embedding_matrix],
                           trainable=False))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(5, activation='softmax'))
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.build(input_shape=(None, maxlen))
model.summary()
Model: "sequential_2"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ embedding_1 (Embedding)         │ (None, 100, 50)        │     1,905,550 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ global_max_pooling1d            │ (None, 50)             │             0 │
│ (GlobalMaxPooling1D)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 10)             │           510 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_5 (Dense)                 │ (None, 5)              │            55 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 1,906,115 (7.27 MB)
 Trainable params: 565 (2.21 KB)
 Non-trainable params: 1,905,550 (7.27 MB)
In [62]:
# Note: for simplicity we use the test set as validation data here;
# in practice you would use a separate validation set (e.g. validation_split=0.2)
history = model.fit(X_train, y_train,
                    epochs=20,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history, val=1)
Training Accuracy: 0.7827
Testing Accuracy:  0.7569
No description has been provided for this image

With frozen embeddings the model has far fewer trainable parameters, so it overfits much less: the gap between training and test accuracy is small, and the test accuracy is similar to (or even higher than) that of the model in question 17. Let’s now see how this performs if we allow the embedding to be fine-tuned by using trainable=True:

In [63]:
model = Sequential()
model.add(layers.Embedding(vocab_size, embedding_dim,
                           weights=[embedding_matrix],
                           trainable=True))
model.add(layers.GlobalMaxPool1D())
model.add(layers.Dense(10, activation='relu'))
model.add(layers.Dense(5, activation='softmax'))
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.build(input_shape=(None, maxlen))
model.summary()
Model: "sequential_3"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ embedding_2 (Embedding)         │ (None, 100, 50)        │     1,905,550 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ global_max_pooling1d_1          │ (None, 50)             │             0 │
│ (GlobalMaxPooling1D)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_6 (Dense)                 │ (None, 10)             │           510 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_7 (Dense)                 │ (None, 5)              │            55 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 1,906,115 (7.27 MB)
 Trainable params: 1,906,115 (7.27 MB)
 Non-trainable params: 0 (0.00 B)

Because it is a multiclass classification problem, log loss is used as the loss function (categorical_crossentropy in keras). The efficient ADAM optimization algorithm is used. The model is fit for 20 epochs with a small batch size of 10 documents, which means many weight updates per epoch.

In [64]:
# Note: for simplicity we use the test set as validation data here;
# in practice you would use a separate validation set (e.g. validation_split=0.2)
history = model.fit(X_train, y_train,
                    epochs=20,
                    verbose=False,
                    validation_data=(X_test, y_test),
                    batch_size=10)
loss, accuracy = model.evaluate(X_train, y_train, verbose=False)
print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=False)
print("Testing Accuracy:  {:.4f}".format(accuracy))
plot_history(history, val=1)
Training Accuracy: 0.9803
Testing Accuracy:  0.8453
No description has been provided for this image