Deep learning for text classification with Keras

The IMDB dataset

In this example, we'll work with the IMDB dataset: a set of 50,000 highly polarized reviews from the Internet Movie Database. They are divided into 25,000 reviews for training and 25,000 reviews for testing, each set consisting of 50% negative and 50% positive reviews.

Why use separate training and test sets? Because you should never test a machine learning model on the same data you trained it on! Just because a model performs well on its training data does not mean it will perform well on data it has never seen. And what you care about is the performance of your model on new data (because you already know the labels of your training data – obviously you don't need your model to predict them Is). For example, it's possible that your model simply wears out. to memorize A mapping between your training samples and their targets, which the model will be useless for the task of predicting targets from data it has never seen before. We will go into this point in more detail in the next chapter.

Like the MNIST dataset, the IMDB dataset is packaged with Keras. This has already been implemented: the review (arrangement of words) has been converted into a sequence of numbers, where each number stands for a specific word in the dictionary.

The following code will load the dataset (about 80 MB of data will be downloaded to your machine when you run it for the first time).

library(keras)
imdb <- dataset_imdb(num_words = 10000)
train_data <- imdb$train$x
train_labels <- imdb$train$y
test_data <- imdb$test$x
test_labels <- imdb$test$y

Argument num_words = 10000 This means you will only keep the top 10,000 most frequent words in the training data. Rare words will be discarded. This allows you to work with vector data of a manageable size.

Variables train_data And test_data Lists of reviews are; Each review is a list of index words (encoding word order). train_labels And test_labels are lists of 0s and 1s, where 0 means negative and means 1. positive:

int [1:218] 1 14 22 16 43 530 973 1622 1385 65 ...

[1] 1

Since you are limiting yourself to the top 10,000 most frequent words, no word index will exceed 10,000:

[1] 9999

For kicks, here's how you can quickly decode one of these reviews into English words:

# Named list mapping words to an integer index.
word_index <- dataset_imdb_word_index()  
reverse_word_index <- names(word_index)
names(reverse_word_index) <- word_index

# Decodes the review. Note that the indices are offset by 3 because 0, 1, and 
# 2 are reserved indices for "padding," "start of sequence," and "unknown."
decoded_review <- sapply(train_data[[1]], function(index) {
  word <- if (index >= 3) reverse_word_index[[as.character(index - 3)]]
  if (!is.null(word)) word else "?"
})
cat(decoded_review)

? this film was just brilliant casting location scenery story direction
everyone's really suited the part they played and you could just imagine
being there robert ? is an amazing actor and now the same being director
? father came from the same scottish island as myself so i loved the fact
there was a real connection with this film the witty remarks throughout
the film were great it was just brilliant so much that i bought the film
as soon as it was released for ? and would recommend it to everyone to 
watch and the fly fishing was amazing really cried at the end it was so
sad and you know what they say if you cry at a film it must have been 
good and this definitely was also ? to the two little boy's that played'
the ? of norman and paul they were just brilliant children are often left
out of the ? list i think because the stars that play them all grown up
are such a big profile for the whole film but these children are amazing
and should be praised for what they have done don't you think the whole
story was so lovely because it was true and was someone's life after all
that was shared with us all

Data preparation

You cannot feed lists of integers into a neural network. You have to convert your lists to tensors. There are two ways to do this:

• Pad your lists so that they are all the same length, convert them to integer tensors of the form (samples, word_indices)and then use as the first layer in your network a layer capable of handling such integer tensors (the “embedding” layer, which we describe in detail later in the book).
• Do a hot encode to convert your lists into vectors of 0s and 1s. This would mean, for example, turning the order [3, 5] In a 10,000-dimensional vector that would be all 0s except for indices 3 and 5, which would be 1s. Then you can use a dense layer as the first layer in your network, capable of handling floating-point vector data.

Let's go with the post-solution of vectorizing the data, which you'll do manually for maximum clarity.

vectorize_sequences <- function(sequences, dimension = 10000) {
  # Creates an all-zero matrix of shape (length(sequences), dimension)
  results <- matrix(0, nrow = length(sequences), ncol = dimension) 
  for (i in 1:length(sequences))
    # Sets specific indices of results[i] to 1s
    results[i, sequences[[i]]] <- 1 
  results
}

x_train <- vectorize_sequences(train_data)
x_test <- vectorize_sequences(test_data)

Here is how the samples look now:

 num [1:10000] 1 1 0 1 1 1 1 1 1 0 ...

You should also convert your labels from numeric to numeric, which is straightforward:

Now the data is ready to be fed into the neural network.

Building your network

The input data is a vector, and the labels are scalars (1s and 0s): this is the simplest setup you'll ever encounter. One type of network that performs well on this type of problem is a simple stack of fully connected (“dense”) layers. relu Activations: layer_dense(units = 16, activation = "relu").

The argument given to each dense layer (16) is the number of hidden units of the layer. Oh Hidden unit A layer has one dimension in the representation space. You will recall from Chapter 2 that with each such dense layer a relu The activation tensor implements the following sequence of operations.

output = relay(dot(w, input) + b);

The 16 hidden units mean the weight matrix. W There will be a shape (input_dimension, 16): with the dot product W will represent the input data in a 16-dimensional representation space (and then you add the bias vector b And apply relu operation). You can intuitively understand the dimensionality of your representation space as “how much freedom you are allowing the network when learning internal representations.” More hidden units (a higher-dimensional representation space) allows your network to learn more complex representations, but it makes the network more computationally expensive and can lead to learning unwanted patterns. is (patterns that will improve performance on training data but not test data).

There are two important architectural decisions regarding such dense layer stacks:

• How many layers to use?
• How many hidden units to choose for each layer.

In Chapter 4, you will learn formal rules to guide you in making these choices. For now, you'll have to trust me with the following architecture choices:

• The two middle layers have 16 hidden units each.
• A third layer that will output a scalar prediction related to the sentiment of the current review.

Intermediate layers will use relu as their activation function, and the final layer will use sigmoid activation to assign a probability (score between 0 and 1, indicating how likely the sample target is “1”: how much of the review is positive likely). Oh relu (corrected linear unit) is a function that aims to zero out negative values.

A sigmoid “squashes” arbitrary values. [0, 1] interval, outputting something that can be interpreted as a probability.

Here's what the network looks like.

Here is the Keras implementation, like the MNIST example you saw earlier.

library(keras)

model <- keras_model_sequential() %>% 
  layer_dense(units = 16, activation = "relu", input_shape = c(10000)) %>% 
  layer_dense(units = 16, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")

Activation functions

Note that without the activation function e.g relu (Also called a Non-linearity), the dense layer will consist of two linear operations – a dot product and an addition:

output = dot ( w , input ) + b

So layer could only learn. Linear transformations (affine transformations) of the input data: Hypothesis space A layer will be the sum of all possible linear transformations of the input data in a 16-dimensional space. Such a hypothesis space is very limited and would not benefit from multiple layers of representation, since a deep stack of linear layers would still implement a linear process: adding more layers would not expand the hypothesis space. .

To access a richer hypothesis space that benefits from a deeper representation, you need a nonlinear or activation function. relu The most popular function in deep learning is activation, but there are many other candidates, all of which come with similarly weird names: prelu, eluand so on.

Loss function and optimizer

Finally, you need to choose a loss function and an optimizer. Since you have a binary classification problem and the output of your network is a probability (you terminate your network with a single unit layer with a sigmoid activation), it is better to use binary_crossentropy loss This is not the only viable choice: you can use, for example, mean_squared_error. But when you're working with models that generate probabilities, crossentropy is usually the best choice. Crossentropy is a quantity from the field of information theory that measures the distance between a probability distribution, or in this case the ground truth distribution, and your predictions.

This is the step where you configure the model. rmsprop Reformer and binary_crossentropy loss function. Remember that you will also monitor accuracy during training.

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)

You are passing your optimizer, loss function, and matrix as strings, which is possible because rmsprop, binary_crossentropyAnd accuracy Packaged as part of Keras. Sometimes you want to configure your optimizer parameters or pass a custom loss function or metric function. The former can be done by passing it as an optimization instance. optimizer Argument:

model %>% compile(
  optimizer = optimizer_rmsprop(lr=0.001),
  loss = "binary_crossentropy",
  metrics = c("accuracy")
) 

Custom loss and matrix functions can be provided by passing the function object as loss and/or metrics Arguments

model %>% compile(
  optimizer = optimizer_rmsprop(lr = 0.001),
  loss = loss_binary_crossentropy,
  metrics = metric_binary_accuracy
) 

Validating your point of view

To monitor the accuracy of the model on the data during training, you will create a validation set by separating 10,000 samples from the original training data.

val_indices <- 1:10000

x_val <- x_train[val_indices,]
partial_x_train <- x_train[-val_indices,]

y_val <- y_train[val_indices]
partial_y_train <- y_train[-val_indices]

Now you will train the model for 20 positions (20 iterations on all samples x_train And y_train tensor), in small batches of 512 samples. At the same time, you'll monitor the damage and integrity of the 10,000 samples you've separated. You do this by passing authentication data as validation_data Argument

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)

history <- model %>% fit(
  partial_x_train,
  partial_y_train,
  epochs = 20,
  batch_size = 512,
  validation_data = list(x_val, y_val)
)

On the CPU, this will take less than 2 seconds per time – training is finished in 20 seconds. At the end of each round, there is a short pause as the model calculates its loss and accuracy on 10,000 samples of validation data.

Note that the call fit() Return a history thing gave history The object is a plot() The method that enables us to visualize the training and validation metrics by distance:

Accuracy is plotted on the top panel and loss on the bottom panel. Note that your own results may vary slightly due to the different random initialization of your network.

As you can see, the training loss decreases with each round, and the training accuracy increases with each round. This is what you would expect when running a gradient descent optimization – the amount you are trying to minimize should decrease with each iteration. But this is not a case of loss of validity and accuracy: they seem to have peaked in the fourth period. This is an example of what we warned against earlier: a model that performs well on training data is not necessarily a model that performs well on data it has never seen before. In other words, what you see. Overfitting: After the second round, you are over-optimizing the training data, and you end up learning representations that are specific to the training data and don't generalize to data outside the training set.

In this case, to prevent overfitting, you can stop training after three periods. In general, you can use a number of techniques to reduce overfitting, which we'll cover in Chapter 4.

Let's train a new network from scratch for four periods and then evaluate it on test data.

model <- keras_model_sequential() %>% 
  layer_dense(units = 16, activation = "relu", input_shape = c(10000)) %>% 
  layer_dense(units = 16, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)

model %>% fit(x_train, y_train, epochs = 4, batch_size = 512)
results <- model %>% evaluate(x_test, y_test)

$loss
[1] 0.2900235

$acc
[1] 0.88512

This fairly naive approach achieves an accuracy of 88%. With the most advanced methods, you should be able to get close to 95%.

Generating forecasts

After training the network, you'll want to use it in a practical setting. You can make the reviews more likely to be positive by using them. predict Method:

 [1,] 0.92306918
 [2,] 0.84061098
 [3,] 0.99952853
 [4,] 0.67913240
 [5,] 0.73874789
 [6,] 0.23108074
 [7,] 0.01230567
 [8,] 0.04898361
 [9,] 0.99017477
[10,] 0.72034937

As you can see, the network is confident for some samples (0.99 or higher, or 0.01 or lower) but less confident for others (0.7, 0.2).

More experiments

The following experiments should help convince you that the architecture choices you've made are all reasonable, even though there's still room for improvement.

• You used two hidden layers. Try using one or three hidden layers, and see how doing so affects validation and test accuracy.
• Try using layers with more hidden units or less hidden units: 32 units, 64 units, etc.
• Try using mse loss function instead of binary_crossentropy.
• Try using tanh Instead of activation (an activation popular in the early days of neural networks). relu.

finish

Here's what you should take away from this example:

• You usually need to do a bit of pre-processing on your raw data to be able to feed it – as a tensor – into a neural network. A sequence of words can be encoded as a binary vector, but there are other encoding options.
• Densely layered piles with relu Activations can solve a wide range of problems (including emotion classification), and you'll use them often.
• In a binary classification problem (two output classes), your network should end up with a dense layer consisting of one unit and one sigmoid Activation: The output of your network should be a scalar between 0 and 1, encoding a probability.
• For a binary classification problem with such a scalar sigmoid output, the loss function you should use is binary_crossentropy.
• gave rmsprop Optimizer is generally a pretty good choice, no matter what your problem is. That's one less thing for you to worry about.
• As they get better at their training data, neural networks eventually start overfitting and eventually get exponentially worse results on data they've never seen before. Be sure to always monitor performance on data that is outside of the training set.