Pre-Training In A Nutshell

In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.
The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In essence, the model can perform a new task based on prior experience.
Three pre-training methods include Word2vec, GPT, and BERT. Each model has its own way of learning the data to make predictions.

In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.

Table of Contents

Pre-training, a key component of the current AI paradigm

Pre-trained has turned out to be one of the most important aspects of the current AI paradigm, where large language models, to transform into general-purpose engines, need pre-training.

Pre-training, therefore, through a transformer architecture, becomes the stepping stone to make the AI model extremely versatile and able to generalize across tasks, which is the core innovation of what made AI commercially viable right now.

Understanding pre-training

Pre-training in artificial intelligence is at least partly inspired by how humans learn. Instead of having to learn a topic from scratch, we transfer and repurpose existing knowledge to understand new ideas and navigate different tasks.

In an AI model, a similar process unfolds. The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In effect, the model can perform a new task based on prior experience.

One of the most critical aspects of pre-training is task-relatedness, or the idea that the task the model learns initially must be similar to the task it will perform in the future. For example, a model trained for object detection could not be later used to predict the weather.

Pre-training methods

Here are some of the ways pre-training is conducted in the natural language processing space.

Word2vec

Developed by Google, Word2vec is a tool that produces static word embedding and can be trained on millions of words by measuring word-to-word similarity. Word2Vec is part of a family of related models that are trained to construct linguistic word contexts.

The model, released in 2013, can detect synonymous words once trained and suggest additional words for a partial sentence.

How is Word2vec trained?

Word2vec utilizes a shallow neural network with the one-hot embedding of each word serving as both its input and output. To better understand what one-hot embedding looks like in practice, consider the following example.

If a dictionary has five words, {‘the’, ‘cat’, ‘ate’, ‘its’, ‘dinner’}, then the one-hot embedding of the word “cat” is [0, 1, 0, 0, 0]. One potential way to train the model is to predict the one-hot embedding of a word as output and the one-hot embedding of the surrounding word as input.

Alternatively, Word2vec can be trained by predicting the surrounding words as output with a target word serving as input. In any case, a parameter matrix is generated once the training is complete. This matrix serves as a word-embedding dictionary that provides each word’s embedding of the training data.

It should also be noted that Word2vec is not an algorithm or model itself but instead refers to the Skip-gram and Continuous Bag of Words (CBOW) models. Both models are architectures that use neural networks to learn the underlying word representations for each word.

GPT

GPT is a transformer-decoder-based language model based on the core premise of self-attention. To compute a representation of a given input sequence, the model can attend to different positions of that sequence.

GPT is trained over two stages. In the first stage, creator OpenAI uses a language modeling objective on unlabeled data to learn the initial parameters. Then, those parameters are adapted to a target task (otherwise referred to as a training example) using the corresponding supervised objective.

An example of how GPT is trained

GPT utilizes static word embedding as input in addition to several layers of the transformer decoder.

Consider a five-word sentence: “w1, w2, w3, w4, w5.” If we take w4, for example, the word’s embedding will pass through a decoder layer and thus become a new embedding.

This new embedding incorporates information via attention w4 paid to w1, w2, and w3. Though beyond the scope of this article, think of attention as a new type of embedding that enables the model to predict a sequence of words more accurately from left to right.

BERT

BERT is another transformer-decoder-based language model that is first trained on a large volume of text such as Wikipedia.

BERT is a fine-tuning and encoder-based model that features a bidirectional language model. Instead of the left-to-right word protection that decoder-based models like GPT use, BERT operates based on two new tasks.

The first pretraining task of the model is known as Masked Language Model (MLM), where 15% of the words are randomly masked and BERT is asked to predict them. As we noted, BERT can predict words in either direction.

The second task is related to model input. BERT does not use words as tokens but instead as word pieces. For instance, the word “working” is “work” and “ing” instead of “working”. The model then adds position embedding to avoid a weakness of self-attention where word position information is ignored.

Pre-training applications

Broadly speaking, the applications of pre-training can be categorized into three groups.

1 – Transfer learning

Transfer learning is an application we touched on earlier and is a machine learning technique where a model trained on one task is repurposed for a second, related task.

Transfer learning is a popular approach in deep learning because of the vast time and computational resources required to develop neural networks from scratch.

To that end, transfer learning is an optimization method that facilitates rapid progress because the model has already been trained on a related task. However, it only works in deep learning if the model features learned in the first task are of a general nature.

2 – Classification

Pre-trained models can also be used in classification tasks such as image classification, which is the process of labeling images based on their features and characteristics.

Here, models work to identify similar features and objects in an image and assign labels to any that are present. The models are pre-trained on millions of labeled images and then fine-tuned to precisely recognize the features of each object.

Two examples of image classification models include the University of Oxford’s VGG-16 and ResNet-50, a convolutional neural network (CNN) that is 50 layers deep and based on 23 million parameters for precise classification.

3 – Feature extraction

Feature extraction is a process that seeks to reduce the number of variables required to describe vast datasets. Feature extraction reduces the computational resources required to process these datasets by reducing an initial set of raw data into more manageable groups.

This is achieved by employing various methods that combine and/or select variables into features that are informative, non-redundant, and can facilitate subsequent learning and generalization steps.

Note that the smaller, resultant dataset must still describe the original data set in a way that is accurate and complete. In other words, features must contain relevant information from the input data to enable a task to be performed even with reduced representation.

Key Takeaways:

Pre-Training in AI: Pre-training is a crucial process in the field of artificial intelligence, where a model is trained on one task to learn parameters that can then be used for other tasks. It enables models to become versatile and generalize across different tasks, making them commercially viable and effective.
Inspiration from Human Learning: The concept of pre-training is inspired by how humans learn and transfer existing knowledge to understand new ideas and tasks. Similarly, AI models are trained on one task to leverage that knowledge for performing other tasks.
Task-Relatedness in Pre-Training: One of the critical factors in pre-training is task-relatedness. The initial task that the model learns must be similar to the task it will perform in the future. For example, a model trained for object detection cannot be used to predict weather.
Pre-Training Methods:
- Word2vec: Developed by Google, Word2vec produces static word embeddings that can detect synonymous words and suggest words for incomplete sentences.
- GPT (Generative Pre-trained Transformer): GPT is a transformer-decoder-based language model that uses self-attention and is trained in two stages, initially on unlabeled data and then on a target task.
- BERT (Bidirectional Encoder Representations from Transformers): BERT is another transformer-based model trained on large text volumes and uses tasks like Masked Language Model and word piece input for fine-tuning.
Applications of Pre-Training:
- Transfer Learning: Pre-trained models can be repurposed for related tasks, saving time and computational resources needed to develop models from scratch.
- Classification: Pre-trained models are used in tasks like image classification, where they recognize features and assign labels to objects in images.
- Feature Extraction: Feature extraction reduces data dimensions, making it easier to process large datasets by selecting relevant variables for subsequent learning.