Pre-Training In A Nutshell

In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.

Pre-training, a key component of the current AI paradigm

Pre-trained has turned out to be one of the most important aspects of the current AI paradigm, where large language models, to transform into general-purpose engines, need pre-training.

Pre-training, therefore, through a transformer architecture, becomes the stepping stone to make the AI model extremely versatile and able to generalize across tasks, which is the core innovation of what made AI commercially viable right now.

Understanding pre-training

Pre-training in artificial intelligence is at least partly inspired by how humans learn. Instead of having to learn a topic from scratch, we transfer and repurpose existing knowledge to understand new ideas and navigate different tasks.

In an AI model, a similar process unfolds. The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In effect, the model can perform a new task based on prior experience.

One of the most critical aspects of pre-training is task-relatedness, or the idea that the task the model learns initially must be similar to the task it will perform in the future. For example, a model trained for object detection could not be later used to predict the weather.

Pre-training methods

Here are some of the ways pre-training is conducted in the natural language processing space.

Word2vec

Developed by Google, Word2vec is a tool that produces static word embedding and can be trained on millions of words by measuring word-to-word similarity. Word2Vec is part of a family of related models that are trained to construct linguistic word contexts.

The model, released in 2013, can detect synonymous words once trained and suggest additional words for a partial sentence.

GPT

GPT is a transformer-decoder-based language model based on the core premise of self-attention. To compute a representation of a given input sequence, the model can attend to different positions of that sequence.

GPT is trained over two stages. In the first stage, creator OpenAI uses a language modeling objective on unlabeled data to learn the initial parameters. Then, those parameters are adapted to a target task (otherwise referred to as a training example) using the corresponding supervised objective.

BERT

BERT is another transformer-decoder-based language model that is first trained on a large volume of text such as Wikipedia.

BERT is a fine-tuning and encoder-based model that features a bidirectional language model. Instead of the left-to-right word protection that decoder-based models like GPT use, BERT operates based on two new tasks.

The first pretraining task of the model is known as Masked Language Model (MLM), where 15% of the words are randomly masked and BERT is asked to predict them. As we noted, BERT can predict words in either direction.

The second task is related to model input. BERT does not use words as tokens but instead as word pieces. For instance, the word “working” is “work” and “ing” instead of “working”. The model then adds position embedding to avoid a weakness of self-attention where word position information is ignored.

Key takeaways

In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.
The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In essence, the model can perform a new task based on prior experience.
Three pre-training methods include Word2vec, GPT, and BERT. Each model has its own way of learning the data to make predictions.