Pre-training is the first, general stage of training an AI model on a very large unlabelled dataset, before it is adapted to specific tasks. For language models it means learning to predict text. OpenAI researchers led by Alec Radford showed in 2018 that generative pre-training followed by task-specific fine-tuning beat purpose-built models on most of the language tasks they tested (Radford et al., 2018), the recipe behind the GPT family.
Collect and filter a very large dataset; quality and diversity matter as much as size.
02
Self-supervised objective
The model learns by predicting hidden or next pieces of the data, so no human labels are needed.
03
Scale under a budget
Model size, data and compute are traded off; research found many large models had been trained on too little data for their size (Hoffmann et al., 2022).
04
Hand over to post-training
The pre-trained model is then fine-tuned or trained with feedback for specific tasks and behaviour.
Where you see it today
The data wall
Researchers projected that, if trends continued, models would be trained on datasets roughly the size of the whole stock of public human text sometime between 2026 and 2032 (Villalobos et al., 2022). That is why labs look to synthetic data, other data types and better data efficiency.
Training data becomes a contract
Data that was once scraped is increasingly licensed. In February 2024 Reddit expanded its partnership with Google, giving structured access to its posts through its Data API, including for training models.
Training models on text produced by earlier models can make the tails of the original data disappear, a defect the authors call model collapse (Shumailov et al., 2024). Genuine human data becomes more valuable as the web fills with generated text.
The shift to post-training
Recent gains have come increasingly from what happens after pre-training: instruction tuning, human feedback and reinforcement learning on tasks with checkable answers, such as maths and code. Pre-training still sets the ceiling the later stages work within.
Pre-training vs later training stages
Pre-training
What the model learnsGeneral patterns of language, code and knowledge
Main inputVery large unlabelled corpus
Supervised fine-tuning
What the model learnsTo follow instructions or do a task
What the model learnsNothing new; it spends more effort per query
Main inputExtra inference compute
How to use it
When choosing a model, ask what it was pre-trained on, not just how it scores after tuning.
Fine-tune or prompt a strong pre-trained model before considering training your own.
Treat proprietary human data, such as support logs or expert notes, as an asset for post-training.
Check licensing and consent for any data used in training before you rely on the model commercially.
Keep a human-written evaluation set that no model has seen, so you can judge new versions fairly.
The Business Engineer lens
Pre-training turned compute and data into general capability, and it set the cost structure of frontier AI. The Business Engineer tracks how value is moving across pre-training, post-training and inference, because where the next gains come from decides who carries the bill and who captures the margin.
It is the first stage of training a model on a very large unlabelled dataset, so it learns general patterns before being adapted to specific tasks.
What does the P in GPT stand for?
Pre-trained: GPT stands for Generative Pre-trained Transformer, building on OpenAI's 2018 work on generative pre-training.
What is the difference between pre-training and fine-tuning?
Pre-training builds broad capability from huge unlabelled data; fine-tuning adapts the pre-trained model to a task or behaviour using smaller, curated data.
Will AI run out of training data?
Researchers have projected that public human text could become a constraint within this decade if trends continue, which is pushing labs toward licensed, synthetic and other kinds of data.
The complete explainer, cleaned up and reformatted for mobile.
In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.
The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In essence, the model can perform a new task based on prior experience.
Three pre-training methods include Word2vec, GPT, and BERT. Each model has its own way of learning the data to make predictions.
In the context of AI, pre-training describes the process of training a model with one task so that it can form parameters to use in other tasks.
Pre-training, a key component of the current AI paradigm
Pre-trained has turned out to be one of the most important aspects of the current AI paradigm, where large language models, to transform into general-purpose engines, need pre-training.
Pre-training, therefore, through a transformer architecture, becomes the stepping stone to make the AI model extremely versatile and able to generalize across tasks, which is the core innovation of what made AI commercially viable right now.
Understanding pre-training
Pre-training in artificial intelligence is at least partly inspired by how humans learn. Instead of having to learn a topic from scratch, we transfer and repurpose existing knowledge to understand new ideas and navigate different tasks.
In an AI model, a similar process unfolds. The model is first trained on a task or dataset with the resultant parameters used to train another model on a different task or dataset. In effect, the model can perform a new task based on prior experience.
One of the most critical aspects of pre-training is task-relatedness, or the idea that the task the model learns initially must be similar to the task it will perform in the future. For example, a model trained for object detection could not be later used to predict the weather.
Pre-training methods
Here are some of the ways pre-training is conducted in the natural language processing space.
Word2vec
Developed by Google, Word2vec is a tool that produces static word embedding and can be trained on millions of words by measuring word-to-word similarity. Word2Vec is part of a family of related models that are trained to construct linguistic word contexts.
The model, released in 2013, can detect synonymous words once trained and suggest additional words for a partial sentence.
How is Word2vec trained?
Word2vec utilizes a shallow neural network with the one-hot embedding of each word serving as both its input and output. To better understand what one-hot embedding looks like in practice, consider the following example.
If a dictionary has five words, {‘the’, ‘cat’, ‘ate’, ‘its’, ‘dinner’}, then the one-hot embedding of the word “cat” is [0, 1, 0, 0, 0]. One potential way to train the model is to predict the one-hot embedding of a word as output and the one-hot embedding of the surrounding word as input.
Alternatively, Word2vec can be trained by predicting the surrounding words as output with a target word serving as input. In any case, a parameter matrix is generated once the training is complete. This matrix serves as a word-embedding dictionary that provides each word’s embedding of the training data.
It should also be noted that Word2vec is not an algorithm or model itself but instead refers to the Skip-gram and Continuous Bag of Words (CBOW) models. Both models are architectures that use neural networks to learn the underlying word representations for each word.
GPT
GPT is a transformer-decoder-based language model based on the core premise of self-attention. To compute a representation of a given input sequence, the model can attend to different positions of that sequence.
GPT is trained over two stages. In the first stage, creator OpenAI uses a language modeling objective on unlabeled data to learn the initial parameters. Then, those parameters are adapted to a target task (otherwise referred to as a training example) using the corresponding supervised objective.
An example of how GPT is trained
GPT utilizes static word embedding as input in addition to several layers of the transformer decoder.
Consider a five-word sentence: “w1, w2, w3, w4, w5.” If we take w4, for example, the word’s embedding will pass through a decoder layer and thus become a new embedding.
This new embedding incorporates information via attention w4 paid to w1, w2, and w3. Though beyond the scope of this article, think of attention as a new type of embedding that enables the model to predict a sequence of words more accurately from left to right.
BERT
BERT is another transformer-decoder-based language model that is first trained on a large volume of text such as Wikipedia.
BERT is a fine-tuning and encoder-based model that features a bidirectional language model. Instead of the left-to-right word protection that decoder-based models like GPT use, BERT operates based on two new tasks.
The first pretraining task of the model is known as Masked Language Model (MLM), where 15% of the words are randomly masked and BERT is asked to predict them. As we noted, BERT can predict words in either direction.
The second task is related to model input. BERT does not use words as tokens but instead as word pieces. For instance, the word “working” is “work” and “ing” instead of “working”. The model then adds position embedding to avoid a weakness of self-attention where word position information is ignored.
Pre-training applications
Broadly speaking, the applications of pre-training can be categorized into three groups.
1 – Transfer learning
Transfer learning is an application we touched on earlier and is a machine learning technique where a model trained on one task is repurposed for a second, related task.
Transfer learning is a popular approach in deep learning because of the vast time and computational resources required to develop neural networks from scratch.
To that end, transfer learning is an optimization method that facilitates rapid progress because the model has already been trained on a related task. However, it only works in deep learning if the model features learned in the first task are of a general nature.
2 – Classification
Pre-trained models can also be used in classification tasks such as image classification, which is the process of labeling images based on their features and characteristics.
Here, models work to identify similar features and objects in an image and assign labels to any that are present. The models are pre-trained on millions of labeled images and then fine-tuned to precisely recognize the features of each object.
Two examples of image classification models include the University of Oxford’s VGG-16 and ResNet-50, a convolutional neural network (CNN) that is 50 layers deep and based on 23 million parameters for precise classification.
3 – Feature extraction
Feature extraction is a process that seeks to reduce the number of variables required to describe vast datasets. Feature extraction reduces the computational resources required to process these datasets by reducing an initial set of raw data into more manageable groups.
This is achieved by employing various methods that combine and/or select variables into features that are informative, non-redundant, and can facilitate subsequent learning and generalization steps.
Note that the smaller, resultant dataset must still describe the original data set in a way that is accurate and complete. In other words, features must contain relevant information from the input data to enable a task to be performed even with reduced representation.
Key Takeaways
Pre-Training in AI: Pre-training is a crucial process in the field of artificial intelligence, where a model is trained on one task to learn parameters that can then be used for other tasks. It enables models to become versatile and generalize across different tasks, making them commercially viable and effective.
Inspiration from Human Learning: The concept of pre-training is inspired by how humans learn and transfer existing knowledge to understand new ideas and tasks. Similarly, AI models are trained on one task to leverage that knowledge for performing other tasks.
Task-Relatedness in Pre-Training: One of the critical factors in pre-training is task-relatedness. The initial task that the model learns must be similar to the task it will perform in the future. For example, a model trained for object detection cannot be used to predict weather.
Pre-Training Methods:
Word2vec: Developed by Google, Word2vec produces static word embeddings that can detect synonymous words and suggest words for incomplete sentences.
GPT (Generative Pre-trained Transformer): GPT is a transformer-decoder-based language model that uses self-attention and is trained in two stages, initially on unlabeled data and then on a target task.
BERT (Bidirectional Encoder Representations from Transformers): BERT is another transformer-based model trained on large text volumes and uses tasks like Masked Language Model and word piece input for fine-tuning.
Applications of Pre-Training:
Transfer Learning: Pre-trained models can be repurposed for related tasks, saving time and computational resources needed to develop models from scratch.
Classification: Pre-trained models are used in tasks like image classification, where they recognize features and assign labels to objects in images.
Feature Extraction: Feature extraction reduces data dimensions, making it easier to process large datasets by selecting relevant variables for subsequent learning.
Transfer Learning
Description– Transfer learning is a machine learning technique where a model trained on one task is reused as the starting point for a model on a second related task. – It involves leveraging knowledge gained from one domain to solve a different, but related, problem. – Transfer learning can accelerate model training and improve performance, especially when labeled data is limited for the target task.
When to Apply– When working with limited labeled data for a specific task, transfer learning can provide a head start by utilizing knowledge learned from a related task or domain. – It is particularly useful in scenarios where the source and target tasks share similarities, allowing for the transfer of learned features or representations. – Transfer learning can be applied across various domains, including natural language processing, computer vision, and speech recognition, to improve model performance and efficiency.
Fine-Tuning
Description– Fine-tuning involves taking a pre-trained model and further training it on a new dataset or task. – It allows for adapting the learned representations of the pre-trained model to better fit the specific nuances of the target task. – Fine-tuning typically involves adjusting the model’s parameters using gradient descent with the new data while retaining most of the knowledge learned during pre-training.
When to Apply– When the pre-trained model needs to be customized for a specific task or domain with a smaller dataset. – Fine-tuning is useful when the pre-trained model captures relevant features or patterns that can be adapted to the target task, leading to improved performance and faster convergence during training. – It is applied in scenarios where the pre-trained model’s general knowledge can be effectively transferred to the target task with minimal adjustments.
Self-Supervised Learning
Description– Self-supervised learning is a learning paradigm where models are trained to predict certain parts of the input data from other parts of the same data. – It does not require explicit human annotation of the data, as the supervision signal is generated from the input data itself. – Self-supervised learning can be used for pre-training models on large amounts of unlabeled data, which can then be fine-tuned on labeled data for specific tasks.
When to Apply– When working with large amounts of unlabeled data, self-supervised learning can be used to pre-train models to learn useful representations or features from the data. – It is particularly effective in domains where labeled data is scarce or expensive to obtain, as it allows for leveraging readily available unlabeled data for pre-training. – Self-supervised learning can be applied to various modalities, including text, images, and audio, to learn rich representations that capture underlying structures or semantics in the data.
Unsupervised Learning
Description– Unsupervised learning is a machine learning paradigm where models are trained on data without explicit supervision or labeled outputs. – It aims to discover patterns, structures, or representations in the data without predefined target labels. – Unsupervised learning techniques include clustering, dimensionality reduction, and generative modeling, which can be used for pre-training models on unlabeled data.
When to Apply– When working with large amounts of unlabeled data, unsupervised learning techniques can be used to pre-train models to capture underlying patterns or structures in the data. – Unsupervised pre-training serves as a foundation for subsequent supervised or semi-supervised learning tasks, where labeled data is available for fine-tuning or further training. – Unsupervised learning is applied in various domains, including anomaly detection, data compression, and feature learning, to uncover hidden insights or representations from unlabeled data.
Generative Adversarial Networks (GANs)
Description– Generative Adversarial Networks (GANs) are a class of neural networks used for generating new data samples from a given distribution. – GANs consist of two neural networks, a generator and a discriminator, trained simultaneously in a zero-sum game framework. – GANs can be used for unsupervised learning tasks, such as generating realistic images, which can then be used for pre-training models on unlabeled data.
When to Apply– When generating synthetic data samples to augment the training data for pre-training models. – GANs can be used to generate diverse and realistic data samples, which can improve the generalization and robustness of pre-trained models. – GANs are applied in various domains, including computer vision, natural language processing, and healthcare, to generate data for training and evaluation purposes.
Autoencoders
Description– Autoencoders are neural network models trained to learn efficient representations of input data by reconstructing it from a compressed latent space. – They consist of an encoder network that maps the input data to a latent representation and a decoder network that reconstructs the input data from the latent representation. – Autoencoders can be used for unsupervised learning tasks, such as dimensionality reduction and feature learning, which can be beneficial for pre-training models on unlabeled data.
When to Apply– When learning compact and informative representations of input data from large unlabeled datasets. – Autoencoders can capture salient features or patterns in the data, which can be transferred to downstream tasks through pre-training. – Autoencoders are applied in various domains, including image and audio processing, anomaly detection, and recommendation systems, to learn meaningful representations of input data for subsequent analysis or modeling tasks.
Word Embeddings
Description– Word embeddings are dense vector representations of words in a continuous vector space, learned from large text corpora using unsupervised learning techniques. – They capture semantic relationships between words and can be used to represent words in a way that preserves their contextual meaning. – Word embeddings pre-trained on large text corpora, such as Word2Vec, GloVe, and FastText, can be used as feature inputs for downstream natural language processing tasks.
When to Apply– When working with text data, word embeddings pre-trained on large text corpora can be used to initialize the embedding layer of neural network models. – Pre-trained word embeddings capture semantic similarities between words, which can improve the performance of NLP models on various tasks, such as sentiment analysis, named entity recognition, and machine translation. – Word embeddings are widely used in NLP applications to represent textual data in a dense and semantically meaningful way, facilitating effective model training and inference.
Pre-Trained Language Models
Description– Pre-trained language models are large-scale neural network models trained on vast amounts of text data using unsupervised learning techniques. – They learn to generate contextualized representations of words, phrases, and sentences by predicting missing words or masked tokens in the input text. – Pre-trained language models, such as BERT, GPT, and XLNet, capture rich semantic information from text data, which can be fine-tuned on specific natural language understanding or generation tasks with labeled data.
When to Apply– When working with natural language processing tasks, pre-trained language models can be used as feature extractors or initialized as the backbone of neural network architectures. – Pre-trained language models capture contextual information from text data, which can improve the performance of downstream NLP tasks, such as text classification, question answering, and language translation. – Pre-trained language models have become a cornerstone of modern NLP research and applications, enabling state-of-the-art performance on a wide range of language understanding and generation tasks.
Data Augmentation
Description– Data augmentation is a technique used to artificially increase the size and diversity of a training dataset by applying various transformations to the original data samples. – It helps improve the generalization and robustness of machine learning models by exposing them to different variations of the input data during training. – Data augmentation techniques include geometric transformations, color jittering, cropping, rotation, and noise injection, which can be applied to images, audio, text, and other types of data.
When to Apply– When training machine learning models on limited labeled data, data augmentation can be used to generate additional training samples with minimal manual annotation efforts. – Data augmentation introduces diversity and variability into the training dataset, which can help models learn invariant features and improve their performance on unseen data. – Data augmentation is widely used in computer vision, speech recognition, natural language processing, and other domains to enhance the robustness and generalization of machine learning models.
Multi-Task Learning
Description– Multi-task learning is a machine learning paradigm where a single model is trained on multiple related tasks simultaneously. – It allows the model to leverage shared representations and learn task-specific features across different tasks, leading to improved performance and generalization. – Multi-task learning can be used for pre-training models on diverse datasets with multiple objectives, which can enhance the model’s ability to transfer knowledge to downstream tasks.
When to Apply– When working with multiple related tasks or datasets, multi-task learning can be used to jointly train a single model on all tasks, leveraging shared knowledge and representations. – Multi-task learning encourages the model to learn common features and representations across tasks, which can improve its performance on individual tasks and facilitate knowledge transfer between tasks. – Multi-task learning is applied in various domains, including computer vision, natural language processing, and healthcare, to learn robust and generalizable models from diverse datasets and objectives.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
