InstructGPT is a family of OpenAI language models fine-tuned to follow instructions using reinforcement learning from human feedback (RLHF). OpenAI researchers led by Long Ouyang described it in the March 2022 paper Training language models to follow instructions with human feedback: labelers wrote example answers and ranked model outputs, and GPT-3 was tuned toward the preferred ones.
Starting from labeler-written prompts and prompts sent to the OpenAI API, labelers showed the desired behaviour and GPT-3 learned from those examples.
02
Preference data
Labelers ranked several outputs per prompt, which is easier and more consistent than writing perfect answers.
03
Reinforcement learning
A reward model trained on the rankings guided further tuning of the model.
04
The headline result
In the paper's human evaluations, outputs from a 1.3-billion-parameter InstructGPT model were preferred to those from the 175-billion-parameter GPT-3.
Where you see it today
From human to AI feedback
Anthropic’s Constitutional AI replaced human labels for harmful outputs with a written list of principles and AI-generated preference judgments, then trained with RL on those (Bai et al., 2022).
Simpler preference tuning
Direct Preference Optimization showed the same preference objective can be solved with a simple classification loss, without a separate RL loop (Rafailov et al., 2023).
RL for reasoning
DeepSeek-R1 used reinforcement learning on tasks with checkable answers, such as maths and code, to develop reasoning without human-labelled reasoning traces (DeepSeek-AI, 2025).
The side effect: sycophancy
Training on human approval can reward agreement over accuracy. A study of five AI assistants found they consistently told users what matched their stated views, and that humans and preference models sometimes preferred convincing sycophantic answers to correct ones (Sharma et al., 2023).
InstructGPT and later post-training methods
Supervised fine-tuning
Feedback signalHuman-written example answers
What it changedModel imitates good responses
RLHF (InstructGPT)
Feedback signalHuman rankings turned into a reward model
What it changedModel optimised toward what people prefer
Constitutional AI
Feedback signalAI judgments guided by written principles
What it changedFar fewer human labels for harmlessness
Direct Preference Optimization
Feedback signalPreference pairs used directly
What it changedNo separate reward model or RL loop
RL with verifiable rewards
Feedback signalAutomatic checks of answers
What it changedReasoning on maths and code
How to use it
When judging a model, ask how it was post-trained, not only how large it is.
Collect your own preference data: have experts rank model answers on your real tasks.
Write explicit principles for tone and limits, as Constitutional AI does, before tuning or prompting.
Test for sycophancy: state a wrong belief and see whether the model corrects you.
Prefer tasks with checkable answers when you want to improve a model with feedback loops.
The Business Engineer lens
InstructGPT showed that post-training, not only scale, decides how useful a model feels, and that idea now shapes frontier economics as labs spend more on reinforcement learning after pre-training. The Business Engineer’s view is that whoever owns good feedback signals on real tasks holds a durable advantage.
A family of OpenAI models, described in 2022, that fine-tuned GPT-3 to follow instructions using human demonstrations and reinforcement learning from human feedback.
How is InstructGPT different from GPT-3?
GPT-3 was trained to predict text; InstructGPT was further tuned on human feedback, and in OpenAI's evaluations people preferred a much smaller InstructGPT model's outputs to GPT-3's.
What is RLHF?
Reinforcement learning from human feedback: people rank model outputs, a reward model learns those preferences, and the language model is optimised to score well on it.
What replaced RLHF?
It was extended rather than replaced: AI feedback (Constitutional AI), direct preference methods such as DPO, and RL on verifiable tasks are now used alongside it.
The complete explainer, cleaned up and reformatted for mobile.
InstructGPT is the successor to the GPT-3 large language model — as explored in the intelligence factory race between AI labs — (LLM) developed by OpenAI. InstructGPT is a model which uses reinforcement learning from human feedback that gets incorporated into the GPT model to make it more reliable.
From GPT-3 to IntructGPT
GPT-3 has been an incredible turning point for the current AI paradigm, where machine learning models could be turned into general-purpose engines via an architecture called a transformer.
The transformer architecture – sometimes referred to as the transformer neural network or transformer model – is an architecture that endeavors to solve sequence-to-sequence tasks while easily handling long-range dependencies.
Along the way, OpenAI figured out a few other components that could be plugged into to make these large language models effective.
Indeed, in-context learning via prompting and learning from human feedback proved effective additions, which OpenaAI used to move from its GPT model to what would later become InstructGPT.
InstructGPT, thus, is the underlying stack that sits beneath ChatGPT. Its core difference with GPT is that InstructGPT uses a human feedback approach in the fine-tuning process, where humans show a set of outputs to the GPT model once it has been pre-trained thourhg the InstructGPT framework.
In the InstructGPT framework, humans iterate on a much smaller dataset, acting in a few ways.
First by producing the desired output and second by comparing it with that generated by GPT.
Second, by labeling the output coming from GPT from human feedback.
Third, by showing that output to the GPT model to instruct it toward the desired outcome on narrower tasks and sort of questions.
This is how we get (mostly) from the GPT model to the InstructGPT model, which has now become a standard within OpenAI’s technology.
Understanding InstructGPT
InstructGPT is the result of an overhaul of the GPT-3 language model. Responding to user complaints about GPT-3, creator OpenAI made the new and improved model:
Better at following English instructions.
Less inclined to spread misinformation (more truthful), and
Less likely to produce toxic results or those that reflect harmful sentiments.
The problem with GPT-3 arose because it was trained to predict the next word from a large dataset and not to safely perform the task the user wanted. To address the problem, OpenAI used a technique known as reinforcement learning from human feedback (RLHF).
With reinforcement learning, an AI agent learns to make decisions by performing actions in an environment and receiving feedback as rewards or penalties.
OpenAI shared already back in 2017 how this process was instrumental in developing safe AI systems. And yet this same methodology proved quite effective also to make these AI systems way more effective for specific tasks.
To be sure, reinforcement learning from human feedback wasn’t a discovery of OpenAI but an achievement of academia.
Yet, what the OpenAI team was good at, was in scaling this approach.
Back then, the team at OpenAI trained an algorithm with 900 bits of feedback from a human evaluator to make it learn to backflip.
Of course, that doesn’t seem a huge achievement for a simple and narrow task, and yet, this was the embryonic stage of what would later turn into something like ChatGPT.
The three steps in InstructGPT training
The RLHF process can best be described as a 3-step feedback cycle between the person, reinforcement learning, and the model’s understanding of the goal.
Source: OpenAI
To better understand this process, let’s explain each step.
Step 1 – Collect human-written demonstration data and train a supervised policy
Once a prompt has been sampled from a dataset, a labeler demonstrates desirable output behavior. These can be submitted by GPT-3 users, but OpenAI researchers also guide labelers based on written instructions, informal conservation, and feedback on specific examples where necessary.
Then, the data are used to refine GPT-3 by training supervised learning baselines.
Step 2 – Collect comparison data and train the reward model
Next, a dataset of human-labeled comparisons between two outputs on a larger set of prompts is collected. Several model outputs are sampled from a prompt, and the labeler ranks each output from best to worst.
The reward model (RM) is then trained on this dataset to clarify which output OpenAI’s labelers prefer.
Step 3 – Use the reward model as a reward function to fine-tune the GPT-3 policy
In step three, a new prompt is sampled from the dataset, and based on the above, the policy generates an output and calculates a reward. The reward is maximized by the company’s Proximal Policy Optimization (PPO) algorithm.
The result is that InstructGPT is much better at following instructions.
This is what the whole process looks like:
InstructGPT vs. GPT-3
Instruct GPT-3 is the model of choice for OpenAI labelers despite it having 100x fewer parameters than the model on which it is based.
The company also noted that “at the same time, we show that we don’t have to compromise on GPT-3’s capabilities, as measured by our model’s performance on academic NLP evaluations.”
InstructGPT models were in beta mode on the API for over twelve months and are now its default language models. Moving forward, OpenAI believes that model refinement with humans in the loop is the most effective way to improve reliability and safety.
Key takeaways
InstructGPT is the successor to the GPT-3 large language model (LLM) developed by OpenAI. It was developed in response to user complaints about the toxic or harmful results generated by GPT-3.
To address the problem, OpenAI used a technique known as reinforcement learning from human feedback (RLHF). The process is best described as a 3-step feedback cycle between a human, reinforcement learning, and the model’s understanding of the goal.
Despite the increase in performance, it is worth noting that Instruct GPT3 is the model of choice for OpenAI labelers despite it having 100x fewer parameters.
Key Highlights on InstructGPT
Introduction to InstructGPT:
InstructGPT is the successor to OpenAI’s GPT-3 model.
Developed to address user complaints regarding GPT-3, especially around toxic or misleading outputs.
Uses reinforcement learning from human feedback (RLHF) for enhanced reliability and safety.
From GPT-3 to InstructGPT:
GPT-3 was based on the transformer architecture, a major milestone in AI.
InstructGPT incorporates in-context learning via prompting and learns from human feedback.
The main distinction between GPT and InstructGPT is the latter’s use of human feedback during fine-tuning.
Understanding InstructGPT:
Designed to better follow English instructions, reduce misinformation, and minimize toxic outputs.
It uses a three-step RLHF process, enhancing the model’s alignment with human-desired outputs.
Three Steps in InstructGPT Training:
Step 1: Collect human-written demonstration data and refine the model using supervised learning.
Step 2: Use human-labeled data to train a reward model that assesses the model’s outputs.
Step 3: Fine-tune the model using the reward model to maximize the desired output.
InstructGPT vs. GPT-3:
InstructGPT is preferred by OpenAI labelers despite having 100x fewer parameters than GPT-3.
OpenAI believes that refining models with human involvement is the best way to improve reliability and safety.
Overall Significance:
InstructGPT models, after being in beta mode for over a year, are now the default language models for OpenAI.
The focus is on ensuring the AI understands and executes user instructions without compromising on safety and reliability.
Connected · InstructGPT
Where to go next from InstructGPT
Picked for this page by topic. Every link checked live.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
The Business Engineer readThe Economics of Reinforcement LearningRead on businessengineer.ai →
