Business Pill 22 · How a model learns what people prefer
A language model that has only read text can continue a sentence, but that is not the same as being helpful. RLHF is the step that taught it what people consider a good answer.
A short explainer video. The numbers in it are round numbers for illustration.
A language model starts by reading enormous amounts of text. That teaches it to continue any sentence. But continuing a sentence is not the same as being helpful. Ask the raw model a question and it may ramble, dodge, or answer with another question.
How a Judge Is Trained
So people step in. The model writes several answers to the same question. Human reviewers compare them and say which one is better, thousands of times, on thousands of questions.
Those comparisons are used to train a second model, a judge. Its only job is to predict which answer a person would prefer. Now the judgment of thousands of reviewers can be applied millions of times, automatically.
The Loop
Then the loop begins. The model answers, the judge scores it, and the model is adjusted to make high-scoring answers more likely. It tries again, round after round, and drifts toward what people prefer.
This is reinforcement learning from human feedback, RLHF for short. It is the step that turned a text predictor into an assistant you can talk to.
Why It Matters
It matters for business for two reasons. First, the reviewers define what good means: their taste becomes the model’s taste.
Second, the model learns to please. It is rewarded for answers people like, which are not always the answers that are correct. That is why a model can sound confident and agreeable and still be wrong. It was trained on preference, not on truth.
RLHF builds on Pretraining. RLHF starts from a model that learned to continue sentences by reading enormous text, then trains it toward helpful answers.
RLHF builds on Data Labeling. RLHF depends on human reviewers judging answers, the people-supply-the-answers work of data labeling; their taste becomes the model’s.
Guardrails contrasts with RLHF. RLHF shapes behaviour inside the model; a guardrail is a rule enforced by software around the model, checking that it behaved.
Preference Gap(lesson unlocks soon) contrasts with RLHF. RLHF teaches a model what reviewers in general prefer; the preference gap is about one person’s taste, closed only by knowing them.
The Business Engineer framework maps where each part of the AI stack sits and who holds the leverage. These pills are short lessons from the same library.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
