NeurIPS 2022 · 2022
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin
OpenAI
What this paper is
InstructGPT (Ouyang et al., 2022) introduced the RLHF pipeline behind ChatGPT-style models: supervised fine-tuning on human demonstrations, a reward model trained on human preference comparisons, then PPO optimization against it. The 1.3B InstructGPT outputs were preferred over 175B GPT-3, showing alignment training beats raw scale for instructions.
Abstract summary
Introduces InstructGPT and the RLHF (Reinforcement Learning from Human Feedback) pipeline: (1) collect demonstrations from human labelers for supervised fine-tuning, (2) collect human preference comparisons over model outputs to train a reward model, (3) optimize the LM against the reward model via PPO. Shows that this pipeline dramatically improves helpfulness, truthfulness, and harmlessness compared to the raw GPT-3 baseline, at a fraction of the parameter count.
Our summary in our own words, see the canonical source links below for the original abstract.
Why we cite this paper
RLHF is the alignment technique that turned raw foundation models into the instruct-tuned helpful-by-default behavior that ChatGPT, Claude, and Gemini exhibit. Our engineering-acceleration tracking of frontier-AI labs (Anthropic, OpenAI, etc.) and the agentic AI categories operates on a substrate where this paper's pipeline is the alignment baseline.
Where this matters for deal flow
Key findings
- 1A 1.3B-parameter InstructGPT model outperforms 175B-parameter GPT-3 on human-preference evaluations after RLHF.
- 2Three-stage pipeline (SFT → reward model → PPO) became the de-facto alignment recipe for major frontier labs.
- 3Helpfulness, truthfulness, and harmlessness can be simultaneously improved without major capability loss.
- 4Modern alternatives (DPO, KTO, RLAIF) achieve similar results without the explicit reward-model step but inherit the framing.
Canonical sources
Related glossary terms
Frequently Asked Questions
What is RLHF?▾
Reinforcement Learning from Human Feedback, the training technique that aligns LLMs to human-preferred outputs after pretraining. See /define/rlhf for the full term definition.
Why is this paper considered foundational?▾
InstructGPT formalized the RLHF pipeline that ChatGPT, Claude, and Gemini training pipelines use as the alignment baseline. The paper turned LLMs from raw text-prediction models into instruction-following assistants.
What are the three stages of the RLHF pipeline?▾
(1) supervised fine-tuning on human-written demonstrations, (2) training a reward model on human preference comparisons over model outputs, and (3) optimizing the language model against that reward model with PPO reinforcement learning.
Did a smaller InstructGPT model beat GPT-3?▾
Yes. The paper reports that a 1.3B-parameter InstructGPT model was preferred by human evaluators over the 175B-parameter GPT-3 baseline, a roughly 100× parameter reduction at higher human-preference quality.
See who is building on this, before the round prices it in
We track engineering acceleration across the AI & Machine Learning sector this paper informs: commit velocity, contributor influx, and repo-creation pulse, surfacing breakout teams 21 to 47 days before the fundraise is public.
Five breakout startups, every Sunday, before the round gets crowded
The free Acceleration Watch: five venture-backed teams accelerating on the engineering signal, translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.
Other research papers
NeurIPS 2017 · 2017
Attention Is All You Need
NeurIPS 2020 · 2020
Language Models are Few-Shot Learners
NeurIPS 2020 · 2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
ICLR 2022 · 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv preprint · 2022
Constitutional AI: Harmlessness from AI Feedback
NeurIPS 2022 · 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Read our own methodology paper
Code-Side Sourcing methodology, replicable on the open dataset.
Read /methodologyRelated papers
More in Research Paper