RLHF
Reinforcement Learning from Human Feedback — aligning AI with human preferences for helpful, safe output.
Teaching AI What "Good" Means
A raw language model trained on internet text will confidently produce harmful, biased, or unhelpful content. RLHF is the process that transforms a raw model into an aligned assistant — one that's helpful, harmless, and honest.
This is why ChatGPT and Claude feel different from a raw GPT or base model. RLHF is the "fine-tuning with human judgment" step.
LLM produces multiple responses to the same prompt. Each response varies in quality and helpfulness.
Human annotators compare responses and pick the better one. These rankings train a Reward Model.
RL algorithm (PPO/GRPO) updates the LLM to favor responses the Reward Model scores highly.
Be the Human Annotator
The model generated two responses. Click the one you think is better:
Prompt: "Explain quantum computing in simple terms."
This is exactly what human annotators do thousands of times. Their preferences train the Reward Model.
The RLHF Pipeline
RLHF in the Wild
Uses RLHF extensively to align GPT-4 with user expectations for helpfulness and safety.
Anthropic pioneered Constitutional AI (CAI), an extension of RLHF using AI-generated feedback.
Llama, Mistral, and other open models use RLHF/DPO to create their "chat" variants.
Test Your Understanding
Q1.What does RLHF stand for?
Q2.What is the role of the Reward Model?
Q3.Why is RLHF necessary?