RLHF (Reinforcement Learning from Human Feedback)

RLHF (Reinforcement Learning from Human Feedback) is the training technique that uses human preference labels to fine-tune a language model's behaviour toward helpful, harmless, and instruction-following outputs. The pipeline has three phases: (1) **supervised fine-tuning** on demonstrations of desired behaviour, (2) training a **reward model** on pairs of outputs that humans ranked against each other, and (3) **reinforcement-learning fine-tuning** (typically PPO) of the language model against the reward model's scores, with a KL-divergence penalty to prevent drift too far from the supervised checkpoint. The technique was crucial to moving GPT-3 to ChatGPT — the base GPT-3 was a capable next-token predictor, but it took RLHF to make it reliably follow instructions and behave as a usable assistant.

The modern landscape has diversified beyond classical PPO-based RLHF. **DPO** (Direct Preference Optimisation) bypasses the explicit reward-modelling step by directly optimising on preference pairs — simpler, often cheaper, often equally effective. **Constitutional AI** (Anthropic) trains the model to critique and revise its own outputs against written principles, with RL from AI feedback (RLAIF) augmenting or replacing human labels. **RLAIF** more broadly uses a strong LLM to generate preference labels at scale. **DPO-like variants** — IPO, KTO, SimPO, ORPO — each tweak the loss function. The practical effect is that preference-based fine-tuning is now a commodity technique with multiple interchangeable implementations.

For APAC mid-market teams, the relevant question is almost never whether to run RLHF from scratch — base models already shipped with extensive preference tuning from their vendors. The live question is **preference fine-tuning for specialisation**: adapting a base model to your tone, your safety boundaries, your task format, using a few thousand preference pairs you collect internally. DPO is usually the right tool — less infrastructure, more stable training, comparable results to PPO-based RLHF in most published comparisons.

The non-obvious operational note: **preference data quality dominates technique choice**. A thousand carefully-curated preference pairs beat ten thousand sloppily-labelled ones. The most common failure mode is labelling pairs where both responses are acceptable on different dimensions — the gradient becomes noise and the model learns nothing useful. Invest in labelling guidelines and inter-annotator agreement before scaling preference-data collection.