How did humans teach AI to be helpful instead of just clever?

The answer

InstructGPT, published by OpenAI in March 2022, demonstrated that reinforcement learning from human feedback (RLHF) could dramatically improve a language model's ability to follow instructions and be helpful. The process works in three stages: first, human labelers write ideal responses to prompts; second, a reward model is trained on human rankings of different outputs; third, the language model is fine-tuned using proximal policy optimization (PPO) against this reward model. A 1.3B-parameter InstructGPT model was preferred by human evaluators over the 175B-parameter GPT-3, proving that alignment techniques could substitute for raw scale.