Study Guide · Reinforcement Learning · 7 min read

How Reinforcement Learning Works Under the Hood

RL's vocabulary is small — agent, environment, state, action, reward, policy — but those pieces assemble into everything from game-playing champions to the alignment layer behind modern chatbots.

The agent–environment loop

Each step, the agent observes a state, picks an action according to its policy (its current strategy), and lands in a new state with a reward. Learning updates the policy to favor actions that historically led to high rewards.

Two schools dominate: value-based methods learn how good each state-action pair is and act greedily on that map; policy-based methods learn the strategy directly. Modern systems often blend both.

Simulators make it safe

Trial-and-error against real warehouses or real bodies is slow and dangerous, so most RL trains in simulators — millions of virtual years of practice compressed into days — before any real-world deployment.

The catch is the 'reality gap': simulations never match reality perfectly, so engineers add randomness during training so policies survive real-world imperfection.

RLHF: teaching language models preferences

The most influential RL application today is RLHF — reinforcement learning from human feedback. Humans rank model responses; a reward model learns those preferences; RL nudges the language model toward helpful, harmless behavior.

This is why assistants refuse harmful requests and explain patiently instead of completing text like an autocomplete engine — the base model's raw behavior was reshaped by learned human preference.

Key Points

  • The loop is always: observe → act → receive reward → improve policy.
  • Simulators provide safe practice at scale; train for the reality gap.
  • Value-based and policy-based methods are the two main algorithm families.
  • RLHF converted RL from games research into the backbone of assistant alignment.

All study guides for this term: Reinforcement Learning, Explained Simply · How Reinforcement Learning Works Under the Hood · Reinforcement Learning in the Real World