How Reinforcement Learning Trains AI Through Reward and Penalty

The AI That Mastered Go Without Studying Human Games

In 2016, DeepMind's AlphaGo defeated 18-time world champion Lee Sedol four games to one — a milestone widely considered a decade premature by the Go community. A year later, AlphaGo Zero surpassed it entirely, having learned exclusively through self-play from random moves, never seeing a single human game. Starting from scratch, AlphaGo Zero defeated the original AlphaGo 100 games to none after just 40 days of training. This achievement demonstrated that reinforcement learning, given sufficient compute and a well-defined reward signal, can discover strategies that transcend the entire recorded history of human expertise in a domain.

Reinforcement learning (RL) occupies a distinct position among machine learning paradigms. Supervised learning requires labeled examples. Unsupervised learning finds patterns in unlabeled data. RL learns from the consequences of actions — optimizing behavior through a feedback loop of states, actions, and rewards that mirrors the trial-and-error structure of biological learning.

The Markov Decision Process Framework

Every reinforcement learning problem is formalized as a Markov Decision Process (MDP). An MDP defines the elements of the learning problem precisely.

• State (S): A representation of the environment's current configuration — the board position in Go, the pixel frame in an Atari game, the joint angles and velocities of a robot arm
• Action (A): The set of choices available to the agent — which tile to play, which button to press, which direction to move joints
• Transition function P(s'|s,a): The probability of transitioning to state s' after taking action a in state s — may be known (model-based RL) or unknown (model-free RL)
• Reward function R(s,a,s'): The scalar signal the agent receives after each transition — the fundamental signal that defines what the agent should optimize
• Discount factor γ: A value between 0 and 1 that determines how much the agent weights future rewards relative to immediate rewards

The Markov property — that the future state depends only on the current state and action, not on history — makes the MDP framework tractable. In practice, many real environments violate this property, motivating the use of memory-augmented architectures like LSTMs in RL agents.

Value Functions and the Bellman Equation

The agent's goal is to maximize cumulative discounted reward. Two value functions formalize this objective:

The state value function V(s) represents the expected cumulative reward from state s, following the current policy: V(s) = E[R_t + γR_{t+1} + γ²R_{t+2} + ... | S_t = s]

The action-value function Q(s,a) extends this to value state-action pairs: the expected cumulative reward when taking action a in state s, then following the policy thereafter. Q-values are directly useful for action selection — the agent picks the action with the highest Q-value in each state.

The Bellman equation recursively defines these values: Q(s,a) = R(s,a) + γ × max_{a'} Q(s',a'). Q-learning uses this equation iteratively, updating Q-value estimates from experience until they converge to the true optimal values.

Key RL Algorithms Compared

The Exploration-Exploitation Dilemma

Every RL agent faces a fundamental tension: exploit known high-reward behaviors, or explore new actions that might yield higher rewards. Too much exploitation traps the agent in local optima. Too much exploration wastes time on suboptimal actions. This balance is one of RL's core unsolved challenges.

• ε-greedy: Take the best-known action with probability 1-ε, and a random action with probability ε; ε is typically annealed from 1.0 to 0.01 over training
• Upper Confidence Bound (UCB): Selects actions based on both estimated value and uncertainty — favoring under-explored actions systematically
• Intrinsic motivation: Auxiliary curiosity rewards for visiting novel states, enabling exploration in sparse-reward environments where external rewards are rare
• Entropy regularization: Used in SAC and PPO, rewards policies for maintaining high action-probability entropy — encouraging diverse behavior without explicit exploration heuristics

Sparse Rewards: The Hard Problem

RL works beautifully when rewards are frequent and informative. It struggles catastrophically when rewards are sparse. A robot learning to solve a Rubik's Cube receives zero reward for the thousands of valid moves that don't complete the puzzle, making it nearly impossible to learn that progress is being made.

Hindsight Experience Replay (HER) addresses this for goal-conditioned tasks by retroactively relabeling failed episodes as successful attempts toward the actual achieved outcome, generating useful learning signal from every experience regardless of whether the original goal was met. OpenAI used this technique, combined with domain randomization and PPO, to train the Dactyl robotic hand to solve a Rubik's Cube in 2019.

Reinforcement learning's unique strength is its capacity to discover strategies that no human expert designed or anticipated. That same openness creates its fundamental risk: agents optimize ruthlessly for the reward signal specified, not the intent behind it. An RL agent rewarded for high game score may discover the one action that freezes the game clock. This reward misspecification problem — where the reward function is a flawed proxy for true desired behavior — is one of the central research problems in AI safety.