Deep reinforcement learning

What Is Deep Reinforcement Learning?

Deep reinforcement learning is the combination of reinforcement learning, which studies how an agent should act in an environment to maximize cumulative reward, with deep neural networks used as function approximators for the quantities that reinforcement learning needs to estimate. The underlying problem is normally formalized as a Markov decision process defined by states, actions, transition dynamics, a reward function, and a discount factor. Classical reinforcement learning stores value estimates in a table indexed by state, which becomes impossible once the state is an image, a sensor stream, or a continuous joint configuration. Deep networks replace that table with a parameterized mapping learned from experience, allowing the same algorithms to operate on raw high-dimensional observations.

The field sits between control theory, dynamic programming, and machine learning. Its distinguishing feature relative to supervised learning is that the training signal is evaluative rather than instructive: the agent is told how good an outcome was, not what the correct action would have been, and it must attribute delayed reward across a sequence of decisions. That credit assignment problem, combined with the need to explore actions whose value is still unknown, drives most of the algorithmic design.

Value-Based Methods

Value-based methods learn an estimate of expected return and derive a policy by acting greedily with respect to it. The deep Q-network introduced in Playing Atari with Deep Reinforcement Learning applied a convolutional network to Q-learning, using an experience replay buffer to decorrelate consecutive samples and a periodically frozen target network to stabilize the bootstrapped regression target. The extended results published as human-level control through deep reinforcement learning showed a single architecture and hyperparameter setting reaching professional human scores across 49 Atari 2600 games from pixel input alone. Later refinements addressed known failure modes: double Q-learning for the overestimation bias in the max operator, dueling architectures separating state value from action advantage, prioritized replay for sampling informative transitions, and distributional formulations that model the full return distribution rather than its mean.

Policy Gradient and Actor-Critic Methods

Value-based methods struggle when the action space is continuous, since the greedy maximization becomes an optimization problem at every step. Policy gradient methods instead parameterize the policy directly and ascend an estimate of the gradient of expected return. Variance in that estimate is the central difficulty, and actor-critic architectures reduce it by pairing the policy with a learned value baseline. Trust region constraints and the clipped surrogate objective of proximal policy optimization limit how far a policy update may move, which keeps training stable enough for large-scale use. Off-policy actor-critic algorithms including deep deterministic policy gradients and soft actor-critic reuse stored transitions and add entropy regularization to maintain exploration.

Search, Models, and Sample Efficiency

Model-free agents can require millions of environment interactions, which is acceptable in simulation and prohibitive on physical hardware. Two responses dominate. Model-based methods learn the transition dynamics and plan within the learned model, amortizing real experience across many imagined rollouts. Search-based methods combine learned value and policy networks with explicit lookahead, the approach used in the AlphaGo system that defeated a professional Go player by guiding Monte Carlo tree search with deep networks trained through self-play. Transfer from simulation to hardware adds domain randomization and system identification to close the reality gap.

Applications

Deep reinforcement learning has applications in a range of fields, including: