What Are Conditional Random Fields?

Conditional random fields, commonly abbreviated CRFs, are undirected probabilistic graphical models that define the conditional distribution of a set of output labels given an observed input sequence or structure. Rather than modeling how the observations were generated, a CRF models only what matters for prediction: the probability of a label assignment given the data. This discriminative framing lets the model use arbitrary, overlapping, and interdependent features of the input without needing to account for the statistical dependencies among those features.

The formulation was introduced by John Lafferty, Andrew McCallum, and Fernando Pereira in a 2001 paper on probabilistic models for segmenting and labeling sequence data. It was motivated by two limitations of earlier sequence models: hidden Markov models require strong independence assumptions about observations, and maximum entropy Markov models suffer from label bias, a failure mode in which states with few outgoing transitions absorb probability mass regardless of the evidence. CRFs avoid both by normalizing over entire label sequences rather than per state.

Linear-Chain CRFs and Sequence Labeling

The most widely used form is the linear-chain CRF, in which output labels form a sequence and each label depends on its immediate neighbors and on the full observation sequence. The model assigns a score to a candidate label sequence as a weighted sum of feature functions defined over label pairs and the input, then converts scores to probabilities with a global normalizing constant known as the partition function. Because the observation is conditioned on rather than generated, a feature can inspect any part of the input: a word's capitalization, its suffix, the identity of tokens several positions away, or the output of an external gazetteer. This flexibility made linear-chain CRFs the standard approach for part-of-speech tagging, named entity recognition, shallow parsing, and gene and protein mention detection through the 2000s and early 2010s.

Inference and Parameter Estimation

Exact inference in a linear chain is tractable through dynamic programming. The forward-backward algorithm computes the marginal probability of each label and the partition function, and the Viterbi algorithm recovers the single highest-scoring label sequence. Training maximizes the conditional log-likelihood of the labeled data, a convex objective that guarantees a single global optimum, usually optimized with limited-memory BFGS or stochastic gradient descent and regularized to prevent weights from growing without bound on rare features. The tutorial introduction to conditional random fields by Charles Sutton and Andrew McCallum sets out these procedures alongside the practical issues that arise at scale, including feature induction, handling very large label sets, and approximate inference when the graph contains cycles and exact computation becomes intractable.

General Graph Structures and Neural CRFs

The chain is a special case. A CRF can be defined over any undirected graph, and skip-chain, tree-structured, and grid-structured variants are used where labels have longer-range or two-dimensional dependencies. Grid CRFs applied to image pixels support semantic segmentation and stereo matching, where inference relies on approximate methods such as loopy belief propagation, graph cuts, or mean-field updates. In current practice the feature functions are frequently supplied by a neural network rather than hand-engineered, an arrangement introduced in work on bidirectional LSTM-CRF models for sequence tagging. The neural component learns representations from raw tokens or pixels, while the CRF layer enforces valid transitions between labels, which matters for tagging schemes where certain label sequences are structurally impossible.

Applications

Conditional random fields have applications in a wide range of fields, including:

• Natural language processing, for named entity recognition and part-of-speech tagging
• Information extraction from documents such as citations, resumes, and clinical notes
• Bioinformatics, for gene prediction and protein secondary structure labeling
• Computer vision, for semantic segmentation and image denoising
• Speech and handwriting recognition, where output symbols form a constrained sequence
• Activity recognition from wearable and ambient sensor streams