What Is Dimensionality Reduction?

Dimensionality reduction is a class of techniques in machine learning and data analysis concerned with transforming high-dimensional data into a lower-dimensional representation while preserving as much relevant structure as possible. As datasets grow in the number of measured features, distances between data points become less meaningful, classification accuracy degrades, and computational cost rises sharply. This phenomenon, often called the curse of dimensionality, motivates the search for compact representations that retain the statistical signal embedded in raw data.

The field draws on linear algebra, information theory, and manifold geometry. Its methods divide broadly into two families: linear projections, which find low-dimensional subspaces, and nonlinear methods, which learn curved manifolds in the original feature space. Both families share the goal of reducing the input dimension while minimizing information loss.

Linear Projection Methods

Linear methods project data onto a subspace defined by a set of orthogonal directions. Principal component analysis (PCA) is the most widely used representative: it finds the directions of maximum variance in the data by computing the eigenvectors of the covariance matrix, then retains only the top-k eigenvectors as the new coordinate axes. Linear discriminant analysis (LDA) takes a supervised variant of this idea, choosing projections that maximize the ratio of between-class to within-class scatter rather than overall variance. Singular value decomposition (SVD) underlies both PCA and a wide range of factor models, making it one of the foundational tools across the discipline. A quantitative survey of dimension reduction techniques published in IEEE Transactions on Visualization and Computer Graphics benchmarks these linear methods against nonlinear alternatives across dozens of datasets.

Nonlinear and Manifold-Based Methods

When data lie on a curved low-dimensional manifold embedded in a high-dimensional space, linear projections discard geometric structure that is essential for downstream tasks. Nonlinear methods address this by preserving local neighborhood relationships or topological properties. Isomap approximates geodesic distances along the manifold surface; locally linear embedding (LLE) reconstructs each point as a weighted combination of its neighbors and seeks a low-dimensional map that preserves those weights. t-distributed stochastic neighbor embedding (t-SNE) converts pairwise distances into probability distributions and minimizes their Kullback-Leibler divergence, making it particularly useful for visualization. Uniform manifold approximation and projection (UMAP) achieves similar visualization quality at lower computational cost and has become the default choice for exploratory analysis of large datasets. A comprehensive analysis of dimensionality reduction techniques on big data at IEEE Access documents the trade-offs in scalability and distortion across these methods.

Feature Selection

Feature selection differs from feature extraction: instead of constructing new coordinates, it selects a subset of the original variables for retention. Filter methods rank features by statistical criteria such as mutual information or correlation with the target label and keep the top scorers. Wrapper methods evaluate candidate subsets by training a classifier and measuring held-out performance. Embedded methods, such as the L1 regularization used in Lasso regression, integrate feature selection directly into the training objective. The arXiv survey on dimensionality reduction techniques by Maaten and colleagues provides a systematic account of how these selection strategies interact with extraction-based approaches in practice. A more recent review in PMC covering dimensionality reduction algorithms and their limitations surveys emerging hybrid methods that combine selection and extraction in a single pipeline.

Applications

Dimensionality reduction has applications in a wide range of fields, including:

• Image classification and computer vision, where pixel features are compressed before training
• Bioinformatics and genomics, where gene expression matrices can have tens of thousands of features
• Natural language processing, including latent semantic analysis of document-term matrices
• Data visualization, for projecting high-dimensional embeddings to two or three dimensions
• Anomaly detection and signal processing, where noise dimensions are removed before analysis