What Is Model Compression?

Model compression is the set of algorithm design techniques used to reduce the size, memory footprint, and computational cost of a trained machine learning model while preserving as much of its predictive accuracy as possible. The motivating observation is that large neural networks are heavily overparameterized: many weights contribute little to the function the network computes, and the numerical precision used during training is far higher than inference requires. Compression exploits that slack, trading a controlled amount of accuracy for reductions in parameter count, arithmetic operations, and energy per inference.

The problem became central as deployment targets diverged from training hardware. A network trained on a cluster of accelerators may need to run on a microcontroller, a phone, or a latency-bounded server tier where memory bandwidth rather than arithmetic throughput is the binding constraint. Compression is therefore evaluated on several axes at once: compression ratio, floating-point operations per inference, measured wall-clock latency on the target device, and the accuracy gap against the uncompressed baseline.

Pruning

Pruning removes parameters or whole structural units from a trained network. Unstructured pruning zeroes individual weights, typically those with the smallest magnitude, producing a sparse weight matrix that shrinks well under compression but needs sparse kernel support to yield an actual speedup. Structured pruning removes entire filters, channels, attention heads, or layers, which leaves a smaller dense network that any runtime can execute faster without special support. A detailed survey of deep neural network pruning organizes the field by what is removed, by the criterion used to score importance, and by whether pruning happens before training, during training, or after convergence, and notes that iterative prune-and-retrain schedules generally recover more accuracy than single-shot removal.

Quantization

Quantization reduces the numerical precision of weights and activations, most commonly from 32-bit floating point to 8-bit integers, and in aggressive settings to 4 bits, ternary values, or single bits. The mapping from real values to integers is defined by a scale and a zero point, which may be shared across a tensor or computed per channel. Post-training quantization applies this mapping to an already trained model using a small calibration set, while quantization-aware training simulates the rounding during the forward pass so the optimizer can compensate. The gain is twofold: an 8-bit tensor occupies a quarter of the memory of a 32-bit one, and integer arithmetic units are cheaper and faster than floating-point units on most inference silicon.

Knowledge Distillation and Low-Rank Factorization

Knowledge distillation trains a small student network to reproduce the output distribution of a larger teacher rather than the hard training labels. The teacher's softened probabilities carry information about class similarity that one-hot labels do not, and matching intermediate feature maps often transfers more than matching outputs alone. Low-rank factorization takes a different route, replacing a weight matrix with the product of two thinner matrices or decomposing a convolution kernel along its tensor modes. These methods compose with the previous two, and pipelines that apply pruning, quantization, and knowledge distillation together generally reach a better accuracy-per-byte point than any single technique, provided the order and the retraining budget are chosen carefully. Work on goal-specific pruning combined with self-distillation shows the same effect when the compression objective is tied to a downstream task rather than to generic accuracy.

Applications

Model compression has applications wherever inference cost matters, including:

• On-device inference for mobile phones, wearables, and hearing aids
• Embedded and microcontroller deployment under tight memory budgets
• Automotive perception stacks with hard latency deadlines
• Large language model serving, where memory bandwidth limits throughput
• Federated learning, where model updates must cross constrained network links
• Energy-constrained sensing platforms and remote instrumentation