What Is Object Segmentation?

Object segmentation is a computer vision task concerned with partitioning an image into regions that correspond to distinct objects or object instances. Unlike object detection, which draws bounding boxes around objects, segmentation produces a pixel-level assignment that delineates the precise boundaries of each region. The field draws on image processing, statistical inference, and deep learning, and has become a foundational capability in vision systems that need fine-grained spatial understanding of a scene.

Semantic Segmentation

Semantic segmentation assigns a class label to every pixel in an image, categorizing regions as belonging to a particular type such as road, building, sky, or pedestrian, without distinguishing individual instances of that type. All pixels belonging to the "car" category receive the same label regardless of how many cars appear in the image. Early approaches relied on Markov random fields and graph cuts to enforce spatial consistency. Convolutional encoder-decoder architectures such as the Fully Convolutional Network (FCN) and U-Net later dominated by learning dense feature maps at multiple resolutions. Dilated convolutions allow networks to capture large receptive fields without loss of spatial resolution, an important property for dense prediction tasks. Semantic segmentation remains the primary approach in applications where category-level region labeling is sufficient, such as medical organ segmentation or terrain classification in satellite imagery.

Instance Segmentation

Instance segmentation distinguishes individual object instances within the same category, producing a separate mask for each one. A scene containing three pedestrians yields three distinct masks, not a single "pedestrian" region. Mask R-CNN, introduced by He et al. in 2017, extended the Faster R-CNN detection framework with a parallel branch that predicts a binary mask for each proposed region of interest. This two-stage approach remains influential, though subsequent architectures have explored single-stage alternatives that regress masks directly without explicit region proposals, reducing latency. The difficulty of instance segmentation lies in separating touching or overlapping objects that share similar appearance, which requires the model to encode identity information beyond category membership. A review of deep learning methods for object detection and segmentation examines how convolutional and transformer-based methods address this challenge.

Panoptic Segmentation

Panoptic segmentation unifies semantic and instance segmentation into a single task by assigning each pixel both a class label and, for countable object instances, an instance identifier. Background regions such as sky or road receive only a class label (called "stuff"), while foreground objects such as people or vehicles receive both a class label and a unique instance ID (called "things"). This formulation was introduced by Kirillov et al. in the 2019 CVPR paper Panoptic Segmentation, which also defined the panoptic quality metric for evaluation. Recent architectures such as Panoptic FPN and Mask2Former use shared backbone features with separate prediction heads for the two subtasks, and transformer-based designs have further improved the handling of complex multi-object scenes. Panoptic segmentation is currently the most information-rich output format available from a single scene understanding model, making it attractive for tasks requiring awareness of both foreground objects and scene context. NCBI's StatPearls collection on visual cortex anatomy documents how the human visual system similarly performs separate processing for object identity and spatial layout, a biological analogy that has informed model design.

Applications

Object segmentation has applications in a wide range of fields, including:

• Autonomous vehicles, for lane detection and pedestrian boundary delineation at the pixel level
• Medical imaging, including tumor boundary delineation and organ segmentation in CT and MRI
• Robotics, enabling grasping planners to determine object shape and contact points
• Satellite and aerial imagery analysis, for land-use classification and damage assessment
• Augmented reality, for precise object cutout and scene composition