Supervised Evaluation of Image Segmentation and Object Proposal Techniques

This paper tackles the supervised evaluation of image segmentation and object proposal algorithms. It surveys, structures, and deduplicates the measures used to compare both segmentation results and object proposals with a ground truth database; and proposes a new measure: the precision-recall for objects and parts. To compare the quality of these measures, eight state-of-the-art object proposal techniques are analyzed and two quantitative meta-measures involving nine state of the art segmentation methods are presented. The meta-measures consist in assuming some plausible hypotheses about the results and assessing how well each measure reflects these hypotheses. As a conclusion of the performed experiments, this paper proposes the tandem of precision-recall curves for boundaries and for objects-and-parts as the tool of choice for the supervised evaluation of image segmentation. We make the datasets and code of all the measures publicly available.

• Publication year

  2016

• Venue

  IEEE Transactions on Pattern Analysis and Machine Intelligence

• Publication date

  2016-07-01

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1109/TPAMI.2015.2481406PMID 26415155
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

  2015cited by this paper
• Generating Object Segmentation Proposals Using Global and Local Search

  2014cited by this paper
• RIGOR: Reusing Inference in Graph Cuts for Generating Object Regions

  2014cited by this paper
• Category-Independent Object Proposals with Diverse Ranking

  2014cited by this paper
• Multiscale Combinatorial Grouping

  2014influential reference
• Image segmentation evaluation and its application to object detection

  2014cited by this paper
• How to Evaluate Foreground Maps

  2014cited by this paper
• Simultaneous Detection and Segmentation

  2014cited by this paper
• Geodesic Object Proposals

  2014cited by this paper
• Selective Search for Object Recognition

  2013cited by this paper
• Measures and Meta-Measures for the Supervised Evaluation of Image Segmentation

  2013cited by this paper
• Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

  2013cited by this paper
• Image Segmentation by Cascaded Region Agglomeration

  2013cited by this paper
• SLIC Superpixels Compared to State-of-the-Art Superpixel Methods

  2012cited by this paper
• CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts

  2012cited by this paper
• Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration

  2012cited by this paper
• Diagnosing Error in Object Detectors

  2012cited by this paper
• Supervised Assessment of Segmentation Hierarchies

  2012cited by this paper
• Semantic segmentation using regions and parts

  2012cited by this paper
• Contour Detection and Hierarchical Image Segmentation

  2011influential reference
• Image segmentation by figure-ground composition into maximal cliques

  2011cited by this paper
• Left Ventricular Segmentation Challenge from Cardiac MRI: A Collation Study

  2011cited by this paper
• Measuring the Objectness of Image Windows

  2011cited by this paper
• SLIC Superpixels ?

  2010cited by this paper
• On Parameter Learning in CRF-Based Approaches to Object Class Image Segmentation

  2010cited by this paper
• A comparative evaluation of interactive segmentation algorithms

  2010cited by this paper
• Optimal Contour Closure by Superpixel Grouping

  2010cited by this paper
• Region Merging Techniques Using Information Theory Statistical Measures

  2010cited by this paper
• Object recognition as ranking holistic figure-ground hypotheses

  2010cited by this paper
• Recognition using regions

  2009cited by this paper
• Benchmarking Image Segmentation Algorithms

  2009cited by this paper
• Binary Partition Trees for Object Detection

  2008cited by this paper
• Beyond sliding windows: Object localization by efficient subwindow search

  2008cited by this paper
• Improving Spatial Support for Objects via Multiple Segmentations

  2007cited by this paper
• Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration

  2007cited by this paper
• Toward Objective Evaluation of Image Segmentation Algorithms

  2007cited by this paper
• New benchmark for image segmentation evaluation

  2007cited by this paper
• Distance Measures for Image Segmentation Evaluation

  2006cited by this paper
• The PASCAL Visual Object Classes Challenge

  2006cited by this paper
• Using Multiple Segmentations to Discover Objects and their Extent in Image Collections

  2006cited by this paper
• Image-Segmentation Evaluation From the Perspective of Salient Object Extraction

  2006cited by this paper
• Evaluating Multi-Object Tracking

  2005cited by this paper
• Toward a generic evaluation of image segmentation

  2005cited by this paper
• Comparing clusterings: an axiomatic view

  2005cited by this paper
• Efficient Graph-Based Image Segmentation

  2004cited by this paper
• Statistical validation of image segmentation quality based on a spatial overlap index.

  2004cited by this paper
• Evaluating Image Segmentation Algorithms Using the Pareto Front

  2002cited by this paper
• An empirical approach to grouping and segmentation

  2002cited by this paper
• Shape Matching and Object Recognition Using Shape Contexts

  2002cited by this paper
• Mean Shift: A Robust Approach Toward Feature Space Analysis

  2002cited by this paper
• A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

  2001cited by this paper
• Shape matching: similarity measures and algorithms

  2001cited by this paper
• Assignment problem in edge detection performance evaluation

  2000cited by this paper
• Performance criteria for graph clustering and Markov cluster experiments

  2000cited by this paper
• Normalized cuts and image segmentation

  1997cited by this paper
• An Experimental Comparison of Range Image Segmentation Algorithms

  1996cited by this paper
• Geodesic Saliency of Watershed Contours and Hierarchical Segmentation

  1996cited by this paper
• Quantitative methods of evaluating image segmentation

  1995cited by this paper
• A fast algorithm for MDL-based multi-band image segmentation

  1994cited by this paper
• A method of establishing group of equal amplitude in plant sociobiology based on similarity of species content and its application to analyses of the vegetation on Danish commons

  1948cited by this paper
• Measures of the Amount of Ecologic Association Between Species

  1945cited by this paper
• Zarys metod statystycznych : w zastosowaniu do antropologii

  year unknowncited by this paper
• Etude comparative de la distribution florale dans une portion des Alpes et des Jura

  year unknowncited by this paper

• Towards an Agnostic Superpixel Segmentation Framework

  2025cites this paper
• E-MSLDNet: a dual-enhanced network with edge-aware and multiscale feature fusion for precision landslide detection in complex terrain

  2025cites this paper
• DualSight: multi-stage instance segmentation framework for improved precision

  2025cites this paper
• Longest similar subsequence-modified Hausdorff distance: A new robust approach for measuring distances between linear features

  2025cites this paper
• Zero-shot image segmentation for scene objects based on the L0 gradient minimization and adaptive superpixel method

  2025cites this paper
• 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding

  2025cites this paper
• Image Segmentation Techniques: An In-Depth Review and Analysis

  2024cites this paper
• A data-driven active learning approach to reusing ML solutions in scientific applications

  2024cites this paper
• Video Object Segmentation

  2024cites this paper
• S2FB IoU: Improving Boundary-based Object-Centric Image Segmentation Quality Evaluation

  2024cites this paper
• A Review on Covid-19 Detection Using Artificial Intelligence from Chest CT Scan Slices

  2024cites this paper
• Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning

  2024cites this paper
• Measuring Hierarchiness of Image Segmentations

  2024cites this paper
• A novel framework for concealed weapons detection using passive terahertz imaging

  2024cites this paper
• Exploiting multi-scale contextual prompt learning for zero-shot semantic segmentation

  2023cites this paper
• Gumbel (EVI)-Based Minimum Cross-Entropy Thresholding for the Segmentation of Images with Skewed Histograms

  2023cites this paper
• Pixel Difference Unmixing Feature Networks for Edge Detection

  2023cites this paper
• A fuzzy approach to evaluate image segmentation based on image complexity

  2023influential citation
• Saliency and superpixel improved detection and segmentation of concealed objects for passive terahertz images

  2023cites this paper
• An automatic quality evaluator for video object segmentation masks

  2022cites this paper
• Otsu Thresholding Model Using Heterogeneous Mean Filters for Precise Images Segmentation

  2022cites this paper
• Easing the Reuse of ML Solutions by Interactive Clustering-based Autotuning in Scientific Applications

  2022cites this paper
• Automatic generation of areas of interest using multimodal geospatial data from an on-demand food delivery platform (industrial paper)

  2022cites this paper
• Learnable Pixel Clustering Via Structure and Semantic Dual Constraints for Unsupervised Image Segmentation

  2022cites this paper
• Toward Field Soil Surveys: Identifying and Delineating Soil Diagnostic Horizons Based on Deep Learning and RGB Image

  2022cites this paper
• From colouring-in to pointillism: revisiting semantic segmentation supervision

  2022cites this paper
• Improvement in the Between-Class Variance Based on Lognormal Distribution for Accurate Image Segmentation

  2022cites this paper
• MMSHRs: a morphology model of suspicious hyperthermic regions for degree of severity prediction from breast thermograms

  2022influential citation
• Assessing Hierarchies by Their Consistent Segmentations

  2022cites this paper
• Auto-CSC: A Transfer Learning Based Automatic Cell Segmentation and Count Framework

  2022cites this paper
• Improving Minimum Cross-Entropy Thresholding for Segmentation of Infected Foregrounds in Medical Images Based on Mean Filters Approaches

  2022cites this paper
• Machine Learning and Deep Learning Techniques for Optic Disc and Cup Segmentation – A Review

  2022cites this paper
• Objective Object Segmentation Visual Quality Evaluation: Quality Measure and Pooling Method

  2022cites this paper
• A bio-inspired contour detection model using multiple cues inhibition in primary visual cortex

  2022cites this paper
• A Boosted Minimum Cross Entropy Thresholding for Medical Images Segmentation Based on Heterogeneous Mean Filters Approaches

  2022cites this paper
• Bidirectional Multiscale Refinement Network for Crisp Edge Detection

  2022cites this paper
• Towards a dynamic soil survey: Identifying and delineating soil horizons in-situ using deep learning

  2021cites this paper
• Region Quality Based Scale-aware Selection for Hierarchical Image Segmentation

  2021cites this paper
• Revisiting the Coco Panoptic Metric to Enable Visual and Qualitative Analysis of Historical Map Instance Segmentation

  2021cites this paper
• Cloud detection of space-borne video remote sensing using improved Unet method

  2021cites this paper
• Iterated Watersheds, A Connected Variation of K-Means for Clustering GIS Data

  2021cites this paper
• Superpixel-based object boundary gimmicking using optimized conditional random fields with random associations

  2021cites this paper
• Supervised quality evaluation of binary partition trees for object segmentation

  2021cites this paper
• Texture Segmentation Benchmark

  2021cites this paper
• Decoupling Zero-Shot Semantic Segmentation

  2021cites this paper
• Near-Infrared Elbow Vein Extraction Algorithm Based on Multiscale Adaptive Filter

  2021cites this paper
• Task Switching Network for Multi-task Learning

  2021cites this paper
• A tutorial on applications of power watershed optimization to image processing

  2021cites this paper
• Deep Superpixel Cut for Unsupervised Image Segmentation

  2021influential citation
• A Graph-Based Method for Joint Instance Segmentation of Point Clouds and Image Sequences

  2021cites this paper
• Learning Crisp Boundaries Using Deep Refinement Network and Adaptive Weighting Loss

  2021cites this paper
• Region Detection Rate: An Applied Measure for Surface Defect Localization

  2021cites this paper
• Object scale selection of hierarchical image segmentation with deep seeds

  2020influential citation
• Power Spectral Clustering

  2020cites this paper
• Evaluation of the Sample Clustering Process on Graphs

  2020cites this paper
• Gradient-driven unsupervised video segmentation using deep learning techniques

  2020cites this paper
• Multi-Label Classification Scheme Based on Local Regression for Retinal Vessel Segmentation

  2020cites this paper
• Particulate Matter Estimation Based on Haze Degree Factor

  2020cites this paper
• Inception U-Net Architecture for Semantic Segmentation to Identify Nuclei in Microscopy Cell Images

  2020cites this paper
• Super-BPD: Super Boundary-to-Pixel Direction for Fast Image Segmentation

  2020influential citation
• Image segmentation evaluation: a survey of methods

  2020cites this paper
• Fault diagnosis method of mechanical power system based on image processing technology

  2020cites this paper
• Learning to realign hierarchy for image segmentation

  2020cites this paper
• Indoor scene segmentation algorithm based on full convolutional neural network

  2020cites this paper
• A vertical profile imaging method for quantifying rock fragments in gravelly soil

  2020cites this paper
• Detection of brain tumour in 2D MRI: implementation and critical review of clustering-based image segmentation methods

  2020cites this paper
• Enhancing Generic Segmentation with Learned Region Representations - Appendices

  2020cites this paper
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference

  2020cites this paper
• Computer Medical Image Segmentation Based on Neural Network

  2020cites this paper
• Accurate road segmentation in remote sensing images using dense residual learning and improved focal loss

  2020cites this paper
• Optimising performance in clinical capsule endoscopy

  2020cites this paper
• DoMars16k: A Diverse Dataset for Weakly Supervised Geomorphologic Analysis on Mars

  2020cites this paper
• Boundaries and Region Representations Fusion

  2020cites this paper
• Self-propagating video segmentation using patch matching and enhanced Onecut

  2020cites this paper
• Efficient Human Detection Algorithm using Color & Depth information with Accurate Outer Boundary Matching

  2019cites this paper
• Delineation of agricultural fields in smallholder farms from satellite images using fully convolutional networks and combinatorial grouping

  2019cites this paper
• M2U-Net: Effective and Efficient Retinal Vessel Segmentation for Real-World Applications

  2019cites this paper
• Fast fully automatic heart fat segmentation in computed tomography datasets

  2019cites this paper
• Efficient Algorithms for Hierarchical Graph-Based Segmentation Relying on the Felzenszwalb-Huttenlocher Dissimilarity

  2019cites this paper
• Enhancing Generic Segmentation With Learned Region Representations

  2019cites this paper
• Revisiting the Isoperimetric Graph Partitioning Problem

  2019cites this paper
• Large-Scale Interactive Object Segmentation With Human Annotators

  2019cites this paper
• Segmentation of 3D FIB-SEM data with pore-back effect

  2019cites this paper
• Evaluation of machine learning applied to the realignment of hierarchies for image segmentation

  2019cites this paper
• Detecting salient objects in non-stationary video image sequence for analyzing user perceptions of digital video contents

  2019cites this paper
• Object Scale Selection of Hierarchical Image Segmentation Using Reliable Regions

  2019cites this paper
• Segmentation and Classification of Multimodal Imagery

  2019cites this paper
• Boundary Matched Human Area Segmentation for Chroma Keying Using Hybrid Depth-Color Analysis

  2019cites this paper
• A study of hierarchical watersheds on graphs with applications to image segmentation. (Une étude sur les hiérarchies de ligne de partage des eaux et ses applications dans la segmentation d'image)

  2019cites this paper
• On the Evaluation and Real-World Usage Scenarios of Deep Vessel Segmentation for Retinography.

  2019cites this paper
• Image correlation method to simulate physical characteristic of particulate matter

  2019cites this paper
• Erratum to “Hierarchizing graph-based image segmentation algorithms relying on region dissimilarity: the case of the Felzenszwalb-Huttenlocher method”

  2019cites this paper
• On the Evaluation and Real-World Usage Scenarios of Deep Vessel Segmentation for Funduscopy

  2019cites this paper
• Fully Automated Segmentation of Lower Extremity Deep Vein Thrombosis Using Convolutional Neural Network

  2019cites this paper
• Human Detection from RGB Depth Image using Active contour and Grow-cut Segmentation

  2019cites this paper
• Color Image Segmentation Using a Fuzzy Inference System

  2019cites this paper
• Hierarchical Segmentation Evaluation of Region-Based Image Hierarchy

  2019cites this paper
• A New Deep Segmentation Quality Assessment Network for Refining Bounding Box Based Segmentation

  2019cites this paper
• Construction of 3D Digital Model of a Rock Sample Based on FIB-SEM Data

  2019cites this paper
• Plant segmentation for growth analysis in temporal datasets

  2018cites this paper