Statistical comparison of image fusion algorithms: Recommendations

Abstract Pixel-level image fusion has been applied in a variety of applications, including multi-modal medical imaging, remote sensing, industrial inspection, video surveillance, and night vision etc. Various algorithms are being proposed for numerous applications which requires a comprehensive method of assessment to discern which methods provide decision support. Currently, the validation or assessment of newly proposed algorithms is done either subjectively or objectively. A subjective assessment is costly and affected by a number of factors that are difficult to control. On the other hand, an objective assessment is carried out with a fusion performance metric which is defined to evaluate the effectiveness and/or efficiency of the fusion operation. There are a number of fusion metrics proposed for fusion processes taking different perspectives. Most image fusion research presents a comparison of the proposed and existing fusion algorithms with selected fusion metric(s) over multiple image data sets. The proposed algorithm advantage is justified by the relative difference with the best or better metric values. However, the statistical significance of such difference is unknown leading to a misperception of the quantitative differences between methods. This paper proposes the use of non-parametric statistical analysis for comparisons of fusion algorithms along with the Image fusion Toolbox Employing Significance Testing (ImTEST). Strategies to use different tests in varied scenarios are presented and recommended. Experiments with recently published algorithms demonstrate the necessity to adopt the statistical comparison to establish a baseline for image fusion research.

• Publication year

  2017

• Venue

  Information Fusion

• Publication date

  2017-07-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1016/j.inffus.2016.12.007
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems

  2016cited by this paper
• STAC: A web platform for the comparison of algorithms using statistical tests

  2015cited by this paper
• A general framework for image fusion based on multi-scale transform and sparse representation

  2015cited by this paper
• Multi-focus image fusion using dictionary-based sparse representation

  2015cited by this paper
• Statistical analysis of the performance assessment results for pixel-level image fusion

  2014cited by this paper
• Image matting for fusion of multi-focus images in dynamic scenes

  2013cited by this paper
• Image Fusion With Guided Filtering

  2013cited by this paper
• Image fusion based on pixel significance using cross bilateral filter

  2013cited by this paper
• Two-Stage Fuzzy Fusion With Applications to Through-the-Wall Radar Imaging

  2013cited by this paper
• URREF reliability versus credibility in information fusion (STANAG 2511)

  2013cited by this paper
• A new image fusion performance metric based on visual information fidelity

  2013cited by this paper
• Qualitative and quantitative comparisons of multispectral night vision colorization techniques

  2012cited by this paper
• Image fusion based on image decomposition using self-fractional Fourier functions

  2012cited by this paper
• Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure

  2012cited by this paper
• Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Study

  2012cited by this paper
• Intelligent financial time series forecasting: A complex neuro-fuzzy approach with multi-swarm intelligence

  2012cited by this paper
• Image Fusion Using Higher Order Singular Value Decomposition

  2012cited by this paper
• Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform

  2012cited by this paper
• Frame Fundamental High-Resolution Image Fusion From Inhomogeneous Measurements

  2012cited by this paper
• Efficient fusion for infrared and visible images based on compressive sensing principle

  2011cited by this paper
• Generalized Random Walks for Fusion of Multi-Exposure Images

  2011cited by this paper
• A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms

  2011cited by this paper
• Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power

  2010cited by this paper
• Towards cognitive image fusion

  2010cited by this paper
• One color contrast enhanced infrared and visible image fusion method

  2010cited by this paper
• Introductory Statistics with R

  2009cited by this paper
• A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability

  2009cited by this paper
• Multifocus image fusion using the nonsubsampled contourlet transform

  2009cited by this paper
• A Total Variation-Based Algorithm for Pixel-Level Image Fusion

  2009cited by this paper
• A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization

  2009cited by this paper
• A Similarity Metric for Assessment of Image Fusion Algorithms

  2008cited by this paper
• Comments on 'Information measure for performance of image fusion'

  2008cited by this paper
• Variational Models for Fusion and Denoising of Multifocus Images

  2008cited by this paper
• KEEL: a software tool to assess evolutionary algorithms for data mining problems

  2008cited by this paper
• A novel similarity based quality metric for image fusion

  2008cited by this paper
• Sensor Management Fusion Using Operating Conditions

  2008cited by this paper
• Adaptive region-based image fusion using energy evaluation model for fusion decision

  2007cited by this paper
• Extending the Depth of Field in Microscopy Through Curvelet-Based Frequency-Adaptive Image Fusion

  2007cited by this paper
• PERFORMANCE ASSESSMENT OF COMBINATIVE PIXEL-LEVEL IMAGE FUSION BASED ON AN ABSOLUTE FEATURE MEASUREMENT

  2007cited by this paper
• A study on the use of statistical tests for experimentation with neural networks: Analysis of parametric test conditions and non-parametric tests

  2007cited by this paper
• Novel Cooperative Neural Fusion Algorithms for Image Restoration and Image Fusion

  2007cited by this paper
• Total variation based image denoising and restoration

  2006cited by this paper
• Fusion of Visible and Infrared Images using Empirical Mode Decomposition to Improve Face Recognition

  2006cited by this paper
• Methods for the assessment of fused images

  2006cited by this paper
• Statistical Comparisons of Classifiers over Multiple Data Sets

  2006cited by this paper
• Image fusion metric based on mutual information and Tsallis entropy

  2006cited by this paper
• Recent Developments in Total Variation Image Restoration

  2004cited by this paper
• Concealed weapon detection using color image fusion

  2003cited by this paper
• Multifocus image fusion using artificial neural networks

  2002cited by this paper
• Objective pixel-level image fusion performance measure

  2000cited by this paper
• Objective image fusion performance measure

  2000cited by this paper
• Handbook of Parametric and Nonparametric Statistical Procedures

  1997cited by this paper
• Image sequence fusion using a shift-invariant wavelet transform

  1997cited by this paper
• New false color mapping for image fusion

  1996cited by this paper
• Exact Maximum A Posteriori Estimation for Binary Images

  1989cited by this paper
• The Laplacian Pyramid as a Compact Image Code

  1983cited by this paper
• Approximations of the critical region of the fbietkan statistic

  1980cited by this paper
• International Journal of Computer Mathematics

  year unknowncited by this paper

• Towards Adulteration Detection in Medicinal Leaves Using an Effective Feature Extractor

  2026cites this paper
• Notices-Driven Situation Awareness for Space and Air Traffic Management

  2025cites this paper
• Situation Awareness Using Fuzzy Analytical Hierarchy Processing

  2025cites this paper
• MHDBN: Mamba-based hybrid dual-branch network for multi-focus image fusion

  2025cites this paper
• Space and Air Traffic Management Situation Awareness with Notices

  2025cites this paper
• Similarity measure for intuitionistic fuzzy sets and its applications in pattern recognition and multimodal medical image fusion

  2025cites this paper
• Self-Adaptive Multi-Population Quadratic Approximation Guided Jaya Optimization Applied to Economic Load Dispatch Problems With or Without Valve-point Effects

  2025cites this paper
• Advancing Capabilities With UAVs: A Comprehensive Review of Sensor Fusion, Path Planning, Energy Management, and Deep Learning Algorithms

  2025cites this paper
• Parallel Adaptive Stochastic Gradient Descent Algorithms for Latent Factor Analysis of High-Dimensional and Incomplete Industrial Data

  2024cites this paper
• Dynamic Digital Twins for Situation Awareness

  2024cites this paper
• Optimization of image recognition and gamification training process for entertainment robots in basketball training games based on tracking technology

  2024cites this paper
• Artificial intelligence based virtual gaming experience for sports training and simulation of human motion trajectory capture

  2024cites this paper
• GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

  2024cites this paper
• Study of the Value of π Probability Sampling by Testing Hypothesis and Experimentally

  2024cites this paper
• An Efficient Network Model for Visible and Infrared Image Fusion

  2023cites this paper
• An Integrated Statistical Process Monitoring and Fuzzy Transformation Approach to Improve Process Performance via Image Data

  2023cites this paper
• A Data-Characteristic-Aware Latent Factor Model for Web Services QoS Prediction

  2022cites this paper
• A hybrid Modified Black Widow Optimization and PSO Algorithm: Application in Feature Selection for Cognitive Radio Networks

  2022cites this paper
• Advances in deep learning for infrared image processing and exploitation

  2022cites this paper
• Data fusion information group (DFIG) model meets AI+ML

  2022cites this paper
• Joint data learning panel summary

  2022cites this paper
• T-Friedman Test: A New Statistical Test for Multiple Comparison with an Adjustable Conservativeness Measure

  2022cites this paper
• A composite framework of deep multiple view human joints feature extraction and selection strategy with hybrid adaptive sunflower optimization‐whale optimization algorithm for human action recognition in video sequences

  2022cites this paper
• A Novel Adaptive Feature Fusion Strategy for Image Retrieval

  2021cites this paper
• Use of the URREF towards Information Fusion Accountability Evaluation

  2021cites this paper
• Scalable Information Fusion Trust

  2021cites this paper
• An Analysis of Image Features Extracted by CNNs to Design Classification Models for COVID-19 and Non-COVID-19

  2021cites this paper
• Certifiable Artificial Intelligence Through Data Fusion

  2021cites this paper
• Linearly decreasing inertia weight PSO and improved weight factor-based clustering algorithm for wireless sensor networks

  2021cites this paper
• An enhanced approach to optimally place the solar powered electric vehicle charging station in distribution network

  2021cites this paper
• Improved Salp Swarm Optimization Algorithm: Application in Feature Weighting for Blind Modulation Identification

  2021cites this paper
• SEDRFuse: A Symmetric Encoder–Decoder With Residual Block Network for Infrared and Visible Image Fusion

  2021cites this paper
• Placement of Public Fast-Charging Station and Solar Distributed Generation with Battery Energy Storage in Distribution Network Considering Uncertainties and Traffic Congestion

  2021cites this paper
• Allocation of EV Public Charging Station in Renewable based Distribution Network using HHO Considering Uncertainties and Traffic Congestion

  2021influential citation
• Multimodal Data Fusion Using Canonical Variates Analysis Confusion Matrix Fusion

  2021cites this paper
• Multisource AI scorecard table analysis of AMIGO

  2021cites this paper
• Evaluation of weighted fusion for scalar images in multi-sensor network

  2021cites this paper
• An adaptive two-scale biomedical image fusion method with statistical comparisons

  2020cites this paper
• Electromagnetic Induction Heating and Image Fusion of Silicon Photovoltaic Cell Electrothermography and Electroluminescence

  2020cites this paper
• Comprehensive measure for evaluating image fusion algorithm

  2020cites this paper
• A novel medical image fusion method based on Rolling Guidance Filtering

  2020cites this paper
• Data fusion methods for materials awareness

  2020cites this paper
• Image fusion for context-aided automatic target recognition

  2020cites this paper
• Quality Assessment by Region and Land Cover of Sharpening Approaches Applied to GF-2 Imagery

  2020cites this paper
• Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion

  2020cites this paper
• Multi-modal Video Fusion for Context-aided Tracking

  2020cites this paper
• A novel 3D medical image super-resolution method based on densely connected network

  2020cites this paper
• Improved Whale optimization Algorithm for SVM Model Selection: Application in Medical Diagnosis

  2020cites this paper
• Integrating Biodiversity, Remote Sensing, and Auxiliary Information for the Study of Ecosystem Functioning and Conservation at Large Spatial Scales

  2020cites this paper
• Multimodal Neuroimaging Game Theoretic Data Fusion in Adversarial Conditions

  2020cites this paper
• A Contrast Adjusted Wavelet and Curvelet Transform Based Fusion of Breast Images

  2020cites this paper
• Autonomy in use for space situation awareness

  2019cites this paper
• Multi-scale fidelity measure for image fusion quality assessment

  2019cites this paper
• Multispectral and hyperspectral image fusion with spatial-spectral sparse representation

  2019cites this paper
• An effective multifocus image fusion method using guided filter

  2019cites this paper
• A privacy-preserving image retrieval method based on deep learning and adaptive weighted fusion

  2019cites this paper
• Fusion of Medical Sensors Using Adaptive Cloud Model in Local Laplacian Pyramid Domain

  2019cites this paper
• Plantar pressure image fusion for comfort fusion in diabetes mellitus using an improved fuzzy hidden Markov model

  2019cites this paper
• Multi-focus Image Fusion Combined with CNN and Algebraic Multi-grid Method

  2019cites this paper
• Research into the Adaptability Evaluation of the Remote Sensing Image Fusion Method Based on Nearest-Neighbor Diffusion Pan Sharpening

  2019cites this paper
• Infrared and Visible Image Fusion Objective Evaluation Method

  2019cites this paper
• The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior

  2019cites this paper
• Automatic machine learning for target recognition

  2019cites this paper
• Dynamic data driven analytics for multi-domain environments

  2019cites this paper
• Multisource deep learning for situation awareness

  2019cites this paper
• Quality Evaluation of Tank Shooter Night Vision Image Fusion Based on Mark Watershed

  2019cites this paper
• Fusion of anatomical and functional images using parallel saliency features

  2018cites this paper
• Multisensor Image Fusion and Enhancement in Spectral Total Variation Domain

  2018cites this paper
• Data-driven predictive analytics for water infrastructure condition assessment and management

  2018cites this paper
• Data fusion in data scarce areas using a back-propagation artificial neural network model: a case study of the South China Sea

  2018cites this paper
• A novel chaotic Jaya algorithm for unconstrained numerical optimization

  2018cites this paper
• Medical image super-resolution via minimum error regression model selection using random forest

  2018cites this paper
• Multispectral Image Fusion and Colorization

  2018cites this paper
• Efficient Misalignment-Robust Multi-Focus Microscopical Images Fusion

  2018cites this paper
• Autonomy in Use for Information Fusion Systems

  2018cites this paper
• Design of a Dynamic Data-Driven System for Multispectral Video Processing

  2018cites this paper
• Handbook of Dynamic Data Driven Applications Systems

  2018cites this paper
• Comparison and Analysis of Image Fusion Algorithms for GF-2 Satellite

  2018cites this paper
• Algebraic Multi-Grid Based Multi-Focus Image Fusion Using Watershed Algorithm

  2018influential citation
• Prediction of pipe performance with stacking ensemble learning based approaches

  2018cites this paper
• Medical images fusion by using weighted least squares filter and sparse representation

  2018cites this paper
• Deep learning for pixel-level image fusion: Recent advances and future prospects

  2018cites this paper
• Fusing synergistic information from multi-sensor images: An overview from implementation to performance assessment

  2018cites this paper
• Scale-Invariant Structure Saliency Selection for Fast Image Fusion

  2018cites this paper
• Research on Intelligent Recognition of Train Image Based on Machine Vision

  2018cites this paper
• An Adaptive Weight Method for Image Retrieval Based Multi-Feature Fusion

  2018cites this paper
• Physics-based and human-derived information fusion for analysts

  2017cites this paper
• Image Fusion-Based Land Cover Change Detection Using Multi-Temporal High-Resolution Satellite Images

  2017cites this paper
• An ensemble method for multiple source images fusion in smart city

  2017cites this paper
• Three-dimensional surface microtopography recovery from a multifocus image sequence using an omnidirectional modified Laplacian operator with adaptive window size.

  2017cites this paper
• The Study of the Value of π Probability Sampling by Testing Hypothesis and Experimentally

  year unknowncites this paper