Cross-Modal Complementarity Learning for Fish Feeding Intensity Recognition via Audio–Visual Fusion

Simple Summary This study presents a novel cross-modal fusion method for accurate recognition of fish feeding intensity in complex underwater environments. Using acoustic and visual data from hydrophones and cameras, we develop a two-stage attention mechanism that adaptively combines complementary information from both modalities to overcome the limitations of single-modal approaches. The first stage enhances individual modal representations through cross-modal interactions, while the second stage dynamically adjusts fusion weights based on environmental conditions and modal reliability. Experimental results demonstrate that our method significantly outperforms existing single-modal and conventional fusion approaches, achieving superior accuracy in underwater scenarios. This work provides robust technical support for intelligent aquaculture monitoring and precision fish farming management.

• Publication year

  2025

• Venue

  Animals

• Publication date

  2025-07-31

• Fields of study

  Medicine, Computer Science, Environmental Science

• Identifiers
  DOI 10.3390/ani15152245PMID 40805035PMCID 12345475
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• MMFINet: A multimodal fusion network for accurate fish feeding intensity assessment in recirculating aquaculture systems

  2025cited by this paper
• Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture

  2025cited by this paper
• Fish Tracking, Counting, and Behaviour Analysis in Digital Aquaculture: A Comprehensive Survey

  2024cited by this paper
• Augmenting Aquaculture Efficiency through Involutional Neural Networks and Self-Attention for Oplegnathus Punctatus Feeding Intensity Classification from Log Mel Spectrograms

  2024cited by this paper
• Investigating the detection of breast cancer with deep transfer learning using ResNet18 and ResNet34

  2024cited by this paper
• A review of aquaculture: From single modality analysis to multimodality fusion

  2024cited by this paper
• Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity

  2024cited by this paper
• Feeding control and water quality monitoring on bioenergetic fish growth modeling: Opportunities and challenges

  2024cited by this paper
• Recent advances in acoustic technology for aquaculture: A review

  2023cited by this paper
• Intelligent fish feeding based on machine vision: A review

  2023cited by this paper
• Detecting the Absence of Lung Sliding in Ultrasound Videos Using 3D Convolutional Neural Networks

  2023cited by this paper
• MobileNetV1-Based Deep Learning Model for Accurate Brain Tumor Classification

  2023cited by this paper
• I3D: Transformer Architectures with Input-Dependent Dynamic Depth for Speech Recognition

  2023cited by this paper
• A review of conservation status of freshwater fish diversity in China.

  2023cited by this paper
• Role of artificial intelligence (AI) in fish growth and health status monitoring: a review on sustainable aquaculture

  2023cited by this paper
• Multimodal Fish Feeding Intensity Assessment in Aquaculture

  2023influential reference
• Feeding intensity assessment of aquaculture fish using Mel Spectrogram and deep learning algorithms

  2023cited by this paper
• Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition

  2022cited by this paper
• Dynamic feeding method for aquaculture fish using multi-task neural network

  2022cited by this paper
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

  2022cited by this paper
• Multi-Class Gastrointestinal Images Classification Using EfficientNet-B0 CNN Model

  2022cited by this paper
• A dual attention network based on efficientNet-B2 for short-term fish school feeding behavior analysis in aquaculture

  2021cited by this paper
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

  2021cited by this paper
• A novel method for peanut variety identification and classification by Improved VGG16

  2021cited by this paper
• Application of computer vision in fish intelligent feeding system—A review

  2020cited by this paper
• Computer Vision Models in Intelligent Aquaculture with Emphasis on Fish Detection and Behavior Analysis: A Review

  2020cited by this paper
• Automatic recognition methods of fish feeding behavior in aquaculture: A review

  2020cited by this paper
• Detecting Affect States Using VGG16, ResNet50 and SE-ResNet50 Networks

  2020cited by this paper
• Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision

  2019cited by this paper
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

  2019cited by this paper
• SlowFast Networks for Video Recognition

  2018cited by this paper
• Audio Set: An ontology and human-labeled dataset for audio events

  2017cited by this paper
• Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

  2017cited by this paper
• Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

  2017cited by this paper
• Video-based emotion recognition using CNN-RNN and C3D hybrid networks

  2016cited by this paper
• Acoustic noise reduces foraging success in two sympatric fish species via different mechanisms

  2014cited by this paper
• A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach

  2006cited by this paper

• Optimized aquaculture feeding through matched-filter audio signal processing and machine learning

  2026cites this paper
• A Review of Artificial Intelligence-Driven Smart Treatment of Aquaculture Effluent: Technical Framework, Application Scenarios, and Development Outlook

  2026influential citation
• From Traditional Machine Learning Models to Multimodal Large Models: A Review of Aquaculture

  2025cites this paper
• FishSegNet-PRL: A Lightweight Model for High-Precision Fish Instance Segmentation and Feeding Intensity Quantification

  2025cites this paper