RNN Fisher Vectors for Action Recognition and Image Annotation

Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. We demonstrate that RNNs can be effectively used in order to encode sequences and provide effective representations. The methodology we use is based on Fisher Vectors, where the RNNs are the generative probabilistic models and the partial derivatives are computed using backpropagation. State of the art results are obtained in two central but distant tasks, which both rely on sequences: video action recognition and image annotation. We also show a surprising transfer learning result from the task of image annotation to the task of video action recognition.

• Publication year

  2015

• Venue

  European Conference on Computer Vision

• Publication date

  2015-12-12

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1007/978-3-319-46466-4_50arXiv 1512.03958
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Bilinear CNN Models for Fine-Grained Visual Recognition

  2015cited by this paper
• UvA-DARE ( Digital Academic Repository ) Event Fisher Vectors : Robust Encoding Visual Diversity of Visual Streams

  2015cited by this paper
• Objects2action: Classifying and Localizing Actions without Any Video Example

  2015cited by this paper
• Associating neural word embeddings with deep image representations using Fisher Vectors

  2015cited by this paper
• Multimodal Convolutional Neural Networks for Matching Image and Sentence

  2015cited by this paper
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

  2015cited by this paper
• NetVLAD: CNN Architecture for Weakly Supervised Place Recognition

  2015cited by this paper
• Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

  2015cited by this paper
• Modeling video evolution for action recognition

  2015cited by this paper
• Deep correlation for matching images and text

  2015cited by this paper
• Action recognition with trajectory-pooled deep-convolutional descriptors

  2015cited by this paper
• Skip-Thought Vectors

  2015cited by this paper
• In Defense of Word Embedding for Generic Text Representation

  2015cited by this paper
• Semantic embedding space for zero-shot action recognition

  2015cited by this paper
• Deep Fishing: Gradient Features from Deep Nets

  2015cited by this paper
• Return of the Devil in the Details: Delving Deep into Convolutional Nets

  2014cited by this paper
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

  2014cited by this paper
• Sequence to Sequence Learning with Neural Networks

  2014cited by this paper
• Deep visual-semantic alignments for generating image descriptions

  2014cited by this paper
• Action Recognition with Stacked Fisher Vectors

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• Microsoft COCO: Common Objects in Context

  2014cited by this paper
• Learning Spatiotemporal Features with 3D Convolutional Networks

  2014cited by this paper
• Show and tell: A neural image caption generator

  2014cited by this paper
• Large-Scale Video Classification with Convolutional Neural Networks

  2014cited by this paper
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

  2014influential reference
• Grounded Compositional Semantics for Finding and Describing Images with Sentences

  2014cited by this paper
• Learning a Recurrent Visual Representation for Image Caption Generation

  2014cited by this paper
• From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

  2014influential reference
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

  2014cited by this paper
• Explain Images with Multimodal Recurrent Neural Networks

  2014cited by this paper
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

  2014cited by this paper
• Long-term recurrent convolutional networks for visual recognition and description

  2014cited by this paper
• Beyond Gaussian Pyramid: Multi-skip Feature Stacking for action recognition

  2014cited by this paper
• Neural Machine Translation by Jointly Learning to Align and Translate

  2014cited by this paper
• Two-Stream Convolutional Networks for Action Recognition in Videos

  2014cited by this paper
• Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice

  2014cited by this paper
• Fisher Vector Faces in the Wild

  2013cited by this paper
• Dense Trajectories and Motion Boundary Descriptors for Action Recognition

  2013cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

  2013cited by this paper
• Motion Interchange Patterns for Action Recognition in Unconstrained Videos

  2012cited by this paper
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

  2012influential reference
• A Comparative Study of Encoding, Pooling and Normalization Methods for Action Recognition

  2012cited by this paper
• Action recognition by dense trajectories

  2011cited by this paper
• The devil is in the details: an evaluation of recent feature encoding methods

  2011cited by this paper
• HMDB: A large video database for human motion recognition

  2011cited by this paper
• Large-scale image retrieval with compressed Fisher vectors

  2010cited by this paper
• Convolutional Learning of Spatio-temporal Features

  2010cited by this paper
• Improving the Fisher Kernel for Large-Scale Image Classification

  2010cited by this paper
• Learning realistic human actions from movies

  2008cited by this paper
• Fisher Kernels on Visual Vocabularies for Image Categorization

  2007cited by this paper
• Corpus Portal for Search in Monolingual Corpora

  2006cited by this paper
• On Space-Time Interest Points

  2005cited by this paper
• Γ and B

  2004cited by this paper
• Long Short-Term Memory

  1997influential reference
• Canonical ridge and econometrics of joint production

  1976cited by this paper
• Relations Between Two Sets of Variates

  1936cited by this paper
• AND T

  year unknowncited by this paper
• Author manuscript, published in "International Conference on Computer Vision (2013)" Action Recognition with Improved Trajectories

  year unknowncited by this paper
• and as an in

  year unknowncited by this paper
• Ieee Transactions on Pattern Analysis and Machine Intelligence 1 3d Convolutional Neural Networks for Human Action Recognition

  year unknowncited by this paper

• GVSE++: improved gated visual-semantic embedding for few-shot image and sentence matching

  2025cites this paper
• Dual-decoder collaborative learning with multi-hybrid view augmentation for self-supervised 3D action recognition

  2025cites this paper
• SHARDeg: A Benchmark for Skeletal Human Action Recognition in Degraded Scenarios

  2025cites this paper
• Full-Dimensional Optimizable Network: A Channel, Frame and Joint-Specific Network Modeling for Skeleton-Based Action Recognition

  2024cites this paper
• MGSAN: multimodal graph self-attention network for skeleton-based action recognition

  2024cites this paper
• TODO-Net: Temporally Observed Domain Contrastive Network for 3-D Early Action Prediction

  2024cites this paper
• Human Behavior Recognition Based on Attention Mechanism and Bottleneck Residual Dual-Path Spatiotemporal Graph Convolutional Network

  2024cites this paper
• BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind

  2024cites this paper
• MSA-GCN: Exploiting Multi-Scale Temporal Dynamics With Adaptive Graph Convolution for Skeleton-Based Action Recognition

  2024cites this paper
• Lightweight deep learning framework for recognizing construction workers' activities based on simplified node combinations

  2024cites this paper
• Attractive Multimodal Instructions, Describing Easy and Engaging Recipe Blogs

  2024cites this paper
• A Spatial-Temporal Multi-Feature Network (STMF-Net) for Skeleton-Based Construction Worker Action Recognition

  2024cites this paper
• Human Action Recognition (HAR) Using Skeleton-based Quantum Spatial Temporal Relative Transformer Network: ST-RTR

  2024cites this paper
• Graph Contrastive Learning for Skeleton-based Action Recognition

  2023cites this paper
• Graph Contrastive Learning for Skeleton-based Action Recognition

  2023cites this paper
• GA-Net: A Guidance Aware Network for Skeleton-Based Early Activity Recognition

  2023cites this paper
• Graph Diffusion Convolutional Network for Skeleton Based Semantic Recognition of Two-Person Actions

  2023cites this paper
• Image-Text Retrieval With Cross-Modal Semantic Importance Consistency

  2023cites this paper
• STTGC-Net: Spatial-Temporal Transformer with Graph Convolution for Skeleton-Based Action Recognition

  2023cites this paper
• Multi-Stream Fusion Network for Skeleton-Based Construction Worker Action Recognition

  2023cites this paper
• A new tra�c �ow prediction based on multi-scale convolutional neural network combined with LSTM neural network

  2023cites this paper
• Expansive Spatial-Temporal Kernel Convolution Network for Skeleton-Based Action Recognition

  2023cites this paper
• Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer - Current Trends and Research Perspectives

  2023cites this paper
• Improving Video Violence Recognition with Human Interaction Learning on 3D Skeleton Point Clouds

  2023cites this paper
• Flexible Data Integrity Checking With Original Data Recovery in IoT-Enabled Maritime Transportation Systems

  2023cites this paper
• Geometric Deep Neural Network Using Rigid and Non-Rigid Transformations for Landmark-Based Human Behavior Analysis

  2023influential citation
• FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation

  2023cites this paper
• Multi-cue temporal modeling for skeleton-based sign language recognition

  2023cites this paper
• On the Use of Deep Learning for Video Classification

  2023influential citation
• Bidirectional Long Short-Term Memory with Temporal Dense Sampling for human action recognition

  2022cites this paper
• Visual Hierarchical Recognition And Segmentation Of Interactions

  2022influential citation
• Skeleton based human action recognition with relative position encoding

  2022cites this paper
• MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching

  2022cites this paper
• Short-term path signature for skeleton-based action recognition

  2022influential citation
• Stock Prediction Analysis and Risk Conduction Path Research Based on EMD-LSTM Model

  2022cites this paper
• The prediction of PM2.5 concentration based on data decomposition and reconstruction algorithm and neural network model

  2022cites this paper
• Multi-visual information fusion and aggregation for video action classification

  2022cites this paper
• Intelligent Monitoring Model for Fall Risks of Hospitalized Elderly Patients

  2022cites this paper
• Motion Guided Attention Learning for Self-Supervised 3D Human Action Recognition

  2022cites this paper
• A Lightweight Subgraph-Based Deep Learning Approach for Fall Recognition

  2022cites this paper
• A deep architecture for log-Euclidean Fisher vector end-to-end learning with application to 3D point cloud classification

  2022cites this paper
• InfoGCN: Representation Learning for Human Skeleton-based Action Recognition

  2022cites this paper
• Efficient Image and Sentence Matching

  2022cites this paper
• Opleiding Informatica CHAD-net: Continuous Human Action Detection in RGB video

  2022cites this paper
• Predict the trend of economic indicators in time series based on recurrent neural network combined with leading indicators

  2022cites this paper
• On the Post Hoc Explainability of Optimized Self-Organizing Reservoir Network for Action Recognition

  2022cites this paper
• Learning-based restoration sequence ordering for multi-site traffic signal failure

  2022cites this paper
• MSST-RT: Multi-Stream Spatial-Temporal Relative Transformer for Skeleton-Based Action Recognition

  2021cites this paper
• A Novel Image Captioning Model Based on Morphology and Fisher Vectors

  2021cites this paper
• Exploring deep learning for intelligent image retrieval

  2021cites this paper
• Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval

  2021cites this paper
• Effective action recognition with fully supervised and self-supervised methods

  2021cites this paper
• Single-Skeleton and Dual-Skeleton Hypergraph Convolution Neural Networks for Skeleton-Based Action Recognition

  2021cites this paper
• Efficient Fine-Grained Visual-Text Search Using Adversarially-Learned Hash Codes

  2021cites this paper
• Soft Sensing of Feed Pulp Concentration in Thickener Based on Cross Validation LSTM Hyper-parameters Optimization

  2021cites this paper
• Human skeleton representation for 3D action recognition based on complex network coding and LSTM

  2021cites this paper
• Learning to Embed Semantic Similarity for Joint Image-Text Retrieval

  2021cites this paper
• Geometric Deep Neural Network using Rigid and Non-Rigid Transformations for Human Action Recognition

  2021influential citation
• Action Recognition with a Multi-View Temporal Attention Network

  2021cites this paper
• EScALation: a framework for efficient and scalable spatio-temporal action localization

  2021cites this paper
• IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition

  2021cites this paper
• Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition

  2021cites this paper
• Attribute-Guided Attention for Referring Expression Generation and Comprehension

  2020cites this paper
• Pictionary-Style Word Guessing on Hand-Drawn Object Sketches: Dataset, Analysis and Deep Network Models

  2020cites this paper
• Image and Sentence Matching via Semantic Concepts and Order Learning

  2020cites this paper
• Matching Image and Sentence With Multi-Faceted Representations

  2020cites this paper
• A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

  2020cites this paper
• Cross-Modal Attention With Semantic Consistence for Image–Text Matching

  2020cites this paper
• Deep Multimodal Image-Text Embeddings for Automatic Cross-Media Retrieval

  2020cites this paper
• Deep Learning: A Breakthrough in Medical Imaging.

  2020cites this paper
• TAEN: Temporal Aware Embedding Network for Few-Shot Action Recognition

  2020cites this paper
• Re-ranking image-text matching by adaptive metric fusion

  2020cites this paper
• A Context Based Deep Temporal Embedding Network in Action Recognition

  2020cites this paper
• Multi-Modality Cross Attention Network for Image and Sentence Matching

  2020cites this paper
• Visual-Semantic Matching by Exploring High-Order Attention and Distraction

  2020cites this paper
• Spatial Temporal Transformer Network for Skeleton-based Action Recognition

  2020cites this paper
• Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition

  2020cites this paper
• Cross‐modal retrieval with dual multi‐angle self‐attention

  2020cites this paper
• AGA-LSTM: An Optimized LSTM Neural Network Model Based on Adaptive Genetic Algorithm

  2020cites this paper
• Human Interaction Learning on 3D Skeleton Point Clouds for Video Violence Recognition

  2020cites this paper
• KShapeNet: Riemannian network on Kendall shape space for Skeleton based Action Recognition

  2020cites this paper
• Skeleton-based action recognition via spatial and temporal transformer networks

  2020cites this paper
• Human Action Recognition From Various Data Modalities: A Review

  2020cites this paper
• Audiovisual Transformer with Instance Attention for Audio-Visual Event Localization

  2020cites this paper
• Automatic Image Description Generation Using Deep Multimodal Embeddings

  2020cites this paper
• Modality-Invariant Image-Text Embedding for Image-Sentence Matching

  2019cites this paper
• ACMM: Aligned Cross-Modal Memory for Few-Shot Image and Sentence Matching

  2019cites this paper
• GestureVLAD: Combining Unsupervised Features Representation and Spatio-Temporal Aggregation for Doppler-Radar Gesture Recognition

  2019cites this paper
• Local fusion networks with chained residual pooling for video action recognition

  2019cites this paper
• Signal-Based Intelligent Hydraulic Fault Diagnosis Methods: Review and Prospects

  2019cites this paper
• Cross-Modality Feature Learning via Convolutional Autoencoder

  2019cites this paper
• Topic-Oriented Image Captioning Based on Order-Embedding

  2019cites this paper
• Language Features Matter: Effective Language Representations for Vision-Language Tasks

  2019cites this paper
• Deep Neural Networks for Human Behavior Understanding

  2019cites this paper
• Multi-Level Visual-Semantic Alignments with Relation-Wise Dual Attention Network for Image and Text Matching

  2019cites this paper
• Dense Convolutional Networks for Efficient Video Analysis

  2019cites this paper
• A Spatiotemporal Heterogeneous Two-Stream Network for Action Recognition

  2019cites this paper
• CycleMatch: A cycle-consistent embedding network for image-text matching

  2019cites this paper
• D3-LND: A two-stream framework with discriminant deep descriptor, linear CMDT and nonlinear KCMDT descriptors for action recognition

  2019cites this paper
• Learning to recognize human actions: from hand-crafted to deep-learning based visual representations

  2019cites this paper