Deep Patch Representations with Shared Codebook for Scene Classification

Scene classification is a challenging problem. Compared with object images, scene images are more abstract, as they are composed of objects. Object and scene images have different characteristics with different scales and composition structures. How to effectively integrate the local mid-level semantic representations including both object and scene concepts needs to be investigated, which is an important aspect for scene classification. In this article, the idea of a sharing codebook is introduced by organically integrating deep learning, concept feature, and local feature encoding techniques. More specifically, the shared local feature codebook is generated from the combined ImageNet1K and Places365 concepts (Mixed1365) using convolutional neural networks. As the Mixed1365 features cover all the semantic information including both object and scene concepts, we can extract a shared codebook from the Mixed1365 features, which only contain a subset of the whole 1,365 concepts with the same codebook size. The shared codebook can not only provide complementary representations without additional codebook training but also be adaptively extracted toward different scene classification tasks. A method of fusing the encoded features with both the original codebook and the shared codebook is proposed for scene classification. In this way, more comprehensive and representative image features can be generated for classification. Extensive experimentations conducted on two public datasets validate the effectiveness of the proposed method. Besides, some useful observations are also revealed to show the advantage of shared codebook.

• Publication year

  2019

• Venue

  ACM Trans. Multim. Comput. Commun. Appl.

• Publication date

  2019-01-24

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3231738
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Visual Tracking With Weighted Adaptive Local Sparse Appearance Model via Spatio-Temporal Context Learning

  2018cited by this paper
• Multi-Scale Multi-Feature Context Modeling for Scene Recognition in the Semantic Manifold

  2017cited by this paper
• Strokelets: A Learned Multi-Scale Mid-Level Representation for Scene Text Recognition

  2016cited by this paper
• Places: An Image Database for Deep Scene Understanding

  2016cited by this paper
• Scene Recognition with CNNs: Objects, Scales and Dataset Bias

  2016cited by this paper
• Object based Scene Representations using Fisher Scores of Local Subspace Projections

  2016cited by this paper
• Hyperlink-Aware Object Retrieval

  2016cited by this paper
• Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation

  2016cited by this paper
• Harvesting Discriminative Meta Objects with Deep CNN Features for Scene Classification

  2015cited by this paper
• Multi-scale pyramid pooling for deep convolutional representation

  2015cited by this paper
• Joint multi-feature spatial context for scene recognition in the semantic manifold

  2015cited by this paper
• An Object-Level High-Order Contextual Descriptor Based on Semantic, Spatial, and Scale Cues

  2015cited by this paper
• Scene classification with semantic Fisher vectors

  2015influential reference
• INSTRE

  2015cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014influential reference
• Learning Deep Features for Scene Recognition using Places Database

  2014cited by this paper
• Very Deep Convolutional Networks for Large-Scale Image Recognition

  2014cited by this paper
• USB: Ultrashort Binary Descriptor for Fast Visual Matching and Retrieval

  2014cited by this paper
• Learning Object-to-Class Kernels for Scene Classification

  2014cited by this paper
• Multi-scale Orderless Pooling of Deep Convolutional Activation Features

  2014influential reference
• Latent Dirichlet Allocation Models for Image Classification

  2013cited by this paper
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

  2013cited by this paper
• Mid-level Visual Element Discovery as Discriminative Mode Seeking

  2013cited by this paper
• Blocks That Shout: Distinctive Parts for Scene Classification

  2013cited by this paper
• Max-Margin Multiple-Instance Dictionary Learning

  2013cited by this paper
• Object Bank: An Object-Level Image Representation for High-Level Visual Recognition

  2013cited by this paper
• Learning Hierarchical Semantic Description Via Mixed-Norm Regularization for Image Understanding

  2012cited by this paper
• Image categorization using Fisher kernels of non-iid image models

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012influential reference
• Context aware topic model for scene recognition

  2012cited by this paper
• Scene Recognition on the Semantic Manifold

  2012cited by this paper
• Holistic Context Models for Visual Recognition

  2012influential reference
• Improving the Fisher Kernel for Large-Scale Image Classification

  2010influential reference
• Object Bank: A High-Level Image Representation for Scene Classification & Semantic Feature Sparsification

  2010cited by this paper
• SUN database: Large-scale scene recognition from abbey to zoo

  2010cited by this paper
• Aggregating local descriptors into a compact image representation

  2010cited by this paper
• Locality-constrained Linear Coding for image classification

  2010influential reference
• Kernel Descriptors for Visual Recognition

  2010cited by this paper
• Linear spatial pyramid matching using sparse coding for image classification

  2009influential reference
• Holistic context modeling using semantic co-occurrences

  2009cited by this paper
• Recognizing indoor scenes

  2009cited by this paper
• Efficient Match Kernel between Sets of Features for Visual Recognition

  2009cited by this paper
• Kernel Codebooks for Scene Categorization

  2008cited by this paper
• Semantic Modeling of Natural Scenes for Content-Based Image Retrieval

  2007cited by this paper
• Bridging the Gap: Query by Semantic Example

  2007cited by this paper
• Fisher Kernels on Visual Vocabularies for Image Categorization

  2007cited by this paper
• Spatial Latent Dirichlet Allocation

  2007cited by this paper
• Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories

  2006cited by this paper
• A Bayesian hierarchical model for learning natural scene categories

  2005cited by this paper
• A Semantic Typicality Measure for Natural Scene Categorization

  2004cited by this paper
• Distinctive Image Features from Scale-Invariant Keypoints

  2004cited by this paper
• Visual categorization with bags of keypoints

  2002cited by this paper
• Semantic organization of scenes using discriminant structural templates

  1999cited by this paper
• Submitted to Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Classemes and Other Classifier-based Features for Efficient Object Categorization

  year unknowncited by this paper

• LFSD-VQVAE: 3D Face Generation Based on Local Feature Swapping

  2025cites this paper
• Context-Aware Dynamic Integration for Scene Recognition

  2025cites this paper
• Feature selection through adaptive sparse learning for scene recognition

  2024cites this paper
• Can using a pre-trained deep learning model as the feature extractor in the bag-of-deep-visual-words model always improve image classification accuracy?

  2024cites this paper
• Dialogue Situation Recognition in Everyday Conversation From Audio, Visual, and Linguistic Information

  2023cites this paper
• Deep feature voting: a semantic-driven and local context-aware approach for image classification

  2023cites this paper
• Resolution-Agnostic Remote Sensing Scene Classification With Implicit Neural Representations

  2023cites this paper
• Scene recognition using multiple representation network

  2022cites this paper
• A Novel Multi-Modal Network-Based Dynamic Scene Understanding

  2022cites this paper
• AABLSTM: A Novel Multi-task Based CNN-RNN Deep Model for Fashion Analysis

  2022cites this paper
• Indoor Scene Classification Algorithm Based on an Object Vector for Robot Applications

  2022cites this paper
• Recent advances in scene image representation and classification

  2022influential citation
• Scene Recognition With Objectness, Attribute, and Category Learning

  2022cites this paper
• Semantic embedding: scene image classification using scene-specific objects

  2022cites this paper
• Deep Learning for Scene Classification: A Survey

  2021cites this paper
• Visual Semantic-Based Representation Learning Using Deep CNNs for Scene Recognition

  2021cites this paper
• Developing efficient transfer learning strategies for robust scene recognition in mobile robotics using pre-trained convolutional neural networks

  2021cites this paper
• Dialogue Situation Recognition for Everyday Conversation Using Multimodal Information

  2021cites this paper
• A New Foreground-Background based Method for Behavior-Oriented Social Media Image Classification

  2021cites this paper
• Recognition of varying size scene images using semantic analysis of deep activation maps

  2021cites this paper
• Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

  2021cites this paper
• Scene recognition: A comprehensive survey

  2020cites this paper
• Scene Recognition with Comprehensive Regions Graph Modeling

  2019cites this paper
• Semantic-Aware Scene Recognition

  2019cites this paper
• FOSNet: An End-to-End Trainable Deep Neural Network for Scene Recognition

  2019cites this paper
• Hierarchical data generator based on tree-structured stick breaking process for benchmarking clustering methods

  2016cites this paper