Dynamics of scene representations in the human brain revealed by magnetoencephalography and deep neural networks

Human scene recognition is a rapid multistep process evolving over time from single scene image to spatial layout processing. We used multivariate pattern analyses on magnetoencephalography (MEG) data to unravel the time course of this cortical process. Following an early signal for lower-level visual analysis of single scenes at ~100ms, we found a marker of real-world scene size, i.e. spatial layout processing, at ~250ms indexing neural representations robust to changes in unrelated scene properties and viewing conditions. For a quantitative explanation that captures the complexity of scene recognition, we compared MEG data to a deep neural network model trained on scene classification. Representations of scene size emerged intrinsically in the model, and resolved emerging neural scene size representation. Together our data provide a first description of an electrophysiological signal for layout processing in humans, and a novel quantitative model of how spatial layout representations may emerge in the human brain.

• Publication year

  2015

• Venue

  bioRxiv

• Publication date

  2015-11-23

• Fields of study

  Biology, Medicine, Computer Science

• Identifiers
  DOI 10.1016/j.neuroimage.2016.03.063PMID 27039703PMCID 5542416
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• The time course of natural scene perception with reduced attention.

  2016cited by this paper
• Similarity-Based Fusion of MEG and fMRI Reveals Spatio-Temporal Dynamics in Human Cortex During Visual Object Recognition

  2015cited by this paper
• Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans?

  2015cited by this paper
• Object Detectors Emerge in Deep Scene CNNs

  2014influential reference
• Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models

  2014cited by this paper
• Low-Level Image Properties of Visual Objects Predict Patterns of Neural Response across Category-Selective Regions of the Ventral Visual Pathway

  2014cited by this paper
• Explaining the hierarchy of visual representational geometries by remixing of features from many computational vision models

  2014cited by this paper
• Patterns of response to visual scenes are linked to the low-level properties of the image

  2014cited by this paper
• Characterizing the dynamics of mental representations: the temporal generalization method.

  2014cited by this paper
• Learning Deep Features for Scene Recognition using Places Database

  2014cited by this paper
• A channel for 3D environmental shape in anterior inferotemporal cortex.

  2014cited by this paper
• Parametric Coding of the Size and Clutter of Natural Scenes in the Human Brain.

  2014influential reference
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream

  2014cited by this paper
• Performance-optimized hierarchical models predict neural responses in higher visual cortex

  2014cited by this paper
• Object-Specific Semantic Coding in Human Perirhinal Cortex

  2014cited by this paper
• Resolving human object recognition in space and time

  2014influential reference
• A Toolbox for Representational Similarity Analysis

  2014cited by this paper
• Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation

  2014cited by this paper
• Caffe: Convolutional Architecture for Fast Feature Embedding

  2014cited by this paper
• The dynamics of invariant object recognition in the human visual system.

  2014cited by this paper
• Representational dynamics of object vision: the first 1000 ms.

  2013influential reference
• Author ' s personal copy Representational geometry : integrating cognition , computation , and the brain

  2013cited by this paper
• The Neural Representation Benchmark and its Evaluation on Brain and Machine

  2013cited by this paper
• From Image Statistics to Scene Gist: Evoked Neural Activity Reveals Transition from Low-Level Natural Image Structure to Scene Category

  2013cited by this paper
• Direct recordings of grid-like neuronal activity in human spatial navigation

  2013cited by this paper
• A network for scene processing in the macaque temporal lobe.

  2013cited by this paper
• How does the brain solve visual object recognition?

  2012cited by this paper
• ImageNet classification with deep convolutional neural networks

  2012cited by this paper
• A New Role for the Parahippocampal Cortex in Representing Space

  2011cited by this paper
• Modality-independent coding of spatial layout in the human brain.

  2011cited by this paper
• Cognitive neuroscience: scene layout from vision and touch.

  2011influential reference
• Decoding Representations of Scenes in the Medial Temporal Lobes

  2011cited by this paper
• A new neural framework for visuospatial processing

  2011cited by this paper
• Constructing scenes from objects in human occipitotemporal cortex

  2011cited by this paper
• Disentangling Scene Content from Spatial Boundary: Complementary Roles for the Parahippocampal Place Area and Lateral Occipital Complex in Representing Real-World Scenes

  2011cited by this paper
• Neural representations for object perception: structure, category, and adaptive coding.

  2011cited by this paper
• Real-World Scene Representations in High-Level Visual Cortex: It's the Spaces More Than the Places

  2011cited by this paper
• Evidence for grid cells in a human memory network

  2010cited by this paper
• Establishing the Boundaries: The Hippocampal Contribution to Imagining Scenes

  2010cited by this paper
• ImageNet: A large-scale hierarchical image database

  2009cited by this paper
• Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory

  2008cited by this paper
• Place cells, grid cells, and the brain's spatial representation system.

  2008cited by this paper
• Representational Similarity Analysis – Connecting the Branches of Systems Neuroscience

  2008influential reference
• Latency and Selectivity of Single Neurons Indicate Hierarchical Processing in the Human Medial Temporal Lobe

  2008cited by this paper
• Untangling invariant object recognition.

  2007cited by this paper
• A feedforward architecture accounts for rapid categorization

  2007influential reference
• Nonparametric statistical testing of EEG- and MEG-data.

  2007influential reference
• A Cortical Region Consisting Entirely of Face-Selective Cells

  2006cited by this paper
• An event-related potential component sensitive to images of the human body

  2006cited by this paper
• Neuromagnetic recordings reveal the temporal dynamics of auditory spatial processing in the human cortex.

  2006cited by this paper
• A Cortical Region Consisting Entirely of Face-Selective Cells

  2006cited by this paper
• Coding of visual objects in the ventral stream.

  2006cited by this paper
• A comparison of random field theory and permutation methods for the statistical analysis of MEG data

  2005influential reference
• Object recognition with features inspired by visual cortex

  2005cited by this paper
• Generalization in vision and motor control

  2004cited by this paper
• The neural correlates of perceiving human bodies: an ERP study on the body-inversion effect

  2004cited by this paper
• Nonparametric Permutation Tests for Functional Neuroimaging

  2003influential reference
• Stages of processing in face perception: an MEG study

  2002cited by this paper
• Human cortical representation of virtual auditory space: differences between sound azimuth and elevation

  2002cited by this paper
• Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope

  2001influential reference
• A Cortical Area Selective for Visual Processing of the Human Body

  2001cited by this paper
• Faculty Opinions recommendation of A cortical area selective for visual processing of the human body.

  2001cited by this paper
• Hierarchical models of object recognition in cortex

  1999influential reference
• The parahippocampal place area: recognition, navigation, or encoding?

  1999cited by this paper
• An area within human ventral cortex sensitive to "building" stimuli: evidence and implications.

  1998cited by this paper
• Signal timing across the macaque visual system.

  1998cited by this paper
• A cortical representation of the local visual environment

  1998cited by this paper
• The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception

  1997cited by this paper
• Evoked Potential Studies of Face and Object Processing

  1996cited by this paper
• Electrophysiological Studies of Face Perception in Humans

  1996cited by this paper
• Speed of processing in the human visual system

  1996cited by this paper
• Face recognition in human extrastriate cortex.

  1994cited by this paper
• Human Brain Mapping 15:1–25(2001) � Nonparametric Permutation Tests For Functional Neuroimaging:

  year unknowninfluential reference

• Spatiotemporal representations of contextual associations for real-world objects

  2025cites this paper
• Wordsworth: A generative word dataset for comparison of speech representations in humans and neural networks

  2025cites this paper
• The timecourse of inter-object contextual facilitation

  2025influential citation
• The impact of scene inversion on early scene-selective activity.

  2025cites this paper
• Where do I go? Decoding temporal neural dynamics of scene processing and visuospatial memory interactions using convolutional neural networks

  2025cites this paper
• Natural Scene and Object Perception Based on Statistical Image Features: Psychophysics and EEG

  2025cites this paper
• Investigating the temporal dynamics and modelling of mid-level feature representations in humans

  2025cites this paper
• Temporal misalignment in scene perception: Divergent representations of locomotive action affordances in human brain responses and DNNs

  2025influential citation
• Core dimensions of human material perception

  2025cites this paper
• Distinguishing the roles of edge, color, and other surface information in basic and superordinate scene representation

  2025cites this paper
• Artificial neural networks for magnetoencephalography: a review of an emerging field

  2025influential citation
• Cortical Encoding of Spatial Structure and Semantic Content in 3D Natural Scenes

  2025cites this paper
• Brain-guided convolutional neural networks reveal task-specific representations in scene processing

  2025cites this paper
• The brain prioritizes the basic level of object category abstraction

  2025cites this paper
• Metabolic quantum limit to the information capacity of magnetoencephalography

  2025cites this paper
• Where do I go? Decoding temporal neural dynamics of scene processing and visuospatial memory interactions using CNNs

  2024cites this paper
• Tool Representations in Human Visual Cortex

  2024cites this paper
• Human Visual Pathways for Action Recognition versus Deep Convolutional Neural Networks: Representation Correspondence in Late but Not Early Layers

  2024cites this paper
• Development of Character Recognition Model Inspired by Visual Explanations

  2024cites this paper
• Enhanced and idiosyncratic neural representations of personally typical scenes

  2024cites this paper
• Representations of imaginary scenes and their properties in cortical alpha activity

  2024cites this paper
• Visual Recognition Memory of Scenes Is Driven by Categorical, Not Sensory, Visual Representations

  2024cites this paper
• Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

  2024cites this paper
• Spatial contextual information modulates affordance processing and early electrophysiological markers of scene perception

  2024cites this paper
• Multi-Task Collaborative Network: Bridge the Supervised and Self-Supervised Learning for EEG Classification in RSVP Tasks

  2024cites this paper
• Shared representations of human actions across vision and language

  2024cites this paper
• The best of two worlds: Decoding and source-reconstructing M/EEG oscillatory activity with a unique model

  2023cites this paper
• Seeking the neural representation of statistical properties in print during implicit processing of visual words

  2023cites this paper
• Oscillations in an artificial neural network convert competing inputs into a temporal code

  2023cites this paper
• Brain decoding: toward real-time reconstruction of visual perception

  2023cites this paper
• Interpretable many-class decoding for MEG

  2023cites this paper
• Memorability shapes perceived time (and vice versa)

  2023cites this paper
• Evaluating Robustness Against Adversarial Attacks: A Representational Similarity Analysis Approach

  2023cites this paper
• Visual number sense for real-world scenes shared by deep neural networks and humans

  2023cites this paper
• Transforming the neuroscience of language: estimating pattern-to-pattern transformations of brain activity

  2023cites this paper
• Visual features are processed before navigational affordances in the human brain

  2023cites this paper
• THE REAL-WORLD SIZE OF OBJECT-CONSTELLATIONS

  2023influential citation
• Investigating the Effects of Fairness Interventions Using Pointwise Representational Similarity

  2023cites this paper
• Rapid Processing of Observed Touch through Social Perceptual Brain Regions: An EEG-fMRI Fusion Study

  2023cites this paper
• Predicting the neural response of primary visual cortex (v1) using deep learning approach

  2023cites this paper
• Objects sharpen visual scene representations: evidence from MEG decoding

  2023cites this paper
• Interpretable full-epoch multiclass decoding for M/EEG

  2023influential citation
• EEG Decoding Reveals Neural Predictions for Naturalistic Material Behaviors

  2023cites this paper
• Empirically Identifying and Computationally Modeling the Brain–Behavior Relationship for Human Scene Categorization

  2023influential citation
• Words are all you need? Capturing human sensory similarity with textual descriptors

  2022cites this paper
• The Neural Representation of a Repeated Standard Stimulus in Dyslexia

  2022cites this paper
• Characterizing dynamic neural representations of scene attractiveness

  2022cites this paper
• Spatiotemporal neural dynamics of object recognition under uncertainty in humans

  2022cites this paper
• The neuroconnectionist research programme

  2022cites this paper
• Temporal dynamics of the neural representation of hue and luminance polarity

  2022cites this paper
• Spectral brain signatures of aesthetic natural perception in the alpha and beta frequency bands

  2022cites this paper
• Effects of natural scene inversion on visual-evoked brain potentials and pupillary responses: a matter of effortful processing of unfamiliar configurations.

  2022cites this paper
• Deep representation-based transfer learning for deep neural networks

  2022cites this paper
• How does artificial intelligence contribute to iEEG research?

  2022cites this paper
• On the similarities of representations in artificial and brain neural networks for speech recognition

  2022cites this paper
• EEG decoding of multidimensional information from emotional faces

  2022cites this paper
• Functional selectivity for naturalistic social interaction perception in the 1 human superior temporal sulcus 2 3

  2021cites this paper
• Face perception foundations for pattern recognition algorithms

  2021cites this paper
• Dynamic Electrode-to-Image (DETI) mapping reveals the human brain’s spatiotemporal code of visual information

  2021cites this paper
• Temporal dynamics of the neural representation of hue and luminance contrast

  2021influential citation
• Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder

  2021cites this paper
• The Neural Correlates of Image Texture in the Human Vision Using Magnetoencephalography

  2021cites this paper
• Rapid Extraction of the Spatial Distribution of Physical Saliency and Semantic Informativeness from Natural Scenes in the Human Brain

  2021cites this paper
• ImageNet-trained deep neural networks exhibit illusion-like response to the Scintillating grid

  2021influential citation
• Parts and Wholes in Scene Processing

  2021cites this paper
• Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing

  2021cites this paper
• Exploring the Similarity of Representations in Model-Agnostic Meta-Learning

  2021cites this paper
• Inter-Subject MEG Decoding for Visual Information with Hybrid Gated Recurrent Network

  2021cites this paper
• Category systems for real-world scenes

  2021cites this paper
• Unraveling Representations in Scene-selective Brain Regions Using Scene-Parsing Deep Neural Networks

  2020cites this paper
• Artificially-generated scenes demonstrate the importance of global scene properties for scene perception.

  2020cites this paper
• Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

  2020cites this paper
• Naturalistic stimuli reveal a dominant role for agentic action in visual representation

  2020cites this paper
• A naturalistic viewing paradigm using 360° panoramic video clips and real-time field-of-view changes with eye-gaze tracking

  2020cites this paper
• Emergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks

  2020cites this paper
• Rapid contextualization of fragmented scene information in the human visual system

  2020influential citation
• Unravelling Representations in Scene-selective Brain Regions Using Scene Parsing Deep Neural Networks

  2020cites this paper
• Disentangling the Independent Contributions of Visual and Conceptual Features to the Spatiotemporal Dynamics of Scene Categorization

  2020cites this paper
• Quasi-synchronization of neural networks with diffusion effects via intermittent control of regional division

  2020cites this paper
• What shapes feature representations? Exploring datasets, architectures, and training

  2020cites this paper
• Temporal dynamics of the neural mechanisms for encoding hue and luminance contrast uncovered by magnetoencephalography

  2020influential citation
• Causal Evidence for a Double Dissociation between Object- and Scene-Selective Regions of Visual Cortex: A Preregistered TMS Replication Study

  2020cites this paper
• Visual Place Recognition by spatial matching of high-level CNN features

  2020cites this paper
• Detection of early stages of Alzheimer's disease based on MEG activity with a randomized convolutional neural network

  2020cites this paper
• Assessing Task-dependent Flexibility and the Temporal Dynamics of Object Categorization

  2020influential citation
• Mapping the Scene and Object Processing Networks by Intracranial EEG

  2020cites this paper
• Comparative Analysis of the Types of Processing of Visual Information from the Point of View of Cognitive Science

  2020cites this paper
• Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space

  2020cites this paper
• Classification of mechanisms underlying cardiac arrhythmias by deep learning

  2020cites this paper
• Brain Inspired Neural Network Models of Visual Motion Perception and Tracking in Dynamic Scenes

  2020cites this paper
• Temporal dynamics of the neural representation of hue and luminance polarity

  2020cites this paper
• Time-resolved estimation of strength of motor imagery representation by multivariate EEG decoding

  2020cites this paper
• Multilayer perceptron based deep neural network for early detection of coronary heart disease

  2020cites this paper
• Spatio-temporal Dynamics of Visual Object Recognition in the Human Brain

  2020cites this paper
• Modelling Guided Attention Using Learned Target-Category Associations

  2020cites this paper
• Angular Visual Hardness

  2020cites this paper
• Reconstructing feedback representations in ventral visual pathway with a generative adversarial autoencoder

  2020cites this paper
• A neural mechanism for contextualizing fragmented inputs during naturalistic vision

  2019cites this paper
• A Survey on Deep Learning based Brain Computer Interface: Recent Advances and New Frontiers

  2019cites this paper
• Time-Resolved Correspondences Between Deep Neural Network Layers and EEG Measurements in Object Processing

  2019cites this paper