Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception. There is a need for extensive neuroimaging datasets to facilitate the study of dynamic human visual perception. Here, the authors present a repository of whole-brain fMRI responses to over 1000 short naturalistic video clips across ten human subjects.

• Publication year

  2024

• Venue

  Nature Communications

• Publication date

  2024-07-24

• Fields of study

  Medicine, Computer Science

• Identifiers
  DOI 10.1038/s41467-024-50310-3PMID 39048577PMCID 11269733
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar, PubMed

• No claims are published for this paper.

• No concepts are published for this paper.

• Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

  2024cited by this paper
• Hierarchical organization of social action features along the lateral visual pathway

  2024cited by this paper
• Hierarchical organization of social action features along the lateral visual pathway.

  2023cited by this paper
• Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity

  2023cited by this paper
• Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

  2023cited by this paper
• THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

  2023cited by this paper
• A large-scale fMRI dataset for human action recognition

  2023cited by this paper
• Improving the accuracy of single-trial fMRI response estimates using GLMsingle

  2022influential reference
• Multidimensional memory topography in the medial parietal cortex identified from neuroimaging of thousands of daily memory videos

  2022cited by this paper
• A Highly Selective Response to Food in Human Visual Cortex Revealed by Hypothesis-Free Voxel Decomposition

  2022cited by this paper
• A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity

  2022cited by this paper
• GIT: A Generative Image-to-text Transformer for Vision and Language

  2022cited by this paper
• A spatio-temporal integrated model based on local and global features for video expression recognition

  2021cited by this paper
• Temporal stability of stimulus representation increases along rodent visual cortical hierarchies

  2021cited by this paper
• The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning

  2021cited by this paper
• Your head is there to move you around: Goal-driven models of the primate dorsal pathway

  2021cited by this paper
• Dynamics of fMRI patterns reflect sub-second activation sequences and reveal replay in human visual cortex

  2021cited by this paper
• Two 'what' pathways for action and object recognition.

  2021influential reference
• Computational models of category-selective brain regions enable high-throughput tests of selectivity

  2021cited by this paper
• Is Space-Time Attention All You Need for Video Understanding?

  2021cited by this paper
• A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

  2021cited by this paper
• Selective responses to faces, scenes, and bodies in the ventral visual pathway of infants.

  2021cited by this paper
• Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions

  2021influential reference
• NeuroGen: Activation optimized image synthesis for discovery neuroscience

  2021cited by this paper
• Memorability: An image-computable measure of information utility

  2021cited by this paper
• Optical-flow-based framework to boost video object detection performance with object enhancement

  2021cited by this paper
• Evidence for a Third Visual Pathway Specialized for Social Perception.

  2020cited by this paper
• TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Devices

  2020cited by this paper
• Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence.

  2020cited by this paper
• A temporal decomposition method for identifying venous effects in task-based fMRI

  2020influential reference
• Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability

  2020influential reference
• Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies

  2020cited by this paper
• A naturalistic neuroimaging database for understanding the brain using ecological stimuli

  2020cited by this paper
• Object representations in the human brain reflect the co-occurrence statistics of vision and language

  2020cited by this paper
• Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding

  2019influential reference
• Population response magnitude variation in inferotemporal cortex predicts image memorability

  2019influential reference
• A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time

  2019cited by this paper
• An integrative computational architecture for object-driven cortex.

  2019influential reference
• THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images

  2019influential reference
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

  2019cited by this paper
• Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex

  2019cited by this paper
• Eidetic 3D LSTM: A Model for Video Prediction and Beyond

  2019cited by this paper
• Recurrence is required to capture the representational dynamics of the human visual system

  2019cited by this paper
• Distinct representations in occipito‐temporal, parietal, and premotor cortex during action perception revealed by fMRI and computational modeling

  2019cited by this paper
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

  2019influential reference
• Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs

  2019influential reference
• FMRIPrep: a robust preprocessing pipeline for functional MRI

  2018cited by this paper
• Image memorability across longer time intervals

  2018cited by this paper
• Neural correlates of action: Comparing meta‐analyses of imagery, observation, and execution

  2018cited by this paper
• MobileNetV2: Inverted Residuals and Linear Bottlenecks

  2018cited by this paper
• The perceptual neural trace of memorable unseen scenes

  2018cited by this paper
• BOLD5000, a public fMRI dataset while viewing 5000 visual images

  2018cited by this paper
• Dissociating neural markers of stimulus memorability and subjective recognition during episodic retrieval

  2018cited by this paper
• Moments in Time Dataset: One Million Videos for Event Understanding

  2018cited by this paper
• Places: A 10 Million Image Database for Scene Recognition

  2018cited by this paper
• Methods for computing the maximum performance of computational models of fMRI responses

  2018cited by this paper
• Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks

  2018cited by this paper
• Temporal Shift Module for Efficient Video Understanding

  2018cited by this paper
• Interpreting encoding and decoding models.

  2018cited by this paper
• Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

  2017cited by this paper
• Parietal area BA7 integrates motor programs for reaching, grasping, and bimanual coordination.

  2017cited by this paper
• Seeing it all: Convolutional network layers map the function of the human visual system

  2017cited by this paper
• Increasingly complex representations of natural movies across the dorsal stream are shared between subjects

  2017cited by this paper
• Object Representations in Human Visual Cortex Formed Through Temporal Integration of Dynamic Partial Shape Views

  2017cited by this paper
• Action Categories in Lateral Occipitotemporal Cortex Are Organized Along Sociality and Transitivity

  2017cited by this paper
• A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation

  2016cited by this paper
• MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites

  2016cited by this paper
• The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments

  2016cited by this paper
• Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

  2016cited by this paper
• Mindboggling morphometry of human brains

  2016cited by this paper
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

  2016cited by this paper
• Enriching Word Vectors with Subword Information

  2016cited by this paper
• A multi-modal parcellation of human cerebral cortex

  2016cited by this paper
• Convolutional Two-Stream Network Fusion for Video Action Recognition

  2016cited by this paper
• Color-Biased Regions of the Ventral Visual Pathway Lie between Face- and Place-Selective Regions in Humans, as in Macaques

  2016cited by this paper
• Probabilistic Maps of Visual Topography in Human Cortex.

  2015cited by this paper
• Deep Residual Learning for Image Recognition

  2015cited by this paper
• Deep neural networks: a new framework for modelling biological vision and brain information processing

  2015cited by this paper
• Learning Computational Models of Video Memorability from fMRI Brain Imaging

  2015cited by this paper
• Similarity-Based Fusion of MEG and fMRI Reveals Spatio-Temporal Dynamics in Human Cortex During Visual Object Recognition

  2015cited by this paper
• Adjudicating between face-coding models with individual-face fMRI responses

  2015cited by this paper
• A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives.

  2015cited by this paper
• Understanding and Predicting Image Memorability at a Large Scale

  2015cited by this paper
• Generic decoding of seen and imagined objects using hierarchical visual features

  2015cited by this paper
• The lateral occipitotemporal cortex in action.

  2015cited by this paper
• EmoNets: Multimodal deep learning approaches for emotion recognition in video

  2015cited by this paper
• A hierarchy of intrinsic timescales across primate cortex

  2014cited by this paper
• Implied motion perception from a still image in infancy

  2014cited by this paper
• Machine learning for neuroimaging with scikit-learn

  2014influential reference
• A Toolbox for Representational Similarity Analysis

  2014cited by this paper
• ImageNet Large Scale Visual Recognition Challenge

  2014cited by this paper
• Performance-optimized hierarchical models predict neural responses in higher visual cortex

  2014cited by this paper
• Methods to detect, characterize, and remove motion artifact in resting state fMRI

  2014cited by this paper
• Temporal Integration Windows for Naturalistic Visual Sequences

  2014cited by this paper
• Accurate decoding of sub-TR timing differences in stimulations of sub-voxel regions from multi-voxel response patterns

  2013cited by this paper
• Fixational eye movements during viewing of dynamic natural scenes

  2013cited by this paper
• Functional connectomics from resting-state fMRI.

  2013cited by this paper
• Functional properties of the left parietal tool use region

  2013cited by this paper
• An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data

  2013cited by this paper
• The minimal preprocessing pipelines for the Human Connectome Project

  2013cited by this paper

• The Human Brain as a Dynamic Mixture of Expert Models in Video Understanding

  2026influential citation
• How does longer temporal context enhance multimodal narrative video processing in the brain?

  2026cites this paper
• BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction

  2026cites this paper
• A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learning

  2025cites this paper
• Sex-specific dynamic changes of serum short-chain fatty acids in rhesus monkeys during aging

  2025influential citation
• Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage

  2025cites this paper
• Probing the content of semantic representations in body-selective regions

  2025cites this paper
• Many-Two-One: Diverse Representations Across Visual Pathways Emerge from A Single Objective

  2025cites this paper
• ADA: A decoding algorithm for temporally-variable brain responses

  2025cites this paper
• Incorporating multi-modal prompt learning into foundation models enhances predictability of visual fMRI responses to dynamic natural stimuli

  2025cites this paper
• Spatio-temporal bidirectional Long Short-Term Memory-based category decoding of natural dynamic facial expression stimuli from visual functional magnetic resonance imaging data.

  2025cites this paper
• Behavioral and neural mechanisms underlying contextual processing in naturalistic short video viewing

  2025cites this paper
• MOSAIC: A scalable framework for fMRI dataset aggregation and modeling of human vision

  2025cites this paper
• Auditory representations of words during silent visual reading

  2025cites this paper
• One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

  2025influential citation
• A Survey of fMRI to Image Reconstruction

  2025cites this paper
• A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision

  2025cites this paper
• Investigating the temporal dynamics and modelling of mid-level feature representations in humans

  2025cites this paper
• Trial-level Representational Similarity Analysis

  2025cites this paper
• Electroencephalogram-based floor vibration serviceability evaluation using convolutional neural network and ensemble learning

  2025cites this paper
• Synchrony and subjective experience: the neural correlates of the stream of consciousness.

  2025cites this paper
• A large-scale MEG and EEG dataset for object recognition in naturalistic scenes

  2025cites this paper
• Cross-Subject Mind Decoding from Inaccurate Representations

  2025cites this paper
• Investigating the role of modality and training objective on representational alignment between transformers and the brain

  2025influential citation
• Neural representation of action features across sensory modalities: A multimodal fMRI study

  2025cites this paper
• Durrmeyer deep neural networks: Bridging deep learning and dynamic brain functional connectivity

  2025cites this paper
• Even the "Devil" has Rights!

  2024cites this paper
• In silico discovery of representational relationships across visual cortex

  2024cites this paper
• Neuro-3D: Towards 3D Visual Decoding from EEG Signals

  2024cites this paper
• An fMRI dataset in response to large-scale short natural dynamic facial expression videos

  2024cites this paper
• Reanimating Images using Neural Representations of Dynamic Stimuli

  2024cites this paper
• Principles of intensive human neuroimaging.

  2024cites this paper
• An OpenMind for 3D medical vision self-supervised learning

  2024cites this paper
• Healthy aging delays and dedifferentiates high-level visual representations

  2024cites this paper
• Hierarchical organization of social action features along the lateral visual pathway.

  2023cites this paper
• A Systematic Comparison of fMRI-to-video Reconstruction Techniques

  year unknowninfluential citation