Overview

‘… there are three times: a time present of things past; a time present of things present; and a time present of things future. [...] The time present of things past is memory; the time present of things present is direct experience; the time present of things future is anticipation.’ – St. Augustine in Confessions 4th century.

How does the brain anticipate the future in our dynamic world? 

In dynamic environments (e.g., traffic or sports), our brain is faced with a continuous stream of changing sensory input. Adaptive behavior in such environments requires our brain to predict unfolding external sensory dynamics. Imagine trying to catch a ball thrown at you. While predictive processing theories propose such dynamic prediction, empirical evidence is often limited to static snapshots and indirect consequences of predictions, and studies often use simple, highly controlled, (static) stimuli. 

One important reason is that naturalistic cognitive neuroscience comes with methodological challenges:

  • what features of complex naturalistic input does the brain represent?
  • when does the brain represent a given time point of continuous input?
  • noisy neuroimaging signals necessitate many repetitions
  • gaze fixation is often required

In the Naturalistic Prediction Group, we try to tackle these challenges and answer important outstanding questions about how our brain anticipates the future in a dynamic world. 

The general strategy we follow is to quantify how well patterns of continuous brain activation (measured with M/EEG, but also indirectly with eye-movements) match information contained in naturalistic continuous stimuli such as audio-visual movies (captured with theoretical or computational stimulus models).  

Research Directions

  • How does familiarity with input affect feature-specific neural prediction in naturalistic dynamic input? In this project we investigate how familiarity with the input affects naturalistic predictions. In a first (finished) study we presented participants in the MEG scanner with videos of action sequences and found that reducing stimulus familiarity by either up-down inversion or temporal piecewise scrambling of the action sequences, impairs neural predictions in a hierarchical level-specific manner, such that inversion selectively impairs predictions at the highest hierarchical level, while piecewise scrambling impairs all predictions. In a second (currently ongoing) study we ask how higher-level semantic/narrative familiarity affects predictions. We tackle this question by presenting participants with 42 minutes of a naturalistic audio-visual movie two times in a row and aim to compare what our brain predicts when in the first vs. second repetition. Currently 40 participants have been collected, and an additional 12 are planned for this summer, thus resulting in a large valuable dataset of 52 participants watching a naturalistic audiovisual movie twice. This data could also be used to answer other interesting research questions. 
     
  • How do high-level conceptual predictions influence low-level perceptual predictions in naturalistic movie viewing? In this research line we investigate whether rich semantic/narrative context improves low-level perceptual predictions. We additionally investigate whether we can capture high-level (semantic) predictions (i.e., what will happen in a story)? For this research line we use the same 42-minute audiovisual movie data as presented above and aim to retrieve continuous semantic representations from the movie using artificial neural networks. We will then investigate whether and when this semantic information is represented in the brain. Additionally, we ask whether better semantic knowledge/predictions also improve low-level perceptual predictions. 
     
  • How do attention and familiarity with stimulus dynamics influence prediction across hierarchical levels in naturalistic movie watching? In this ongoing study we first ask again the question of familiarity, but regarding low-level (pixelwise) dynamics. In previous studies we found evidence that these low-level dynamics are predicted. Here we ask whether we are able to do so because we are highly familiar with the dynamics of the world around us. In this study we showed 4 different 16-minute silent movie segments, 2 of which are played backwards in time, to reduce the observer’s familiarity with low-level stimulus dynamics. Second, we ask how attention affects naturalistic predictions for both low-level perceptual and high-level semantic features. In 2 out of 4 movie segments participants pay attention to the movie, while in the other 2, although receiving the exact same input, they do not pay attention to the movie. One expectation is that low-level perceptual predictions happen automatically, and therefore independent of attention, while higher-level predictions rely on attention. When data collection for this study is finished, it will encompass an additional valuable MEG dataset with 52 participants. 
     
  • Category-specific distractibility of visual working memory - In this study we investigate how vulnerable information held in visual working memory is to visual distraction, and whether this vulnerability depends on the content of the distraction. Using behavioral performance measures (accuracy and reaction time), we already found that the amount of distractibility depends on whether the object category of the distractor matches the visual working memory content, suggesting that distraction in part takes place at the hierarchical level of object category, rather than only at low perceptual levels. This dataset also contains eye-tracker data, and one open question is whether we can find the same pattern of distractibility in the pupil dilation in response to the distractor, i.e., is pupil dilation larger for category-matching distractors? 

Members

  • Ingmar de Vries, Principal Investigator
  • Tiziano Causin, Master student
  • Pardis Rahanandeh, Master student
  • Thuy Thien Thanh Tran, Master student

Publications

  • De Vries, I.E.J. & Wurm, M.F. (2023) Predictive neural representations of naturalistic dynamic input. Nature Communications 14 (1), 3858
  • De Vries, I.E.J, Marinato, G., Baldauf, D. (2021) Decoding object-based auditory attention from source-reconstructed MEG alpha oscillations. Journal of Neuroscience 41 (41), 8603-8617
  • De Vries, I.E.J, Slagter, H.A., Olivers, C.N.L. (2020) Oscillatory control over representational states in working memory. Trends in cognitive sciences 24 (2), 150-162

For a complete list, see Ingmar de Vries’ personal page or my Google Scholar page.

Ongoing Collaboration

  • Dr. Moritz Wurm - Centre for Mind/Brain Sciences, University of Trento, IT
  • Prof. Floris de Lange - Donders Centre for Cognitive Neuroimaging, Radboud University, NL
  • Dr. Christoph Huber-Huber - Centre for Mind/Brain Sciences, University of Trento, IT
  • Dr. Eva Berlot - Donders Centre for Cognitive Neuroimaging, Radboud University, NL
  • Dr. Daniel Baldauf - Centre for Mind/Brain Sciences, University of Trento, IT

The PI is also involved in several ongoing worldwide replication efforts that try to estimate the scope of the replication crisis within EEG research, specifically: 

  • Lead analyst in three #EEGmanylabs replication efforts. 
  • One of hundreds of worldwide analysts in the #EEGmanypipelines project 
  • One of hundreds of worldwide analysts in the #EEGmanyanalyst project