Exploring spatio-temporal neural basis of scene processing with MEG/EEG using a convolutional neural network

摘要

Human brains can efficiently process rich information in visual scenes. The neural mechanisms underlying such proficiency may involve not only feedforward processing in the hierarchical visual cortex, but also top-down feedback. To understand these mechanisms, we explored the nature of the visual scene features processed at different brain locations and different time points using high-temporal-resolution MEG and EEG - in separate sessions - while participants viewed briefly presented (200ms) photographs of scenes. We used linear regression to quantify the correlations between neural signals and visual features of the same images, where these features were derived from a convolutional neural network (CNN) with 8 hierarchically organized layers. Next we tested whether variance in the neural signals was explained at each time point and each location by features in different layers, thereby creating a spatio-temporal profile describing the significance of correlation with different CNN layers. For both the MEG and EEG sensor data, we observed that the majority of layers exhibited significant correlations from 60~450 ms after the stimulus onset. When contrasting low-level Layer1 with higher-level Layer6, we found that Layer1 demonstrated greater significance early on (before 120 ms), while Layer6 showed greater significance somewhat later (after 150 ms). In a preliminary analysis of source localized MEG data, we again observed sustained significance for the majority of layers, as well as early greater significance of Layer1 in lower-level visual cortex and later greater significance of Layer6 in higher-level visual cortex. This early to late, lower- to higher-level progression indicates feedforward information flow. Additionally, the sustained significance of low- and high-level layers, which was maintained until at least 400 ms, indicates possible non-feedforward neural responses during scene processing. We are also using connectivity analysis to further investigate if there is top-down feedback from frontal lobe and inferior temporal lobe to the visual cortex.