Visual decoding and encoding are crucial aspects in investigating the representation of visual information in the human brain. This paper proposes a bidirectional model for decoding and encoding of visual stimulus based on manifold representation of the temporal and spatial information extracted from magnetoencephalographic data.
In the proposed decoding process, principal component analysis is applied to extract temporal principal components (TPCs) from the visual cortical activity estimated by a beamforming method. The spatial distribution of each TPC is in a high-dimensional space and can be mapped to the corresponding spatiotemporal component (STC) on a low-dimensional manifold. Once the linear mapping between the STC and the wavelet coefficients of the stimulus image is determined, the decoding process can synthesize an image resembling the stimulus image. The encoding process is performed by reversing the mapping or transformation in the decoding model and can predict the spatiotemporal brain activity from a stimulus image.
In our experiments using visual stimuli containing eleven combinations of checkerboard patches, the information of spatial layout in the stimulus image was revealed in the embedded manifold. The correlation between the reconstructed and original images was 0.71 and the correlation map between the predicted and original brain activity was highly correlated to the map between the original brain activity for different stimuli (r = 0.89). These results suggest that the temporal component is important in visual processing and manifolds can well represent the information related to visual perception.
Figure 1. The proposed bidirectional model for decoding and encoding. The model contains manifold and wavelet representation for brain activity and stimulus image, respectively. In the decoding process, TPCs are calculated by applying PCA to the time course of brain activity. LLE is then applied to the spatial distribution of each TPC to obtain STC lying on a manifold. Following linear mapping, STC is transformed into wavelet coefficients, which are used to reconstruct the stimulus image through the weighted summation of wavelet functions. In the encoding process, a test stimulus image is decomposed into wavelet coefficients according to the wavelet representation. The coefficients are then transformed into STCs using the inverse of linear mapping. Then reverse LLE is applied to the STC to obtain TPCs and the brain activity is predicted by applying inverse PCA to the TPCs.
Figure 2. Results of stimulus image reconstruction. (a) Presented and reconstructed images of each single trial data for Subject S1. Each stimulus image was presented for 250 ms and MEG data from 60 ms to 160 ms after the stimulus onset were used to reconstruct the image. (b) Reconstructed stimulus images after averaging and binarization. Presented are the single trial results of eleven stimulus images. The averaged image was obtained from five reconstructed images and the results after binarization are presented in the rightmost column. (c) Correlation map showing the similarity between the presented and reconstructed images. Similarity value was calculated as the spatial correlation coefficient of presented and reconstructed images for each subject followed by averaging across seven subjects. (d) Similarity between the presented and reconstructed images for each subject. Similarity values for the basic and composite image sets are indicated by black and gray bars, respectively, and the white bars present mean similarity values across all stimulus images. (e) Correlation coefficient map between the presented stimulus images. The correlation coefficient between the correlation maps shown in (c) and (e) was 0.92 (p b 0.0001).
Determining the means by which perceived stimuli are compactly represented in the human brain is a difficult task. This study aimed to develop techniques for the construction of the neural manifold as a representation of visual stimuli. Approach. We propose a supervised locally linear embedding method to construct the embedded manifold from brain activity, taking into account similarities between corresponding stimuli. In our experiments, photographic portraits were used as visual stimuli and brain activity was calculated from magnetoencephalographic data using a source localization method.
The results of 10 × 10-fold crossvalidation revealed a strong correlation between manifolds of brain activity and the orientation of faces in the presented images, suggesting that high-level information related to image content can be revealed in the brain responses represented in the manifold. Significance. Our experiments demonstrate that the proposed method is applicable to investigation into the inherent patterns of brain activity.