Project 2. Comparable semantic representation in neural and computer systems

Publication

Zhang, L., Wang, L., Yang, J., Qian, P., Wang, X., Qiu, X., ... & Tian, X. (2020). Can computers understand words like humans do? Comparable semantic representation in neural and computer systems. bioRxiv.
https://doi.org/10.1101/843896v2

Research Background

Semantic representation has been studied independently in neuroscience and computer science. A deep understanding of human neural computations and the revolution to strong artificial intelligence appeal for a joint force in the language domain. We investigated comparable representational formats of lexical semantics between these two complex systems with fine temporal resolution neural recordings. We found semantic representations generated from computational models significantly correlated with EEG responses at an early stage of a typical semantic processing time window in a two-word semantic priming paradigm. Moreover, three representative computational models differentially predicted EEG responses along the dynamics of word processing. Our study provided a finer-grained understanding of the neural dynamics underlying semantic processing and developed an objective biomarker for assessing human-like computation in computational models. Our novel framework trailblazed a promising way to bridge across disciplines in the investigation of higher-order cognitive functions in human and artificial intelligence.

fig2-1

Figure 2.1 The Brain Dictionary. Adapted from Huth et al., 2016.

My Contribution: Experiment Design, Preprocessing, Database Construction, Data Collection & Analysis

1. Constructed word stimulus set by filtering out true words from the English Lexicon Project and selecting non-words from Wuggy database (word stimuli dataset, English version)
2. Revised Python codes to present new experimental paradigms
3. Redesigned Python codes to include event codes (triggers)
4. Wrote Python codes to analyze cosine similarity between two words using GloVe models from NLP
5. Conducted EEG experiments independently: recruited participants (n=18), prepared participants' scalps, added EEG cream, used Brain Pycorder to monitor data collection
6. Used easyeeg package to preprocess raw datasets: artifact rejection and correction; averaging; filtering
7. Analyzed preprocessed data to detect the amplitude and latency of ERP waves, GFP, and topography

fig2-2

Fig 2.2 Semantic representations in the human brain and word embedding models

fig2-3

Fig 2.3 Experimental procedure and single-trial correlation analysis

My Relevant Skills

1. neurolinguistics
2. natural language processing, python programming
3. ERP experiments design, EEG preprocessing, EEG data analysis

fig2-4

Fig 2.4 Python programming

Results

fig2-5

• The validity of the EEG data was examined by checking ERP components using data averaged across all trials. ERP waveforms were plotted in several channels. One representative waveform shown here was from channel Cz.
• Early visual responses N1 and P2 as well as semantics-related N400 effects were shown in responses to both target and prime, consistent with well-established literature (Kutas and Federmeier 2011).

fig2-6

• The dynamics of Global Field Power(GFP) was also plotted as a geometric mean of electric potentials across all sensors (Lehmann and Skrandies 1980) to visualize the distributed energy fluctuation.
• Similar evolution of ERP components was observed in the dynamics of GFP which included activity of all sensors, demonstrating the reliability of elicited data without the potential pitfalls of subjective bias.

fig2-7

• The topographies across time for ERP responses to prime and target as well as the differences between the two (i.e., target minus prime) were also plotted to visualize the dynamics of activation patterns. Consistent evolution of response patterns were demonstrated in topographic responses to prime and target, which suggests that common cognitive functions unfolded over time during the reading of these words at prime and target positions. Topographic differences between target and prime showed magnitude differences in sensors over frontal and temporo-parietal regions around 275 ms

fig2-8

• From the correlation between EEG responses in each of the 32 channels at each of the 800 milliseconds and cosine similarities computed from computational models GloVe, we obtained a 32 × 800-dimensional matrix of r values for GloVe.
• To estimate the overall predictability of each model, we averaged the absolute r values across channels, yielding a line of temporal progression of r for each word embedding model.
• We find that In Chinese version, GloVe-generated semantic similarity values significantly correlated with EEG response differences between 226 and 306 ms, while in English version, GloVe-generated semantic similarity values significantly correlated with EEG response differences between around 420 and 440 ms.

fig2-9