How Large Language Models Align with the Brain: A Multidataset Neuroimaging Study

Abstract Natural language processing (NLP) models, especially transformers, are approaching human-level performance. However, it remains unclear whether and how these models align with the human brain. Although some neuroimaging studies using linguistic stimuli have reported significant correlations between brain activity and model representations, interpreting and comparing such findings is challenging due to methodological inconsistencies. Here, we reanalyse six fMRI datasets comprising a total of 74 participants, covering written sentences, written narratives, and audio narratives, using a consistent analysis pipeline. Each dataset is evaluated against a range of NLP models using both voxelwise encoding models and representational similarity analysis. We find positive model-brain correlations across all datasets. Further, we also find that, while transformers typically achieve higher correlations than simple word embedding models, the difference is modest and unrelated to model size or complexity. Our findings raise the possibility that previously reported differences between models are driven by methodological choices rather superior alignment to the brain.