Word2Vec: The Geometry of Meaning

This presentation provides a comprehensive, structured exploration of how language can be transformed from discrete symbolic representations to continuous vector spaces through predictive learning. It begins by contrasting traditional frequency-based approaches with modern neural methods, illustrating the shift from isolated word counting to context-driven learning, where meaning emerges from relationships among words rather than from standalone definitions. The work introduces the core philosophy of contextual representation, emphasising that a word’s meaning is defined by its surrounding words, and develops the Word2Vec framework as a unified family of models comprising Continuous Bag-of-Words (CBOW) and Skip-Gram, both operating within a shared neural architecture. It further explains how embedding matrices transform input words into dense vector representations used for context prediction. The presentation addresses the computational limitations of full softmax and introduces negative sampling as an efficient alternative, reformulating learning as a binary classification task between true and noisy word pairs, and supports this with an optimally designed noise distribution based on the 3/4 power law. Through visualisations of training dynamics, cosine similarity matrices, and geometric vector spaces, it demonstrates how semantic relationships emerge during training and can be interpreted through vector arithmetic. The work concludes by positioning Word2Vec as a foundational advancement that bridges statistical language modelling and modern deep learning, establishing continuous vector representations as the basis for contemporary natural language processing systems.