Sem2 Week 9

Because there is no actual link between a word and its real-world meaning — vector semantics only approximates meaning through patterns of context, not through any genuine understanding

The problem of connecting a symbol (like the word ‘cat’) to its actual real-world referent. Vector semantics has no symbol grounding — it only relates symbols to other symbols

The idea that humans ground symbols in meaning through physical experience of the world via our bodies and senses — something pure vector semantics lacks entirely

A thought experiment: if you follow rules to correctly answer questions in a language you don’t understand, does that count as understanding? Searle argues no — correct behaviour does not imply genuine comprehension

Both manipulate symbols according to rules/patterns without any grounding in real-world meaning — they can produce correct outputs without understanding in Searle’s sense

Synonyms (close in space), similarity rankings (angle between vectors), and different senses of the same word (different regions of the semantic space)

A vector that is mostly zeros — in count-based context vectors this happens because most words do not co-occur with most other words

They are treated as completely unrelated dimensions even though they are synonyms — the vector has no way to know that co-occurring with ‘car’ and co-occurring with ‘automobile’ are essentially the same information

• Learning the embedding; 2. Attention / transformers; 3. Fine-tuning to build a better chatbot

A dense, low-dimensional vector representing a word, learned by a neural network rather than counted directly — the dimensions are abstract learned features, not individual words

The unit the model works with — can be a whole word, a sub-word fragment, or a common multi-word phrase. The model can learn its own tokenisation from data

A neural network trained to predict which words appear near a given target word. The hidden layer weights learned during training become the word embeddings

Input: one-hot vector of size N (vocabulary); Hidden layer: 300 linear neurons; Output layer: softmax classifier with N neurons giving probabilities over the vocabulary for likely neighbouring words

A learning method where the correct answers are already present in the raw data — no manual labelling needed. Word2Vec is self-supervised because neighbouring words in text serve as the training targets

They are dense and low-dimensional (e.g. 300 vs 50,000), capture relationships between similar words like synonyms, and generalise better — ‘car’ and ‘automobile’ end up close in the space

Neural network embeddings emerged in the 2000s; Word2Vec specifically was introduced in 2013 in ‘Efficient Estimation of Word Representations in Vector Space’

Buckets = learned embedding dimensions; recipe book = training data; words (apple, cherry, butter) = tokens fed into the model

Too few: the model cannot capture enough distinctions between words (underfitting). Too many: the vectors become sparse again and the model may overfit or fail to generalise

Each word gets one fixed embedding regardless of context — so ‘litter’ (cat litter vs littermates) maps to the same vector even when its meaning is completely different

Attention generates a new embedding for each word that is adjusted based on its specific context within a particular sentence, rather than using a single average embedding across all uses

A mechanism that evaluates different word groupings within a sentence and assigns context-sensitive meanings to groups of words according to how they are grammatically combined

A method for projecting high-dimensional vectors down into 2D so that relationships between words can be visualised

Paris - France + Italy = Rome; the direction between country and capital is consistent across different countries, showing that relational structure is encoded geometrically in the embedding space

‘man’ - ‘computer programmer’ + ‘woman’ yields ‘homemaker’ — the model reflects gender biases present in its training data (news text)