2. Vectors

Why this layer exists

A language model never sees a token. It sees a row of a matrix. The embedding layer of every model you run is a matrix of shape (V, d), one row per vocabulary entry; a token id selects a row, and from that point until the final projection back onto the vocabulary, everything the model computes is arithmetic on vectors of length $d$. When you call /v1/embeddings on a llama-server instance, the JSON array of floats you get back is a vector of the same kind: not a row of the embedding matrix, but the model’s final hidden states, each of length $d$, pooled over the input into one.

The operation on those vectors that the rest of the book keeps returning to is the dot product. Attention, which chapter 10 derives as a soft dictionary lookup, scores every earlier position by the dot product of a query vector with a key vector. Retrieval, which Part V builds into the agent’s memory, embeds a query with dixie’s honcho-embed, compares it against every stored vector by cosine similarity (chapter 28), and then replaces the exhaustive comparison with an index that approximates it (chapter 29). A similarity threshold in a memory config, the scores a top-k retrieval ranks by, and an attention score before its softmax are all the same quantity: a sum of elementwise products, possibly divided by two lengths.

This chapter builds that quantity from the definition, derives why it measures an angle, and uses it to build a first embedding: word vectors from nothing but counts of which words appear near which others.

Mechanism

Vectors as lists and as arrows

A vector $\mathbf{a}$ of dimension $d$ is an ordered list of $d$ real numbers, shape (d,):

$\mathbf{a} = (a_1, a_2, \ldots, a_d) \in \mathbb{R}^d.$

That is the whole algebraic definition, and it is the one the code uses: a numpy array of length $d$. The geometric reading is an arrow from the origin to the point with those coordinates. In two or three dimensions you can draw it; in 1,024 you cannot. The pictures in this chapter are two-dimensional, and so is the one derivation that uses a picture: any two vectors span a plane, and the angle identity below is derived in coordinates chosen for that plane. Every other statement is derived from the list form and holds in any dimension.

Two operations define the space. Addition is elementwise: for $\mathbf{a}, \mathbf{b}$ both of shape (d,),

$\mathbf{a} + \mathbf{b} = (a_1 + b_1, \ldots, a_d + b_d),$

shape (d,). Geometrically, place the tail of $\mathbf{b}$ at the head of $\mathbf{a}$; the sum is the arrow from the origin to where you end up. Scaling by a scalar $c$ multiplies every entry:

$c\,\mathbf{a} = (c\,a_1, \ldots, c\,a_d),$

shape (d,). A positive $c$ stretches or shrinks the arrow without turning it; a negative $c$ also reverses it. Subtraction is addition of $(-1)\,\mathbf{b}$, and $\mathbf{a} - \mathbf{b}$ is the arrow from the head of $\mathbf{b}$ to the head of $\mathbf{a}$. That last fact carries the derivation below.

Vectors of different lengths cannot be added, and numpy will not always tell you. Adding arrays of shapes (3,) and (1,) broadcasts the single entry across all three and returns a result of shape (3,) with no error. Chapter 3 covers broadcasting properly; in this chapter every function checks its shapes explicitly.

The dot product

The dot product of two vectors of the same dimension is the sum of their elementwise products:

$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{d} a_i b_i .$

Both inputs have shape (d,); the result is a scalar. Three properties follow directly from the definition, and every later derivation leans on them.

1. Symmetry. $\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}$, because $a_i b_i = b_i a_i$ term by term.
2. Linearity in each argument. For scalars $c$ and vectors $\mathbf{a}, \mathbf{b}, \mathbf{e}$ of shape (d,), $(c\,\mathbf{a} + \mathbf{e}) \cdot \mathbf{b} = \sum_i (c\,a_i + e_i) b_i = c \sum_i a_i b_i + \sum_i e_i b_i = c\,(\mathbf{a} \cdot \mathbf{b}) + \mathbf{e} \cdot \mathbf{b}$. By symmetry the same holds in the second argument.
3. Positivity. $\mathbf{a} \cdot \mathbf{a} = \sum_i a_i^2 \ge 0$, since each term is a square.

Read algebraically, the dot product is a weighted sum: $\mathbf{b}$ supplies weights and $\mathbf{a}$ supplies values, or the other way round. That reading is the one attention uses. The geometric reading takes a derivation.

Length

The norm (Euclidean length) of $\mathbf{a}$ is

$\lVert \mathbf{a} \rVert = \sqrt{\mathbf{a} \cdot \mathbf{a}} = \sqrt{\sum_{i=1}^{d} a_i^2 } ,$

a non-negative scalar. In two dimensions this is Pythagoras: the arrow to $(a_1, a_2)$ is the hypotenuse of a right triangle with legs $a_1$ and $a_2$. In three dimensions apply Pythagoras twice, first to $(a_1, a_2)$ and then to that hypotenuse and $a_3$, and the pattern extends one coordinate at a time to any $d$. The square root exists because of positivity. Two consequences used later: scaling a vector scales its length by the absolute value of the scalar,

$\lVert c\,\mathbf{a} \rVert = \sqrt{(c\,\mathbf{a}) \cdot (c\,\mathbf{a})} = \sqrt{c^2\,(\mathbf{a} \cdot \mathbf{a})} = \lvert c \rvert\,\lVert \mathbf{a} \rVert ,$

where the middle step applies linearity once in each argument; and the distance between the heads of two arrows is $\lVert \mathbf{a} - \mathbf{b} \rVert$. A unit vector has norm 1. Any nonzero $\mathbf{a}$ can be turned into one by dividing by its length, $\hat{\mathbf{a}} = \mathbf{a} / \lVert \mathbf{a} \rVert$, shape (d,), and by the scaling rule $\lVert \hat{\mathbf{a}} \rVert = \lVert \mathbf{a} \rVert / \lVert \mathbf{a} \rVert = 1$.

The dot product measures an angle

Take two nonzero vectors $\mathbf{a}$ and $\mathbf{b}$ of shape (d,). However large $d$ is, the two arrows and the origin lie in a single plane, so there is a well-defined angle $\theta \in [0, \pi]$ between them. The claim is

$\mathbf{a} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert \cos\theta .$

The proof computes the length of the third side of the triangle, $\lVert \mathbf{a} - \mathbf{b} \rVert$, twice: once geometrically, once algebraically.

Geometrically. Work in the plane of the two arrows and choose coordinates in it so that $\mathbf{b}$ lies along the first axis. Rotating the picture changes no lengths and no angles. Then the head of $\mathbf{b}$ is at $(\lVert \mathbf{b} \rVert, 0)$, and the head of $\mathbf{a}$, at distance $\lVert \mathbf{a} \rVert$ from the origin and angle $\theta$ from the first axis, is at $(\lVert \mathbf{a} \rVert \cos\theta,\ \lVert \mathbf{a} \rVert \sin\theta)$. The first coordinate is the projection drawn dashed in the figure: how far along $\mathbf{b}$ the head of $\mathbf{a}$ sits. The squared distance between the two heads is the sum of the squared coordinate differences:

$\lVert \mathbf{a} - \mathbf{b} \rVert^2 = \left(\lVert \mathbf{a} \rVert \cos\theta - \lVert \mathbf{b} \rVert\right)^2 + \left(\lVert \mathbf{a} \rVert \sin\theta\right)^2 .$

Expand the first square:

$\lVert \mathbf{a} - \mathbf{b} \rVert^2 = \lVert \mathbf{a} \rVert^2 \cos^2\theta - 2 \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \cos\theta + \lVert \mathbf{b} \rVert^2 + \lVert \mathbf{a} \rVert^2 \sin^2\theta .$

Collect the two $\lVert \mathbf{a} \rVert^2$ terms and use $\cos^2\theta + \sin^2\theta = 1$:

$\lVert \mathbf{a} - \mathbf{b} \rVert^2 = \lVert \mathbf{a} \rVert^2 + \lVert \mathbf{b} \rVert^2 - 2 \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \cos\theta .$

This is the law of cosines. No case split is needed for obtuse angles: when $\theta > \pi/2$, $\cos\theta$ is negative and the head of $\mathbf{a}$ projects behind the origin, which the coordinates handle on their own.

Algebraically. Expand the same squared length with the dot product, using linearity in each argument and then symmetry:

$\lVert \mathbf{a} - \mathbf{b} \rVert^2 = (\mathbf{a} - \mathbf{b}) \cdot (\mathbf{a} - \mathbf{b}) = \mathbf{a} \cdot \mathbf{a} - \mathbf{a} \cdot \mathbf{b} - \mathbf{b} \cdot \mathbf{a} + \mathbf{b} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert^2 - 2\,\mathbf{a} \cdot \mathbf{b} + \lVert \mathbf{b} \rVert^2 .$

Equate the two right-hand sides. $\lVert \mathbf{a} \rVert^2$ and $\lVert \mathbf{b} \rVert^2$ appear in both and cancel, leaving

$-2\,\mathbf{a} \cdot \mathbf{b} = -2 \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \cos\theta ,$

and dividing by $-2$ gives the claim. The dot product is the length of $\mathbf{b}$ times the length of the projection of $\mathbf{a}$ onto it, $\lVert \mathbf{a} \rVert \cos\theta$, signed by which side of the perpendicular it lands on. Three cases are worth having as reflexes:

• $\mathbf{a} \cdot \mathbf{b} > 0$: the angle is acute; the vectors point broadly the same way.
• $\mathbf{a} \cdot \mathbf{b} = 0$: the vectors are orthogonal; neither has any component along the other.
• $\mathbf{a} \cdot \mathbf{b} < 0$: the angle is obtuse; they point broadly apart.

Since $\lvert \cos\theta \rvert \le 1$, the identity also gives $\lvert \mathbf{a} \cdot \mathbf{b} \rvert \le \lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert$, the Cauchy–Schwarz inequality.

Cosine similarity

Solve the identity for the angle’s cosine:

$\cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert} \in [-1, 1] .$

This is cosine similarity, a scalar that depends only on direction. It is undefined when either vector is zero, since a zero vector has no direction; the code raises rather than invent a value.

The raw dot product mixes two things: how aligned the vectors are, and how long they are. Scale $\mathbf{a}$ by 10 and $\mathbf{a} \cdot \mathbf{b}$ grows by 10 while the angle is unchanged. For word vectors built from counts, length tracks frequency: a word seen 10 times as often, in the same mix of contexts, has counts roughly 10 times larger in every coordinate, and its dot product with everything is roughly 10 times larger. Ranking neighbours by raw dot product would rank frequent words first regardless of meaning. Dividing by both norms removes that: cosine similarity is invariant to scaling either argument by a positive constant (exercise c), so a word’s frequency no longer inflates its similarity to anything.

Dividing by the norms is also a factorisation. $\cos(\mathbf{a}, \mathbf{b}) = \hat{\mathbf{a}} \cdot \hat{\mathbf{b}}$: normalise every vector once, and every subsequent cosine is a plain dot product. The nearest function below relies on this.

Words as vectors

The distributional hypothesis is that words occurring in similar contexts have similar meanings. Harris made the structural version of the argument in “Distributional Structure” (1954); Firth’s “A Synopsis of Linguistic Theory” (1957) gave it the line that is usually quoted, “you shall know a word by the company it keeps.” It turns meaning, which cannot be measured, into context, which can be counted.

The simplest way to count it: fix a vocabulary of $V$ words and a window width $w$. For every position in a token stream, look at the tokens up to $w$ positions to the left and right, and for each one add 1 to a table entry. The result is the co-occurrence matrix $\mathbf{C}$, shape (V, V), where

$C_{ij} = \text{number of times word } j \text{ appears within } w \text{ positions of an occurrence of word } i .$

Row $i$, shape (V,), is word $i$‘s vector: its context profile over the whole vocabulary. $\mathbf{C}$ is symmetric, $C_{ij} = C_{ji}$, because the relation “within $w$ positions of” is symmetric: if positions $p$ and $q$ hold words $i$ and $j$ with $0 < \lvert p - q \rvert \le w$, the pair is visited once from $p$, adding 1 to $C_{ij}$, and once from $q$, adding 1 to $C_{ji}$.

A small example makes the hypothesis concrete. The lab’s tests use this corpus:

the cat sat on the mat . the dog sat on the rug . the cat ate . the dog ate .

The sorted vocabulary is . ate cat dog mat on rug sat the. With $w = 1$, cat occurs twice, preceded both times by the and followed once by sat and once by ate. dog has exactly the same neighbours. Both rows are $(0, 1, 0, 0, 0, 0, 0, 1, 2)$, their cosine is exactly 1, and nearest must find dog as the neighbour of cat with similarity 1.0. The test test_nearest_finds_distributional_twin pins that.

What raw counts get wrong

Cosine similarity removes the frequency of the word whose row you are looking at. It does nothing about the frequency of the context words, which are the columns. Any word that takes a determiner has large counts in the the column, and that one column pulls all of those words’ vectors toward the same direction.

The lab corpus (below) shows how strong the effect is. It is generated from a grammar in which animals (cat, dog, bird) take one set of verbs and royals (king, queen, prince) take a disjoint set, so the only thing an animal and a royal share is the function words around them: the, a, ., and the adverbs. With $w = 2$, those five shared columns hold 88% of the squared length of the cat row and of the king row. The verb columns, which are the only place the two groups differ, hold the remaining 12%. The cosine of cat and king comes out at 0.878, against 0.999 or more within each group: high, for two words that never share a verb.

The fix is to reweight each count by how surprising it is, measuring whether word $j$ appears near word $i$ more often than it would if the two were independent. That reweighting is positive pointwise mutual information (PPMI), and deriving it needs probability. Chapter 4 builds that, and one of its exercises applies PPMI to this matrix.

Walkthrough

The reference module is py/tinygpt/vectors.py. Every function accepts anything numpy can turn into an array, including plain lists.

def dot(a, b):
    """Sum of elementwise products of two vectors of equal length."""
    a, b = np.asarray(a, dtype=np.float64), np.asarray(b, dtype=np.float64)
    if a.ndim != 1 or a.shape != b.shape:
        raise ValueError(f"dot needs two vectors of equal length, got {a.shape} and {b.shape}")
    total = 0.0
    for x, y in zip(a, b):
        total += x * y
    return float(total)

The loop is the definition, written out. np.dot(a, b) or a @ b would compute the same number faster, and every later chapter uses them; here the point is that there is nothing inside the operator except this loop. The conversion to float64 fixes the arithmetic type, so integer inputs such as [1, 2, 3] are summed in floating point; float(total) returns a Python float rather than a numpy scalar.

The shape check is explicit because numpy is permissive in ways that hide bugs. a.ndim != 1 rejects a matrix passed where a vector was meant; a.shape != b.shape rejects vectors of different lengths, including a (3,) array paired with a (1,) array that elementwise arithmetic would broadcast. Checking only a.ndim is enough, since equal shapes imply equal dimensionality.

def norm(a):
    """Euclidean length: the square root of a vector's dot product with itself."""
    return float(np.sqrt(dot(a, a)))


def cosine(a, b):
    """Cosine of the angle between a and b, in [-1, 1]."""
    na, nb = norm(a), norm(b)
    if na == 0.0 or nb == 0.0:
        raise ValueError("cosine is undefined for a zero vector")
    return dot(a, b) / (na * nb)

norm is defined through dot, exactly as the mechanism defines it, so a bug in either shows up in both. cosine computes both norms first and refuses a zero vector with a ValueError. The alternatives are returning nan, which propagates silently through any sort or mean it touches, or returning 0, which asserts “unrelated” about a vector that has no direction at all. Raising puts the decision with the caller.

def cooccurrence(tokens, window):
    """Count how often each word appears within `window` positions of each other word.

    Returns (vocab, counts) where vocab is sorted and counts has shape
    (len(vocab), len(vocab)); counts[i, j] is the number of times vocab[j]
    appeared near vocab[i].
    """
    if window < 1:
        raise ValueError("window must be at least 1")
    vocab = sorted(set(tokens))
    index = {w: i for i, w in enumerate(vocab)}
    counts = np.zeros((len(vocab), len(vocab)))
    for i, w in enumerate(tokens):
        for j in range(max(0, i - window), min(len(tokens), i + window + 1)):
            if j != i:
                counts[index[w], index[tokens[j]]] += 1
    return vocab, counts

The vocabulary is sorted so the row order, and therefore every test and every printed result, is deterministic across runs and Python versions; set iteration order is not. The inner loop visits every position in the window except the centre, clipped at both ends of the token stream. Clipping removes only positions that do not exist, so the symmetry argument above holds at the edges too: a pair near the start of the stream is still visited from both of its positions.

The cost is $O(N w)$ time for $N$ tokens and $O(V^2)$ memory for the dense matrix. For the 17-word lab corpus that is 289 floats. For a 50,000-word vocabulary it is $2.5 \times 10^9$ floats, 20 GB in float64, almost all of them zero. Real systems use sparse matrices or, more usually, skip the count matrix entirely and learn dense vectors of a few hundred to a few thousand dimensions, which is what embedding models such as the one on dixie produce.

def normalize_rows(m):
    """Scale each row to unit length; all-zero rows stay zero."""
    m = np.asarray(m, dtype=np.float64)
    lengths = np.sqrt((m * m).sum(axis=1, keepdims=True))
    return np.divide(m, lengths, out=np.zeros_like(m), where=lengths > 0)


def nearest(matrix, vocab, word, k):
    """The k words whose rows are most cosine-similar to word's row, best first."""
    if word not in vocab:
        raise KeyError(f"{word!r} is not in the vocabulary")
    unit = normalize_rows(matrix)
    q = vocab.index(word)
    sims = unit @ unit[q]  # (V, d) @ (d,) -> (V,)
    ranked = sorted(
        ((vocab[i], float(sims[i])) for i in range(len(vocab)) if i != q),
        key=lambda pair: (-pair[1], pair[0]),
    )
    return ranked[:k]

normalize_rows divides each row of $\mathbf{M}$, shape (V, d), by its length. lengths has shape (V, 1) because of keepdims=True, so the division broadcasts one length across each row: this is the deliberate kind of broadcasting, and the explicit shape makes it visible. The where=lengths > 0 argument leaves an all-zero row as zeros rather than dividing by zero. That is a different policy from cosine, and on purpose: normalize_rows is a batch operation over a whole vocabulary, where one word that never appeared with anything should not abort the rest, and a zero row has similarity 0 to everything, which among count vectors ranks it last.

nearest normalises once and then does all of its work in one line of the listing above, sims = unit @ unit[q], commented (V, d) @ (d,) -> (V,). The shape comment names the general case, a (V, d) embedding matrix; for a co-occurrence matrix $d = V$, and for an embedding model $d$ is its output width. Write $\mathbf{U}$ for unit, shape (V, d), and $\hat{\mathbf{u}}_i$ for its row $i$, shape (d,). A matrix times a vector produces one entry per row of the matrix, and entry $i$ is that row dotted with the vector:

$(\mathbf{U} \hat{\mathbf{u}}_q)_i = \sum_{m=1}^{d} U_{im} \, (\hat{\mathbf{u}}_q)_m = \hat{\mathbf{u}}_i \cdot \hat{\mathbf{u}}_q = \cos(\text{row } i, \text{row } q) ,$

where the last step is the factorisation from the mechanism: both rows are unit length, so their dot product is their cosine. One matrix–vector product, shape (V,), holds the cosine of the query against every word in the vocabulary, itself included. Chapter 3 treats matrix–vector products as a subject in their own right; this is the first place one earns its keep.

The ranking excludes the query’s own row, sorts by descending similarity, and breaks ties by word so equal scores come out in a fixed order. Slicing with [:k] caps the result at the number of candidates, so k=100 on a 9-word vocabulary returns 8 pairs rather than failing. The full sort is $O(V \log V)$; np.argpartition would find the top $k$ in $O(V)$, and chapters 28 and 29 care about that difference. An unknown word raises KeyError, the same exception a dictionary lookup would.

Exercises

(a) Compute $\cos(\mathbf{a}, \mathbf{b})$ by hand for $\mathbf{a} = (1, 2, 2)$ and $\mathbf{b} = (0, 3, 4)$, and the angle between them.

(b) Prove that $\lVert \mathbf{a} \rVert = 0 \Leftrightarrow \mathbf{a} = \mathbf{0}$.

(c) Show that $\cos(c\,\mathbf{a}, \mathbf{b}) = \cos(\mathbf{a}, \mathbf{b})$ for any $c > 0$, and by symmetry the same for scaling $\mathbf{b}$. What happens for $c < 0$?

(d) In high dimensions, random vectors are nearly orthogonal. Draw 10,000 pairs of vectors with independent standard Gaussian entries for $d = 2, 10, 100, 1000$, compute the cosine of each pair, and report the mean absolute cosine and the standard deviation. Explain the trend.

import numpy as np

rng = np.random.default_rng(0)
for d in [2, 10, 100, 1000]:
    a = rng.standard_normal((10_000, d))
    b = rng.standard_normal((10_000, d))
    cos = (a * b).sum(axis=1) / (np.linalg.norm(a, axis=1) * np.linalg.norm(b, axis=1))
    print(f"d={d:5d}  mean |cos| = {np.abs(cos).mean():.3f}  std = {cos.std():.3f}  1/sqrt(d) = {1 / np.sqrt(d):.3f}")

d=    2  mean |cos| = 0.633  std = 0.705  1/sqrt(d) = 0.707
d=   10  mean |cos| = 0.258  std = 0.315  1/sqrt(d) = 0.316
d=  100  mean |cos| = 0.079  std = 0.099  1/sqrt(d) = 0.100
d= 1000  mean |cos| = 0.025  std = 0.032  1/sqrt(d) = 0.032

(e) For unit vectors $\mathbf{a}$ and $\mathbf{b}$, derive $\lVert \mathbf{a} - \mathbf{b} \rVert^2 = 2 - 2\cos\theta$, and explain why ranking by Euclidean distance and ranking by cosine similarity give the same order.

Lab

Implement every function in py/labs/ch02/starter.py, then run

make lab CH=02 IMPL=mine

until it passes. Read py/labs/ch02/test_lab.py first. It fixes details the prose leaves open: dot must raise ValueError for mismatched shapes, cosine must raise ValueError for a zero vector, cooccurrence must reject a window below 1, and nearest must exclude the query word, return results in descending order of similarity, and raise KeyError for a word outside the vocabulary.

The last test runs on py/data/tiny-corpus.txt, 2,000 sentences generated by py/data/make_tiny_corpus.py. The generator uses a fixed random seed, so the file is reproducible; the two noun groups never share a verb, which gives distributional similarity real structure to find. A few lines of it:

a queen quickly commands a king .
the queen quickly rules a queen .
a cat quickly bites the bird .

Once your implementation passes, look at what it learned. Save this script outside the repository, for example as /tmp/neighbours.py:

import pathlib

from tinygpt.vectors import cooccurrence, nearest

tokens = pathlib.Path("data/tiny-corpus.txt").read_text().split()
vocab, counts = cooccurrence(tokens, window=2)
print(len(tokens), "tokens,", len(vocab), "words:", " ".join(vocab))
for word in ["king", "cat", "quickly", "the"]:
    hits = nearest(counts, vocab, word, k=3)
    print(f"{word:8}", "  ".join(f"{w} {s:.3f}" for w, s in hits))

and run it from py/ with PYTHONPATH=. uv run python /tmp/neighbours.py. As written it imports the reference; change the import to from labs.ch02.starter import cooccurrence, nearest to run yours, and the output must be identical:

13332 tokens, 17 words: . a bird bites cat chases commands dog king prince queen quickly quietly rules sniffs summons the
king     queen 1.000  prince 0.999  dog 0.879
cat      dog 0.999  bird 0.999  queen 0.879
quickly  quietly 0.998  . 0.935  queen 0.726
the      a 0.999  bird 0.557  king 0.556

Each line is the distributional hypothesis at work, and each is worth explaining before moving on.

• king and cat find their own group first, at similarities of 0.999 and above. Within a group the words are interchangeable in the grammar, so their context profiles differ only by sampling noise.
• quickly neighbours quietly although the generator treats them as unrelated strings. They fill the same slot, between a subject noun and a verb, so they see the same contexts. The model has no notion of meaning beyond “can stand in the same place”, and here that is enough to recover a part of speech.
• the neighbours a for the same reason: both fill the determiner slot. The two never appear within two positions of each other (a determiner is always separated from the next one by at least a noun and a verb, or a noun and a full stop), so their similarity comes entirely from shared contexts, not from co-occurring.
• The third neighbour of king is dog, at 0.879, and the third of cat is queen. The groups are separated, but by a margin of 0.12, and the cross-group similarity is high because, as the mechanism showed, 88% of each row’s squared length sits in the columns for the, a, ., and the adverbs. The same effect puts . second for quickly: both are dominated by determiner counts. PPMI, in chapter 4, is the correction.

Optionally, point the same code at your own writing. Change the script to take a directory as a command-line argument, read every text file under it, and tokenise with re.findall(r"[a-z']+", text.lower()). Keep the path out of the repository and out of anything you commit. Before calling cooccurrence, drop every token that is not among the 2,000 most frequent: the dense (V, V) matrix for 2,000 words is 32 MB, while an unrestricted vocabulary of 50,000 words would need 20 GB. Then ask for the neighbours of words you use often, and see how many of the top hits are neighbours in meaning and how many are neighbours only because they sit next to the same function words.

Further reading

• Zellig S. Harris, “Distributional Structure” (1954). The structural argument that the distribution of a word over contexts carries information about its meaning.
• J. R. Firth, “A Synopsis of Linguistic Theory, 1930–1955” (1957). The source of “you shall know a word by the company it keeps.”
• Gilbert Strang, Introduction to Linear Algebra (5th ed., 2016), chapter 1. Vectors, lengths, dot products, and the angle between two vectors.
• Supplementary video series: 3Blue1Brown, Essence of Linear Algebra (2016). The geometric picture of vectors, and of dot products as projections, drawn in motion.