Word embeddings (Word2Vec & GloVe)

Click any word. Its nearest neighbor lights up — and it's always the word that's closest in meaning, not spelling or length. "King" sits nearest to "prince," not to "man" or "queen," because this toy space has two axes: how royal, and which gender. Real embeddings have hundreds of dimensions learned from text, but the idea is identical — position encodes meaning, and nearby points mean similar things.

With word vectors $a$ and $b$, distance is just ordinary Euclidean distance:

• $d(a, b)$ — the Euclidean distance between two word vectors.
• $a, b$ — two word vectors being compared (e.g. the embedding coordinates of two different words).
• $a_1, a_2, \dots$ — the individual coordinate values of vector $a$ along each embedding dimension.
• $b_1, b_2, \dots$ — the individual coordinate values of vector $b$ along each embedding dimension.

1. Nearest-neighbor search

   A nearest-neighbor search is nothing more than computing this for every other word and keeping the smallest.

2. The formula isn't the magic

   What makes embeddings powerful isn't the distance formula — it's that a well-trained embedding space arranges words so this simple formula lines up with human judgments of similarity.

Click through several words and read the full ranked list. Notice the axes this space actually encodes: moving along one direction changes royalty (commoner → prince → king), moving along another changes gender (woman → man). Nobody labeled those axes by hand — they just happen to be the directions distance is measured along.

If position encodes meaning, then directions should too. Take king's vector, subtract man's, add woman's: $\text{king} - \text{man} + \text{woman}$.

1. Subtract man from king

   Removes whatever direction encodes "male royalty $\to$ male."

2. Add woman

   Adds back "female" in its place — leaving pure "female royalty."

3. Land on queen

   In this space, that arithmetic lands exactly on queen's coordinates. This is the single most famous fact about word embeddings: analogies work as vector arithmetic, because the relationship between two words is encoded as a direction, and that direction transfers to other word pairs.