an embedding is a list of numbers that says where something lies on a map of meaning.
your sheet: 2 numbers per card · the model*: 384 numbers per token and per text
before a computer
can search for
meaning, it has to
know where each
text belongs.
can search for
meaning, it has to
know where each
text belongs.
image: ai-generated (gpt-image-2.5-flare)
close in meaning, close on the map
a place is a list of numbers
key idea
* all-MiniLM-L6-v2 · huggingface.co/sentence-transformers/all-MiniLM-L6-v2
a table with one row per token
similarity is the angle between two arrows
* all-MiniLM-L6-v2 · huggingface.co/sentence-transformers/all-MiniLM-L6-v2
measured with the model*
* all-MiniLM-L6-v2 · huggingface.co/sentence-transformers/all-MiniLM-L6-v2
king − man + woman
the same works elsewhere
paris − france + germany
→
berlin 0.79
puppy − dog + cat
→
kitten 0.87
doctor − hospital + school
→
teacher 0.82
walked − walk + swim
→
swimming 0.91 swam 0.88
not always perfect
what the model* reads
* all-MiniLM-L6-v2 · huggingface.co/sentence-transformers/all-MiniLM-L6-v2
four steps from tokens to one list
the same token, two meanings
* all-MiniLM-L6-v2 · huggingface.co/sentence-transformers/all-MiniLM-L6-v2
where the numbers come from
the idea behind it
“you shall know a word by the company it keeps.”
John Rupert Firth, 1957
384 directions, drawn on two
search by meaning
one word as the question
key idea
a search by meaning finds texts that share no word with the question.
about the same, or the same?
key idea
a map of meaning knows what a text is about, but not whether it is true.
similarity is one sum
how a language model picks the next token
key idea
a language model predicts where the next token should lie on its map and takes a token that lies close to that place.