What is an embedding?

What is an embedding?

An embedding is a list of numbers, a vector, that stands in for a piece of data — a word, a sentence, an image. It lets a computer measure how alike two things are. Items with similar meaning end up with similar vectors, placed close together in a high-dimensional space. Distance in that space becomes a proxy for semantic similarity.

The idea predates deep learning. But Mikolov et al. (2013) made it practical at scale with word2vec, training vectors for words from a 1.6-billion-word corpus in under a day. Since then the same trick has been extended from single words to whole sentences and, separately, to combinations of text and images.

How do word vectors work?

Word2vec, proposed by Mikolov et al. (2013), learns a vector for each word by having a simple model predict context from a word, or a word from its context, over a huge amount of text. The paper introduced two architectures. Continuous bag-of-words (CBOW) predicts the current word from surrounding words, while continuous skip-gram predicts surrounding words from the current one. Skip-gram, trained on 783 million words at 300 dimensions, scored 55.9% on syntactic word-relationship questions and 50.0% on semantic ones in the paper's own test set. That was notably stronger on semantic relationships than CBOW's 15.5%.

The paper's best-known finding is that the resulting vectors encode relationships as directions, not just proximity. Subtracting the vector for "Man" from "King" and adding "Woman" lands near the vector for "Queen". The same arithmetic recovers country-capital pairs, such as Paris minus France plus Italy landing near Rome. Mikolov et al. trained their largest models, 1000 dimensions on 6 billion words, using a distributed system called DistBelief, reaching 65.6% total accuracy on their combined semantic-syntactic test.

How do sentence embeddings differ from word embeddings?

A word embedding covers one word; a sentence embedding compresses a whole sentence into one fixed-size vector so that two sentences can be compared directly. Reimers and Gurevych (2019) showed that BERT produces poor sentence vectors by default, despite setting state-of-the-art scores on sentence-pair tasks. Averaging its output layer scored only 54.81 on seven semantic textual similarity (STS) benchmarks. That was worse than simply averaging older GloVe word vectors, which scored 61.32.

Their fix, Sentence-BERT (SBERT), adds a pooling step and fine-tunes BERT with a siamese network on natural language inference data. A siamese network is two copies of the same network, sharing weights, processing two sentences at once. SBERT-large reached an average of 76.55 on the same seven STS tasks, beating the Universal Sentence Encoder's 71.22 and InferSent's 65.01.

The gain is as much about speed as accuracy. Finding the most similar pair among 10,000 sentences takes BERT about 65 hours of pairwise comparison, because every pair must be fed through the network together. SBERT instead embeds each sentence once and compares vectors with cosine similarity. That cuts the same search to about 5 seconds.

What about multimodal embeddings?

Neither paper reviewed here covers multimodal embeddings — vectors that place images, audio or other data types in the same space as text. That topic is outside the documents this page draws on.

How are embeddings used for search?

Embedding-based search, often called semantic or vector search, finds matches by nearness in vector space rather than by shared keywords. Reimers and Gurevych frame this directly: once sentences are embedded, comparison costs almost nothing. A collection can be indexed and queried without passing every pair through a heavy network. They cite Johnson et al.'s (2017) optimised index structures as a way to cut a search over 40 million candidate questions from 50 hours to a few milliseconds.

This separation of embedding, which is slow and done once per item, from comparison, which is fast and done per query, is what makes embeddings practical for retrieval, clustering and deduplication at scale. SBERT's authors note it also helps for clustering, something the BERT cross-encoder architecture cannot do without that quadratic blow-up of pairwise passes.

What are the limits of these embeddings?

Mikolov et al. acknowledge their accuracy metric demands an exact match to the correct word. So perfect scores are effectively impossible, since their model has no notion of word morphology and synonyms count as errors. Word2vec also assigns one vector per word regardless of context, so a word's different senses are blended into a single point in space.

SBERT has its own failure mode. On the Argument Facet Similarity corpus, which asks whether two arguments share both a claim and a reasoning, cross-topic evaluation showed SBERT's score drop by about 7 points of Spearman correlation. That was compared with BERT's cross-encoder, which can directly compare sentences word-by-word rather than relying on separately computed vectors. Vectors built for speed still trade away some of that direct comparison.