Tim VanReenen

linkedin.com/in/tvanreenen | github.com/tvanreenen

Embeddings — A Mathematical Rosetta Stone for Generative AI

This is one of those posts that I put together for my own edification in my personal Obsidian vault. But it ulimately derived from a Jupyter notebook that I put together when I was first trying to understand some of the foundational concepts behind generative AI.

One of the first big questions I had was:

How do you turn open-ended input — like text, audio, or video — into something a predictive model like GPT can actually work with?

The answer begins with embeddings.

Embedding Concepts in Space

In mathematical terms, embeddings are represented as vectors — long lists of numbers where each value reflects how much a concept aligns with a particular, learned dimension of meaning.

For example, let's take the phrase "hot dog" and see what its embedding looks like when processed through OpenAI's text-embedding-3-small model:

embedding('hot dog')

This returns a vector — an array of 1,536 numbers — that uniquely points to the concept of "hot dog" in high-dimensional space. Each number describes how strongly "hot dog" aligns with a particular learned dimension of meaning.

array([-0.03104307, -0.04807128, -0.01409304, ..., -0.00020377, 0.0029014 , -0.0072846 ], shape=(1536,))

Understanding Dimensionality

But what do all the number really mean? Where GPS has 2 dimensions (latitude and longitude), and physical space has 3 (x, y, z), large language models like GPT operate in high-dimensional space — in this case, 1,536 dimensions.

The impact of each dimension on meaning isn't explicitly labeled (like "happiness" or "animal-ness"). But during training, the model learns to organize these dimensions in ways that reflect meaning. Each one ends up capturing some subtle influence on a concept's overall representation.

Comparing Similarity

Beyond embedding a single concept in space. A powerful feature of embeddings is that we can compare concepts mathematically. One popular technique is cosine similarity, which measures the angle between two vectors rather than their literal distance.

Cosine similarity returns values from -1 to 1:

• Closer to 1 indicates that the vectors are pointed in the same direction (meaning very similar)
• Closer to 0 indicates that they are orthogonal or have no similarity
• Closer to -1 indicates that they are in exactly opposite directions (meaning they are dissimilar)

With concepts represented mathematically, this one little tool gives us the ability to uncover how concepts are related beneath the surface — not just by spelling or syntax, but by highly nuanced dimensions of meaning.

To explore this, let's start with a simple list of words (or tokens).

words = [
    "man", "woman", "king", "queen", "boy", "girl", "prince", "princess",
    "father", "mother", "husband", "wife", "son", "daughter", "friend",
    "enemy", "teacher", "student", "boss", "employee", "doctor", "nurse",
    "potatoes", "cheese", "hamburger", "french fries", "cheeseburger",
    "banana", "chocolate", "cake", "cheesecake", "bacon", "pancakes",
    "avocado", "salad", "peanut butter", "toast", "burrito", "sushi",
    "pizza", "ice cream", "cookie", "hot dog",
    "soda", "water", "milk", "la croix", "coffee", "latte", "espresso",
    "mocha", "tea", "hot chocolate", "cappuccino", "matcha", "smoothie",
    "kombucha", "wine", "beer", "cocktail", "lemonade",
    "red", "blue", "green", "yellow", "brown", "black", "white", "orange",
    "pink", "purple", "beige", "gray", "neon", "clear", "golden",
    "crispy", "fizzy", "sweet", "salty", "creamy", "hot", "cold", "spicy",
    "refreshing", "bitter", "sour", "smooth", "crunchy", "bubbly", "strong",
    "light", "dark"
]

Let's embed the word "burrito" and see which other words from our list are most similar:

find_similar_concepts(embedding('burrito'), filter=["burrito"])

We can see that "burrito" clusters with other savory foods — especially those in the fast food or protein-heavy category.

Refining Meaning Through Vector Arithmetic

Now that we've seen how embeddings represent individual concepts in high-dimensional space and how they can be compared, a natural next step is to explore how meaning can be manipulated mathematically.

One powerful property of vectors is that they support meaningful arithmetic. By adding, subtracting, or blending vectors, we can produce new representations that reflect combined or transformed meanings — a fundamental mechanism used in the internal computation of models like GPT.

Let's look at a few examples within our word list.

find_similar_concepts(
  embedding('milk') + embedding('espresso'),
  filter=["milk", "espresso"]
)

find_similar_concepts(
  embedding('father') - embedding('man') + embedding('woman'),
  filter=["father", "man", "woman"]
)

These examples further demonstrate how embeddings capture both meaning and similarity. But in language, meaning isn't just about what words are present — it's also about where they appear. The phrases "dog bites man" and "man bites dog" contain the same words, but carry very different meanings.

To account for this, after converting tokens into vectors, GPT models apply positional embeddings — learned vectors that encode each word's position in the sequence. Just as we combined "milk" and "espresso" to create a new concept like "cappuccino", GPT adds a learned position vector to each token's meaning vector — effectively saying, "this word means this, here."

This positional context is essential to how GPT models understand and generate coherent language — but it's a deeper topic, likely worth a post of its own.

Words vs Tokens

Just to clarify — when I mention tokens, it's important to know that an embedding doesn't always represent a single word. In fact, embeddings can capture the meaning of a whole string of words, a phrase, or even a sentence.

There are many different strategies for how text is split into tokens, depending on the model and the task — but that's a topic for another post.

For now, it'll suffice to demonstrate the same vector-based similarity of full phrases against out word list.

find_similar_concepts(calculate_embedding('dessert you eat with a spoon'))

find_similar_concepts(calculate_embedding('members of a family'))

Take Away

A core idea behind large language models (LLMs) is that they are mathematical models of human understanding.

Rather than relying on strict logic or fixed rules, these models represent meaning as positions in a high-dimensional space — where the geometry between points reflects relationships between concepts. This allows computers to interpret meaning, not just match literal words.

And it doesn't stop at language. Today's models are being trained to understand meaning across images, audio, and video — enabling multimodal understanding, a kind of modern-day Rosetta Stone for human perception.

I don't know about you, but as a technologist, I find this to be an incredibly exciting time.

After years of working in a world defined by literal, logical, and deterministic systems, we're now stepping into an era shaped by probabilistic, semantic, and conceptual ones. Just as exciting is how accessible and cost-effective these capabilities have become — not just for researchers, but for everyday developers and creators.

The very nature of how we interact with computers is evolving — and it's happening faster, and at a greater scale, than anything we've seen before.

Appendix

For those interested in replicating the examples shown in this post, here's the initial setup using Python and the OpenAI API:

import os
import pandas as pd
import numpy as np
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def calculate_cosine_similarity(a, b):
  return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def find_similar_concepts(query_embedding, top_n=5, filter=[]):
    df["similarity"] = df["embedding"].apply(lambda emb: calculate_cosine_similarity(query_embedding, emb))
    result = df[~df["word"].isin(filter)].copy()
    return result.sort_values("similarity", ascending=False).head(top_n).reset_index(drop=True)

def calculate_embedding(text):
  return np.array(client.embeddings.create(input=[text.replace("\n", " ")], model="text-embedding-3-small").data[0].embedding)

def embedding(word):
  return np.array(df.loc[df['word'] == word, 'embedding'].item())

df = pd.DataFrame({
    "word": words,
    "embedding": [calculate_embedding(word) for word in words]
})