How large language models work, step by step

Large language models can seem to think, but every reply comes from one mechanical loop repeated once per token. This guide follows a single short sentence through that loop, from raw text to the next word, with small worked numbers at each step. Then it covers how the model learned its behavior, why scale matters, and what the mechanics explain about hallucinations and prompt sensitivity.

The pipeline at a glance

Our running example: you send a model the text "She sat on the river bank and", and it must produce the next word. Here's what happens before that word appears.

1. A tokenizer splits the text into tokens and maps each one to an integer ID.
2. Each ID is swapped for an embedding, a long list of numbers.
3. The vectors pass through a stack of transformer layers, where self-attention lets each token gather context from the tokens before it.
4. The final vector becomes a raw score, or logit, for every token in the vocabulary.
5. The scores become probabilities, and a sampling rule picks one token.
6. That token is appended to the input, and the loop repeats until the reply is done.

Steps 1 through 5 are one forward pass. A 500-word answer needs several hundred of them, one per token, which is why replies stream out piece by piece.

From text to vectors: tokens and embeddings

Tokenization

Models don't see letters or words; they see tokens. Before training, a tokenizer is built with a method such as byte-pair encoding: it starts from individual bytes and repeatedly merges the most frequent neighboring pairs until it reaches a fixed vocabulary, typically on the order of 100,000 to a few hundred thousand entries.

Common words become single tokens, usually including the space in front of them. With a typical modern tokenizer, our sentence most likely becomes seven tokens: "She", " sat", " on", " the", " river", " bank" and " and". A rarer word such as "riverbank" might be split into pieces, and numbers or code often break in ways that look arbitrary.

Each token's ID is just a row number in the vocabulary. Tokenization also explains everyday quirks: you pay per token, and many non-English languages need more tokens per word.

Embeddings

Each token ID selects a row from an embedding table, a learned list of numbers that represents the token. Vectors range from a few hundred numbers in tiny models to more than 10,000 in the largest published ones. Training places tokens that are used in similar ways near each other, so " river" ends up closer to " stream" than to " invoice".

Attention has no built-in sense of word order, so position must be added. Most current open models use rotary position embedding (RoPE), which rotates each token's query and key vectors by an amount that depends on its position.

Transformer layers and self-attention

The embeddings now pass through a stack of identical transformer layers: 24 in OpenAI's gpt-oss-20b and 36 in gpt-oss-120b, for example. Each layer has two main parts. Self-attention lets every token pull in information from earlier tokens, and a feed-forward network then processes each token's vector on its own; the feed-forward parts hold most of the parameters and appear to store much of the model's factual knowledge.

The intuition behind attention

Take the token " bank". On its own it's ambiguous: it could be the edge of a river or a place that holds money. Attention lets " bank" look back over the sentence and decide which earlier tokens matter for interpreting it.

For each token, the layer computes three vectors from its current state. The query describes what this token is looking for, the key advertises what each token offers, and the value is the content a token passes along when it's selected. Comparing one token's query with every key produces relevance scores, and those scores decide how much of each value gets blended in.

A small worked example

Real query and key vectors typically have 64 to 128 dimensions per head, but two are enough to see the arithmetic. Suppose the query for " bank" is [1.0, 2.0] and each token's key is as shown below. Each score is the dot product of the query with a key, divided by the square root of the vector size (here √2, about 1.41) to keep values in a stable range, and a softmax then turns the scores into weights that sum to 100%.

Nearly 59% of the weight lands on " river". The new vector for " bank" is the weighted sum of all six value vectors, so it's dominated by information from " river", and later layers see a " bank" that leans toward the riverside meaning. Nobody programmed that rule; training adjusted the query and key projections until useful patterns like this one emerged.

Heads, masks and depth

Each layer runs many attention heads in parallel, so one head can track grammar while another follows references. A causal mask stops each token from attending to later ones, which lets training score every position at once. After the final layer, the vector at " and" carries what the model needs to predict the next token.

Logits, probabilities and sampling

From logits to probabilities

The model then multiplies that final vector by an output matrix, producing one raw score, or logit, for every token in the vocabulary. A softmax converts the scores into probabilities by exponentiating each one and dividing by the total, so higher scores get disproportionately more weight.

Pretend the vocabulary had only five candidates. Here's the distribution for the next token at three temperatures.

Temperature

Temperature divides every logit by a number before the softmax. Below 1 it sharpens the distribution: at 0.5, " watched" climbs from 51% to 76%. Above 1 it flattens it, and at 1.5 the long shot " cried" more than doubles to 5.5%. A temperature near 0 approximates greedy decoding, which always takes the top token.

Top-p

Top-p, or nucleus sampling, trims the unlikely tail instead of reshaping the whole curve. It keeps the smallest set of top tokens whose probabilities add up to at least p, rescales them, and samples from that set. With p set to 0.8, the first three candidates cover 89.8%, so " fished" and " cried" are dropped, and the survivors become 57.1%, 25.7% and 17.2%.

In practice, adjust one of these settings at a time. Low temperatures suit extraction and classification, where you want the same answer every run, while moderate values suit drafting and brainstorming.

The generation loop and the KV cache

Suppose the sampler picks " watched". The model appends it to the input and runs again, now predicting the token after "and watched". This repeats until the model emits a special end-of-sequence token or hits the output limit you set.

Generation has two phases with different costs. In prefill, the model processes your whole prompt in parallel, which determines the time to the first token. In decode, it produces one token per step, and that sequential loop sets the output speed you see as tokens per second.

Why the KV cache exists

Without a shortcut, each step would recompute keys and values for every earlier token. The KV cache stores those keys and values for every layer, so each step computes them only for the newest token and reads the rest from memory.

The cache trades compute for memory. In a configuration shared by several 7B-to-8B open models, with 32 layers and 8 key-value heads of 128 dimensions each, stored at 16-bit precision, every token needs 2 × 32 × 8 × 128 × 2 bytes, or 128 KiB. A single 128K-token conversation therefore needs 16 GiB of cache on top of the model's weights.

That's why long contexts are expensive to serve and why labs keep shrinking the cache. DeepSeek's model card for V4.1-Flash, for example, reports a global KV cache of about 890 bytes per token, roughly a quarter of its predecessor's.

Providers extend the idea across requests with prompt caching: when calls share a long prefix, such as a system prompt, repeat reads are billed at a steep discount. As of October 2026, cached input on Claude Opus 5.5 and GPT-6.1 Sol costs 5% of the normal input price.

How the model learned to do this

Everything above happens at inference time with fixed parameters. The parameters themselves come from training stages that run the same forward pass but add a feedback signal.

Pretraining: predict the next token

During pretraining, the model reads trillions of tokens and predicts the next one at every position. The loss is the negative logarithm of the probability it gave the correct token: assigning 25% costs about 1.39, while 90% costs only 0.11. Backpropagation then nudges every parameter to lower the average loss across trillions of these predictions.

Average loss is often reported as perplexity, the exponential of the loss. A loss of 1.39 corresponds to a perplexity of 4, as uncertain as choosing among four equally likely options.

Supervised fine-tuning

A pretrained base model continues text but doesn't reliably follow instructions. Supervised fine-tuning, also called instruction tuning, trains it on curated prompt-and-response examples with the same next-token loss, usually counting only the response tokens. This is where it learns chat formatting, when to decline and how to call tools.

Preference training: RLHF and DPO

Next comes learning from comparisons. In RLHF, people rank alternative responses, a separate reward model learns to predict those rankings, and reinforcement learning pushes the LLM toward higher-reward answers while a penalty keeps it close to its starting point. OpenAI's 2022 InstructGPT work used this recipe, and evaluators preferred its tuned 1.3B model over the 175B GPT-3.

Direct preference optimization (DPO), introduced in 2023, reaches a similar goal more simply. It skips the separate reward model and the reinforcement learning loop, training directly on preferred and rejected pairs with a classification-style loss. Many open-weight models use DPO or a variant because it's cheaper and more stable to run.

Reinforcement learning for reasoning

Reasoning models add large-scale reinforcement learning on tasks whose answers can be checked automatically, such as math problems with known solutions or code with unit tests. DeepSeek's R1 paper, published in January 2025, reported that this kind of training, without human-written reasoning examples, produced behaviors such as self-reflection and verification. The model learns that spending tokens on intermediate steps raises its chance of reward, and that's the visible thinking you see in reasoning models.

Why scale matters: scaling laws and mixture-of-experts

Scaling laws

In 2020, Kaplan and colleagues at OpenAI showed that a language model's loss falls along a smooth power law as you increase parameters, training data and compute. These scaling laws are why labs commit huge budgets with some confidence: they can forecast a large model's loss from small pilot runs.

In 2022, DeepMind's Chinchilla paper refined the recipe: for a fixed compute budget, parameters and training tokens should grow together, with data doubling whenever model size doubles. Its 70B model, trained on about four times as much data as the 280B Gopher for the same compute, outperformed it.

Current practice goes further, training smaller models on far more data than that recipe suggests, because a model that's cheap to run matters more than one that's cheapest to train. Reasoning models add a second axis: more compute at inference time, through longer thinking, also improves results on hard problems.

Mixture-of-experts

Mixture-of-experts (MoE) models gain capacity without a matching increase in compute per token. Each MoE layer replaces one feed-forward network with many smaller expert networks plus a router that sends each token to only a few of them. Sparse expert layers were scaled up for language models in a 2017 paper by Shazeer and colleagues, and they now underpin most of the largest open-weight models.

OpenAI's gpt-oss-120b shows the trade-off: it has 117 billion parameters and 128 experts per layer, but each token passes through only 4 experts, so about 5.1 billion parameters, roughly 4% of the total, do the work. Compute per token resembles a 5B dense model, yet all 117B parameters must sit in memory.

MoE saves compute, not memory, and the gap can be extreme. Moonshot lists its open-weight Kimi K3 at 2.8 trillion total parameters with 104 billion active.

What the mechanics explain about behavior

Once you know the pipeline, many puzzling behaviors become predictable. Each one below has a mechanical cause and a practical response.

Hallucinations about rare facts follow from the mechanics: facts that appear in few training documents leave a weak imprint, so the model can blend plausible fragments into a fluent but false answer. A 2025 paper by Kalai and colleagues adds that common training and evaluation methods reward confident guessing over abstaining, much as a multiple-choice exam rewards a guess over a blank.

Sensitivity to wording has the same root: each sampled token becomes input for the next step, so one unlucky early choice can steer an entire answer. That's also why asking a model to work through a problem before answering, known as chain-of-thought, often helps: it gives the final tokens more relevant context to attend to.

Common mistakes

These misconceptions about the mechanics lead to bad design decisions. Each one has a simple correction.

• Expecting temperature 0 to guarantee identical output. It narrows the choice but doesn't remove variation from batching and hardware.
• Assuming the model remembers earlier conversations. The model keeps nothing between calls; apps resend history or saved notes in the prompt, and anything outside the context window is invisible.
• Equating size with freshness. A bigger model isn't more current, because its knowledge still stops at the training cutoff.
• Sizing hardware by active parameters. An MoE model with 5B active parameters may still need memory for 117B.
• Reading attention weights as explanations. They show where information flowed in one layer, not why the model answered as it did.

Next steps

To see how these mechanics translate into limits and cost, read context windows explained and try the token counter on your own prompts. To turn the behavior insights into better results, the prompt engineering guide covers concrete techniques. If you're designing a production system and want an experienced reviewer, you can get expert help.