Deep learning glossary

143 terms, each from the explainer where it is used.

Active parameters: The subset of a model's total weights actually used to process a single token, which determines inference compute cost. Mixture of experts →
Adaptation method: The specific way a scenario is turned into a prompt for a model, which HELM found can swing accuracy by tens of percentage points for the same model and task. Benchmarks and evaluation →
Adapter layers: Small trainable modules inserted into a frozen pretrained model so that only they, not the full model, are updated during fine-tuning. Fine-tuning →
Adversarial suffix: A string of tokens appended to a harmful query, optimised to push the model toward an affirmative rather than a refusing response. Jailbreaks and prompt injection →
AI agent: A language model that takes actions in a loop, observing results and deciding the next step, rather than producing one answer. AI agents and tool use →
Alignment: Training a language model to act in accordance with a user's intention, including both explicit instructions and implicit goals like truthfulness and harmlessness. Alignment →
Alignment tax: The performance regression on certain public NLP benchmarks that an alignment fine-tuning procedure can cause. Alignment →
Apache 2.0 licence: A permissive open-source licence Mistral AI applied to both Mixtral 8x7B and its instruct version, allowing free academic and commercial use. Open-weight models →
Attack Success Rate (ASR): The metric Zou et al. (2023) use to measure how often an attack causes the target behaviour or exact target string. Jailbreaks and prompt injection →
Attention: A mechanism that maps a query and a set of key-value pairs to an output, weighting each value by how well its key matches the query. Attention →
Automatic differentiation: A family of techniques, more general than backpropagation, for computing exact numerical derivatives of a function expressed as a computer program. Neural networks →
AWQ: Activation-aware weight quantization, a method that scales up the small fraction of weight channels identified as salient by activation magnitude to reduce quantization error. Quantization →
Backpropagation: An efficient method, dating to the 1960s-70s and first applied to neural networks in 1981, for computing how each weight should change to reduce error. Neural networks →
BM25: A sparse, word-overlap-based retrieval method used as a non-learned baseline against dense retrievers. Retrieval-augmented generation →
Bradley-Terry model: A model, used by Christiano et al. (2017) and later papers, that converts a difference in predicted reward between two options into a probability that one is preferred, similar to a chess Elo gap. RLHF and preference tuning →
Byte-pair encoding (BPE): A word-segmentation algorithm, adapted from a data-compression method by Gage (1994), that repeatedly merges the most frequent adjacent pair of symbols in a training corpus to build a fixed-size subword vocabulary. Tokenization →
Calibration: A measure of whether a model's stated confidence in an answer matches how often that answer is actually correct. Benchmarks and evaluation →
Catastrophic forgetting: The loss of a model's previously learned general knowledge when it is fine-tuned too aggressively on new data. Fine-tuning →
CBOW: Continuous bag-of-words, a word2vec architecture that predicts the current word from the words around it. Embeddings →
Chain of thought: A series of intermediate natural language reasoning steps a model generates before its final answer. Reasoning models →
Chain-of-thought prompting: A method, used as a baseline in the ReAct paper, where a model writes out reasoning steps without taking any actions in an environment. AI agents and tool use →
Chain-of-thought reasoning: Prompting a model to reason step by step before giving an answer, which Bai et al. found improved the accuracy of AI-generated harmlessness judgments. Alignment →
Circuit: A set of connected features, wired through a network's weights, that together carry out a specific computation. Interpretability →
Coding agent: A system that reads an issue and a codebase and generates a patch, as evaluated by SWE-bench's execution-based tests. AI agents and tool use →
Constitutional AI (CAI): A method from Bai et al. (2022) that trains a model to be harmless using a written list of principles instead of human labels identifying harmful outputs. Alignment →
Context window: The number of tokens a language model can take as input at once, per Liu et al. (2023). Context windows and long context →
Convolutional neural network: A network, first introduced as the Neocognitron in 1979, in which a filter's weights are replicated across an input such as an image. Neural networks →
Cosine similarity: A similarity measure between two vectors used by SBERT to compare sentence embeddings. Embeddings →
Credit assignment path: A chain of possibly causal links between an action and its effect, used to measure whether learning in a network is deep or shallow. Neural networks →
Decode: The autoregressive phase that generates each remaining output token one at a time, dependent on all previous tokens in the sequence. Inference and serving →
Dense Passage Retriever (DPR): A bi-encoder retriever that represents queries and documents as vectors with separate BERT models and scores relevance by their inner product. Retrieval-augmented generation →
DPO (Direct Preference Optimization): An algorithm from Rafailov et al. (2023) that fine-tunes a language model directly on preference pairs with a classification loss, skipping both reward-model training and reinforcement learning. RLHF and preference tuning →
Embedding: A vector of numbers representing a word, sentence or other item such that similar items have similar vectors. Embeddings →
Expert capacity factor: A multiplier, used in Switch Transformers, that sets how many tokens each expert may process before excess tokens are dropped. Mixture of experts →
Exposure bias: The mismatch between training, where a decoder is fed the true previous tokens, and inference, where it must condition on its own possibly wrong past outputs. Hallucination →
Extrinsic hallucination: Generated output that the source can neither confirm nor contradict, whether or not it happens to be true. Hallucination →
Faithfulness: Staying consistent and truthful to a provided source; the opposite of hallucination. Hallucination →
Feature: An underlying concept a network represents, which in superposition may be spread across many neurons rather than confined to one. Interpretability →
Few-shot, one-shot, zero-shot: Settings that vary how many task examples, from several down to none, are given to a model at inference time. Large language models →
Fine-tuning: Further training a pretrained model on a smaller, task-specific dataset rather than training from scratch. Fine-tuning →
FlashAttention: An IO-aware exact attention algorithm that uses tiling to avoid storing the full attention matrix in GPU high-bandwidth memory. Attention →
Gating network: The trainable component that scores experts for an input and decides which ones receive it. Mixture of experts →
GPT-judge: A GPT-3-6.7B model finetuned by Lin, Hilton and Evans (2021) to automatically classify TruthfulQA answers as true or false, matching human judgment 90-96% of the time. Hallucination →
GPTQ: A one-shot post-training quantization method that uses approximate second-order information to quantize each layer while compensating remaining weights for rounding error. Quantization →
Gradual unfreezing: A fine-tuning technique from ULMFiT that unfreezes a model's layers one at a time, starting from the last, to avoid catastrophic forgetting. Fine-tuning →
Greedy Coordinate Gradient (GCG): An optimisation method from Zou et al. (2023) that uses token-level gradients plus exact evaluation of candidate substitutions to find adversarial suffixes. Jailbreaks and prompt injection →
Group Relative Policy Optimization (GRPO): The reinforcement learning algorithm DeepSeek-AI used to train DeepSeek-R1, which scores a sampled group of model outputs against their group average rather than using a separate value model. Reasoning models →
Group-wise quantization: A scheme that assigns separate quantization scales to small blocks of consecutive weights rather than one scale per tensor, improving precision at low bit widths. Quantization →
Grouped-query attention (GQA): An attention variant, introduced by Ainslie et al. (2023), that divides query heads into groups which each share one key and value head, interpolating between multi-head and multi-query attention. The KV cache →
HBM (high-bandwidth memory): The GPU's relatively large but comparatively slow main memory, as distinct from much smaller and faster on-chip SRAM. Attention →
HELM (Holistic Evaluation of Language Models): A 2022 Stanford benchmarking effort that scores 30 language models on 16 core scenarios using seven categories of metric, not accuracy alone. Benchmarks and evaluation →
Imitative falsehood: A false answer that has high likelihood under a model's training distribution because it mimics a common human misconception, as defined by Lin, Hilton and Evans (2021). Hallucination →
In-context learning: A pretrained model's ability to perform a new task from instructions or examples given in its prompt, without any gradient updates. Large language models →
Index hot-swapping: Replacing a retrieval model's document index with an updated one so its knowledge changes without retraining the model. Retrieval-augmented generation →
Indirect prompt injection: An attack, introduced by Greshake et al. (2023), that places malicious instructions in data an LLM-integrated application retrieves, rather than in the user's own prompt. Jailbreaks and prompt injection →
InstructGPT: The model produced by Ouyang et al. (2022) by fine-tuning GPT-3 with demonstration data and reinforcement learning from human feedback. Alignment →
Instruction tuning: Fine-tuning a model on paired instructions and responses so it learns to follow user commands. Fine-tuning →
Intrinsic hallucination: Generated output that directly contradicts the source content it was supposed to be based on. Hallucination →
Iteration-level scheduling: A batching technique that adds and removes requests from a batch after every decoding step rather than waiting for the whole batch to finish. Inference and serving →
Jailbreak: A prompt engineered to make an aligned language model generate content it was trained to refuse. Jailbreaks and prompt injection →
KL penalty: A per-token penalty, used in Ouyang et al. (2022)'s PPO training, that discourages the fine-tuned policy from straying too far from the original supervised model's output distribution. RLHF and preference tuning →
KV cache: The cached key and value vectors from a sequence's earlier tokens, stored so the attention mechanism does not recompute them for every new token. Inference and serving →
Large language model (LLM): A neural network trained on large amounts of text that generates language by predicting the next token. Large language models →
Layer: The set of neurons reachable by paths of the same length from the network's input, in a feedforward network. Neural networks →
LLM-integrated application: A system, such as Bing Chat or GitHub Copilot, that connects a language model to external data, tools, or APIs. Jailbreaks and prompt injection →
Load-balancing loss: An auxiliary training loss that encourages tokens to be spread evenly across experts rather than concentrated on a few. Mixture of experts →
Long-term decay: A property of RoPE, shown by Su et al. (2021), in which the attention score between two tokens shrinks as the distance between their positions grows. Context windows and long context →
LoRA (Low-Rank Adaptation): A fine-tuning method that freezes pretrained weights and trains a pair of small, low-rank matrices to represent the needed weight update. Fine-tuning →
Lossless tokenization: SentencePiece's design in which decoding a tokenized sequence exactly reproduces the normalized input text, including whitespace, with no information lost. Tokenization →
Maximum Inner Product Search (MIPS): The search problem of finding the documents whose vectors have the highest inner product with a query vector, solvable approximately in sub-linear time. Retrieval-augmented generation →
Mechanistic interpretability: The branch of interpretability that tries to identify the specific internal parts and circuits of a network responsible for a given behaviour. Interpretability →
Memory fragmentation: Wasted GPU memory from reserving a KV cache chunk sized for a request's maximum length, identified by Kwon et al. (2023) as leaving only 20.4% to 38.2% of allocated cache memory actually used. The KV cache →
Mixture of experts (MoE): A layer made of many smaller sub-networks, called experts, with a router that selects a sparse subset to process each input. Mixture of experts →
MMLU (Massive Multitask Language Understanding): A 2020 benchmark of 15,908 multiple-choice questions across 57 subjects, designed to test knowledge and reasoning a model picks up during pretraining. Benchmarks and evaluation →
Multi-head attention: Several scaled dot-product attention computations run in parallel on different learned projections of the queries, keys and values, then combined. Attention →
Multi-query attention (MQA): An attention variant, proposed by Shazeer (2019), in which all query heads share a single set of key and value vectors, sharply cutting KV cache size and memory bandwidth. The KV cache →
Neuron: A simple processing unit in a neural network that produces a real-valued activation from its weighted inputs. Neural networks →
Next-token prediction: The training objective where a model learns to guess the next word, or token, given everything written before it. Large language models →
Non-parametric memory: An external, editable knowledge source, such as a Wikipedia dump, that a model consults at inference time rather than storing in its weights. Retrieval-augmented generation →
Open weights: The practice of publishing a trained model's learned parameters so others can download and run it. Open-weight models →
Oracle retrieval: A SWE-bench evaluation setting that hands the model the exact files the real pull request edited, rather than files found by search. AI agents and tool use →
Out-of-vocabulary (OOV) word: A word that does not appear anywhere in the training text, which a word-level model cannot translate or generate directly. Tokenization →
Outcome-supervised reward model (ORM): A reward model trained using only whether a solution's final result was correct. Reasoning models →
Outlier features: A small number of hidden-state dimensions in large transformers that carry unusually large magnitude values and dominate quantization error. Quantization →
PagedAttention: An attention algorithm that stores a sequence's key-value cache in non-contiguous, fixed-size memory blocks, inspired by operating-system virtual memory paging. Inference and serving →
Parameters: The adjustable numerical weights of a neural network that are set during training and determine what it has learned. Large language models →
Parametric memory: Knowledge stored implicitly in a model's trained weights, which can only be updated by further training. Retrieval-augmented generation →
Perplexity: A language-modeling score measuring how well a model predicts held-out text, used throughout these papers to track accuracy lost to quantization. Quantization →
Phase change: A shift reported in the toy model between representing features individually and packing them into superposition, depending on conditions like feature sparsity. Interpretability →
Polysemanticity: A neuron that responds to several unrelated concepts at once, making its individual activity hard to interpret. Interpretability →
Position encoding: A mechanism that tells a self-attention model where each token sits in a sequence, since self-attention is otherwise position-agnostic, per the RoFormer paper. Context windows and long context →
Post-training: The stages applied after pretraining, such as supervised fine-tuning and reinforcement learning with human feedback, that adapt a model for dialogue or other uses. Large language models →
Post-training quantization (PTQ): A compression approach that quantizes an already-trained model without further retraining, usually needing only a small calibration dataset. Quantization →
PPO (proximal policy optimization): The reinforcement learning algorithm Ouyang et al. (2022) used to fine-tune a supervised language model to maximise reward-model score. RLHF and preference tuning →
Prefill: The phase of generation that processes the full input prompt at once, computing the first output token's probability in parallel across all prompt tokens. Inference and serving →
Pretraining: The initial stage of training an LLM on a broad corpus of text with no task-specific labels. Large language models →
Process-supervised reward model (PRM): A reward model trained using feedback on the correctness of each individual step in a solution, as introduced for large-scale use by Lightman et al. (2023). Reasoning models →
QLoRA: A method that backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, cutting memory use without losing performance, according to Dettmers et al., 2023. Fine-tuning →
Quantization: Converting a model's weights from a higher-precision format, such as 16-bit, to a lower-precision one, such as 4-bit, to save memory. Fine-tuning →
Query-aware contextualization: A technique tested by Liu et al. (2023) of placing the query both before and after the input data, which sharply improved key-value retrieval but not multi-document QA. Context windows and long context →
Query, key, value: The three vector roles in attention: the query represents what is being looked up, keys represent what each position offers, and values are the content retrieved. Attention →
ReAct: A prompting method from Yao et al. (2022) that interleaves written reasoning steps (thoughts) with actions and environment observations. AI agents and tool use →
Recurrent neural network: A network whose connections form cycles, allowing it to process and retain memories of sequences of inputs over time. Neural networks →
Reinforcement learning with human feedback (RLHF): A fine-tuning procedure that uses a reward model trained on human preference comparisons to align a language model's outputs with what people prefer. Large language models →
Retrieval-augmented generation (RAG): A fine-tuning approach that combines a pretrained parametric generator with a non-parametric, searchable document index, retrieved at answer time. Retrieval-augmented generation →
Retriever-reader setup: A system described by Liu et al. (2023) in which a retrieval model fetches documents and a language model reads them to answer a query. Context windows and long context →
Reward hacking: A failure mode where a trained policy finds shortcuts that exploit a reward model or reward rule rather than genuinely improving, flagged as a risk by both the DeepSeek-R1 and Let's Verify Step by Step papers. Reasoning models →
Reward model: A model trained on human comparison data to predict which of two outputs a human would prefer, then used as the reward signal during reinforcement learning. Alignment →
RLAIF (reinforcement learning from AI feedback): A variant of RLHF in which an AI feedback model, rather than human labelers, judges which of two responses better follows a given principle. Alignment →
RLHF (reinforcement learning from human feedback): A technique that uses human preference rankings of model outputs as a reward signal to fine-tune a model with reinforcement learning. Alignment →
RLHF (Reinforcement Learning with Human Feedback): A training procedure that uses human preference comparisons to further align a fine-tuned model's behaviour. Open-weight models →
Rotary Position Embedding (RoPE): A method from Su et al. (2021) that encodes position by rotating a token's query and key vectors by an angle tied to that position, so attention depends on relative distance. Context windows and long context →
Round-to-nearest (RTN): The simplest quantization baseline, which rounds each weight to the closest value on the lower-bit grid with no error compensation. Quantization →
Salient span masking: A REALM pretraining technique that masks named entities and dates, rather than random tokens, to force the model to rely on retrieved world knowledge. Retrieval-augmented generation →
Scaled dot-product attention: The attention function used in the transformer, which scores queries against keys by dot product, divides by the square root of the key dimension, and applies a softmax. Attention →
Self-attention: An attention mechanism relating different positions within a single sequence to compute a representation of that sequence. Attention →
Semantic textual similarity (STS): A set of benchmark tasks that score how well a system's similarity judgement between two sentences matches human-labelled gold scores. Embeddings →
Sentence-BERT (SBERT): A modification of BERT, proposed by Reimers and Gurevych (2019), that uses siamese networks to produce sentence embeddings comparable by cosine similarity. Embeddings →
SentencePiece: A language-independent subword tokenizer and detokenizer, described by Kudo and Richardson (2018), that trains directly from raw sentences and supports both BPE and a unigram language model algorithm. Tokenization →
Siamese network: A network structure using two copies of the same model with shared weights to process two inputs for comparison, as used to fine-tune SBERT. Embeddings →
Skip-gram: A word2vec architecture that predicts the surrounding words given the current word. Embeddings →
Soft alignment: A weighted combination over all source positions, rather than a hard choice of a single position, used to decide what to attend to. Attention →
Sparse routing: The gating mechanism that picks only the top-scoring experts for each token and skips computing the rest. Mixture of experts →
Subword unit: A sequence of characters shorter than a full word, used so that rare or unseen words can still be represented by combining known pieces. Tokenization →
Superposition: A network storing more features than it has neurons by representing them as overlapping combinations of neurons rather than one feature per neuron. Interpretability →
SWE-bench: A benchmark from Jimenez et al. (2023) of 2,294 real GitHub issues, where a model must edit a codebase so the repository's own tests pass. AI agents and tool use →
Switch routing: The Switch Transformer approach of routing each token to exactly one expert rather than several. Mixture of experts →
Test-time compute: Computation spent while a model generates its answer, which reasoning models can scale up by producing longer chains of thought on harder problems. Reasoning models →
Throughput: The number of requests or tokens a serving system completes per unit of time, as distinct from the latency of any single request. Inference and serving →
Token: A discrete unit of text, often a subword piece, that a neural network reads or generates one at a time. Tokenization →
Tool calling: A model's ability to invoke an external function, such as a calculator or search engine, and use its output to continue generating text. AI agents and tool use →
Toolformer: A GPT-J-based model from Schick et al. (2023) that taught itself which API calls help predict future text, using its own filtered self-generated examples. AI agents and tool use →
Total (sparse) parameters: The full count of weights across all experts in a model, which determines its memory footprint. Mixture of experts →
Train-test contamination: The risk that a model has already seen a benchmark's questions, or close variants, during training, inflating its apparent score. Benchmarks and evaluation →
Transferability: The property, demonstrated by Zou et al. (2023), that a suffix optimised against one set of models can also succeed against different, unseen models. Jailbreaks and prompt injection →
TruthfulQA: A 817-question, 38-category benchmark designed by Lin, Hilton and Evans (2021) to test whether models generate truthful answers rather than imitating human misconceptions. Hallucination →
U-shaped performance curve: The pattern Liu et al. (2023) found where models answer best when relevant information is at the start or end of the input and worst when it is in the middle. Context windows and long context →
Unigram language model (subword algorithm): A subword segmentation algorithm, proposed by Kudo (2018) and implemented in SentencePiece, that trains and segments with cost linear in the size of the input data. Tokenization →
Uptraining: A conversion process from Ainslie et al. (2023) that mean-pools an existing multi-head checkpoint's key and value heads into fewer heads, then retrains briefly to recover quality. The KV cache →
vLLM: The LLM serving engine built on PagedAttention by Kwon et al. (2023), reported to improve throughput 2-4x over prior systems at the same latency. The KV cache →
Weight: A real-valued, adjustable parameter attached to a connection between two neurons, found by learning. Neural networks →
Word2vec: A pair of model architectures (CBOW and skip-gram), proposed by Mikolov et al. (2013), that learn word vectors by predicting context words from a target word or vice versa. Embeddings →
Zero-shot / few-shot evaluation: Testing a model with no example questions, or with only a handful, so it must rely on knowledge from pretraining rather than task-specific fine-tuning. Benchmarks and evaluation →