AI · Reference

AI glossary

Plain-language definitions for the terms that appear in these articles. Start here when the machinery behind a coding assistant is unfamiliar.

For worked explanations, start with model architectures or the Local AI Toolkit.

Activation: An intermediate numerical value produced while a network processes an input. Activations vary with the request; learned weights normally remain unchanged during inference.
Active parameters: Parameters involved in the computation selected for a token, particularly in MoE models. This count does not replace the total weight-storage requirement.
Agent harness: The system around a model that prepares context, coordinates tool execution, handles results, and manages the task loop. It is distinct from the inference server executing model requests.
Agent loop: A repeated process in which a model receives context, produces output or requests an action, and receives the action's result before continuing. The harness coordinates execution and stopping.
Attention: A learned way to combine information across accessible sequence positions. Conventional attention compares queries and keys to weight values; other attention formulations can have different execution and storage properties.
Backpropagation: An application of the chain rule to compute how a loss changes with model parameters. Training uses these gradients with an optimizer; ordinary inference does not update weights this way.
Bandwidth: The rate at which data can move through a memory system or connection. A link rating, measured payload throughput, and local memory bandwidth describe different things.
Batching: Executing work for multiple inputs or active requests together to use hardware more efficiently. Aggregate throughput can improve while the latency experienced by an individual request changes.
Cache breakpoint: A boundary that a provider can use to identify the end of a reusable prompt prefix. It does not make every document before it independently reusable in arbitrary contexts.
Chunked prefill: Scheduling a long prompt's processing in smaller pieces so it can share execution time with other work, such as ongoing decoding. It changes scheduling; prefix caching reduces repeated computation.
Compute backend: An engine's implementation path for executing model operations through CPU code or a GPU platform such as CUDA, HIP, Vulkan, or Metal. Device and operation support depend on the build.
Context engineering: Selecting, organizing, and updating the information supplied to a model at each step: instructions, relevant files, tool definitions, history, and observed results.
Context window: The amount of context a model and runtime can handle for a request. Prompt material and generated continuation consume the available token budget; a cache hit does not make tokens disappear from context.
CUDA: NVIDIA's parallel-computing platform and programming model. AI applications can use its runtime, libraries, and custom GPU kernels; CUDA is not a requirement for ordinary game rendering.
Decode: Producing a continuation from the current context. In ordinary autoregressive generation, each selected token becomes context for the next step.
Dense model: A model whose ordinary dense layers do not route tokens to a selected subset of expert feed-forward networks. Dense versus MoE is a different distinction from Transformer versus state-space architecture.
Embedding: A numerical vector representing a token or another input. Token embeddings are transformed by later layers; an embedding is not a human-readable definition or a guaranteed map of semantic facts.
Evaluation harness: Infrastructure that runs test tasks against an AI system, records attempts, grades outcomes, and aggregates results. It evaluates the agent and its environment; it is not synonymous with the agent harness.
Eviction: Removing retained cache state to reclaim capacity. A later identical prompt can miss because the state is no longer present, even though its content has not changed.
GGUF: A binary model format in the ggml ecosystem containing tensor data and metadata. It can hold quantized or higher-precision tensors; the extension alone does not establish model or runtime compatibility.
GPU kernel: A program executed on a GPU to perform an operation over data. Fast inference needs kernels compatible with the model's operations, numerical formats, and hardware.
Harness engineering: Designing and improving the model's surrounding control loop and environment: tool access, context handling, permissions, feedback, recovery, continuity, and completion checks.
HIP: A C++ runtime API and GPU kernel language used in AMD's compute ecosystem. Source portability does not mean arbitrary CUDA binaries execute unchanged on AMD hardware.
Inference: Running a trained model on an input to produce a prediction or output. Ordinary inference uses the learned weights without training them again.
Inference server: A running service that accepts model requests, schedules computation, and returns outputs. It manages model loading and request state; it can run on your own computer or on remote hardware.
KV cache: Stored attention keys and values for previously processed token positions. Reusing them avoids reconstructing that state at every generation step. This is intermediate model state, not a collection of saved answers.
Layer splitting: Placing different groups of model layers on different devices. A request's activations pass between groups in layer order. Suitable scheduling can pipeline work from different requests.
Logits: Unnormalized output scores. In next-token generation, the model produces scores over its vocabulary before probabilities and a selection rule determine the next token.
Metal: Apple's graphics and compute API, also used for machine-learning workloads. It is a device-programming interface, distinct from an application or a framework such as MLX.
Mixture of experts (MoE): A model architecture that routes tokens through selected expert subnetworks. Active parameters describe part of the per-token work; they do not replace the total expert-weight storage budget.
OpenAI-compatible API: An interface implementing some OpenAI-style request and response conventions. Supported endpoints, tool behavior, and output capabilities must be checked; the label does not imply identical models or feature coverage.
Parameter: A learned numerical value in a model. Parameter count describes model size, but does not by itself determine resident memory, latency, or answer quality.
PCIe switch: Hardware that routes PCIe transactions between upstream and downstream ports. It is distinct from a passive lane-bifurcation adapter or a signal retimer, and cannot remove a shared upstream bottleneck.
Peer-to-peer transfer: A supported device-to-device data path that avoids requiring the application to stage every transfer in host memory. Real support depends on topology, drivers, runtime, and the operation being performed.
Prefill: Processing the input prompt before continuing generation. For an attention-based model, this also constructs key/value state for prompt positions. Reusable prefix state can avoid repeating part of this work.
Prefix caching: Reusing computation for an eligible matching beginning of a request. Ordinary prefix-cache identity depends on the preceding context as well as the current tokens, model, and relevant execution settings.
Prompt engineering: Shaping instructions and examples to elicit useful model behavior. Clear objectives, constraints, and acceptance cases help make a request assessable; a prompt alone does not enforce tool permissions.
Quantization: Representing numerical values with a reduced-precision format. Weight and KV-cache quantization are separate choices, with format-specific overhead, kernel support, and potential quality effects.
Replica: Another complete serving instance of a model that can handle independent requests. Replicas increase placement and concurrency options; their memory does not automatically pool for one oversized model.
ROCm: AMD's GPU-computing software stack, including runtimes, libraries, and developer tools. Support depends on the exact GPU, operating system, application, and release.
Safetensors: A tensor serialization format designed for safe and efficient loading without pickle-style code execution. Model configuration and tokenizer files may be separate; unrelated repository code is not covered by that guarantee.
Serving engine: The software that executes and schedules model inference. Its supported models, accelerator kernels, memory management, batching, and cache features determine what the server can do.
Softmax: A function that converts a set of real-valued scores into positive values summing to one. It is used for token probabilities and conventional attention weights; it does not verify factual truth.
Tensor parallelism: Dividing operations within model layers across devices. Devices compute shards and exchange or reduce results, making communication and synchronization part of inference.
Time to first token (TTFT): The wait from sending a request until its first output token appears. It can include networking, queueing, preparation, and prompt processing. It is different from the time required to finish the answer.
Token: A unit of model input or output. Depending on the tokenizer, it may represent a word, part of a word, punctuation, or other content. Tokens are not interchangeable with characters or streamed network events.
Tokenizer: The model-specific process that maps text into token identifiers and back. The same text can have different token counts with different tokenizers.
Tool: An operation exposed to an assistant, such as reading a file or running a test. A model can request its use; surrounding software validates and executes the operation subject to permissions.
Unified memory: A platform memory arrangement allowing processors such as a CPU and integrated GPU to access a shared physical pool through supported paths. Usable allocation still depends on the operating system, driver, and runtime.
Vulkan: A cross-platform graphics and compute API. An inference engine can use compute shaders without rendering graphics; portability does not guarantee identical speed or features across GPUs.
Weights: The numerical arrays learned during training. They are reused across requests; they are different from the temporary state created for a particular conversation.