Series
Running AI Yourself: Where the Time and Memory Go
Six practical explainers following a coding assistant from its first repository question to a shared local service: inference, memory budgets, PCIe topology, prompt caching, scheduling, and hardware placement.
Continue exploring: Choosing Your AI Stack.
6 parts · first published
- 01
What Actually Happens When You Send an AI Prompt?
Follow a repository question through tokenization, prefill, and decode to see why memory capacity, bandwidth, and compute solve different problems.
·9 min read - 02
Your Model Fits. Why Does Your Conversation Run Out of Memory?
Build a memory budget for a 128 GB Halo: weights, attention state, runtime buffers, and several people asking questions at once.
·9 min read - 03
Two GPUs, One Narrow Cable: Where Does the Traffic Go?
Follow model traffic through an upstream link and a PCIe switch, and learn what the Franken Strix experiment does and does not prove.
·8 min read - 04
Stop Reprocessing the Same Prompt: Caching for Coding Agents
Arrange stable repository context for reuse, compare provider cache controls, and distinguish API savings from coding-assistant subscription behavior.
·9 min read - 05
Prefix Caching, Chunked Prefill, and the Trouble with Chunks
Trace shared and changed prompts through cache blocks and scheduling slices, and resolve what people mean by chunked prefixing.
·9 min read - 06
One Halo, Several Halos, or a Desktop Full of GPUs?
Choose a serving shape for latency, concurrency, and context capacity without confusing replicas, model splitting, or remote cache transfer.
·10 min read