In AI inference, particularly for Large Language Models (LLMs), processing happens in two distinct phases: Prefill and Decoding.

1. Prefill Phase (The Input Phase)

The prefill phase takes the entire user prompt and processes it all at once to understand the context.

• Action: The system ingests the input text, generates tokens, and calculates attention scores across the entire prompt simultaneously.
• Key Output: It populates the KV Cache (Key-Value Cache), which stores the mathematical context of the prompt so the system does not have to reprocess it later.
• Performance Profile: This phase is compute-bound, meaning it maximizes the GPU's raw processing power (parallel processing) and finishes quickly.

2. Decoding Phase (The Output Phase)

The decoding phase generates the response text sequentially, one single token at a time.

• Action: The system predicts the next most likely token based on the prompt context and all previously generated tokens.
• Key Output: Each newly generated token is appended to the KV Cache to inform the next prediction.
• Performance Profile: This phase is memory-bandwidth bound. The GPU must fetch the entire model weights and the growing KV Cache from its memory for every single token generated, making it the slower and more expensive part of the process.

https://medium.com

How KV Caching Optimizes Inference

During the Decoding phase, the AI model needs to look at all previous tokens to predict the next word. Without KV (Key-Value) Caching, the GPU would have to re-calculate the mathematical representation (the attention context) of every single prompt token and generated token for every single new word it outputs.

KV Caching works like a temporary scratchpad:

• One-Time Calculation: During the prefill phase, the GPU calculates the key and value matrices for the input prompt once
• Store and Reuse: It stores these matrices in the GPU memory (HBM). [7, 8]
• Incremental Updates: For every new token generated during decoding, the GPU only calculates the KV values for that specific token and appends it to the cache.