KV-Cache — AI Glossary

A KV-cache (key-value cache) stores the computed attention key and value matrices from previously processed tokens so the model does not need to recalculate them when generating each new token. Without a KV-cache, the model would recompute attention for the entire input sequence at every generation step. This cache is what makes autoregressive text generation fast enough for real-time conversations.