Prompt Caching — AI Glossary

Prompt caching is a performance optimization where the model's computed internal representations (key-value attention states) of a static prompt prefix are stored and reused across multiple requests. Instead of recomputing these states every time the same system prompt or reference text is sent, the cached version is loaded directly, significantly reducing latency and computational cost for the repeated portion.