Khazad – a transparent semantic cache for LLM API calls, zero code changes
By Holidays in Europe / June 30, 2026 / No Comments / Uncategorized
Introducing Khazad: A Transparent Semantic Cache for Large Language Model API Calls with Zero Code Changes
In the rapidly evolving landscape of AI-powered applications, optimizing interactions with large language models (LLMs) is paramount. Recognizing this need, I developed Khazad, a novel semantic caching solution designed to enhance the efficiency of LLM API calls — all without requiring modifications to your existing application code.
Streamlining LLM API Optimization
Traditional methods of caching or intercepting API requests often involve wrapping SDKs or deploying proxy servers, which can complicate development workflows and introduce additional layers of complexity. Khazad takes a different approach by seamlessly integrating into your application’s HTTP transport layer. Specifically, it patches the httpx transport mechanism — a common HTTP client used across many Python-based SDKs.
Once initialized, Khazad intercepts outgoing requests to LLM APIs such as OpenAI, Anthropic, Gemini, Azure OpenAI, and Mistral. It then embeds conversational context and evaluates whether a semantically-equivalent response is already stored in Redis, serving it directly when appropriate. This process significantly reduces redundant API calls, speeding up response times and lowering costs.
Key Features and Benefits
- Model-Aware: Recognizes different model types and adjusts caching strategies accordingly.
- Conversation-Aware: Maintains contextual understanding, ensuring relevance across conversation turns.
- Bidirectional Streaming Support: Handles streaming data both inbound and outbound, preserving real-time communication flows.
Ideal Use Cases
Khazad is especially beneficial for applications with repetitive query patterns, including:
– FAQs and support chatbots
– Retrieval-Augmented Generation (RAG) frontend interfaces
– Development and Continuous Integration environments with repeated requests
Technical Specifications
- Compatibility: Python version 3.10 and above
- Database: Redis 8 with vector set support for semantic embeddings
- License: MIT License
Final Thoughts
Khazad offers a zero-effort solution to optimize LLM API integrations, delivering faster responses and cost savings for dynamic AI-driven applications. Its compatibility with existing httpx-based SDKs means you can deploy it effortlessly without altering your existing codebase.
I am eager to gather feedback and collaborate with the community to further refine this tool. For more information, you can explore the project on GitHub:
https://github.com/GuglielmoCerri/khazad