Semantic Cache

Semantic caching stores LLM responses indexed by the semantic meaning of the input. When a similar query arrives, the cached response is returned without calling the LLM — reducing costs and latency.

Quick Start

import { Agent, openai, InMemoryVectorStore, OpenAIEmbedding } from "@agentium/core";

const agent = new Agent({
  name: "assistant",
  model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol"),
  semanticCache: {
    vectorStore: new InMemoryVectorStore(new OpenAIEmbedding()),
    embedding: new OpenAIEmbedding(),
    similarityThreshold: 0.92,
    scope: "agent",
  },
});

// First call: LLM call, result cached
await agent.run("What is the capital of France?");

// Second call: may return from cache if the stored match clears the threshold
await agent.run("What's the capital of France?");

Configuration

interface SemanticCacheConfig {
  vectorStore: VectorStore;        // Any vector store backend
  embedding: EmbeddingProvider;    // Embedding model for similarity
  similarityThreshold?: number;    // 0-1, default 0.92
  ttl?: number;                    // Cache expiry in ms
  collection?: string;             // Default: semantic_cache
  scope?: "global" | "agent" | "session"; // Default: global
  maxEntries?: number;             // Per-instance admission counter
}

Scope

Scope	Behavior
`global`	All agents share one cache
`agent`	Each agent has its own cache partition
`session`	Each session has its own cache partition

Cache scope partitions collection names; it does not verify tenant ownership. Avoid sharing cached private responses across users, and include the application context in your cache design.

How It Works

Before calling the LLM, the input is embedded and searched against the vector store
If a result exceeds the similarityThreshold, it’s returned as a cache hit
Output guardrails still run on cached responses
After an LLM call, the input + output are stored in the vector store (fire-and-forget)
TTL is enforced on lookup — expired entries are evicted lazily

Events

Event	Payload
`cache.hit`	`{ agentName, input, cachedId }`
`cache.miss`	`{ agentName, input }`

Supported Backends

Any VectorStore implementation works: InMemoryVectorStore, QdrantVectorStore, MongoDBVectorStore, PgVectorStore.

Backend Examples

InMemory (Development)

import { Agent, openai, InMemoryVectorStore, OpenAIEmbedding } from "@agentium/core";

const embedding = new OpenAIEmbedding();

const agent = new Agent({
  name: "assistant",
  model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol"),
  semanticCache: {
    vectorStore: new InMemoryVectorStore(embedding),
    embedding,
    similarityThreshold: 0.92,
  },
});

The in-memory cache is lost when the process restarts. Use it for a local experiment with a bounded set of public queries.

Qdrant

import { Agent, openai, QdrantVectorStore, OpenAIEmbedding } from "@agentium/core";

const embedding = new OpenAIEmbedding();

const agent = new Agent({
  name: "assistant",
  model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol"),
  semanticCache: {
    vectorStore: new QdrantVectorStore({
      url: "http://localhost:6333",
    }, embedding),
    embedding,
    similarityThreshold: 0.90,
    ttl: 3600_000, // 1 hour
  },
});

PgVector (PostgreSQL)

import { Agent, openai, PgVectorStore, OpenAIEmbedding } from "@agentium/core";

const embedding = new OpenAIEmbedding();

const agent = new Agent({
  name: "assistant",
  model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol"),
  semanticCache: {
    vectorStore: new PgVectorStore({
      connectionString: "postgresql://localhost:5432/myapp",
    }, embedding),
    embedding,
    similarityThreshold: 0.92,
  },
});

Cache Hit vs Miss Behavior

This function observes an existing Agent configured with semantic caching. It supplies separate session IDs so conversation history does not become the reason for a different answer. A similar input is only a candidate hit: inspect the emitted event rather than assuming the embedding score clears the threshold.

import type { Agent, AgentEventMap } from "@agentium/core";

export async function compareCachedQueries(agent: Agent) {
  const hit = ({ input, cachedId }: AgentEventMap["cache.hit"]) => {
    console.log("hit", input, cachedId);
  };
  const miss = ({ input }: AgentEventMap["cache.miss"]) => console.log("miss", input);
  agent.eventBus.on("cache.hit", hit).on("cache.miss", miss);
  try {
    await agent.run("What is the capital of France?", { sessionId: "cache-first" });
    await agent.run("What's France's capital city?", { sessionId: "cache-second" });
  } finally {
    agent.eventBus.off("cache.hit", hit).off("cache.miss", miss);
  }
}

Cache writes happen asynchronously. The second request can miss if the first write has not settled. Use cache events and representative paraphrases to measure hits. A cache hit still requires an embedding lookup and can become stale when the underlying data or prompt changes.

Tuning `similarityThreshold`

Threshold	Behavior
`0.98+`	Nearly exact matches only
`0.92-0.95`	Good default — catches rephrasings
`0.85-0.90`	Aggressive caching — may return irrelevant results
`< 0.85`	Not recommended — too many false matches

Start with 0.92 and adjust based on your cache hit rate and quality.

Cross-References

Tool Caching — Cache individual tool results (different from semantic cache)
Cost Tracking — Semantic cache reduces LLM costs; track savings with CostTracker

Ship

Identity and boundaries

Recovery and upgrades

Operating the service

Quick Start

Configuration

Scope

How It Works

Events

Supported Backends

Backend Examples

InMemory (Development)

Qdrant

PgVector (PostgreSQL)

Cache Hit vs Miss Behavior

Tuning `similarityThreshold`

Cross-References

​Quick Start

​Configuration

​Scope

​How It Works

​Events

​Supported Backends

​Backend Examples

​InMemory (Development)

​Qdrant

​PgVector (PostgreSQL)

​Cache Hit vs Miss Behavior

​Tuning similarityThreshold

​Cross-References

Quick Start

Configuration

Scope

How It Works

Events

Supported Backends

Backend Examples

InMemory (Development)

Qdrant

PgVector (PostgreSQL)

Cache Hit vs Miss Behavior

Tuning `similarityThreshold`

Cross-References