Historical benchmark results
These results predate the v4.5 accounting changes. They have not been rerun for v4.5. They do not establish current provider prices, complete auxiliary-call costs, or accuracy for other models and APIs. All benchmarks usegpt-4o-mini, identical prompts, and 5 runs per scenario. Agentium and LangChain run on Node.js; Agno runs on Python.
Simple Completion
Tool Calling
Agentium and LangChain produce identical tool schemas (167 prompt tokens). Agentium strips verbose JSON Schema metadata (
$schema, additionalProperties) to keep schemas compact.
Multi-turn Memory
In this historical sample, Agentium reported 39% fewer prompt tokens and 43% lower estimated cost than LangChain. Different prompts, memory settings, models, or tariffs can change the result.
Summary
These rankings apply to the five-run sample above. No statistical confidence interval was recorded.
Optimizations
Tool Schema Caching & Optimization
Tool definitions (Zod-to-JSON Schema conversion) are computed once at agent construction and cached. Verbose JSON Schema metadata ($schema, additionalProperties, description on the root object) is stripped automatically — reducing token overhead without losing semantic information.
Automatic Retry
Transient LLM API failures are automatically retried with exponential backoff + jitter. Retryable errors include HTTP 429, 5xx, and network errors.Token-Based History Trimming
Trim history on memory, not onAgent:
maxContextTokens on Agent.
Background extraction
Whenmemory.userFacts (or profile / entities / learnings) is on, extraction runs after the answer is sent. The user is not waiting on it.
Tiny always-on notes (fileMemory) are injected into the prompt instead of being fetched as a tool.
Streaming usage and cost
Content can finish before a provider sends its final usage. A cumulative usage snapshot must replace the previous snapshot. Adding both snapshots counts the same work twice. Reasoning tokens are a subset of output for APIs that report inclusive output. For example, 19 input tokens and 16 output tokens, including 9 reasoning tokens, give 35 total tokens. They do not give 44. Cache reads and writes have separate quantities and prices. They are part of canonical input, so they must be removed from ordinary input before separate cache rates apply. Provider adapters must map their own input semantics first: Anthropic reports ordinary input separately from its cache counts. Use the cost guide for the supported mappings, unknown-cost states, and storage contract. Fixture checks establish behavior for the tested schemas; they do not establish accuracy for every provider or future API feature.Methodology
- All benchmarks use
gpt-4o-miniwith identical prompts - Each scenario runs 5 times; results are averaged
- Startup time measures framework import + agent initialization
- Cost uses gpt-4o-mini pricing: $0.15/1M input, $0.6/1M output
- Network latency to OpenAI is shared across all frameworks
- Full benchmark scripts are in
benchmarks/in the repository