Skip to main content
This page describes execution optimizations and historical benchmark results. For current usage and price rules, use cost accounting.

Historical benchmark results

These results predate the v4.5 accounting changes. They have not been rerun for v4.5. They do not establish current provider prices, complete auxiliary-call costs, or accuracy for other models and APIs. All benchmarks use gpt-4o-mini, identical prompts, and 5 runs per scenario. Agentium and LangChain run on Node.js; Agno runs on Python.

Simple Completion

Tool Calling

Agentium and LangChain produce identical tool schemas (167 prompt tokens). Agentium strips verbose JSON Schema metadata ($schema, additionalProperties) to keep schemas compact.

Multi-turn Memory

In this historical sample, Agentium reported 39% fewer prompt tokens and 43% lower estimated cost than LangChain. Different prompts, memory settings, models, or tariffs can change the result.

Summary

These rankings apply to the five-run sample above. No statistical confidence interval was recorded.

Optimizations

Tool Schema Caching & Optimization

Tool definitions (Zod-to-JSON Schema conversion) are computed once at agent construction and cached. Verbose JSON Schema metadata ($schema, additionalProperties, description on the root object) is stripped automatically — reducing token overhead without losing semantic information.
For OpenAI models, tools can opt into strict mode for guaranteed valid JSON output:

Automatic Retry

Transient LLM API failures are automatically retried with exponential backoff + jitter. Retryable errors include HTTP 429, 5xx, and network errors.
Default: 3 retries, 500ms initial delay, 10s max delay.

Token-Based History Trimming

Trim history on memory, not on Agent:
There is no maxContextTokens on Agent.

Background extraction

When memory.userFacts (or profile / entities / learnings) is on, extraction runs after the answer is sent. The user is not waiting on it. Tiny always-on notes (fileMemory) are injected into the prompt instead of being fetched as a tool.

Streaming usage and cost

Content can finish before a provider sends its final usage. A cumulative usage snapshot must replace the previous snapshot. Adding both snapshots counts the same work twice. Reasoning tokens are a subset of output for APIs that report inclusive output. For example, 19 input tokens and 16 output tokens, including 9 reasoning tokens, give 35 total tokens. They do not give 44. Cache reads and writes have separate quantities and prices. They are part of canonical input, so they must be removed from ordinary input before separate cache rates apply. Provider adapters must map their own input semantics first: Anthropic reports ordinary input separately from its cache counts. Use the cost guide for the supported mappings, unknown-cost states, and storage contract. Fixture checks establish behavior for the tested schemas; they do not establish accuracy for every provider or future API feature.

Methodology

  • All benchmarks use gpt-4o-mini with identical prompts
  • Each scenario runs 5 times; results are averaged
  • Startup time measures framework import + agent initialization
  • Cost uses gpt-4o-mini pricing: $0.15/1M input, $0.6/1M output
  • Network latency to OpenAI is shared across all frameworks
  • Full benchmark scripts are in benchmarks/ in the repository