Evaluate tool reliability

ReliabilityEval checks run completion, non-empty text, and expected successful tool calls. Use it when the choice of action matters in addition to the final wording.

Quick Start

This host helper receives an Agent already configured with the application’s lookup_order tool. It makes model calls through that Agent; the host owns its lifecycle. Use a deterministic provider and fake service when testing the pipeline itself.

import type { Agent } from "@agentium/core";
import { ReliabilityEval } from "@agentium/eval";

export async function verifyOrderLookup(agent: Agent) {
  return new ReliabilityEval({
    name: "order-lookup", agent, timeoutMs: 30_000,
    cases: [{
      name: "looks-up-an-order", input: "Check the status of order-42 using lookup_order.",
      expectedTools: ["lookup_order"],
    }],
  }).run();
}

Inspect result.failed and each case’s scores. Merely mentioning a tool name in text does not pass the tool assertion. Calls with an error or denial do not count as successful expected calls.

Case Options

Field	Meaning
`expectedTools`	Names that must appear among successful tool calls
`shouldError`	Expect the Agent invocation to throw an execution error

A tool can fail and be handled inside the Agent loop without the invocation throwing. Do not use shouldError: true as a general assertion about a tool error. Timeouts and cancellation are evaluation failures rather than successful expected errors.

Tool Call Match Scorer

Use the scorer inside an EvalSuite when combining tool assertions with application output checks:

import type { Agent } from "@agentium/core";
import { EvalSuite, toolCallMatch } from "@agentium/eval";

export function orderToolSuite(agent: Agent) {
  return new EvalSuite({
    name: "order-tools", agent,
    cases: [{ name: "lookup", input: "Look up order-42 with lookup_order." }],
    scorers: [toolCallMatch(["lookup_order"])],
  });
}

This checks invocation evidence, not that an external system applied an effect. Verify service receipts and idempotency in your application tests. Continue with conversation trajectories when tool order across multiple turns matters.

Build

Agent applications

Harness runtime

Teams and workflows

Data and capabilities

Interfaces and jobs

Voice and telephony

Verify and manage

Evaluate tool reliability

Quick Start

Case Options

Tool Call Match Scorer

​Quick Start

​Case Options

​Tool Call Match Scorer

Quick Start

Case Options

Tool Call Match Scorer