Accuracy Evaluation - Agentium

Overview

AccuracyEval uses an LLM judge to score agent responses against expected answers on a 0.0–1.0 scale.

Quick Start

Install @agentium/core@4.0.0, @agentium/eval@4.0.0, and the optional openai SDK. Set OPENAI_API_KEY; this example makes live provider calls. For a no-key runnable project, start with the quality-gate recipe.

import { AccuracyEval } from "@agentium/eval";
import { Agent, openai } from "@agentium/core";

const agent = new Agent({ name: "qa-bot", model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol") });

const evaluator = new AccuracyEval({
  name: "qa-accuracy",
  agent,
  judge: openai(process.env.OPENAI_FAST_MODEL ?? "gpt-6-luna"),
  cases: [
    { name: "capital", input: "What is the capital of France?", expected: "Paris" },
    { name: "math", input: "What is 2+2?", expected: "4" },
  ],
  threshold: 0.8,
});

try {
  const result = await evaluator.run();
  console.log(`${result.passed}/${result.total} passed`);
  if (result.failed > 0) process.exitCode = 1;
} finally {
  await agent.close();
}

Configuration

Option	Type	Default	Description
`name`	`string`	required	Name of the evaluation
`agent`	`Agent`	required	Agent to evaluate
`judge`	`ModelProvider`	required	Model used for scoring
`cases`	`EvalCase[]`	required	Test cases with input/expected
`threshold`	`number`	`0.7`	Minimum score to pass
`timeoutMs`	`number`	`30000`	Timeout per case