Eval examples

Runnable files: agentium-examples/eval. Guide: Eval

npm install @agentium/eval

1. A small suite

import { Agent, openai } from "@agentium/core";
import { EvalSuite, contains, regexMatch, custom, ConsoleReporter } from "@agentium/eval";

const agent = new Agent({
  name: "eval-target",
  model: openai("gpt-4o-mini"),
  instructions: "Be factual and short.",
});

const suite = new EvalSuite({
  name: "Basic Quality Suite",
  agent,
  cases: [
    { name: "Capital of France", input: "What is the capital of France?", expected: "Paris" },
    { name: "Simple math", input: "What is 15 * 7?", expected: "105" },
  ],
  scorers: [
    contains("Paris"),
    regexMatch(/\d+/),
    custom("non-empty", async (_input, output) => {
      const pass = output.text.length > 10;
      return { score: pass ? 1 : 0, pass, reason: pass ? undefined : "too short" };
    }),
  ],
  threshold: 0.5,
  concurrency: 2,
});

const result = await suite.run();
new ConsoleReporter().report(result);

2. Jev as the judge

The chat agent writes the reply. Jev scores it with noul / score / choice. Do not pass jev() to llmJudge. Guide: Jev as an eval judge. File: eval/jev-judge.ts.

import { Agent, jev, noul } from "@agentium/core";
import { custom } from "@agentium/eval";

const judge = new Agent({ name: "jev-judge", model: jev("jev-latest") });

custom("faithful", async (input, output) => {
  const result = await judge.run(
    JSON.stringify({ ticket: input, reply: output.text }),
    { questions: { faithful: noul("Does the reply stay true to the ticket — no fake systems or denials?") } },
  );
  const s = JSON.parse(result.text).faithful.noul;
  return { score: s, pass: s >= 0.7, reason: `noul=${s}` };
});

TYPESAFE_API_KEY=... npx tsx eval/jev-judge.ts

3. Conversational cases

See agentium-examples/eval/conversational-test.ts and Conversational eval.