> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentium.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval examples

> Score an agent with a suite of cases. Run it in CI.

# Eval examples

Runnable files: [agentium-examples/eval](https://github.com/agentiumOS/agentium-examples/tree/main/eval).

Guide: [Eval](/eval/overview)

```bash theme={null}
npm install @agentium/eval
```

***

## 1. A small suite

```typescript theme={null}
import { Agent, openai } from "@agentium/core";
import { EvalSuite, contains, regexMatch, custom, ConsoleReporter } from "@agentium/eval";

const agent = new Agent({
  name: "eval-target",
  model: openai("gpt-4o-mini"),
  instructions: "Be factual and short.",
});

const suite = new EvalSuite({
  name: "Basic Quality Suite",
  agent,
  cases: [
    { name: "Capital of France", input: "What is the capital of France?", expected: "Paris" },
    { name: "Simple math", input: "What is 15 * 7?", expected: "105" },
  ],
  scorers: [
    contains("Paris"),
    regexMatch(/\d+/),
    custom("non-empty", async (_input, output) => {
      const pass = output.text.length > 10;
      return { score: pass ? 1 : 0, pass, reason: pass ? undefined : "too short" };
    }),
  ],
  threshold: 0.5,
  concurrency: 2,
});

const result = await suite.run();
new ConsoleReporter().report(result);
```

***

## 2. Jev as the judge

The chat agent writes the reply. Jev scores it with `noul` / `score` / `choice`. Do not pass `jev()` to `llmJudge`.

Guide: [Jev as an eval judge](/eval/jev). File: [eval/jev-judge.ts](https://github.com/agentiumOS/agentium-examples/blob/main/eval/jev-judge.ts).

```typescript theme={null}
import { Agent, jev, noul } from "@agentium/core";
import { custom } from "@agentium/eval";

const judge = new Agent({ name: "jev-judge", model: jev("jev-latest") });

custom("faithful", async (input, output) => {
  const result = await judge.run(
    JSON.stringify({ ticket: input, reply: output.text }),
    { questions: { faithful: noul("Does the reply stay true to the ticket — no fake systems or denials?") } },
  );
  const s = JSON.parse(result.text).faithful.noul;
  return { score: s, pass: s >= 0.7, reason: `noul=${s}` };
});
```

```bash theme={null}
TYPESAFE_API_KEY=... npx tsx eval/jev-judge.ts
```

***

## 3. Conversational cases

See [agentium-examples/eval/conversational-test.ts](https://github.com/agentiumOS/agentium-examples/blob/main/eval/conversational-test.ts) and [Conversational eval](/eval/conversational).