> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentium.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Jev as an eval judge

> Score a chat agent with Jev — noul, score, and choice — not a prose LLM judge.

# Jev as an eval judge

## In plain terms

Your **agent under test** is still Claude or GPT. It writes the reply.

**Jev** only scores that reply. You ask typed questions. You get numbers your suite can pass or fail.

```
Case input  →  support agent  →  reply text
                                    ↓
                         Jev: { faithful: 0.91, helpful: 2 }
                                    ↓
                         pass if faithful >= 0.7
```

Think of Jev as a calculator for quality, not a reviewer who writes an essay.

<Note>
  There is no `jevJudge()` helper. Wrap Jev in [`custom()`](/eval/overview#custom-scorer). Do **not** pass `jev()` to `llmJudge` or `AgentJudgeEval` — those ask a chat model to write `{"score", "reason"}`. Jev does not write text.
</Note>

Runnable file: [agentium-examples/eval/jev-judge.ts](https://github.com/agentiumOS/agentium-examples/blob/main/eval/jev-judge.ts).

```bash theme={null}
npm install @agentium/eval @typesafe-ai/sdk
export TYPESAFE_API_KEY="..."
```

***

## The smallest example

The agent under test can be any chat model. Jev is a **second** agent that only judges.

```typescript theme={null}
import { Agent, jev, noul, openai, score } from "@agentium/core";
import { EvalSuite, custom, ConsoleReporter } from "@agentium/eval";

const agent = new Agent({
  name: "support",
  model: openai("gpt-4o-mini"),
  instructions: "Help the customer. Stay on the ticket. Do not invent facts.",
});

const judge = new Agent({
  name: "jev-judge",
  model: jev("jev-latest"),
});

const suite = new EvalSuite({
  name: "support quality",
  agent,
  cases: [
    { name: "double-charge", input: "I was charged twice for last month." },
    { name: "outage", input: "The API returns 500 on /v1/shipments." },
  ],
  scorers: [
    custom("faithful", async (input, output) => {
      const result = await judge.run(
        JSON.stringify({ ticket: input, reply: output.text }),
        {
          questions: {
            faithful: noul(
              "Does the reply stay true to the ticket — no fake systems, no weather, no denying what the customer said?",
            ),
            helpful: score("How useful is this reply?", [
              "unhelpful",
              "partial",
              "useful",
              "excellent",
            ]),
          },
        },
      );
      const a = JSON.parse(result.text);
      const s = a.faithful.noul as number;
      return {
        score: s,
        pass: s >= 0.7,
        reason: `faithful=${s} helpful=${a.helpful.score}`,
      };
    }),
  ],
  threshold: 0.7,
});

const result = await suite.run([new ConsoleReporter()]);
console.log(`${result.passed}/${result.total} passed`);
```

`noul` is already 0–1 — use it as the scorer `score`. `score` is a 0-based rubric index (the live API may return a fraction — `Math.round` it if you treat it as a level).

Write `noul` questions the way you mean them. “Only state facts in the ticket” fails any next step (“I will refund today”) because that promise is not in the ticket. Ask “stay true to the ticket — no invented systems or denials” if a grounded offer to help should pass.

The canned example is meant to go **1 pass / 2 fail**: grounded billing passes; a made-up outage and a weather reply fail.

***

## Which Jev question for which check

| Jev question          | Eval job                      | Pass rule                |
| --------------------- | ----------------------------- | ------------------------ |
| `noul("…")`           | Yes/no quality gate           | `noul >= 0.7`            |
| `score("…", levels)`  | Rubric (helpfulness, safety)  | `Math.round(score) >= 2` |
| `choice("…", labels)` | Did it pick the right bucket? | `choice === expected`    |

Ask several in one `judge.run` — TypeSafe evaluates them in parallel against the same `{ ticket, reply }` state.

***

## Do not use `llmJudge` with Jev

```typescript theme={null}
// Broken — Jev cannot write this JSON
llmJudge({ model: jev("jev-latest"), criteria: ["relevance"] });

// Broken — same chat-shaped prompt
new AgentJudgeEval({ judge: jev("jev-latest"), criteria: ["…"] });
```

Those helpers send a prose prompt and parse `{"score": 0.9, "reason": "…"}`. Jev has nothing to ask, or cannot map a free-form `reason` string.

`jev_evaluate` on [JevToolkit](/toolkits/jev) is also not this. That is a **runtime** pack a chat agent can call. This page is **offline scoring** in `@agentium/eval`.

***

## When Jev is the better judge

Use Jev when:

* You want a **number**, not a paragraph
* The criterion is closed (yes/no, a rubric, a label)
* You want **confidence / probabilities**, not just a winner
* You want cheap, parallel questions on the same reply

Use [`llmJudge`](/eval/overview#llm-as-judge) when you need a **written reason**, an open-ended critique, or a multi-sentence comparison.

***

## See also

* [Eval overview](/eval/overview) — suites, `custom()`, reporters
* [Jev as the model](/models/jev) — `run({ questions })`
* [Jev examples](/examples/jev) — including this recipe
* [Eval examples](/examples/eval)
* [TypeSafe models](https://docs.typesafe.ai/models)