Build a browser agent

@agentium/browser operates web pages through Playwright. Use BrowserAgent when a model should choose browser actions, or BrowserProvider when your application already knows the navigation and interactions. Prefer an application API/tool when the task does not require a UI.

Run your first browser task

This example reads the heading on a public example page. It requires the supported Node runtime, Chromium, an OpenAI key, and network access to both the model API and the page. It does not run offline. In a new project:

npm init -y
npm pkg set type=module
npm install @agentium/core@4.0.0 @agentium/browser@4.0.0 openai playwright
npm install --save-dev typescript tsx @types/node
npx playwright install chromium
export OPENAI_API_KEY="your-key"
export OPENAI_MODEL="gpt-6.1-sol"

Save as read-page.ts:

read-page.ts

import { BrowserAgent } from "@agentium/browser";
import { openai } from "@agentium/core";

const browser = new BrowserAgent({
  name: "page-reader",
  model: openai(process.env.OPENAI_MODEL ?? "gpt-6.1-sol"),
  startUrl: "https://example.com",
  allowedDomains: ["example.com"],
  useDOM: true,
  useVision: false,
  maxSteps: 6,
});

const result = await browser.run("Read the main heading of this page. Report the heading and finish.");
if (!result.success) throw new Error(result.result);
console.log(result.result);
console.log("Final URL:", result.finalUrl);
console.log("Actions:", result.steps.map((step) => step.action.action));

npx tsx read-page.ts

Expect the response to identify the page’s “Example Domain” heading. Inspect finalUrl and the recorded actions as well as the answer. Exact model wording and action count vary. BrowserAgent.run() owns and closes its per-run browser connection when it finalizes; there is no BrowserAgent.close() method in v4. Change the task to read a different fact on the same page. Then try your own permitted page, updating the starting URL and domain policy together. For a visual task, select a vision-capable model and enable vision.

Choose a planning and observation path

Need	Configuration	What changes
Read accessible text and controls	`useDOM: true`, `useVision: false`	The model receives DOM observations without per-step screenshots
Interpret visual layout	`useVision: true` or `"auto"`	The model must support the screenshot input used by the run
Choose from a bounded control set	`planner: "jev"`	Jev selects actions; a text model still supplies text/search strings
Extract page content with another model	`pageExtractionLLM`	The `extract` action can use a separately configured provider
Run known setup actions first	`initialActions`	The host specifies initial navigation/interactions before the model loop
Attach to an existing browser	`cdpUrl`	The host owns the remote browser and its existing account state

DOM indexes identify controls in the current observation. Refresh the observation after navigation or a meaningful DOM change; an earlier index is not a permanent element identifier. Coordinates are a fallback in CSS pixels and depend on the current viewport.

Jev planner

Host factory: supply a text model and configure TypeSafe credentials according to Jev setup. Do not pass jev() as a screenshot model. The returned BrowserAgent owns the browser connection used by each run.

import { BrowserAgent } from "@agentium/browser";
import type { ModelProvider } from "@agentium/core";

export function createJevBrowser(textModel: ModelProvider) {
  return new BrowserAgent({
    name: "control-picker", model: textModel,
    planner: "jev", jevModel: "jev-latest",
    useVision: false, useDOM: true,
    maxActionChoices: 40, maxSteps: 10,
    allowedDomains: ["example.com"],
    startUrl: "https://example.com",
  });
}

Evaluate completion and selected actions for the actual page. A bounded choice set does not establish that every selected action is appropriate for your application’s domain rules.

Use direct browser operations

Host function: pass a URL the host has decided to visit. This example launches Chromium, navigates, reads the DOM, and closes the browser without a model call. It still performs page network I/O and requires the browser binary.

import { BrowserProvider } from "@agentium/browser";

export async function readPageDOM(url: string) {
  const browser = new BrowserProvider();
  try {
    await browser.launch();
    await browser.navigate(url);
    const snapshot = await browser.extractDOM({ maxElements: 50 });
    return { text: snapshot.text, elements: snapshot.elements };
  } finally {
    await browser.close();
  }
}

For tabs, use newTab(), listTabs(), switchTab(), and closeTab() on the provider. Keep the returned tab IDs rather than assuming a particular ID. Refresh each tab’s DOM before using its element indexes.

Compose a browser as an Agent tool

Host function: the parent Agent decides when to invoke a browser task. Both model calls require the supplied provider’s capabilities. This function owns the parent Agent; BrowserAgent owns the connection created by its browser run.

import { Agent, type ModelProvider } from "@agentium/core";
import { BrowserAgent } from "@agentium/browser";

export async function answerFromPage(model: ModelProvider, question: string) {
  const browser = new BrowserAgent({
    name: "page-reader", model, useDOM: true, useVision: false,
    allowedDomains: ["example.com"], startUrl: "https://example.com", maxSteps: 6,
  });
  const assistant = new Agent({
    name: "page-assistant", model, register: false,
    instructions: "Use the browser tool to read the selected page. Cite its URL and acknowledge missing evidence.",
    tools: [browser.asTool()], maxToolRoundtrips: 2,
  });
  try {
    const result = await assistant.run(question);
    if (result.status !== "completed") throw new Error(`Page assistant: ${result.status}`);
    return result.text;
  } finally {
    await assistant.close();
  }
}

asTool() carries the parent identity, policy, and cancellation context. Custom ToolDef execution uses core validation, execution policy, and approval. Native browser click/type/navigation/evaluate actions are not covered by that custom-tool policy. Plan mode rejects browser execution before launch.

Configure the application boundary

Concern	Options	Host responsibility
Task limits	`maxSteps`, `maxFailures`, `maxActionsPerStep`, `maxRepeats`	Check `success` and preserve partial evidence when work stops
Observation	`useDOM`, `useVision`, `viewport`, `maxActionChoices`	Select a model and representation appropriate to the page
Model behavior	`model`, `planner`, `jevModel`, `pageExtractionLLM`, `fallbackModel`, `useThinking`	Credentials, capabilities, quality checks, and pricing
Context	`instructions`, `extendSystemMessage`, `overrideSystemMessage`, `historyWindow`, `memory`, `skills`	Trusted instructions, scoped state, and retained content
Navigation	`startUrl`, `searchEngine`, `directlyOpenUrl`, `initialActions`, `allowedDomains`, `prohibitedDomains`	Intended browsing scope and external effects
Execution	`tools`, `executionPolicy`, `approval`, `approvalManager`, `allowEvaluate`	Authorize custom tools and understand native-action limits
Browser lifetime	`headless`, `waitAfterAction`, `cdpUrl`	Local dependencies or ownership of a remote browser
Account state	`storageState`, `credentials`	Identity, file access, retention, and session reuse
Browser environment	`stealth`, `humanize`	Test the chosen browser configuration; success on a site is not guaranteed
Evidence	`recordVideo`, `eventBus`, `logLevel`, `costTracker`	Content handling, exporter destination, and ownership

This is a configuration guide, not a replacement for the exported BrowserAgentConfig, BrowserRunOpts, BrowserRunOutput, and BrowserAction types in the installed package. See the package map for matching versions.

Account state and credentials

storageState loads a Playwright state file; saveStorageState on a run writes one for a later connection. Saving state does not guarantee a login succeeded or that a future session remains authenticated. Verify the application’s signed-in state and treat the file as account data. CredentialVault substitutes selected values during actions so the prompt can use placeholders. This does not mean every page string, screenshot, recording, or tool result is redacted. Filled form values are excluded from DOM labels, while arbitrary page text and screenshot pixels can still contain sensitive data. Decide what may leave your browser environment before enabling observation or recording. Domain restrictions constrain the navigation paths checked by the runtime; they are not a complete network isolation policy for every browser subresource or page effect. allowEvaluate enables JavaScript execution in page context and is disabled by default. Local browser actions and host tool execution have different authorization boundaries.

Recorded evidence and cost

Read steps, finalUrl, finalScreenshot, optional extractedContent, and optional videoPath from the result. A step screenshot and a final screenshot serve different purposes; inspect the output rather than assuming one image proves the entire task. Supply model-keyed prices when using CostTracker; follow cost patterns. A budget estimate is not a guaranteed cap on provider billing. Step limits and a bounded workload are independent controls.

Connect a local Socket.IO interface

The v4 browser gateway is a browser-specific adapter. It does not share all the authenticated text gateway’s session ownership and cancellation contracts. Host integration: the host supplies a Socket.IO server and browser instance. Use this for a trusted local interface; bind its underlying HTTP server to loopback and close the server when finished.

import type { BrowserAgent } from "@agentium/browser";
import { createBrowserGateway } from "@agentium/transport";
import type { Server } from "socket.io";

export function attachLocalBrowserView(io: Server, browser: BrowserAgent) {
  createBrowserGateway({
    io, agents: { reader: browser },
    streamScreenshots: false,
  });
}

Direction	Event	Meaning
Client → server	`browser.start`	Request a task for a configured agent name
Client → server	`browser.stop`	Stop forwarding the tracked run’s output
Server → client	`browser.started`	Request accepted
Server → client	`browser.action`, `browser.step`	Action and step observations
Server → client	`browser.screenshot`	Screenshot forwarding when enabled
Server → client	`browser.done`, `browser.error`, `browser.stopped`	Completion, error, or forwarding-stop notification

In v4, the gateway does not pass its abort signal into BrowserAgent.run(). Stop/disconnect therefore does not establish that browser execution or an external effect stopped. A host needing execution cancellation should call run() with its own signal and preserve the identity/ownership boundary. The optional authMiddleware can control admission; it does not automatically isolate shared browser instances or their event streams between users.

Diagnose a failed browser task

Symptom	First check
Browser executable missing	Run `npx playwright install chromium` in the environment used by the application
Model request fails	Credentials, model access, and the selected input modality
Wrong control is selected	Current DOM observation, indexes, viewport, and page changes
`success: false` with partial steps	Result message, action history, and configured limits
Logged-out page after state reuse	Whether authentication actually succeeded and the saved state is still valid
Expected text absent from traces	Capture settings and which events/output fields the host retains

The documentation examples are type-checked; this does not establish live model, target-site, or login success. Test against an owned development page before connecting an application account.

More patterns

Continue with browser examples, core tool patterns, sandbox execution, and shipping profiles. Browser automation, API tools, and code execution solve different integration tasks.

Build

Agent applications

Harness runtime

Teams and workflows

Data and capabilities

Interfaces and jobs

Voice and telephony

Verify and manage

Build a browser agent

Run your first browser task

Choose a planning and observation path

Jev planner

Use direct browser operations

Compose a browser as an Agent tool

Configure the application boundary

Account state and credentials

Navigation and page execution

Recorded evidence and cost

Connect a local Socket.IO interface

Diagnose a failed browser task

More patterns

​Run your first browser task

​Choose a planning and observation path

​Jev planner

​Use direct browser operations

​Compose a browser as an Agent tool

​Configure the application boundary

​Account state and credentials

​Navigation and page execution

​Recorded evidence and cost

​Connect a local Socket.IO interface

​Diagnose a failed browser task

​More patterns

Run your first browser task

Choose a planning and observation path

Jev planner

Use direct browser operations

Compose a browser as an Agent tool

Configure the application boundary

Account state and credentials

Navigation and page execution

Recorded evidence and cost

Connect a local Socket.IO interface

Diagnose a failed browser task

More patterns