> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentium.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraper

> Extract text content and links from web pages — no dependencies.

# Web Scraper

Extract text content and links from any web page. Uses native `fetch` and lightweight HTML stripping — no browser or heavy dependencies needed.

***

## Quick Start

```typescript theme={null}
import { Agent, openai, ScraperToolkit } from "@agentium/core";

const scraper = new ScraperToolkit({ maxLength: 10_000 });

const agent = new Agent({
  name: "reader",
  model: openai("gpt-4o"),
  instructions: "Read web pages and summarize their content.",
  tools: [...scraper.getTools()],
});

const result = await agent.run("Summarize the content of https://agentium.dev");
```

***

## Config

<ParamField body="maxLength" type="number" default="15000">
  Max characters of extracted text to return.
</ParamField>

<ParamField body="userAgent" type="string">
  Custom User-Agent header for requests.
</ParamField>

<ParamField body="timeout" type="number" default="15000">
  Request timeout in milliseconds.
</ParamField>

***

## Tools

| Tool           | Description                                                                          |
| -------------- | ------------------------------------------------------------------------------------ |
| `scrape_url`   | Fetch a URL and extract text content. Scripts, styles, nav, and footer are stripped. |
| `scrape_links` | Extract all links from a page. Returns link text and absolute URLs.                  |

***

## Tool Usage Examples

### Scrape and Summarize

```typescript theme={null}
const result = await agent.run(
  "Read https://agentium.dev/docs/getting-started and give me a quick summary"
);

// The agent calls scrape_url with:
// { url: "https://agentium.dev/docs/getting-started" }
//
// Returns extracted text (HTML stripped, max 15000 chars):
// "Getting Started\n\nAgentium is an open-source framework for building AI agents..."
//
// The agent then summarizes the content for the user
```

### Extract Links

```typescript theme={null}
const result = await agent.run(
  "What documentation pages are linked from https://agentium.dev/docs?"
);

// The agent calls scrape_links with:
// { url: "https://agentium.dev/docs" }
//
// Returns:
// [
//   { text: "Getting Started", url: "https://agentium.dev/docs/getting-started" },
//   { text: "Agents", url: "https://agentium.dev/docs/agents/overview" },
//   { text: "Tools", url: "https://agentium.dev/docs/agents/tools" },
//   ...
// ]
```

### Research Agent

Combine the scraper with other toolkits for a research agent:

```typescript theme={null}
import { Agent, openai, ScraperToolkit, HttpToolkit } from "@agentium/core";

const agent = new Agent({
  name: "researcher",
  model: openai("gpt-4o"),
  tools: [
    ...new ScraperToolkit({ maxLength: 15_000 }).getTools(),
    ...new HttpToolkit({ baseUrl: "https://api.example.com" }).getTools(),
  ],
  instructions: `You are a research assistant. Use the scraper to read web pages
and the HTTP toolkit to query APIs. Synthesize information from multiple sources.`,
  toolResultLimit: { maxChars: 20_000, strategy: "summarize", model: openai("gpt-4o-mini") },
});

const result = await agent.run(
  "Research the latest trends in AI agents and summarize the key findings"
);
```