> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentium.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Modal Input

> Send images, audio, and files to agents. MessageContent, ContentPart types, and examples.

# Multi-Modal Input

Agentium agents accept not only text but also **images**, **audio**, and **files**. Use the `MessageContent` type and `ContentPart[]` to send multi-modal input to vision and audio-capable models.

***

## MessageContent Type

Input to `agent.run()` or `agent.stream()` can be:

```typescript theme={null}
type MessageContent = string | ContentPart[];
```

* **string** — Plain text (most common)
* **ContentPart\[]** — Array of text, image, audio, or file parts

***

## ContentPart Types

<CardGroup cols={2}>
  <Card title="TextPart" icon="text">
    `{ type: "text", text: string }`
  </Card>

  <Card title="ImagePart" icon="image">
    `{ type: "image", data: string, mimeType? }`
  </Card>

  <Card title="AudioPart" icon="audio">
    `{ type: "audio", data: string, mimeType? }`
  </Card>

  <Card title="FilePart" icon="file">
    `{ type: "file", data: string, mimeType, filename? }`
  </Card>
</CardGroup>

***

## Image Input

Images can be provided as **base64** or **URL**:

```typescript theme={null}
import { Agent, openai, type ContentPart } from "@agentium/core";

const agent = new Agent({
  name: "VisionAgent",
  model: openai("gpt-4o"),
  instructions: "Describe and analyze images in detail.",
});

// Image via URL
const input: ContentPart[] = [
  { type: "text", text: "What's in this image?" },
  {
    type: "image",
    data: "https://example.com/image.png",
    mimeType: "image/png",
  },
];

// Image via base64
const base64Image = "data:image/png;base64,iVBORw0KGgo...";
const inputBase64: ContentPart[] = [
  { type: "text", text: "Analyze this." },
  { type: "image", data: base64Image, mimeType: "image/png" },
];

const result = await agent.run(input);
```

Supported `mimeType` values: `image/png`, `image/jpeg`, `image/gif`, `image/webp`.

***

## Audio Input

Audio is provided as **base64**-encoded data:

```typescript theme={null}
import { Agent, google, type ContentPart } from "@agentium/core";
import { readFileSync } from "node:fs";

const agent = new Agent({
  name: "AudioAnalyzer",
  model: google("gemini-2.5-flash"),
  instructions: "Transcribe and analyze audio content.",
  structuredOutput: AudioAnalysisSchema,
});

const audioData = readFileSync("sample.mp3");
const base64Audio = audioData.toString("base64");

const result = await agent.run([
  { type: "text", text: "Transcribe and summarize this audio." },
  { type: "audio", data: base64Audio, mimeType: "audio/mp3" },
] as ContentPart[]);
```

Supported `mimeType` values: `audio/mp3`, `audio/wav`, `audio/ogg`, `audio/webm`.

***

## File Input

Generic files (PDFs, documents, etc.) use `FilePart`:

```typescript theme={null}
const input: ContentPart[] = [
  { type: "text", text: "Summarize this document." },
  {
    type: "file",
    data: "https://example.com/doc.pdf",
    mimeType: "application/pdf",
    filename: "document.pdf",
  },
];
```

`data` can be a URL or base64-encoded content.

***

## Example: Vision Agent Analyzing an Image

```typescript theme={null}
import { Agent, openai, type ContentPart } from "@agentium/core";
import { z } from "zod";

const ImageAnalysis = z.object({
  description: z.string().describe("Detailed description of the image"),
  objects: z.array(z.string()).describe("Objects detected"),
  dominantColors: z.array(z.string()).describe("Dominant colors"),
  mood: z.string().describe("Overall mood"),
});

const analyzer = new Agent({
  name: "ImageAnalyzer",
  model: openai("gpt-4o"),
  instructions: "Analyze images and return structured JSON.",
  structuredOutput: ImageAnalysis,
});

const multiModalInput: ContentPart[] = [
  { type: "text", text: "Analyze this image in detail." },
  {
    type: "image",
    data: "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png",
    mimeType: "image/png",
  },
];

const result = await analyzer.run(multiModalInput);
console.log(result.structured);
```

***

## Example: Audio Analysis with Gemini

```typescript theme={null}
import { Agent, google, type ContentPart } from "@agentium/core";
import { readFileSync } from "node:fs";
import { z } from "zod";

const AudioAnalysis = z.object({
  transcription: z.string(),
  language: z.string(),
  speakerCount: z.number(),
  summary: z.string(),
  mood: z.string(),
  topics: z.array(z.string()),
});

const agent = new Agent({
  name: "AudioAnalyzer",
  model: google("gemini-2.5-flash"),
  instructions: "Analyze audio: transcribe, detect language, summarize.",
  structuredOutput: AudioAnalysis,
});

const audioData = readFileSync("audio/sample.mp3");
const base64Audio = audioData.toString("base64");

const result = await agent.run([
  { type: "text", text: "Analyze this audio clip in detail." },
  { type: "audio", data: base64Audio, mimeType: "audio/mp3" },
] as ContentPart[]);

console.log(result.structured);
```

***

## Provider Support Matrix

Not all providers support all content types. When an unsupported type is passed, the provider logs a warning and either skips the content or substitutes a placeholder.

| Content Type       | OpenAI | Anthropic | Google/Vertex | AWS Claude | AWS Bedrock | Azure OpenAI | Azure Foundry   | Ollama |
| ------------------ | ------ | --------- | ------------- | ---------- | ----------- | ------------ | --------------- | ------ |
| **Image** (URL)    | Yes    | Yes       | Yes           | Yes        | No          | Yes          | Model-dependent | No     |
| **Image** (base64) | Yes    | Yes       | Yes           | Yes        | Yes\*       | Yes          | Model-dependent | Yes    |
| **Audio** (base64) | Yes    | No        | Yes           | No         | No          | Yes          | No              | No     |
| **File** (URL)     | Yes    | Yes       | Yes           | Yes        | No          | Yes          | No              | No     |
| **File** (base64)  | Yes    | Yes       | Yes           | Yes        | Yes\*       | Yes          | No              | No     |

<Note>
  * Ollama image support requires a vision-capable model (e.g., `llava`, `bakllava`, `llama3.2-vision`).
  * AWS Bedrock multi-modal support (\*) depends on the specific model. Amazon Nova supports images; document support varies by model.
  * AWS Claude supports the same multi-modal features as the direct Anthropic provider.
  * Azure OpenAI supports the same multi-modal features as the direct OpenAI provider.
  * Azure AI Foundry vision support depends on the model (e.g., `Phi-3.5-vision-instruct` supports images).
</Note>

***

## Reading CSV Data

CSV files can be sent to Anthropic and OpenAI as file input. The model reads and analyzes the data directly:

```typescript theme={null}
import { Agent, anthropic, type ContentPart } from "@agentium/core";
import { readFileSync } from "node:fs";

const agent = new Agent({
  name: "DataAnalyst",
  model: anthropic("claude-sonnet-4-6"),
  instructions: "Analyze data files. Provide insights with specific numbers.",
});

// From a local CSV file
const csvData = readFileSync("sales-data.csv").toString("base64");
const result = await agent.run([
  { type: "text", text: "Analyze this sales data. What are the top 3 products by revenue?" },
  { type: "file", data: csvData, mimeType: "text/csv", filename: "sales-data.csv" },
] as ContentPart[]);

console.log(result.text);
// "Based on the sales data, the top 3 products by revenue are:
//  1. Widget Pro - $142,500 (1,425 units)
//  2. Gadget Plus - $98,200 (982 units)
//  3. Tool Basic - $67,800 (2,260 units)"
```

***

## Analyzing PDFs

PDF documents can be sent via URL (no download needed) or base64:

```typescript theme={null}
import { Agent, anthropic, type ContentPart } from "@agentium/core";

const agent = new Agent({
  name: "DocumentReader",
  model: anthropic("claude-sonnet-4-6"),
  instructions: "Extract key information from documents. Be thorough but concise.",
});

// PDF via URL — Anthropic fetches it directly
const result = await agent.run([
  { type: "text", text: "Summarize the key findings in this research paper." },
  {
    type: "file",
    data: "https://example.com/research-paper.pdf",
    mimeType: "application/pdf",
    filename: "paper.pdf",
  },
] as ContentPart[]);

// PDF via base64
import { readFileSync } from "node:fs";
const pdfData = readFileSync("contract.pdf").toString("base64");

const contractResult = await agent.run([
  { type: "text", text: "What are the payment terms and termination clauses?" },
  { type: "file", data: pdfData, mimeType: "application/pdf", filename: "contract.pdf" },
] as ContentPart[]);
```

***

## XLSX and Binary Formats

Most providers cannot process Excel (`.xlsx`) files directly. Google Gemini is the exception — it handles XLSX natively via `inlineData`.

For other providers, convert to CSV first:

```typescript theme={null}
import { parse } from "xlsx"; // npm install xlsx

function xlsxToCsv(filePath: string): string {
  const workbook = parse(readFileSync(filePath));
  const sheet = workbook.Sheets[workbook.SheetNames[0]];
  return XLSX.utils.sheet_to_csv(sheet);
}

const csvContent = xlsxToCsv("report.xlsx");
const csvBase64 = Buffer.from(csvContent).toString("base64");

const result = await agent.run([
  { type: "text", text: "Analyze this spreadsheet data." },
  { type: "file", data: csvBase64, mimeType: "text/csv", filename: "report.csv" },
] as ContentPart[]);
```

***

## Multi-Modal via HTTP File Upload

When exposing agents via Express, you can accept file uploads and convert them to `ContentPart[]`. The transport layer provides `buildMultiModalInput` for this:

See <a href="/transport/file-upload">File Upload</a> for how to handle `multipart/form-data` and build multi-modal input from uploaded files.