> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentium.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Recover unfinished execution graphs

> Use versioned host catalogs, JSON DAG plans, retained node evidence, root limits, and bounded planning with ExecutionGraphRuntime.

Use `ExecutionGraphRuntime` when a graph must retain per-node progress and reconcile interrupted work. The host supplies a versioned catalog of executable tools, Agents, nested graphs, and bounded repeats. A static plan or optional model planner selects those operations through JSON; a proposal cannot supply code or grants.

This guide describes **Agentium 4.9.0**. Review [release and upgrade guidance](/reference/releases), then start with the [provider-free graph project](/examples/execution-graph) for a complete executable example.

## Choose the recovery boundary

| Requirement | Use | Retained boundary |
| - | - | - |
| Dependency scheduling with trusted callbacks | [Workflow graph](/workflows/graphs) | Complete top-level graph or repeat checkpoint |
| Recover completed nodes and reconcile interrupted dispatch | `ExecutionGraphRuntime` | Fenced node/model attempts, outputs, plan revisions and action evidence |
| General Agent memory, hooks, mutable sessions or handoffs | Ordinary [Agent](/agents/overview) or [HarnessRuntime](/harness/runtime) | Its existing session and execution contract |

The graph adapter reuses Workflow, Agent, ToolExecutor and the durable task protocol. Ordinary `workflowDriver` remains non-durable. Recovery requires stateless JSON-compatible bindings; unsupported mutable or opaque behavior fails preflight.

```mermaid theme={null}
flowchart LR
  H[Host catalog and grants] --> V[Validate complete JSON DAG]
  P[Optional bounded planner] --> V
  V --> W[Workflow and Agent execution]
  W --> R[Shared root admission]
  R --> J[Fenced node journal and action ledger]
  J --> S[Host durable store and receipts]
```

## Bind executable behavior on the host

`GraphBinding` has four variants. Every binding requires `id`, `version`, `input` and `output` schemas.

| Kind | Host binding | Output and recovery contract |
| - | - | - |
| `tool` | A `ToolDef`, including a controlled function in `execute` | JSON output; explicit pure declaration or durable effect connector |
| `agent` | An existing `Agent` and approved catalog tool IDs | `{ text, usage }`; bounded output and supported static continuation |
| `graph` | A versioned nested `ExecutionGraphProgram` | Map of node IDs to owned JSON outputs |
| `repeat` | A nested program, `maxIterations` and pure `while` callback | Persisted guard decisions and iteration plans |

Choose exactly one of `program.plan` or `program.planner`. A plan node names a catalog `binding` and exact `version`, with a JSON `args` object. The entire candidate is validated before its work can dispatch: identities, schemas, references, cycles, operation versions, bytes, nodes, depth, concurrency and supported recovery policy.

`dependsOn` provides control ordering. `inputsFrom` declares data dependencies. Argument references also infer a data dependency:

```json theme={null}
{"$ref":{"node":"producer","path":["field"]}}
```

Declare exact dot-separated `referenceInputs` and `referenceOutputs` paths on bindings; `""` denotes the whole value. Supply a pure `validateTemplate` callback for future-valued inputs. The runtime resolves references and validates concrete arguments before dispatch. Static argument templates refer to producer outputs; root program input informs planning.

## Persist compatible task identity

Each task pins its manifest/catalog digest, program and binding versions, original input, policy revision, grants, budgets and absolute deadline. Plain plans, nested templates, limits, budgets and grants are copied at construction. Opaque schemas, provider/Agent objects and callback implementations remain host-attested by version. Bump versions when instructions, schemas, callable code or captured dependencies change.

Use a verified persistent `DurableTaskStore`, such as `MongoDBDurableTaskStore`, and set `requireDurable: true`. The host owns credentials, database lifecycle, authenticated tenant/actor admission, atomic CAS, fencing and receipt retention. Default `InMemoryDurableTaskStore` advertises `capabilities.durable: false`; it is a local fixture and cannot survive process loss. `requireDurable: true` rejects it.

`create()` admits the task. `run(key, workerId)` acquires fenced ownership and executes or recovers it. `inspect(key)` validates retained state without claiming ownership or writing. `cancel(key, reason)` requests persisted cancellation observed by active lease polling. A completed task returns retained output without another dispatch. [Generated graph reference](/harness/api/execution-graphs) contains complete signatures.

## Recover nodes and effects

Materialized revisions and node incarnations have stable IDs. Inputs, outputs and attempts carry digests and worker fences. Completion persists before successors start. Completed nodes reuse their output; interrupted running work starts under a newer fence; failed or canceled work remains terminal. A changed descriptor or stale fence fails before resumed effects.

Restricted Agent nodes restart from node entry. Complete recorded model responses replay at request-bound positions with stable tool-call IDs. Approved tools are reauthorized. Confirmed raw outputs and receipts reuse stable budget admission IDs without duplicate logical charges. Interrupted model reservations remain conservatively charged as unknown usage. Recovery replenishes neither budgets nor the original deadline.

A tool must declare `recovery: "pure"` or provide a versioned `DurableActionConnector`, host-bound destination, and immutable `resolveResult(resultRef)`. `pure` is the host's assertion that the callback performs no external mutation. The SDK cannot prove that assertion.

For effects, action identity binds node incarnation, call, binding/version, actual parsed arguments, destination, policy/grants and caller identity. The declared connector receives those arguments; the arbitrary tool callback is not substituted for it. Append-only origin evidence selects the originating catalog operation even when two connectors share destination/version.

Root admission occurs before remote dispatch is recorded as executing. Local admission rejection remains known not dispatched. The permit covers dispatch and receipt settlement; confirmed replay and reconciliation need no fresh dispatch allowance.

If a destination accepts an effect and the process dies before the receipt persists, recovery needs confirmed reconciliation, proven absence or destination idempotency. Missing or transient lookup evidence cannot justify blind redispatch. An unresolved outcome raises `DurableRecoveryRequiredError`. Fencing prevents stale local settlement but cannot retract physical remote work. **Exactly-once external execution and remote rollback are not guaranteed.** See [recovery guidance](/ship/recovery).

Result caches, `toModelOutput`, dynamic Agent tool resolvers, handoffs, mutable memory, hooks and unsupported opaque continuation fail preflight. Recovery uses stateless harness configuration; ordinary session/continuity policies remain available through `HarnessRuntime`.

## Bound all controlled work at the root

One root slot covers an actual provider request/stream or controlled tool/function callback. Graph, repeat and Agent containers hold no slot while waiting for children, so capacity one can complete without a parent-child deadlock. Agent tools, nested graphs, repeats and planning share the controller. Nested controlled dispatch from inside a leaf is rejected. FIFO admission checks cancellation and deadline.

| Root limit | What it counts across the task lifetime |
| - | - |
| `maxConcurrentWork` | Active controlled leaf dispatches; model streams hold their slot through iterator closure |
| `maxExpandedNodes` | Unique materialized node incarnations, including pending replacements |
| `maxWorkAttempts` | Fresh leaf dispatches, including Agent/planner models and tools |
| `maxPlanRevisions` | Admitted revisions across root, nested and repeat episodes |

Set explicit ceilings for your application. Model/tool/token budgets apply in addition to root limits. Expansion, attempt and revision counters persist before dispatch; removed nodes and interrupted reservations do not refund allowance. Provider transport retries retain their existing owner: one controlled provider invocation is one root work attempt. Queue/admission/release/exhaustion events are available through `onAdmission` and controller snapshots.

Already-owned stream cleanup may finish after cancellation through the lease's cleanup path, without admitting more work. A permit releases only after the owned operation settles. Receipt lookup and standalone storage/reconciliation do not occupy dispatch slots. Providers/connectors must honor cooperative cancellation and timeout; an SDK lease cannot preempt accepted remote work.

## Add bounded model planning

Replace the static plan with `program.planner` containing `provider`, `hostInstructions`, `maxCorrections`, `maxOutputTokens`, and `stream` bounds. `mode` is `"complete"` by default or `"stream"`. Both paths are tool-free, use authorized model ports and share root admission and harness budgets.

Host instructions retain system authority. Catalog descriptions, inputs, retrieved/tool content and observations are source data. Model text can propose an operation; validation and grants determine whether it may execute. Accepted/rejected proposals and causal diagnostics remain in the journal. Correction and fresh-call ownership stay bounded across recovery. Permanent exhaustion fails with the originating validation cause.

Plan defaults are `maxBytes: 128000`, `maxNodes: 100`, `maxDepth: 32`, and `maxConcurrency: 5`; narrow them with `limits.plan`. `maxDepth` bounds cumulative dependency and nesting depth, including dynamically planned children. Repeat iterations share nesting depth and consume lifetime expansion/revision allowances.

## Stream complete pending revisions

Streaming accepts newline-terminated, self-contained frames with `{ baseRevision, plan }`:

```json theme={null}
{"baseRevision":0,"plan":{"schemaVersion":1,"revision":1,"nodes":[{"id":"result","binding":"echo","version":"1","args":{"value":"hello"}}]}}
```

The frame advances the current revision by one; identical retransmission is idempotent. Unclosed fragments and unresolved prerequisites never execute. Fenced CAS serializes revision commit and node claim.

Only **unclaimed pending** nodes may change or be removed. Claimed/prepared/running/completed/failed/uncertain definitions and relevant edges remain immutable. Replacements retain tombstones and spend new incarnation allowance. Old compiled callbacks cannot execute superseded work; no revision rolls back applied effects.

The producer and scheduler progress independently. At capacity greater than one, a committed prefix can execute while more frames arrive. At capacity one, dispatch waits for the model stream slot. Planning seals only after coherent terminal provider usage and EOF. A malformed, truncated, interrupted or canceled continuation retains committed work and evidence but cannot report planning success. Recovery makes a bounded continuation request from the current revision; sealed plans need no new planning call.

Configure `maxFrameBytes`, cumulative `maxTotalBytes`/`maxFrames`, output-token and correction limits. `maxLatencyMs` defaults to 30 seconds per planning call. All apply alongside root limits and the original deadline.

## Validate your deployment contract

Local fixtures check declared behavior. The SDK's graph acceptance suite also exercises fresh-process SIGKILL windows with a disposable MongoDB adapter and separately records backend and packed-consumer evidence. Those results do not establish live-model planning quality, failover, production performance or your connector's idempotency.

For your adapter, exercise before-dispatch, accepted-before-receipt, completed-node reuse, stale fencing, cancellation, exhausted budgets and interrupted stream recovery. Inspect destination counts, receipts and retained counters. Keep the [complete recipe](/examples/execution-graph), [Workflow graph guide](/workflows/graphs), and [durable storage contract](/durable/overview) as distinct integration starting points.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.