Approves an approval request of the run or of one of its sub-agents (the reference names the run). context is the application’s current context: the policy is checked again with it, and a current denial or policy failure wins over the approval; a policy that now requires another approval refuses it with RequirementChanged. The approved action runs with exactly this context, whether or not a runner was live: the tool body receives it, and an approved sub-agent start starts the child run with it. It does not become the run’s context: the run’s other actions keep the context it was started or recovered with. If the run’s runner is lost before the approved action starts, recovery asks again (see recover).

Works with no process holding the run: the answer is committed to the stored record with compare-and-set, so of concurrent answers exactly one wins and the others get AlreadyAnswered (or RunEnded after a cancel). The caller then emits the commit’s events before approve returns (see fabric/telemetry); the approved work has already started.

An answer to a sub-agent run whose ancestor is stopping or has ended is refused with RunEnded, even if the sub-agent’s own cancellation has not been committed yet. An answer checked just before an ancestor’s cancellation was committed may still commit, but nothing it approved starts: a sub-agent run checks its ancestors again before each tool or sub-agent start, and one that finds an ancestor stopping or ended starts nothing and cancels itself.

An answer after the request’s deadline (agent.with_approval_expiry) is refused with ApprovalExpired: the request expires instead, its action is rejected, and the run goes on.

proof says who answers: approvers.check made it for the request’s requirement (reference.requirement) with the approvers of the agent whose run issued the request (agent.with_approvers). Any other proof, or one older than the proof lifetime, is refused with ProofRefused before the request is checked. The reviewer it names and the approvers’ name are recorded with the answer (run.Approval.reviewer, run.Approval.verifier).

Blocks until the run is no longer Working, or until within passes, and returns its status: Working when the time ran out, Suspended or Finished, or Unattended when work is in flight but no runner known to this store drives it. On an unleased store, Unattended means the runner was lost or handed the run off at shutdown, or the run is driven through another Store; only the application knows which, and recover takes the run over, so call it only when the previous owner is known to be gone. On a leased store a run whose lease is live elsewhere is Working. Unattended means its lease is free or expired, belongs to an earlier process of this store, or belongs to this store with no runner: recover takes it over.

It wakes on commits made through this run’s store and when a runner exits; on a leased store it also reads the run again at least every third of the lease (at most every second), since another node’s commits wake nothing here. If the store’s process stops meanwhile, it waits, until within runs out, for a supervisor to register the next one, and goes on through it; that process knows no runner, so work in flight then reads Unattended. With no store process running when it starts, or none registered again in time, it is StoreUnavailable. A run whose sub-agents work is working; one waiting only on paused sub-agents is suspended on their approvals. A suspended run with an approval request whose deadline passed is not returned as is: the request is rejected first, and the wait goes on with the run. A within of zero (or less) reads the status now; one longer than the runtime’s longest timer (2^32 - 1 ms) is waited in parts.

To wait for the run or for something else at once (a caller’s cancellation, a shutdown message), use await_with.

await, ending early when or receives a message: one receive waits for the run and for the caller’s own messages, so a handler needs no helper process. The message is returned as Interrupted(message) and is consumed; the run is left as it is. A request handler that must stop the run when its caller goes away selects on that signal:

let cancelled = relay_tool.cancelled(call)   // a process.Selector(Nil)
case fabric.await_with(handle, within: duration.seconds(30), or: cancelled) {
  Ok(fabric.Interrupted(Nil)) -> {
    let _ = fabric.cancel(handle)
    Error(Cancelled)
  }
  Ok(fabric.Reached(run.Finished(outcome))) -> Ok(outcome)
  Ok(fabric.Reached(_)) -> Error(NotFinished)
  Error(error) -> Error(Unreadable(error))
}

The selector is checked whenever the wait blocks, and a message already queued for it wins over a status read in the same moment only if it arrived first.

Cancels a run that is active, suspended, or whose runner was lost, and its sub-agent runs through the store. A stopped tool bound with tool.bind_settling is waited for, up to its bound, until its late settlement records what happened. A sub-agent’s cancellation that fails on a store error is tried again with a bounded backoff; one that still fails makes the delegation an uncertain effect, and the sub-agent then accepts no answer or reconciliation (see RunEnded). Cancelling a run that is still stopping asks its sub-agents to cancel again. A sub-agent run that was never stored is stored as cancelled before it started, so a start or a recovery racing the cancellation never runs it. Running tools are stopped and recorded as uncertain effects, never retried; queued actions and pending approvals are recorded as not started. Returns the status right after the cancellation was committed: Working while tools are being stopped or sub-agents cancelled, then Finished(Cancelled) (see await).

Through a Store that does not drive the run (another Store over the same backend), the cancellation is committed to the record at once and reported Finished(Cancelled), but the owner’s tool bodies keep running until its runner next tries to commit and stops; they are recorded as uncertain effects. A runner that does not take the cancellation within the command timeout (a synchronous handler holds it) has its work abandoned in the record the same way, and is then killed, with its model call and running tool bodies (recorded as uncertain effects): it calls no model, starts no tool body or sub-agent, and commits nothing after the cancellation. A handler that cancels the run whose runner it runs in cannot kill that runner; the runner calls no model and starts no sub-agent once its record moved on, and commits nothing more. A sub-agent is cancelled the same way; its delegation becomes an uncertain effect only when the store keeps failing.

Cancels the stored run id with no agent: for a run that cannot be recovered because its agent changed (another identity, or a pending tool that no longer exists). Cancelling a sub-agent run this way does not apply its end to its parent, which needs its agent to map it: recover the parent to apply it (await on the parent reports Unattended until then). A run whose runner is live in this store is cancelled through that runner, as cancel would, which must take it within 5 seconds (there is no agent to configure it); otherwise (a lost runner, or one a handler holds) the work of a lost runner is abandoned (running tools become uncertain effects) and the run ends Cancelled in one commit. Active sub-agent runs are cancelled first, the same way; their delegations are recorded as uncertain effects, since no agent maps their outcome. A sub-agent that is still stopping (it waits for a stopped tool’s settlement) is recorded as such, and ends on its own. A sub-agent run that was never stored is stored as cancelled before it started (naming no agent), and its delegation is recorded as not started.

Cancels the run when owner stops, for a run that must not outlive the process that asked for it: a request handler, a connection, a Relay tool call. A watcher process monitors owner and cancels the run as soon as owner exits, for any reason (also when it had already exited); it then stops. It also stops, cancelling nothing, once the run has finished. A suspended run is checked for its end every 5 seconds.

The watcher is not linked to the caller and is not supervised: a VM that stops takes it with it, and the run is then recovered like any other. Call it once per owner; each call adds one watcher.

Opens the sub-agent run id, a descendant of run, with run’s agent and context: its snapshot, reconciling its uncertain effects, answering or cancelling it. Its end still reaches its parent. Its answer reads as the text it stored (its delegation reads it with the child’s own answer codec).

One line for logs.

Messages appended by this run, in exact stored order, including tool calls, results, answer corrections and opaque provider replay data.

Whether this run’s complete original input equals input. Generated messages, live context and retry correlation do not participate.

Opens the stored run id under agent and context, for a caller that did not start it: a request handler reading, awaiting, answering, reconciling or cancelling a run. It only reads the record and checks that the run can continue under agent (IncompatibleAgent otherwise); it never takes the run over and starts nothing, so a live runner, in this store or behind another Store, keeps its work.

Commands through the handle behave exactly as through the handle start returned: each is checked against the stored record, one that produces work with no runner live in this store starts a runner, and one that needs the live runner of another Store is RunUnattended. A sub-agent run is reached through its root’s handle (child).

recover takes over work in flight. Leased stores permit it at any time; on an unleased store the previous owner must be known to be gone.

The approval requests waiting for an answer: the run’s own, oldest first, then its sub-agents’. They can be answered while other work still runs.

Records what actually happened for an uncertain effect of the run or of one of its sub-agents (UncertainAction.reference names both the run and the action; it is routed through the family like approve). content is what the model will see as that call’s result. When nothing else is pending the run continues with its next model turn. Reconciliation does not consume a turn.

An effect of a run outside this handle’s family, or an action that the run’s current tool batch does not have, is refused with WrongReference; an action that is not uncertain, or a run whose model is being called, with NotReconcilable. A sub-agent run whose ancestor is stopping or has ended accepts none (RunEnded). After a run has finished, use reconcile_stored to retain evidence without resuming work, then settle_stored for its finished ancestors.

Records evidence for an uncertain tool of a finished run, without an agent definition or any execution callbacks. The outcome, transcript, usage and counters stay unchanged. The same content is an idempotent acknowledgement; different content cannot overwrite a saved result. Active runs return RunNotFinished; use reconcile to continue them. A delegation cannot be reconciled by supplying content: settle its child and then use settle_stored on the parent. Returns this run’s updated snapshot, including any remaining uncertain actions.

Opens the stored run id under agent and context. When work was in flight and no runner in this store drives it, recovery takes the work over as a new incarnation (committed with compare-and-set, so concurrent recoveries have one winner): running tools become uncertain effects (replayable ones, tool.with_replay, are started again while they have attempts left), queued tools are dispatched again, and a lost model call is issued again against the turn budget. A suspended or finished run is opened unchanged, except that an approval request whose deadline passed is rejected (the model sees it, and the run goes on). Calling it again is harmless.

An approved tool or sub-agent start that had not started is not run with context: the context its answer was checked with is gone, and a stored approval alone does not authorize a later incarnation. It asks for its approval again, as a new request under the requirement last answered (the old reference is then stale).

Sub-agent runs are recovered with their parent: a child that was never stored is started, a child whose end the parent missed is applied to it, an active child is recovered in turn, and a child that cannot be read or continued becomes an uncertain effect of the delegation.

A run handed off by a drained shutdown (store.supervised) has nothing running: its results are committed, a model call it never issued is issued as the same turn, and its queued tools are dispatched, except approved ones, which ask for their approval again (the handoff already asked).

On a leased store (store.leased) it is safe to call at any time: a run whose lease another node holds live is left exactly as it is (the handle is returned, and the run reads Working), and only a free or expired lease, one of an earlier process of this store, or this store’s own lease whose runner is gone is taken over. This node reads that last case as Unattended; another node reads it as Working until expiry. Of several nodes recovering one run at once, exactly one takes it over. A live foreign parent keeps its runner; eligible children can recover independently, and that parent reads their committed outcomes.

An unleased store knows only the runners it started. Recovering through another unleased Store (for example in another VM) while the run’s runner is still alive takes the run over: the older runner can no longer commit and stops, and its running tools become uncertain effects. On an unleased store, recover only when the previous owner is known to be gone (for example at boot); to read or command a run someone else may be driving, open it.

Rejects an approval request of the run or of one of its sub-agents (the reference names the run): the action does not run, and the model sees reason. A rejection is not checked again, so it takes no context and never runs the policy; the run continues with its own context. It is committed like an approval (see approve), with the same refusals except RequirementChanged; a rejection after the request’s deadline is ApprovalExpired. proof is checked and recorded as for approve.

Settles uncertain delegations of a finished agent family from their saved child outcomes. Descendants must name the exact parent action. Missing or unreadable children return an error; active or uncertain children leave the delegation uncertain. No agent code runs and no work resumes. The returned root snapshot retains every unresolved action; successful observation does not imply that every effect has been settled.

Each record commits independently, from the leaves outward. A failed parent write may follow successful descendant writes; repeat this command to finish propagation. Repeating an unchanged walk performs no writes. For a graph-owned family, settle this agent root, then recover its graph parent to observe the saved outcome. The graph remains cancelled.

The run’s own record, with the status of the run and its sub-agents (see await).

Starts a run of agent in store under id: stores its first record and hands the first model call to a new runner. id is the caller’s: run.new_id() for a fresh run, or an id derived from the work that starts it (run.id_from_parts("job", [job_id])), so that starting again finds the run (AlreadyStarted) instead of starting a second one. A retry that must start afresh rather than continue (a job’s next attempt after a business failure) includes its attempt among the parts. An agent with a family budget (agent.with_family_budget) declares it for this root run.

correlation is carried in every event of the run, its sub-agent runs included, in every model.Request (which fabric/llm puts on its HTTP requests) and in every tool’s tool.Call; it is stored with the run, so a recovered run keeps it. None derives it from the id (correlation.from_key(run.id_to_string(id))); pass the correlation of the request or job that starts the run to join their events.

Starts one run from validated history and a current prompt. Historical calls are context only. The writer must retain the original boundary; unsupported history is refused before any write or execution.