fabric/telemetry

Fabric’s Sinal events: what a run did, for logs, metrics, and traces.

Telemetry is diagnostic, never a source of truth: the run’s record is. Every event is emitted after the commit of the transition it describes, by the process that made the commit (a runner, or the caller of start, approve, reject, cancel, reconcile, recover, or cancel_stored), and never by the pure controller. The lease events of a leased store (lease_lost, renewal_failed) describe no commit: a process its store’s process starts emits each, after the kill or the failed renewal, so that no handler holds up the store. The sweep event is emitted after a scan in a separate process with a 1-second deadline; a blocked synchronous handler is stopped, so scans cannot accumulate blocked emitters. Shutdown emits drain or drain_unavailable with the same bound before the store closes. Every event uses sinal.emit, which follows forwarder routes, so the application chooses where handlers run:

• By default, synchronously in the committing process. A handler that blocks holds up that run’s progress.
• If the application routes [fabric] (or a longer prefix) through a forwarder with forwarder.route, in the forwarder’s process. A blocked handler then stalls the forwarder, never the run, and a handler cannot read the committing process’s dictionary. Route at application start, once the forwarder is supervised, and unroute at shutdown.

A handler that fails (returns an error or raises) is detached by Sinal and telemetry and never affects the run.

A command (approve, reject, cancel, reconcile, cancel_stored) that a live runner applies is answered once its commit is stored, before that commit’s events are emitted. A command applied with no runner (to a suspended run, or a cancellation that takes over a lost or held run) is committed by its caller, which emits the commit’s events itself before the command returns: a synchronous handler then holds up the caller, while a runner the commit started is already working. A synchronous handler running in a runner holds that runner: a command it sends to the run it observes is refused at once with fabric.RunnerBusy, and a command from elsewhere that the runner does not take within the agent’s command timeout is refused the same way and never applied later. A cancellation is the exception: it is committed to the record, abandoning the held runner’s work. A runner held by a handler of another process is then killed, with its model call and tool bodies; a runner whose own handler cancelled its run calls no model and starts no child afterwards, and stops at its next commit. Handlers that call Fabric should run in a forwarder.

An event may be missing: a process that dies between its commit and its emit, or a commit made through a store in another VM, emits nothing here. A routed event is also dropped when its forwarder is full or not running (both counted and reported through the forwarder’s dropped_event, those made while it is down by its next incarnation), and in-flight events are lost if the forwarder stops. Events of one commit are emitted in the order listed below, and one process’s events keep their order through one route; events of different runs, of one run through several processes, or split across a route change have no global order.

| | run_finished | [fabric, run, stop] | the run ended | | drain | [fabric, drain, stop] | the factory stopped; runner exit and handoff evidence were collected within the accounting deadline | | drain_unavailable | [fabric, drain, unavailable] | shutdown accounting could not be read; no successful empty summary is inferred |

A graph run (fabric/graph) emits its own events, after the commit they describe, by the process that made it (its runner, or the caller of graph.start, approve, reject, deliver, reconcile, poll_job, recover or cancel). Events of one commit are emitted in this order:

A late settlement (tool.bind_settling) of a stopped action is observed as that action’s tool_settled; one that resolves an uncertain effect emits nothing, like a reconciliation. A settlement the run refused commits nothing: settlement_refused is emitted by the process that offered it, once it has its answer, with why it was refused. NotAwaited and NotReached mean what the settlement knew is not in the record and needs a person.

Every event of a run (all but renewal_failed, sweep, drain and drain_unavailable) carries the run’s correlation, under the correlation key that every gleam-dream package uses: the one given to fabric.start, or one derived from the run id. It also carries root, the id of the family’s root run: the run itself for a root, its root’s for a sub-agent run, so a sub-agent’s events join its root’s. A sub-agent run’s events carry its root’s correlation too. The same correlation reaches the model’s requests and the tools’ calls, so their own events join the run’s.

A sub-agent or graph child record written before roots were stored names only its parent. A reader of the store derives its root from the stored ancestors, following parent links (at most 64, without repeats) up to a root run or to the first ancestor that stores its root; the run’s next commit stores the derived root, so the walk is paid once. When an ancestor’s record is missing or unreadable, or the chain repeats or is too long, the root is the topmost ancestor that could be read (the parent when none could): the reader emits root_inferred, the run’s events carry that root, and its record stores none, so a later reader derives it again. An unavailable store fails the read instead: no root is inferred from it. root_inferred describes no commit; the reading process emits it each time it reads such a record.

A graph run’s events, its lease_lost included, carry the correlation given to graph.start (or derived from its id) and its family’s root; a graph’s child runs, managed agents included, inherit both.

Metadata carries identifiers and closed kinds only, never arguments, tool results, or model text, with one exception: settlement_refused carries the refused settlement’s summary as the tool gave it to tool.settle, so that a person can reconcile the action. A tool’s summary must therefore not carry secrets; saga/outcome.summary names outcome kinds and step addresses only. turn and call_id identify an action only within its run.

pub type Action {
  Action(run: String, turn: Int, call_id: String, tool: String)
}

pub type ActivationSettled {
  ActivationSettled(
    activation: GraphActivation,
    route: Route,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ActivationStarted {
  ActivationStarted(
    activation: GraphActivation,
    kind: policy.OperationKind,
    deadline_after: option.Option(Int),
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type Answered {
  Approved
  Rejected
  Expired
}

pub type ApprovalAnswered {
  ApprovalAnswered(
    action: Action,
    revision: Int,
    answer: Answered,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ApprovalRequested {
  ApprovalRequested(
    action: Action,
    requirement: String,
    requirement_version: Int,
    revision: Int,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ChildSettled {
  ChildSettled(
    action: Action,
    child: String,
    disposition: Disposition,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ChildStarted {
  ChildStarted(
    action: Action,
    child: String,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type Disposition {
  ModelVisible
  EffectUncertain
  HostFailure
  Withdrawn
}

pub type Drain {
  Drain(
    runners: Int,
    handed_off: Int,
    failed_handoffs: Int,
    pending_handoffs: Int,
    killed: Int,
    exited: Int,
    unobserved: Int,
    elapsed_ms: Int,
  )
}

pub type GraphActivation {
  GraphActivation(
    run: String,
    activation: Int,
    attempt: Int,
    node: String,
    operation: String,
    operation_version: Int,
  )
}

pub type GraphApprovalAnswered {
  GraphApprovalAnswered(
    activation: GraphActivation,
    revision: Int,
    answer: Answered,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type GraphApprovalRequested {
  GraphApprovalRequested(
    activation: GraphActivation,
    requirement: String,
    requirement_version: Int,
    revision: Int,
    expires: option.Option(Int),
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type GraphOutcome {
  GraphCompleted
  GraphFailed
  GraphExhausted
  GraphCancelled
  GraphExpired
}

pub type GraphRunCancelled {
  GraphRunCancelled(
    run: String,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type GraphRunFinished {
  GraphRunFinished(
    run: String,
    outcome: GraphOutcome,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type GraphRunStarted {
  GraphRunStarted(
    run: String,
    graph: String,
    graph_version: Int,
    parent: option.Option(String),
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type LeaseLoss {
  Revoked
  Unrenewed
}

pub type LeaseLost {
  LeaseLost(
    run: String,
    owner: String,
    reason: LeaseLoss,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ModelTurn {
  ModelTurn(
    run: String,
    turn: Int,
    result: TurnResult,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type OutcomeKind {
  Completed
  AnswerInvalid
  Refused
  OutputLimited
  BudgetExhausted
  BudgetUnverifiable
  Cancelled
  Failed
}

pub type RenewalFailed {
  RenewalFailed(owner: String, runs: Int)
}

pub type RootInferred {
  RootInferred(
    run: String,
    root: String,
    ancestor: String,
    problem: RootProblem,
    correlation: correlation.Correlation,
  )
}

pub type RootProblem {
  AncestorMissing
  AncestorUnreadable
  AncestryCycle
  AncestryTooLong
}

pub type Route {
  NextNode
  Answer
  Kept
}

pub type RunCancelled {
  RunCancelled(
    run: String,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunFinished {
  RunFinished(
    run: String,
    outcome: OutcomeKind,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunHandedOff {
  RunHandedOff(
    run: String,
    incarnation: Int,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunRecovered {
  RunRecovered(
    run: String,
    incarnation: Int,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunStarted {
  RunStarted(
    run: String,
    agent: String,
    agent_version: Int,
    parent: option.Option(String),
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunTakenOver {
  RunTakenOver(
    run: String,
    incarnation: Int,
    previous_owner: String,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type RunTotals {
  RunTotals(
    turns: Int,
    input_tokens: Int,
    output_tokens: Int,
    unreported_replies: Int,
  )
}

pub type SettlementRefusal {
  AlreadyRecorded
  NotAwaited
  NotReached
}

pub type SettlementRefused {
  SettlementRefused(
    action: Action,
    offered: Disposition,
    reason: SettlementRefusal,
    summary: String,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type Sweep {
  Sweep(
    claimed: Int,
    recovered: Int,
    unmatched: Int,
    failed: Int,
  )
}

pub type ToolDispatched {
  ToolDispatched(
    action: Action,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type ToolSettled {
  ToolSettled(
    action: Action,
    disposition: Disposition,
    root: String,
    correlation: correlation.Correlation,
  )
}

pub type TurnResult {
  ToolRequest
  FinalAnswer
  AnswerRejected
  Refusal
  Truncated
  Retry
  ModelFailure
  ProtocolViolation
  BudgetStop
}

pub fn activation_settled() -> sinal.Event(Nil, ActivationSettled)

pub fn activation_started() -> sinal.Event(Nil, ActivationStarted)

pub fn approval_answered() -> sinal.Event(Nil, ApprovalAnswered)

pub fn approval_requested() -> sinal.Event(Nil, ApprovalRequested)

pub fn child_settled() -> sinal.Event(Nil, ChildSettled)

pub fn child_started() -> sinal.Event(Nil, ChildStarted)

pub fn drain() -> sinal.Event(Drain, String)

pub fn drain_unavailable() -> sinal.Event(Nil, String)

pub fn graph_approval_answered() -> sinal.Event(
  Nil,
  GraphApprovalAnswered,
)

pub fn graph_approval_requested() -> sinal.Event(
  Nil,
  GraphApprovalRequested,
)

pub fn graph_cancelled() -> sinal.Event(Nil, GraphRunCancelled)

pub fn graph_finished() -> sinal.Event(Nil, GraphRunFinished)

pub fn graph_started() -> sinal.Event(Nil, GraphRunStarted)

pub fn lease_lost() -> sinal.Event(Nil, LeaseLost)

pub fn model_turn() -> sinal.Event(
  option.Option(model.Usage),
  ModelTurn,
)

pub fn renewal_failed() -> sinal.Event(Nil, RenewalFailed)

pub fn root_inferred() -> sinal.Event(Nil, RootInferred)

pub fn run_cancelled() -> sinal.Event(Nil, RunCancelled)

pub fn run_finished() -> sinal.Event(RunTotals, RunFinished)

pub fn run_handed_off() -> sinal.Event(Nil, RunHandedOff)

pub fn run_recovered() -> sinal.Event(Nil, RunRecovered)

pub fn run_started() -> sinal.Event(Nil, RunStarted)

pub fn run_taken_over() -> sinal.Event(Nil, RunTakenOver)

pub fn settlement_refused() -> sinal.Event(Nil, SettlementRefused)

pub fn sweep() -> sinal.Event(Sweep, Nil)

pub fn tool_dispatched() -> sinal.Event(Nil, ToolDispatched)

pub fn tool_settled() -> sinal.Event(Nil, ToolSettled)