Skip to content

Traces

The Ops Feed pulls trace records from system.observability.tracing (canonical) or system.tracing (compat). This page documents the trace path; Logs covers logging.

Trace kinds

IDirectorFeedKind includes five tracing kinds plus log and activity:

'rpc' | 'log' | 'tool-call' | 'message' | 'inference' | 'activity' | 'all'

When the user selects kind === 'all', queryOpsFeed requests the five tracing kinds (rpc, log, tool-call, message, inference) and merges them with the log path. When a specific kind is chosen (other than log or activity), only the trace path runs.

ts
{
  limit:  number,                                           // backendLimit
  kind:   string[],                                         // e.g. ['rpc'] or all five
  source?: string,                                          // selected mount
  text?:   string,                                          // search text (search variant)
}

tracing.query is used when there is no text filter; tracing.search is used when one is present. Both come from serviceMounts(DirectorServiceTargets.observability.tracing) with canonical-first fallback.

Activity feed special case

When kind === 'activity', _queryActivityFeed runs a dedicated path:

  1. If a selected mount is set, ask system.observability.tracing trace.activity { mount, windowMs, limit }. The response goes through _mapActivityFeedRecords which extracts user/assistant/tool spans into a more conversational summary.
  2. If no mount is selected (or the activity probe returned nothing), fall back to the broad five-kind query and run _summarizeActivityRecords on the result.

This path is what powers the "what happened with this actor" view that an operator wants when watching a session unfold.

Detail fan-out (loadOpsDetail)

Selecting any trace row dispatches director:ops-record-selected { record }. DirectorApp then asks DirectorRuntimeAdapter.loadOpsDetail(context, root, record), which builds candidate calls in this order:

  1. trace.request { requestId, limit: 80 } — sibling records for the same request cycle. Accepted only if it returns at least one record.
  2. trace.thread { sessionId, limit: 80 } — full conversation thread.
  3. trace.get { traceId } — exactly the trace records.
  4. trace.activity { mount: source, windowMs: 15min, limit: 50 } — fallback activity summary.

invokeFirst returns the first candidate that succeeds and (for trace.request) returns non-empty. If all fail, a fallback "record metadata only" detail is rendered.

Window semantics

_windowMs and _windowStart:

'15m' = 15 * 60 * 1000
'1h'  = 60 * 60 * 1000
'24h' = 24 * 60 * 60 * 1000
'7d'  = 7  * 24 * 60 * 60 * 1000
'all' = no window filter

_windowStart('all') returns null, and the client-side window filter becomes a no-op.

Dedup and sort

Across the merged log + trace stream:

  • Each record gets a _feedRecordToken from a stable subset (requestId|spanId|traceId|sessionId|ts|op|kind).
  • Duplicates collapse to the first occurrence.
  • Final list is sorted by ts descending and sliced to the requested limit.

Operator notes

  • Activity vs all. "All" gives you raw records (every RPC, every log line); "Activity" gives you a session-conversational summary. Use Activity when investigating a user-driven session; use All for low-level RPC/log forensics.
  • Empty feed but actors are healthy. Confirm system.observability.tracing is mounted (matrix invoke system.observability.tracing $introspect). The compat alias system.tracing is honored, but only one of the two needs to exist.
  • Detail pane empty. Possible if the record lacks a requestId/sessionId/traceId and the actor has no recent activity rows. The fallback metadata view will still display the record itself.

See also

Source: projects/matrix-3/packages/director/src/services/DirectorRuntimeAdapter.ts:549-757, 1830-1977 (queryOpsFeed, loadOpsDetail, mappers).