Evaluation harness

The evaluation harness is the deterministic side of the harness pair: a propose session writes a candidate, an evaluate session runs checks, and the harness records the verdict.

The `evaluate` profile

From BUILT_IN_AGENCY_PROFILES (profiles/AgencyProfile.ts:90-98):

{
  key: 'evaluate',
  title: 'Evaluate',
  description: 'Deterministic validation or scoring of a target',
  prompt: { purpose: 'Evaluate the target against explicit checks' },
  context: { memoryScope: 'session', includeSnapshot: false, includeChildren: 'none' },
  tools: { toolContextRefs: ['test-runner'], allowMatrixInvoke: false },
  runtime: { preferred: 'deterministic' },
  supersedePolicy: 'none',
}

Note runtime.preferred: 'deterministic' and supersedePolicy: 'none' — multiple evaluations can coexist; they are not deduplicated like propose sessions are.

The `test-runner` tool context

test-runner (tools/ToolContextSpec.ts:96-109) ships exactly one tool: trace_query (against system.agents.$traceQuery). The intention is "the evaluator can read traces of the candidate's run, but cannot mutate state." The list will grow once verificationChecks (declared on skills like repo-coding) become invokable as tools — that wiring is not in code today.

How a harness uses evaluation

A coding harness pattern:

import { agencyInvoke, recordHarnessIteration } from '@open-matrix/agents';

// Spawn a propose session
const proposed = await agencyInvoke({
  targetRef,
  profileKey: 'propose',
  message: 'Add a null check around the parse step',
});

// Wait for the propose session to finish (via activity frames)
// ...

// Spawn an evaluate session against the same target
const evaluated = await agencyInvoke({
  targetRef,
  profileKey: 'evaluate',
  message: 'Run the focused tests for the touched file',
});

// Read the result, record the iteration
const score = parseEvalScore(evaluated.finalText);
const iteration = {
  iteration: 1,
  promptHash: hashPrompt(proposed.input),
  agentSessionId: proposed.sessionId,
  profileKey: 'propose',
  score,
  startedAt: proposed.startedAt,
  completedAt: evaluated.completedAt,
};
run = recordHarnessIteration(run, iteration);

This is the pattern the Smithers coding agent implements; it's also the shape a future generic harness would consume.

What "deterministic" means

The runtime.preferred: 'deterministic' hint asks the runtime adapter to favor a deterministic runtime (no LLM nondeterminism — e.g. a pure-test-execution adapter). Today there is no such adapter. The hint is honored only insofar as the underlying tool (trace_query, eventually focused-test runners) is deterministic; the LLM step remains stochastic with temperature: 0 advice baked into the system prompt.

A clean deterministic adapter is target state — when it lands, it would short-circuit inference for evaluations whose check set is fully expressible without natural language.

What's NOT wired

A library of named checks (focused-test, package-build, typecheck, etc.) callable as tools. They appear in SkillDescriptor.contributes.verificationChecks but there is no executor.
A score schema. HarnessIteration.score is a free-form number; what 0.7 vs 0.9 means is convention, not protocol.
Constraint result aggregation. HarnessIteration.constraintResults is Record<string, boolean> — the harness provides the dictionary, the application interprets it.

Evaluation harness ​

The evaluate profile ​

The test-runner tool context ​

How a harness uses evaluation ​

What "deterministic" means ​

What's NOT wired ​