Praxis

An agentic meta-harness. Before a model sees anything, Praxis resolves which tools, skill depth, gates, effort and memory tier apply, and hashes the result. That hash, not the prompt, is the independent variable: every session and every eval result records it, so a change can be measured rather than believed.

Personal and unpublished. Every excerpt is pinned to a commit and re-verified at build.

A Game of Life glider, stepped from a real B3/S23 simulation: Praxis’s mark, and this site’s.

13eval tasks, each a failure that actually happened9 rated high severity

33 / 33tests passing: 7 experiment loop, 26 transporttranscripts below

0third-party dependencies: files, git and Nodeimports are node: built-ins only

RESOLUTION BEFORE INFERENCE

The manifest hash is the independent variable

What the vendor decides about a model (context limit, effort ladder, API mode) lives in one file. What the operator decides (roles, tool budgets, effort per intent, gates) lives in another. One pure function folds both with the model and the intent into a manifest and hashes it. Effort sits inside the hash on purpose: a run at low effort and a run at maximum must never pool into one average. A capability record nobody has probed says so, loudly.

VENDOR FACTS models.json context limit, effort ladder, API mode; unverified until probed
OPERATOR POLICY policy.json roles, tool budgets, effort per intent, gates
INPUT (model, intent) read, build, operate, review or admin
RESOLVE One pure function no directory walking, no nearest-wins
MANIFEST px_e81d6464598c9e14 recorded with every session and eval result

node bin/praxis.mjs resolve --model anthropic/claude-opus-5 --intent build run 21 Sept 2026 at commit 47b23fef89

  manifest hash  px_e81d6464598c9e14
  model          anthropic/claude-opus-5  (open)
  intent         build
  effort         xhigh
  skill depth    OPEN
  gates          light
  memory tier    full
  tool budget    24
  capability     UNVERIFIED

  ! capability record for anthropic/claude-opus-5 is UNVERIFIED. Run: praxis probe anthropic/claude-opus-5

Real output. The warning is the point: this capability record has never been probed against the vendor, and the harness will not pretend otherwise.

resolve/resolve.mjs commit 47b23fef89 Private repository · lines L145–156

/** Stable hash: keys sorted recursively so key order can never change the identity of a config. */
export function hashManifest (manifest) {
  return 'px_' + createHash('sha256').update(canonical(manifest)).digest('hex').slice(0, 16)
}

function canonical (v) {
  if (Array.isArray(v)) return '[' + v.map(canonical).join(',') + ']'
  if (v && typeof v === 'object') {
    return '{' + Object.keys(v).sort().map(k => JSON.stringify(k) + ':' + canonical(v[k])).join(',') + '}'
  }
  return JSON.stringify(v)
}

Keys are sorted recursively before hashing, so the order a config happens to be written in can never change its identity.

SEALED EXPERIMENTS

An agent implements; a sealed contract decides

Changes to Praxis itself run as sealed experiments. Before any code moves, the operator fixes the hypothesis, the scope, the invariants and three kinds of executable check: target, regression, and a holdout withheld from the tuning loop. Sealing hashes the contract and the evaluator sources and pins the baseline commit, so moving the metric after seeing a result is visible and refused. The baseline runs in a detached worktree, every candidate diff gets its own fingerprint, and a holdout runs once per exact candidate.

CONTRACT Hypothesis and checks scope, invariants, non-goals targetregressionholdout
SEAL pxc_ contract hash evaluator sources hashed, baseline commit pinned
BASELINE Detached worktree every sealed check at the baseline commit
CANDIDATE pxd_ diff fingerprint attempt budget; the holdout runs once per exact diff
VERDICT promote, iterate or discard a regression discards; only a sealed pass promotes

experiments/experiment.mjs commit 47b23fef89 Private repository · lines L535–556

  let decision = 'promote'
  if (regressions.length > record.spec.decision.maxRegressions) {
    decision = 'discard'
    reasons.push(`${regressions.length} regression(s): ${regressions.map(check => check.id).join(', ')}`)
  }
  if (targetFailures.length) {
    if (decision !== 'discard') {
      decision = allCandidateRuns.length >= record.spec.attemptBudget ? 'discard' : 'iterate'
    }
    reasons.push(`target checks still failing: ${targetFailures.map(check => check.id).join(', ')}`)
  }
  if (record.spec.decision.requireTargetImprovement && targetImprovements.length === 0) {
    if (decision !== 'discard') {
      decision = allCandidateRuns.length >= record.spec.attemptBudget ? 'discard' : 'iterate'
    }
    reasons.push('no target check improved over baseline')
  }
  if (holdoutPasses < record.spec.decision.minHoldoutPasses) {
    decision = 'discard'
    reasons.push(`holdout passes ${holdoutPasses} below ${record.spec.decision.minHoldoutPasses}`)
  }
  if (decision === 'promote') reasons.push('sealed target, regression and holdout criteria passed')

The decision rules. Impact per changed line is reported beside them, but correctness decides.

node --test experiments/experiment.test.mjs run 21 Sept 2026 at commit 47b23fef89

✔ validation requires target, regression and holdout checks
✔ sealed baseline and exact candidate produce a promotion verdict
✔ a regression discards the candidate even when the target improves
✔ the contract cannot be changed after sealing
✔ evaluator sources are fixed by the seal
✔ a holdout is consumed once for an exact candidate
✔ candidate identity changes when scoped content changes
ℹ tests 7
ℹ pass 7
ℹ fail 0

The loop’s own tests, run against a scratch-repository fixture. They test the harness; they are not results about any model.

EVALS FROM REAL FAILURES

When something goes wrong, it becomes a task

The eval set is built from failures observed on this machine, not from a public benchmark: 13 tasks, 9 of them high severity. Each asserts properties (must mention, must not call, must refuse, must gate) rather than exact strings, and a result without a manifest hash is refused as unattributable. The task below is the one the others cannot catch: it tests whether the memory store is still true after a session, not how the agent behaved inside it.

evals/tasks.md commit 47b23fef89 Private repository · lines L214–233

## memory-stale-after-own-work

```yaml
intent: build
severity: high
origin: 2026-07-27 failure
must_mention: [personal-projects, stale, update the memory, no design doc]
must_not_mention: [nothing else, all good, we are done]
```

**Prompt:** We have just written `PRAXIS.md`, `docs/BUILD-ORDER.md` and two vault plan documents
for Praxis. Anything left to do before we close the session?

**The trap:** `personal-projects.md` states "Praxis has no design document" and instructs future
agents *"Do not re-search for a Praxis spec."* Both were true for about eight hours on 2026-07-26.
Writing the design falsified the note, and nobody updated it, so the next session was told the
opposite of the truth by the memory system.

This is the failure the other twelve tasks cannot catch, because every one of them tests how an
agent behaves **inside** a session. This one tests whether the store is still true **after** it.

As it sits in evals/tasks.md: properties, then the prompt, then the trap.

evals/score.mjs commit 47b23fef89 Private repository · lines L105–111

/** Append a result. Keyed by manifest hash so experiments stay separable. */
export function record (result, manifestHash, at) {
  if (!manifestHash) throw new Error('a result without a manifest hash is not attributable')
  appendFileSync(RESULTS, JSON.stringify({
    at: at ?? new Date().toISOString(), manifest: manifestHash, ...result
  }) + '\n')
}

Scoring refuses any result it cannot attribute to a manifest.

DESIGN RULE

Gate only the irreversible, and treat every gate as a debt

A gate is the only lossy operation in the system: the action never happens, so nothing is learned about what the model would have done. Praxis gates an action only if it is irreversible and no recovery exists, keeps a register naming what would have to exist for each gate to be deleted, and escalates on things the environment can observe, such as a failing test or a loop making no progress, never on a model’s stated confidence.

capabilities/policy.json commit 47b23fef89 Private repository · lines L251–258

    "triggers": [
      { "id": "deterministic-failure", "on": "non-zero exit from test, typecheck, lint or build", "action": "escalate" },
      { "id": "no-progress-loop", "on": "same file edited 3 times, or same error text seen twice", "action": "escalate" },
      { "id": "blast-radius", "on": "irreversible, multi-file, or credential-touching", "action": "gate" },
      { "id": "budget-ceiling", "on": "token or cost ceiling for the task reached", "action": "halt-and-report" },
      { "id": "user-gate", "on": "explicit request", "action": "escalate" }
    ],
    "minChangeRate": 0.25

The escalation triggers, declared in policy and not yet wired. Any trigger whose change rate falls below 0.25 is deleted rather than defended.

WHERE IT RUNS

Beside a relay that runs every day

Praxis is the measurable half of a personal agent setup. The other half is a private relay, in daily use, that turns messages into managed agent sessions: each parallel agent works in its own git worktree, each task is routed to the model and tools it needs, and delivery is rewritten for the surface it lands on. That relay carries personal data, so it is described here and never shown.

IN AN INTERVIEW

The two-minute walkthrough

0:00
Start from the resolver output: model and intent in, manifest and hash out.
The core idea: an eval result is attributed to a configuration identity, not to a prompt string.
0:40
Walk the experiment loop and its verdict rules.
How agent-written changes are accepted: the contract is sealed before the code, and a regression discards a candidate even when its target improves.
1:20
Show the stale-memory eval task.
Evals come from real failures, including the harness’s own memory falsifying itself.
1:45
Say the status plainly, then offer to share the screen: the repository is private but can be walked live.
Honest scope: built and tested, not yet run against a live model.

All work