Add evaluation suites and deterministic regression replay #44

New issue

Open

opened 2026-10-24 23:24:33 +00:00 by TheComputerGuy · 0 comments

TheComputerGuy commented

2026-10-24 23:24:33 +00:00

Owner

Problem

Maintainers need to measure whether a model or configuration change improves task completion without weakening policy or silently increasing cost. Variable provider responses make raw reruns insufficient for regression diagnosis.

Proposed outcome

Define an eval format containing a repository fixture, task, policy, expected tests, forbidden effects, budget, and scoring rules. Support provider-backed runs and deterministic replay from captured provider/tool fixtures, with comparable reports across models and Chock revisions.

Acceptance criteria

Evals can assert repository state, tests, refusals, network destinations, cost, and turns.
Captures are integrity-checked and contain no unredacted secrets.
Reports distinguish model, harness, sandbox, and fixture failures.
CI can compare a candidate against a baseline and enforce thresholds.
Multi-agent and microVM cases are supported.

### Problem Maintainers need to measure whether a model or configuration change improves task completion without weakening policy or silently increasing cost. Variable provider responses make raw reruns insufficient for regression diagnosis. ### Proposed outcome Define an eval format containing a repository fixture, task, policy, expected tests, forbidden effects, budget, and scoring rules. Support provider-backed runs and deterministic replay from captured provider/tool fixtures, with comparable reports across models and Chock revisions. ### Acceptance criteria - Evals can assert repository state, tests, refusals, network destinations, cost, and turns. - Captures are integrity-checked and contain no unredacted secrets. - Reports distinguish model, harness, sandbox, and fixture failures. - CI can compare a candidate against a baseline and enforce thresholds. - Multi-agent and microVM cases are supported.

TheComputerGuy added the

enhancement

label

2026-10-25 11:59:04 +00:00