Track AI evaluation evidence and serving versions

The reviewer's brief

End product: a runnable evaluation service and release registry. It invokes the candidate on labeled cases, saves per-case outcomes, checks category coverage and severe misses, publishes an immutable report, and conditionally promotes or rolls back the active release pointer. The project does not silently redeploy an inference endpoint. The registry records which evaluated release consumers should use.

Define what a passing test means

The application is a three-label classifier: billing, technical, or escalate. For this bounded task, exact label comparison is the oracle. Human-authored fixture labels are independent of the candidate's response. A language-model judge would add uncertainty without helping this exact-match contract.

The demo gate requires at least six cases, all three labels, at least one severe case, and every case passing. Each run accepts at most eight cases to keep synchronous inference bounded. These numbers teach gate behavior. Six fixtures cannot establish production accuracy or statistically reliable rare-failure rates.

Understand the evidence chain

A dataset hash identifies the exact labeled cases used. A candidate identifier hashes the checked runtime manifest: source bytes, exact classifier prompt, model identifier, inference configuration, retrieval marker and policy. AI_SOURCE_COMMIT can add the build commit. Source-byte identity is checked even when that optional field is absent. A report preserves per-case evidence, including severe failures. A release pointer is a small mutable record identifying the approved report. Keep evidence immutable while allowing the pointer to change under concurrency control.

Draw the AWS architecture

Implement the path from cases to rollback

1. Validate the dataset. Require distinct case IDs, bounded text, recognized expected labels, and explicit boolean severity. Reject malformed evidence before spending model calls.
2. Identify the complete candidate. candidate.py hashes the shipped implementation files and records prompt/configuration identity. The local model is named fixture-v1. The Bedrock adapter records its configured model ID. The evaluator checks identity again after the run so a changed candidate cannot silently share one report.
3. Run each case. The model receives the case text, not its expected label or severity. Malformed outputs and provider errors count as failures.
4. Score by case. Preserve expected label, actual label, pass/fail and severity. Compute passed count, severe misses and category coverage separately.
5. Register the report. Save a content-addressed artifact and conditionally create an immutable run ID. Reusing a run ID conflicts rather than rewriting history.
6. Promote deliberately. Read the release revision. Only an eligible report can become active, and only if the revision still matches. Preserve the previous distinct active report. Re-promoting the active report with a current revision is a no-op. A stale revision still conflicts.
7. Roll back explicitly. Swap active and previous under the same revision rule. The old report and its dataset identity remain available. classifier.predict only serves when the running candidate matches that active report. Pointer rollback does not install old code or reconfigure Bedrock.

eligible = (
    enough_cases
    and every_case_passed
    and severe_misses == 0
    and required_categories_present
)

This is a decision rule, not a universal AI metric. The reference exposes it as demo-gate-v1. A product owner must justify thresholds, sample coverage and severity rules for the real application.

Run a full release cycle

python3.12 examples/ai-systems/demo.py evaluation

The session evaluates release-1, promotes it, evaluates release-2, promotes it, and rolls back. The final result is active: release-1, previous: release-2, revision: 3. The fixture always classifies its six known examples correctly. Tests separately inject a broken classifier that returns billing for everything and verify severe misses prevent promotion.

Run cloud_smoke.py evaluation --function "$AI_FUNCTION" from the workbench directory to exercise AWS persistence. With a real model, a failed gate should stop promotion. Investigate the failures. Do not change the labels simply to make a deployment pass.

Follow-up: model version, prompt version, and live traffic diverge

A passing report is useful only if it identifies the thing actually being served. The supplied classifier.predict consumer now resolves the active report and compares its candidate manifest with the running implementation. It also rechecks release revision and candidate identity after inference. A changed prompt under the same model name fails that check. Dataset and policy remain part of the immutable evaluation evidence. A model alias alone is not a reproducibility guarantee.

release-1 → promote → revision 1
release-2 → promote → revision 2, previous release-1
release-2 → promote again → revision 2, previous still release-1
rollback → revision 3, active release-1
classifier.predict → serve only if this runtime matches release-1

Both fixture releases in the demo evaluate the same runtime. For different runtimes, rollback requires loading the matching code/configuration separately. The consumer fails closed instead of pretending a registry write redeploys it.

Senior follow-up: extend the supplied identity-checking consumer with an explicit runtime loader/deployment boundary. Add per-category latency and cost, independent held-out datasets, repeated stochastic runs, and comparison with the current baseline. A new runtime must not borrow approval from an old report.

Staff follow-up: add a canary rollout with exposure accounting, delayed quality signals, rollback authority and ownership. A registry pointer does not automatically roll back downstream data changes. Design compatible output schemas and a reconciliation plan for effects created by the failed release.

Engineer FAQs

Why not use one average quality score? A high average can hide a rare severe failure or an untested category. Keep those outcomes visible and define separate release rules.

Why no LLM judge in the reference? The expected result is a finite label. Exact comparison is simpler and auditable. For open-ended answers, a judge needs independent human calibration, failure recall, severity checks and monitoring for drift.

Can 100% passing regressions be valid? Yes. A useful suite should pass after fixes. Prove that it detects the defects it claims to cover by injecting a known broken candidate and observing the relevant failures.

Are six cases sufficient to ship? No. They demonstrate the mechanics. Production evidence needs representative sampling, rare-risk coverage, untouched evaluation data, uncertainty estimates, and real operating measurements.

Why hash the dataset? Without a stable identity, a later reader cannot tell whether two runs used the same evidence. A hash establishes identity, not label quality or data independence.

What if the model alias changes behind the same name? Pin the most specific supported version and record configuration. Re-evaluate changes. The reference's model ID alone is not a universal reproducibility guarantee.

Does rollback change Bedrock automatically? No. It changes the active report. The supplied classifier.predict consumer enforces that decision by refusing a mismatched runtime. Installing a different runtime or changing cloud traffic is a separate deployment integration, not a side effect of swapping report IDs.

Should evaluation failures be retried until they pass? No. That selects favorable outputs and hides stochastic failure rates. Predefine retry rules for infrastructure faults and record every attempt used in the decision.

What you are expected to hand over

Bring an immutable report, the dataset hash, a blocked broken-candidate run, the promotion/rollback transcript, and an explanation of what the small fixture suite does and does not establish. Include a design for connecting the registry to serving without silently changing the evaluated configuration.

How the review conversation gets harder

Research behind the design

Reviewed September 23, 2026. AWS's evaluation dataset documentation explains reference responses and category-specific reporting. The February 2026 Amazon agent evaluation report describes examining components, tool use and whole-task outcomes. The March 2026 AgentCore evaluation article discusses evaluating interaction flows. This reference implements its own deterministic classifier gate and registry. It does not claim to reproduce Amazon's internal platform.