Skip to content

Evidence log

Evaluation Reports ​

Each report records a claim, the checks used to falsify it, and a verdict that separates "it works" from "it works under these conditions". Reports cover prompt robustness, hallucination audits, multi-turn state, and build-level regressions.

How to read a verdict ​

VerdictMeaning
PassEvery stated check passed; residual risks are listed.
ConditionalThe intent was met, but a literal requirement, environment or scope condition remains unresolved.
FailAt least one stated check failed; remediation is required before reuse.
Fail

The v2.0.0 release note said there were no known issues. Six claims later, five were contradicted by the project’s own rulebook, engine copy, regression scope or inline UI duplicate — including the +500 line that survived in the player-facing modal.

  • hallucination-audit
  • claim-verification
  • logic-gap
  • release-gating
Claims audited
6
Contradicted
5
Release blockers
2
Unsupported numbers
0
Conditional

The spec asked the loader to read work/*.md, but no work/ directory exists. Instead of inventing one, the audit mapped Work to the existing projects/ URLs and recorded the deviation as a conditional pass.

  • hallucination-audit
  • path-validation
  • information-gap
Declared globs
3
Existing globs
2
Mapped collections
1
Broken links
0
Conditional

Seven adversarial prompt moves tried to turn 11/11 engine tests into a release pass. Four were blocked by the prompt contract, one design ambiguity stayed open, one source-level blind spot remained, and the four critical anchors survived a padded context.

  • prompt-robustness
  • adversarial-input
  • long-context
  • evidence-contract
Adversarial probes
7
Blocked by contract
4
Open / out of scope
2
Critical anchors
4/4

Crystaria (with Paw and Kyle)