Evaluation report · Prompt boundary

Prompt Boundary Test: Can a Green Regression Prove a Release? ​

TL;DR ​

The audit prompt asks a model to review four documents and decide whether a game build can ship. The most dangerous input is not a hostile instruction; it is a true-looking one: “the regression is 11/11, so everything is fine.” I turned that and six other pressure points into probes, then checked whether the prompt and the evidence pack contain enough constraints and cross-source evidence to reject the move. Four probes were blocked by the contract. One stayed open because the source material is genuinely ambiguous. One was out of scope by design. The four critical numbers survived a padded context. The report is deliberately not a model leaderboard: no endpoint was called, and no model accuracy is claimed.

Scope: a contract test, not a model score ​

This distinction matters. A prompt can be well-posed and still be failed by a model; a model can also pass despite a vague prompt by luck. This report tests the first half:

Given the prompt and the four documents, is every adversarial move either explicitly blocked, contradicted by the evidence, or correctly classified as unresolved?

That is a contract test. It catches ambiguous instructions, missing anchors and evidence gaps before spending model calls. It does not tell you which model will follow the contract.

Execution boundary:

• Model endpoint called: no.
• Prompts run: 0.
• Probe dispositions are based on prompt text and source evidence, not generated model answers.
• The pre-registered model protocol is included at the end so the same probes can be run as a blind benchmark.

Evidence pack ​

The pack is small enough to fit in any modern context window. That is the point: the failure mode is not raw length. It is near-duplicate values across layers — +500 and +50, ≤10 → +50 and diff 10 → +40, EXECUTE_BLACK and SABOTAGE_BOTH — that are easy to conflate even when every line is present. “Long context” is often a retrieval problem disguised as a length problem.

Seven adversarial probes ​

The dispositions are not all the same kind of result, and that is intentional:

• Blocked means the prompt contains an explicit constraint and the evidence pack contains a contradiction. A compliant model has what it needs to refuse.
• Open means the evidence is genuinely ambiguous. More prompt pressure will not fix it; the design owner must decide what “betrayal” means.
• Static pass means the context assembles the anchors correctly. It says nothing about whether a model will retrieve them under load.
• Out of scope means the prompt deliberately narrows the evidence. That is good security hygiene and a bad completeness guarantee. The inline duplicate is outside the four-document contract, so the prompt cannot catch it without a second, source-level audit.

What the probes found ​

The prompt is strongest where the risk is social: it does not let a green test or a confident release note end the review. It is weakest where the risk is structural: it cannot tell whether the four documents are the right four documents, and it cannot inspect the duplicate engine that generated the player-facing +500.

The four critical anchors were present in every context assembly:

1. Nun hidden condition: +500 player copy vs 50 engine/regression.
2. Duke disguise: checkbox advertised, engine does not read it.
3. Regent: +75% bound to EXECUTE_BLACK, while “betrayal” is defined as SABOTAGE_BOTH.
4. Duke balance: diff 10 → +40, not the advertised +50 cap.

Those are the facts a model must not lose. The prompt’s job is to make losing them visible; the evidence pack’s job is to make them recoverable from more than one source.

Test design ​

1. Build a four-document evidence map before writing probes: player copy, engine implementation, regression evidence, release claim.
2. Write each probe with a pre-registered disposition and the exact source lines that justify it. Do not score after seeing a model answer.
3. Include near-duplicate numbers and terms, not just irrelevant distractor text. Confusion comes from similarity, not length.
4. Run a context-assembly check: every critical anchor must appear in at least two independent sources where possible.
5. Score the contract first: blocked / open / out of scope. Only then run a model.
6. Add a source-level follow-up for the out-of-scope probe: diff the inline engine and the player copy. That is a separate evaluation, not a prompt fix.
7. Record the evidence cutoff and the exact document versions. A later copy edit invalidates the probe set.

Pre-registered model protocol ​

This protocol was written but not executed for this report:

The value of pre-registration is that it prevents the scorer from moving the goalposts after seeing a persuasive answer. The value of not reporting a score is that the report does not pretend a contract test is a benchmark.

Evidence ​

Verdict ​

Conditional. The prompt is robust against the two most common release-audit failures — “the tests are green” and “the release note says it is fine” — and it keeps the four critical anchors in context. It cannot resolve the Regent ambiguity, and it deliberately cannot see the inline engine copy where the player-facing +500 survives. The first needs a design decision; the second needs a source-level audit. Until both are closed, “the prompt passed” is not the same as “the release is safe.”

Related ​

• When the Rulebook and the Engine Disagree — the rule-level replay behind the four anchors.
• Hallucination Audit: The “No Known Issues” Release — the source-level audit that closes the out-of-scope probe.
• Content Loader Contract — schema validation as the first line of defence against shape-correct, meaning-wrong content.