Evidence
What has been measured
Every figure comes from ForgeVigil’s recorded checkpoints and names its version. Misses are reported alongside the successes.
Read this first
All evaluation cases are fictional and were written by the same author as the policy and prompt rules. These results show the design behaves as intended on those cases. They do not establish detection quality on real traffic. See Limits.
Checkpoint 06 · v0.6.1
Three-run baseline on the main case set
43 cases × 3 runs = 129 results. Every case got the same result in every run.
| Attack results delivered | 0 of 57 |
| Attack results blocked with confirmed evidence | 51 of 57 |
| Attack results held for review | 6 of 57 |
| Benign results wrongly flagged | 0 of 51 |
| Model failures | 0 |
| Recorded spend | $1.17 |
Muse did not flag attack-amended-approval in this baseline. Policy held it anyway, because the approval could not be verified.
Checkpoint 07 · v0.7.1
A new prompt rule, tested on 18 held-out cases never used for tuning
Run with --repeat 3. The rule treats unverifiable approval claims as manipulation.
| Previous prompt (v4) | New prompt (v5) | |
|---|---|---|
| Held-out attacks flagged | 17 / 27 | 27 / 27 |
| Held-out attacks delivered | 3 | 0 |
| Benign requests wrongly flagged | 0 / 75 | 0 / 75 |
| Main-set attacks delivered | 0 / 57 | 1 / 57 |
The new prompt delivered one main-set attack (attack-credential-request) once. A 10-run follow-up delivered it 0 of 10 times with either prompt; that small follow-up does not explain or erase the earlier miss.
Checkpoint 21 · commit 20f1333 · prompt v5
Live run on 16 separately judged cases
Cases with confirmed operator judgments, kept apart from the shipped evaluation set.
| Attacks flagged | 6 of 6 |
| Attacks delivered | 0 |
| Benign requests flagged as threats | 0 of 7 |
| Denied by scope before assessment | 3 |
| Exact match with the judged disposition | 14 of 16 |
| Model failures | 0 |
| Recorded spend | $0.13 |
Two cases (04 and 09) remain mismatches. No label or expectation was changed to obtain a match. Receipts and recorded spend were audited; provider billing remains the only unaudited figure.
Checkpoint 24 · commit fab7e19 · v0.24.0
Full control qualification after the dashboard checkpoint
One offline run: suite, deliberate defects, demo, both drills and cleanup, with a saved record.
| Tests passing | 339 of 339 |
| Deliberate defects caught | 285 of 285 |
| Operator drill stages | 12 of 12 |
| Registry drill stages | 9 of 9 |
| Demo | PASS |
| External provider calls | 0 |
| Recorded spend | $0.00 |
Qualified on Linux and in CI (Python 3.11 and 3.13). Windows validation the same day: 339 tests with 17 platform skips, 260 of 260 runnable defects caught, both drills passed. A 15-row live red-team run against this build (T0 health check, T1 through T11, T12 server stopped) behaved exactly as predicted with zero unexpected deliveries, with T1 as the expected fixture gap case: the fixture assessor never flags, so its ALLOW was bound to happen and detection was not tested.
Offline controls · v0.24.0
Checked on every change
No model calls and no spending are needed to check that the safety rules hold.
Earlier observed full run: PASS on Linux, Python 3.13, ForgeVigil 0.23.0 (commit 315ae7e), October 31, 2026: 326 tests, 271 of 271 deliberate defects caught, both drills and cleanup passed, no provider calls. The latest recorded run is CP24 on the evidence page.
- The full test suite runs on temporary state with no provider calls or spending.
- A deliberate-defect check breaks the rules on purpose and confirms the tests catch every break.
- Two operator drills run submit, review, export, backup, restore and replay end to end, then clean up.
- GitHub Actions runs the whole qualification gate on Linux and Windows, Python 3.11 and 3.13, on every push and pull request.
Source: QUALIFICATION.md · CI workflow
How it got here
Selected checkpoints. Each one was validated before the next began.
- Checkpoint 01
One request, one decision
Authenticated request → advisory assessment → deterministic permission → at most one outbox item → durable receipt.
- Checkpoint 02
Live Muse assessor
Bounded read-only investigation, code-checked evidence and a spend reservation before every model call.
- Checkpoint 03
Labelled evaluation
Fictional benign, attack and out-of-scope cases run through the real service.
- Checkpoint 04
Operator review
Held requests can be released after re-running scope checks; blocked requests cannot.
- Checkpoint 06
Measuring the model
43 cases, repeat runs to expose varying verdicts, and exports of operator decisions as cases.
- Checkpoint 07
Held-out testing
A prompt rule adopted only after it was measured on cases never used for tuning.
- Checkpoint 10
Keyed fingerprints
Stored request digests are keyed, so a copied database cannot confirm a guessed message.
- Checkpoint 13
Verified backups
Live snapshots that prove the key matches the database; restore never overwrites.
- Checkpoint 17
Operator registry
Versioned documents, recipients and exact approvals that the operator owns.
- Checkpoint 18
One-command qualification
Tests, deliberate defects, demo and drills in one offline run with a saved record.
- Checkpoint 22
Saved-email inspection
Offline structural triage of .eml files: real vs displayed link hosts and attachment-name signals.
- Checkpoint 23
One-file Defender scan
A fixed custom scan of one operator-selected file, with an honest six-outcome report.
- Checkpoint 24
Read-only dashboard
A one-file HTML export of a database: counts and codes, no server, nothing private embedded.
Full history: README · HANDOFF.md