CONTINUUM · BLIND HOLDOUT
PREREGISTERED · LABEL-HIDDEN · REAL PROVIDERS
The answer key was sealed first.
An independent Bedrock generator created 60 unseen provider-state and attack combinations. The challenge and labels were checksum-addressed in S3 before either arm ran. Only after raw-RAG and Continuum completed real disposable GitHub and S3 effects did a separate evaluator open the labels.
Public gate—unchanged preregistration
Paired cases—120 arm observations
Continuum success—verified provider outcome
Raw-RAG success—same hidden holdout
False promotions—raw-RAG / Continuum
Memory exposure—raw-RAG / Continuum
Citation adoption—raw-RAG / Continuum
Residual effects—duplicate + cleanup
Blind evaluation firewall
01GenerateBedrock creates new paraphrase, poison, stale, and conflict variants.
02SealChallenge, labels, and scoring commitment receive content-addressed S3 keys.
03DenyCandidate IAM is explicitly denied the sealed label object.
04ExecuteBoth arms perform the same GitHub Releases and S3 sandbox incidents.
05EvaluateLabels open only after both arms finish; receipts and outcomes determine score.
Checksum-bound lineage
| Workflow | — |
|---|---|
| Source SHA | — |
| Challenge SHA-256 | — |
| Commitment SHA-256 | — |
| Seal receipt SHA-256 | — |
| Public result SHA-256 | — |
| Models | — |