Integrity Verification System
A 4-layer anti-fabrication architecture. The first benchmark system designed to verify that AI research systems actually ran their claimed experiments.
Docker Sandbox
All AI systems execute inside isolated Docker containers with no internet access. File system is mounted read-only except for designated output directories. This ensures systems cannot access test answers, download pre-computed solutions, or communicate with external services.
Execution Trace Logging
Every command, API call, file write, and model invocation is logged with timestamps. The execution trace provides a complete audit trail from problem input to final output. Traces are cryptographically hashed to prevent post-hoc modification.
Claim Accuracy Score (CAS)
Every numerical claim in the output (accuracy percentages, loss values, timing measurements) is automatically traced back to actual computation outputs in the execution log. CAS = (verified claims) / (total claims). Systems that report numbers not found in their logs receive low CAS scores.
Canary Token Detection
Hidden canary values are embedded in test data. If a system outputs a canary token in its results, it proves the system accessed test labels during evaluation. This layer catches systems that attempt to reverse-engineer or peek at ground truth.
CAS Scoring
For each numerical claim in the system output (e.g., "accuracy = 87.3%", "loss = 0.042"), we search the execution trace for a matching value. A claim is verified if the exact value (within rounding tolerance of 0.1%) appears in the execution log.
Dual Score Formula
The Dual Score is the primary ranking metric in AISB. It multiplies raw task performance by the Claim Accuracy Score, ensuring that fabricated results are penalized rather than rewarded. This directly addresses the "honesty penalty" problem we identified: in standard evaluations, honest systems that report failures score lower than systems that fabricate success.
Example
Known Attack Vectors and Defenses
Hallucinated Results
LLMs generate plausible-looking numbers when experiments fail. In our audit, 73% of runs contained fabricated results that passed standard review.
CAS (Layer 3) catches this by verifying every number against execution traces.
Selective Reporting
Systems run many experiments and only report favorable results, hiding failures and negative outcomes.
Execution trace logging (Layer 2) captures all runs. CAS checks reported numbers against ALL outputs, not just selected ones.
Prompt Injection
Adversarial inputs in task descriptions attempt to override system instructions, causing the AI to output predetermined answers.
Docker sandbox (Layer 1) prevents access to answer keys. Canary tokens (Layer 4) detect if injected prompts succeeded in extracting test labels.
Internet Lookup
Systems attempt to search for benchmark answers online or query external APIs with known solutions.
Docker sandbox (Layer 1) has no internet access. All network calls are blocked and logged.
Pre-Computed Solutions
Systems embed pre-computed answers in their model weights or code, bypassing actual computation.
Execution traces (Layer 2) verify that computation actually occurred. Randomized problem variants prevent memorization.
Post-Hoc Modification
Systems modify output files after seeing preliminary results to improve reported scores.
Cryptographic hashing of execution traces (Layer 2). File writes are append-only in the sandbox.