A passing check
needs a precise question.
Verification is most useful when you can see the system under test, the fixture, the expected behavior, and the limits of the assertion. This guide explains the evidence described in the public theo Arca report.
Offline verification
≠ live model quality.
01three different kinds of evidence
Do not let one kind stand in for another.
- Browser illustration
- The public workbench runs a deterministic simulation with synthetic fixtures and mock tools. It demonstrates its own rules and makes no production AI request. It does not test the production gateway, a detector, or a live model.
- Offline gateway verification
- The Arca report describes the real gateway application exercised against deterministic fake upstreams, including adversarial responses. These checks can establish that the tested build enforces the asserted behavior for those fixtures.
- Live candidate evaluation
- The report describes probes aimed at a deployed candidate to assess model-dependent behavior and serving characteristics. Results belong to the exact build, configuration, probe set, and deployment, not to a model name in the abstract.
02read an assertion
Test a refusal before celebrating an answer.
{
"fixture": "synthetic routing check",
"declared_class": "sensitive",
"eligible_targets": [],
"expected": {
"decision": "refuse",
"inference": "not attempted",
"tool_execution": "not attempted"
}
}What would this check tell us?
Observe the boundary: assert that an empty eligible set produces a refusal before inference or a tool side effect, not just a polite refusal string after execution.
Add a positive control: a separate non-sensitive fixture with an eligible target should be able to take its intended path. A system that rejects everything is not a useful passing test.
Retain the failure detail: record the assertion, configuration identity, and unexpected action without including sensitive fixture content in logs.
03what the report organizes into gates
Read the suite name and the actual assertion.
- Safety & identity
- Checks around sensitive-value handling, scoped re-identification, and served identity. Adversarial fixtures include output split across streaming boundaries.
- Capability & coding
- Tool-schema acceptance and rejection, plus structural domain-engineering checks. Valid structure is not proof of overall correctness, security, or clinical validity.
- Long context
- Planted-value retrieval at selected positions. Compact fixtures do not describe behavior for every input length, document shape, or retrieval task.
- Parity & operations
- Comparisons against a reference and checks against deployment-specific serving budgets. A deterministic offline upstream cannot establish real-world latency or model quality.
04an evidence record worth keeping
Make a result reproducible before making it persuasive.
Attach the build and configuration identity, run date, execution mode, fixture set, assertions, and failure details to a result. Say which components were real and which were replaced by deterministic stand-ins. A copied “pass” without that context loses most of its meaning.
Keep disallowed behavior as a first-class test case. Exercise missing scopes, malformed tool arguments, absent eligible targets, and adversarial content. Make an important failure block the relevant release decision rather than disappearing inside an average.
Then examine the untested edges: new languages, different document formats, changed policies, more complex tools, and people using the workflow in unexpected ways. Passing the same fixtures again does not remove those open questions.
Does an offline test prove that a model gives correct answers?
No. With a deterministic fake upstream, the supplied answer is controlled by the fixture. The check can assess the surrounding system and the harness, not the quality of a deployed model's reasoning.
Does a structural coding check prove the code is safe to ship?
No. An asserted structure can be necessary without being sufficient. Application tests, code review, security analysis, domain validation, and deployment checks still matter.
Can our browser workbench result be used as a release gate?
Not as evidence about production behavior. It is an educational simulation. Build a deployment-specific verification process that exercises your real application boundaries and records the scope of each result.
Bring the claim back to the architecture.
See which parts of the described Arca request path enforce a boundary, and where deployment review is still needed.