The AX-Ray project presents a public diagnostic case that it says revealed and then repeatedly demonstrated causal leakage defects in two public AI models: Zyphra/Zamba2-1.2B and nvidia/Nemotron-H-8B-Base-8K. Both models appear on the AX-Ray dashboard as Causal-LEAK cases, while the framework treats confirmed causal leakage as a deployment-blocking defect, regardless of a model’s score on general capability tests.
The material published on the Hugging Face blog begins with a core thesis: correctly answering standardized-test questions is not enough to judge whether a model is ready for practical use. Deployment safety, according to VIDRAFT’s presentation, includes the correctness of causal relationships, serving-path consistency, robustness against adversarial conditions or long contexts, data integrity, infrastructure security, regulatory readiness, and agentic-system risks.
What Is Causal Leakage?
In an autoregressive language model, representations or probability-value behavior at an earlier position in the sequence are expected to remain unaffected by future tokens that are not available at that position. Causal leakage occurs when later information or information beyond the context changes hidden states, probability values, or evaluation-related behavior in the earlier portion of the sequence.
AX-Ray distinguishes this issue from hallucination, refusal failure, prompt injection, and benchmark contamination. These are important problems, but they do not describe the same defect; causal leakage concerns the correctness of the model’s computational path. According to the material, this defect may affect prefix stability, the correctness of hidden states, probability-value consistency, batched or hybrid sequence processing, cache and serving reliability, confidence in long contexts, evaluation validity, and agentic execution safety.
Why Might Capability Tests Not Detect It?
Most public leaderboards focus on how often a model provides a correct answer, which is a necessary signal but does not cover every property of internal behavior. Causal leakage may remain invisible in question-answering, mathematics, programming, and instruction-following tests because these tests typically observe only the final answer.
By contrast, AX-Ray examines deeper properties, including prefix invariance, serving-path consistency, and whether critical failures should outweigh the overall capability result. The material summarizes the principle clearly: high capability does not automatically mean deployment readiness.
A Multidimensional Diagnostic Framework
AX-Ray organizes its work around three diagnostic axes and 11 operational categories, with a public catalog containing 117 diagnostic records. These records cover technical questions linked to evidence, issue severity, detection direction, remediation direction, and governance context.
- MODEL-SCAN: Examines model correctness, reliability, robustness, safety, data integrity, efficiency, internal structure, and remediation directions.
- AX-SCAN: Focuses on serving, infrastructure, security, compliance, and operational risks.
- AGENT-SCAN: Addresses agentic-deployment risks, such as tool permissions, hijacking, loops, memory contamination, deletion behavior, and autonomy governance.
Other categories include causal safety and serving integrity, reliability, robustness and long contexts, security, safety and alignment, data integrity and evaluation methodology, efficiency, quantization and architecture, inspectable internal structure, and remediation. The operational axis also includes categories specific to cross-engine serving drift, infrastructure security, and regulatory compliance.
Distinguishing a Model Defect from a Serving Problem
AX-Ray also records an outcome audited through the model’s upstage/Solar-Open2-250B API when run through FP8 vLLM. The run showed a reproducible anomaly in token-probability logging within the serving or interface path, but the framework does not present it as confirmed causal leakage at the model level.
According to the material, the internally inspectable D1 and D7 tests remain pending; therefore, the row was marked official_dhs=false, with an indication that it underwent API auditing while the internal inspection was not complete. This separation is a central part of the methodology: serving anomaly, score-logging issue through the interface, and hidden-state leakage at the model level are not one claim.
Transparency and Disclosure Limits
AX-Ray says it discloses leaderboard rows, model-level diagnostic summaries, high-level category scores, item classification, a jurisdiction-linkage structure, and some public reports, in addition to specific causal-leakage cases. However, it does not publish private test recipes, operational threshold details, raw sensitive prompts, raw harmful outputs, patent-protected implementation details, internal evaluation commands, or procedures that could be converted into exploitation methods.
The project presents this balance as an attempt to make safety diagnostics accountable without turning them into a guide for bypassing models or reproducing harmful behavior. It also links the catalog to governance contexts in Korea, the European Union, the United States, Japan, China, the United Arab Emirates, and Saudi Arabia, while emphasizing that AX-Ray is diagnostic guidance rather than an official legal standard.
The importance of the published case lies in its proposal of an additional layer above capability tests: evaluation asks not only whether the model answers well, but also whether its model, serving path, and agentic environment can be trusted during deployment, operation, and governance. The project makes its public data and diagnostic space available through FINAL-Bench/AX-RAY, while details of the private tests remain unpublished.