Silent data errors, or Silent Data Errors/Silent Data Corruption, reveal a growing gap between a chip passing conventional manufacturing tests and its ability to produce correct results throughout its operational life. A processor may pass ATPG tests, transition-fault and stuck-at-fault tests, and at-speed structural tests, then execute a calculation incorrectly without recording an obvious failure, before the corrupted result reaches an application or an AI training task.
This assessment is based on an analysis published by Semiconductor Engineering on September 10, 2026, bringing together views and findings from Advantest, Siemens EDA, NXP, Synopsys, Intel, Meta, Google, and proteanTecs. It does not concern the launch of a specific product, but rather a shift in how processor quality is defined, how test coverage is evaluated, and how processors are managed after reaching the data center.
A Rare Problem at the Chip Level, Widespread at Fleet Scale
Silent data errors appear rare when measured on an individual device, but they become frequent and costly when millions of processors operate at high utilization rates. Early analyses from Google and Meta indicate that these errors may affect one server in every thousand, equivalent to a level ranging from 100 to 1,000 defective parts per million. Even a rate of 10 FIT, meaning one failure per billion operating hours, can mean an error occurring approximately every four days when 10 million devices are deployed.
The danger is that the error may appear as an incorrect numerical result or an undefined value, then cause database corruption, unexpected behavior in an AI model, or contradictory analytical results. Because the processor itself may not send an error signal, the fault may not be detected until an illogical result appears at the end of a lengthy task.
Why Do Conventional Tests Fail?
Silent errors are associated with marginal defects such as high-resistance metal connections, weak bridge defects, and timing and voltage variations, in addition to the effects of aging, radiation, temperature conditions, and workload. Their likelihood increases with advanced process nodes, where margins narrow and connections become smaller and more resistive. Packages based on chiplets also increase the complexity of verification paths.
The industry estimates that about 80% of corrupted execution errors are associated with defects that escape zero-time testing, while the remaining 20% appear intermittently or as a result of aging. However, testing all possible combinations of voltage, frequency, temperature, age, and workload type is impractical. Linking a failure that appears at the system level to a specific defect pattern in chip testing can also take weeks and require collaboration among design, test, failure-analysis, and system-integration teams.
Siemens EDA points out that relying on a single-input transition when testing delay faults may not simulate actual functional operation, because switching multiple inputs can produce greater delays. Tests therefore need to target multiple voltage, temperature, and frequency corners rather than relying solely on structural fault models that do not cover every usage condition.
Deeper Testing from the Factory to the Data Center
This problem is redefining the concept of test coverage. Instead of counting the known manufacturing defects that a test can detect, coverage is also becoming linked to the probability of detecting an incorrect computational answer during realistic operation and under an actual workload. As a result, companies are turning to system-level functional tests, workload-aware tests, and mission-mode tests, alongside improvements to built-in test design and monitoring of on-chip margins.
Intel’s experience illustrates the scale of the challenge. After testing 1.2 million processors across five generations of Intel Xeon processors, the company needed more than 1,000 functional tests in the DCDiag suite and 5,000 synthetic stress tests to detect silent-error defects. The tests were not evenly distributed across defects: approximately 50% of defective parts could be detected using only 5% of the tests, while detecting 90% of errors required more than half of the 1,000 tests. The results also showed that more than 70% of defects were detected by only one test, and that an effective test recipe for one product generation could not be transferred directly to the next generation.
What Is Changing in Practice Across Fleets?
The response does not stop at the factory gate. Data center operators use layers of software screening and field testing to isolate servers or cores that exhibit abnormal behavior. At Meta, the Fleetscanner program takes a server out of service and runs it through computational tests with known results, while the Ripple program executes short patterns during normal operation. Hardware Sentinel analyzes application exceptions and system behavior without allocating test workloads, and Meta reported that it improved detection by 40% compared with methods based on testing across different architectures, applications, and data centers.
Google uses a range of safeguards, including end-to-end validation of test sets, redundant computations and comparisons, invariant checks and assertions, verification of data in transit, and periodic verification of stored data. Application-level measurements, such as the Spanner approach, can also detect corruption and remove suspect devices from the fleet, while screening methods are modified when devices return in order to identify cores vulnerable to the problem before it worsens.
From Shipping Tests to Silicon Lifecycle Management
These trends are driving continuous monitoring of timing, voltage, temperature, and degradation margins, along with the use of time-series analysis and machine learning to detect deviations before they turn into silent errors. Parametric measurements and machine-learning models may help isolate devices that deviate from their expected profile, while diverse instruction tests, redundant execution, and comparisons between cores can improve detection opportunities.
However, this approach does not eliminate the limitations. No single method captures every error mechanism, test results do not automatically transfer from one design to another, and root-cause diagnosis remains difficult when the effect appears in an application far from the physical defect. The analysis also indicates that the sharing of failure data among manufacturers, test-tool providers, data center operators, and universities remains limited because of commercial and legal considerations. Therefore, the real change is not the addition of a single test, but the construction of a chain of visibility extending from manufacturing to operation, while recognizing that processor quality is not fully determined at the moment it is shipped.