Skip to content
The methodOpen data
Back to the collection

Silent Data Corruptions at Scale

Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon

Show all 7 authors

Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, Sriram Sankar

2021 · arXiv

Distributed systems
Read the paper
HN score
0
Mentions
6
Readers
6
Months
6
What’s in a score?

From the abstract

Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

sitkack
Neutral mention · rule
Read the paper below. Absolutely frightening, we all basically swimming in quicksand. ECC should be mandatory! Not using it is like huffing leaded gas. Silent Data Corruptions at Scale https://arxiv.org/abs/2102.11245
Original comment ECC RAM should be a human right
fulafel
Neutral mention · rule
These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on empirical data at Facebook)
Original comment Single random bit flip causes error in certificate transparency log
freehrtradical
Neutral mention · rule
The article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read: https://arxiv.org/pdf/2102.11245.pdf > After a few iterations, it became obvious that the computation of 𝐼𝑛𝑡(1.153)=0 as an input to the math.pow function in Scala would always produce a result of 0 on Core 59 of the CPU. However, if the computation was attempted with a different input value set 𝐼𝑛𝑡(1.152)=142 the result was accurate.
Original comment Cores that don’t count [pdf]

Keep following the thread.

More in Distributed systems

More from the same topics.