See Wikipedia:Erasure_code for a general discussion of erasure codes.
We don’t need any of this stuff for version 1.0 of the distributed wiki. But it might help reduce hardware costs for very large systems.
Say we want to guarantee no files are lost when any one disk fails. And we want to be able to store N disks full of files.
Say we want to guarantee no files are lost when any 2 disks fail. And we want to be able to store N disks full of files.
“RAID 5” and “RAID 6” use one particular “erasure code”. Would perhaps one of these other erasure codes work better for a distributed wiki? Wikipedia:Erasure_code
“A Reed-Solomon Code for Disk Storage, and Efficient Recovery Computations for Erasure-Coded Disk Storage” by M.S.Manasse, C.A.Thekkath, and A. Silverberg wdas.pdf “… assuming independent failures, correctable within a day, a code which allows up to three failures per group of 250 disks should experience a data loss once every 50,000 years in a set of 10,000 disks.” “… we present a simplified … triple-erasure correcting Reed-Solomon code …”
“Decentralized Erasure Codes for Distributed Networked Storage” http://www.eecs.berkeley.edu/~adim/DPR_IT.pdf
“In the Koh-i-Noor project, we are investigating how to construct large, inexpensive, reliable disks. We are using somewhat-novel erasure codes (similar to standard Reed-Solomon codes) and parallel reconstruction techniques during repair. … In particular, we expect to build clusters of up to 256 disks, each attached to a separate processor. Each processor is connected to two independent networks … … assuming a MTTF for individual disk drives of 50 years. If we suffer no cabling failures, no dependent failures of hardware, and a similar MTTF for CPUs to those of disks, we might hope to be able to defer any maintenance on any cluster for up to five years.
Triple-erasure-correcting codes give us an expected time of 50,000 years until we experience data loss on any block in the petabyte. “ http://research.microsoft.com/research/sv/kohinoor/
“Asymptotically Optimal Erasure-Resilient Codes for Large Disk Arrays (2000)” by Yeow Meng Chee, Charles J. Colbourn, Alan C. H. Ling http://citeseer.ist.psu.edu/chee00asymptotically.html
“Erasure Codes for Storage Applications” by James S. Plank. http://www.cs.utk.edu/~plank/plank/papers/FAST-2005.html
What is the “Erasure Code Internet Protocol” ? “ECIP and S-PACK are UDP based protocols that were developed as a replacement for TCP/IP and RTP, for transmission of audio and video conferencing steaming, broadcast or VOD and VOIP over the Internet.” – http://ecip.com/