WO2014060572A2 - Method and system for storing a file on a plurality of servers - Google Patents
Method and system for storing a file on a plurality of servers Download PDFInfo
- Publication number
- WO2014060572A2 WO2014060572A2 PCT/EP2013/071823 EP2013071823W WO2014060572A2 WO 2014060572 A2 WO2014060572 A2 WO 2014060572A2 EP 2013071823 W EP2013071823 W EP 2013071823W WO 2014060572 A2 WO2014060572 A2 WO 2014060572A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- servers
- file
- chunks
- replies
- reconstruction
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/183—Provision of network file services by network file servers, e.g. by using NFS, CIFS
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
- G06F11/1096—Parity calculation or recalculation after configuration or reconfiguration of the system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
- Retry When Errors Occur (AREA)
Abstract
The present invention relates to a method for storing a file on a plurality of servers, wherein the number of servers is n and the maximum number of servers which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers, and wherein n equals 2t+b+1, comprising the steps of a) Dividing the file into a plurality of chunks, wherein the number of chunks is equal to or greater than the number of servers n, b) Sending n chunks of the file to the n servers, wherein one chunk is sent to each server, c) Determining the number of replies r from the n servers indicating successful storage of the respective chunks, d) Checking if the number of replies r matches a terminating condition, and if not e) Generating a new file based on one or more chunks of the old file, a reconstruction threshold of the old file and the number of replies, f) Perform steps a)-e) with the new file as file for these steps, until the terminating condition in step d) is fulfilled, wherein the terminating condition is based on the difference between the reconstruction thresholds of the new file of step e) and the old file of step a) and the maximum number of servers which might be fail. The present invention further relates to a system for storing a file on a plurality of servers.
Description
METHOD AND SYSTEM FOR STORING A FILE
ON A PLURALITY OF SERVERS
The present invention relates to a method for storing a file on a plurality of servers wherein the number of servers is n and the maximum number of servers which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers, and wherein n equals 2t+b+1. The present invention further relates to a system for storing a file on a plurality of servers, wherein the number of servers is n and the maximum number of servers which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers and wherein n equals 2t+b+1 , comprising the plurality of servers and a writer for writing the file onto the servers, preferably for performing with a method according to one of the claims 1 -13.
In distributed storage systems, for example a RAID system, a file is dispersed on a plurality of servers, in case of a RAID system on a plurality of hard discs. The file is dispersed in such a way that for example in a RAID system when a hard disc fails or more general is simply unavailable, the number of dispersed file fragments on the remaining hard discs is large enough to restore or reconstruct the dispersed file from the file parts stored on the remaining operating hard discs.
Unavailability of entities like servers in distributed computing systems or for example hard discs in a RAID system can be distinguished into byzantine failures and crashes. Byzantine failures are arbitrary faults occurring for example during an execution of an algorithm by the distributed system. When a byzantine failure has occurred the distributed system may respond in an unpredictable way. Byzantine failures may e.g. arise from malware or hackers that attack storage servers or from manufacturer faults.
The other type of failure is a crash leading to unavailability at least temporarily. A crash may also be a intended shutdown of a server, for example for maintenance reasons.
However unavailability of entities in distributed systems occurs only occasionally. Such a worst case scenario would include unpredictable message delays, for example due to a network partition or a swamped server. In most cases the distributed system is functioning: The communication is synchronous and messages are delivered within respected time bounds in the distributed system Further a distributed computing system is conventionally configured to tolerate a large number of server failures although the occurrence of actual failures is rather low.
Conventional storage protocols like byzantine storage protocols described in James Hendricks, Gregory R. Ganger, and Michael K. Reiter. 2007, "Low- overhead byzantine fault-tolerant storage", in Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (SOSP Ό7) proposes handling of worst case scenarios. One of the disadvantages is however that a large overhead communication with respect to the information exchanged is necessary leading to a high blow-up factor A further disadvantage is that the proposed methods therein are inflexible relating only to byzantine failures of servers.
It is therefore an objective of the present invention to provide a method and a system for storing a file on a plurality of servers with reduced communication overhead for dispersing the file onto the plurality of servers. It is a further objective of the present invention to provide a method and a system for storing a file on a plurality of servers providing an optimized blow up factor with respect to the amount of information sent.
It is an even further objective of the present invention to provide a method and a system for storing a file on a plurality of servers which are more flexible, in particular with regard to failure types and/or error correction codes.
The aforementioned objectives are accomplished by a method of claim 1.
In claim 1 a method for storing a file on a plurality of servers is defined, wherein the number of servers is n and the maximum number of servers which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers, and wherein n equals 2t+b+1.
According to claim 1 the method is characterized by the steps of
a) Dividing the file into a plurality of chunks, wherein the number of chunks is equal to or greater than the number of servers n,
b) Sending n chunks of the file to the n servers, wherein one chunk is sent to each server,
c) Determining the number of replies r from the n servers indicating successful storage of the respective chunks,
d) Checking if the number of replies r matches a terminating condition, and if not
e) Generating a new file based on one or more chunks of the old file, a reconstruction threshold of the old file and the number of replies,
f) Perform steps a)-e) with the new file as file for these steps, until the terminating condition in step d) is fulfilled,
wherein the terminating condition is based on the difference between the reconstruction thresholds of the new file of step e) and the old file of step a) and the maximum number of servers which might be fail.
The aforementioned objectives are also accomplished by a system of claim 14.
In claim 14 a system for storing a file on a plurality of servers is defined, wherein the number of servers is n and the maximum number of servers which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers, and wherein n equals 2t+b+1 , comprising the plurality of servers and a writer for writing the file onto the servers, preferably for performing with a method according to one of the claims 1 -13.
According to claim 14 the system is characterized by dividing means, preferably a writer, operable to divide the file into a plurality of chunks, wherein the number of chunks is equal to or greater than the number of servers n, to sending means,
preferably the writer, operable to send n chunks of the file to the n servers, wherein one chunk is sent to each server, and deternnining means, preferably the writer, operable to determine the number of replies r from the n servers indicating successful storage of the respective chunks, checking means, preferably the writer, operable to check whether the number of replies r matches a terminating condition, generating means, preferably the writer, operable to generate a new file based on one or more chunks of the old file, a reconstruction threshold of the old file and the number of replies, and recursive means operable to operate recursively the dividing means, the sending means, the determining means, the checking means and the generation means with the new file as file, until the terminating condition is fulfilled, wherein the terminating condition is based on the difference between the reconstruction thresholds of the new file and the old file and the maximum number of servers which might be fail. According to the invention it has been recognized that synchrony is exploited to achieve a communication blow-up factor which is optimal. For example the blowup factor is at most n/(r-t) where r is the number of responsive servers with n-t < r < n in a synchronous execution. In an asynchronous execution a low bound of n/(n-2t) is achieved. A blow up factor of n/(r-t) is optimal because the file must be recoverable despite a number of failed servers t out of the r responsive servers.
According to the invention it has been further recognized that flexibility is enhanced since generic erasure coding schemes may be used, i.e. the invention is not limited or restricted to a specific type of erasure coding technique. Further both crashed servers and byzantine servers and/or a combination of both may be respected.
In other words synchrony is exploited for providing a method and a system for a distributed storage with an optimal communication and blow-up factor and enables to recursively apply erasure coding on data parameters in particular by the number of received replies as a reconstruction threshold.
Further features, advantages and preferred embodiments are described in the following subclaims.
According to a preferred embodiment in step a) when a total number of generated chunks is greater then the number of servers, the number of chunks generated in addition to the n chunks is dependent on the number of servers which might fail and a reconstruction threshold for the file. This enables to generate in addition to n so-called main chunks a number of auxiliary chunks. These auxiliary chunks are then preferably constructed in one step together with the main chunks allowing the use of verification techniques for examples to calculate cross check sums accompanying the chunk in a reply. This enables to verify if the corresponding chunk on the server was somehow modified.
According to a further preferred embodiment for the first performing of step a) the reconstruction threshold is based on an estimated number of responsive servers. This enables to provide a number of chunks for the file which are sufficient to reconstruct the file based on the number of responsive servers. For example the estimation may be performed on historic data on availability of a server or any other data indicating responsiveness like communication between a writer and the respective servers or the like. According to a further preferred embodiment the estimated number is greater than a sum of byzantine servers b and servers t-b that might crash. If the number of responsive servers is r, the number of servers that might be fail is t and the number of byzantine servers is b then the number of responsive servers is represented by r > t + b + 1.
According to a further preferred embodiment the total number of chunks in step a) is 2 t + th, wherein t is the maximum of servers that might be fail and th is the reconstruction threshold for the corresponding file. One of the advantages is, that then the total number of chunks may be different in each round of steps a)-f), so that reconstruction of the overall file, i.e. the original file, is ensured.
According to a further preferred embodiment the one or more chunks for generation of the new file are solely based on the one or more of the prior non- sent remaining chunks. This enables to only count the number of responsive
servers but not to determine which chunk of the already sent chunks was stored successfully. Thus, efficiency is enhanced.
According to a further preferred embodiment chunks in addition to the generated n chunks are only generated to the extent that the termination condition is not matched. This further leverages efficiency, since - when the number of responsive servers is high - only a few chunks are used for the next round whereas if the number of responsive servers is low then more chunks are generated to ensure consistency for reconstruction. To provide chunks to the extended termination condition is not matched, i.e. to create further chunks on demand, rateless codes such as online codes, described in P. Maymounkov, "Online Codes", Technical Report TR2002-833, Technical report, New York University, 2002, which is incorporated by reference herein, can be used to create them on demand. According to a further preferred embodiment the generation of the new file is performed by concatenating one or more chunks. This allows in an easy and efficient way to generate a new file.
According to a further preferred embodiment a timeout threshold is used when determining the number of replies according to step c), preferably wherein the timeout threshold starts after a predetermined number of received replies, preferably wherein the predetermined number corresponds to the number of replies needed for reconstruction of the file. When a timeout threshold is used a termination condition for waiting for the number of replies of responsive servers is enabled. This allows to further perform the next step even if a certain number of servers has not replied yet. If preferably the timeout threshold starts after a predetermined number of received replies it is ensured that at least it is waited until a number of replies is received. Preferably this number corresponds to the number of replies needed for reconstruction of the file. In this case it is ensured that the number of replies of responsive servers is high enough to reconstruct the file. Thus resending of the already sent chunks to all servers is not necessary and the next step may be performed.
According to a further preferred embodiment the timeout threshold is dynamically adapted in each round of steps a)-e), preferably wherein the adaption is based on connection conditions between a writer of the file and the servers and/or of server conditions. By dynamically adapting the timeout threshold, flexibility in general is enhanced. Further a dynamic adaption of the timeout threshold enables an optimized waiting time for a number of replies of responsive servers in each round. If the adaption is based on information concerning connection conditions or server conditions this may be used for adapting the timeout threshold: For example if a server usually has a low latency time for responding but during one round a load of the server is increasing, then the timeout may be increased in the next steps to ensure that a predefined number of additional replies is received in any case. Preferably latency may be used for determining a timeout threshold by analyzing historic latency times and then taking for example the timeout in such a way that in 95% the servers reply within that time period.
According to a further preferred embodiment adaption information is encoded in the replies from the servers. Therefore a writer may analyze the reply and extract the necessary information for adapting the timeout. Additional data traffic after the reply is avoided.
According to a further preferred embodiment the termination condition is fulfilled if either the reconstruction threshold for file of an actual round with regard to rounds of steps a)-e) is greater than or equal to the reconstruction threshold of the prior round or the number of rounds has exceeded the number of servers that might fail. This enables that in any case after t+1 rounds the original file is stored on the servers in such a way, that it can be reconstructed from the servers even if t servers have failed. If the reconstruction threshold of the actual round is greater than or equal to the reconstruction threshold then the file is stored on enough servers so that it can be correctly reconstructed by readers later on. Then the write operation is completed.
According to a further preferred embodiment the new file is generated based on the first one or more chunks of the prior non-sent remaining chunks. This enables a fast selection of chunks for the new file.
There are several ways how to design and further develop the teaching of the present invention in an advantageous way. To this end it is to be referred to the patent claims subordinate to patent claim 1 on the one hand and to the following explanation of preferred embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the preferred embodiments of the invention by the aid of the figure, generally preferred embodiments and further developments of the teaching will be explained. In the drawings
Fig. 1 shows a first embodiment of a method according to the present invention and
Fig. 2 shows a second embodiment of a method according to the present invention.
Fig. 1 shows a first embodiment of a method according to the present invention.
In Fig. 1 the storing of a file F using rateful codes is shown.
For storing the file F with rateful erasure coding a writer estimates the number of responsive servers r which is greater than or equal to t + b + 1 assuming that n = 2t + b + 1 servers among which at most b might be byzantine and the rest t - b may crash are provided. Further a set of clients is assumed leveraging them for sharing data via read/write functionality. Neither servers nor clients communicate with each other. Even further asynchronous transfer respectively communication is assumed in which no assumption is made on the time it takes to transmit a message between a client and a server. After the number of responsive servers r is estimated, the writer computes a reconstruction threshold m i such that m i = r - 1.
Then a sequence of k rounds with the following steps are performed, wherein 1 < k < t + 1.
The writer encodes the file F into n' = n + δι = 2t + b + 1 + mi - (b + 1 ) chunks such that mi of the generated chunks are sufficient to reconstruct the file F. In the following mi is the reconstruction threshold in the first round k = 1. The total number n' of generated chunks C-i, C2 is greater or equal than the total number of servers n. Therefore the chunks C-i, C2 can be divided into n main chunks Ci and n' - n = mi - (b + 1 ) auxiliary chunks C2. The blow-up factor BFi is then n/m-i.
The writer then selects the first n chunks C-i, i.e. the main chunks C-i, and sends them to the servers S-i, S2, Sn one to each server S-i, S2, Sn denoted with reference sign Wi and Wk for round k. Since some of the servers S-i, S2, S3, Sn may be unresponsive, the writer counts the number of replies r it receives, wherein the replies are indicated with reference sign Ri and Rk for round k. In order to avoid blocking by failed servers, the writer waits for an expiry of a predetermined time period, preferably after the writer has already received n - t replies from the servers S-i, S2, S3, Sn, wherein receiving of n-t replies is sufficient for reconstruction of the file F. After receiving the number n of replies in the first round k=1 then the reconstruction threshold nri2 for the next round k=2 is set to nri2 = n - t. This ensures that when the number of replies received from the servers S-i, S2, S3, Sn is greater than the estimated number of responsive servers, i.e. nri2≥ m-i, and the file F is stored at enough servers S-i, S2, S3, Sn, so that it can be correctly reconstructed by readers later. In this case the write operation is already completed.
If the number of replies n is smaller than the number of estimated replies r from the initial step then additional chunks are needed to be stored into the servers S-i, S2, S3, .... Sn for the file F to be recoverable.
These additional chunks are selected by taking the first chunks of the auxiliary chunks C21 , i.e. the first mi - nri2 chunks among the mi - (b + 1 ) constructed auxiliary chunks C2. The writer concatenates the first mi - nri2 auxiliary chunks C2 and proceeds to round k=2 with the concatenated chunks forming a new file Δ2 with smaller size than the original file F.
In round k=2 instead of the file F the generated file Δ2 consisting of the mi - nri2 concatenated auxiliary chunks C2 is encoded into n' = 2 t + nri2 chunks so that the file Δ2 can be reconstructed with nri2 chunks, wherein nri2 = n - t. These resulting chunks of the file Δ2 are smaller than the auxiliary chunks C2 of the preliminary round k=1.
These chunks of smaller side, i.e. the first n chunks of the generated n' chunks in the second round k=2 are then (re)sent to the n servers S-i , S2, S3, Sn one chunk to each server S-i , S2, S3, Sn. Similarily the writer then counts the number of replies Γ2 received. The reconstruction threshold for the next round nri3 is then set to nri3=r2 - 1.
After that, it is checked whether nri3 is greater than or equal to nri2 or nri3 ≥ nri2 respectively. If this terminating condition TC is fulfilled then F is finally stored on enough servers S-i , S2, S3, , Sn, so that it can be correctly reconstructed by readers later. If the terminating condition TC is not fulfilled, i.e. nri3 < nri2 then a further round k=3 is performed as long as either the maximum number of iterations k is reached, i.e. k = t + 1 or nrik≥ nrik-1. For example in the worst case mi is set to t + b + 1 and in the second step nri2 is set to mi - 1 , ... rrik = m-i - k - 1 . If k is equal to t + 1 then mt+i = mi - t = b+1 . Since n - t servers S-i, S2, S3, Sn are responsive, i.e. n - t replies are provided, the file F can be encoded such that it can be recovered from n - 2t = b + 1 chunks ensuring its reconstructability. Therefore in round k the file k is encoded into n'k = 2t + nrik chunks so it can be reconstructed with nrik chunks, n'k is always greater than or equal to n. The first chunks of the resulting chunks are then recent to the n servers S-i , S2, S3, Sn and the number of replies rk received is counted and nrik+ 1 is then set to rk - 1.
The terminating condition TC is fulfilled if nrik+i ≥ nrik: F is then stored at enough servers S-i , S2, S3, Sn, so that it can be correctly reconstructed later. On the other hand if nrik+1 < nrik then the further round k + 1 is performed. The reference sign δι , δ2, δ3, ... indicates the number of auxiliary chunks when dividing the file Ak in the respective round k.
Further in Fig. 1 the corresponding blow-up factors BFk in the corresponding rounds k are shown. For example in the first round k=1 the blow-up factor BFi is equal to n/m-i , in round k=2 the blow-up factor BF2 is (n/nri2) (mi - nri2) / m-i . Therefore the blow-up factor in round k is BFk = n/nrik (nrik-1 - nrik) / nrik-1. The resulting blow-up factor BF can be computed as the sum of the corresponding blow-up factors during each of the k rounds of the write. In detail it is the sum of the number of bits sent during each rounds of the write over IFI:
BF = BFi + BF2 + ... = n/mi + n/nri2 * (mi-nri2)/nni + n/nri3 * (m2-m3) m2
+ ... + n/mk * (mk-i-mk)/mk-i
= n/mk
Fig. 2 shows a second embodiment of a method according to the present invention.
In Fig. 2 a storing method using rateless erasure codes is shown. The steps performed in Fig. 2 are similar to the steps in Fig. 1 . However, instead of encoding the files Ak with reconstruction threshold nrik with n + 5k chunks Ci, C2 prior to sending the first n chunks to the servers S-i , S2, S3, Sn, the auxiliary chunks C2 for generating the file Ak are generated after receiving the number of replies r from the servers S-i , S2, S3, Sn, so that the auxiliary chunks C2 for the file Ak are only generated to the extent that they are necessary. This is the main difference to the method shown in Fig. 1 : In Fig. 1 in round k=1 the writer creates a set of auxiliary chunks C2 for the case that there are less available servers n than estimated denoted with r. If the estimate was correct, which is the common case, the resources spent to create these additional chunks C2 are wasted since they are not used for storing.
In Fig. 2 these auxiliary chunks C2 are created only if necessary: The additional chunks C2 are generated by performing an encoding procedure with the file Ak-i , the reconstruction threshold nrik - 1 and the number of chunks to be created rrik - 1 - nrik. These nrik-1 - nrik created new chunks C2 form then the file Ak for the next step. A further difference is, that this generated file Ak is then only encoded into n chunks corresponding to the number of servers. Therefore instead of performing an encoding procedure Encode(Ak, nrik, n + 5k) in each round k the encoding procedure according to Fig. 2 is to produce the chunks for sending them to the servers S-i , S2, Sn as follows: Encode {Ak, nrik, n). The resulting blow-up factor BF corresponds to the blow-up factor of Fig. 1 and amounts exactly n/nrik in total.
In summary the present invention leverages synchrony in order to construct an asynchronous distributed storage protocol with an optimal communication blow-up factor. The present invention further recursively applies erasure coding on data parameters by a number of received replies as a reconstruction threshold.
Further the present invention minimizes the overhead with respect to information exchanged over the network in existing asynchronous distributed storage protocols. The present invention further applies to generic erasure coding schemes and is not particularly restricted to a specific type of erasure coding technique, in particular applies to both crash and byzantine models.
The present invention has inter alia the following advantages: The present invention provides an optimal blow-up factor with respect to the amount of information sent over a communication channel. The present invention may further be used in conjunction with both rateful and rateless codes. Even further the present invention applies in scenarios featuring crash-only servers, byzantine servers or a combination of crash-only and byzantine servers.
Many modifications and other embodiments of the invention set forth herein will come to mind the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
Claims
1. A method for storing a file (F) on a plurality of servers (Si, S2, Sn), wherein the number of servers (Si, S2, Sn) is n and the maximum number of servers (Si, S2, Sn) which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers (S-i, S2,
Sn), and wherein n equals 2t+b+1 ,
characterized by the steps of
a) Dividing the file (F, Δ-ι) into a plurality of chunks (Ci, C2), wherein the number of chunks (n + δι , n) is equal to or greater than the number of servers n,
b) Sending n chunks (Ci) of the file (F) to the n servers (Si, S2, Sn), wherein one chunk (Ci) is sent to each server (S-i, S2, Sn) c) Determining the number of replies r from the n servers (S-i, S2, Sn) indicating successful storage of the respective chunks (C-i), d) Checking if the number of replies r matches a terminating condition (TC), and if not
e) Generating a new file (Δ2) based on one or more chunks (C-i, C2) of the old file (F), a reconstruction threshold (m-i, nri2, ...) of the old file (F) and the number of replies r,
f) Perform steps a)-e) with the new file (Δ2) as file for these steps, until the terminating condition in step d) is fulfilled, wherein the terminating condition (TC) is based on the difference between the reconstruction thresholds of the new file (Δ-ι, Δ2,..., Δκ) of step e) and the old file (F) of step a) and the maximum number of servers t which might be fail.
2. The method according to claim 1 , characterized in that in step a) when the total number of generated chunks (C-i, C2) is greater than the number of servers n, the number of chunks (δ-ι) generated in addition to the n chunks (C-i) is dependent on the number of servers (t) which might be fail and a reconstruction threshold (m-i, nri2, ...) for the file (Δ-ι, Δ2).
3. The method according to one of the claims 1 -2, characterized in that for the first performing of step a) the reconstruction threshold (m-i) is based on an estimated number of responsive servers (Si, S2, Sn).
4. The method according to claim 3, characterized in that the estimated number is greater than a sum of byzantine servers b and servers t-b that might crash.
5. The method according to one of the claims 2-4, characterized in that the total number of chunks in step a) is
2t + th,
wherein t is the maximum number of servers that might be fail and th is the reconstruction threshold for the corresponding file (F).
6. The method according to one of the claims 1 -5, characterized in that the one or more chunks (C-i) for generation of the new file (Δ2) are solely based on one or more of the prior non-sent remaining chunks (C2).
7. The method according to claim 1 , characterized in that chunks (C2) in addition to the generated n chunks (C-i) are only generated to the extent that the termination condition (TC) is not matched.
8. The method according to one of the claims 1 -7, characterized in that the generation of the new file (Δ2, Δ3, ... ) is performed by concatenating one or more chunks (C2).
9. The method according to one of the claims 1 -8, characterized in that a timeout threshold is used when determining the number of replies according to step c), preferably wherein the timeout threshold starts after a predetermined number of received replies, preferably wherein the predetermined number corresponds to the number of replies needed for reconstruction of the file (F).
10. The method according to claim 9, characterized in that the timeout threshold is dynamically adapted in each round of steps a)-e), preferably wherein the adaption is based on connection conditions between a writer of the file and the servers (S-i, S2, Sn) and/or of server conditions.
1 1. The method according to claim 10, characterized in that adaption information is encoded in the replies from the servers (S-i , S2, Sn).
12. The method according to one of the claims 1 -1 1 , characterized in that the termination condition (TC) is fulfilled if either the reconstruction threshold (nri2, nri3) for a file (F , Δ-ι) of an actual round with regard to rounds of steps a)-e) is greater than or equal to the reconstruction threshold (m-i , nri2, ...) of the prior round or the number of rounds has exceeded the number of servers (t) that might be fail.
13. The method according to one of the claims 1 -12, characterized in that the new file (Δ2, Δ3) is generated based on the first one or more chunks (C21) of the prior non-sent remaining chunks (C2).
14. A system for storing a file (F) on a plurality of servers (S-i, S2, Sn), wherein the number of servers (S-i, S2, Sn) is n and the maximum number of servers (S-i, S2, Sn) which might be fail is t, preferably including a predefined number b of byzantine failures and a number t-b of crashes of the servers (S-i , S2, Sn), and wherein n equals 2t+b+1 , comprising the plurality of servers (S-i , S2, Sn) and a writer for writing the file onto the servers (S-i, S2, Sn), preferably for performing with a method according to one of the claims 1 -13,
characterized by
dividing means, preferably a writer, operable to divide the file (F, Δ-ι) into a plurality of chunks (C-i, C2), wherein the number of chunks (n + δ-ι , n) is equal to or greater than the number of servers n, sending means, preferably the writer, operable to send n chunks of the file (F) to the n servers (S-i, S2, Sn), wherein one chunk (C-i , C2) is sent to each server (S-i , S2, Sn),
determining means, preferably the writer, operable to determine the number of replies r from the n servers (S-i , S2, Sn) indicating successful storage of the respective chunks (C-i, C2),
checking means, preferably the writer, operable to check if the number of replies r matches a terminating condition (TC),
generating means, preferably the writer, operable to generate a new file (Δ2, Δ3) based on one or more chunks (C-i , C2) of the old file (F, Δ-ι) , a reconstruction threshold (mi , m2, ...) of the old file and the number of replies r,
and recursive means operable to operate recursively the dividing means, the sending means, the determining means, the checking means and the generation means with the new file (Δ2) as file, until the terminating condition is fulfilled, wherein the terminating condition (TC) is based on the difference between the reconstruction thresholds of the new file (Δ2, .. . , Δκ) and the old file (F) and the maximum number of servers t which might be fail.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP13794823.8A EP2842037B1 (en) | 2012-10-19 | 2013-10-18 | Method and system for storing a file on a plurality of servers |
US14/419,700 US10055427B2 (en) | 2012-10-19 | 2013-10-18 | Method and system for storing a file on a plurality of servers |
US16/027,447 US10445297B2 (en) | 2012-10-19 | 2018-07-05 | Method and system for storing a file on a plurality of servers |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP12189208.7 | 2012-10-19 | ||
EP12189208 | 2012-10-19 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/419,700 A-371-Of-International US10055427B2 (en) | 2012-10-19 | 2013-10-18 | Method and system for storing a file on a plurality of servers |
US16/027,447 Continuation US10445297B2 (en) | 2012-10-19 | 2018-07-05 | Method and system for storing a file on a plurality of servers |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2014060572A2 true WO2014060572A2 (en) | 2014-04-24 |
WO2014060572A3 WO2014060572A3 (en) | 2014-06-12 |
Family
ID=49626903
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/EP2013/071823 WO2014060572A2 (en) | 2012-10-19 | 2013-10-18 | Method and system for storing a file on a plurality of servers |
Country Status (3)
Country | Link |
---|---|
US (2) | US10055427B2 (en) |
EP (1) | EP2842037B1 (en) |
WO (1) | WO2014060572A2 (en) |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8572429B2 (en) * | 2007-10-09 | 2013-10-29 | Cleversafe, Inc. | Optimistic data writing in a dispersed storage network |
-
2013
- 2013-10-18 WO PCT/EP2013/071823 patent/WO2014060572A2/en active Application Filing
- 2013-10-18 US US14/419,700 patent/US10055427B2/en not_active Expired - Fee Related
- 2013-10-18 EP EP13794823.8A patent/EP2842037B1/en not_active Not-in-force
-
2018
- 2018-07-05 US US16/027,447 patent/US10445297B2/en not_active Expired - Fee Related
Non-Patent Citations (2)
Title |
---|
JAMES HENDRICKS; GREGORY R. GANGER; MICHAEL K. REITER: "Low- overhead byzantine fault-tolerant storage", PROCEEDINGS OF TWENTY-FIRST ACM SIGOPS SYMPOSIUM ON OPERATING SYSTEMS PRINCIPLES (SOSP '07, 2007 |
P. MAYMOUNKOV: "Online Codes", TECHNICAL REPORT TR2002-833, TECHNICAL REPORT, NEW YORK UNIVERSITY, 2002 |
Also Published As
Publication number | Publication date |
---|---|
US10055427B2 (en) | 2018-08-21 |
EP2842037A2 (en) | 2015-03-04 |
EP2842037B1 (en) | 2018-05-23 |
US20180357256A1 (en) | 2018-12-13 |
WO2014060572A3 (en) | 2014-06-12 |
US10445297B2 (en) | 2019-10-15 |
US20150220562A1 (en) | 2015-08-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6649340B2 (en) | Data storage method, data recovery method, related devices and systems | |
US7266716B2 (en) | Method and recovery of data using erasure coded data from stripe blocks | |
CN108170555B (en) | Data recovery method and equipment | |
EP3014451B1 (en) | Locally generated simple erasure codes | |
US7310703B2 (en) | Methods of reading and writing data | |
US7284088B2 (en) | Methods of reading and writing data | |
US8060772B2 (en) | Storage redundant array of independent drives | |
JP5852674B2 (en) | Method for detecting data loss in data transfer between information devices | |
CN102866935B (en) | Instant replication method and storage system based on ISCSI (internet small computer system interface) | |
KR20120112981A (en) | Receiving node and method of reducing retransmission of data frame | |
US20170308332A1 (en) | Method for writing data into storage system and storage system | |
EP4075252A1 (en) | Data storage method, apparatus, and system and data reading method, apparatus, and system in storage system | |
US10445297B2 (en) | Method and system for storing a file on a plurality of servers | |
CN105117303A (en) | Data recovery method and equipment | |
US20160080111A1 (en) | Receiver, transmitter and data transmission system | |
CN114595092B (en) | Distributed storage system, data reconstruction method, device and storage medium | |
CN110431538B (en) | Storage system control | |
CN110083478B (en) | Data recovery method, data recovery system and solid state disk | |
US20240134739A1 (en) | Method and apparatus for storing blockchain data based on error correction code | |
CN117992281A (en) | Disaster recovery backup method, system and electronic equipment | |
US11409603B2 (en) | Method of storing items of data | |
KR20240056400A (en) | Method and apparatus for storing blockchain data based on error correction coding | |
CN109901953B (en) | Data reconstruction method and device | |
CN117806558A (en) | Data protection method, storage array controller and storage device | |
CN117494161A (en) | Mobile hard disk encryption/decryption method, system and medium capable of recovering error data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13794823 Country of ref document: EP Kind code of ref document: A2 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2013794823 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14419700 Country of ref document: US |