CN115756947A - Data consistency verification method, device and system of distributed data storage system - Google Patents
Data consistency verification method, device and system of distributed data storage system Download PDFInfo
- Publication number
- CN115756947A CN115756947A CN202211492639.7A CN202211492639A CN115756947A CN 115756947 A CN115756947 A CN 115756947A CN 202211492639 A CN202211492639 A CN 202211492639A CN 115756947 A CN115756947 A CN 115756947A
- Authority
- CN
- China
- Prior art keywords
- data
- check
- blocks
- block
- consistency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Detection And Correction Of Errors (AREA)
Abstract
The embodiment of the specification provides a data consistency checking method, a device and a system for a distributed data storage system. When consistency check is carried out, acquiring the total data stored in the distributed data storage system, wherein the acquired total data comprises data blocks stored in each data storage device and check blocks stored in each data check device, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions when data storage is carried out. Subsequently, checking consistency of the check blocks by using all the acquired data blocks; and using part or all of the stored check blocks and part of the data blocks to perform data block consistency check. Then, a data consistency check result for the distributed data storage system is determined based on the check block consistency check result and/or the data block consistency check result.
Description
Technical Field
Embodiments of the present disclosure relate generally to the field of data storage, and more particularly, to a method and an apparatus for performing data consistency check on a distributed data storage system, and a distributed data storage system.
Background
In distributed storage systems, multiple copy redundancy mechanisms are often employed to ensure data availability. HDFS, such as Hadoop, typically employs a three-copy mechanism, where a first copy is stored on a data storage node in a local chassis, another copy is stored on another data storage node in the same chassis, and a third copy is stored on a data storage node in a different chassis. The three-copy mechanism can tolerate two copies loss at most without data loss. However, the three-copy mechanism would cause an additional overhead of 200% of the storage space and would also cause waste on other resources, such as extra bandwidth consumption when writing data. In addition, for example, for cold data, because the query frequency is low, the second copy and the third copy are rarely accessed, but still occupy the same amount of storage space, thereby causing a waste of storage space.
Therefore, for some scenarios that require reduced storage cost (e.g., cold data storage), erasure Coding (EC) technology is usually adopted to implement data redundancy while reducing the storage cost of multiple copies. The EC code is a coding fault-tolerant technology, and makes each data block generate relevance by partitioning data and calculating a check block. When part of the data blocks are lost, the lost data blocks can be recovered by the remaining data blocks and the check blocks.
In order to prevent data loss caused by software bug or storage device failure in the distributed data storage system, data consistency check is usually performed on the distributed data storage system periodically, that is, the whole data is scanned periodically in the background to check the correctness of each data block.
Disclosure of Invention
The embodiment of the specification provides a method and a device for carrying out data consistency check on a distributed data storage system and the distributed data storage system. In the method and the device for checking the data consistency of the distributed data storage system, check block consistency check is performed by using all the data blocks, and data block consistency check is performed by using part or all of the stored check blocks and part of the stored data blocks, so that the check block consistency check and the data block consistency check are combined to realize the data consistency check for the distributed data storage system.
According to an aspect of an embodiment of the present specification, there is provided a method for performing data consistency check on a distributed data storage system, including: acquiring full data stored in a distributed data storage system, wherein the distributed data storage system comprises at least two data storage devices and at least two data check devices, the full data comprises data blocks stored in each data storage device and check blocks stored in each data check device, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions during data storage; checking consistency of the check block by using all the acquired data blocks; using part or all of the stored check blocks and part of the data blocks to carry out data block consistency check; and determining a data consistency check result for the distributed data storage system based on the check block consistency check result and/or the data block consistency check result.
Optionally, in an example of the above aspect, the determining a data consistency check result for the distributed data storage system based on the check block consistency check result and/or the data block consistency check result may include: determining that a data consistency check for the distributed data storage system fails in response to the parity check block consistency check result indicating that the parity check block consistency check fails or the data block consistency check result indicating that the data block consistency check fails, or determining that a data consistency check for the distributed data storage system succeeds in response to the parity check block consistency check result indicating that the parity check block consistency check succeeds and the data block consistency check result indicating that the data block consistency check succeeds.
Optionally, in an example of the above aspect, checking for consistency of the chunk using all the obtained data chunks may include: and performing check block consistency check once by using all the acquired data blocks. In addition, performing a data chunk consistency check using some or all of the stored parity chunks and partial data chunks may include: and performing data block consistency check once by using part or all of the stored check blocks and part of the data blocks.
Optionally, in an example of the above aspect, the checking once the consistency check of the chunk using all the obtained data chunks may include: performing check block calculation by using all the acquired data blocks to calculate any one of the check blocks; and checking consistency of the check blocks once according to the calculated check blocks and the stored corresponding check blocks.
Optionally, in an example of the above aspect, performing a data chunk consistency check once using part or all of the stored parity chunks and partial data chunks may include: and performing data block consistency check once by using all other check blocks except the check block which completes the consistency check of the check blocks and part of the data blocks in the stored check blocks.
Optionally, in an example of the above aspect, performing a data block consistency check using all other parity blocks of the stored parity blocks except for the parity block for which the parity block consistency check is completed and the partial data block may include: performing redundant decoding by using all other check blocks and partial data blocks except the check block which completes the consistency check of the check blocks in the stored check blocks to recover any one data block in the data blocks which do not participate in the redundant decoding; and performing a data block consistency check once according to the recovered data block and the stored corresponding data block.
Optionally, in an example of the above aspect, the redundancy coding includes redundancy coding based on an RS code, and a difference between the number of the data check devices and the number of the data storage devices is not greater than 2.
Optionally, in an example of the above aspect, a total number of check blocks and data blocks used in the data block consistency check is equal to a total number stored by the data storage device.
Optionally, in an example of the above aspect, the check block consistency check and the data block consistency check may be performed sequentially or in parallel.
Optionally, in one example of the above aspect, the method is performed by one of the at least two data storage devices, the at least two data verification devices, and a third party device.
According to another aspect of embodiments of the present specification, there is provided an apparatus for performing data consistency check on a distributed data storage system, including: the system comprises a full data acquisition unit, a full data acquisition unit and a data storage unit, wherein the full data acquisition unit is used for acquiring full data stored in a distributed data storage system, the distributed data storage system comprises at least two data storage devices and at least two data check devices, the full data comprises data blocks stored by the data storage devices and check blocks stored by the data check devices, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions; the check block checking unit is used for checking the consistency of the check blocks by using all the acquired data blocks; the data block checking unit is used for checking the consistency of the data blocks by using part or all of the stored check blocks and part of the data blocks; and the consistency checking unit is used for determining a data consistency checking result aiming at the distributed data storage system based on a checking block consistency checking result and/or a data block consistency checking result.
Optionally, in one example of the above aspect, the consistency check unit determines that the data consistency check for the distributed data storage system fails in response to the check block consistency check result indicating that the check block consistency check fails or the data block consistency check result indicating that the data block consistency check fails. In response to the check block consistency check result indicating that the check block consistency check is successful and the data block consistency check result indicating that the data block consistency check is successful, the consistency check unit determines that data consistency check for the distributed data storage system is successful.
Optionally, in an example of the above aspect, the check block checking unit performs check block consistency check once using all the acquired data blocks, and/or the data block checking unit performs check block consistency check once using part or all of the stored check blocks and part of the data blocks.
Optionally, in an example of the above aspect, the check block checking unit may include: the check block calculation module is used for calculating the check blocks by using all the acquired data blocks so as to calculate any one of the check blocks; and the check block checking module is used for checking the consistency of the check blocks once according to the calculated check blocks and the stored corresponding check blocks.
Alternatively, in one example of the above aspect, the data block checking unit may perform the data block consistency check once using all other parity blocks of the stored parity blocks except for the parity block for which the parity block consistency check is completed and a part of the data blocks.
Optionally, in an example of the above aspect, the data block checking unit may include: the data block recovery module is used for carrying out redundancy decoding on all other check blocks and part of data blocks except the check block which completes the consistency check of the check blocks in the stored check blocks so as to recover any one data block in the data blocks which do not participate in the redundancy decoding; and the data block checking module is used for executing data block consistency check once according to the recovered data block and the stored corresponding data block.
According to another aspect of embodiments herein, there is provided a distributed data storage system comprising: at least two data storage devices, each storing a respective data block; each data checking device stores a checking block obtained by respectively using different redundancy coding functions to perform redundancy coding on all stored original data blocks during data storage; and means for performing a data consistency check on the distributed data storage system as described above.
According to another aspect of embodiments of the present specification, there is provided an apparatus for performing data consistency check on a distributed data storage system, including: at least one processor, a memory coupled with the at least one processor, and a computer program stored in the memory, the at least one processor executing the computer program to implement the method for data consistency checking for a distributed data storage system as described above.
According to another aspect of embodiments herein, there is provided a computer readable storage medium storing executable instructions that when executed cause a processor to perform a method for data consistency checking for a distributed data storage system as described above.
According to another aspect of embodiments of the present specification, there is provided a computer program product comprising a computer program for execution by a processor to implement the method for data consistency checking for a distributed data storage system as described above.
Drawings
A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the drawings, similar components or features may have the same reference numerals.
FIG. 1 shows a block schematic diagram of a distributed data storage system according to an embodiment of the present description.
FIG. 2 illustrates an example flow diagram of a method for data consistency checking for a distributed data storage system in accordance with an embodiment of this specification.
Fig. 3A, 3B, and 3C illustrate example schematics of a comparison of operational processes between a data consistency check method according to embodiments of the present description and the prior art.
FIG. 4 illustrates an example block diagram of a data consistency check apparatus in accordance with an embodiment of this specification.
FIG. 5 illustrates an example block diagram of a parity chunk check unit, according to an embodiment of this specification.
FIG. 6 illustrates an example block diagram of a data block checking unit in accordance with an embodiment of this specification.
FIG. 7 illustrates an example schematic diagram of a computer-system-implemented data consistency check apparatus in accordance with an embodiment of the present description.
Detailed Description
The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand the subject matter described herein and are not intended to limit the scope, applicability, or examples set forth in the claims. Changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as needed. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with respect to some examples may also be combined in other examples.
As used herein, the term "include" and its variants mean open-ended terms in the sense of "including, but not limited to. The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first," "second," and the like may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. The definition of a term is consistent throughout the specification unless the context clearly dictates otherwise.
For a distributed data storage system with n data storage devices and m data verification devices, a common implementation scheme is to read out all n original data blocks and recalculate m verification blocks based on the n original data blocks read out when performing data consistency verification. And then, comparing the calculated m check blocks with corresponding check blocks stored in the distributed data storage system respectively, and if the m check blocks are inconsistent, indicating that data damage occurs. In the implementation scheme, the data consistency check can be completed only by m times of coding operation, so that the operation amount of the data consistency check is high.
Another common solution is to store an additional check value (e.g., CRC) for each data block and each check block, which may be stored with the original data block/check block. When data consistency check is performed, all original data blocks and check blocks need to be read, check values of each data block/check block are recalculated, and whether the check values are consistent or not is compared. In the implementation scheme, the data consistency check can be completed only by m + n times of CRC check calculation, so that the operation amount of the data consistency check is high.
In view of this, the present specification provides a method, an apparatus and a distributed data storage system for data consistency check of the distributed data storage system. In the method and the device for checking data consistency of the distributed data storage system, the check block consistency check is executed by using all the data blocks and the data block consistency check is executed by using part or all of the stored check blocks and part of the stored data blocks, so that the check block consistency check and the data block consistency check are combined to realize the data consistency check for the distributed data storage system.
A method, an apparatus, and a distributed data storage system for data consistency check of the distributed data storage system according to embodiments of the present specification are described below with reference to the accompanying drawings.
FIG. 1 illustrates a block schematic diagram of a distributed data storage system 100 in accordance with embodiments of the present description.
As shown in FIG. 1, the distributed data storage system 100 includes at least two data storage devices 110-1 to 110-n, at least two data verification devices 120-1 to 120-m, and a data consistency check apparatus 130. The at least two data storage devices 110-1 to 110-n, the at least two data verification devices 120-1 to 120-m, and the data consistency check apparatus 130 may communicate with each other via the network 140 for data transfer. In some embodiments, the network 140 may be any one or more of a wired network or a wireless network. Examples of network 140 may include, but are not limited to, a cable network, a fiber optic network, a telecommunications network, an intranet, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a bluetooth network, a zigbee network (zigbee), near Field Communication (NFC), an intra-device bus, an intra-device line, and the like, or any combination thereof. In some embodiments, direct communication may be possible between the at least two data storage devices 110-1 to 110-n, the at least two data verification devices 120-1 to 120-m, and the data consistency check apparatus 130 without the network 140.
In distributed data storage, each of the n data storage devices stores one original data block, and stores n original data blocks in total. Each data verification device 120 may have a redundancy coding algorithm (redundancy coding function F) for calculating the check block, and the redundancy coding functions of the respective data verification devices 120 are different from each other. For example, data check device 120-1 may have a redundancy coding function F1, and data check device 120-m may have a redundancy coding function Fm. In this specification, the redundancy coding used may include erasure codes. Examples of erasure codes may include, for example, but are not limited to, RS (Reed-Solomon) codes.
During data storage, each data storage device sends the respective stored original data block to each data verification device 120. After receiving n original data blocks, each data checking device 120 redundantly encodes the received n original data blocks using a respective redundancy coding function, thereby obtaining respective check blocks and performing local storage.
After the data storage is performed as above, in order to prevent data loss caused by a software bug or a storage device failure occurring in the distributed data storage system 100, the data consistency check apparatus 130 typically performs a data consistency check on the distributed data storage system periodically, that is, periodically scans the entire amount of data in the background to perform the data consistency check. In some embodiments, the data consistency check apparatus 130 may be deployed in any one of the at least two data storage devices, in any one of the at least two data check devices, or in a third party device.
FIG. 2 illustrates an example flow diagram of a method 200 for data consistency checking for a distributed data storage system in accordance with embodiments of the present description.
As shown in FIG. 2, at 210, the full amount of data stored in the distributed data storage system is obtained. The acquired full data comprises data blocks stored by the data storage devices and check blocks stored by the data check devices. The check block of each data check device is obtained by respectively using different redundancy coding functions to perform redundancy coding on all stored original data blocks during data storage.
The redundancy coding process according to the embodiment of the present specification is explained below by taking an RS code as an example. The RS code uses an encoding algorithm based on finite Field operations, also known as Galois Field (GF Field).
If the data block stored in the data storage device is represented as a vector D and the check block stored in the data check device is represented as a vector C, the coding function adopted by the data check device is represented as a matrix F, and each data check device i has the coding function F i As a row of elements in the matrix F, the check block can be calculated according to the expression FD = C.
Assuming that the data storage system has n data storage devices and m data verification devices, the encoding function matrix F may be defined as an m n Vandermonde matrix, where F i,j ＝j i-1 The above expression can thus be modified to:
in some embodiments, the data consistency checking means may be disposed in one of the at least two data storage devices. In this case, the data consistency check apparatus may acquire the stored data blocks from the other data storage devices of the at least two data storage devices and the stored check blocks from the respective data check devices, and compose the full amount of data together with the data blocks stored by itself.
In some embodiments, the data consistency check apparatus may be disposed in one of the at least two data check devices. In this case, the data consistency check apparatus may acquire the stored check blocks from the other data check device of the at least two data check devices and acquire the stored data blocks from the respective data storage devices, and compose the full amount of data together with the check blocks stored by itself.
In some embodiments, the data consistency check apparatus may be deployed in a third party device. In this case, the data consistency check apparatus may acquire the stored data blocks from the respective data storage devices and the stored check blocks from the respective data check devices, thereby obtaining the full amount of data.
After the full amount of data is acquired as above, a check block consistency check is performed at 220 using all of the acquired data blocks. In some embodiments, multiple check block consistency checks may be performed using all of the retrieved data blocks. In some embodiments, the check block consistency check may be performed only once using all of the retrieved data blocks.
In some embodiments, parity block calculations may be performed using all of the acquired data blocks to calculate at least one parity block. For example, a corresponding redundancy coding function (e.g., any of the RS coding functions F described above) used in calculating parity blocks for data storage may be used i ) And performing redundancy coding on all the acquired data blocks according to the coding mode to calculate a check block c i ′ . It is noted that the calculated parity chunks may correspond to any one or more of the stored parity chunks.
In a case that the data consistency check apparatus is deployed in one of the at least two data storage devices, the data consistency check apparatus may obtain a corresponding redundancy coding function from the one of the at least two data check devices to perform check block calculation. In addition, the data consistency check device may also store all the redundant coding functions in advance, and then select the corresponding redundant coding function to perform check block calculation.
The data consistency checking device is deployed in one data checking device of at least two data checking devices. In this case, the data consistency check apparatus may obtain a corresponding redundant coding function from one of the at least two data check devices to perform check block calculation. In addition, the data consistency check device may also store all the redundant coding functions in advance, and then select the corresponding redundant coding function to perform check block calculation. If check block consistency check is performed only once, the data consistency check apparatus may perform check block calculation using a redundancy coding function that itself has.
In a case that the data consistency check apparatus is deployed in the third-party device, the data consistency check apparatus may obtain a corresponding redundancy coding function from one of the at least two data check devices to perform check block calculation. In addition, the data consistency check device may also store all the redundant coding functions in advance, and then arbitrarily select the corresponding redundant coding function to perform check block calculation.
After the check blocks are calculated as above, check block consistency check is performed according to the calculated check blocks and the stored corresponding check blocks. And if all the calculated check blocks are consistent with the stored corresponding check blocks, the check block consistency check is considered to be successful. And if at least one inconsistency exists between each calculated check block and the corresponding stored check block, the check block consistency check is considered to fail.
At 230, a data chunk consistency check is performed using some or all of the stored parity chunks and some of the stored data chunks. In some embodiments, multiple data chunk consistency checks may be performed using some or all of the stored parity chunks and portions of the stored data chunks. In some embodiments, a data chunk consistency check may be performed once using some or all of the stored parity chunks and some of the stored data chunks.
In some embodiments, a data chunk consistency check may be performed once using all other parity chunks and partial data chunks of the stored parity chunks except for the parity chunk that completed the consistency check. In some embodiments, the total number of parity chunks and data chunks used by the data chunk consistency check process may be equal to the total number stored by the data storage device. For example, assume that there are n data storage devices and m data verification devices, the data blocks stored by the n data storage devices are D1 to Dn, respectively, and the check blocks stored by the m data verification devices are C1 to Cm, respectively, and the check block C is calculated in step 220 using all the data blocks D1 to Dn 1 ′ . In performing the data block consistency check, the data block consistency check may be performed using the remaining m-1 check codes C2 through Cm and n-m +1 data blocks.
In some embodiments, some or all of the stored parity blocks and some of the stored data blocks may be used for redundancy decoding to recover at least one of the data blocks not participating in redundancy decoding. The above-described redundancy decoding process may be performed in a manner corresponding to the redundancy encoding process.
To describe the above redundant decoding process, the matrix A is defined as
Wherein I is an identity matrix, F is the redundant coding function matrix, and a vector E is defined as
Where D is the data block vector and C is the check block vector, resulting in the equation AD = E, i.e.,
each of the data storage device and the data verification device has a corresponding row of matrix a and vector E. When data in the device is lost, it may be reflected by removing the corresponding row of the device from matrix A and vector E, thereby resulting in matrix A ′ Sum vector E ′ And satisfies the following equation: a. The ′ D＝E ′ 。
Assuming there are m devices failing, matrix A ′ Is an n × n matrix. Since matrix F is defined as a Vandermonde matrix, it is possible to ensure that a subset of the matrix rows of matrix a are linearly independent. Thus, matrix A ′ Is a non-singular matrix such that a Gaussian elimination method can be used to derive from matrix A ′ The value of D is calculated so that the data blocks stored at all data storage devices can be recovered.
After the data blocks are recovered as above, a data block consistency check is performed based on each of the recovered data blocks and the corresponding stored data block. And if the recovered data blocks are consistent with the stored corresponding data blocks, the data block consistency check is considered to be successful. If at least one inconsistency exists between each of the recovered data blocks and the corresponding stored data block, the data block consistency check is deemed to have failed.
In some embodiments, the data block consistency check may be performed once using all other check blocks of the stored check blocks except the check block that completes the check block consistency check, and the partial data block. For example, all other check blocks except the check block which completes the check of check block consistency and partial data blocks in the stored check blocks can be used for redundancy decoding to recover any one data block in the data blocks which do not participate in redundancy decoding. Then, a data block consistency check is performed once according to the recovered data block and the stored corresponding data block.
At 240, a data consistency check result for the distributed data storage system is determined based on the check block consistency check result and/or the data block consistency check result. Determining that the data consistency check for the distributed data storage system fails if the check block consistency check result indicates that the check block consistency check fails or the data block consistency check result indicates that the data block consistency check fails. And if the check block consistency check result indicates that the check block consistency check is successful and the data block consistency check result indicates that the data block consistency check is successful, determining that the data consistency check for the distributed data storage system is successful.
Optionally, in some embodiments, for the RS code-based data consistency check scheme, the number m of the data check devices may be set to be not greater than 2, that is, n +2 is greater than or equal to m, so as to ensure that the recovery and checking of all the data blocks and check blocks can be covered by the one-time check block consistency check process and the one-time data block consistency check process, thereby completing the data consistency check process of the distributed data storage system.
Fig. 3A, 3B, and 3C illustrate example diagrams of comparison of operational processes between a data consistency check method according to an embodiment of the present specification and the prior art. In the schemes shown in fig. 3A, 3B, and 3C, the distributed data storage device includes 8 data storage devices and 4 data verification devices.
In the existing data consistency check scheme shown in fig. 3A, all 8 original data blocks need to be read out, and 4 check blocks C0', C1', C2', and C3' need to be recalculated based on the 8 original data blocks D0, D1, D2, D3, D4, D5, D6, and D7 that are read out. And then, comparing the calculated 4 check blocks with corresponding check blocks stored in the distributed data storage system respectively, and if the check blocks are inconsistent, indicating that data damage occurs. In this implementation, 4 encoding operations are required to complete the data consistency check.
In the data consistency check scheme according to the embodiment of the present specification shown in fig. 3B, after all 8 original data blocks D0, D1, D2, D3, D4, D5, D6, and D7 and 4 check blocks C0, C1, C2, and C3 are read out, first, a check block C3 'is calculated using the read out 8 original data blocks D0, D1, D2, D3, D4, D5, D6, and D7, and then the calculated check block C3' is compared with the stored check block C3 for consistency. If the data is inconsistent, the data consistency check fails. If the data is consistent, data block D0 'may be recovered using the read parity blocks C0, C1, C2, C3 and data blocks D4, D5, D6, and D7, and the recovered data block D0' may be compared for consistency with the stored data block D0. If the data is inconsistent, the data consistency check fails. If the data are consistent, the data consistency verification is successful, and data loss or data tampering does not occur.
In the data consistency check scheme according to the embodiment of the present specification shown in fig. 3C, after all 8 original data blocks D0, D1, D2, D3, D4, D5, D6, and D7 and 4 check blocks C0, C1, C2, and C3 are read out, first, a check block C3 'is calculated using the read out 8 original data blocks D0, D1, D2, D3, D4, D5, D6, and D7, and then the calculated check block C3' is compared with the stored check block C3 for consistency. If the data is inconsistent, the data consistency check fails. If the data is consistent, data block D0 'may be recovered using the read parity blocks C0, C1, C2 and data blocks D3, D4, D5, D6, and D7, and the recovered data block D0' may be compared for consistency with the stored data block D0. If the data is inconsistent, the data consistency check fails. And if the data are consistent, the data consistency is successfully checked, and data loss or data tampering does not occur.
It is noted that in some embodiments, the check block consistency check and the data block consistency check may be performed sequentially or in parallel. For example, a check for chunk consistency check may be performed first, followed by a data chunk consistency check. Alternatively, the data chunk consistency check may be performed first, followed by a check chunk consistency check. In this case, as long as the result of the consistency check performed first indicates that the consistency check failed, it is determined that the data consistency check for the distributed data storage system failed, and the subsequent process need not be performed. In some embodiments, the check block consistency check and the data block consistency check may be performed in parallel.
In the above-described data consistency check scheme, the parity chunk consistency check is performed using all of the data chunks and the data chunk consistency check is performed using some or all of the stored parity chunks and some of the stored data chunks, thereby combining the parity chunk consistency check with the data chunk consistency check to implement the data consistency check for the distributed data storage system.
By using the data consistency check scheme, the consistency check of the check blocks is performed once by using all the data blocks, and the consistency check of the data blocks is performed once by using part or all of the stored check blocks and part of the stored data blocks, so that the data consistency check of the distributed data storage system is realized, the calculation amount during the data consistency check can be reduced, and the data consistency check of the distributed data storage system is accelerated.
By using the data consistency check scheme, the consistency check of the check block is performed once by using all the data blocks, and the consistency check of the data blocks is performed once by using all the remaining check blocks except the check block which completes the consistency check in the stored check blocks and part of the stored data blocks, so that the data consistency check of the distributed data storage system is realized, the recovery and the check of all the data blocks and the check blocks can be covered by using the consistency check process of the check blocks and the consistency check process of the data blocks, and the data consistency check process of the distributed data storage system is further completed.
By using the data consistency checking scheme, the difference between the number m of the data checking devices and the number n of the data storage devices is set to be not more than 2, namely n +2 is not less than or equal to m, so that the recovery and checking of all the data blocks and the check blocks can be covered by using the one-time check block consistency checking process and the one-time data block consistency checking process, and the data consistency checking process of the distributed data storage system is further completed.
FIG. 4 illustrates an example block diagram of a data consistency check apparatus 400 in accordance with an embodiment of this specification. As shown in fig. 4, the data consistency check apparatus 400 includes a full data acquisition unit 410, a check block check unit 420, a data block check unit 430, and a consistency check unit 440.
The full data obtaining unit 410 is configured to obtain full data stored in the distributed data storage system, where the obtained full data includes data blocks stored by each data storage device and check blocks stored by each data check device. The operation of the full data acquisition unit 410 may refer to the operation described above with reference to 210 of fig. 2.
The check block checking unit 420 is configured to perform check block consistency checking using all the acquired data blocks. The operation of the check block checking unit 420 may refer to the operation described above with reference to 220 of fig. 2.
The data block checking unit 430 is configured to perform a data block consistency check using part or all of the stored parity blocks and the partial data blocks. The operation of the data block checking unit 430 may refer to the operation described above with reference to 230 of fig. 2.
The consistency check unit 440 is configured to determine a data consistency check result for the distributed data storage system based on the check block consistency check result and/or the data block consistency check result. The operation of the consistency check unit 440 may refer to the operation described above with reference to 240 of FIG. 2.
In some embodiments, in response to the check block consistency check result of the check block checking unit 420 indicating that the check block consistency check failed or the data block consistency check result of the data block checking unit 430 indicating that the data block consistency check failed, the consistency checking unit 440 determines that the data consistency check for the distributed data storage system failed. In response to the check block consistency check result of the check block checking unit 420 indicating that the check block consistency check is successful and the data block consistency check result of the data block checking unit 430 indicating that the data block consistency check is successful, the consistency checking unit 440 determines that the data consistency check for the distributed data storage system is successful.
In some embodiments, the parity chunk check unit 420 may perform a parity chunk consistency check once using all of the acquired data chunks. Fig. 5 illustrates an example block diagram of a parity chunk check unit 500 in accordance with an embodiment of this specification.
As shown in fig. 5, the parity check block checking unit 500 includes a parity block calculation module 510 and a parity block check module 520. The check block calculation module 510 performs check block calculation using all the acquired data blocks to calculate any one of the check blocks. Then, the check block checking module 520 performs a check block consistency check according to the calculated check block and the stored corresponding check block.
In some embodiments, the data chunk checking unit 430 may perform a data chunk consistency check once using some or all of the stored parity chunks and partial data chunks. In some embodiments, the data block checking unit 430 may perform a data block consistency check once using all of the stored parity blocks except for the parity block that completes the parity block consistency check and a part of the data blocks.
Fig. 6 illustrates an example block diagram of a data block checking unit 600 in accordance with an embodiment of this specification.
As shown in fig. 6, the data block checking unit 600 includes a data block restoring module 610 and a data block checking unit 620. The data block recovery module 610 performs redundancy decoding using all other parity blocks except for the parity block that completes the parity check of the parity block among the stored parity blocks and a part of the data blocks to recover any one of the data blocks that does not participate in the redundancy decoding. The data block checking module 620 then performs a data block consistency check based on the recovered data block and the stored corresponding data block.
In some embodiments, the data consistency check apparatus 400 may be applied to one data storage device of at least two data storage devices. In this case, the full data obtaining unit 410 may obtain the stored data blocks from the other data storage devices of the at least two data storage devices and the stored parity blocks from the respective data parity devices, and compose the full data together with the data blocks stored by itself.
In some embodiments, the data consistency check apparatus 400 may be applied to one data check device of at least two data check devices. In this case, the full data obtaining unit 410 may obtain the stored parity chunks from the other data verification devices of the at least two data verification devices and the stored data chunks from the respective data storage devices, and compose the full data together with the self-stored parity chunks.
In some embodiments, the data consistency check apparatus 400 may be applied to a third party device. In this case, the full data obtaining unit 410 may obtain the stored data chunks from the respective data storage devices and the stored parity chunks from the respective data verification devices to obtain the full data.
As described above with reference to fig. 1 to 6, a method and a data consistency checking apparatus for checking data consistency of a distributed data storage system according to an embodiment of the present specification are described. The above data consistency check device may be implemented by hardware, or may be implemented by software, or a combination of hardware and software.
FIG. 7 illustrates an example schematic diagram of a computer-system-implemented data consistency check apparatus 700 in accordance with an embodiment of the present description. As shown in fig. 7, the data consistency check apparatus 700 may include at least one processor 710, a storage (e.g., a non-volatile storage) 720, a memory 730, and a communication interface 740, and the at least one processor 710, the storage 720, the memory 730, and the communication interface 740 are connected together via a bus 760. The at least one processor 710 executes at least one computer-readable instruction (i.e., the elements described above as being implemented in software) stored or encoded in memory.
In one embodiment, computer-executable instructions are stored in the memory that, when executed, cause the at least one processor 710 to: acquiring full data stored in a distributed data storage system, wherein the distributed data storage system comprises at least two data storage devices and at least two data check devices, the full data comprises data blocks stored in the data storage devices and check blocks stored in the data check devices, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions during data storage; checking consistency of the check block by using all the acquired data blocks; using part or all of the stored check blocks and part of the data blocks to carry out data block consistency check; and determining a data consistency check result aiming at the distributed data storage system based on the check block consistency check result and/or the data block consistency check result.
It should be appreciated that the computer-executable instructions stored in the memory, when executed, cause the at least one processor 710 to perform the various operations and functions described above in connection with fig. 1-6 in the various embodiments of the present description.
According to one embodiment, a program product, such as a machine-readable medium (e.g., a non-transitory machine-readable medium), is provided. A machine-readable medium may have instructions (i.e., elements described above as being implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described above in connection with fig. 1-6 in the various embodiments of the present specification. Specifically, a system or apparatus may be provided which is provided with a readable storage medium on which software program code implementing the functions of any of the above embodiments is stored, and which causes a computer or processor of the system or apparatus to read out and execute the instructions stored in the readable storage medium.
In this case, the program code itself read from the readable medium can realize the functions of any of the above-described embodiments, and thus the machine-readable code and the readable storage medium storing the machine-readable code form part of the present invention.
Examples of the readable storage medium include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs), magnetic tapes, nonvolatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or the cloud by a communication network.
According to an embodiment, a computer program product is provided, which comprises a computer program that, when executed by a processor, causes the processor to perform the various operations and functions described above in connection with fig. 1-6 in the various embodiments of the present description.
It will be understood by those skilled in the art that various changes and modifications may be made in the above-disclosed embodiments without departing from the spirit of the invention. Accordingly, the scope of the invention should be determined from the following claims.
It should be noted that not all steps and units in the above flows and system structure diagrams are necessary, and some steps or units may be omitted according to actual needs. The execution order of the steps is not fixed, and can be determined as required. The apparatus structures described in the foregoing embodiments may be physical structures or logical structures, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities separately, or some units may be implemented by some components in multiple independent devices together.
In the above embodiments, the hardware units or modules may be implemented mechanically or electrically. For example, a hardware unit, module or processor may comprise permanently dedicated circuitry or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operations. The hardware units or processors may also include programmable logic or circuitry (e.g., a general purpose processor or other programmable processor) that may be temporarily configured by software to perform the corresponding operations. The specific implementation (mechanical, or dedicated permanent, or temporarily set) may be determined based on cost and time considerations.
The detailed description set forth above in connection with the appended drawings describes exemplary embodiments but does not represent all embodiments that may be practiced or fall within the scope of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration," and does not mean "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described technology. However, the techniques may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described embodiments.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (20)
1. A method for data consistency checking of a distributed data storage system, comprising:
acquiring full data stored in a distributed data storage system, wherein the distributed data storage system comprises at least two data storage devices and at least two data check devices, the full data comprises data blocks stored in each data storage device and check blocks stored in each data check device, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions during data storage;
checking consistency of the check block by using all the acquired data blocks;
using part or all of the stored check blocks and part of the data blocks to carry out data block consistency check; and
determining a data consistency check result for the distributed data storage system based on the check block consistency check result and/or the data block consistency check result.
2. The method of claim 1, wherein the determining a data consistency check result for the distributed data storage system based on a check block consistency check result and/or a data block consistency check result comprises:
determining that the data consistency check for the distributed data storage system fails, or in response to the parity check result indicating that the parity check fails or the data chunk consistency check result indicating that the data chunk consistency check fails, determining that the data consistency check for the distributed data storage system fails, or
Determining that a data consistency check for the distributed data storage system is successful in response to the check block consistency check result indicating that the check block consistency check is successful and the data block consistency check result indicating that the data block consistency check is successful.
3. The method of claim 1, wherein using all of the retrieved data chunks for check chunk consistency checking comprises:
checking once the block consistency using all the acquired data blocks, and/or
The data block consistency check by using part or all of the stored check blocks and part of the data blocks comprises the following steps:
and performing data block consistency check once by using part or all of the stored check blocks and part of the data blocks.
4. The method of claim 3, wherein performing a check block consistency check using all of the retrieved data blocks comprises:
performing check block calculation by using all the acquired data blocks to calculate any one of the check blocks; and
and checking the consistency of the check blocks once according to the calculated check blocks and the stored corresponding check blocks.
5. The method of claim 4, wherein performing a data chunk consistency check using some or all of the stored parity chunks and partial data chunks comprises:
and performing data block consistency check once by using all other check blocks except the check block which completes the consistency check of the check blocks and part of the data blocks in the stored check blocks.
6. The method of claim 5, wherein performing a data chunk consistency check using all of the stored parity chunks and the partial data chunks except for the parity chunk that completes the parity chunk consistency check comprises:
performing redundant decoding by using all other check blocks and partial data blocks except the check block which completes the consistency check of the check blocks in the stored check blocks to recover any one data block in the data blocks which do not participate in the redundant decoding; and
and performing data block consistency check once according to the recovered data block and the stored corresponding data block.
7. The method of claim 3, wherein the redundancy coding comprises redundancy coding based on an RS code, and a difference between the number of data check devices and the number of data storage devices is no greater than 2.
8. The method of claim 1, wherein a total number of check blocks and data blocks used in the data block consistency check is equal to a total number stored by the data storage device.
9. The method of claim 1, wherein the check block consistency check and the data block consistency check are performed sequentially or in parallel.
10. The method of claim 1, wherein the method is performed by one of the at least two data storage devices, the at least two data verification devices, and a third party device.
11. An apparatus for data consistency checking for a distributed data storage system, comprising:
the system comprises a full data acquisition unit, a full data acquisition unit and a data storage unit, wherein the full data acquisition unit is used for acquiring full data stored in a distributed data storage system, the distributed data storage system comprises at least two data storage devices and at least two data check devices, the full data comprises data blocks stored by the data storage devices and check blocks stored by the data check devices, and the check blocks of each data check device are obtained by coding all stored original data blocks by using different redundancy coding functions;
the check block checking unit is used for checking the consistency of the check blocks by using all the acquired data blocks;
the data block checking unit is used for checking the consistency of the data blocks by using part or all of the stored check blocks and part of the data blocks; and
and the consistency checking unit determines a data consistency checking result aiming at the distributed data storage system based on a checking block consistency checking result and/or a data block consistency checking result.
12. The apparatus of claim 11, wherein the consistency check unit is to determine that a data consistency check for the distributed data storage system fails or that the consistency check unit fails in response to the check block consistency check result indicating that the check block consistency check fails or the data block consistency check result indicating that the data block consistency check fails
In response to the check block consistency check result indicating that the check block consistency check is successful and the data block consistency check result indicating that the data block consistency check is successful, the consistency check unit determines that the data consistency check for the distributed data storage system is successful.
13. The apparatus of claim 11, wherein the parity chunk checking unit performs a parity chunk consistency check once using all the acquired data chunks, and/or the data chunk checking unit performs a data chunk consistency check once using part or all of the stored parity chunks and part of the data chunks.
14. The apparatus of claim 13, wherein the check block checking unit comprises:
the check block calculation module is used for calculating the check blocks by using all the acquired data blocks so as to calculate any one of the check blocks; and
and the check block checking module is used for carrying out check block consistency check once according to the calculated check block and the stored corresponding check block.
15. The apparatus of claim 14, wherein the data block checking unit performs a data block consistency check once using all of the stored parity blocks except for the parity block for which the consistency check of the parity blocks is completed and a part of the data blocks.
16. The apparatus of claim 15, wherein the data block checking unit comprises:
the data block recovery module is used for performing redundancy decoding on all other check blocks and part of the data blocks except the check block which completes the consistency check of the check blocks in the stored check blocks so as to recover any one data block in the data blocks which do not participate in the redundancy decoding; and
and the data block checking module executes data block consistency check once according to the recovered data block and the stored corresponding data block.
17. A distributed data storage system comprising:
at least two data storage devices, each storing a respective data block;
each data checking device stores a checking block obtained by respectively using different redundancy coding functions to perform redundancy coding on all stored original data blocks during data storage; and
apparatus for performing a data consistency check on a distributed data storage system as claimed in any one of claims 11 to 16.
18. An apparatus for data consistency checking for a distributed data storage system, comprising:
at least one processor for executing a program code for the at least one processor,
a memory coupled to the at least one processor, an
A computer program stored in the memory for execution by the at least one processor to implement the method for data consistency checking of a distributed data storage system as claimed in any one of claims 1 to 10.
19. A computer readable storage medium storing executable instructions that when executed cause a processor to perform a method for data consistency checking a distributed data storage system as claimed in any one of claims 1 to 10.
20. A computer program product comprising a computer program for execution by a processor to implement a method for data consistency checking of a distributed data storage system as claimed in any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211492639.7A CN115756947A (en) | 2022-11-25 | 2022-11-25 | Data consistency verification method, device and system of distributed data storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211492639.7A CN115756947A (en) | 2022-11-25 | 2022-11-25 | Data consistency verification method, device and system of distributed data storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115756947A true CN115756947A (en) | 2023-03-07 |
Family
ID=85338181
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211492639.7A Pending CN115756947A (en) | 2022-11-25 | 2022-11-25 | Data consistency verification method, device and system of distributed data storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115756947A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116756162A (en) * | 2023-06-28 | 2023-09-15 | 蝉鸣科技(西安)有限公司 | Method and system for guaranteeing data consistency |
-
2022
- 2022-11-25 CN CN202211492639.7A patent/CN115756947A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116756162A (en) * | 2023-06-28 | 2023-09-15 | 蝉鸣科技(西安)有限公司 | Method and system for guaranteeing data consistency |
| CN116756162B (en) * | 2023-06-28 | 2024-03-12 | 蝉鸣科技(西安)有限公司 | Method and system for guaranteeing data consistency |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11531593B2 (en) | Data encoding, decoding and recovering method for a distributed storage system | |
| US20200136652A1 (en) | Hierarchical Data Integrity Verification of Erasure Coded Data in a Distributed Computing System | |
| US8527840B2 (en) | System and method for restoring damaged data programmed on a flash device | |
| US9600365B2 (en) | Local erasure codes for data storage | |
| US8386841B1 (en) | Systems and methods for improving redundant storage fault tolerance | |
| CN109643258B (en) | Multi-node repair using high-rate minimal storage erase code | |
| US8862847B2 (en) | Distributed storage method, apparatus, and system for reducing a data loss that may result from a single-point failure | |
| US9459957B2 (en) | Offloading node CPU in distributed redundant storage systems | |
| WO2014121593A1 (en) | Distributed storage method, device and system | |
| CN111697976B (en) | RS erasure correcting quick decoding method and system based on distributed storage | |
| WO2023165536A1 (en) | Data storage method and system, device, and medium | |
| EP3852275A1 (en) | Method, device and apparatus for storing data, computer readable storage medium | |
| CN112000512B (en) | Data restoration method and related device | |
| WO2022127289A1 (en) | Method and system for performing check recovery based on gaussian elimination, device, and medium | |
| CN115756947A (en) | Data consistency verification method, device and system of distributed data storage system | |
| US20230306041A1 (en) | Asynchronous Data Replication in a Storage Network | |
| WO2013185271A1 (en) | Data reconstruction method and apparatus against byzantine failure in network storage, and method and apparatus for restoring failure data | |
| US10637508B2 (en) | Optimal construction of regenerating code through rate-matching | |
| CN109828723B (en) | Distributed storage system and accurate data restoration method and device thereof | |
| US9268640B1 (en) | Limiting data loss on parity RAID groups | |
| US20170161141A1 (en) | Method and apparatus for correcting data in multiple ecc blocks of raid memory | |
| US9405626B1 (en) | At risk data caching (ARDC) | |
| US10379952B2 (en) | Data recovery and regeneration using parity code | |
| Galinanes et al. | Ensuring data durability with increasingly interdependent content | |
| CN111949440A (en) | Data recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |