CN115422133A

CN115422133A - Data processing method and device

Info

Publication number: CN115422133A
Application number: CN202210986636.2A
Authority: CN
Inventors: 李雷; 陈亮; 魏舒展; 赵亚飞; 董元元
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-08-17
Filing date: 2022-08-17
Publication date: 2022-12-02

Abstract

An embodiment of the present specification provides a data processing method and an apparatus, wherein the data processing method includes: creating check nodes with the same hierarchical structure based on attribute information of system nodes, and storing check data to the check nodes, wherein the check data is generated according to first subdata and second subdata stored in the system nodes; under the condition that a failure node exists in a distributed system, determining a target system node having a repair relation with the failure node in the distributed system; extracting target original data stored by the target system node and the check data stored by the check node; and generating failure original data according to the target original data and the check data, and storing the failure original data to the failure node.

Description

Data processing method and device

Technical Field

The embodiment of the specification relates to the technical field of computer storage, in particular to a data processing method and device.

Background

With the development of internet technology, social networks, video and file sharing, cloud storage and other related products are more and more widely applied to daily life of people. As audience populations grow, the data produced by products grows exponentially. In the face of the storage problem of mass data, the traditional centralized storage cannot meet the actual requirements in the aspects of cost and reliability. The distributed storage system has advantages of high scalability, low cost, high reliability, etc., and is widely used and studied in the industry and academia. Meanwhile, in order to avoid data loss, the distributed storage system is developed from the initial use of a copy technology to achieve data fault tolerance to the adoption of erasure codes to store data, so that higher storage efficiency is achieved under the given fault tolerance. Although the distributed storage system in the prior art can improve the storage efficiency of the database through the erasure codes, the erasure codes can be constructed only by requiring an exponential-level packet data support, and the complexity and the resource occupation are very high; meanwhile, in the repair phase, more repair bandwidth needs to be occupied to complete data repair, which greatly affects the performance of the distributed storage system. There is therefore a need for an effective solution to the above problems.

Disclosure of Invention

In view of this, the embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, so as to solve the technical deficiencies of the prior art.

According to a first aspect of embodiments herein, there is provided a data processing method comprising:

creating check nodes with the same hierarchical structure based on attribute information of system nodes, and storing check data to the check nodes, wherein the check data is generated according to first subdata and second subdata stored in the system nodes;

under the condition that a failure node exists in a distributed system, determining a target system node having a repair relation with the failure node in the distributed system;

extracting target original data stored by the target system node and the check data stored by the check node;

and generating failure original data according to the target original data and the check data, and storing the failure original data to the failure node.

According to a second aspect of embodiments of the present specification, there is provided a data processing apparatus comprising:

the system comprises a creating module, a verification module and a verification module, wherein the creating module is configured to create check nodes with the same hierarchical structure based on attribute information of system nodes and store check data to the check nodes, and the check data are generated according to first subdata and second subdata stored in the system nodes;

the system comprises a determining module and a judging module, wherein the determining module is configured to determine a target system node which has a repair relation with a failure node in a distributed system under the condition that the failure node is detected to exist in the distributed system;

an extraction module configured to extract target raw data stored by the target system node and the check data stored by the check node;

and the repairing module is configured to generate failure original data according to the target original data and the check data and store the failure original data to the failure node.

According to a third aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is for storing computer-executable instructions which, when executed by the processor, implement the steps of the data processing method described above.

According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method described above.

According to a fifth aspect of embodiments herein, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method described above.

In order to improve data storage efficiency and reduce storage cost in a scenario supporting data fault tolerance, a check node may be created based on attribute information of a system node in a distributed system, check data generated according to first subdata and second subdata stored in the system node is stored in the check node, and creation of the check node may be controlled by the system node attribute, so as to achieve a purpose of reducing storage cost. And then if the failure node exists in the distributed system, determining a target system node having a repair relation with the failure node in the distributed system, extracting target original data stored by the target system node and check data stored by the check node, generating failure original data corresponding to the failure node by combining the target original data and the check data, and storing the failure original data in the failure node to repair the data in the failure node. In the process, the data repair processing is completed by determining the target system node with the repair relation and combining the check node, so that the repair is completed under the condition of less system nodes, the repair bandwidth of the system nodes can be effectively reduced, and the storage efficiency and the reliability of the distributed system are effectively improved.

Drawings

FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present description;

FIG. 2 is a schematic diagram of encoding in a data processing method according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of encoding in another data processing method provided in one embodiment of the present specification;

FIG. 4 is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present specification;

fig. 6 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if," as used herein, may be interpreted as "at \8230; \8230when" or "when 8230; \823030when" or "in response to a determination," depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

Distributed storage system: a Distributed Storage System provides a Distributed caching System to provide access to the Distributed Storage System and local data buffering to reduce network stress, i.e., a System that stores large amounts of data in a Distributed manner on multiple Storage nodes.

Erasure coding: the (EC) is a data protection method that divides data into fragments, expands, encodes, and stores redundant data blocks in different locations, such as disks, storage nodes, or other geographical locations. The basic principle of the method is to fragment stored data, generate k + m parts of data from k parts of original data in a certain check calculation mode, and restore the k parts of data into the original data from any k parts of data in the k + m parts of data, so that the system can still recover the original data even if part of data is lost.

MDS codes: (maximum distance separable code) refers to a code that reaches the Singelton bound. If M = q ^n+1-d Then, the q-ary (n, M, d) code is called as the maximum distance separable code, and M is the total number of different code words in the (n, M, d) code; another equivalent description is that when C is [ n, k, d ]]In the case of linear codes, a sufficient requirement for the MDS code to be C is d = n-k +1.

Singleton Limit: (Singleton bound) is a measure of the codewords, and is an upper bound on the number of codewords given the length of the codeword and the minimum distance, and q-ary (n, M) codes with the minimum distance d are usually called q-ary (n, M, d) codes, where different total numbers of codewords are denoted as a (n, d), and the singelton bound indicates: q is less than or equal to A (n, d) ^n+1-d 。

In the present specification, a data processing method is provided, and the present specification relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

In practical application, the distributed storage system supports data fault tolerance, reduces storage cost, and adopts erasure codes to store data. RS (Reed-Solomon) code is a maximum distance divisible code and is widely used in data storage due to its maturity. In an actual distributed storage system, node failure is a common phenomenon. In order to maintain the reliability of the system, the failed node needs to be repaired in time, and the repair bandwidth of the (n, k) RS code is k times of the data storage capacity of a single node. Local Repair Codes (LRC) and Minimum Storage regeneration codes (MSR codes) are effective means for reducing repair bandwidth, and the latter has the highest Storage efficiency (lowest Storage cost) due to MDS properties. The construction of the MSR code requires an exponential-level sub-packet level, and discontinuous data needs to be read in the node repair and degradation reading processes, which is high in complexity, so that the MSR code is difficult to popularize in actual storage and application.

Fig. 1 shows a flowchart of a data processing method provided in accordance with an embodiment of the present specification, which specifically includes the following steps.

Step S102, creating check nodes with the same hierarchical structure based on attribute information of system nodes, and storing check data to the check nodes, wherein the check data is generated according to first subdata and second subdata stored in the system nodes.

Specifically, the distributed system specifically refers to a distributed storage system; correspondingly, the system node specifically refers to a node included in the distributed storage system, and is used for storing data; the data types that can be stored by the system node include, but are not limited to, transaction data, user information data, video data, image data, and the like, and the data stored in different scenarios may be selected according to actual application scenarios, which is not limited herein. Correspondingly, the check node specifically refers to a node used for storing check data in the distributed system, and the check node is different from the data directly stored upstream and issued by the system node, but generates the check data based on the data stored by the system node and is dedicated for storing the check data, so that decoding recovery can be performed through the check data stored in the check node under the condition that the system node fails. It should be noted that the number of check nodes in the distributed system is smaller than the number of system nodes, and the hierarchical structures of the check nodes and the system nodes are the same, so as to support the data reading distribution requirement during the repair process. Correspondingly, the original data specifically refers to data stored by the system node and issued by an upstream, and correspondingly, the check data specifically refers to data generated according to the original data, and the check data can be combined with the original data stored by other system nodes to complete data recovery processing in the failed node.

Based on this, in order to support fault tolerance of data while reducing the storage cost of the distributed system, so that after data is lost, data repair processing can be completed by using less repair bandwidth, a check node adapted to the current distributed system can be created based on the attribute information of the system nodes in the distributed system, and the adaptation degree can achieve the purpose of occupying less storage space, thereby realizing that data stored in any system node can be repaired, namely forming a check node with the same hierarchical structure as the system nodes. After the check node is created, check data can be generated according to first subdata and second subdata stored in the system node, wherein the first subdata and the second subdata are original data stored in the system node; the check data is then stored to the check nodes for possible repair processing operations in the event of a system node failure.

It should be noted that, because the system node in the distributed system may increase data in real time, after each time the original data is updated, the check data needs to be updated at the same time, so as to ensure that the check data stored in the check node is generated based on the latest original data. On this basis, the check data stored in the check node may be updated after the original data is updated each time, or may be updated according to a preset time node, which is not limited herein.

Furthermore, in the process of creating check nodes, in order to enable the number of check nodes to match the number of system nodes of the distributed system and achieve that data recovery can be performed on any failed system node in any state, the creation of a set number of check nodes needs to be completed by combining the number of system nodes in the distributed system; in this embodiment, the specific implementation manner is as follows:

acquiring attribute information of system nodes in the distributed system; determining the number of system nodes contained in the distributed system according to the attribute information; creating a set number of check nodes according to preset check node creating conditions and the number of the system nodes; wherein the hierarchical structure of the check node and the system node is the same.

Specifically, the number of system nodes specifically refers to the total number of all available system nodes included in the distributed system; correspondingly, the preset check node creation condition specifically refers to a condition for determining a set number, which defines how many check nodes should be created in the distributed system at different scales, so as to support repairability of the distributed system data by using the minimum storage resources. Wherein, the preset check node creation condition may be that k is greater than or equal to 2 (r-1), i.e. r is greater than or equal to k/2+1; wherein k is the number of system nodes, and r is the number of check nodes.

Based on the method, firstly, attribute information corresponding to system nodes in the distributed system is obtained, the number of the system nodes which can be used for supporting the upstream in the distributed system can be determined according to the attribute information, then, the set number of the check nodes which can be created in the current distributed system is determined according to the preset check node creating conditions and the number of the system nodes, and then, the set number of the check nodes is created in the distributed system. In practical application, when a check node is created, a disaster recovery system stage can be selected in a distributed system as the check node, and a new storage space can be established as the check node.

For example, in the case that the distributed storage system includes 8 system nodes, 3 check nodes may be created according to a preset check node creation condition; or, in the case that the distributed storage system includes 6 system nodes, 2 check nodes may be created according to a preset check node creation condition.

In summary, by combining the preset check node creation conditions and reasonably creating the set number of check nodes for the distributed system, the repairable processing of data can be supported by using less storage resources, so that the storage cost is reduced, and the reliability of the distributed system is effectively improved.

Furthermore, after the check node is created and completed for the distributed system, check data needs to be stored in the check node, and the check data is a basis for repairing data stored in the failed node after any system node fails, so that when the check data is generated, the check data needs to be completed according to the relationship between the system node and the check node; in this embodiment, the specific implementation manner is as in step S1022 to step S1028.

Step S1022, a first system child node and a second system child node are determined in the system node.

Specifically, the first system child node specifically refers to a system node in the system nodes, which is used for generating the check data required by the first check child node, and the second system child node specifically refers to a system node in the system nodes, which is used for generating the check data required by the second check child node; for example, the first syndrome node and the second syndrome node may generate the check data by combining the original data stored in the system nodes 1,2, and 3 when generating the first syndrome node, and the second syndrome node may generate the check data by combining the original data stored in the system nodes 2,3, and 4 when generating the check data required by different syndrome nodes, the system nodes 2 and 3 may be used as the first system syndrome node or the second system syndrome node.

Further, in the process of determining the first system child node and the second system child node, in consideration that different system child nodes need to create check data required by different check child nodes for different check child nodes, in order to achieve a mechanism that can be repaired by using less storage resources to support data, the created check nodes may be divided into two different types, namely, a first check child node having a direct check relationship with a system node and a second check child node having an indirect check relationship with the system node, and then the system nodes corresponding to the respective check child nodes are selected as the first system child node and the second system child node, which in this embodiment is specifically implemented as follows:

taking a syndrome node in the check nodes, which has a direct check relationship with the system node, as a first syndrome node, and taking a syndrome node in the check nodes, which has an indirect check relationship with the system node, as a second syndrome node; determining the first system child node in the system node according to the first syndrome child node, and determining the second system child node in the system node according to the second syndrome child node.

Specifically, the direct check relationship specifically refers to a preset check node type, and is used for determining that original data required by the check node of the type is data stored by all system nodes; correspondingly, the indirect check relation specifically refers to another preset check type, and is used for determining that the original data required by the check node of the type is data stored by the local system node, so as to distinguish the original data and facilitate the subsequent creation of check data aiming at different syndrome nodes; correspondingly, the first syndrome node is a check node corresponding to the direct check relationship; the second syndrome node is the check node corresponding to the indirect check relationship.

Based on this, after the check nodes are created for the distributed system, a first check child node which needs to combine the data stored in all the system nodes for creating check data can be selected from the set number of created check nodes, and meanwhile, a second check child node which needs to combine the data stored in the local system nodes for creating check data is selected. After the first syndrome node and the second syndrome node are determined, a first system syndrome node associated with the first syndrome node and a second system syndrome node associated with the second syndrome node can be determined in the system nodes, so that creation and storage of the check data can be completed according to the mapping relation between the syndrome node and the system nodes.

The data processing method is described by taking an example that the distributed storage system includes 6 system nodes and 2 check nodes, and the same or corresponding description contents in this embodiment can be referred to for descriptions in other scenarios, which is not described in detail herein.

The distributed storage system comprises nodes 1 to 8, wherein the nodes 1 to 6 are system nodes, and the node 7 and the node 8 are check nodes. When the check data is created for the check node, it may be determined that the creation of the check data can be implemented by the node 7 in combination with all data stored by the nodes 1 to 6, and then the node 7 is used as a first check child node, and the creation of the check data can be implemented by the node 8 in combination with local data, and then the node 8 is used as a second check child node. Further, the nodes 1 to 6 are all associated with the node 7, and then the nodes 1 to 6 are used as the first system child nodes corresponding to the node 7; correspondingly, nodes 1-6 are also all associated with node 8, and then nodes 1-6 can be regarded as second system child nodes corresponding to node 8.

In summary, by determining the syndrome node and the system sub-node by combining the direct check relationship and the indirect check relationship, the mapping relationship between the syndrome node and the system sub-node can be determined in the determination process, the generation of check data can be performed according to the mapping relationship conveniently in the follow-up process, and the data waste caused by the excessive data loading is avoided.

Step S1024, generating first check subdata according to the first original subdata stored in the first system sub-node.

Specifically, after the first system sub-node and the second system sub-node are determined, further, first syndrome data may be generated according to the first original syndrome data stored in the first system sub-node, so that the check data for storing in the check node may be formed subsequently by combining with the second check data. The first original subdata specifically refers to original data stored in a first system sub node, and correspondingly, the first check subdata specifically refers to part of check data stored in a check node.

Further, in the process of generating the first syndrome data, in order to save repair bandwidth, a small amount of original syndrome data needs to be combined in the generation stage, so that the data processing method provided by this embodiment generates the first syndrome data on the basis of packet numbers of the system subnode and the syndrome subnode, and the specific implementation manner is as follows:

determining a first system node sub-packet in the first system sub-node, and generating first check sub-packet data corresponding to a first check node sub-packet in the first check sub-node according to first original sub-packet data stored in the first system node sub-packet; determining a second system node sub-packet in the first system sub-node, and generating second check sub-packet data corresponding to a second check node sub-packet in the first check sub-node according to second original sub-packet data stored in the second system node sub-packet; the first original sub-packet data and the second original sub-packet data form the first original sub-data, and the first check sub-packet data and the second check sub-packet data form the first check sub-data.

Specifically, because the system subnodes and the check subnodes are both segmented with the packet number of 2, each system subnode and each check subnode are composed of two parts of sub-packets, namely, a first system node sub-packet is one sub-packet of two sub-packets of a first system subnode, a second system node sub-packet is another sub-packet, and the first system node sub-packet and the second system node sub-packet form a first system subnode; correspondingly, the first original sub-packet data is original data stored in the first system node sub-packet, the second original sub-packet data is original data stored in the second system node sub-packet, and the first original sub-packet data and the second original sub-packet data form original data stored in a first system sub-node. Correspondingly, the first check node sub-packet is one sub-packet of the two sub-packets of the first check sub-node, and the second check node sub-packet is the other sub-packet. The first check node sub-package and the second check node sub-package form a first check child node; correspondingly, the first check sub-packet data is check data needing to be written in the first check node sub-packet, the second check sub-packet data is check data needing to be written in the second check node sub-packet, and the first check sub-packet data and the second check sub-packet data form first check sub-data. I.e. the check data stored in the first syndrome node.

Based on this, after determining the first system subnode associated with the first syndrome subnode, in order to save storage resources, each system subnode and each syndrome subnode are packetized by using the number of the packetized data as 2, and based on this, when generating the required first syndrome data for the first syndrome subnode, the first system node may be first determined to be packetized in the first system subnode, and first syndrome packet data corresponding to the first syndrome node packetized packet in the first syndrome subnode is generated according to first original packetized data stored in the first system node packetized packet; determining a second system node sub-packet in the first system sub-node, and generating second check sub-packet data corresponding to a second check node sub-packet in the first check sub-node according to second original sub-packet data stored in the second system node sub-packet; the first original sub-packet data and the second original sub-packet data form first original subdata, and the first check sub-packet data and the second check sub-packet data form first check subdata; the generation of the data which can be combined with the subpackage storage of different layer structures aiming at the first check child node is realized.

In specific implementation, the first check sub-packet data can be determined by a formula (1):

wherein p1,1 represents first check packet data; a1, j represents first original packet data; k represents the number of system child nodes; j belongs to {1,2 \8230, k } and represents system node sub-package;

accordingly, the second calibration packet data may be determined by equation (2):

wherein p2,1 represents the second check packet data; a2, j denote the second original packetization data.

In practical application, referring to the schematic diagram shown in fig. 2, in a scenario where k system nodes and r check nodes are provided, the k system nodes may be divided into t =2 (r-1) groups, each group includes m = k/2 (r-1) system nodes, and t check data p1,2, p2, \ 8230, p1, r, p2, r respectively correspond to the t groups of system nodes, that is, the check data is obtained by linearly combining k original data of a row where the check data is located and m data of different rows in the corresponding group.

Following the above example, in a case that the system node k is 2 check nodes r with 6, and the packet number of each node is 2, the generating of the check data corresponding to the node 7 includes: referring to fig. 3, firstly, taking p1,1 in the node 7 as first check packet data to be generated, determining that the data needs to be generated by combining the first system node packets corresponding to the nodes 1 to 6 according to the formula (1), that is, data a1,1 stored in the node 1 by packet in the first system node and data a1,2, \\8230storedin the node 2 by packet in the first system node are needed to be combined, and data a1,6 stored in the node 6 by packet in the first system node; after loading into data { a1,1; a1,2; after a1,6, determining p1,1= a1,1+ a1,1, 2+ a1,3+ a1,4+ a1,5+ a1,6 according to formula (1), then encoding the data by combining the expression, and obtaining the first check sub-packet data p1,1 in the node 7. Similarly, the p2 and 1 in the node 7 are used as second check packet data to be generated, and it is determined according to the formula (2) that the data a2 and 1 stored in the node 1 by the second system node and the data a2,2 and 823030stored in the node 2 by the second system node by the node 2 need to be combined and generated by the second system node packet corresponding to the nodes 1 to 6, and the data a2 and 6 stored in the node 6 by the second system node packet; after loading into data { a2,1; a2,2; after 8230a 2,6}, determining p2,1= a2,1+ a2,2+ a2,3+ a2,4+ a2,5+ a2,6 according to the formula (2), and then encoding the data by combining the expression, thereby obtaining second checking and subpackaging data p2,1 in the node 7. Wherein "+" and "-" are addition and subtraction operations in the finite field Fq.

In summary, by packetizing the system subnode and the syndrome subnode with the number of the packets being 2, and generating the check data for the syndrome subnode, the generation of the check data can be completed on the premise of using less data, and less storage resources are occupied, so that the occupation of the repair bandwidth in the data repair stage is increased, the repair efficiency is increased, and the reliability of the distributed system is ensured.

Step S1026, generating second parity data according to a preset parity parameter and the second primitive data stored in the second system sub node.

Specifically, after the generation of the first syndrome node data is completed, further, second syndrome data needs to be generated for the second syndrome node, and since an indirect repair relationship exists between the second syndrome node and the second system node, in the encoding stage, the second syndrome data needs to be generated by combining the preset syndrome parameters and the second original syndrome data stored in the second system node. The check parameters specifically refer to non-zero elements combined in the encoding stage. Correspondingly, the second original subdata specifically refers to original data stored in a second system sub-node, and correspondingly, the second check subdata specifically refers to part of check data stored in a check node, and the check data is composed of the first check subdata and the second original subdata.

Further, in the process of generating the second check sub data, in order to save repair bandwidth and ensure data availability check in the repair stage, a small amount of original sub data needs to be combined in the generation stage, so that the data processing method provided in this embodiment generates the second check sub data based on two packet numbers of the system sub node and the check sub node, and the specific implementation manner is as follows:

determining, in the second system child node, a first global system node sub-packet and a first local system node sub-packet, and a second global system node sub-packet and a second local system node sub-packet; generating third check sub-packet data corresponding to a third check node sub-packet in the second check sub-node according to preset check parameters, first global sub-packet data stored in the first global system node sub-packet and first local sub-packet data stored in the first local system node sub-packet; generating fourth check sub-packet data corresponding to a fourth check node sub-packet in the second check sub-node according to preset check parameters, second global sub-packet data stored in the second global system node sub-packet and second local sub-packet data stored in the second local system node sub-packet; wherein the third check sub-packet data and the fourth check sub-packet data constitute the second check sub-data.

Specifically, because the system subnodes and the check subnodes are both segmented by the number of the sub-packets being 2, each system subnode and each check subnode are composed of two part sub-packets, namely, the first global system node sub-packet is one sub-packet of two sub-packets of all the second system subnodes, the second all-system node sub-packet is the other sub-packet of all the second system subnodes, and the first global system node sub-packet and the second all-system node sub-packet constitute all the second system subnodes; correspondingly, the first local system node sub-packet is one of two sub-packets of a part of the second system sub-node, the second local system node sub-packet is another sub-packet of a part of the second system sub-node, and the first local system node sub-packet and the second local system node sub-packet constitute a part of the second system sub-node.

Correspondingly, the first global packet data is original data stored in the first global system node packet; the second global packet data is original data stored in a second global system node packet; the first global packet data and the second global packet data constitute original data stored in all the second system sub-stages. Correspondingly, the first local packet data is original data stored in the first local system node packet; the second local sub-packet data is original data stored in a second local system node sub-packet; the first partial packet data and the second partial packet data constitute a part of original data stored in the second system child node.

Correspondingly, the third check node sub-packet is one sub-packet of the two sub-packets of the second check sub-node, and the fourth check node sub-packet is the other sub-packet. The third check node sub-packet and the fourth check node sub-packet form a second check child node; correspondingly, the third check sub-packet data is check data needing to be written in the third check node sub-packet, the fourth check sub-packet data is check data needing to be written in the fourth check node sub-packet, and the third check sub-packet data and the fourth check sub-packet data form second check sub-data. I.e. the check data stored in the second check child node.

Based on this, after determining the second system child node associated with the second parity child node, in order to save storage resources, each system child node and each parity child node are packetized with the number of packets being 2, and based on this, when generating the second parity data necessary for the second parity child node, the first global system node packet and the first local system node packet, and the second global system node packet and the second local system node packet may be determined in the second system child node.

Further, third check sub-packet data corresponding to a third check node sub-packet in the second check sub-node is generated according to preset check parameters, first global sub-packet data stored in the first global system node sub-packet and first local sub-packet data stored in the local system node sub-packet.

Furthermore, fourth check sub-packet data corresponding to a fourth check node sub-packet in the second syndrome node is generated according to the preset check parameters, second global sub-packet data stored in the second global system node sub-packet and second local sub-packet data stored in the second local system node sub-packet; and the third check sub-packet data and the fourth check sub-packet data form second check sub-data.

In specific implementation, the third check sub-packet data may be determined by formula (3):

wherein p1 and r represent third check sub-packet data;

representing a verification parameter; ax, j denotes first global packetization data; m represents the local system node sub-package number; r belongs to {2,3 \8230, r } represents check node sub-packaging; iota represents a local system node sub-package; a2, iota represents the first partial packet data;

accordingly, the fourth parity packetization data can be determined by equation (4):

wherein p2, r represents fourth check sub-packet data; a1, iota represents second local sub-package data; ay, j denotes second global packetization data.

In the practical application of the method, the air conditioner,

is k (r-1) nonzero elements in the finite field Fq by selecting proper elements

The encoding process for generating the syndrome data can be made to have MDS property.

Following the above example, in a case that the system node k is 2 check nodes r with 6, and the packet number of each node is 2, the generating of the check data corresponding to the node 8 includes: referring to fig. 3, first, p1,2 in the node 8 is taken as the third check sub-packet data to be generated, according to the above disclosureEquation (3) determines that first system node packets corresponding to nodes 1 to 6 need to be combined, and second system node packets corresponding to nodes 1,2, and 3 need to be generated, that is, data a1,1 stored in the first system node packet in node 1, data a1,2, \8230storedin the first system node packet in node 2, data a1,6 stored in the first system node packet in node 6, data a2,1 stored in the second system node packet in node 1, data a2,2 stored in the second system node packet in node 2, and data a2,3 stored in the second system node packet in node 3 need to be combined; after loading into data { a1,1; a1,2; a1,3; a1,4; a1,5; a1,6; a2,1; a2,2; a2, 3), according to the formula (3)

Then, the data may be encoded by combining the expression, so as to obtain the third check sub-packet data p1,2 in the node 8.

Similarly, taking p2,2 in the node 8 as fourth check packet data to be generated, determining that it is necessary to combine the second system node packets corresponding to the nodes 1 to 6, respectively, and generate the first system node packets corresponding to the nodes 4, 5, and 6, respectively, according to the above formula (4), that is, it is necessary to combine the data a2,1 stored in the second system node packets in the node 1, the data a2, \82303030storedin the second system node packets in the node 2, the data a2,6 stored in the second system node packets in the node 6, the data a1,4 stored in the first system node packets in the node 4, the data a1,5 stored in the first system node packets in the node 5, and the data a1,6 stored in the first system node packets in the node 6; after loading into data { a2,1; a2,2; a2,3; a2,4; a2,5; a2,6; a1,4; a1,5; a1, 6), according to the formula (4)

Then the data may be entered at this point in conjunction with the expressionAnd performing line coding processing to obtain fourth check sub-packet data p2 and 2 in the node 8. Wherein,

the coding process can have MDS property by selecting proper value for non-zero element in the finite field Fq.

In sum, by combining the preset check parameters and the original data corresponding to different sub-packets and generating the check data corresponding to the second check sub-node, the generation of the check data in the encoding stage can be realized to have the MDS property, so that the node repair node is ensured, the data recovery can be completed by combining less data, and the repair bandwidth is effectively reduced.

Step S1028, generating the check data according to the first check subdata and the second check subdata, where the first primitive subdata and the second primitive subdata constitute primitive data stored in the system node.

Specifically, after the first syndrome data and the second syndrome data are obtained, the check data for storing in the check node may be generated according to the first syndrome data and the second syndrome data, so that the repair processing operation of the system node may be completed by combining the check data when the system node fails.

In conclusion, by generating the check data by taking the number of the sub-packets as 2, the storage efficiency of the distributed system is higher and the consumption of storage resources is saved under the condition of reducing the repair bandwidth of the system nodes.

And step S104, under the condition that the failure node exists in the distributed system, determining a target system node having a repair relation with the failure node in the distributed system.

Specifically, after the generation of the check node and the check data is completed, the distributed system may call the system node to perform data storage processing according to the upstream data storage request. On this basis, if it is detected that a failed node exists in the distributed system, it indicates that data stored in the system node will be lost, and in order to avoid affecting the upstream and downstream, data recovery may be completed in combination with erasure codes.

The failed node specifically refers to a node in the distributed system, which cannot continue data storage, and causes of the failed node include, but are not limited to, power failure, disk damage, restart, and the like. Correspondingly, the target system node specifically refers to a system node that needs to be used when data stored by the failed node can be repaired, and the relationship between the target system node and the system node is the repair relationship.

In practical application, in order to complete node recovery under the premise of downloading the minimum data in consideration of different numbers of failed nodes at the same time under different scenes, different processing levels can be divided according to the number of the failed nodes, that is, different recovery strategies are determined according to the number of the failed nodes and are used for recovering data corresponding to the failed nodes. In specific implementation, during the classification, it may be determined whether the number of failed nodes is greater than a threshold 1, if the number of failed nodes is greater than 1, a first repair processing policy is selected, and if the number of failed nodes is less than or equal to 1, a second repair processing policy is selected, where data required by the first repair processing policy and the second repair processing policy may be determined by combining the above equations (1) to (4).

Furthermore, the data required during the repair processing are from the system node and the check node, so that the target system node corresponding to the failed node can be determined by determining the source of the required data.

Step S106, extracting the target original data stored by the target system node and the check data stored by the check node.

Specifically, after the target system node having the association relationship with the failure node is determined, the check node is considered to be used for repairing the failure node, so that the target original data stored in the target system node and the check data stored in the check node can be extracted, and the data repairing processing on the failure node can be completed subsequently by combining the check data and the target original data.

And S108, generating failure original data according to the target original data and the check data, and storing the failure original data to the failure node.

Specifically, on the basis of obtaining the target original data stored in the target system node and the check data stored in the check node, further, the failure original data may be generated according to the check data and the target original data, and stored in the failed node after the restart, so that the recovery processing operation on the original data stored in the failed node may be completed.

Further, during the repair process, if the number of failed nodes is one, in order to save the repair bandwidth, the target system node associated with the failed node needs to be accurately screened out first, and in this embodiment, the specific implementation manner is as follows:

determining at least two candidate system nodes of the check node in association with the check node creation stage; and taking the candidate system nodes except the failed node in the at least two candidate system nodes as target system nodes having a repair relation with the failed node.

Specifically, the candidate system node specifically refers to a check node to which the original data used in the check node creating stage belongs. Based on this, in order to recover the failed node, it may be determined that the check node creating stage associates at least two candidate system nodes of the check node, and then, other nodes except the failed node are all used as target system nodes for performing data repair processing subsequently.

In conclusion, the target system node is determined by selecting the candidate system node associated with the check node, so that the association degree between the target system node and the failure node can be ensured, and the data restoration precision can be improved.

Furthermore, when generating the failure original data, considering that the number of the packets of the failure node is 2, in the recovery stage, it is necessary to perform a repair processing operation on each packet, and in this embodiment, the specific implementation manner is as follows:

determining first target original subdata and second target original subdata in the target original data, and determining third check subdata and fourth check subdata in the check data; the data volume of the second target original subdata is smaller than that of the first target original subdata; generating first failure original subdata corresponding to first failure node sub packets in the failure nodes according to the first target original subdata and the third check subdata; generating second failure original subdata corresponding to second failure node sub-packets in the failure nodes according to the second target original subdata and the fourth check subdata; and generating the failure original data according to the first failure original subdata and the second failure original subdata.

Specifically, the first target original subdata specifically refers to original data required to be used for decoding a first failure node sub-packet in a failure node, and the first target original subdata is from a first target system node sub-packet of a target system node; correspondingly, the second target original subdata specifically refers to original data required by decoding a second failure node sub-packet in the failure node, and the second target original subdata is from a second target system node sub-packet of the target system node. Correspondingly, the third check subdata specifically refers to check data stored in check subdomain packets in the check nodes; the fourth syndrome data specifically refers to the syndrome data stored in another syndrome node sub-packet in the check nodes.

Based on the method, first target original subdata and second target original subdata are determined in target original data, and third check subdata and fourth check subdata are determined in check data; the data volume of the second target original subdata is smaller than that of the first target original subdata; secondly, generating first failure original subdata corresponding to first failure node sub packets in failure nodes according to the first target original subdata and the third check subdata; generating second failure original subdata corresponding to second failure nodes in a sub-packet mode according to the second target original subdata and the fourth check subdata; and generating failure original data according to the first failure original subdata and the second failure original subdata.

In specific implementation, for the case of single system node failure, bandwidth reduction method repair exists, taking system node 1 failure as an example, data { a1,2; a1,3; 8230a 1, k and p1,1, the data a1,1 in the node 1 can be calculated according to the formula (1)

And node 1 still has failed data a2,1, so it is determined according to equations (3) and (4) that data { a2,2; a2,3; 8230a 2, m and p1,2, the data a2,1 in the node 1 can be calculated according to the formula (3)

And finally writing the decoded a1,1 into the first sub-packet in the node 1, and writing the a2,1 into the second sub-packet in the node 1, so as to finish the recovery of the data of the node 1. In the process, the data amount required to be downloaded by the repair node 1 is determined to be k + k/2 (r-1), similarly, the repair of any system node i can be calculated according to the formula (1), (2) or (3), (4) and reading corresponding data, and the data a1, i and a2, i stored by the node i are obtained.

In addition, when any system node fails, that is, when the number of failed nodes is greater than 1, the equations (1) to (4) may be transformed to obtain a transformation equation set:

wherein the variables in the equation set are { a1,1; a2,1; a1,2; a2,2; 8230a 1, r; a2, r, according to MDS property, the coefficient matrix full rank corresponding to the unknown quantity, therefore, downloading all data of the nodes r + 1-r + k can solve the unique solution of the equation set, namely, the decoding of the data is realized, and the data recovery is carried out on the failed node.

Following the above example, as shown in fig. 3, in the event of a failure of node 1, the data is updated by downloading data { a1,2; a1,3; a1,4; a1,5; a1,6} and p1,1, a1,1= p1,1-a1,2-a1,3-a1,4-a1,5-a1,6 is calculated according to the formula (1). Then, downloading the data { a2,2; a2,3} and p1,2, can be calculated according to equation (3)

According to a1,1 and a2,1, the recovery of the data of the node 1 can be completed.

In the case where the node 1 and the node 2 fail simultaneously, the equation system can be obtained by converting the equations (1) to (4):

a1,1+a1,2+a1,3+a1,4+a1,5+a1,6-p1,1＝0

a2,1+a2,2+a2,3+a2,4+a2,5+a2,6-p2,1＝0

and determining the unknowns a1, a1,2, a2,1 and a2,2 through the equation set, wherein the unknowns are a1, a1,2, a2,1 and a2,2, and according to the MDS property, the coefficient matrix corresponding to the unknowns is full-rank, so that the only solution of the equation can be obtained by downloading all data on the nodes 3-6 and used for repairing the nodes 1 and 2.

The following describes the data processing method further with reference to fig. 4 by taking an application of the data processing method provided in this specification in a scenario where the number of system nodes is 6 and the number of check nodes is 2 as an example. Fig. 4 shows a flowchart of a processing procedure of a data processing method according to an embodiment of the present specification, which specifically includes the following steps.

And step S402, acquiring attribute information of system nodes in the distributed system.

Step S404, determining the number of system nodes contained in the distributed system according to the attribute information.

Step S406, creating a set number of check nodes according to preset check node creating conditions and the number of the system nodes.

Step S408, storing verification data to the verification node, wherein the verification data is generated according to the original data stored in the system node.

Specifically, the generation of the verification data includes: determining a first system sub-node and a second system sub-node in the system node; generating first check subdata according to first original subdata stored in the first system sub-node; generating second check-up subdata according to preset check-up parameters and second original subdata stored in the second system sub-node; generating the check data according to the first check subdata and the second check subdata; and the first original subdata and the second original subdata form the original data.

Further, a syndrome node having a direct checking relationship with the system node in the check nodes is used as a first syndrome node, and a syndrome node having an indirect checking relationship with the system node in the check nodes is used as a second check node; determining the first system child node in the system node according to the first syndrome node, and determining the second system child node in the system node according to the second syndrome node.

Under the condition that the number of sub packets of the system sub-node in the system node and the number of sub packets of the check sub-node in the check node are both two, generating first check sub-data according to first original sub-data stored in the first system sub-node includes:

Further, determining in the second system child node a first global system node sub-packet and a first local system node sub-packet, and a second global system node sub-packet and a second local system node sub-packet; generating third check sub-packet data corresponding to a third check node sub-packet in the second check sub-node according to preset check parameters, first global sub-packet data stored in the first global system node sub-packet and first local sub-packet data stored in the first local system node sub-packet; generating fourth check sub-packet data corresponding to a fourth check node sub-packet in the second check sub-node according to preset check parameters, second global sub-packet data stored in the second global system node sub-packet and second local sub-packet data stored in the second local system node sub-packet; wherein the third check sub-packet data and the fourth check sub-packet data constitute the second check sub-data.

Step S410, under the condition that the failure node exists in the distributed system, determining a target system node having a repair relation with the failure node in the distributed system.

Step S412, extracting the target original data stored in the target system node and the check data stored in the check node.

And step S414, generating failure original data according to the target original data and the check data, and storing the failure original data to the failure node.

When the number of the failed nodes is one, the determining, in the distributed system, a target system node having a repair relationship with the failed nodes includes: determining at least two candidate system nodes of the check node in a stage of establishing the check node; and taking the candidate system nodes except the failed node in the at least two candidate system nodes as target system nodes having a repair relation with the failed node.

Further, determining first target original subdata and second target original subdata in the target original data, and determining third check subdata and fourth check subdata in the check data; the data volume of the second target original subdata is smaller than that of the first target original subdata; generating first failure original subdata corresponding to first failure node sub packets in the failure nodes according to the first target original subdata and the third check subdata; generating second failure original subdata corresponding to second failure node sub-packets in the failure nodes according to the second target original subdata and the fourth check subdata; and generating the failure original data according to the first failure original subdata and the second failure original subdata.

In summary, in order to improve the data storage efficiency and reduce the storage cost in the scenario supporting data fault tolerance, a check node may be created based on the attribute information of the system node in the distributed system, and check data generated according to the first subdata and the second subdata stored in the system node is stored in the check node, so that the creation of the check node may be controlled by the attribute of the system node, and the purpose of reducing the storage cost is achieved. And then if the failure node exists in the distributed system, determining a target system node having a repair relation with the failure node in the distributed system, extracting target original data stored by the target system node and check data stored by the check node, generating failure original data corresponding to the failure node by combining the target original data and the check data, and storing the failure original data in the failure node to repair the data in the failure node. In the process, the data repair processing is completed by determining the target system node with the repair relation and combining the check node, so that the repair is completed under the condition of less system nodes, the repair bandwidth of the system nodes can be effectively reduced, and the storage efficiency and the reliability of the distributed system are effectively improved.

Corresponding to the above method embodiment, the present specification further provides an embodiment of a data processing apparatus, and fig. 5 shows a schematic structural diagram of a data processing apparatus provided in an embodiment of the present specification. As shown in fig. 5, the apparatus includes:

a creating module 502 configured to create check nodes with the same hierarchical structure based on attribute information of a system node, and store check data to the check nodes, wherein the check data is generated according to first subdata and second subdata stored in the system node;

a determining module 504 configured to determine a target system node having a repair relationship with a failed node in a distributed system in case of detecting that the failed node exists in the distributed system;

an extracting module 506 configured to extract target raw data stored by the target system node and the check data stored by the check node;

and a repair module 508 configured to generate failure original data according to the target original data and the check data, and store the failure original data to the failure node.

In an optional embodiment, the creating module 502 is further configured to:

acquiring attribute information of system nodes in the distributed system; determining the number of system nodes contained in the distributed system according to the attribute information; creating a set number of check nodes according to preset check node creating conditions and the number of the system nodes; wherein the hierarchical structure of the check nodes and the system nodes is the same.

In an optional embodiment, the creating module 502 is further configured to:

determining a first system child node and a second system child node in the system node; generating first check subdata according to first original subdata stored in the first system sub-node; generating second check sub data according to preset check parameters and second original sub data stored in the second system sub node; generating the check data according to the first check subdata and the second check subdata; and the first original subdata and the second original subdata form original data stored by the system node.

In an optional embodiment, the creating module 502 is further configured to:

taking a syndrome node in the check nodes having a direct check relationship with the system node as a first syndrome node, and taking a syndrome node in the check nodes having an indirect check relationship with the system node as a second syndrome node; determining the first system child node in the system node according to the first syndrome child node, and determining the second system child node in the system node according to the second syndrome child node.

In an optional embodiment, in a case that the number of packets of the system child node in the system node and the parity child node in the parity node are both two, the creating module 502 is further configured to:

In an optional embodiment, the creating module 502 is further configured to:

determining, in the second system child node, a first global system node sub-packet and a first local system node sub-packet, and a second global system node sub-packet and a second local system node sub-packet; generating third check sub-packet data corresponding to a third check node sub-packet in the second check sub-node according to preset check parameters, first global sub-packet data stored in the first global system node sub-packet and first local sub-packet data stored in the first local system node sub-packet; generating fourth check sub-packet data corresponding to a fourth check node sub-packet in the second syndrome node according to preset check parameters, second global sub-packet data stored in the second global system node sub-packet and second local sub-packet data stored in the second local system node sub-packet; wherein the third check sub-packet data and the fourth check sub-packet data constitute the second check sub-data.

In an optional embodiment, in case that the number of failed nodes is one, the determining module 504 is further configured to:

In an optional embodiment, the repair module 508 is further configured to:

In an optional embodiment, the first check packet data is determined by formula (1):

wherein p1,1 represents first check packet data; a1, j represents first original subpackage data; k represents the number of system child nodes; j belongs to {1,2 \8230; k } represents system node subpackaging;

accordingly, the second calibration packet data is determined by equation (2):

wherein p2,1 represents the second check packet data; a2, j represent the second original packetization data.

In an optional embodiment, the third parity packetization data is determined by equation (3):

wherein p1 and r represent third check sub-packet data;

accordingly, the fourth parity data is determined by equation (4):

wherein p2, r represents fourth check sub-packet data; a1, iota represents second partial packet data; ay, j denotes second global packetization data.

In order to improve data storage efficiency and reduce storage cost in a scenario supporting data fault tolerance, a data processing apparatus provided in this specification may create a check node based on attribute information of a system node in a distributed system, and store check data generated according to first subdata and second subdata stored in the system node to the check node, so that creation of the check node may be controlled by the system node attribute, and a purpose of reducing storage cost is achieved. And then if the failure node exists in the distributed system, determining a target system node having a repair relation with the failure node in the distributed system, extracting target original data stored by the target system node and check data stored by the check node, generating failure original data corresponding to the failure node by combining the target original data and the check data, and storing the failure original data in the failure node to repair the data in the failure node. In the process, the data repair processing is completed by determining the target system node with the repair relation and combining the check node, so that the repair is completed under the condition of less system nodes, the repair bandwidth of the system nodes can be effectively reduced, and the storage efficiency and the reliability of the distributed system are effectively improved.

The above is a schematic configuration of a data processing apparatus of the present embodiment. It should be noted that the technical solution of the data processing apparatus belongs to the same concept as the technical solution of the data processing method, and for details that are not described in detail in the technical solution of the data processing apparatus, reference may be made to the description of the technical solution of the data processing method.

FIG. 6 illustrates a block diagram of a computing device 600 provided in accordance with one embodiment of the present specification. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is coupled to the memory 610 via a bus 630 and a database 650 is used to store data.

Computing device 600 also includes access device 640, access device 640 enabling computing device 600 to communicate via one or more networks 660. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 640 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 600, as well as other components not shown in FIG. 6, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device structure shown in FIG. 6 is for illustration purposes only and is not intended to limit the scope of the present description. Other components may be added or replaced as desired by those skilled in the art.

Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 600 may also be a mobile or stationary server.

Wherein the processor 620 is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of the data processing method described above.

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which when executed by a processor implement the steps of the data processing method described above.

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data processing method, and for details that are not described in detail in the technical solution of the storage medium, reference may be made to the description of the technical solution of the data processing method.

An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the data processing method.

The above is a schematic scheme of a computer program of the present embodiment. It should be noted that the technical solution of the computer program and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the data processing method.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, and software distribution medium, etc. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Furthermore, those skilled in the art will appreciate that the embodiments described in this specification are presently preferred and that no acts or modules are required in the implementations of the disclosure.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the teaching of the embodiments of the present disclosure. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of data processing, comprising:

2. The method of claim 1, the creating hierarchically identical check nodes based on attribute information of system nodes, comprising:

acquiring attribute information of system nodes in the distributed system;

determining the number of system nodes contained in the distributed system according to the attribute information;

creating a set number of check nodes according to preset check node creating conditions and the number of the system nodes; wherein the hierarchical structure of the check nodes and the system nodes is the same.

3. The method of claim 1, the generating of the verification data, comprising:

determining a first system sub-node and a second system sub-node in the system node;

generating first check subdata according to first original subdata stored in the first system sub-node;

generating second check sub data according to preset check parameters and second original sub data stored in the second system sub node;

generating the check data according to the first check subdata and the second check subdata;

and the first original subdata and the second original subdata form original data stored by the system node.

4. The method of claim 3, the determining of the first and second system sub-nodes in the system node, comprising:

taking a syndrome node in the check nodes having a direct check relationship with the system node as a first syndrome node, and taking a syndrome node in the check nodes having an indirect check relationship with the system node as a second syndrome node;

determining the first system child node in the system node according to the first syndrome node, and determining the second system child node in the system node according to the second syndrome node.

5. The method of claim 4, wherein, in the case that the number of packets of a system sub-node in the system node and the number of packets of a check sub-node in the check node are both two, the generating first check sub-data according to the first original sub-data stored in the first system sub-node comprises:

determining a first system node sub-packet in the first system sub-node, and generating first check sub-packet data corresponding to a first check node sub-packet in the first check sub-node according to first original sub-packet data stored in the first system node sub-packet;

determining a second system node sub-packet in the first system sub-node, and generating second check sub-packet data corresponding to a second check node sub-packet in the first check sub-node according to second original sub-packet data stored in the second system node sub-packet;

the first original sub-packet data and the second original sub-packet data form the first original sub-data, and the first check sub-packet data and the second check sub-packet data form the first check sub-data.

6. The method of claim 5, wherein generating second parity data according to the preset parity parameters and second primitive data stored in the second system sub-node comprises:

determining, in the second system child node, a first global system node sub-packet and a first local system node sub-packet, and a second global system node sub-packet and a second local system node sub-packet;

generating third check sub-packet data corresponding to a third check node sub-packet in the second check sub-node according to preset check parameters, first global sub-packet data stored in the first global system node sub-packet and first local sub-packet data stored in the first local system node sub-packet;

generating fourth check sub-packet data corresponding to a fourth check node sub-packet in the second check sub-node according to preset check parameters, second global sub-packet data stored in the second global system node sub-packet and second local sub-packet data stored in the second local system node sub-packet;

wherein the third check sub-packet data and the fourth check sub-packet data constitute the second check sub-data.

7. The method according to any one of claims 1-6, wherein in case the number of failed nodes is one, the determining a target system node having a repair relationship with the failed node in the distributed system comprises:

determining at least two candidate system nodes of the check node in a stage of establishing the check node;

and taking the candidate system nodes except the failed node in the at least two candidate system nodes as target system nodes having a repair relation with the failed node.

8. The method of claim 7, the generating stale raw data from the target raw data and the check data, comprising:

determining first target original subdata and second target original subdata in the target original data, and determining third check subdata and fourth check subdata in the check data; the data volume of the second target original subdata is smaller than that of the first target original subdata;

generating first failure original subdata corresponding to first failure node sub packets in the failure nodes according to the first target original subdata and the third check subdata;

generating second failure original subdata corresponding to second failure node sub-packets in the failure nodes according to the second target original subdata and the fourth check subdata;

and generating the failure original data according to the first failure original subdata and the second failure original subdata.

9. The method of claim 6, the first parity packetization data determined by equation (1):

wherein, p1,1 represents the first check sub-packet data; a1, j represents first original subpackage data; k represents the number of system child nodes; j belongs to {1,2 \8230, k } and represents system node sub-package;

accordingly, the second check packet data is determined by equation (2):

10. The method of claim 9, the third parity packetization data determined by equation (3):

wherein p1 and r represent third check sub-packet data;

representing a verification parameter; ax, j denotes first global packetization data; m represents the local system node sub-package number; r belongs to {2,3 \8230, r } represents check node sub-packaging; iota represents local system node sub-package; a2, iota represents the first local packet data;

correspondingly, the fourth check packet data is determined by formula (4):

11. A data processing apparatus comprising:

the system comprises a creating module, a judging module and a judging module, wherein the creating module is configured to create check nodes with the same hierarchical structure based on attribute information of system nodes and store check data into the check nodes, and the check data are generated according to first subdata and second subdata stored in the system nodes;

12. A computing device, comprising:

a memory and a processor;

the memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions, which when executed by the processor, implement the steps of the method of any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 10.