CN109426587B

CN109426587B - Data recovery method and device

Info

Publication number: CN109426587B
Application number: CN201710742705.4A
Authority: CN
Inventors: 叶敏; 林鹏; 汪渭春; 林起芊
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2017-08-25
Filing date: 2017-08-25
Publication date: 2020-08-28
Anticipated expiration: 2037-08-25
Also published as: CN109426587A; WO2019037587A1

Abstract

The embodiment of the invention discloses a data recovery method and a device, wherein in the scheme, a storage space is divided into a plurality of storage units, and metadata of the storage space comprises index information of the plurality of storage units; when the metadata is restored, the index information of the storage unit contained in the storage space pointed by the metadata to be restored is read, the metadata is restored by using the read index information, and then the data can be read by using the metadata.

Description

Data recovery method and device

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data recovery method and apparatus.

Background

Erasure Coding (EC) is a data protection method, and an erasure coding strategy can be represented by n ═ k + m, where k represents the number of original fragments, m represents the number of redundant fragments, and n represents the total number of data fragments. Specifically, the data to be stored can be divided into k original segments, and expansion and encoding are performed based on the k original segments to obtain m redundant segments; respectively storing the k original fragments and the m redundant fragments into each storage unit; thus, if a storage unit fails to cause data loss, the lost data segment can be recovered according to the original segment and the redundant segment which are not lost.

In the above scheme, after the erasure code strategy is used to store the data to be stored, the metadata corresponding to the data is usually recorded. Metadata, that is, data describing data (data about data), is mainly related information describing data attributes, and is used to support functions such as indicating storage locations, history data, resource searching, file recording, and the like. When the data is subsequently read, the metadata can be utilized to combine the various pieces of data into the data.

If the metadata is lost due to system abnormality or hardware damage, the data corresponding to the lost metadata cannot be read.

Disclosure of Invention

An object of the embodiments of the present invention is to provide a data recovery method and apparatus, so as to recover metadata after the metadata is lost.

In order to achieve the above object, an embodiment of the present invention discloses a data recovery method, which is applied to a management server in a distributed system, and the method includes:

determining a storage space pointed by metadata to be recovered, wherein the storage space is located in a storage server of the distributed system;

determining a storage unit contained in the storage space based on pre-acquired data structure information; wherein, the data structure information includes: the corresponding relation between the storage space and each storage unit contained in the storage space; the storage unit is the minimum unit for storing the data fragments, a plurality of storage units form a storage space, and the metadata of the storage space comprises the index information of the storage units;

reading the index information of the determined storage unit; the read index information comprises a first identifier of a storage space where the corresponding storage unit is located and a first attribute of the data fragment stored in the corresponding storage unit;

determining the first identifier as a storage space identifier in each piece of index information contained in the metadata to be restored;

determining the attribute of the data fragment in each piece of index information contained in the metadata to be recovered according to the first attribute and a preset data storage strategy;

and recovering to obtain the metadata to be recovered based on the storage space identification and the data segment attribute in each piece of index information contained in the metadata to be recovered.

In order to achieve the above object, an embodiment of the present invention further discloses a data recovery apparatus, which is applied to a management server in a distributed system, and the apparatus includes:

the first determination module is used for determining a storage space pointed by metadata to be recovered, and the storage space is positioned in a storage server of the distributed system;

the second determining module is used for determining a storage unit contained in the storage space based on the pre-acquired data structure information; wherein, the data structure information includes: the corresponding relation between the storage space and each storage unit contained in the storage space; the storage unit is the minimum unit for storing the data fragments, a plurality of storage units form a storage space, and the metadata of the storage space comprises the index information of the storage units;

the first reading module is used for reading the index information of the determined storage unit; the read index information comprises a first identifier of a storage space where the corresponding storage unit is located and a first attribute of the data fragment stored in the corresponding storage unit;

a third determining module, configured to determine the first identifier as a storage space identifier in each piece of index information included in the metadata to be restored;

a fourth determining module, configured to determine, according to the first attribute and a preset data storage policy, a data fragment attribute in each piece of index information included in the metadata to be restored;

and the first recovery module is used for recovering and obtaining the metadata to be recovered based on the storage space identifier and the data segment attribute in each piece of index information contained in the metadata to be recovered.

In order to achieve the above object, the embodiment of the present invention further discloses a management server, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing any data recovery method when executing the program stored in the memory.

In order to achieve the above object, an embodiment of the present invention further discloses a distributed system, including: the data fragment storage system comprises a storage server and a management server, wherein the storage server comprises at least one storage space, each storage space comprises a plurality of storage units, and each storage unit is the minimum unit for storing data fragments;

the storage server is used for storing the index information of each storage unit in the storage server;

the management server is used for determining a storage space pointed by metadata to be recovered, and the storage space is positioned in a storage server of the distributed system; determining a storage unit contained in the storage space based on pre-acquired data structure information; wherein, the data structure information includes: the corresponding relation between the storage space and each storage unit contained in the storage space; the metadata of one storage space comprises index information of a plurality of storage units in the one storage space; reading the index information of the determined storage unit; the read index information comprises a first identifier of a storage space where the corresponding storage unit is located and a first attribute of the data fragment stored in the corresponding storage unit; determining the first identifier as a storage space identifier in each piece of index information contained in the metadata to be restored; determining the attribute of the data fragment in each piece of index information contained in the metadata to be recovered according to the first attribute and a preset data storage strategy; and recovering to obtain the metadata to be recovered based on the storage space identification and the data segment attribute in each piece of index information contained in the metadata to be recovered.

In order to achieve the above object, an embodiment of the present invention further discloses a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any of the above data recovery methods.

By applying the embodiment of the invention, one storage space is divided into a plurality of storage units, and the metadata of the storage space comprises the index information of the plurality of storage units; when the metadata is restored, the index information of the storage unit contained in the storage space pointed by the metadata to be restored is read, the metadata is restored by using the read index information, and then the data can be read by using the metadata.

Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a distributed system according to an embodiment of the present invention;

fig. 2 is a first flowchart of a data recovery method according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of index information according to an embodiment of the present invention;

fig. 4 is a schematic flowchart of a second data recovery method according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a data recovery apparatus according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of a management server according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In order to solve the above technical problem, embodiments of the present invention provide a data recovery method and apparatus, where the method may be applied to a management server in a distributed system. A distributed system may be as shown in fig. 1, including a storage server that stores data and a management server that stores metadata. The metadata, that is, data (data about data) describing the data, here mainly describes attribute information of the data in the storage server. The data in the storage server can only be read by the metadata in the management server. If the management server fails, resulting in loss of metadata, the data stored in the storage server cannot be read. By using the embodiment of the invention, the metadata lost in the management server can be recovered.

First, a data recovery method provided by an embodiment of the present invention is described in detail below.

Fig. 2 is a schematic flow chart of a data recovery method according to an embodiment of the present invention, including:

s201: and determining a storage space pointed by the metadata to be restored, wherein the storage space is positioned in a storage server of the distributed system.

The storage space in this embodiment is a Bucket in the storage server, and one Bucket may correspond to only one erasure code policy. For example, a storage server may include a plurality of buckets: bucket1 and Bucket 2; data stored using erasure coding policy 4+1 may be written to Bucket1 and data stored using erasure coding policy 4+2 may be written to Bucket 2. Or the storage server comprises Bucket1, Bucket2, Bucket3 and Bucket 4; data stored using erasure code policy 4+1 may be written to Bucket1 or Bucket2, and data stored using erasure code policy 4+2 may be written to Bucket3 or Bucket 4. The specific storage manner is not limited.

In the embodiment of the present invention, the metadata has a corresponding relationship with a Bucket, the metadata may describe data stored in the corresponding Bucket, and the metadata may also be referred to as metadata of the Bucket, or the metadata points to the Bucket. For example, as shown in fig. 1, if metadata 1 corresponds to Bucket1, then metadata 1 describes data stored in Bucket1, and metadata 1 points to Bucket 1; if there is a corresponding relationship between metadata 2 and Bucket2, metadata 2 describes data stored in Bucket2, and metadata 2 points to Bucket1 … … in the embodiment of the present invention, only recovering metadata of one Bucket is taken as an example for explanation.

After detecting that the metadata is lost, the user can send a metadata recovery instruction to the management server, and the management server determines a storage space to which the metadata to be recovered points according to the received instruction; or, the management server may also automatically detect whether metadata is lost, and if so, determine a storage space to which the metadata to be recovered points; or, the other devices may also detect whether metadata is lost in the management server, and if so, send a metadata recovery instruction to the management server, and the management server determines, according to the received instruction, a storage space to which the metadata to be recovered points.

S202: and determining the storage units contained in the storage space based on the data structure information acquired in advance. Wherein, the data structure information includes: the corresponding relation between the storage space and each storage unit contained in the storage space; the storage unit is the minimum unit for storing the data fragments, a plurality of storage units form a storage space, and the metadata of the storage space comprises the index information of the plurality of storage units.

In the embodiment of the invention, each Bucket in the storage server is divided into a plurality of storage units. The storage Unit is a Unit, and the Unit is the minimum Unit for storing the data segment. After dividing a packet into a plurality of units, the division condition may be recorded as data structure information, where the data structure information includes a correspondence relationship between the packet and each Unit included in the packet. The management server may obtain the data structure information in advance, and determine, based on the data structure information, a Unit included in a Bucket to which the metadata to be restored points.

Each storage Unit corresponds to index information, the index information is called a key or a Unit key, and the key can contain an identifier of a Bucket where the key is located, an attribute of a data fragment, and the like. The identifier of the Bucket may be the name of the Bucket or others, and is not limited specifically; the attribute of the data segment may include an identifier of the corresponding target data, a data amount of the target data, a corresponding erasure code policy, and the like, which is not limited specifically. The target data refers to complete data corresponding to the data fragments stored in the Unit, that is, data before segmentation or data after combination.

The metadata of a Bucket includes keys of a plurality of units into which the Bucket is divided. For example, as shown in fig. 1, metadata 1 describes data stored in Bucket1, and assuming that Bucket1 is divided into 6 units, metadata 1 contains keys of the 6 units.

S203: and reading the index information of the determined storage unit. The read index information includes a first identifier of a storage space where the corresponding storage unit is located and a first attribute of the data segment stored by the corresponding storage unit.

In the embodiment of the invention, the index information of the storage unit can be stored in the storage server, so that even if the management server fails, the index information in the storage server can be read, and the metadata in the management server can be recovered. Alternatively, it is also reasonable that the index information of the storage unit is stored in the management server, and the metadata and the index information in the management server are stored in different areas. Alternatively, the index information and the metadata may be stored in the same location, and the metadata may be recovered by using only part of the index information by applying the embodiment of the present invention. The specific storage location of the index information is not limited.

In S203, one key may be read, or a plurality of keys may be read. As an implementation manner, a key of each Unit included in the Bucket to which the metadata points may be read, and since the metadata includes the key of each Unit, each read key may be directly copied, so that the metadata to be restored is obtained.

As another embodiment, a key of the partial Unit included in the Bucket, to which the metadata points, may also be read, where the "key of the partial Unit" may be one or more keys. In this case, the metadata to be restored may be restored based on the read key.

For convenience of description, the Bucket identifier included in the key read in S203 is referred to as a first identifier, and the attribute of the data fragment included in the key read in S203 is referred to as a first attribute.

S204: and determining the first identifier as a storage space identifier in each piece of index information contained in the metadata to be restored.

According to the foregoing description, multiple keys included in one piece of metadata belong to the same Bucket, and therefore, Bucket identifiers included in the multiple keys are the same and are all the first identifiers.

S205: and determining the data fragment attribute in each piece of index information contained in the metadata to be recovered according to the first attribute and a preset data storage strategy.

As described above, the attributes of a data fragment may include an identification of the corresponding target data, a data amount of the target data, a corresponding erasure coding policy, and so on. The identification of the target data in the key of the Unit where the multiple data fragments obtained by splitting one piece of target data are located is the same as the data volume of the target data, and the erasure code strategies corresponding to the same Bucket are the same. Therefore, the attributes of the data segments in the plurality of keys can be recovered based on the attributes of the data segments in one key.

S206: and recovering to obtain the metadata to be recovered based on the storage space identification and the data segment attribute in each piece of index information contained in the metadata to be recovered.

As described above, since the metadata includes the key pointing to each Unit in the Bucket, the information in each key is determined, and the metadata is recovered.

In one embodiment, the metadata of a Bucket may include one or more pieces of Stripe data (strips), each piece of Stripe data including keys of units. The Stripe data may be a Stripe table, or may be in other forms, and is not limited in particular.

The data structure information in this embodiment includes a correspondence relationship between a packet and each Stripe included therein, and a correspondence relationship between each Stripe and each Unit included therein.

In this embodiment, the key includes not only the identifier of the Bucket where the key is located and the attribute of the data fragment, but also the identifier of the corresponding stripe data. For convenience of description, the identification of the slice data contained in the key read in S203 is referred to as a second identification.

In this embodiment, before S206, the method may further include: determining a stripe identifier in each piece of index information contained in the metadata to be restored according to the second identifier or the second identifier and a preset stripe identifier rule; s206 includes: and recovering to obtain at least one piece of stripe data based on the storage space identifier, the data fragment attribute and the stripe identifier in each piece of index information contained in the metadata to be recovered.

The metadata includes one or more pieces of stripe data, and the metadata is recovered by recovering the one or more pieces of stripe data.

If the Bucket pointed by the metadata to be restored only contains one piece of stripe data, the stripe identification in each key contained in the metadata is the same and is the second identification. The identification of the strip in each key contained in the metadata can be determined only from the second identification.

And if the Bucket pointed by the metadata to be restored contains multiple pieces of strip data, determining the strip identification in each key contained in the metadata according to the second identification and a preset strip identification rule.

For example, the target data X is large, X is divided into a plurality of slices for storage, and it is assumed that X is divided into three pieces of data X1, X2, and X3, X1 corresponds to slice data S1, X2 corresponds to slice data S2, and X3 corresponds to slice data S3. Assuming that only the stripe marks S1 and S3 are read in S203 and the preset stripe mark rule is that the marks are marked according to the order of numbers from small to large, it can be determined that the stripe mark in the key of the Unit storing X2 is S2.

In this embodiment, the data segment attributes may include an in-band sequence number and a data offset in addition to the identifier of the target data, the data amount of the target data, and the corresponding erasure correction code policy. For convenience of description, an identifier of target data included in the first attribute is referred to as a third identifier, a data amount of the target data included in the first attribute is referred to as a first data amount, an erasure coding policy included in the first attribute is referred to as a first erasure coding policy, an intra-stripe sequence number included in the first attribute is referred to as a first intra-stripe sequence number, and a data offset included in the first attribute is referred to as a first data offset; in this case, S205 may include:

determining a target data identifier in each piece of index information contained in the metadata to be recovered according to the third identifier;

determining the data volume of target data in each piece of index information contained in the metadata to be restored according to the first data volume;

determining the first erasure code strategy as an erasure code strategy in each piece of index information contained in the metadata to be recovered;

determining the intra-strip sequence number in each piece of index information contained in the metadata to be recovered according to the first intra-strip sequence number and a sequencing rule in a preset data storage strategy;

and determining the data offset in each piece of index information contained in the metadata to be recovered according to the first data offset and an offset rule in a preset data storage strategy.

For example, assume that a packet corresponds to an erasure coding policy of 4+1, and each Stripe in the packet corresponds to 5 units with a capacity of 1 MB. And storing the data A, wherein the data A to be stored is 11.5MB, and the target data volume is 11.5 MB. Dividing the data A into three parts of data A1(4MB), A2(4MB) and A3(3.5MB), wherein each part of data corresponds to a Stripe; and storing the three data by using an erasure code strategy 4+ 1. Taking A1 as an example:

allocating stripes first:

{ < OSD _1, wwn _1>, < OSD _2, wwn _2>, < OSD _3, wwn _3>, < OSD _4, wwn _4>, < O SD _5, wwn _5> }, where OSD is a storage server, OSD _1, OSD _2, etc. are storage server identifications, wwn can be disks, wwn _1, wwn _2, etc. can be disk identifications.

According to the erasure code strategy 4+1, slicing and redundancy processing are performed on the data a1 to obtain an original fragment and a redundant fragment. Each data segment in the stripe corresponds to a quadruple < OSD, wwn, key, value >, wherein key represents the index information of the Unit where the data segment is located, and value represents the value or content of the data segment.

The stripe data corresponding to the record data a1 is:

{<OSD_1,wwn_1,key_1,value_1>,<OSD_2,wwn_2,key_2,value_2>,<OSD_3,wwn_3,key_3,value_3>,<OSD_4,wwn_4,key_4,value_4>,<OSD_5,wwn_5,key_5,value_5>}。

in some embodiments, the same disk may belong to different storage servers at different times, and therefore, in this case, the storage server identifier may not be included in the recorded stripe, that is, the recorded stripe data may also be:

{<wwn_1,key_1>,<wwn_2,key_2>,<wwn_3,key_3>,<wwn_4,key_4>,<wwn_5,key_5>}。

key _1 and key _2 in the stripe data are index information in this embodiment, and the index information may be as shown in fig. 3, and include: storage space identifier (Bucket), Stripe identifier (Stripe-id), attribute of data fragment { identifier (Object-id) of corresponding target data, erasure code policy (EC-type), sequence number within Stripe (EC-seq), data Offset (Offset of Stripe in data a), data size of target data (Object _ len, data size of complete data corresponding to Stripe data) }, and the like, without limitation.

The data offset may represent a corresponding position of the stripe in the target data, for example, the stripe data offset corresponding to a1 may be 0MB, the stripe data offset corresponding to a2 may be 4MB, and the stripe data offset corresponding to A3 may be 8MB, which indicates that when the data a is divided into three copies of data, the three copies of data have a sequence of a1, a2, and A3. Alternatively, it is reasonable to indicate the precedence order by a strip identifier.

The storage processes of A2 and A3 are similar and are not described in detail. The metadata of the Bucket may contain these three pieces of stripe data: the stripe data corresponding to A1, the stripe data corresponding to A2, and the stripe data corresponding to A3.

The process of recovering metadata using keys is described as follows:

assume that the key includes: the storage space name (Bucket), the data identification (Object-id), the Stripe identification (Stripe-id), the corresponding erasure coding strategy (EC-type), the intra-Stripe sequence number (EC-seq), the data Offset (Offset, the Offset of the Stripe in the data A), and the size of the target data volume (Object _ len, the data volume of the complete data corresponding to the Stripe data).

Continuing with the above example, assume that the above generated metadata, i.e., the three pieces of stripe data, is lost. It is assumed that one key read in S203 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 1, Offset: 0MB, Object _ len: 11.5 MB.

The first identifier of a Bucket in the key is 1, and thus, the Bucket identifier in each key included in the metadata is 1. The third identifier of the target data in the key is a, and therefore, the target data in each key contained in the metadata is identified as a. The first erasure code policy in this key is 4+1, and therefore the erasure code policy in each key contained in the metadata is 4+ 1. The first data amount in this key is 11.5MB, and therefore, the data amount of the target data in each key included in the metadata is 11.5 MB.

And an erasure code strategy of 4+1 indicates that one piece of stripe data corresponds to 5 units, wherein 4 units store original data fragments, 1 Unit stores redundant data fragments, and the maximum data amount of the data in the management server, which points to the data in the storage server, is 4MB assuming that the capacity of one Unit is 1 MB. Therefore, it is determined that three pieces of stripe data need to be allocated to store the target data a of 11.5MB, and when the metadata is restored, three pieces of stripe data are also restored.

The second identifier of the stripe data in the key is S1, and assuming that the preset stripe identification rule is that the sequence numbers start to be +1 from 1, it is determined that the identifiers of the three pieces of stripe data are: s1, S2 and S3.

Since the amount of data of one piece of stripe data in the management server pointing to the storage data in the storage server is 4MB at most, and the first data offset in the key is 0MB, it is assumed that the offset rule in the preset data storage policy is: "the data offset gradually increases from small to large according to the stripe identifier", the data offsets of the three stripe data are determined to be: the data offset of S1 is 0MB, the data offset of S2 is 4MB, and the data offset of S3 is 8 MB.

The first intra-strip serial number in the key is 1, and assuming that the sequence rule in the preset data storage strategy is sequence +1, the intra-strip serial numbers of the key in each piece of strip data are determined to be: 1. 2, 3, 4 and 5.

Assuming that the storage space corresponding to the metadata to be restored is located in the disk wwn _1, it is determined that the three pieces of stripe data are specifically:

the first part:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, wherein key _1 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 1, Offset: 0MB, Object _ len: 11.5 MB; key _2 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 2, Offset: 0MB, Object _ len: 11.5 MB; key _3 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 3, Offset: 0MB, Object _ len: 11.5 MB; key _4 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 4, Offset: 0MB, Object _ len: 11.5 MB; key _5 includes: bucket: 1, Object-id: a, Stripe-id: s1, EC-type: 4+1, EC-seq: 5, Offset: 0MB, Object _ len: 11.5 MB.

And (2) second part:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, wherein key _1 includes: bucket: 1, Object-id: a, Stripe-id: s2, EC-type: 4+1, EC-seq: 1, Offset: 4MB, Object _ len: 11.5 MB; key _2 includes: bucket: 1, Object-id: a, Stripe-id: s2, EC-type: 4+1, EC-seq: 2, Offset: 4MB, Object _ len: 11.5 MB; key _3 includes: bucket: 1, Object-id: a, Stripe-id: s2, EC-type: 4+1, EC-seq: 3, Offset: 4MB, Object _ len: 11.5 MB; key _4 includes: bucket: 1, Object-id: a, Stripe-id: s2, EC-type: 4+1, EC-seq: 4, Offset: 4MB, Object _ len: 11.5 MB; key _5 includes: bucket: 1, Object-id: a, Stripe-id: s2, EC-type: 4+1, EC-seq: 5, Offset: 4MB, Object _ len: 11.5 MB.

And (3) third part:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, wherein key _1 includes: bucket: 1, Object-id: a, Stripe-id: s3, EC-type: 4+1, EC-seq: 1, Offset: 8MB, Object _ len: 11.5 MB; key _2 includes: bucket: 1, Object-id: a, Stripe-id: s3, EC-type: 4+1, EC-seq: 2, Offset: 8MB, Object _ len: 11.5 MB; key _3 includes: bucket: 1, Object-id: a, Stripe-id: s3, EC-type: 4+1, EC-seq: 3, Offset: 8MB, Object _ len: 11.5 MB; key _4 includes: bucket: 1, Object-id: a, Stripe-id: s3, EC-type: 4+1, EC-seq: 4, Offset: 8MB, Object _ len: 11.5 MB; key _5 includes: bucket: 1, Object-id: a, Stripe-id: s3, EC-type: 4+1, EC-seq: 5, Offset: 8MB, Object _ len: 11.5 MB.

These three pieces of stripe data are the metadata of Bucket 1.

In one embodiment of the present invention, the metadata may include target data information in addition to the stripe data. In the above example, the data a is divided into three pieces of data a1, a2, and A3, each piece of data corresponding to one piece of stripe data; in this case, a piece of target data information indicating which pieces of strip data a complete piece of data correspond to may be stored for the data a.

In this embodiment, after at least one piece of stripe data is recovered based on the storage space identifier, the data segment attribute, and the stripe identifier in each piece of index information included in the metadata to be recovered, stripe data with the same target data identifier in the metadata to be recovered may be further combined into one piece of target data information.

It can be understood that the stripe data with the same target data identifier corresponds to the same complete data, the stripe data with the same target data identifier forms a piece of target data information, and when the data is subsequently read, the stripe data forming the data can be obtained only by searching the piece of target data information, so that the reading efficiency is improved.

With the embodiment of the present invention shown in fig. 2, a storage space is divided into a plurality of storage units, and metadata of the storage space includes index information of the plurality of storage units; when the metadata is restored, the index information of the storage unit contained in the storage space pointed by the metadata to be restored is read, the metadata is restored by using the read index information, and then the data can be read by using the metadata.

In one embodiment, the metadata includes a plurality of pieces of stripe data, each piece of stripe data including index information of a plurality of storage units. If a system exception occurs or hardware is damaged, the index information in the stripe data may be lost.

An embodiment of the present invention further provides a stripe data recovery method, as shown in fig. 4, including:

s401: incomplete stripe data is determined.

In the embodiment shown in fig. 4, the incomplete slice data is the slice data with lost index information.

As an embodiment, when determining incomplete stripe data, target stripe data with lost index information may be determined first; judging whether the target stripe data is fully written: if the target stripe data is full, directly determining the target stripe data as incomplete stripe data; if the storage unit corresponding to the lost index information in the target stripe data is not fully written, further judging whether the storage unit corresponding to the lost index information in the target stripe data is an empty storage unit, and if not, determining the target stripe data as incomplete stripe data.

As can be understood by those skilled in the art, if all storage units corresponding to a slice of stripe data are fully written, it indicates that the slice of stripe data is fully written, and the slice of stripe data is full stripe data; conversely, if the storage units corresponding to a slice of stripe data are not fully written, it indicates that the slice of stripe data is not fully written, and the storage units in which data is not written in the not fully written stripe data are called empty storage units.

If the target stripe data is not full stripe data and the index information lost in the not full stripe data is index information of an empty storage unit, in this case, there is no data loss, and the user can read data normally, and it is not necessary to execute the present scheme to perform data recovery, so that the not full stripe data is not determined to be incomplete stripe data.

S402: based on the data structure information, the storage unit pointed to by the incomplete stripe data is determined as a target storage unit.

The data structure information includes the corresponding relationship between the stripe data and each Unit included in the stripe data, and therefore, the Unit pointed to by the incomplete stripe data can be determined as the target Unit based on the data structure information. The target Unit and the Unit of the lost key correspond to the same stripe data, namely the incomplete stripe data.

As an embodiment, before performing S402, the number of index information lost in the incomplete stripe data may be counted; judging whether the number is larger than m, wherein m is the number of redundant fragments in the erasure code strategy corresponding to the incomplete stripe data; if not, executing S402 again; if so, reconstructing target data corresponding to the incomplete strip data; storing the target data by using a preset erasure code strategy k + m; and obtaining new stripe data corresponding to the target data.

In general, the erasure coding strategy can be represented by n ═ k + m, where k represents the number of original fragments, m represents the number of redundant fragments, and n represents the total number of data fragments. It will be understood by those skilled in the art that if the number of missing index information in a slice of stripe data is greater than m, the missing index information generally cannot be recovered, and in this case, other information present in the stripe data may be used for data reconstruction.

Assuming that the erasure code strategy corresponding to the data a is 4+1, the original stripe data is:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, where key _2, key _3 are lost and the incomplete stripe data is:

{<wwn_1,key_1>,<wwn_2,>,<wwn_3,>,<wwn_4,key_4>,<wwn_5,key_5>}。

the number 2 of the lost index information is greater than m (1), in this case, the data segments which are not lost are read by using other information existing in the stripe data, and data reconstruction is performed by using the read data segments to obtain target data, which is reconstructed data a. Data a may then be restored using erasure coding policy 4+1, which, in turn, regenerates new stripe data for data a, similar to the process described above. The new stripe data may be generated by the storage server from which the management server obtains the new stripe data, or the new stripe data may be generated by the management server.

On the other hand, the erasure code policy corresponding to the data a is 4+1, and the original stripe data is: { < stripe _ id, wwn _1, key _1>, < stripe _ id, wwn _2, key _2>, < stripe _ id, wwn _3, key _3>, < stripe _ id, wwn _4, key _4>, < stripe _ id, wwn _5, key _5> }, where key _2 is lost and incomplete stripe data is:

{<stripe_id,wwn_1,key_1>,<stripe_id,wwn_2,>,<stripe_id,wwn_3,key_3>,<stripe_id,wwn_4,key_4>,<stripe_id,wwn_5,key_5>}。

the number 1 of lost index information is not more than m (1), and in this case, S402 and subsequent steps are performed.

S403: and reading the index information of the target storage unit to obtain a target storage space identifier, a target strip identifier and a target attribute of the data fragment.

As shown in fig. 3, the index information (key) may include a storage space identifier (Bucket), a Stripe identifier (Stripe-id), an attribute of the data fragment { an identifier (Object-id) of the corresponding destination data, an erasure code policy (EC-type), an intra-Stripe sequence number (EC-seq), a data Offset (Offset, Offset of the Stripe in the data a), a data size of the destination data (Object _ len, data size of the complete data corresponding to the Stripe data) }, and the like, which is not limited specifically.

For convenience of description, the storage space identifier read in S403 is referred to as a target storage space identifier, the stripe identifier read in S303 is referred to as a target stripe identifier, and the attribute of the data segment read in S403 is referred to as a target attribute.

S404: and determining the target storage space identification as the storage space identification in the lost index information.

It can be understood that the lost key and the key read in S403 belong to the same stripe data, and therefore, the same storage space identifier as that in the key read in S403 is the target storage space identifier.

S405: and determining the target stripe identification as the stripe identification in the lost index information.

It can be understood that the lost key and the key read in S403 belong to the same stripe data, and therefore, the lost key and the stripe identifier in the key read in S403 are the same and are both identified for the target stripe.

S406: and determining the attribute of the data fragment in the lost index information according to the target attribute.

As an embodiment, the data fragment attribute includes: the corresponding erasure code strategy, the sequence number in the strip, the data quantity of the corresponding target data and the data offset.

For convenience of description, the erasure code policy read in S403 is referred to as a target erasure code policy, the intra-stripe sequence number read in S403 is referred to as a target sequence number, the data amount read in S403 is referred to as a target data amount, and the data offset read in S402 is referred to as a target data offset.

S406 may include: determining the target erasure code strategy as an erasure code strategy in the lost index information;

determining the in-band sequence number in the lost index information according to the target sequence number and a preset sequencing rule;

determining the target data amount as a target data amount in the lost index information;

and determining the data offset in the lost index information according to the target data offset and a preset offset rule.

Assuming that the target erasure code policy read in S403 is 4+1, the target sequence number includes: 1. 3, 4, 5, the target data size is 4 MB.

It can be understood that (1), the erasure code strategies corresponding to the same piece of stripe data are the same, and therefore, the read target erasure code strategy 4+1 can be directly used as the erasure code strategy in the lost index information. (2) The sequencing rule in the preset data storage strategy can be understood as 1-5 sequence arrangement, 1, 3, 4 and 5 are read, and the sequence number in the strip corresponding to the lost index information can be determined to be 2. (3) Since the data amount of the target data corresponding to the same piece of stripe data is the same, the read target data amount of 4MB can be directly used as the data amount in the lost index information. (4) The offset rule in the preset data storage policy may be: each slice is offset by 4MB more than the previous slice, in this example, data a corresponds to only one slice, and the data offset for this slice is 0 MB.

S407: and recovering the incomplete stripe data based on the storage space identification, the stripe identification and the data segment attribute in the lost index information.

Continuing with the above example, it has been determined that the missing index information includes: erasure code strategy 4+1, serial number 2 in the stripe, data volume 4MB and data offset 0 MB; the determined contents are supplemented into the incomplete stripe data, and the recovery of the incomplete stripe data is completed.

Therefore, by applying the embodiment shown in fig. 4 of the present invention, incomplete stripe data, that is, stripe data with lost index information, is determined, and the lost index information is recovered and obtained by using the index information existing in the incomplete stripe data; therefore, if the index information is lost in the stripe data due to system abnormity or hardware damage and the like, the lost index information can be recovered by applying the scheme.

In one embodiment, after the lost index information is recovered, the stored data corresponding to the lost index information may be reconstructed.

In this embodiment, the data length corresponding to the lost index information may be determined; and reconstructing the stored data corresponding to the lost index information according to the data length and the lost index information.

The data length corresponding to the index information, that is, the data length of the data stored in the corresponding storage unit; determining the data length corresponding to the lost index information may include: and acquiring the capacity of a storage unit in advance, and determining the data length of the data stored in each lost storage unit according to the capacity of the storage unit and the target data volume, wherein the lost storage unit is the storage unit corresponding to the lost index information.

Those skilled in the art can understand that when the erasure code strategy is used for storing data, the data is generally divided averagely, and if the data cannot be divided averagely, the last bit is divided after being supplemented with 0; therefore, the data length in the memory cells other than the trailing memory cell is equal and equal to the memory cell capacity, and the data length of the trailing memory cell may be equal to, and may be smaller than, the memory cell capacity since the trailing memory cell may contain "0" data.

Based on this, in this embodiment, the memory cells corresponding to the stripe data are divided into a tail memory cell and a non-tail memory cell: for non-trailing memory cells, the corresponding data length is equal to the memory cell size, and for trailing memory cells, the corresponding data length may be equal to, and may be less than, the memory cell size.

Specifically, the data length corresponding to the tail storage unit can be calculated by the following formula: and Object _ len-4X Y + X, where Object _ len is the target data size, X is the storage unit capacity, and Y is the number of stripes corresponding to the stored complete data.

The tail memory cell in this embodiment refers to: when data is divided, the storage location where the data segment of "0" is located may be supplemented finally.

After the data length of the data stored in each lost storage unit is determined, the data stored in the lost storage unit, that is, the storage data corresponding to the lost index information, may be reconstructed by using the lost index information and the determined data length.

It can be understood that if the index information is lost in the stripe data, the data corresponding to the lost index information cannot be read. Therefore, after the lost index information is recovered, the storage data corresponding to the lost index information is reconstructed.

Assuming that the data volume of the data B is 4MB, the capacity of each storage unit is 1MB, and the original stripe data corresponding to the data B is as follows:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, where key _2 is lost, the lost storage unit is unit2, and the incomplete stripe data is: { < wwn _1, key _1>, < wwn _2, >, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }.

By applying the embodiment shown in fig. 4, the lost index information obtained by recovery includes: erasure code strategy 4+1, intra-stripe sequence number 2, and data volume 4 MB. From the sequence number 2 in the stripe, the lost memory cell is unit2, unit2 is a non-tail memory cell, and the data length corresponding to unit2 is 1MB (memory cell capacity).

Reading index information corresponding to storage units (unit1, unit3, unit4 and unit5) except for the lost storage unit in the stripe data, and further obtaining data corresponding to the index information; according to the read data, the recovered lost index information and the determined data length (1MB) corresponding to the unit2, reconstructing the stored data in the unit2, namely reconstructing the data corresponding to the lost index information.

By applying the embodiment, incomplete strip data, namely strip data with lost index information, is determined, the lost index information is recovered and obtained by utilizing the index information in the incomplete strip data, and the storage data corresponding to the lost index information is reconstructed; therefore, if the index information is lost in the strip data due to system abnormity or hardware damage and the like, the lost index information is restored by applying the scheme, and then the data corresponding to the index information is reconstructed, so that the data is restored, and the data corresponding to the strip data can be read.

In the above embodiment, for simplicity of explanation, the data B corresponds to one slice, and if the data B is large, it may correspond to a plurality of slices. For example, if one slice corresponds to 5 storage units with a capacity of 1MB and the data B to be stored is 11.5MB (target data amount), the data B may be divided into three data sets, i.e., data B1(4MB), data B2(4MB) and data B3(3.5MB), and the three data sets may be stored by using erasure coding policy 4+ 1. Taking B1 as an example, B1 is divided into 4 original fragments (each fragment is 1MB), and expansion and encoding are performed based on the 4 original fragments to obtain 1 redundant fragment; the 5 data segments (4 original segments and 1 redundant segment) are stored in the 5 storage units with the capacity of 1MB, respectively, and the 5 data segments correspond to one stripe.

Object _ len is 11.5MB, X is 1MB, Y is 3, and the data length corresponding to the tail memory cell is 11.5-4 × 1 × 3+1 — 0.5 MB. It can be understood that when B3 is stored, B3 is divided into 4 original fragments, the data length sizes of the 4 original fragments are 1MB, 1MB and 0.5MB respectively, and the 4 original fragments are expanded and encoded to obtain 1 redundant fragment, the data length size is 1 MB; the 5 data segments (4 original segments and 1 redundant segment) are stored in 5 storage units with a capacity of 1MB, and the 5 data segments correspond to one slice of stripe data.

In the embodiment of the present invention, the data length corresponding to each lost memory cell is determined: if the missing memory cell is a non-tail memory cell, the data length corresponding to the non-tail memory cell is 1MB (memory cell capacity), and if the missing memory cell is a tail memory cell, the data length corresponding to the tail memory cell is Object _ len-4X Y + X, which is 0.5 MB.

Assume that the original stripe data corresponding to B3 is:

{ < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, key _4>, < wwn _5, key _5> }, where key _4 is lost, the lost storage unit is unit4, and the incomplete stripe data corresponding to B3 is: { < wwn _1, key _1>, < wwn _2, key _2>, < wwn _3, key _3>, < wwn _4, >, < wwn _5, key _5> }.

By applying the embodiment shown in fig. 4, the lost index information obtained by recovery includes: erasure code strategy 4+1, intra-stripe sequence number 4, and data volume 4 MB. From the sequence number 4 in the stripe, the missing memory cell is unit4, unit4 is the tail memory cell, and the data length corresponding to unit4 is Object _ len-4 × Y + X ═ 0.5 MB.

Reading index information of storage units (unit1, unit2, unit3 and unit5) except for the lost storage unit in the stripe data, and further obtaining data corresponding to the index information; and reconstructing the stored data in the unit4 according to the read data, the recovered lost index information and the determined data length (0.5MB) corresponding to the unit4, namely reconstructing the data corresponding to the lost index information.

Corresponding to the foregoing method embodiment, an embodiment of the present invention further provides a data recovery apparatus, as shown in fig. 5, including:

a first determining module 501, configured to determine a storage space to which metadata to be restored points, where the storage space is located in a storage server of the distributed system;

a second determining module 502, configured to determine, based on data structure information acquired in advance, a storage unit included in the storage space; wherein, the data structure information includes: the corresponding relation between the storage space and each storage unit contained in the storage space; the storage unit is the minimum unit for storing the data fragments, a plurality of storage units form a storage space, and the metadata of the storage space comprises the index information of the storage units;

a first reading module 503, configured to read index information of the determined storage unit; the read index information comprises a first identifier of a storage space where the corresponding storage unit is located and a first attribute of the data fragment stored in the corresponding storage unit;

a third determining module 504, configured to determine the first identifier as a storage space identifier in each piece of index information included in the metadata to be restored;

a fourth determining module 505, configured to determine, according to the first attribute and a preset data storage policy, a data fragment attribute in each piece of index information included in the metadata to be restored;

a first restoring module 506, configured to restore to obtain the metadata to be restored based on the storage space identifier and the data segment attribute in each piece of index information included in the metadata to be restored.

As an embodiment, the data structure information may include: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units; the read index information also comprises a second identifier of the corresponding stripe data;

the apparatus may further include:

a fifth determining module (not shown in the figure), configured to determine, according to the second identifier, or the second identifier and a preset stripe identifier rule, a stripe identifier in each piece of index information included in the metadata to be restored;

the first recovery module 506 may specifically be configured to:

and recovering to obtain at least one piece of stripe data based on the storage space identifier, the data fragment attribute and the stripe identifier in each piece of index information contained in the metadata to be recovered.

As an implementation manner, the data fragment attribute includes an identifier of corresponding target data, a data amount of the target data, a corresponding erasure code policy, a sequence number in a stripe, and a data offset; the first attribute comprises a third identifier of corresponding target data, a first data volume of the corresponding target data, a first erasure code strategy, a first in-band sequence number and a first data offset;

the fourth determining module 505 may specifically be configured to:

As an embodiment, the metadata to be recovered further includes at least one piece of target data information, and the apparatus may further include:

and the combining module (not shown in the figure) is used for combining the stripe data with the same target data identification in the metadata to be recovered into one piece of target data information.

As an embodiment, the data structure information may include: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units;

the apparatus may further include: a sixth determining module, a seventh determining module, a second reading module, an eighth determining module, a ninth determining module, a tenth determining module, and a second restoring module (not shown in the drawings), wherein,

a sixth determining module, configured to determine incomplete stripe data, where the incomplete stripe data is stripe data with lost index information;

a seventh determining module, configured to determine, based on the data structure information, a storage unit to which the incomplete stripe data points, as a target storage unit;

the second reading module is used for reading the index information of the target storage unit to obtain a target storage space identifier, a target stripe identifier and a target attribute of the data fragment;

an eighth determining module, configured to determine the target storage space identifier as a storage space identifier in the lost index information;

a ninth determining module, configured to determine the target stripe identifier as a stripe identifier in the lost index information;

a tenth determining module, configured to determine, according to the target attribute, a data fragment attribute in the lost index information;

and the second recovery module is used for recovering the incomplete stripe data based on the storage space identifier, the stripe identifier and the data fragment attribute in the lost index information.

As an embodiment, the data fragment attribute includes a corresponding erasure code policy, a sequence number in a stripe, a data amount of corresponding target data, and a data offset; the target attribute comprises a target erasure code strategy, a target serial number, a target data volume and a target data offset;

the tenth determining module may be specifically configured to:

determining the target erasure code strategy as an erasure code strategy in the lost index information;

determining the target data amount as the data amount in the lost index information;

As an embodiment, the apparatus may further include: a statistics module and a decision module (not shown), wherein,

the counting module is used for counting the number of lost index information in the incomplete stripe data;

the judging module is used for judging whether the number is larger than m, wherein m is the number of redundant fragments in the erasure code strategy corresponding to the incomplete strip data; and if not, triggering the seventh determining module.

As an embodiment, the apparatus may further include: a first reconstruction module, a storage module, and an acquisition module (not shown), wherein,

the first reconstruction module is used for reconstructing target data corresponding to the incomplete strip data under the condition that the judgment module judges that the number is larger than m;

the storage module is used for storing the target data by utilizing a preset erasure code strategy k + m;

and the obtaining module is used for obtaining new stripe data corresponding to the target data.

As an embodiment, the sixth determining module may be specifically configured to:

determining target stripe data of lost index information;

judging whether the target stripe data is fully written;

if the target stripe data is full, determining the target stripe data as incomplete stripe data;

if the target stripe data is not fully written, judging whether a storage unit corresponding to the lost index information in the target stripe data is an empty storage unit, and if not, determining the target stripe data to be incomplete stripe data.

As an embodiment, the apparatus may further include:

a second reconstructing module (not shown in the figure) configured to determine a data length corresponding to the lost index information; and reconstructing the stored data corresponding to the lost index information according to the data length and the lost index information.

With the embodiment of the present invention shown in fig. 5, a storage space is divided into a plurality of storage units, and metadata of the storage space includes index information of the plurality of storage units; when the metadata is restored, the index information of the storage unit contained in the storage space pointed by the metadata to be restored is read, the metadata is restored by using the read index information, and then the data can be read by using the metadata.

The embodiment of the present invention further provides a management server, as shown in fig. 6, which includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,

a memory 603 for storing a computer program;

the processor 601 is configured to implement any of the above-described data recovery methods when executing the program stored in the memory 603.

The aforementioned communication bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the management server and other devices.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.

An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the computer program implements any of the above data recovery methods.

An embodiment of the present invention further provides a distributed system, as shown in fig. 1, including: the data fragment storage system comprises a storage server and a management server, wherein the storage server comprises at least one storage space, each storage space comprises a plurality of storage units, and each storage unit is the minimum unit for storing data fragments;

As an embodiment, the data structure information includes: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units;

the management server may be further configured to:

determining incomplete stripe data, wherein the incomplete stripe data is stripe data losing index information;

determining a storage unit pointed by the incomplete stripe data as a target storage unit based on the data structure information;

reading the index information of the target storage unit to obtain a target storage space identifier, a target strip identifier and a target attribute of the data fragment;

determining the target storage space identifier as a storage space identifier in the lost index information;

determining the target stripe identification as a stripe identification in the lost index information;

determining the attribute of the data fragment in the lost index information according to the target attribute;

and recovering the incomplete strip data based on the storage space identification, the strip identification and the data segment attribute in the lost index information.

The management server in the distributed system may also be configured to perform any of the data recovery methods described above.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiment of the distributed system shown in fig. 1, the embodiment of the data recovery apparatus shown in fig. 5, the embodiment of the management server shown in fig. 6, and the embodiment of the computer-readable storage medium, since they are substantially similar to the embodiments of the data recovery method shown in fig. 2 to 4, the description is relatively simple, and relevant points can be referred to the partial description of the embodiment of the data recovery method shown in fig. 2 to 4.

Those skilled in the art will appreciate that all or part of the steps in the above method embodiments may be implemented by a program to instruct relevant hardware to perform the steps, and the program may be stored in a computer-readable storage medium, which is referred to herein as a storage medium, such as: ROM/RAM, magnetic disk, optical disk, etc.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A data recovery method is applied to a management server in a distributed system, and the method comprises the following steps:

2. The method of claim 1, wherein the data structure information comprises: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units; the read index information also comprises a second identifier of the corresponding stripe data;

before the obtaining of the metadata to be restored based on the storage space identifier and the data segment attribute in each piece of index information included in the metadata to be restored is restored, the method further includes:

determining a stripe identifier in each piece of index information contained in the metadata to be restored according to the second identifier or the second identifier and a preset stripe identifier rule;

the recovering the metadata to be recovered based on the storage space identifier and the data segment attribute in each piece of index information included in the metadata to be recovered includes:

3. The method of claim 2, wherein the data fragment attribute comprises an identifier of corresponding target data, a data size of the target data, a corresponding erasure coding policy, a sequence number within a stripe, and a data offset; the first attribute comprises a third identifier of corresponding target data, a first data volume of the corresponding target data, a first erasure code strategy, a first in-band sequence number and a first data offset;

the determining, according to the first attribute and a preset data storage policy, a data fragment attribute in each piece of index information included in the metadata to be restored includes:

4. The method according to claim 3, wherein the metadata to be restored further includes at least one piece of target data information, and after at least one piece of stripe data is restored based on the storage space identifier, the data segment attribute, and the stripe identifier in each piece of index information included in the metadata to be restored, the method further includes:

and forming strip data with the same target data identification in the metadata to be recovered into a piece of target data information.

5. The method of claim 1, wherein the data structure information comprises: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units;

the method further comprises the following steps:

6. The method of claim 5, wherein the data segment attributes comprise corresponding erasure coding strategies, sequence numbers within stripes, data amounts and data offsets of corresponding target data; the target attribute comprises a target erasure code strategy, a target serial number, a target data volume and a target data offset;

the determining the data segment attribute in the lost index information according to the target attribute includes:

7. The method of claim 5, wherein after determining incomplete stripe data, the method further comprises:

counting the number of lost index information in the incomplete stripe data;

judging whether the number is larger than m, wherein m is the number of redundant fragments in the erasure code strategy corresponding to the incomplete stripe data;

and if not, executing the step of determining the storage unit pointed by the incomplete stripe data as a target storage unit based on the data structure information.

8. The method of claim 7, wherein in the case that the number is determined to be greater than m, the method further comprises:

reconstructing target data corresponding to the incomplete strip data;

storing the target data by using a preset erasure code strategy k + m;

and obtaining new stripe data corresponding to the target data.

9. The method of claim 5, wherein determining incomplete band data comprises:

determining target stripe data of lost index information;

judging whether the target stripe data is fully written;

10. The method of claim 6, further comprising, after the recovering the incomplete stripe data based on the storage space identifier, the stripe identifier and the data segment attribute in the lost index information:

determining the data length corresponding to the lost index information;

and reconstructing the stored data corresponding to the lost index information according to the data length and the lost index information.

11. A data recovery apparatus, applied to a management server in a distributed system, the apparatus comprising:

12. The apparatus of claim 11, wherein the data structure information comprises: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units; the read index information also comprises a second identifier of the corresponding stripe data;

the device further comprises:

a fifth determining module, configured to determine, according to the second identifier or the second identifier and a preset stripe identifier rule, a stripe identifier in each piece of index information included in the metadata to be restored;

the first recovery module is specifically configured to:

13. The apparatus of claim 12, wherein the data fragment attribute comprises an identifier of corresponding target data, a data size of the target data, a corresponding erasure coding policy, a sequence number within a stripe, and a data offset; the first attribute comprises a third identifier of corresponding target data, a first data volume of the corresponding target data, a first erasure code strategy, a first in-band sequence number and a first data offset;

the fourth determining module is specifically configured to:

14. The apparatus of claim 13, wherein the metadata to be recovered further comprises at least one piece of target data information, and the apparatus further comprises:

and the combination module is used for combining the strip data with the same target data identification in the metadata to be recovered into a piece of target data information.

15. The apparatus of claim 11, wherein the data structure information comprises: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units;

the device further comprises:

16. The apparatus of claim 15, wherein the data segment attributes comprise corresponding erasure coding strategies, sequence numbers within stripes, data amounts and data offsets of corresponding target data; the target attribute comprises a target erasure code strategy, a target serial number, a target data volume and a target data offset;

the tenth determining module is specifically configured to:

17. The apparatus of claim 15, further comprising:

18. The apparatus of claim 17, further comprising:

19. The apparatus of claim 15, wherein the sixth determining module is specifically configured to:

determining target stripe data of lost index information;

judging whether the target stripe data is fully written;

20. The apparatus of claim 16, further comprising:

the second reconstruction module is used for determining the data length corresponding to the lost index information; and reconstructing the stored data corresponding to the lost index information according to the data length and the lost index information.

21. A distributed system, comprising: the data fragment storage system comprises a storage server and a management server, wherein the storage server comprises at least one storage space, each storage space comprises a plurality of storage units, and each storage unit is the minimum unit for storing data fragments;

22. The system according to claim 21, wherein the data structure information comprises: the method comprises the steps of obtaining a corresponding relation between a storage space and each piece of strip data contained in the storage space, and obtaining a corresponding relation between each piece of strip data and each storage unit contained in the strip data; the metadata of one storage space comprises at least one piece of stripe data, and each piece of stripe data comprises index information of a plurality of storage units;

the management server is further configured to: