CN109144766B

CN109144766B - Data storage and reconstruction method and device and electronic equipment

Info

Publication number: CN109144766B
Application number: CN201710512246.0A
Authority: CN
Inventors: 叶敏; 林鹏; 汪渭春; 林起芊
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2017-06-28
Filing date: 2017-06-28
Publication date: 2023-05-02
Anticipated expiration: 2037-06-28
Also published as: CN109144766A

Abstract

The embodiment of the invention provides a data storage and reconstruction method and device and electronic equipment, wherein the data storage method comprises the following steps: when a storage instruction aiming at target data is received, determining an erasure code strategy corresponding to the target data based on a preset first proportion, wherein the erasure code strategy comprises a first erasure code strategy and/or a second erasure code strategy, the security level of the first erasure code strategy is higher than that of the second erasure code strategy, and the first proportion is as follows: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy; slicing and redundancy processing are carried out on the target data based on the determined erasure coding strategy, and a plurality of data fragments corresponding to the target data are obtained; and respectively storing the plurality of data fragments in storage objects of different storage devices. By applying the embodiment of the invention, the reconstruction efficiency can be effectively improved.

Description

Data storage and reconstruction method and device and electronic equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for data storage and reconstruction, and an electronic device.

Background

Currently, to secure stored data, it is often necessary to store the data based on Erasure Coding (EC).

For example, in storing target data, an erasure coding strategy is utilized: 4+1, dividing the target data into 4 original segments, and expanding and encoding based on the 4 original segments to obtain 1 redundant segment, and further storing the 4 original segments and the 1 redundant segment into each storage object respectively. Thus, if the data in a certain storage object is lost, the lost data segment can be reconstructed according to the original segment and the redundant segment which are not lost, so that the safety of the target data can be ensured.

However, since the existing storage objects are often located in the storage device, and the capacity of the storage device can reach 12TB (Terabyte), when the storage device storing 12TB data in the storage manner fails, a large amount of data needs to be acquired from other storage objects through the network, and then the 12TB data is restored by using the erasure coding policy, so that the reconstruction of the 12TB data stored in the storage device can be realized, and the reconstruction efficiency is very low.

Disclosure of Invention

The embodiment of the invention aims to provide a data storage and reconstruction method and device and electronic equipment so as to improve reconstruction efficiency.

In a first aspect, an embodiment of the present invention provides a data storage method, where the method may include:

when a storage instruction aiming at target data is received, determining an erasure code strategy corresponding to the target data based on a preset first proportion, wherein the erasure code strategy comprises a first erasure code strategy and/or a second erasure code strategy, the security level of the first erasure code strategy is higher than that of the second erasure code strategy, and the first proportion is as follows: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

slicing and redundancy processing are carried out on the target data based on the determined erasure coding strategy, and a plurality of data fragments corresponding to the target data are obtained;

and respectively storing the plurality of data fragments in storage objects of different storage devices.

Optionally, in an implementation manner of the embodiment of the present invention, the step of storing the plurality of data segments in storage objects of different storage devices respectively may include:

Generating a first mapping relation, wherein at least records in the first mapping relation: the corresponding relation between the plurality of data fragments and each storage object is the corresponding relation between the plurality of data fragments and the erasure coding strategy, wherein each storage object is positioned in different storage devices;

and respectively storing the plurality of data fragments in the storage objects based on the first mapping relation.

Optionally, the first mapping relationship further records: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments.

Optionally, in another implementation manner of the embodiment of the present invention, the step of storing the plurality of data segments in storage objects of different storage devices respectively may include:

obtaining a target stripe corresponding to the determined erasure coding strategy, wherein when the erasure coding strategy corresponds to N data segments, the N data segments=K original segments+M redundant segments, the target stripe comprises N storage objects, and the N storage objects are positioned in different storage devices;

generating a second mapping relation, wherein at least records in the second mapping relation: the corresponding relation between the plurality of data fragments and each storage object in the target stripe and the corresponding relation between the target stripe and the determined erasure coding strategy, wherein each storage object is positioned in different storage devices;

And respectively storing the plurality of data fragments into each storage object in the target stripe based on the second mapping relation.

Optionally, when the determined erasure coding policy includes a first erasure coding policy and a second erasure coding policy, the step of slicing and redundancy processing the target data based on the determined erasure coding policy to obtain a plurality of data segments corresponding to the target data may include:

dividing the target data into first target sub-data and second target sub-data;

slicing and redundancy processing are carried out on the first target sub-data based on the first erasure coding strategy, so that a plurality of first data fragments corresponding to the first target sub-data are obtained; slicing and redundancy processing are carried out on the second target sub-data based on the second erasure coding strategy, so that a plurality of second data fragments corresponding to the second target sub-data are obtained;

accordingly, the step of storing the plurality of data fragments in the storage objects of different storage devices respectively may include:

and respectively storing the first data fragments into first storage objects of different storage devices, and respectively storing the second data fragments into second storage objects of different storage devices.

In a second aspect, an embodiment of the present invention provides a data reconstruction method, where the method may include:

detecting whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies;

when the target storage device is detected to be offline, determining a storage object in which a data fragment obtained based on the second erasure coding strategy is located; wherein the target storage device is one storage device of the at least two storage devices, and the target storage device includes: a storage object storing a data fragment obtained based on the first erasure coding strategy and a storage object storing a data fragment obtained based on the second erasure coding strategy; the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy;

and reconstructing the data fragments in the determined storage object.

Optionally, in an implementation manner of the embodiment of the present invention, the step of determining a storage object where the data segment obtained based on the second erasure coding policy is located may include:

determining a plurality of storage objects corresponding to the target storage device;

Determining a storage object in which a data fragment obtained based on a second erasure coding strategy is located from the plurality of storage objects based on a first mapping relation stored in advance; wherein, at least the first mapping relation records: and the corresponding relation between the plurality of data fragments and each storage object is the corresponding relation between the plurality of data fragments and the erasure coding strategy, wherein each storage object is positioned in different storage devices.

Optionally, the first mapping relationship may further record:

the corresponding relation between the identification corresponding to the target data and the plurality of data fragments;

accordingly, the step of reconstructing the data segment in any one of the determined storage objects may include:

determining the identification of target data corresponding to the storage object based on the first mapping relation;

determining target storage objects which are not offline in all storage objects corresponding to the identification in the at least two storage devices; reading a data fragment from the target storage object;

and reconstructing the data fragments stored in the storage object according to the second erasure coding strategy and the read data fragments.

Optionally, in another implementation manner of the embodiment of the present invention, the step of determining a storage object where the data segment obtained based on the second erasure coding policy is located may include:

determining a storage object in which a data fragment obtained based on a second erasure coding strategy is located from the plurality of storage objects based on a second mapping relation stored in advance; wherein, at least the second mapping relation records: and the corresponding relation between the plurality of data fragments and each storage object in the target stripe and the corresponding relation between the target stripe and the determined erasure coding strategy, wherein each storage object in the target stripe is positioned in different storage devices.

Optionally, the step of reconstructing the data segment in the determined storage object may include:

determining a target strip corresponding to any storage object determined based on the second mapping relation;

determining target storage objects which are not offline in all storage objects contained in the target stripe;

Reading a data fragment from the target storage object;

Optionally, after the step of reconstructing the data segment stored in the storage object according to the second erasure coding policy and the read data segment, the method may further include:

and storing the data fragments obtained by reconstructing the storage object, and updating the target strips corresponding to the storage object based on the storage positions of the data fragments obtained by reconstructing.

In a third aspect, an embodiment of the present invention further provides a data storage device, where the device may include:

the first determining unit is configured to determine, when a storage instruction for target data is received, an erasure coding policy corresponding to the target data based on a preset first proportion, where the erasure coding policy includes a first erasure coding policy and/or a second erasure coding policy, and a security level of the first erasure coding policy is higher than that of the second erasure coding policy, and the first proportion is: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

The processing unit is used for slicing and redundancy processing the target data based on the determined erasure coding strategy to obtain a plurality of data fragments corresponding to the target data;

and the storage unit is used for respectively storing the plurality of data fragments in storage objects of different storage devices.

Optionally, in an implementation manner of the embodiment of the present invention, the storage unit may include:

the first generating subunit is configured to generate a first mapping relationship, where at least a record is recorded in the first mapping relationship: the corresponding relation between the plurality of data fragments and each storage object is the corresponding relation between the plurality of data fragments and the erasure coding strategy, wherein each storage object is positioned in different storage devices;

and the first storage subunit is used for respectively storing the plurality of data fragments in the storage objects based on the first mapping relation.

Optionally, the first mapping relationship may further include: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments.

Optionally, in another implementation manner of the embodiment of the present invention, the storage unit may include:

The method comprises the steps of obtaining a subunit, wherein the subunit is used for obtaining a target strip corresponding to the determined erasure coding strategy, and when the erasure coding strategy corresponds to N data segments, the N data segments=K original segments+M redundant segments, the target strip contains N storage objects, and the N storage objects are positioned in different storage devices;

a second generating subunit, configured to generate a second mapping relationship, where at least a record is recorded in the second mapping relationship: the corresponding relation between the plurality of data fragments and each storage object in the target stripe and the corresponding relation between the target stripe and the determined erasure coding strategy, wherein each storage object is positioned in different storage devices;

and the second storage subunit is used for respectively storing the plurality of data fragments into each storage object in the target stripe based on the second mapping relation.

Optionally, when the erasure coding policy determined by the first determining unit includes a first erasure coding policy and a second erasure coding policy, the processing unit is specifically configured to:

dividing the target data into first target sub-data and second target sub-data;

correspondingly, the storage unit is specifically configured to:

In a fourth aspect, an embodiment of the present invention further provides a data reconstruction apparatus, where the apparatus may include:

the detection unit is used for detecting whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies;

the second determining unit is used for determining a storage object where the data fragment obtained based on the second erasure coding strategy is located when the target storage device is detected to be offline; wherein the target storage device is one storage device of the at least two storage devices, and the target storage device includes: a storage object storing a data fragment obtained based on the first erasure coding strategy and a storage object storing a data fragment obtained based on the second erasure coding strategy; the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy;

And the reconstruction unit is used for reconstructing the data fragments in the determined storage object.

Optionally, in an implementation manner of the embodiment of the present invention, the second determining unit may include:

a first determining subunit, configured to determine a plurality of storage objects corresponding to the target storage device;

a second determining subunit, configured to determine, from the plurality of storage objects, a storage object in which a data segment obtained based on a second erasure coding policy is located, based on a first mapping relation stored in advance; wherein, at least the first mapping relation records: and the corresponding relation between the plurality of data fragments and each storage object is the corresponding relation between the plurality of data fragments and the erasure coding strategy, wherein each storage object is positioned in different storage devices.

Optionally, the first mapping relationship may further record:

correspondingly, the reconstruction unit is specifically configured to:

Alternatively, in another implementation manner of the embodiment of the present invention, the second determining unit may include:

a third determining subunit, configured to determine a plurality of storage objects corresponding to the target storage device;

a fourth determining subunit, configured to determine, from the plurality of storage objects, a storage object in which a data segment obtained based on a second erasure coding policy is located, based on a second mapping relationship stored in advance; wherein, at least the second mapping relation records: and the corresponding relation between the plurality of data fragments and each storage object in the target stripe and the corresponding relation between the target stripe and the determined erasure coding strategy, wherein each storage object in the target stripe is positioned in different storage devices.

Optionally, in an embodiment of the present invention, the reconstruction unit may include:

a fifth determining subunit, configured to determine, for any one of the storage objects determined based on the second mapping relationship, a target stripe corresponding to the storage object based on the second mapping relationship;

A sixth determining subunit, configured to determine, among all storage objects included in the target stripe, a target storage object that is not offline;

a reading subunit, configured to read a data segment from the target storage object;

and the reconstruction subunit is used for reconstructing the data fragments stored in the storage object according to the second erasure coding strategy and the read data fragments.

Optionally, in an embodiment of the present invention, the apparatus may further include: an updating unit; the updating unit is specifically configured to:

and after the reconstruction subunit reconstructs the data segments stored in the storage object according to the second erasure coding strategy and the read data segments, storing the data segments reconstructed for the storage object, and updating the target strips corresponding to the storage object based on the storage positions of the data segments obtained by reconstruction.

In a fifth aspect, an embodiment of the present invention provides an electronic device that may include a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory perform communication with each other via the communication bus,

The memory is used for storing a computer program;

the processor is configured to implement the method steps of the data storage method according to any one of the first aspect when executing the program stored in the memory.

In a sixth aspect, an embodiment of the present invention provides an electronic device that may include a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory perform communication with each other through the communication bus,

the memory is used for storing a computer program;

the processor is configured to implement the method steps of the data reconstruction method according to any one of the second aspect when executing the program stored in the memory.

In a seventh aspect, an embodiment of the present invention further provides a storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements the method steps of the data storage method according to any one of the first aspects.

In an eighth aspect, an embodiment of the present invention further provides a storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements the method steps of the data reconstruction method according to any one of the second aspects.

In a ninth aspect, embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of the data storage method of any of the first aspects described above.

In a tenth aspect, embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of the data reconstruction method according to any one of the second aspects above.

In the data storage method provided by the embodiment of the invention, when a storage instruction is received, a corresponding erasure code strategy is determined for target data indicated to be stored by the storage instruction according to a preset first proportion, and the determined erasure code can be the first erasure code strategy and/or the second erasure code strategy. Then, slicing and redundancy processing can be carried out on the target data according to the determined erasure coding strategy to obtain a plurality of data fragments corresponding to the target data; then determining the storage objects to be written by the plurality of data fragments, wherein each written storage object is located in a different storage device. Thus, each time data is stored, the obtained data fragments can be written into the storage objects positioned in different storage devices; and, most storage devices can be made to have both data segments written according to the first erasure coding strategy and data segments written according to the second erasure coding strategy. Therefore, after the data is stored according to the data storage method provided by the embodiment of the invention, once the storage device is offline, the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy, so that only the data segment corresponding to the second erasure coding strategy in the storage device needs to be reconstructed on the premise that the security of the stored data can be ensured, the data quantity required to be reconstructed is reduced, and the reconstruction efficiency can be improved.

In addition, based on the data storage method, the embodiment of the invention also provides a data reconstruction method, in the data reconstruction method, when the target storage device is offline, because the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy, on the premise that the security of the stored data can be ensured, only the data segment corresponding to the second erasure coding strategy in the target storage device is required to be reconstructed, and the data segment corresponding to the first erasure coding strategy in the target storage device is not required to be reconstructed, so that the data quantity required to be reconstructed is reduced, and the reconstruction efficiency can be improved.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a data storage method according to an embodiment of the present invention;

FIG. 2 is a flow chart of a data reconstruction method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a data storage device according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a data reconstruction device according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of another electronic device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order to solve the problems in the prior art, the embodiment of the invention provides a data storage and reconstruction method and device, an electronic device and a storage medium.

The following first describes a data storage method provided by an embodiment of the present invention.

It should be noted that, the execution body for executing the data storage method provided in the embodiment of the present invention may be a server, which is not limited to this. Moreover, functional software for implementing the data storage method provided by the embodiment of the invention can be: the special data storage software arranged in the server can also be: it is reasonable to locate functional plug-ins in the existing data storage software in the server.

Where the server may be a management server in a data processing system, this is reasonable.

Referring to fig. 1, the data storage method provided by the embodiment of the present invention may include the following steps:

s101: when a storage instruction aiming at target data is received, determining an erasure code strategy corresponding to the target data based on a preset first proportion, wherein the erasure code strategy comprises a first erasure code strategy and/or a second erasure code strategy, the security level of the first erasure code strategy is higher than that of the second erasure code strategy, and the first proportion is as follows: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

it can be understood that the first proportion is preset in the server, and the number of times of using the first erasure coding strategy can be controlled through the first proportion, so that the proportion of the number of times of using the first erasure coding strategy to the total number of times of using the erasure coding strategy meets the first proportion. Wherein the total number of times the erasure coding strategy is used = the number of times the first erasure coding strategy is used + the number of times the second erasure coding strategy is used.

For example, assuming that the preset first proportion has a value of 50%, it is indicated that the server should ensure that the number of times of using the first erasure coding policy is equal to the number of times of using the second erasure coding policy as much as possible in the process of storing data.

It will thus be appreciated that each time a store instruction for target data is received, the server needs to determine which erasure coding strategy is employed for the target data corresponding to the current store instruction based on the first ratio. For example, assuming that the server stores 99 data before storing the target data, wherein 50 data are stored using a first erasure coding strategy, and 49 data are stored using a second erasure coding strategy, when storing the target data, in order to match the first ratio as much as possible, the server stores the target data using the second erasure coding strategy.

Of course, the storage instruction received by the server may also be: store instructions for a plurality of target data. In this case, the server may determine, based on the first ratio, an erasure coding policy corresponding to each of the plurality of target data. For example, assuming that the value of the first proportion is 50% and the number of the plurality of target data is 100, it may be determined that each of any 50 data in the 100 target data corresponds to the first erasure coding policy, and each of the remaining 50 data corresponds to the second erasure coding policy, so that the number of times of using the first erasure coding policy and the number of times of using the second erasure coding policy may be ensured to be equal as much as possible.

It should be noted that, the security level of the first erasure coding policy is higher than that of the second erasure coding policy means that: the number of redundant fragments corresponding to the first erasure coding strategy is greater than the number of redundant fragments corresponding to the second erasure coding strategy.

For example, assume that the server is preset with: the first erasure code strategy is: 4+2, namely the first erasure coding strategy corresponds to 4 original segments and 2 redundant segments; and preset: the second erasure coding strategy is: 4+1, the second erasure coding strategy corresponds to 4 original segments and 1 redundant segment.

It should be noted that, the first erasure coding strategy described above: 4+2, and a second erasure coding strategy: 4+1 is merely an example, and the setting of the first erasure coding policy and the second erasure coding policy is not limited thereto.

S102: slicing and redundancy processing are carried out on the target data based on the determined erasure coding strategy, and a plurality of data fragments corresponding to the target data are obtained;

for example, when the determined erasure coding strategy is the first erasure coding strategy, and the preset first erasure coding strategy is: and 4+2, slicing and redundancy processing are carried out on the target data by utilizing the first erasure coding strategy, so that 4 original fragments and 2 redundant fragments corresponding to the target data can be obtained.

It should be noted that the specific number of the plurality of data segments is determined by the determined erasure coding strategy. For example, when the determined first erasure code strategy is: 4+2, the number of the plurality of data segments is 6; when the determined first erasure code strategy is: 4+3, the number of the plurality of data segments is 7; etc. And are not illustrated herein.

S103: and respectively storing the plurality of data fragments in storage objects of different storage devices.

For example, after obtaining 6 data segments (4 original segments and 2 redundant segments) corresponding to the target data, the 6 data segments may be respectively stored in the storage objects of 6 different storage devices. Thus, when any 1 (or 2) of the 6 storage devices are offline, the data fragments can still be read from the other 5 storage devices (or 4 storage devices), so that the target data can be recovered by using the read data fragments, and the safety of the target data is improved.

It should be noted that, as known from probabilistic knowledge, when the more storage instructions are received, that is, the more data are stored, the more storage objects corresponding to the first erasure coding policy in each storage device that the server is responsible for managing, the ratio of the number of storage objects in each storage device is closer to the first ratio.

It should be noted that, the storage device according to the embodiment of the present invention includes, but is not limited to: magnetic disks, SSDs (Solid State Drives ), ISCSI (Internet Small Computer System Interface, small computer System interface) devices, FC (Fiber Channel) devices, magnetic tape, and RAID (Redundant Arrays of Independent Disks, disk array) devices. It is reasonable that the server may be responsible for managing one or more of the storage devices described above.

An implementation of storing the plurality of data fragments in the storage objects of different storage devices, respectively, is described below.

Optionally, in an embodiment of the present invention, the step of storing the plurality of data segments in storage objects of different storage devices respectively may include:

It will be appreciated that in such an implementation, the first mapping relationship may be generated each time the plurality of data segments are derived based on the determined erasure coding policy: the corresponding relation between the plurality of data fragments and the determined erasure coding strategy and the corresponding relation between the plurality of data fragments and a plurality of storage objects positioned in different storage devices.

For example, when based on the second erasure coding strategy: 4+1, after obtaining 5 data segments corresponding to the target data, respectively establishing the obtained 5 data segments and a first erasure coding strategy: 4+1, and establishing a one-to-one correspondence between the 5 data segments and 5 storage objects located in 5 different storage devices, thereby generating and obtaining the first mapping relation.

In this way, the 5 data segments may be stored according to the generated first mapping relation. After the storage is completed, the data segments corresponding to the storage objects in each storage device managed by the server and the erasure coding strategies corresponding to the data segments can be determined through the first mapping relation, so that the erasure coding strategies corresponding to the storage objects in each storage device can be determined. That is, when the target storage device in each storage device goes offline, the target storage object corresponding to the second erasure coding policy in the target storage device may be determined based on the first mapping relationship, so that only the data segment corresponding to the target storage object may be reconstructed, and thus the reconstruction efficiency may be improved.

It may be appreciated that, in this implementation, in order to facilitate subsequent reconstruction of the lost data segment, the first mapping relationship may further be recorded with: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments. Thus, when a storage device storing a certain data segment of the target data is offline, the identification of the target data corresponding to the lost data segment and the erasure coding policy can be determined. Therefore, when the lost data segment erasure coding strategy is determined to be the second erasure coding strategy, namely, the corresponding security level is lower, other data segments corresponding to the target data can be searched according to the identification, and further the lost data segment can be recovered according to the searched data segment and the determined second erasure coding strategy, so that the security of the target data is ensured.

Optionally, in a further embodiment of the present invention, the step of storing the plurality of data segments in storage objects of different storage devices respectively may include:

As known in the art, any stripe may include a number of memory objects located on different memory devices. Therefore, after determining the erasure coding policy corresponding to the target data, a target stripe corresponding to the erasure coding policy can be obtained, wherein the number of storage objects in different storage devices contained in the target stripe is determined by the erasure coding policy.

For example, when the determined erasure coding policy is the first erasure coding policy: 4+2, then the first erasure coding strategy: the target stripe corresponding to 4+2 contains 6 storage objects located in different storage devices. For another example, when the determined erasure coding policy is the second erasure coding policy: 4+1, then the second erasure coding strategy: the target stripe corresponding to 4+1 contains 5 storage objects located in different storage devices.

Thus, in this case, the target stripe to which the target data corresponds can be obtained based on the first scale. For example, when it is determined that the target data corresponds to a first erasure coding policy based on the first ratio: after 4+2, a target stripe of type 4+2 (i.e., a target stripe containing 6 storage objects located on different storage devices) may be obtained at this time. When it is determined that the target data corresponds to a second erasure coding scheme based on the first ratio: after 4+1, a target stripe of type 4+1 (i.e., a target stripe containing 5 storage objects located on different storage devices) may be obtained at this time.

And, after obtaining the plurality of data segments based on the determined erasure coding policy, a second mapping relationship may be generated: and the corresponding relation between the plurality of data fragments and each storage object in the target stripe and the corresponding relation between the target stripe and the determined erasure coding strategy.

For example, when based on the first erasure coding strategy: 4+2, after obtaining 6 data segments corresponding to the target data, establishing a corresponding relation between each obtained data segment and a target stripe a, and establishing the target stripe a and the first erasure coding strategy: and 4+2, thereby generating and obtaining the second mapping relation.

Thus, after the second mapping relationship is generated, the 6 data segments may be stored based on the second mapping relationship. And after the storage is completed, determining each target stripe corresponding to the storage object in each storage device managed by the server and the erasure code policy corresponding to each target stripe through the second mapping relationship, so as to determine the erasure code policy corresponding to the storage object in each storage device. That is, when the target storage device in each storage device goes offline, the target storage object corresponding to the second erasure coding policy in the target storage device may be determined based on the second mapping relationship, so that only the data segment corresponding to the target storage object may be reconstructed, and thus the reconstruction efficiency may be improved.

Optionally, when the determined erasure coding policy includes a first erasure coding policy and a second erasure coding policy, the step of slicing and redundancy processing the target data based on the determined erasure coding policy to obtain a plurality of data segments corresponding to the target data includes:

dividing the target data into first target sub-data and second target sub-data;

accordingly, the step of storing the plurality of data segments into the storage objects of different storage devices respectively includes:

For example, assume that the server stores 100 data before storing the target data, wherein 50 data are stored using a first erasure coding policy and 50 data are stored using a second erasure coding policy. In addition, when the data amount of the target data is detected to be large at this time, in order to match the first proportion as much as possible when the target data is stored, the target data may be divided into first target sub-data and second target sub-data, the first target sub-data may be stored based on a first erasure coding policy, and the second target sub-data may be stored based on a second erasure coding policy, which is reasonable.

It is also reasonable to note that the data storage method described above may also be performed jointly by individual servers in the data processing system. Wherein, the data processing system may include: management server, slice calculation server and storage server. The management server may be configured to determine, when a storage instruction for target data is received, an erasure code policy corresponding to the target data based on a preset first proportion; the slice calculation server may be configured to perform slicing and redundancy processing on the target data based on the determined erasure coding policy, to obtain a plurality of data segments corresponding to the target data; the storage server may be used to store the plurality of data fragments, as is reasonable and not described in detail herein.

Corresponding to the data storage method, the embodiment of the invention also provides a data reconstruction method.

It should be noted that, an execution body for executing the data reconstruction method provided by the embodiment of the present invention may be a server. Moreover, functional software for implementing the data reconstruction method provided by the embodiment of the present invention may be: the special data reconstruction software set in the server may also be: it is reasonable to provide a functional plug-in the existing data reconstruction software in the server.

Referring to fig. 2, the data reconstruction method provided by the embodiment of the present invention may include the following steps:

s201: detecting whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies;

it will be appreciated that the server may detect whether the storage device it manages is down, i.e. may detect whether at least two storage devices it manages are down. Wherein the specific number of the at least two storage devices may be determined by the number of storage devices that the server is responsible for managing.

S202: when the target storage device is detected to be offline, determining a storage object in which a data fragment obtained based on the second erasure coding strategy is located; wherein the target storage device is one storage device of the at least two storage devices, and the target storage device includes: a storage object storing a data fragment obtained based on the first erasure coding strategy and a storage object storing a data fragment obtained based on the second erasure coding strategy; the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy;

It will be appreciated that the server is able to detect that the target storage device is down when the target storage device fails.

For example, assume that there are 100 storage objects in the target storage device, wherein the data segments stored in 50 of the 100 storage objects are based on a first erasure coding policy: 4+2, the stored data segments in the remaining 50 storage objects are based on a second erasure coding policy: 4+1.

S203: and reconstructing the data fragments in the determined storage object.

In combination with the above example, when the target storage device is detected to be offline, only 50 storage objects of the data fragment obtained based on the second erasure coding policy may be determined to be stored in the target storage device. That is, only 50 storage objects corresponding to 50 data segments with a lower security level in the target storage device need to be determined. Therefore, only the data segments stored in the 50 determined storage objects need to be reconstructed, the number of the data segments needing to be reconstructed is reduced by 50%, and the reconstruction efficiency is improved.

It should be noted that, among the 100 storage objects in the target storage device, the number of storage objects for storing the data segments based on the first erasure coding policy and the number of storage objects for storing the data segments based on the second erasure coding policy are largely determined by the first ratio preset when storing the data. When the data amount stored by the server is larger, the number of storage objects corresponding to the first erasure coding policy in the target storage device is closer to the first ratio than the total number of storage objects in the target storage device, that is, the reconstruction efficiency can be controlled by setting the first ratio. The larger the value set by the first proportion is, the higher the reconstruction efficiency is when the target storage device is offline.

The manner in which the storage object in which the data fragment obtained based on the second erasure coding scheme is located is determined will be described below.

Optionally, in an embodiment of the present invention, the step of determining a storage object where the data segment obtained based on the second erasure coding policy is located may include:

It can be understood that, because the first mapping relationship is stored in the server in advance, when the offline of the target storage device is detected, the target storage object corresponding to the second erasure coding policy in the target storage device can be determined based on the first mapping relationship, so that only the data segment corresponding to the target storage object can be reconstructed, and thus the reconstruction efficiency can be improved.

For example, it is assumed that 100 storage objects may be determined to exist in the target storage device, so that according to the first mapping relationship, data segments corresponding to the 100 storage objects may be determined, so as to determine an erasure coding policy corresponding to the determined data segments, and then only the data segments stored based on the second erasure coding policy are reconstructed.

Optionally, the first mapping relationship further records:

correspondingly, the step of reconstructing the data segment in any one of the determined storage objects comprises the following steps:

It may be appreciated that, in this implementation, in order to facilitate subsequent reconstruction of the lost data segment, the pre-stored first mapping relationship may further be recorded with: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments. Thus, when a storage device storing a certain data segment of the target data is offline, the identification of the target data corresponding to the lost data segment and the erasure coding policy can be determined. Therefore, when the erasure coding policy of the lost data segment is determined to be the second erasure coding policy, that is, the security level corresponding to the lost data segment is lower, other data segments corresponding to the target data can be searched according to the identifier, and then the lost data segment can be recovered according to the searched data segment and the determined second erasure coding policy.

Optionally, in another embodiment of the present invention, the step of determining a storage object where the data segment obtained based on the second erasure coding policy is located may include:

It will be appreciated that, when the target storage device is detected to be offline, all storage objects corresponding to the target storage device may be determined. And since the second mapping relation is preset, the following is recorded: the corresponding relation between the plurality of data fragments and each storage object in the target stripe, and the corresponding relation between the target stripe and the determined erasure coding strategy. Therefore, according to the first mapping relationship and the determined all storage objects, all target stripes corresponding to the target storage device can be determined, and then the storage object with the corresponding erasure coding policy being the second erasure coding policy in all target stripes is determined, so that only the data segments in the storage object with the second erasure coding policy in the target storage device can be reconstructed, and the reconstruction efficiency can be improved.

It should be noted that the number of storage objects located in different storage devices included in the target stripe is determined by the erasure coding policy. Wherein, since the strip belongs to the prior art, the form of the target strip is not specifically described herein.

reading a data fragment from the target storage object;

It may be appreciated that, after determining the storage object based on the second mapping relationship, the target stripe corresponding to the storage object may be determined again according to the second mapping relationship. Thus, it is possible to determine a target memory object which is not offline among all the memory objects contained in the target stripe. And further, the data can be read from the target storage object, and the data segment corresponding to the storage object is reconstructed according to the second erasure coding strategy and the read data segment.

It can be appreciated that, since only the data segment stored based on the second erasure coding policy in the target storage device needs to be reconstructed, the amount of data to be reconstructed is reduced. Then the amount of data resulting from the reconstruction to be stored is also reduced.

In addition, assuming that the target storage device contains 100 storage objects, when each storage object corresponds to one target stripe, the target storage device corresponds to 100 target stripes. In this way, when the target storage device is offline, in the embodiment of the invention, since the data segment in each storage object in the target storage device does not need to be reconstructed, only the target stripe corresponding to the storage object needing to reconstruct the data segment needs to be updated, and the number of target stripes needing to be updated is reduced.

It should also be noted that it is reasonable that the above-described data reconstruction method may also be performed jointly by the various servers in the data processing system. Wherein, the data processing system may include: management server, audit server and storage server. The management server may be configured to detect whether at least two storage devices are offline; the storage server may be configured to determine, when detecting that the target storage device is offline, a storage object in which a data segment obtained based on the second erasure coding policy is located; the audit server may be used to reconstruct the data segments in the determined storage objects, as is reasonable and not described in detail herein.

Corresponding to the above-mentioned data storage method embodiment, the embodiment of the present invention further provides a data storage device, referring to fig. 3, where the device may include:

a first determining unit 301, configured to determine, when a storage instruction for target data is received, an erasure coding policy corresponding to the target data based on a preset first proportion, where the erasure coding policy includes a first erasure coding policy and/or a second erasure coding policy, and a security level of the first erasure coding policy is higher than that of the second erasure coding policy, and the first proportion is: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

the processing unit 302 is configured to perform slicing and redundancy processing on the target data based on the determined erasure coding policy, so as to obtain a plurality of data segments corresponding to the target data;

the storage unit 303 is configured to store the plurality of data fragments in storage objects of different storage devices, respectively.

Optionally, in one embodiment of the present invention, the storage unit 303 includes:

Optionally, in another embodiment of the present invention, the storage unit 303 includes:

Optionally, when the erasure coding policy determined by the first determining unit includes a first erasure coding policy and a second erasure coding policy, the processing unit 302 is specifically configured to:

Dividing the target data into first target sub-data and second target sub-data;

accordingly, the storage unit 303 is specifically configured to:

Corresponding to the above data reconstruction method, the embodiment of the present invention further provides a data reconstruction device, see fig. 4, where the device may include:

a detecting unit 401, configured to detect whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies;

a second determining unit 402, configured to determine, when it is detected that the target storage device is offline, a storage object where a data segment obtained based on the second erasure coding policy is located; wherein the target storage device is one storage device of the at least two storage devices, and the target storage device includes: a storage object storing a data fragment obtained based on the first erasure coding strategy and a storage object storing a data fragment obtained based on the second erasure coding strategy; the security level of the first erasure coding strategy is higher than that of the second erasure coding strategy;

A reconstruction unit 403, configured to reconstruct the data segment in the determined storage object.

Optionally, in one embodiment of the present invention, the second determining unit 402 includes:

Optionally, the first mapping relationship further records:

accordingly, the reconstruction unit 403 is specifically configured to:

Optionally, in still another embodiment of the present invention, the second determining unit 402 includes:

Optionally, the reconstruction unit 403 includes:

Optionally, the apparatus further comprises: an updating unit; the updating unit is specifically configured to:

Corresponding to the above-described method embodiments, the present application further provides an electronic device, see fig. 5, which includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, where the processor 501, the communication interface 502, and the memory 503 perform communication with each other through the communication bus 504,

A memory 503 for storing a computer program;

the processor 501, when configured to execute the program stored in the memory 503, may perform any of the method steps of the data storage method described above.

The communication bus 504 mentioned above for the electronic device may be: peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated PCI) bus or extended industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) bus, etc. The communication bus 504 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface 502 is used for communication between the electronic device and other devices described above.

The memory 503 may include: random access memory (Random Access Memory, RAM) may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 503 may also be at least one memory device located remotely from the aforementioned processor 501.

The processor 501 may be a general purpose processor, including: a central processing unit (Central Processing Unit, CPU for short), a network processor (Net work Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Corresponding to the above-described method embodiments, the present application further provides an electronic device, see fig. 6, which includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 perform communication with each other through the communication bus 604,

a memory 603 for storing a computer program;

the processor 601, when executing the program stored on the memory 603, may perform any of the method steps of the data reconstruction method described above.

The communication bus 604 mentioned above for the electronic device may be: peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated PCI) bus or extended industry standard architecture (Extended Industry Standard Architecture, abbreviated EISA) bus, etc. The communication bus 604 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface 602 is used for communication between the electronic device and other devices described above.

The memory 603 may include: random access memory (Random Access Memory, RAM) may also include non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 603 may also be at least one storage device located remotely from the processor 601.

The processor 601 may be a general purpose processor, including: a central processing unit (Central Processing Unit, CPU for short), a network processor (Net work Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Corresponding to the above method embodiments, the present application further provides a storage medium, in which a computer program is stored, where the computer program is executed by a processor to implement the method steps of any one of the above data storage methods. Wherein the storage medium refers to a computer-readable storage medium.

Corresponding to the above method embodiments, the present application further provides a storage medium, where a computer program is stored, where the computer program is executed by a processor to implement the method steps of any one of the above data reconstruction methods. Wherein the storage medium refers to a computer-readable storage medium.

Corresponding to the above-described method embodiments, the present application further provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of any of the data storage methods described above.

Corresponding to the above method embodiments, the present application further provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of any of the above described data reconstruction methods.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method of data storage, the method comprising:

storing the plurality of data fragments in storage objects of different storage devices respectively;

when the determined erasure coding strategy includes a first erasure coding strategy and a second erasure coding strategy, the step of slicing and redundancy processing the target data based on the determined erasure coding strategy to obtain a plurality of data segments corresponding to the target data includes:

dividing the target data into first target sub-data and second target sub-data;

the step of storing the plurality of data fragments in the storage objects of different storage devices respectively includes:

and respectively storing the plurality of first data fragments into first storage objects of different storage devices, respectively storing the plurality of second data fragments into second storage objects of different storage devices, wherein each storage device comprises the first storage object and the second storage object.

2. The method of claim 1, wherein the step of storing the plurality of data segments in the memory objects of the different memory devices, respectively, comprises:

3. The method according to claim 2, wherein the first mapping relationship is further recorded with: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments.

4. The method of claim 1, wherein the step of storing the plurality of data segments in the memory objects of the different memory devices, respectively, comprises:

5. A method of data reconstruction, the method comprising:

detecting whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies; the erasure code strategy comprises a first erasure code strategy and/or a second erasure code strategy; the erasure coding strategy is determined based on a preset first ratio, wherein the first ratio is as follows: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

And reconstructing the data fragments in the determined storage object.

6. The method of claim 5, wherein the step of determining the storage object in which the data segment based on the second erasure coding policy resides comprises:

7. The method of claim 6, wherein the first mapping relationship is further recorded with:

8. The method of claim 5, wherein the step of determining the storage object in which the data segment based on the second erasure coding policy resides comprises:

9. The method of claim 8, wherein the step of reconstructing the determined data segments in the storage object comprises:

reading a data fragment from the target storage object;

10. The method of claim 9, wherein after the step of reconstructing the data segments stored in the storage object according to the second erasure coding policy and the read data segments, the method further comprises:

11. A data storage device, the device comprising:

a storage unit, configured to store the plurality of data fragments in storage objects of different storage devices, respectively;

when the erasure coding policy determined by the first determining unit includes a first erasure coding policy and a second erasure coding policy, the processing unit is specifically configured to:

dividing the target data into first target sub-data and second target sub-data;

the storage unit is specifically configured to:

12. The apparatus of claim 11, wherein the storage unit comprises:

13. The apparatus of claim 12, wherein the first mapping relationship further includes: and the corresponding relation between the identification corresponding to the target data and the plurality of data fragments.

14. The apparatus of claim 11, wherein the storage unit comprises:

15. A data reconstruction apparatus, the apparatus comprising:

the detection unit is used for detecting whether at least two storage devices are offline; wherein the data stored by the at least two storage devices are stored based on erasure coding policies; the erasure code strategy comprises a first erasure code strategy and/or a second erasure code strategy; the erasure coding strategy is determined based on a preset first ratio, wherein the first ratio is as follows: the using times of the first erasure code strategy are in proportion to the total using times of the erasure code strategy;

16. The apparatus according to claim 15, wherein the second determining unit comprises:

17. The apparatus of claim 16, wherein the first mapping relationship is further recorded with:

correspondingly, the reconstruction unit is specifically configured to:

18. The apparatus according to claim 15, wherein the second determining unit comprises:

19. The apparatus of claim 18, wherein the reconstruction unit comprises:

20. The apparatus of claim 19, wherein the apparatus further comprises: an updating unit; the updating unit is specifically configured to:

21. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus,

the memory is used for storing a computer program;

The processor is configured to implement the method steps of any of claims 1-4 when executing a program stored on a memory.

22. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus,

the memory is used for storing a computer program;

the processor is configured to implement the method steps of any of claims 5-10 when executing a program stored on a memory.