WO2021057377A1

WO2021057377A1 - Data storage method and data storage device

Info

Publication number: WO2021057377A1
Application number: PCT/CN2020/111903
Authority: WO
Inventors: 杨艳伟; 孙荣宗
Original assignee: 华为技术有限公司
Priority date: 2019-09-27
Filing date: 2020-08-27
Publication date: 2021-04-01
Also published as: CN112578992B; CN112578992A

Abstract

A data storage method, the data storage method comprising: sending a first control instruction, the first control instruction instructing to install data processing software in N storage devices and creating storage resource pools and virtual machines in any storage device among the N storage devices, any virtual machine using a created storage resource pool in a corresponding storage device thereof to store data, any virtual machine serving as a selectable data node for the data processing software, and the N being an integer greater than or equal to two (101); acquiring a configuration instruction, the configuration instruction comprising configuring a number of copies M for data storage, and a storage awareness policy, the storage awareness policy comprising determining M data nodes for storing data, the M data nodes being located in M different storage devices, the M being an integer smaller than or equal to N (102); according to the configuration instruction, determining the M data nodes for saving data to be stored (103); and storing the data in the M data nodes (104). Configuring a storage awareness policy is advantageous for improving the reliability of data storage.

Description

Data storage method and data storage device

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China, the application number is 201910926872.3, and the title of the invention is "a data storage method and data storage device" on September 27, 2019, the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of computer networks, and in particular to a data storage method and data storage device.

Background technique

In order to improve the security of data storage, multiple copies are usually used when storing data. One data file is copied into multiple copies and stored in multiple servers or magnetic arrays, and the data files are saved to multiple computers. Take the server as an example. As long as any server that saves a copy of the data file is available, the data file can be accessed, which can prevent a single server from being caused by network failure, disk damage, power outage, downtime, etc. Data loss and inaccessibility issues.

The multi-copy storage method achieves data storage reliability through high redundancy. It should be noted that although this method has the advantage of easy data recovery, sometimes multiple backups are stored in the same data storage device during storage. When the data storage device that saves multiple backup data is out of power, downtime, etc., the number of copies that can be used is not the total number of copies minus one, but the total number of copies minus the current data storage device The number of copies stored, that is, the number of copies actually available is less than expected, which reduces the reliability of storage.

Therefore, how to improve the reliability of data storage is a problem that needs to be solved.

Summary of the invention

The embodiments of the present application provide a data storage method and a data storage device, which can improve the reliability of data storage.

In the first aspect, an embodiment of the present application provides a data storage method, and the method includes the following steps:

Send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, create a storage resource pool and a virtual machine in any one of the N storage devices, any one of the The virtual machine uses the created storage resource pool in its corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is an integer greater than or equal to 2;

Acquire a configuration instruction, the configuration instruction includes: setting the number of copies M for data storage, and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, and the M data nodes are located in M In different storage devices, the M is an integer less than or equal to the N;

Determining, according to the configuration instruction, M data nodes for storing data to be stored;

Storing the data to be stored in the M data nodes.

In the data storage method provided by the embodiments of the present application, when multiple copies are stored, different copies are located in different storage devices. In this way, when a storage device storing backup data fails, the number of copies actually reduced is one, which is compared with the current one. In some technologies, when a problem occurs in a storage device that stores one copy, it may cause multiple copies to be unusable, which improves storage reliability.

In some possible implementation manners, the storage resource pools created in the N storage devices adopt erasure code EC encoding. In the embodiment of the present application, EC coding is used in the storage resource pool created by the storage device. When data is lost or damaged, the lost data can be calculated to ensure that some data can still be used after loss. Compared with a traditional distributed system that can continue to provide services after 3 copies of data after hardware failures and other faults occur, the embodiment of the present application saves storage space and improves storage utilization.

In some possible implementations, when the EC encoding adopts 8 data blocks and 1 check block 8D1P mode, the storage utilization rate of the hard disk is 88.89%, and when the EC encoding adopts 4D1P mode, the storage utilization rate of the hard disk is 80%. When the EC encoding adopts the 8D2P mode, the storage utilization rate of the hard disk is 80%, and when the EC encoding adopts the 4D2P mode, the storage utilization rate of the hard disk is 66.67%.

In a second aspect, an embodiment of the present application provides a data storage device, and the storage device includes:

The sending unit is configured to send a first control instruction that instructs to install data processing software in N storage devices, and create a storage resource pool and a virtual machine in any one of the N storage devices , Any one of the virtual machines uses the created storage resource pool in the corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is greater than or equal to 2 Integer.

The acquiring unit is configured to acquire a configuration instruction, the configuration instruction includes: setting the number of copies M for data storage, and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, the M Data nodes are located in M different storage devices, and the M is an integer less than or equal to the N.

The determining unit is configured to determine M data nodes for storing data to be stored according to the configuration instruction.

The processing unit is configured to store the data to be stored in the M data nodes.

When the data storage device provided in the embodiment of the present application performs multiple copy storage, different copies are located in different storage devices. In this way, when a storage device storing backup data fails, the number of copies actually reduced is one, which is compared with the current one. When there is a problem with the storage device that stores one copy in the technology, it may cause multiple copies to be unusable, which increases the reliability of storage.

In some possible implementation manners, the storage device includes: a distributed server or a magnetic array.

In some possible implementation manners, the data processing software includes: distributed processing software Hadoop.

In some possible implementations, the M=2.

In a third aspect, an embodiment of the present application provides a data storage system, including N storage devices such as the data storage device described in the second aspect or any one of the possible implementation manners of the second aspect, where N is greater than or An integer equal to 2.

In a fourth aspect, an embodiment of the present application provides an electronic device, including:

One or more processors;

Storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect or any one of the possible implementation manners of the first aspect.

In the fifth aspect, the embodiments of the present application provide a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the implementation is as in the first aspect or any one of the possible implementation manners of the first aspect The method described.

Description of the drawings

FIG. 1 is a schematic flowchart of a data storage method provided by an embodiment of the present application.

Fig. 2 is a schematic flowchart of a data storage method provided by another embodiment of the present application.

FIG. 3 is a schematic diagram of the interaction flow of a data storage method provided by an embodiment of the present application.

Fig. 4 is a schematic structural diagram of a data storage device provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in this application, those of ordinary skill in the art can also obtain other embodiments.

Please refer to FIG. 1. FIG. 1 is a data processing method provided by an embodiment of the present application, which includes the following steps.

101. Send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, and create a storage resource pool and a virtual machine in any one of the N storage devices, any The virtual machine uses the created storage resource pool in the corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is an integer greater than or equal to 2.

For example, the storage device may be a distributed server or a magnetic array.

102. Obtain a configuration instruction, where the configuration instruction includes: setting the number of copies M for data storage and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, the M data nodes Located in M different storage devices, the M is an integer less than or equal to the N.

For example, if N is 3 and M is 2, the first control instruction instructs to install data processing software in three storage devices, and create storage resource pools and virtual machines in any one of the three storage devices. Any virtual machine uses the created storage resource pool in its corresponding storage device to create data, and the three created virtual machines can be used as optional data nodes of the data processing software. In some possible embodiments, the data processing software may be distributed processing software Hadoop.

103. Determine, according to the configuration instruction, M data nodes for storing data to be stored.

For example, if M is 2, two data nodes for storing the data to be stored are determined according to the configuration instruction.

104. Store the data to be stored in the M data nodes.

For example, if M is 2, the data to be stored is saved to a certain number of data nodes.

In the data storage method provided by the embodiments of the present application, when multiple copies are stored, different copies are located in different storage devices. In this way, when a storage device storing backup data fails, the number of copies actually reduced is one, which is compared with the current one. When there is a problem with the storage device that stores one copy in the technology, it may cause multiple copies to be unusable, which increases the reliability of storage.

Please refer to FIG. 2, which is a schematic flowchart of a data processing method provided by another embodiment of the present application. Including the following steps:

201. Send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, create a storage resource pool and a virtual machine in any one of the N storage devices, any The virtual machine uses the created storage resource pool in its corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is an integer greater than or equal to 2, and the storage The resource pool is coded with erasure code EC.

For example, the storage device may be a distributed server or a magnetic array. The following describes the distributed server as an example.

In some possible implementations, when the EC encoding adopts 8 data block and 1 parity block 8D1P mode, the storage utilization rate of the hard disk is 88.89%. When the EC encoding adopts 4D1P mode, the storage utilization rate of the hard disk is 80%. When the encoding adopts the 8D2P mode, the storage utilization rate of the hard disk is 80%, and when the EC encoding adopts the 4D2P mode, the storage utilization rate of the hard disk is 66.67%.

202. Obtain a configuration instruction, where the configuration instruction includes: setting the number of copies M for data storage, and a storage-aware strategy; the storage-aware strategy includes: determining M data nodes for storing data, and the M data nodes Located in M different storage devices, the M is an integer less than or equal to the N.

203. Determine, according to the configuration instruction, M data nodes for storing data to be stored.

204. Store the data to be stored in the M data nodes.

In the embodiment of the present application, EC coding is used in the storage resource pool created by the storage device. When data is lost or damaged, the lost data can be calculated to ensure that some data can still be used after loss. Compared with a traditional distributed system that can continue to provide services after 3 copies of data after hardware failures and other faults occur, the embodiment of the present application saves storage space and improves storage utilization.

Please refer to FIG. 3, which is a schematic diagram of the interaction flow of the data storage method provided by an embodiment of the present application. As shown in FIG. 3, when data storage is performed in this embodiment, the following steps are included.

301. Install Software Defined Storage (SDS) on the first rack Rack1 and the second rack Rack2, and create different storage resource pools on Rack1 and Rack2 respectively, using EC (N: N+M) Encoding. In this embodiment, the EC encoding adopts the 8D1P mode.

In this embodiment, as shown in FIG. 3, a storage resource pool 1 can be created by SDS, and the storage resource pool 1 adopts EC coding in the 8D1P mode. And the resource pool 2 is created by SDS, and the resource pool 2 adopts the EC code of the 8D1P mode.

SDS is a storage architecture that can separate storage software and hardware. Unlike traditional Network Attached Storage (NAS) or Storage Area Network (SAN) systems, SDS is generally executed on industry standard systems or x86 systems, thereby eliminating software dependence on proprietary hardware Sex. SDS usually uses a distributed architecture to improve reliability and scalability, so SDS is sometimes called distributed storage. In fact, the difference between the two is obvious. Distributed storage refers to the architecture, which emphasizes that the architecture is distributed; SDS refers to software-defined storage, which emphasizes the decoupling of software and hardware.

SDS has the following advantages: (1) Software and hardware decoupling. The storage hardware is a commercial off-the-shelf (COTS) COTS, which avoids vendor lock-in, and purchases software and hardware hierarchically to reduce equipment procurement costs. (2) Strong scalability. SDS adopts a distributed architecture, and the storage specifications are theoretically unlimited, and the storage specifications increase linearly with the number of servers (horizontal expansion). SAN is limited by the processing capacity of the controller, and the specifications of a single set of magnetic arrays are limited. After the storage specifications exceed the specifications of the magnetic array, a set of storage equipment must be added (vertical expansion). (3) High reliability.

302. Create a virtual machine used by Hadoop, where the disk of the virtual machine uses a storage pool on the rack where the virtual machine is located.

Specifically, the virtual machine disk on Rack1 uses storage resource pool 1, and the virtual machine on Rack2 uses storage resource pool 2.

Hadoop is a distributed system infrastructure often used in the prior art. The Hadoop Distributed File System (HDFS) divides nodes into two categories, Name Node and Data Node. The NameNode manages the namespace of the file system. It maintains the file system tree and all files and directories in the entire tree. This information is permanently stored on the local disk in the form of two files: the namespace mirror file and the edit log file. The NameNode records the data node information where each block in each file is located, but it does not permanently store the location information of the block. This information is reconstructed by the data node when the system is started.

303. Install Hadoop in the virtual machine created in step 302, and these virtual machines are included as DataNodes by Hadoop.

304. Configure Hadoop's rack awareness strategy, and configure the number of replicas to 2.

When data is written to Hadoop, one DataNode is taken from each of the two racks, so that it is written to two different storage pools. Rack-level reliability guarantees data dual-active and improves overall reliability.

It is understandable that when the storage system is actually built, the number of racks may not be limited to two, and the same strategy may be adopted for multiple racks.

4, FIG. 4 is an embodiment of the present application provides a data storage device 400, the storage device 400 includes: a sending unit 401, configured to send a first control instruction, the first control instruction instructs N storage devices Install data processing software in the N storage devices, create a storage resource pool and a virtual machine in any one of the N storage devices, and any one of the virtual machines uses the created storage resource pool in the corresponding storage device to store data, Any one of the virtual machines is used as an optional data node of the data processing software, and the N is an integer greater than or equal to 2.

The obtaining unit 402 is configured to obtain a configuration instruction. The configuration instruction includes: setting the number of copies M for data storage and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, the The M data nodes are located in M different storage devices, and the M is an integer less than or equal to the N.

The determining unit 403 is configured to determine M data nodes for storing data to be stored according to the configuration instruction.

The processing unit 404 is configured to store the data to be stored in the M data nodes.

In some possible implementation manners, the storage resource pools created in N storage devices are encoded with erasure code EC. In the embodiment of the present application, EC coding is used in the storage resource pool created by the storage device. When data is lost or damaged, the lost data can be calculated to ensure that some data can still be used after loss. Compared with a traditional distributed system that can continue to provide services after 3 copies of data after hardware failures and other faults occur, the embodiment of the present application saves storage space and improves storage utilization.

An embodiment of the present application also provides a data storage system, including N storage devices and an embodiment corresponding to any of the foregoing data storage devices, where N is an integer greater than or equal to 2. The data storage device is shown in FIG. 4, the storage device includes: a sending unit 401, configured to send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, A storage resource pool and a virtual machine are created in any one of the storage devices, and any one of the virtual machines uses the created storage resource pool in the corresponding storage device to store data, and any one of the virtual machines is used as the data processing An optional data node of the software, where the N is an integer greater than or equal to 2.

In some possible implementation manners, the storage resource pools created in N storage devices are coded with erasure code EC. In the embodiment of the present application, EC coding is used in the storage resource pool created by the storage device. When data is lost or damaged, the lost data can be calculated to ensure that some data can still be used after loss. Compared with a traditional distributed system that can continue to provide services after 3 copies of data after hardware failures and other faults occur, the embodiment of the present application saves storage space and improves storage utilization.

In some possible implementations, when the EC encoding uses 8 data blocks and 1 check block 8D1P mode, the storage utilization rate of the hard disk is 88.89%, and when the EC encoding uses 4D1P mode, the storage utilization rate of the hard disk is 80%. When the EC encoding adopts the 8D2P mode, the storage utilization rate of the hard disk is 80%, and when the EC encoding adopts the 4D2P mode, the storage utilization rate of the hard disk is 66.67%.

The embodiment of the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs; when the one or more programs are used by the one or more processors Execution, so that the one or more processors implement the data storage method described in any of the foregoing method embodiments. The method includes:

Storing the data to be stored in the M data nodes.

In some possible embodiments, the storage resource pools created in the N storage devices adopt erasure code EC encoding.

In some possible embodiments, the storage device includes a distributed server or a magnetic array.

In some possible embodiments, the EC encoding includes 8 data blocks 1 check block 8D1P mode, 4D1P mode, 8D2P mode, or 4D2P mode.

In some possible embodiments, the data processing software includes: distributed processing software Hadoop.

In some possible embodiments, the M=2.

In the data storage method provided by the embodiments of the present application, when multiple copies are stored, different copies are located in different storage devices. In this way, when a storage device storing backup data fails, the number of copies actually reduced is one, which is relative to the current one. In some technologies, when a storage device that stores one copy has a problem, it may cause multiple copies to be unusable, which improves storage reliability. When the storage resource pools created in N storage devices are coded with erasure code EC, when data is lost or damaged, the lost data can be calculated to ensure that some data can still be used after loss. Compared with a traditional distributed system that can continue to provide services after 3 copies of data after hardware failures and other faults occur, the embodiment of the present application saves storage space and improves storage utilization.

The embodiment of the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data storage method as described in any of the foregoing method embodiments is implemented. The method includes:

Storing the data to be stored in the M data nodes.

In some possible embodiments, the M=2.

It should be understood that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in a sequence other than the content illustrated or described herein. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to what is clearly listed. Those steps or modules may include other steps or modules that are not clearly listed or are inherent to these processes, methods, products, or equipment.

What is disclosed above is only a preferred embodiment of the present invention. Of course, it cannot be used to limit the scope of rights of the present invention. A person of ordinary skill in the art can understand all or part of the process of implementing the above embodiments and follow the rights of the present invention. The equivalent changes required are still within the scope of the invention.

Claims

A data storage method, characterized in that the method includes the following steps:

Send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, create a storage resource pool and a virtual machine in any one of the N storage devices, any one of the The virtual machine uses the created storage resource pool in its corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is an integer greater than or equal to 2;

Acquire a configuration instruction, the configuration instruction includes: setting the number of copies M for data storage, and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, and the M data nodes are located in M In different storage devices, the M is an integer less than or equal to the N;

Determining, according to the configuration instruction, M data nodes for storing data to be stored;

Storing the data to be stored in the M data nodes.
The data storage method according to claim 1, wherein:

The storage resource pools created in the N storage devices are coded with erasure code EC.
The data storage method according to claim 2, wherein the storage device comprises: a distributed server or a magnetic array.
The data storage method according to claim 2, wherein the EC encoding includes 8 data blocks 1 check block 8D1P mode, 4D1P mode, 8D2P mode, or 4D2P mode.
The data storage method according to claim 1, wherein the data processing software comprises: distributed processing software Hadoop.
The data storage method according to any one of claims 1 to 5, wherein the M=2.
A data storage device, characterized in that it comprises:

The sending unit is configured to send a first control instruction, the first control instruction instructs to install data processing software in N storage devices, and create a storage resource pool and a virtual machine in any one of the N storage devices , Any one of the virtual machines uses the created storage resource pool in the corresponding storage device to store data, any one of the virtual machines is used as an optional data node of the data processing software, and the N is greater than or equal to 2 Integer

The obtaining unit is configured to obtain a configuration instruction, the configuration instruction includes: setting the number of copies M for data storage, and a storage awareness strategy; the storage awareness strategy includes: determining M data nodes for storing data, the M Data nodes are located in M different storage devices, where M is an integer less than or equal to N;

A determining unit, configured to determine M data nodes for storing data to be stored according to the configuration instruction;

The processing unit is configured to store the data to be stored in the M data nodes.
8. The data storage device according to claim 7, wherein the storage resource pools created in the N storage devices are coded with erasure code EC.
The data storage device according to claim 8, wherein the storage device comprises: a distributed server or a magnetic array.
The data storage device according to claim 8, wherein the EC coding includes 8 data blocks 1 check block 8D1P mode, 4D1P mode, 8D2P mode, or 4D2P mode.
The data storage device according to claim 7, wherein the data processing software comprises: distributed processing software Hadoop.
The data storage device according to any one of claims 7 to 11, wherein the M=2.
An electronic device, characterized in that it comprises:

One or more processors;

Storage device for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to any one of claims 1 to 6.
A computer readable medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 is realized.