WO2019170004A1

WO2019170004A1 - Data storage system, method and device

Info

Publication number: WO2019170004A1
Application number: PCT/CN2019/075696
Authority: WO
Inventors: 汪渭春; 王伟; 林鹏
Original assignee: 杭州海康威视系统技术有限公司
Priority date: 2018-03-09
Filing date: 2019-02-21
Publication date: 2019-09-12
Also published as: CN110244904B; CN110244904A

Abstract

Provided are a data storage system, method and device. The storage system comprises storage nodes, wherein the storage nodes are connected via an SAS network; and a target storage node is used for acquiring data to be stored, and carrying out slice processing on the data to be stored to obtain slice data, and is also used for determining a first-type storage node for storing the obtained slice data, acquiring a first-type storage position, in the determined first-type storage node, for storing the slice data, and writing, via the SAS network, the obtained slice data into the acquired first-type storage position. By means of the technical solution provided in the embodiments of the present application, after slice processing is carried out on data to be stored, a target storage node can directly write slice data obtained by means of slice processing into a corresponding storage position respectively, thereby avoiding the occurrence of a transmission delay caused by data transmission, and thus improving the data storage efficiency.

Description

Data storage system, method and device

The present application claims priority to Chinese Patent Application No. 201101193323.5, entitled "A Data Storage System, Method, and Apparatus", filed on March 9, 2018, the entire contents of in.

Technical field

The present application relates to the field of data storage technologies, and in particular, to a data storage system, method, and apparatus.

Background technique

With the increasing scale of computer applications, the demand for storage space is increasing day by day. As a result, cloud storage systems based on cloud storage technologies have emerged. Compared with traditional storage systems, cloud storage systems are scalable and have improved storage. effectiveness.

Each storage node included in the cloud storage system has two types of network ports: a service network port and a data network port. The service network port is used to receive data that needs to be stored. For example, the monitoring data of each monitoring device in the monitoring network. The data network port is used for data transmission between devices in the storage system. Moreover, both the service network port and the data network port are connected by electrical ports, wherein the electrical port can be an ordinary network cable interface or a coaxial cable interface, and the 100 Mbps Ethernet electrical port and the Gigabit Ethernet can be selected according to requirements. Electric port and so on.

After a storage node S in the cloud storage system receives the data to be stored through the service network port, considering a load balancing factor, etc., a part of the data can be stored locally, and then the other data is sent to other storage nodes through the data network port. storage.

When the storage node S stores its local data to other storage nodes, the local data of the storage node S needs to be transmitted to other storage nodes through the network, and then the other storage nodes write the received data to other storage nodes. Local storage. Although the storage node S can store its local data to other storage nodes in the above manner, the storage node S needs to transmit its local data to other storage nodes through the data network port, that is, the electrical port. There is a data delay in this process, resulting in inefficient data storage.

Summary of the invention

The present application provides a data storage system, method and apparatus to solve the problem of data transmission delay, resulting in low data storage efficiency. The specific technical solutions are as follows:

In a first aspect, the embodiment of the present application provides a data storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;

a target storage node, configured to acquire data to be stored, perform slice processing on the data to be stored to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type storage node And storing the first type of storage location of the slice data, and writing the obtained slice data to the acquired first type of storage location through the SAS network.

Optionally, the target storage node acquires the first type of storage location for storing the slice data in the determined first type of storage node, including:

The target storage node is configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: The location of the determined first type of storage node for storing slice data.

Optionally, the target storage node is configured to determine a first type of storage node for storing the obtained slice data, including:

The target storage node is configured to determine a first type of storage node for storing the obtained slice data according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes a management node;

The target storage node is configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence relationship between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored; sending the generated storage index to the management node;

The management node is configured to receive and store a storage index sent by the target storage node.

Optionally, the management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type of storage node; a stored storage index, determining slice data corresponding to the data to be read and a second type of storage location of each slice data, and sending the second type of storage location to the second type storage node;

The second type of storage node is configured to receive the second type of storage location sent by the management node, read slice data from the second type of storage location, and perform restoration processing on the read slice data. Obtaining the data to be read, and sending the data to be read to the target client.

Optionally, the management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage node included in the storage system, to reconstruct the fault. a storage node storing data in the location as a storage node of the third type;

The third type of storage node is configured to reconstruct data stored in the fault location, obtain reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated data. storage location.

Optionally, the management node is specifically configured to acquire a fault location in the fault storage node, and determine, according to a local storage index, data corresponding to the fault location, and data corresponding to the fault location. Slicing data, and obtaining a storage location corresponding to each determined slice data, and transmitting the obtained storage location to a third type of storage node, wherein the third type of storage node is: a data amount of data stored according to the fault location And a storage node selected from the storage node included in the storage system for reconstructing data stored in the fault location;

The third type storage node is configured to receive a storage location sent by the management node, read corresponding slice data from the received storage location, and perform reconfiguration processing on the read slice data to obtain a weight Constructing data, allocating a storage location for the reconstructed data, and writing the reconstructed data to the allocated storage location.

In a second aspect, the embodiment of the present application provides a data storage system, where the storage system includes: a slicing server, at least two storage nodes, and each storage node is connected through a SAS network;

The slice server is configured to acquire data to be stored, perform slice processing on the data to be stored, and obtain slice data; determine, from the at least two storage nodes, a target storage node that receives the slice data, and obtain the obtained Sending slice data to the target storage node;

The target storage node is configured to receive the slice data sent by the slice server, determine a first type of storage node used to store the slice data, and acquire the determined first type storage node for storing The first type of storage location of the slice data is written to the acquired first type of storage location by the SAS network.

Optionally, the target storage node acquires a first type of storage location for storing slice data in the determined first type of storage node, including:

Optionally, the target storage node is configured to determine a first type of storage node for storing slice data, including:

The target storage node is configured to determine a first type of storage node for storing slice data according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes a first management node;

The target storage node is configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the data to be stored and each slice data obtained by the slice processing; sending the generated storage index to the first management node;

The first management node is configured to receive and store a storage index sent by the target storage node.

Optionally, the first management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type of storage node; Determining, according to the locally stored storage index, the slice data corresponding to the data to be read and the second type of storage location of each slice data, and sending the second type of storage location to the second type storage node;

The second type of storage node is configured to receive the second type of storage location sent by the first management node, read slice data from the second type of storage location, and perform the read slice data. The restoration process is performed to obtain the data to be read, and the data to be read is sent to the target client.

Optionally, the first management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage node included in the storage system for reconfiguring a storage node storing data stored in the fault location as a storage node of the third type;

Optionally, the first management node is specifically configured to acquire a fault location in the fault storage node, determine, according to a local storage index, data corresponding to the fault location, and data stored in the fault location. Corresponding slice data, and obtaining a storage location corresponding to each determined slice data, and sending the obtained storage location to a third type storage node, wherein the third type storage node is: storing data according to the fault location a quantity of data, from a storage node included in the storage system, a storage node selected to reconstruct data stored in the fault location;

The storage node of the third type is configured to receive a storage location sent by the first management node, read corresponding slice data from the received storage location, and perform reconstruction processing on the read slice data. Reconstructed data is obtained, a storage location is allocated for the reconstructed data, and the reconstructed data is written to the allocated storage location.

Optionally, the storage system further includes a second management node;

Determining, by the slice server, a target storage node that receives the slice data from the at least two storage nodes, including:

The slice server is configured to send a second storage request to the second management node;

The second management node is configured to receive the second storage request sent by the slice server, determine a target storage node from the storage nodes included in the storage system, and send the identifier of the determined target storage node To the slicing server;

The slice server is configured to receive the identifier sent by the second management node, and determine the target storage node corresponding to the identifier.

Optionally, the second management node determines the target storage node from the storage nodes included in the storage system, including:

The second management node is configured to determine a target storage node from the storage nodes included in the storage system according to at least one of the following information:

a storage node that has been allocated by the second management node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

In a third aspect, the embodiment of the present application provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;

The method includes:

Obtain data to be stored;

And performing slice processing on the data to be stored to obtain slice data;

Determining a first type of storage node for storing the obtained slice data;

Obtaining, in the determined first type of storage node, a first type of storage location for storing the slice data;

The obtained slice data is written into the acquired first type of storage location through the SAS network.

Optionally, the step of obtaining the first type of storage location for storing the slice data in the determined first type of storage node includes:

Sending a first storage request to the determined first type of storage node;

Receiving the determined first type of storage location sent by the first type of storage node, where the first type of storage location is: a location allocated by the first type of storage node for storing slice data.

Optionally, the step of determining a first type of storage node for storing the obtained slice data includes:

A first type of storage node for storing the obtained slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes: a management node; after the step of writing the obtained slice data to the acquired first type storage location by using the SAS network, the method further includes:

Generating a storage index of each of the obtained slice data, wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the to-be-stored data Corresponding relationship between each slice data obtained by slicing the data;

The generated storage index is sent to the management node.

In a fourth aspect, the embodiment of the present application provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes and a slicing server, and each storage node is connected through a SAS network. ;

The method includes:

Receiving, by the slice server, slice data sent by the slice server, where the slice server performs slice processing on the data to be stored;

Determining a first type of storage node for storing the slice data;

The slice data is written to the acquired first type of storage location through the SAS network.

Optionally, the step of acquiring the first type of storage location for storing the slice data in the determined first type of storage node includes:

Sending a first storage request to the determined first type of storage node;

Optionally, the step of determining a first type of storage node for storing slice data, comprising:

A first type of storage node for storing slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes a management node;

After the step of writing the slice data to the acquired first type of storage location through the SAS network, the method further includes:

Generating a storage index of each of the received slice data, wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the Corresponding relationship between each slice data obtained by slice processing of the data to be stored;

The generated storage index is sent to the management node.

In a fifth aspect, the embodiment of the present application provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;

The device includes:

a first acquiring module, configured to acquire data to be stored;

a slice processing module, configured to slice the data to be stored to obtain slice data;

a first determining module, configured to determine a first type of storage node for storing the obtained slice data;

a second acquiring module, configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data;

And a first writing module, configured to write the obtained slice data to the acquired first type of storage location through the SAS network.

Optionally, the second obtaining module includes:

a first sending submodule, configured to send a first storage request to the determined first type of storage node;

a first receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.

Optionally, the first determining module is specifically configured to:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes a management node;

The device also includes:

a first generation module, configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, Corresponding relationship between the data to be stored and each slice data obtained by slicing the data to be stored;

The first sending module is configured to send the generated storage index to the management node.

In a sixth aspect, the embodiment of the present application provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes and a slicing server, and each storage node is connected through a SAS network. ;

The device includes:

a receiving module, configured to receive the slice data sent by the slice server, where the slice data is: the data obtained by the slice server to slice the processed data;

a second determining module, configured to determine a first type of storage node for storing the slice data;

a third acquiring module, configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data;

And a second writing module, configured to write the slice data to the acquired first type of storage location through the SAS network.

Optionally, the third obtaining module includes:

a second sending submodule, configured to send a first storage request to the determined first type of storage node;

a second receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.

Optionally, the second determining module is specifically configured to:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, the storage system further includes a management node;

The device also includes:

a second generation module, configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes: a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored;

And a second sending module, configured to send the generated storage index to the management node.

In a seventh aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

a memory for storing a computer program;

The processor, when executed to execute a program stored on the memory, implements one of the data storage method steps described in any of the above.

In an eighth aspect, the embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a computer program, and when the computer program is executed by the processor, implements one of the foregoing data. Store method steps.

In the technical solution provided by the embodiment of the present application, the storage system includes at least two storage nodes, and each storage node is connected through a SAS network, and the target storage node is configured to acquire data to be stored, and perform slice processing on the stored data to obtain slice data. Determining a first type of storage node for storing the obtained slice data, and acquiring a first type of storage location for storing slice data in the determined first type of storage node, and writing the obtained slice data through a SAS network The first type of storage location obtained. After the data to be stored is sliced, the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application and the technical solutions of the prior art, the following description of the embodiments and the drawings used in the prior art will be briefly introduced. Obviously, the drawings in the following description are only Some embodiments of the application may also be used to obtain other figures from those of ordinary skill in the art without departing from the scope of the invention.

1 is a schematic diagram of a first structure of a data storage system according to an embodiment of the present application;

2 is a schematic diagram of a second structure of a data storage system according to an embodiment of the present application;

FIG. 3 is a first flowchart of a data storage method according to an embodiment of the present application;

FIG. 4 is a second flowchart of a data storage method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a first structure of a data storage device according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a second structure of a data storage device according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

In order to avoid the transmission delay caused by the data transmission, the data storage efficiency can be improved. The embodiment of the present application provides a data storage system, method, and device, where the storage system includes at least two storage nodes, and each storage node Connected through a SAS network, the target storage node is a storage node that acquires data to be stored;

a target storage node, configured to acquire data to be stored, perform slice processing on the stored data to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type of storage node for use in The first type of storage location of the sliced data is stored, and the obtained sliced data is written into the acquired first type of storage location through the SAS network.

After the data to be stored is sliced, the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.

A data storage system provided by an embodiment of the present application is first introduced.

A data storage system is provided in the embodiment of the present application. As shown in FIG. 1 , the storage system 100 includes: at least two storage nodes 110, each storage node 110 is connected through a SAS network, and the target storage node is configured to acquire data to be stored. Storage node.

The storage system 100 and the target storage node are respectively described below.

First, the storage system 100

The storage system 100 can include at least two storage nodes 110.

For the storage nodes included in the storage system 100, each storage node may include multiple hard disks, and the hard disks are used to store corresponding data. When the data is stored in the storage node, the storage node can store the data on different hard disks separately, and can also store the data on the same hard disk.

Each storage node in the storage system 100 is interconnected. The interconnection may be interconnected through a SAS network, and each storage node includes at least a SAS interface.

Among them, SAS (Serial Attached SCSI) is a serial connection SCSI interface. Among them, SCSI (Small Computer System Interface) is a small computer system interface, which is a general interface standard. For example, SCSI can be used for computers and hard disks. Interface standard between devices such as floppy drives, optical drives, printers, and scanners.

One way for each storage node to connect through the SAS network can be connected by means of a SAS switch. Each storage node in the storage system is connected to the SAS switch, so that the storage nodes can be connected to each other through the SAS switch. Storage nodes can be connected to SAS switches through their respective SAS interfaces. The manner in which the storage nodes are connected through the SAS network may include other methods in addition to the manner of the above-mentioned SAS switch, which is not limited herein.

In the case where the storage nodes in the storage system 100 are connected through the SAS network, the storage nodes are visible to each other, that is, for each storage node, the hard disks included in other storage nodes in the storage system 100 can be seen. Seen here can be understood as: each storage node can perform write operations and read operations on each hard disk that is seen. That is to say, each storage node can see all the hard disks of the storage system 100, and each storage node can directly write data to the hard disk that can be seen, or can read data from the hard disk that can be seen.

For example, the storage system includes a storage node A, a storage node B, and a storage node C, and the storage node A, the storage node B, and the storage node C are connected by a SAS network, and the storage node A includes a hard disk A1, a hard disk A2, and a storage node. B includes the hard disk B1 and the hard disk B2. The storage node C includes the hard disk C1 and the hard disk C2. Then, for the storage node A, the hard disk B1, the hard disk B2, the hard disk C1, and the hard disk C2 can be seen, and the data can be directly written to the hard disk. In each of the B1, the hard disk B2, the hard disk C1, and the hard disk C2, the respective stored data may be read from the hard disk B1, the hard disk B2, the hard disk C1, and the hard disk C2.

In the storage system 100, each storage node is connected by a SAS network, and each storage node can perform read and write operations, thereby solving the problem of mutual exclusion between storage nodes.

Second, the target storage node

The steps performed by the target storage node are described separately below.

The first step is to acquire data to be stored.

The storage system 100 may further include an access node. The manner in which the target storage node obtains the data to be stored may be: the target storage node receives the data to be stored sent by the access node in the storage system. The access node is connected to the client external to the storage system 100, and is configured to receive data sent by the client. For example, the access node may be connected to the monitoring device for receiving video data sent by the monitoring device.

In an implementation manner, the access node in the storage system is configured to receive data to be stored sent by the client, and after receiving the data to be stored, the access node may send the to-be-stored data to the storage node, where the storage node is the target. Storage node. The access node may select a target storage node for receiving data to be stored from at least two storage nodes included in the storage system by using a preset policy, and the policy may be customized.

The access node receives the data to be stored and the data to be stored sent by the target storage node and the access node can be implemented through the service network port. The service network port is used to receive service data, and the service data may be from the outside of the storage system. Storing data.

For example, the service data may be the video data of the monitoring device, and the monitoring device sends the video data to the access node in the storage system through the service network port. At this time, the video data is the data to be stored, and the access node receives the received data. The video data is sent to the target storage node through its own service network port, so that the target storage node can obtain the video data to be stored.

In the second step, after acquiring the data to be stored, the target storage node performs slice processing on the stored data to obtain slice data.

In an implementation manner, the target storage node may perform data processing on the data by using Erasure Coding (EC).

By performing the slicing process on the stored data by the erasure code technique, a preset number N pieces of data can be obtained, and according to the obtained N pieces of data, a preset number of M pieces can be obtained. From the sum of the data slice and the test slice M + N, is the number of slice data obtained. The type of the erasure code in the erasure code technology may include multiple types, and N and M may be determined by the type of the erasure code. In different erasure code types, N and M may be different.

For example, an RS (Reed-Solomon) code is an erasure code commonly used in a storage system. The N code of the RS code is 5, and M is 3. The data is sliced by using the RS code, and 5 pieces of data can be obtained. 3 test pieces, and the final slice data obtained is 8.

For the slice using the erasure code technique, the obtained M check slices can function as a fault tolerance. In the obtained M+N slice data, when the number of slice data in which there is an error is less than or equal to M, the N pieces of data can be restored by the reconstruction algorithm in the erasure code technique. Therefore, the erasure code technology can play a role of fault tolerance for the slice data, improving the reliability of the data storage.

For example, the slice data obtained by the erasure code technique includes five data slices and three check slices, wherein the five data slices are: data slice A, data slice B, data slice C, and data slice D. , data piece E, three check pieces are: test piece A, test piece B, test piece C. When there is an error in the data slice A, the data slice B, and the check slice A, the data slice A, the data slice B, the data slice C, the data slice D, and the data slice E can be restored by the reconstruction algorithm in the erasure code technique. .

The manner in which the target storage node performs the slicing process is not limited to the erasure code technique in the above embodiment, and may be other types of slice processing, which is not limited herein.

In a third step, a first type of storage node for storing the obtained slice data is determined, and a first type of storage location for storing slice data in the determined first type of storage node is obtained.

The target storage node may determine, according to the number of slice data and the data volume of each slice data, a first type of storage node for storing slice data, and obtain the determined first type of storage node for storing slice data. The first type of storage location.

When the target storage node determines the first type of storage node, first, the number of the first type of storage nodes may be determined. The manner of determining the number of the first type of storage nodes may be in various manners.

In the first manner, it can be determined that the number of the first type of storage nodes is the same as the number of the slice data. For example, when the number of slice data obtained by the target storage node is eight, the number of the first type of storage nodes determined is also eight. .

In the first mode, it can be considered that one slice data corresponds to one first type storage node, that is, only one slice data is stored in each of the determined first type storage nodes, and is stored in each first type storage node. The slice data is not the same.

For example, the slice data obtained by the target storage node includes: slice data A, slice data B, and slice data C, and the determined three first type storage nodes for storing the slice data are: the first type storage node 1, The first type storage node 2 and the first type storage node 3, and the target storage node assigns the slice data A to the first type storage node 1, the slice data B to the first type storage node 2, and distributes the slice data C To the first type of storage node 3, then the first type of storage node 1 stores slice data A, the first type of storage node 2 stores slice data B, and the first type of storage node 3 stores slice data C.

In a second manner, it may be determined that the number of storage nodes of the first type is smaller than the number of slice data. In this manner, at least one storage node of the first type of storage nodes that are determined needs to store more than one slice. data. Wherein, the first type of storage nodes that need to store more than one slice data may be determined according to the storage state of each storage node. The storage state may include the remaining storage space of the storage node, load pressure, and the like.

For example, the number of slice data is four, and the number of the first type of storage nodes determined is three, namely: a first type storage node 1, a first type storage node 2, and a first type storage node 3, and Among the three first type storage nodes, compared with the first type storage node 2 and the first type storage node 3, the remaining storage space of the first type storage node 1 is larger, and the load pressure is smaller, therefore, the first can be determined. One type of storage node 1 stores two of the slice data.

After the target storage node determines the number of the first type of storage nodes, the determined number of storage nodes may be selected from the storage system 100 according to a preset policy, and the selected storage node is the first type of storage node. The preset policy may be customized. Specifically, the manner of selecting a storage node according to the preset policy is described in detail in the following embodiments, and details are not described herein.

In addition, when selecting the first type of storage node, the amount of data of each slice data may also be considered, and the remaining storage space of the selected first type of storage node is at least greater than the data amount of the corresponding stored slice data.

After the target storage node determines the first type of storage node, the first type of storage location for storing the slice data in the determined first type of storage node may be obtained.

The storage location may be a hard disk in the storage node, and different storage locations are different hard disks. When there are multiple partitions in each hard disk, the storage location may also be a partition specifically referring to the hard disks in the storage node.

For example, the storage node includes the hard disk 1 and the hard disk 2, and the data is stored in the hard disk 1. At this time, the storage location of the data may refer to the hard disk 1. When the hard disk 1 includes the partition 1 and the partition 2, the data is specifically stored in the partition of the hard disk 1. In this case, at this time, the storage location of the data refers to the partition 1 of the hard disk 1.

The storage node may include multiple hard disks, and the storage state of each hard disk is different. After determining the storage node of the first type, the storage state of each hard disk in the storage node of the first type may be selected to store the slice data. The hard disk is the first type of storage location.

The storage state of the hard disk may be the remaining storage space, the load pressure, etc. When the hard disk for storing the slice data is selected, the storage state may be selected according to only one of the storage states, and may also be selected according to multiple storage states of the hard disk. It is not limited here.

For example, the hard disk with the largest remaining storage space is selected from the first type of storage nodes as the hard disk storing the slice data only according to the remaining storage space. Considering the remaining storage space and load pressure, the hard disk with the largest remaining storage space and the lowest load pressure can be selected from the first type of storage nodes as the hard disk storing the slice data.

In addition, the first type of storage location for storing the slice data in the first type of storage node may be determined by the first type of storage node itself, or may be specified by the target storage node, and then applied to the corresponding first type of storage node. This part is described in detail in the following embodiments and will not be described in detail herein.

In the fourth step, the obtained slice data is written into the acquired first type storage location through the SAS network.

In the storage system 100, each storage node may be connected through a SAS network. Therefore, the target storage node can know the storage location in each of the other storage nodes in the storage system 100, the target storage node can read the corresponding data from the other storage nodes, and can also write the data to the storage location in the other storage nodes. .

Therefore, after acquiring the slice data and the first type of storage location corresponding to each slice data, the target storage node may directly write the slice data into the corresponding first type storage location.

For example, the target storage node obtains the slice data A, and determines that the slice data A corresponds to the first type storage location A of the first type storage node A, and the target storage node can directly write the slice data A through the SAS network. The first type of storage node A is stored in location A.

In one embodiment, the first type of storage location may be determined by the first type of storage node itself.

After the target storage node determines the first type of storage node, the target storage node may send the first storage request to the determined first type of storage node. After receiving the first storage request, the first type of storage node may allocate a corresponding storage location according to the first storage request, where the storage location is the first type of storage location, and the first type of storage node stores the allocated first category. The location is sent to the target storage node.

The first storage request sent by the target storage node may include any one or more of data type, data volume, data identifier, and the like of the slice data to be stored.

The data type may include video data, image data, and the like. The amount of data reflects the size of the storage space occupied by the slice data, and the storage node allocates the corresponding storage space according to the data amount of the slice data. The data identification can be used to verify the slice data to ensure that the slice data is written to the correct storage location.

In an embodiment, the target storage node may be configured to determine, according to at least one of the following information, a first type of storage node used to store the obtained slice data: a storage node that has been allocated by the target storage node, and a storage system The survival state of each storage node, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.

The following four types of information are introduced separately.

1. The target storage node determines the first type of storage node based on the storage node that has been allocated by the target storage node.

The target storage node can record the identity of each of the determined first type of storage nodes.

The first implementation manner is a centralized policy, that is, when the target storage node determines the first type of storage node, the storage node that is the same as the storage node of the first type determined last time may be selected.

For example, the storage system 100 includes: a storage node 1, a storage node 2, a storage node 3, a storage node 4, and a storage node 5. The last determined first type of storage node recorded by the target storage node is the storage node 1 and the storage. Node 2, then, when the target storage node determines the first type of storage node this time, the storage node 1 and the storage node 2 are still selected as the first type of storage node.

When the first type of storage node determined last time fails, the storage space is full, etc., the first type of storage node determined last time can no longer be used as the first type of storage node, but can pass the target. The storage node determines the other basis of the first type of storage node to re-determine the first type of storage node.

The second implementation manner is a decentralized strategy, that is, when the target storage node determines the first type of storage node, the storage node different from the previously determined first type storage node may be selected. It can be divided into two cases: in the first case, a storage node different from the first type of storage node determined last time can be selected; in the second case, it can be selected differently from the first type of storage node determined many times before. Storage node.

The following two cases are described separately.

In the first case, when determining the first type of storage node, the target storage node may select a storage node different from the first type of storage node determined last time. That is to say, the determined first type of storage node may be different from the first type of storage node determined last time.

For example, the storage system 100 includes: a storage node 1, a storage node 2, a storage node 3, a storage node 4, and a storage node 5. The last determined storage node recorded by the target storage node is a storage node 1 and a storage node 2, Then, when the target storage node determines the first type of storage node, the storage node 3, the storage node 4, and the storage node 5 may be selected as the storage node of the first type.

In the second case, when determining the first type of storage node, the target storage node may select a storage node different from the first type of storage node determined a plurality of times before. Among them, the number of times before can be customized.

In order to make the storage of the slice data in the storage nodes included in the storage system 100 distributed, the target storage node may employ a rotation policy when determining the first type of storage nodes.

An order sorting table for each storage node included in the storage system 100 may be preset, and the sorting table records the sorting of the identifiers corresponding to the storage nodes included in the storage system 100, and the sorting rules may be customized. Each time the target storage node determines the first type of storage node, the identifier may be allocated according to the identifier sorting table, and the identifier of the first type of storage node determined each time is: the first type of storage node determined last time Identifies the next identity in the identity sort table.

When the determined remaining storage space of the first type of storage node is insufficient to store the slice data, the first type of storage node may be re-determined.

2. The target storage node determines the first type of storage node according to the survival state of each storage node in the storage system.

The survival state of the storage node is: a fault state or an intact state of the storage node.

The storage node can periodically report the survival status of the storage node to the target storage node. For the target storage node, the storage state of each storage node can also be known in real time. In this way, when the target storage node determines the first type of storage node, the failed storage node can be eliminated in time, and only the target storage node is determined from the intact storage node. In this way, the efficiency of slicing data storage can be improved.

3. The target storage node determines the first type of storage node based on: the remaining storage space of each storage node in the storage system.

The remaining storage space of each storage node may be sorted, and for a storage node with a large remaining storage space, it may be preferentially determined as the first type of storage node. In this way, the target storage node may update the order of the remaining storage spaces of each storage node after each determination of the storage node of the first type, and the obtained new remaining storage space sorting may be used as the next determination of the first type of storage. The basis of the node. In this way, the remaining storage space of each storage node in the storage system 100 can be equalized.

4. The target storage node determines the basis of the first type of storage node as: the load pressure of each storage node in the storage system.

The load pressure of each storage node included in the storage system 100 may be sorted, and for a storage node with a small load pressure, it may be preferentially determined as the first type of storage node. In this way, the target storage node may update the load pressure order of each storage node after each determination of the storage node of the first type, and the obtained new load pressure ordering may be used as the basis for determining the first type of storage node next time. . In this way, the load pressure of each storage node in the storage system 100 can be equalized.

For the above four types of information, the basis of the first type of storage node may be separately determined as the target storage node, and any two, three or four combinations may be used as the basis for the target storage node to determine the first type of storage node. There is no limit here.

In an implementation manner, the storage system may further include a management node, where the management node in the storage system may be one or multiple. In a case where a plurality of management nodes are included in the storage system, the collection of the plurality of management nodes is a management cluster. Therefore, it can be considered that the storage system includes a management cluster, and the management cluster includes a plurality of management nodes.

The target storage node and the management node in this embodiment are respectively introduced below.

First, the target storage node

On the basis of the embodiment corresponding to FIG. 1 above, in the embodiment, after the target storage node performs the slice processing on the data to be stored to obtain each slice data, the correspondence between the obtained slice data and the data to be stored may be recorded. Relationship, the correspondence is a correspondence relationship of storage index records of one slice data.

For example, the slice data obtained by the target storage node after the data A is sliced includes: slice data 1, slice data 2, and slice data 3. The correspondence relationship of the target storage node records includes: data A corresponds to slice data 1, and data A and The slice data 2 corresponds, and the data A corresponds to the slice data 3.

After the target storage node writes the received slice data to the corresponding first type storage location through the SAS network, the corresponding relationship between each slice data and the first type of storage location stored by the slice data may be recorded. The relationship is also another correspondence of the storage index records of one slice of data.

For example, the slice data obtained by the target storage node includes: slice data A, slice data B, and slice data C, wherein the storage index 1 of the slice data A is the slice data A corresponding to the storage location A, and the storage index 2 of the slice data B The storage index 3 of the slice data C corresponds to the storage location C, and the storage index generated by the target storage node includes the storage index 1, the storage index 2, and the storage index 3.

In this way, the storage index generated by the target storage node includes: a storage index corresponding to each slice data obtained by the target storage node.

Second, the management node

When the management system includes the management cluster, a management node may be selected from the management cluster according to a preset policy, and used to receive and store the storage index sent by the target storage node.

The preset policy may be: determining, according to at least one of the following information, a management node for receiving and storing a storage index sent by the target storage node: the allocated management node, and the survival status of each management node in the management cluster. Manage the load pressure of each management node in the cluster.

The policy for determining the management node is similar to the policy for determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the first embodiment, the second base, and the fourth base in the foregoing embodiments, and details are not described herein again.

With this embodiment, the management node can more accurately and quickly query the storage location of storing the slice data in the storage system through the stored storage index, thereby improving data processing efficiency.

In one embodiment, based on the above embodiments, the management node and the second type storage node in the embodiment are respectively introduced below.

First, the management node

The management node can also be used to receive a read request from the target client for the data to be read.

The target client may be a device external to the storage system that sends a read request, for example, a client device such as a computer, a mobile phone, or a tablet. The data to be read may be data stored in the storage system.

The management node selects the second type of storage node according to a preset policy. In an implementation manner, the second type storage node may be selected from the storage nodes included in the storage system according to at least one of the following information: The storage node allocated by the node, the surviving state of each storage node in the storage system, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.

The policy for the management node to select the second type of storage node is similar to the strategy for determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the foregoing embodiment, the second, the third, and the fourth, and will not be described herein. .

The management node may determine the slice data corresponding to the data to be read and the second type of storage location of each slice data according to the locally stored storage index.

For example, the management node may query the obtained slice data of the data to be read according to the locally stored storage index, including: slice data A, slice data B, and slice data C, and may query and obtain a storage location corresponding to each slice data: slice data. A is stored in the storage location A, the slice data B is stored in the storage location B, and the slice data C is stored in the storage location C. Then, the storage location A, the storage location B, and the storage location C are the second type of storage location.

Second, the second type of storage node

The storage nodes included in the storage system 100 are connected by a SAS network, and the second type of storage nodes can read data from other storage nodes in the storage system. Thus, in this embodiment, after receiving the second type of storage location sent by the management node, the second type of storage node can read the stored slice data from the second type of storage location.

For example, the second type of storage location sent by the management node includes a storage location A, a storage location B, and a storage location C, wherein the storage location A stores slice data A, the storage location B stores slice data B, and the storage location C stores slice Data C, then, for the second type of storage node, the slice data A is read from the storage location A, the slice data B is read from the storage location B, and the slice data C is read from the storage location C.

The second type of storage node can perform data restoration processing on the read slice data to obtain data to be read required by the target client.

The second type of storage node performing data restoration processing may be an erasure code technique, and the number of slice data read by the second type storage node may be N+M, where N is data. The number of slices, M is the number of slices.

The number of slice data read by the second type of storage node may be less than N+M and greater than or equal to N, because the fault tolerance characteristic of the erasure code technology itself can restore the slice data to N+M, and then perform data restoration. Processing to obtain data to be read.

In one embodiment, based on the above embodiments, the management node and the third type storage node in the embodiment are respectively introduced below.

First, the management node

The management node can perform overall management on the storage nodes included in the storage system 100, and can obtain state information of each storage node in real time. The status information of the storage node may include at least one of failure information of the storage node, remaining storage space information, and hard disk status information included in the storage node.

There are many ways to obtain the status information of a storage node.

The first type of obtaining may be that each storage node actively sends its own state information to the management node, where the storage node may send according to a preset frequency, and the preset frequency may be customized. For example, in order to ensure that the state information of each storage node acquired by the management node is real-time, the frequency of the transmission may be set higher; in addition, when the storage node is in a sudden state such as a failure or a storage is full, the storage node may immediately Send the information of the emergency to the management node.

In the second method, the management node actively obtains the state information of each storage node from each storage node in the storage system.

The manner in which the management node obtains the state information of the storage node is not limited to the above two types, and may also include other acquisition modes, which are not limited herein.

According to any one of the foregoing manners, the management node may obtain the fault information of the faulty storage node. In this embodiment, the storage node fault may be a storage location fault in the storage node, causing the data stored in the storage location to be damaged. Can't read.

The fault information may include the data amount of the data stored in the fault location, and the fault information may further include at least one of the following information: a data type of the data stored in the fault location, data identifier information, and the like.

In an implementation manner, the management node acquires the fault location in the fault storage node, determines the data corresponding to the fault location according to the local storage index, and the slice data corresponding to the data stored in the fault location, and obtains the determined slice data. Corresponding storage location, the obtained storage location is sent to the third type of storage node.

Specifically, the fault location acquired by the management node may be: after the faulty storage node that identifies the fault identifies the fault location, the fault location is reported to the management node. After obtaining the fault location, the management node may determine, according to the correspondence between the slice data in the locally stored storage index and the first type of storage location corresponding to the slice data, the data corresponding to the fault location may be determined; Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored, thereby determining the slice data corresponding to the data stored in the fault location; and finally corresponding to the slice data according to the slice data in the storage index. For a correspondence of storage locations, the storage locations corresponding to the determined slice data can be obtained. The determined storage location is sent to a third type of storage node.

The management node may select the third type of storage node from the storage nodes included in the storage system according to the data amount of the data stored in the fault location. In order to facilitate the reconstruction process and to store the reconstructed data, the available storage space of the third type of storage node is larger than the data amount of the data stored by the faulty location. The third type of storage node may be selected according to a preset policy, and the third type of storage node may be selected from the storage nodes included in the storage system according to at least one of the following information: The storage node allocated by the node, the surviving state of each storage node in the storage system, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.

The policy of selecting the storage node of the third type is similar to the policy of determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the foregoing embodiment according to the first, the second, the third and the fourth, and will not be described herein. .

The selected third type of storage node may perform reconstruction processing on the data, and the reconstruction processing may be an erasure coding technique.

Second, the third type of storage node

In order to recover the data stored in the fault location, the third type of storage node may perform reconstruction processing on the data stored in the fault location, and an implementation manner of the reconstruction processing may be stored in the fault location by using erasure code technology. The data is reconstructed, and then the reconstructed data is obtained, and the obtained reconstructed data is the original data stored in the fault location.

For example, the data stored in the fault location is data A, and the reconstructed data obtained by the reconstruction process by the erasure code technique is data A.

In an implementation manner, the third type storage node receives the storage location sent by the management node, reads the corresponding slice data from the received storage location, and reconstructs the read slice data to obtain the reconstructed data. .

After the third type of storage node obtains the reconstructed data, the corresponding storage location may be allocated for the reconstructed data. The first allocation mode, the storage location allocated by the storage node of the third type is a storage location in a storage node, so that the reconstructed data can be directly stored to the allocated storage location. The remaining storage space of the allocated storage location is larger than the amount of data of the reconstructed data.

For example, if the reconstructed data obtained by the third type of storage node is data A, and the data amount of the data A is 10M, then the third type of storage node may allocate a storage location for the data A, and the remaining space of the storage location is greater than 10M. In this way, data A can be written directly to the storage location.

The second allocation method can allocate multiple storage locations for the reconstructed data and perform slice storage on the reconstructed data. In this allocation method, the reconstructed data may be first sliced to obtain slice data, and each slice data is separately written into the allocated storage location. The third type of storage node may perform slice processing by means of erasure coding technology.

For the above two modes of distribution, the storage location allocated for the reconstruction data may be the hard disk of the storage node of the third type of storage node or the hard disk of other storage nodes of the storage system 100. This is not limited here.

With this embodiment, even if the storage location in the storage node included in the storage system fails, the data in the fault location can be restored and reconstructed, and the data is reconstructed and stored in other non-faulty hard disks. In this way, the problem of data loss due to a storage location failure is avoided, and the reliability of the stored data can be improved.

As shown in FIG. 2, the storage system 100 includes: a slicing server 120, at least two storage nodes 110, and each storage node 110 is connected through a SAS network;

The slice server 120 is configured to acquire data to be stored, perform slice processing on the stored data, and obtain slice data; determine, from at least two storage nodes, a target storage node that receives the slice data, and send the obtained slice data to the target storage node;

a target storage node, configured to receive slice data sent by the slice server 120, determine a first type of storage node for storing slice data, and obtain a first type of storage location for storing slice data in the determined first type of storage node. And the received slice data is written into the first type of storage location obtained by the SAS network.

The slice data obtained by the target storage node in the embodiment is sent by the slice server 120, and after the target storage node receives the slice data, the operation performed by the target storage node in the embodiment corresponding to FIG. 1 is performed. For details, refer to the description of the target storage node in the embodiment corresponding to FIG. 1 above, and details are not described herein again.

Targeting the slicing server 120

The slice server 120 is configured to acquire data to be stored, perform slice processing on the stored data, and obtain slice data; determine a target storage node that receives the slice data from the at least two storage nodes, and send the obtained slice data to the target storage node.

The respective steps performed by the slicing server 120 will be described below.

In the first step, the data to be stored is obtained, and the data to be stored is sliced to obtain slice data.

For the step, refer to the description of the embodiment in FIG. 1 that the target storage node is used to obtain the data to be stored, and the data to be sliced is processed to obtain the slice data, and details are not described herein.

In a second step, the target storage node that receives the slice data is determined from the at least two storage nodes, and the obtained slice data is sent to the target storage node.

In one embodiment, the storage system 100 further includes a second management node. After the slice data is sliced to obtain the slice data, the slice server 120 may send a second storage request to the second management node, where the second storage request is a request by the slice server 120 to allocate the storage node to the second management node. The second storage request may carry related information of the slice data, for example, the number of slice data obtained by the slice server 120, the size of the slice data, and the like.

The second management node receives the second storage request sent by the slicing server 120, and determines the target storage node from the storage nodes included in the storage system according to the second storage request.

In an implementation manner, the second management node may determine, from the storage nodes included in the storage system, a target storage node according to at least one of the following information: a storage node that has been allocated by the second management node, and each storage in the storage system. The surviving state of the node, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.

The implementation manner is similar to the policy for determining the storage node of the first type in the implementation manner corresponding to the foregoing embodiment of FIG. 1, and may refer to the basis of the foregoing, the second, the third, and the fourth according to the foregoing embodiments, and details are not described herein again.

After determining the target storage node, the second management node sends the determined identifier of the target storage node to the slice server 120. The identifier may be at least one of an IP (Internet Protocol) address, a MAC (Media Access Control) address, a code, and the like.

After receiving the identifier sent by the second management node, the slice server 120 can determine the target storage node corresponding to the identifier, and send the obtained slice data to the target storage node.

In one embodiment, the target storage node obtains the first type of storage location for storing the slice data in the determined first type of storage node, and may include:

And a target storage node, configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined first The location allocated by the class storage node to store the slice data.

The embodiment in this embodiment is the same as the embodiment corresponding to FIG. 1 above, and may be referred to the foregoing embodiment, and details are not described herein again.

In an embodiment, the target storage node is configured to determine a first type of storage node for storing slice data, and may include:

a target storage node, configured to determine, according to at least one of the following information, a first type of storage node for storing slice data: a storage node that has been allocated by the target storage node, a survival state of each storage node in the storage system, and storage The remaining storage space of each storage node in the system, and the load pressure of each storage node in the storage system.

In an embodiment, the storage system may further include a first management node, where the first management node and the second management node may be the same management node, and may be different management nodes, which are not limited herein. a storage node for generating a obtained storage index of each slice data, wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, and data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored; and transmitting the generated storage index to the first management node.

In an implementation manner, on the basis of the foregoing implementation manner, the first management node may further be configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system. As a second type of storage node, according to the locally stored storage index, the slice data corresponding to the data to be read and the second type storage location of each slice data are determined, and the second type storage location is sent to the second type storage node.

The second type of storage node may be configured to receive the second type of storage location sent by the first management node, read the slice data from the second type of storage location, and perform the reduction processing on the read slice data to obtain a to-be-read Data and send the data to be read to the target client.

In an embodiment, the first management node may be further configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage location node included in the storage system for reconstructing the fault location. A storage node that stores data as a third type of storage node.

The third type of storage node can also be used to reconstruct the data stored in the fault location, obtain the reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated storage location.

In an implementation manner, on the basis of the foregoing implementation manner, the first management node is specifically configured to acquire a fault location in the fault storage node, determine, according to the local storage index, data corresponding to the fault location, and store the fault location. Data corresponding to the slice data, and obtain the storage location corresponding to each slice data, and send the obtained storage location to the third type storage node, and the third type of storage node is: the data volume of the data stored according to the fault location And a storage node selected from the storage node included in the storage system for reconstructing data stored in the fault location.

The third type of storage node may be configured to receive a storage location sent by the first management node, read corresponding slice data from the received storage location, and reconstruct the processed slice data to obtain a reconstruction. Data, allocate storage locations for reconstructed data, and write reconstructed data to the allocated storage locations.

The implementation in this embodiment is the same as the implementation in the foregoing embodiment, and may be referred to an implementation in the foregoing embodiment, and details are not described herein.

Corresponding to the system embodiment corresponding to FIG. 1 above, the embodiment of the present application further provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node passes through a SAS network. connection;

As shown in FIG. 3, the method includes the following steps:

S301. Acquire data to be stored.

S302, performing slice processing on the stored data to obtain slice data;

S303. Determine a first type of storage node used to store the obtained slice data.

S304. Acquire a first type of storage location for storing slice data in the determined first type of storage node.

S305. Write the obtained slice data to the acquired first type storage location through the SAS network.

Optionally, in an implementation manner, the step of acquiring the first type of storage location (S304) for storing the slice data in the determined first type of storage node may include the following steps:

Sending a first storage request to the determined first type of storage node;

Receiving the determined first type of storage location sent by the first type of storage node, the first type of storage location is: the determined location of the first type of storage node for storing slice data.

Optionally, in an implementation manner, the step of determining a first type of storage node (S303) for storing the obtained slice data may include the following steps:

a storage node that has been allocated by the target storage node;

The survival status of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, in an implementation manner, the storage system may further include a management node;

After the step of writing the obtained slice data to the acquired first type storage location (S305) through the SAS network, the method may further include the following steps:

Generating a storage index of each slice data obtained, wherein the storage index record of one slice data includes a correspondence relationship between the slice type data and the first type of storage location corresponding to the slice data, and the data to be stored and the data to be stored are sliced Processing the corresponding relationship of each slice data;

The generated storage index is sent to the management node.

Corresponding to the system embodiment corresponding to FIG. 2 above, the embodiment of the present application further provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes, a slicing server, and each storage node. Connected through the SAS network, the target storage node is: a storage node that receives the slice data in the storage node included in the storage system;

As shown in FIG. 4, the method includes the following steps:

S401: Receive slice data sent by the slice server, where the slice data is: data obtained by slice processing the slice data to be stored by the slice server;

S402. Determine a first type of storage node for storing slice data.

S403. Acquire a first type of storage location for storing slice data in the determined first type of storage node.

S404. Write the obtained slice data to the acquired first type storage location through the SAS network.

Optionally, in an implementation manner, the step of acquiring the first type of storage location (S403) for storing the slice data in the determined first type of storage node may include the following steps:

Sending a first storage request to the determined first type of storage node;

Optionally, in an implementation manner, the step of determining a first type of storage node (S402) for storing slice data may include the following steps:

a storage node that has been allocated by the target storage node;

The survival status of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, in an implementation manner, the storage system further includes a management node;

After the step of writing the obtained slice data to the acquired first type storage location (S404) through the SAS network, the method may further include the following steps:

The generated storage index is sent to the management node.

Corresponding to the method embodiment corresponding to FIG. 3 above, the embodiment of the present application further provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node passes through a SAS network. Connection, the target storage node is a storage node having data to be stored;

As shown in Figure 5, the device includes:

The first obtaining module 510 is configured to acquire data to be stored.

The slice processing module 520 is configured to perform slice processing on the data to be stored to obtain slice data.

a first determining module 530, configured to determine a first type of storage node for storing the obtained slice data;

a second obtaining module 540, configured to acquire a first type of storage location for storing slice data in the determined first type of storage node;

The first writing module 550 is configured to write the obtained slice data into the acquired first type of storage location through the SAS network.

Optionally, in an implementation manner, the second obtaining module 540 may include:

a first receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined location of the first type of storage node for storing slice data .

Optionally, in an implementation manner, the first determining module 530 is specifically configured to:

a storage node that has been allocated by the target storage node;

The survival status of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

Optionally, in an implementation manner, the storage system may further include a management node; the device may further include:

a first generation module, configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes a correspondence between the slice type data and the first type of storage location corresponding to the slice data, and a data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored;

Corresponding to the method embodiment corresponding to FIG. 4 above, the embodiment of the present application further provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes, a slicing server, and each storage node. Connected through the SAS network, the target storage node is: a storage node that receives the slice data in the storage node included in the storage system;

As shown in Figure 6, the device includes:

The receiving module 610 is configured to receive the slice data sent by the slice server, where the slice data is: the data obtained by the slice server to slice the data to be stored;

a second determining module 620, configured to determine a first type of storage node for storing slice data;

a third obtaining module 630, configured to acquire a first type of storage location for storing slice data in the determined first type of storage node;

The second writing module 640 is configured to write the obtained slice data to the acquired first type of storage location through the SAS network.

Optionally, in an implementation manner, the third obtaining module 630 may include:

a second receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined location of the first type of storage node for storing slice data .

Optionally, in an implementation manner, the second determining module 620 is specifically configured to:

a storage node that has been allocated by the target storage node;

The survival status of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.

a second generation module, configured to generate a storage index of each slice data obtained, where the storage index record of one slice data includes a correspondence between the slice data and a first type of storage location corresponding to the slice data, and a data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored;

The embodiment of the present application further provides an electronic device, as shown in FIG. 7, including a processor 710 and a memory 720;

a memory 720, configured to store a computer program;

The processor 710 is configured to perform the following steps when executing the program stored on the memory 720:

Obtain data to be stored;

Slice the processed data to obtain the sliced data;

Determining a first type of storage node for storing the obtained slice data;

Obtaining a first type of storage location for storing slice data in the determined first type of storage node;

An electronic device provided by an embodiment of the present application may further perform a data storage method according to any one of the foregoing embodiments. For details, refer to the embodiments corresponding to FIG. 3 and FIG. 3, and details are not described herein again.

In still another embodiment provided by the present application, there is provided a computer readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the above-described FIG. 3 and FIG. A data storage method according to any of the corresponding embodiments.

The embodiment of the present application further provides an electronic device, which can implement the following steps:

Receiving slice data sent by the slice server, where the slice data is: data obtained by slicing the data to be stored by the slice server;

Determining a first type of storage node for storing slice data;

An electronic device provided by an embodiment of the present application may further perform a data storage method according to any one of the foregoing embodiments. For details, refer to the embodiments corresponding to FIG. 4 and FIG. 4, and details are not described herein again.

In still another embodiment provided by the present application, there is provided a computer readable storage medium having instructions stored therein, when executed on a computer, causing the computer to perform the above-described FIG. 4 and FIG. 4 A data storage method according to any of the corresponding embodiments.

The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or may be a digital signal processing (DSP), dedicated integration. Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component.

It should be noted that, in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply such entities or operations. There is any such actual relationship or order between them. Furthermore, the term "comprises" or "comprises" or "comprises" or any other variations thereof is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that comprises a plurality of elements includes not only those elements but also Other elements, or elements that are inherent to such a process, method, item, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional elements in the process, method, item, or device that comprises the element.

The various embodiments in the present specification are described in a related manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the method embodiment and the device embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the system embodiment.

The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalents, improvements, etc., which are made within the spirit and principles of the present invention, should be included in the present invention. Within the scope of protection.

Claims

A data storage system, characterized in that the storage system comprises at least two storage nodes, and each storage node is connected by a serial connection small computer system interface SAS network;

a target storage node, configured to acquire data to be stored, perform slice processing on the data to be stored to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type storage node And storing the first type of storage location of the slice data, and writing the obtained slice data to the acquired first type of storage location through the SAS network.
The storage system according to claim 1, wherein the target storage node acquires the first type of storage location for storing slice data in the determined first type of storage node, including:

The target storage node is configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: The location of the determined first type of storage node for storing slice data.
The storage system according to claim 1, wherein the target storage node is configured to determine a first type of storage node for storing the obtained slice data, including:

The target storage node is configured to determine a first type of storage node for storing the obtained slice data according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The storage system according to claim 1, wherein the storage system further comprises a management node;

The target storage node is configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence relationship between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored; sending the generated storage index to the management node;

The management node is configured to receive and store a storage index sent by the target storage node.
A storage system according to claim 4, wherein

The management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type storage node; according to the locally stored storage index Determining the slice data corresponding to the data to be read and the second type of storage location of each slice data, and sending the second type of storage location to the second type of storage node;

The second type of storage node is configured to receive the second type of storage location sent by the management node, read slice data from the second type of storage location, and perform restoration processing on the read slice data. Obtaining the data to be read, and sending the data to be read to the target client.
A storage system according to claim 4, wherein

The management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a data used to reconstruct the fault location from the storage nodes included in the storage system. Storage node, as a third type of storage node;

The third type of storage node is configured to reconstruct data stored in the fault location, obtain reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated data. storage location.
A storage system according to claim 6 wherein:

The management node is specifically configured to acquire a fault location in the fault storage node, and determine, according to a local storage index, data corresponding to the fault location, and slice data corresponding to the data stored in the fault location, and Obtaining a storage location corresponding to each determined slice data, and transmitting the obtained storage location to a third type of storage node, wherein the third type of storage node is: according to the data amount of the data stored in the fault location, from the a storage node selected by the storage system for reconstructing data stored in the fault location;

The third type storage node is configured to receive a storage location sent by the management node, read corresponding slice data from the received storage location, and perform reconfiguration processing on the read slice data to obtain a weight Constructing data, allocating a storage location for the reconstructed data, and writing the reconstructed data to the allocated storage location.
A data storage system, characterized in that: the storage system comprises: a slicing server, at least two storage nodes, and each storage node is connected by a serial connection small computer system interface SAS network;

The slice server is configured to acquire data to be stored, perform slice processing on the data to be stored, and obtain slice data; determine, from the at least two storage nodes, a target storage node that receives the slice data, and obtain the obtained Sending slice data to the target storage node;

The target storage node is configured to receive the slice data sent by the slice server, determine a first type of storage node used to store the slice data, and acquire the determined first type storage node for storing The first type of storage location of the slice data is written to the acquired first type of storage location by the SAS network.
The storage system according to claim 8, wherein the target storage node acquires a first type of storage location for storing slice data in the determined first type of storage node, including:

The target storage node is configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: The location of the determined first type of storage node for storing slice data.
The storage system according to claim 8, wherein the target storage node is configured to determine a first type of storage node for storing slice data, including:

The target storage node is configured to determine a first type of storage node for storing slice data according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The storage system according to claim 8, wherein the storage system further comprises a first management node;

The target storage node is configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes: a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored; sending the generated storage index to the first management node;

The first management node is configured to receive and store a storage index sent by the target storage node.
A storage system according to claim 11 wherein:

The first management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type storage node; Storing an index, determining slice data corresponding to the data to be read and a second type of storage location of each slice data, and sending the second type of storage location to the second type of storage node;

The second type of storage node is configured to receive the second type of storage location sent by the first management node, read slice data from the second type of storage location, and perform the read slice data. The restoration process is performed to obtain the data to be read, and the data to be read is sent to the target client.
A storage system according to claim 11 wherein:

The first management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage node included in the storage system for reconstructing the fault location a storage node that stores data as a third type of storage node;

The third type of storage node is configured to reconstruct data stored in the fault location, obtain reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated data. storage location.
A storage system according to claim 13 wherein:

The first management node is specifically configured to acquire a fault location in the fault storage node, and determine, according to a local storage index, data corresponding to the fault location, and slice data corresponding to the data stored in the fault location. And obtaining the determined storage location corresponding to each slice data, and sending the obtained storage location to the third type storage node, wherein the third type storage node is: according to the data amount of the data stored in the fault location, a storage node selected by the storage system for reconstructing data stored in the fault location;

The storage node of the third type is configured to receive a storage location sent by the first management node, read corresponding slice data from the received storage location, and perform reconstruction processing on the read slice data. Reconstructed data is obtained, a storage location is allocated for the reconstructed data, and the reconstructed data is written to the allocated storage location.
The storage system according to claim 8, wherein the storage system further comprises a second management node;

Determining, by the slice server, a target storage node that receives the slice data from the at least two storage nodes, including:

The slice server is configured to send a second storage request to the second management node;

The second management node is configured to receive the second storage request sent by the slice server, determine a target storage node from the storage nodes included in the storage system, and send the identifier of the determined target storage node To the slicing server;

The slice server is configured to receive the identifier sent by the second management node, and determine the target storage node corresponding to the identifier.
The storage system according to claim 15, wherein the second management node determines the target storage node from the storage nodes included in the storage system, including:

The second management node is configured to determine a target storage node from the storage nodes included in the storage system according to at least one of the following information:

a storage node that has been allocated by the second management node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
A data storage method, which is applied to a target storage node in a storage system, the storage system includes at least two storage nodes, and each storage node is connected by a serial connection small computer system interface SAS network;

The method includes:

Obtain data to be stored;

And performing slice processing on the data to be stored to obtain slice data;

Determining a first type of storage node for storing the obtained slice data;

Obtaining, in the determined first type of storage node, a first type of storage location for storing the slice data;

The obtained slice data is written into the acquired first type of storage location through the SAS network.
The method according to claim 17, wherein the step of acquiring the first type of storage location for storing slice data in the determined first type of storage node comprises:

Sending a first storage request to the determined first type of storage node;

Receiving the determined first type of storage location sent by the first type of storage node, where the first type of storage location is: a location allocated by the first type of storage node for storing slice data.
The method according to claim 17, wherein said step of determining a first type of storage node for storing the obtained slice data comprises:

A first type of storage node for storing the obtained slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The method of claim 17, wherein the storage system further comprises a management node;

After the step of writing the obtained slice data to the acquired first type storage location through the SAS network, the method further includes:

Generating a storage index of each of the obtained slice data, wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the to-be-stored data Corresponding relationship between each slice data obtained by slicing the data;

The generated storage index is sent to the management node.
A data storage method is characterized in that it is applied to a target storage node in a storage system, the storage system includes: at least two storage nodes, a slicing server, and each storage node is connected to a small computer system interface SAS network through a serial connection connection;

The method includes:

Receiving, by the slice server, slice data sent by the slice server, where the slice server performs slice processing on the data to be stored;

Determining a first type of storage node for storing the slice data;

Obtaining, in the determined first type of storage node, a first type of storage location for storing the slice data;

The slice data is written to the acquired first type of storage location through the SAS network.
The method according to claim 21, wherein the step of acquiring the first type of storage location for storing the slice data in the determined first type of storage node comprises:

Sending a first storage request to the determined first type of storage node;

Receiving the determined first type of storage location sent by the first type of storage node, where the first type of storage location is: a location allocated by the first type of storage node for storing slice data.
The method according to claim 21, wherein the step of determining a first type of storage node for storing slice data comprises:

A first type of storage node for storing slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The method of claim 21, wherein the storage system further comprises a management node;

After the step of writing the slice data to the acquired first type of storage location through the SAS network, the method further includes:

Generating a storage index of each of the received slice data, wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the Corresponding relationship between each slice data obtained by slice processing of the data to be stored;

The generated storage index is sent to the management node.
A data storage device, which is applied to a target storage node in a storage system, the storage system includes at least two storage nodes, and each storage node is connected by a serial connection small computer system interface SAS network;

The device includes:

a first acquiring module, configured to acquire data to be stored;

a slice processing module, configured to slice the data to be stored to obtain slice data;

a first determining module, configured to determine a first type of storage node for storing the obtained slice data;

a second acquiring module, configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data;

And a first writing module, configured to write the obtained slice data to the acquired first type of storage location through the SAS network.
The device according to claim 25, wherein the second obtaining module comprises:

a first sending submodule, configured to send a first storage request to the determined first type of storage node;

a first receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.
The device according to claim 25, wherein the first determining module is specifically configured to:

A first type of storage node for storing the obtained slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The apparatus according to claim 25, wherein said storage system further comprises a management node;

The device also includes:

a first generation module, configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, Corresponding relationship between the data to be stored and each slice data obtained by slicing the data to be stored;

The first sending module is configured to send the generated storage index to the management node.
A data storage device is characterized in that it is applied to a target storage node in a storage system, the storage system includes: at least two storage nodes, a slicing server, and each storage node is connected to a small computer system interface SAS network through a serial connection connection;

The device includes:

a receiving module, configured to receive the slice data sent by the slice server, where the slice data is: the data obtained by the slice server to slice the processed data;

a second determining module, configured to determine a first type of storage node for storing the slice data;

a third acquiring module, configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data;

And a second writing module, configured to write the slice data to the acquired first type of storage location through the SAS network.
The device according to claim 29, wherein the third obtaining module comprises:

a second sending submodule, configured to send a first storage request to the determined first type of storage node;

a second receiving submodule, configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.
The device according to claim 29, wherein the second determining module is specifically configured to:

A first type of storage node for storing slice data is determined according to at least one of the following information:

a storage node that has been allocated by the target storage node;

a survival state of each storage node in the storage system;

The remaining storage space of each storage node in the storage system;

The load pressure of each storage node in the storage system.
The apparatus according to claim 29, wherein said storage system further comprises a management node;

The device also includes:

a second generation module, configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes: a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored;

And a second sending module, configured to send the generated storage index to the management node.
An electronic device, comprising: a processor and a memory;

a memory for storing a computer program;

The processor, when executed to execute a program stored on the memory, implements the method steps of any of claims 17-20.
A computer readable storage medium, wherein the computer readable storage medium stores a computer program, the computer program being executed by a processor to implement the method steps of any one of claims 17-20.
An electronic device, comprising: a processor and a memory;

a memory for storing a computer program;

The processor, when executed to execute a program stored on the memory, implements the method steps of any of claims 21-24.
A computer readable storage medium, wherein the computer readable storage medium stores a computer program, the computer program being executed by a processor to implement the method steps of any one of claims 21-24.