WO2019170004A1 - Système, procédé et dispositif de stockage de données - Google Patents

Système, procédé et dispositif de stockage de données Download PDF

Info

Publication number
WO2019170004A1
WO2019170004A1 PCT/CN2019/075696 CN2019075696W WO2019170004A1 WO 2019170004 A1 WO2019170004 A1 WO 2019170004A1 CN 2019075696 W CN2019075696 W CN 2019075696W WO 2019170004 A1 WO2019170004 A1 WO 2019170004A1
Authority
WO
WIPO (PCT)
Prior art keywords
storage
data
type
node
storage node
Prior art date
Application number
PCT/CN2019/075696
Other languages
English (en)
Chinese (zh)
Inventor
汪渭春
王伟
林鹏
Original Assignee
杭州海康威视系统技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视系统技术有限公司 filed Critical 杭州海康威视系统技术有限公司
Publication of WO2019170004A1 publication Critical patent/WO2019170004A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present application relates to the field of data storage technologies, and in particular, to a data storage system, method, and apparatus.
  • cloud storage systems based on cloud storage technologies have emerged. Compared with traditional storage systems, cloud storage systems are scalable and have improved storage. effectiveness.
  • Each storage node included in the cloud storage system has two types of network ports: a service network port and a data network port.
  • the service network port is used to receive data that needs to be stored. For example, the monitoring data of each monitoring device in the monitoring network.
  • the data network port is used for data transmission between devices in the storage system.
  • both the service network port and the data network port are connected by electrical ports, wherein the electrical port can be an ordinary network cable interface or a coaxial cable interface, and the 100 Mbps Ethernet electrical port and the Gigabit Ethernet can be selected according to requirements. Electric port and so on.
  • a storage node S in the cloud storage system After a storage node S in the cloud storage system receives the data to be stored through the service network port, considering a load balancing factor, etc., a part of the data can be stored locally, and then the other data is sent to other storage nodes through the data network port. storage.
  • the storage node S When the storage node S stores its local data to other storage nodes, the local data of the storage node S needs to be transmitted to other storage nodes through the network, and then the other storage nodes write the received data to other storage nodes. Local storage. Although the storage node S can store its local data to other storage nodes in the above manner, the storage node S needs to transmit its local data to other storage nodes through the data network port, that is, the electrical port. There is a data delay in this process, resulting in inefficient data storage.
  • the present application provides a data storage system, method and apparatus to solve the problem of data transmission delay, resulting in low data storage efficiency.
  • the specific technical solutions are as follows:
  • the embodiment of the present application provides a data storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;
  • a target storage node configured to acquire data to be stored, perform slice processing on the data to be stored to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type storage node And storing the first type of storage location of the slice data, and writing the obtained slice data to the acquired first type of storage location through the SAS network.
  • the target storage node acquires the first type of storage location for storing the slice data in the determined first type of storage node, including:
  • the target storage node is configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: The location of the determined first type of storage node for storing slice data.
  • the target storage node is configured to determine a first type of storage node for storing the obtained slice data, including:
  • the target storage node is configured to determine a first type of storage node for storing the obtained slice data according to at least one of the following information:
  • the storage system further includes a management node
  • the target storage node is configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence relationship between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored; sending the generated storage index to the management node;
  • the management node is configured to receive and store a storage index sent by the target storage node.
  • the management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type of storage node; a stored storage index, determining slice data corresponding to the data to be read and a second type of storage location of each slice data, and sending the second type of storage location to the second type storage node;
  • the second type of storage node is configured to receive the second type of storage location sent by the management node, read slice data from the second type of storage location, and perform restoration processing on the read slice data. Obtaining the data to be read, and sending the data to be read to the target client.
  • the management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage node included in the storage system, to reconstruct the fault.
  • a storage node storing data in the location as a storage node of the third type;
  • the third type of storage node is configured to reconstruct data stored in the fault location, obtain reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated data. storage location.
  • the management node is specifically configured to acquire a fault location in the fault storage node, and determine, according to a local storage index, data corresponding to the fault location, and data corresponding to the fault location. Slicing data, and obtaining a storage location corresponding to each determined slice data, and transmitting the obtained storage location to a third type of storage node, wherein the third type of storage node is: a data amount of data stored according to the fault location And a storage node selected from the storage node included in the storage system for reconstructing data stored in the fault location;
  • the third type storage node is configured to receive a storage location sent by the management node, read corresponding slice data from the received storage location, and perform reconfiguration processing on the read slice data to obtain a weight Constructing data, allocating a storage location for the reconstructed data, and writing the reconstructed data to the allocated storage location.
  • the embodiment of the present application provides a data storage system, where the storage system includes: a slicing server, at least two storage nodes, and each storage node is connected through a SAS network;
  • the slice server is configured to acquire data to be stored, perform slice processing on the data to be stored, and obtain slice data; determine, from the at least two storage nodes, a target storage node that receives the slice data, and obtain the obtained Sending slice data to the target storage node;
  • the target storage node is configured to receive the slice data sent by the slice server, determine a first type of storage node used to store the slice data, and acquire the determined first type storage node for storing The first type of storage location of the slice data is written to the acquired first type of storage location by the SAS network.
  • the target storage node acquires a first type of storage location for storing slice data in the determined first type of storage node, including:
  • the target storage node is configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: The location of the determined first type of storage node for storing slice data.
  • the target storage node is configured to determine a first type of storage node for storing slice data, including:
  • the target storage node is configured to determine a first type of storage node for storing slice data according to at least one of the following information:
  • the storage system further includes a first management node
  • the target storage node is configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the data to be stored and each slice data obtained by the slice processing; sending the generated storage index to the first management node;
  • the first management node is configured to receive and store a storage index sent by the target storage node.
  • the first management node is further configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system as the second type of storage node; Determining, according to the locally stored storage index, the slice data corresponding to the data to be read and the second type of storage location of each slice data, and sending the second type of storage location to the second type storage node;
  • the second type of storage node is configured to receive the second type of storage location sent by the first management node, read slice data from the second type of storage location, and perform the read slice data.
  • the restoration process is performed to obtain the data to be read, and the data to be read is sent to the target client.
  • the first management node is configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage node included in the storage system for reconfiguring a storage node storing data stored in the fault location as a storage node of the third type;
  • the third type of storage node is configured to reconstruct data stored in the fault location, obtain reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated data. storage location.
  • the first management node is specifically configured to acquire a fault location in the fault storage node, determine, according to a local storage index, data corresponding to the fault location, and data stored in the fault location. Corresponding slice data, and obtaining a storage location corresponding to each determined slice data, and sending the obtained storage location to a third type storage node, wherein the third type storage node is: storing data according to the fault location a quantity of data, from a storage node included in the storage system, a storage node selected to reconstruct data stored in the fault location;
  • the storage node of the third type is configured to receive a storage location sent by the first management node, read corresponding slice data from the received storage location, and perform reconstruction processing on the read slice data. Reconstructed data is obtained, a storage location is allocated for the reconstructed data, and the reconstructed data is written to the allocated storage location.
  • the storage system further includes a second management node
  • Determining, by the slice server, a target storage node that receives the slice data from the at least two storage nodes including:
  • the slice server is configured to send a second storage request to the second management node
  • the second management node is configured to receive the second storage request sent by the slice server, determine a target storage node from the storage nodes included in the storage system, and send the identifier of the determined target storage node To the slicing server;
  • the slice server is configured to receive the identifier sent by the second management node, and determine the target storage node corresponding to the identifier.
  • the second management node determines the target storage node from the storage nodes included in the storage system, including:
  • the second management node is configured to determine a target storage node from the storage nodes included in the storage system according to at least one of the following information:
  • the embodiment of the present application provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;
  • the method includes:
  • the obtained slice data is written into the acquired first type of storage location through the SAS network.
  • the step of obtaining the first type of storage location for storing the slice data in the determined first type of storage node includes:
  • the step of determining a first type of storage node for storing the obtained slice data includes:
  • a first type of storage node for storing the obtained slice data is determined according to at least one of the following information:
  • the storage system further includes: a management node; after the step of writing the obtained slice data to the acquired first type storage location by using the SAS network, the method further includes:
  • the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the to-be-stored data Corresponding relationship between each slice data obtained by slicing the data;
  • the generated storage index is sent to the management node.
  • the embodiment of the present application provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes and a slicing server, and each storage node is connected through a SAS network. ;
  • the method includes:
  • the slice data is written to the acquired first type of storage location through the SAS network.
  • the step of acquiring the first type of storage location for storing the slice data in the determined first type of storage node includes:
  • the step of determining a first type of storage node for storing slice data comprising:
  • a first type of storage node for storing slice data is determined according to at least one of the following information:
  • the storage system further includes a management node
  • the method further includes:
  • the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, the data to be stored, and the Corresponding relationship between each slice data obtained by slice processing of the data to be stored;
  • the generated storage index is sent to the management node.
  • the embodiment of the present application provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node is connected through a SAS network;
  • the device includes:
  • a first acquiring module configured to acquire data to be stored
  • a slice processing module configured to slice the data to be stored to obtain slice data
  • a first determining module configured to determine a first type of storage node for storing the obtained slice data
  • a second acquiring module configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data
  • a first writing module configured to write the obtained slice data to the acquired first type of storage location through the SAS network.
  • the second obtaining module includes:
  • a first sending submodule configured to send a first storage request to the determined first type of storage node
  • a first receiving submodule configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.
  • the first determining module is specifically configured to:
  • a first type of storage node for storing the obtained slice data is determined according to at least one of the following information:
  • the storage system further includes a management node
  • the device also includes:
  • a first generation module configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, Corresponding relationship between the data to be stored and each slice data obtained by slicing the data to be stored;
  • the first sending module is configured to send the generated storage index to the management node.
  • the embodiment of the present application provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes and a slicing server, and each storage node is connected through a SAS network. ;
  • the device includes:
  • a receiving module configured to receive the slice data sent by the slice server, where the slice data is: the data obtained by the slice server to slice the processed data;
  • a second determining module configured to determine a first type of storage node for storing the slice data
  • a third acquiring module configured to acquire, in the determined first type of storage node, a first type of storage location for storing the slice data
  • a second writing module configured to write the slice data to the acquired first type of storage location through the SAS network.
  • the third obtaining module includes:
  • a second sending submodule configured to send a first storage request to the determined first type of storage node
  • a second receiving submodule configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined storage of slice data allocated by the first type of storage node s position.
  • the second determining module is specifically configured to:
  • a first type of storage node for storing slice data is determined according to at least one of the following information:
  • the storage system further includes a management node
  • the device also includes:
  • a second generation module configured to generate a storage index of each of the received slice data, where the storage index record of the slice data includes: a correspondence between the slice data and the first type of storage location corresponding to the slice data, Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored;
  • a second sending module configured to send the generated storage index to the management node.
  • an embodiment of the present application provides an electronic device, including a processor and a memory;
  • a memory for storing a computer program
  • the processor when executed to execute a program stored on the memory, implements one of the data storage method steps described in any of the above.
  • the embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a computer program, and when the computer program is executed by the processor, implements one of the foregoing data. Store method steps.
  • the storage system includes at least two storage nodes, and each storage node is connected through a SAS network, and the target storage node is configured to acquire data to be stored, and perform slice processing on the stored data to obtain slice data. Determining a first type of storage node for storing the obtained slice data, and acquiring a first type of storage location for storing slice data in the determined first type of storage node, and writing the obtained slice data through a SAS network The first type of storage location obtained. After the data to be stored is sliced, the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • FIG. 1 is a schematic diagram of a first structure of a data storage system according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of a second structure of a data storage system according to an embodiment of the present application.
  • FIG. 3 is a first flowchart of a data storage method according to an embodiment of the present application.
  • FIG. 4 is a second flowchart of a data storage method according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a first structure of a data storage device according to an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of a second structure of a data storage device according to an embodiment of the present disclosure.
  • FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
  • the embodiment of the present application provides a data storage system, method, and device, where the storage system includes at least two storage nodes, and each storage node Connected through a SAS network, the target storage node is a storage node that acquires data to be stored;
  • a target storage node configured to acquire data to be stored, perform slice processing on the stored data to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type of storage node for use in The first type of storage location of the sliced data is stored, and the obtained sliced data is written into the acquired first type of storage location through the SAS network.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • a data storage system provided by an embodiment of the present application is first introduced.
  • the storage system 100 includes: at least two storage nodes 110, each storage node 110 is connected through a SAS network, and the target storage node is configured to acquire data to be stored. Storage node.
  • a target storage node configured to acquire data to be stored, perform slice processing on the stored data to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type of storage node for use in The first type of storage location of the sliced data is stored, and the obtained sliced data is written into the acquired first type of storage location through the SAS network.
  • the storage system 100 and the target storage node are respectively described below.
  • the storage system 100 First, the storage system 100
  • the storage system 100 can include at least two storage nodes 110.
  • each storage node may include multiple hard disks, and the hard disks are used to store corresponding data.
  • the storage node can store the data on different hard disks separately, and can also store the data on the same hard disk.
  • Each storage node in the storage system 100 is interconnected.
  • the interconnection may be interconnected through a SAS network, and each storage node includes at least a SAS interface.
  • SAS Serial Attached SCSI
  • SCSI Small Computer System Interface
  • SCSI Small Computer System Interface
  • SCSI Small Computer System Interface
  • SCSI can be used for computers and hard disks. Interface standard between devices such as floppy drives, optical drives, printers, and scanners.
  • each storage node to connect through the SAS network can be connected by means of a SAS switch.
  • Each storage node in the storage system is connected to the SAS switch, so that the storage nodes can be connected to each other through the SAS switch.
  • Storage nodes can be connected to SAS switches through their respective SAS interfaces.
  • the manner in which the storage nodes are connected through the SAS network may include other methods in addition to the manner of the above-mentioned SAS switch, which is not limited herein.
  • each storage node can perform write operations and read operations on each hard disk that is seen. That is to say, each storage node can see all the hard disks of the storage system 100, and each storage node can directly write data to the hard disk that can be seen, or can read data from the hard disk that can be seen.
  • the storage system includes a storage node A, a storage node B, and a storage node C, and the storage node A, the storage node B, and the storage node C are connected by a SAS network, and the storage node A includes a hard disk A1, a hard disk A2, and a storage node.
  • B includes the hard disk B1 and the hard disk B2.
  • the storage node C includes the hard disk C1 and the hard disk C2. Then, for the storage node A, the hard disk B1, the hard disk B2, the hard disk C1, and the hard disk C2 can be seen, and the data can be directly written to the hard disk.
  • the respective stored data may be read from the hard disk B1, the hard disk B2, the hard disk C1, and the hard disk C2.
  • each storage node is connected by a SAS network, and each storage node can perform read and write operations, thereby solving the problem of mutual exclusion between storage nodes.
  • a target storage node configured to acquire data to be stored, perform slice processing on the stored data to obtain slice data, determine a first type of storage node for storing the obtained slice data, and obtain the determined first type of storage node for use in The first type of storage location of the sliced data is stored, and the obtained sliced data is written into the acquired first type of storage location through the SAS network.
  • the first step is to acquire data to be stored.
  • the storage system 100 may further include an access node.
  • the manner in which the target storage node obtains the data to be stored may be: the target storage node receives the data to be stored sent by the access node in the storage system.
  • the access node is connected to the client external to the storage system 100, and is configured to receive data sent by the client.
  • the access node may be connected to the monitoring device for receiving video data sent by the monitoring device.
  • the access node in the storage system is configured to receive data to be stored sent by the client, and after receiving the data to be stored, the access node may send the to-be-stored data to the storage node, where the storage node is the target.
  • the access node may select a target storage node for receiving data to be stored from at least two storage nodes included in the storage system by using a preset policy, and the policy may be customized.
  • the access node receives the data to be stored and the data to be stored sent by the target storage node and the access node can be implemented through the service network port.
  • the service network port is used to receive service data, and the service data may be from the outside of the storage system. Storing data.
  • the service data may be the video data of the monitoring device, and the monitoring device sends the video data to the access node in the storage system through the service network port.
  • the video data is the data to be stored, and the access node receives the received data.
  • the video data is sent to the target storage node through its own service network port, so that the target storage node can obtain the video data to be stored.
  • the target storage node After acquiring the data to be stored, the target storage node performs slice processing on the stored data to obtain slice data.
  • the target storage node may perform data processing on the data by using Erasure Coding (EC).
  • EC Erasure Coding
  • a preset number N pieces of data can be obtained, and according to the obtained N pieces of data, a preset number of M pieces can be obtained. From the sum of the data slice and the test slice M + N, is the number of slice data obtained.
  • the type of the erasure code in the erasure code technology may include multiple types, and N and M may be determined by the type of the erasure code. In different erasure code types, N and M may be different.
  • an RS (Reed-Solomon) code is an erasure code commonly used in a storage system.
  • the N code of the RS code is 5, and M is 3.
  • the data is sliced by using the RS code, and 5 pieces of data can be obtained. 3 test pieces, and the final slice data obtained is 8.
  • the obtained M check slices can function as a fault tolerance.
  • the N pieces of data can be restored by the reconstruction algorithm in the erasure code technique. Therefore, the erasure code technology can play a role of fault tolerance for the slice data, improving the reliability of the data storage.
  • the slice data obtained by the erasure code technique includes five data slices and three check slices, wherein the five data slices are: data slice A, data slice B, data slice C, and data slice D.
  • data piece E, three check pieces are: test piece A, test piece B, test piece C.
  • the manner in which the target storage node performs the slicing process is not limited to the erasure code technique in the above embodiment, and may be other types of slice processing, which is not limited herein.
  • a first type of storage node for storing the obtained slice data is determined, and a first type of storage location for storing slice data in the determined first type of storage node is obtained.
  • the target storage node may determine, according to the number of slice data and the data volume of each slice data, a first type of storage node for storing slice data, and obtain the determined first type of storage node for storing slice data.
  • the first type of storage location may be determined, according to the number of slice data and the data volume of each slice data, a first type of storage node for storing slice data, and obtain the determined first type of storage node for storing slice data. The first type of storage location.
  • the number of the first type of storage nodes may be determined.
  • the manner of determining the number of the first type of storage nodes may be in various manners.
  • the number of the first type of storage nodes is the same as the number of the slice data. For example, when the number of slice data obtained by the target storage node is eight, the number of the first type of storage nodes determined is also eight. .
  • one slice data corresponds to one first type storage node, that is, only one slice data is stored in each of the determined first type storage nodes, and is stored in each first type storage node.
  • the slice data is not the same.
  • the slice data obtained by the target storage node includes: slice data A, slice data B, and slice data C
  • the determined three first type storage nodes for storing the slice data are: the first type storage node 1, The first type storage node 2 and the first type storage node 3, and the target storage node assigns the slice data A to the first type storage node 1, the slice data B to the first type storage node 2, and distributes the slice data C
  • the first type of storage node 1 stores slice data A
  • the first type of storage node 2 stores slice data B
  • the first type of storage node 3 stores slice data C.
  • the number of storage nodes of the first type is smaller than the number of slice data.
  • at least one storage node of the first type of storage nodes that are determined needs to store more than one slice. data.
  • the first type of storage nodes that need to store more than one slice data may be determined according to the storage state of each storage node.
  • the storage state may include the remaining storage space of the storage node, load pressure, and the like.
  • the number of slice data is four
  • the number of the first type of storage nodes determined is three, namely: a first type storage node 1, a first type storage node 2, and a first type storage node 3, and Among the three first type storage nodes, compared with the first type storage node 2 and the first type storage node 3, the remaining storage space of the first type storage node 1 is larger, and the load pressure is smaller, therefore, the first can be determined.
  • One type of storage node 1 stores two of the slice data.
  • the determined number of storage nodes may be selected from the storage system 100 according to a preset policy, and the selected storage node is the first type of storage node.
  • the preset policy may be customized. Specifically, the manner of selecting a storage node according to the preset policy is described in detail in the following embodiments, and details are not described herein.
  • the amount of data of each slice data may also be considered, and the remaining storage space of the selected first type of storage node is at least greater than the data amount of the corresponding stored slice data.
  • the target storage node determines the first type of storage node
  • the first type of storage location for storing the slice data in the determined first type of storage node may be obtained.
  • the storage location may be a hard disk in the storage node, and different storage locations are different hard disks. When there are multiple partitions in each hard disk, the storage location may also be a partition specifically referring to the hard disks in the storage node.
  • the storage node includes the hard disk 1 and the hard disk 2, and the data is stored in the hard disk 1.
  • the storage location of the data may refer to the hard disk 1.
  • the storage location of the data refers to the partition 1 of the hard disk 1.
  • the storage node may include multiple hard disks, and the storage state of each hard disk is different. After determining the storage node of the first type, the storage state of each hard disk in the storage node of the first type may be selected to store the slice data.
  • the hard disk is the first type of storage location.
  • the storage state of the hard disk may be the remaining storage space, the load pressure, etc.
  • the storage state may be selected according to only one of the storage states, and may also be selected according to multiple storage states of the hard disk. It is not limited here.
  • the hard disk with the largest remaining storage space is selected from the first type of storage nodes as the hard disk storing the slice data only according to the remaining storage space.
  • the hard disk with the largest remaining storage space and the lowest load pressure can be selected from the first type of storage nodes as the hard disk storing the slice data.
  • the first type of storage location for storing the slice data in the first type of storage node may be determined by the first type of storage node itself, or may be specified by the target storage node, and then applied to the corresponding first type of storage node. This part is described in detail in the following embodiments and will not be described in detail herein.
  • the obtained slice data is written into the acquired first type storage location through the SAS network.
  • each storage node may be connected through a SAS network. Therefore, the target storage node can know the storage location in each of the other storage nodes in the storage system 100, the target storage node can read the corresponding data from the other storage nodes, and can also write the data to the storage location in the other storage nodes. .
  • the target storage node may directly write the slice data into the corresponding first type storage location.
  • the target storage node obtains the slice data A, and determines that the slice data A corresponds to the first type storage location A of the first type storage node A, and the target storage node can directly write the slice data A through the SAS network.
  • the first type of storage node A is stored in location A.
  • the storage system includes at least two storage nodes, and each storage node is connected through a SAS network, and the target storage node is configured to acquire data to be stored, and perform slice processing on the stored data to obtain slice data. Determining a first type of storage node for storing the obtained slice data, and acquiring a first type of storage location for storing slice data in the determined first type of storage node, and writing the obtained slice data through a SAS network The first type of storage location obtained. After the data to be stored is sliced, the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the first type of storage location may be determined by the first type of storage node itself.
  • the target storage node may send the first storage request to the determined first type of storage node.
  • the first type of storage node may allocate a corresponding storage location according to the first storage request, where the storage location is the first type of storage location, and the first type of storage node stores the allocated first category. The location is sent to the target storage node.
  • the first storage request sent by the target storage node may include any one or more of data type, data volume, data identifier, and the like of the slice data to be stored.
  • the data type may include video data, image data, and the like.
  • the amount of data reflects the size of the storage space occupied by the slice data, and the storage node allocates the corresponding storage space according to the data amount of the slice data.
  • the data identification can be used to verify the slice data to ensure that the slice data is written to the correct storage location.
  • the target storage node may be configured to determine, according to at least one of the following information, a first type of storage node used to store the obtained slice data: a storage node that has been allocated by the target storage node, and a storage system The survival state of each storage node, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.
  • the target storage node determines the first type of storage node based on the storage node that has been allocated by the target storage node.
  • the target storage node can record the identity of each of the determined first type of storage nodes.
  • the first implementation manner is a centralized policy, that is, when the target storage node determines the first type of storage node, the storage node that is the same as the storage node of the first type determined last time may be selected.
  • the storage system 100 includes: a storage node 1, a storage node 2, a storage node 3, a storage node 4, and a storage node 5.
  • the last determined first type of storage node recorded by the target storage node is the storage node 1 and the storage. Node 2, then, when the target storage node determines the first type of storage node this time, the storage node 1 and the storage node 2 are still selected as the first type of storage node.
  • the first type of storage node determined last time fails, the storage space is full, etc.
  • the first type of storage node determined last time can no longer be used as the first type of storage node, but can pass the target.
  • the storage node determines the other basis of the first type of storage node to re-determine the first type of storage node.
  • the second implementation manner is a decentralized strategy, that is, when the target storage node determines the first type of storage node, the storage node different from the previously determined first type storage node may be selected. It can be divided into two cases: in the first case, a storage node different from the first type of storage node determined last time can be selected; in the second case, it can be selected differently from the first type of storage node determined many times before. Storage node.
  • the target storage node when determining the first type of storage node, may select a storage node different from the first type of storage node determined last time. That is to say, the determined first type of storage node may be different from the first type of storage node determined last time.
  • the storage system 100 includes: a storage node 1, a storage node 2, a storage node 3, a storage node 4, and a storage node 5.
  • the last determined storage node recorded by the target storage node is a storage node 1 and a storage node 2, Then, when the target storage node determines the first type of storage node, the storage node 3, the storage node 4, and the storage node 5 may be selected as the storage node of the first type.
  • the target storage node may select a storage node different from the first type of storage node determined a plurality of times before. Among them, the number of times before can be customized.
  • the target storage node may employ a rotation policy when determining the first type of storage nodes.
  • An order sorting table for each storage node included in the storage system 100 may be preset, and the sorting table records the sorting of the identifiers corresponding to the storage nodes included in the storage system 100, and the sorting rules may be customized.
  • the identifier may be allocated according to the identifier sorting table, and the identifier of the first type of storage node determined each time is: the first type of storage node determined last time Identifies the next identity in the identity sort table.
  • the first type of storage node may be re-determined.
  • the target storage node determines the first type of storage node according to the survival state of each storage node in the storage system.
  • the survival state of the storage node is: a fault state or an intact state of the storage node.
  • the storage node can periodically report the survival status of the storage node to the target storage node.
  • the storage state of each storage node can also be known in real time. In this way, when the target storage node determines the first type of storage node, the failed storage node can be eliminated in time, and only the target storage node is determined from the intact storage node. In this way, the efficiency of slicing data storage can be improved.
  • the target storage node determines the first type of storage node based on: the remaining storage space of each storage node in the storage system.
  • the remaining storage space of each storage node may be sorted, and for a storage node with a large remaining storage space, it may be preferentially determined as the first type of storage node.
  • the target storage node may update the order of the remaining storage spaces of each storage node after each determination of the storage node of the first type, and the obtained new remaining storage space sorting may be used as the next determination of the first type of storage.
  • the basis of the node In this way, the remaining storage space of each storage node in the storage system 100 can be equalized.
  • the target storage node determines the basis of the first type of storage node as: the load pressure of each storage node in the storage system.
  • the load pressure of each storage node included in the storage system 100 may be sorted, and for a storage node with a small load pressure, it may be preferentially determined as the first type of storage node. In this way, the target storage node may update the load pressure order of each storage node after each determination of the storage node of the first type, and the obtained new load pressure ordering may be used as the basis for determining the first type of storage node next time. . In this way, the load pressure of each storage node in the storage system 100 can be equalized.
  • the basis of the first type of storage node may be separately determined as the target storage node, and any two, three or four combinations may be used as the basis for the target storage node to determine the first type of storage node. There is no limit here.
  • the storage system may further include a management node, where the management node in the storage system may be one or multiple.
  • the collection of the plurality of management nodes is a management cluster. Therefore, it can be considered that the storage system includes a management cluster, and the management cluster includes a plurality of management nodes.
  • the target storage node and the management node in this embodiment are respectively introduced below.
  • the correspondence between the obtained slice data and the data to be stored may be recorded. Relationship, the correspondence is a correspondence relationship of storage index records of one slice data.
  • the slice data obtained by the target storage node after the data A is sliced includes: slice data 1, slice data 2, and slice data 3.
  • the correspondence relationship of the target storage node records includes: data A corresponds to slice data 1, and data A and The slice data 2 corresponds, and the data A corresponds to the slice data 3.
  • the target storage node After the target storage node writes the received slice data to the corresponding first type storage location through the SAS network, the corresponding relationship between each slice data and the first type of storage location stored by the slice data may be recorded.
  • the relationship is also another correspondence of the storage index records of one slice of data.
  • the slice data obtained by the target storage node includes: slice data A, slice data B, and slice data C, wherein the storage index 1 of the slice data A is the slice data A corresponding to the storage location A, and the storage index 2 of the slice data B
  • the storage index 3 of the slice data C corresponds to the storage location C
  • the storage index generated by the target storage node includes the storage index 1, the storage index 2, and the storage index 3.
  • the storage index generated by the target storage node includes: a storage index corresponding to each slice data obtained by the target storage node.
  • a management node may be selected from the management cluster according to a preset policy, and used to receive and store the storage index sent by the target storage node.
  • the preset policy may be: determining, according to at least one of the following information, a management node for receiving and storing a storage index sent by the target storage node: the allocated management node, and the survival status of each management node in the management cluster. Manage the load pressure of each management node in the cluster.
  • the policy for determining the management node is similar to the policy for determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the first embodiment, the second base, and the fourth base in the foregoing embodiments, and details are not described herein again.
  • the management node can more accurately and quickly query the storage location of storing the slice data in the storage system through the stored storage index, thereby improving data processing efficiency.
  • the management node and the second type storage node in the embodiment are respectively introduced below.
  • the management node can also be used to receive a read request from the target client for the data to be read.
  • the target client may be a device external to the storage system that sends a read request, for example, a client device such as a computer, a mobile phone, or a tablet.
  • the data to be read may be data stored in the storage system.
  • the management node selects the second type of storage node according to a preset policy.
  • the second type storage node may be selected from the storage nodes included in the storage system according to at least one of the following information: The storage node allocated by the node, the surviving state of each storage node in the storage system, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.
  • the policy for the management node to select the second type of storage node is similar to the strategy for determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the foregoing embodiment, the second, the third, and the fourth, and will not be described herein. .
  • the management node may determine the slice data corresponding to the data to be read and the second type of storage location of each slice data according to the locally stored storage index.
  • the management node may query the obtained slice data of the data to be read according to the locally stored storage index, including: slice data A, slice data B, and slice data C, and may query and obtain a storage location corresponding to each slice data: slice data.
  • A is stored in the storage location A
  • the slice data B is stored in the storage location B
  • the slice data C is stored in the storage location C.
  • the storage location A, the storage location B, and the storage location C are the second type of storage location.
  • the storage nodes included in the storage system 100 are connected by a SAS network, and the second type of storage nodes can read data from other storage nodes in the storage system.
  • the second type of storage node after receiving the second type of storage location sent by the management node, the second type of storage node can read the stored slice data from the second type of storage location.
  • the second type of storage location sent by the management node includes a storage location A, a storage location B, and a storage location C, wherein the storage location A stores slice data A, the storage location B stores slice data B, and the storage location C stores slice Data C, then, for the second type of storage node, the slice data A is read from the storage location A, the slice data B is read from the storage location B, and the slice data C is read from the storage location C.
  • the second type of storage node can perform data restoration processing on the read slice data to obtain data to be read required by the target client.
  • the second type of storage node performing data restoration processing may be an erasure code technique, and the number of slice data read by the second type storage node may be N+M, where N is data.
  • the number of slices, M is the number of slices.
  • the number of slice data read by the second type of storage node may be less than N+M and greater than or equal to N, because the fault tolerance characteristic of the erasure code technology itself can restore the slice data to N+M, and then perform data restoration. Processing to obtain data to be read.
  • the management node and the third type storage node in the embodiment are respectively introduced below.
  • the management node can perform overall management on the storage nodes included in the storage system 100, and can obtain state information of each storage node in real time.
  • the status information of the storage node may include at least one of failure information of the storage node, remaining storage space information, and hard disk status information included in the storage node.
  • the first type of obtaining may be that each storage node actively sends its own state information to the management node, where the storage node may send according to a preset frequency, and the preset frequency may be customized. For example, in order to ensure that the state information of each storage node acquired by the management node is real-time, the frequency of the transmission may be set higher; in addition, when the storage node is in a sudden state such as a failure or a storage is full, the storage node may immediately Send the information of the emergency to the management node.
  • the management node actively obtains the state information of each storage node from each storage node in the storage system.
  • the manner in which the management node obtains the state information of the storage node is not limited to the above two types, and may also include other acquisition modes, which are not limited herein.
  • the management node may obtain the fault information of the faulty storage node.
  • the storage node fault may be a storage location fault in the storage node, causing the data stored in the storage location to be damaged. Can't read.
  • the fault information may include the data amount of the data stored in the fault location, and the fault information may further include at least one of the following information: a data type of the data stored in the fault location, data identifier information, and the like.
  • the management node acquires the fault location in the fault storage node, determines the data corresponding to the fault location according to the local storage index, and the slice data corresponding to the data stored in the fault location, and obtains the determined slice data. Corresponding storage location, the obtained storage location is sent to the third type of storage node.
  • the fault location acquired by the management node may be: after the faulty storage node that identifies the fault identifies the fault location, the fault location is reported to the management node.
  • the management node may determine, according to the correspondence between the slice data in the locally stored storage index and the first type of storage location corresponding to the slice data, the data corresponding to the fault location may be determined; Corresponding relationship between the stored data and each slice data obtained by the slice processing of the data to be stored, thereby determining the slice data corresponding to the data stored in the fault location; and finally corresponding to the slice data according to the slice data in the storage index.
  • the storage locations corresponding to the determined slice data can be obtained.
  • the determined storage location is sent to a third type of storage node.
  • the management node may select the third type of storage node from the storage nodes included in the storage system according to the data amount of the data stored in the fault location. In order to facilitate the reconstruction process and to store the reconstructed data, the available storage space of the third type of storage node is larger than the data amount of the data stored by the faulty location.
  • the third type of storage node may be selected according to a preset policy, and the third type of storage node may be selected from the storage nodes included in the storage system according to at least one of the following information: The storage node allocated by the node, the surviving state of each storage node in the storage system, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.
  • the policy of selecting the storage node of the third type is similar to the policy of determining the storage node of the first type in the foregoing embodiment, and may refer to the basis of the foregoing embodiment according to the first, the second, the third and the fourth, and will not be described herein. .
  • the selected third type of storage node may perform reconstruction processing on the data, and the reconstruction processing may be an erasure coding technique.
  • the third type of storage node may perform reconstruction processing on the data stored in the fault location, and an implementation manner of the reconstruction processing may be stored in the fault location by using erasure code technology.
  • the data is reconstructed, and then the reconstructed data is obtained, and the obtained reconstructed data is the original data stored in the fault location.
  • the data stored in the fault location is data A
  • the reconstructed data obtained by the reconstruction process by the erasure code technique is data A.
  • the third type storage node receives the storage location sent by the management node, reads the corresponding slice data from the received storage location, and reconstructs the read slice data to obtain the reconstructed data. .
  • the corresponding storage location may be allocated for the reconstructed data.
  • the first allocation mode the storage location allocated by the storage node of the third type is a storage location in a storage node, so that the reconstructed data can be directly stored to the allocated storage location.
  • the remaining storage space of the allocated storage location is larger than the amount of data of the reconstructed data.
  • the third type of storage node may allocate a storage location for the data A, and the remaining space of the storage location is greater than 10M. In this way, data A can be written directly to the storage location.
  • the second allocation method can allocate multiple storage locations for the reconstructed data and perform slice storage on the reconstructed data.
  • the reconstructed data may be first sliced to obtain slice data, and each slice data is separately written into the allocated storage location.
  • the third type of storage node may perform slice processing by means of erasure coding technology.
  • the storage location allocated for the reconstruction data may be the hard disk of the storage node of the third type of storage node or the hard disk of other storage nodes of the storage system 100. This is not limited here.
  • the storage system 100 includes: a slicing server 120, at least two storage nodes 110, and each storage node 110 is connected through a SAS network;
  • the slice server 120 is configured to acquire data to be stored, perform slice processing on the stored data, and obtain slice data; determine, from at least two storage nodes, a target storage node that receives the slice data, and send the obtained slice data to the target storage node;
  • a target storage node configured to receive slice data sent by the slice server 120, determine a first type of storage node for storing slice data, and obtain a first type of storage location for storing slice data in the determined first type of storage node. And the received slice data is written into the first type of storage location obtained by the SAS network.
  • the slice data obtained by the target storage node in the embodiment is sent by the slice server 120, and after the target storage node receives the slice data, the operation performed by the target storage node in the embodiment corresponding to FIG. 1 is performed.
  • the operation performed by the target storage node in the embodiment corresponding to FIG. 1 is performed.
  • the slice server 120 is configured to acquire data to be stored, perform slice processing on the stored data, and obtain slice data; determine a target storage node that receives the slice data from the at least two storage nodes, and send the obtained slice data to the target storage node.
  • the data to be stored is obtained, and the data to be stored is sliced to obtain slice data.
  • the target storage node is used to obtain the data to be stored, and the data to be sliced is processed to obtain the slice data, and details are not described herein.
  • the target storage node that receives the slice data is determined from the at least two storage nodes, and the obtained slice data is sent to the target storage node.
  • the storage system 100 further includes a second management node.
  • the slice server 120 may send a second storage request to the second management node, where the second storage request is a request by the slice server 120 to allocate the storage node to the second management node.
  • the second storage request may carry related information of the slice data, for example, the number of slice data obtained by the slice server 120, the size of the slice data, and the like.
  • the second management node receives the second storage request sent by the slicing server 120, and determines the target storage node from the storage nodes included in the storage system according to the second storage request.
  • the second management node may determine, from the storage nodes included in the storage system, a target storage node according to at least one of the following information: a storage node that has been allocated by the second management node, and each storage in the storage system. The surviving state of the node, the remaining storage space of each storage node in the storage system, and the load pressure of each storage node in the storage system.
  • the implementation manner is similar to the policy for determining the storage node of the first type in the implementation manner corresponding to the foregoing embodiment of FIG. 1, and may refer to the basis of the foregoing, the second, the third, and the fourth according to the foregoing embodiments, and details are not described herein again.
  • the second management node After determining the target storage node, the second management node sends the determined identifier of the target storage node to the slice server 120.
  • the identifier may be at least one of an IP (Internet Protocol) address, a MAC (Media Access Control) address, a code, and the like.
  • the slice server 120 After receiving the identifier sent by the second management node, the slice server 120 can determine the target storage node corresponding to the identifier, and send the obtained slice data to the target storage node.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the target storage node obtains the first type of storage location for storing the slice data in the determined first type of storage node, and may include:
  • a target storage node configured to send a first storage request to the determined first type of storage node, and receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined first The location allocated by the class storage node to store the slice data.
  • the target storage node is configured to determine a first type of storage node for storing slice data, and may include:
  • a target storage node configured to determine, according to at least one of the following information, a first type of storage node for storing slice data: a storage node that has been allocated by the target storage node, a survival state of each storage node in the storage system, and storage The remaining storage space of each storage node in the system, and the load pressure of each storage node in the storage system.
  • the storage system may further include a first management node, where the first management node and the second management node may be the same management node, and may be different management nodes, which are not limited herein.
  • a storage node for generating a obtained storage index of each slice data wherein the storage index record of the slice data includes: a correspondence between the slice data and a first type of storage location corresponding to the slice data, and data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored; and transmitting the generated storage index to the first management node.
  • the first management node is configured to receive and store a storage index sent by the target storage node.
  • the first management node may further be configured to receive a read request of the target client for the data to be read, and select a storage node from the storage nodes included in the storage system.
  • a second type of storage node according to the locally stored storage index, the slice data corresponding to the data to be read and the second type storage location of each slice data are determined, and the second type storage location is sent to the second type storage node.
  • the second type of storage node may be configured to receive the second type of storage location sent by the first management node, read the slice data from the second type of storage location, and perform the reduction processing on the read slice data to obtain a to-be-read Data and send the data to be read to the target client.
  • the first management node may be further configured to obtain a data amount of data stored in a fault location of the fault storage node, and select, according to the data volume, a storage location node included in the storage system for reconstructing the fault location.
  • the third type of storage node can also be used to reconstruct the data stored in the fault location, obtain the reconstructed data, allocate a storage location for the reconstructed data, and write the reconstructed data to the allocated storage location.
  • the first management node is specifically configured to acquire a fault location in the fault storage node, determine, according to the local storage index, data corresponding to the fault location, and store the fault location.
  • the third type of storage node may be configured to receive a storage location sent by the first management node, read corresponding slice data from the received storage location, and reconstruct the processed slice data to obtain a reconstruction. Data, allocate storage locations for reconstructed data, and write reconstructed data to the allocated storage locations.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the embodiment of the present application further provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node passes through a SAS network. connection;
  • the method includes the following steps:
  • the step of acquiring the first type of storage location (S304) for storing the slice data in the determined first type of storage node may include the following steps:
  • the first type of storage location is: the determined location of the first type of storage node for storing slice data.
  • the step of determining a first type of storage node (S303) for storing the obtained slice data may include the following steps:
  • a first type of storage node for storing the obtained slice data is determined according to at least one of the following information:
  • the storage system may further include a management node
  • the method may further include the following steps:
  • the storage index record of one slice data includes a correspondence relationship between the slice type data and the first type of storage location corresponding to the slice data, and the data to be stored and the data to be stored are sliced Processing the corresponding relationship of each slice data;
  • the generated storage index is sent to the management node.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the embodiment of the present application further provides a data storage method, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes, a slicing server, and each storage node.
  • the target storage node is: a storage node that receives the slice data in the storage node included in the storage system;
  • the method includes the following steps:
  • S401 Receive slice data sent by the slice server, where the slice data is: data obtained by slice processing the slice data to be stored by the slice server;
  • S402. Determine a first type of storage node for storing slice data.
  • the step of acquiring the first type of storage location (S403) for storing the slice data in the determined first type of storage node may include the following steps:
  • the first type of storage location is: the determined location of the first type of storage node for storing slice data.
  • the step of determining a first type of storage node (S402) for storing slice data may include the following steps:
  • a first type of storage node for storing slice data is determined according to at least one of the following information:
  • the storage system further includes a management node
  • the method may further include the following steps:
  • the storage index record of one slice data includes a correspondence relationship between the slice type data and the first type of storage location corresponding to the slice data, and the data to be stored and the data to be stored are sliced Processing the corresponding relationship of each slice data;
  • the generated storage index is sent to the management node.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the embodiment of the present application further provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes at least two storage nodes, and each storage node passes through a SAS network. Connection, the target storage node is a storage node having data to be stored;
  • the device includes:
  • the first obtaining module 510 is configured to acquire data to be stored.
  • the slice processing module 520 is configured to perform slice processing on the data to be stored to obtain slice data.
  • a first determining module 530 configured to determine a first type of storage node for storing the obtained slice data
  • a second obtaining module 540 configured to acquire a first type of storage location for storing slice data in the determined first type of storage node
  • the first writing module 550 is configured to write the obtained slice data into the acquired first type of storage location through the SAS network.
  • the second obtaining module 540 may include:
  • a first sending submodule configured to send a first storage request to the determined first type of storage node
  • a first receiving submodule configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined location of the first type of storage node for storing slice data .
  • the first determining module 530 is specifically configured to:
  • a first type of storage node for storing the obtained slice data is determined according to at least one of the following information:
  • the storage system may further include a management node; the device may further include:
  • a first generation module configured to generate a storage index of each of the obtained slice data, where the storage index record of the slice data includes a correspondence between the slice type data and the first type of storage location corresponding to the slice data, and a data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored;
  • the first sending module is configured to send the generated storage index to the management node.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the embodiment of the present application further provides a data storage device, which is applied to a target storage node in a storage system, where the storage system includes: at least two storage nodes, a slicing server, and each storage node.
  • the target storage node is: a storage node that receives the slice data in the storage node included in the storage system;
  • the device includes:
  • the receiving module 610 is configured to receive the slice data sent by the slice server, where the slice data is: the data obtained by the slice server to slice the data to be stored;
  • a second determining module 620 configured to determine a first type of storage node for storing slice data
  • a third obtaining module 630 configured to acquire a first type of storage location for storing slice data in the determined first type of storage node
  • the second writing module 640 is configured to write the obtained slice data to the acquired first type of storage location through the SAS network.
  • the third obtaining module 630 may include:
  • a second sending submodule configured to send a first storage request to the determined first type of storage node
  • a second receiving submodule configured to receive the first type of storage location sent by the determined first type of storage node, where the first type of storage location is: the determined location of the first type of storage node for storing slice data .
  • the second determining module 620 is specifically configured to:
  • a first type of storage node for storing slice data is determined according to at least one of the following information:
  • the storage system may further include a management node; the device may further include:
  • a second generation module configured to generate a storage index of each slice data obtained, where the storage index record of one slice data includes a correspondence between the slice data and a first type of storage location corresponding to the slice data, and a data to be stored Corresponding relationship with each slice data obtained by slicing the data to be stored;
  • a second sending module configured to send the generated storage index to the management node.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • the embodiment of the present application further provides an electronic device, as shown in FIG. 7, including a processor 710 and a memory 720;
  • a memory 720 configured to store a computer program
  • the processor 710 is configured to perform the following steps when executing the program stored on the memory 720:
  • the obtained slice data is written into the acquired first type of storage location through the SAS network.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • An electronic device provided by an embodiment of the present application may further perform a data storage method according to any one of the foregoing embodiments.
  • a data storage method may be performed according to any one of the foregoing embodiments.
  • a computer readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the above-described FIG. 3 and FIG. A data storage method according to any of the corresponding embodiments.
  • the embodiment of the present application further provides an electronic device, which can implement the following steps:
  • the obtained slice data is written into the acquired first type of storage location through the SAS network.
  • the target storage node can directly write the slice data obtained by the slice processing to the corresponding storage location, so as to avoid the data transmission. Transmission delay, which in turn increases data storage efficiency.
  • An electronic device provided by an embodiment of the present application may further perform a data storage method according to any one of the foregoing embodiments.
  • a data storage method may be performed according to any one of the foregoing embodiments.
  • FIG. 4 A data storage method according to any of the corresponding embodiments.
  • the communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.
  • the communication interface is used for communication between the above electronic device and other devices.
  • the memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
  • RAM random access memory
  • NVM non-volatile memory
  • the memory may also be at least one storage device located away from the aforementioned processor.
  • the above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or may be a digital signal processing (DSP), dedicated integration.
  • CPU central processing unit
  • NP network processor
  • DSP digital signal processing
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention concerne un système, un procédé et un dispositif de stockage de données. Le système de stockage comprend des nœuds de stockage qui sont connectés via un réseau SAS ; et un nœud de stockage cible est utilisé pour acquérir des données à stocker et pour exécuter un traitement de mise en tranche sur les données à stocker pour obtenir des données en tranche, et est également utilisé pour déterminer un nœud de stockage de premier type pour stocker les données en tranche obtenues, acquérir une position de stockage de premier type, dans le nœud de stockage de premier type déterminé, pour stocker les données en tranche, et pour écrire, via le réseau SAS, les données en tranche obtenues dans la position de stockage de premier type acquise. Au moyen de la solution technique fournie par les modes de réalisation de la présente invention, après qu'un traitement de mise en tranche est exécuté sur des données à stocker, un nœud de stockage cible peut écrire directement des données en tranche obtenues au moyen du traitement de mise en tranche respectivement à une position de stockage correspondante, ce qui évite l'apparition d'un retard de transmission causé par une transmission de données et améliore ainsi l'efficacité de stockage des données.
PCT/CN2019/075696 2018-03-09 2019-02-21 Système, procédé et dispositif de stockage de données WO2019170004A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810193323.5A CN110244904B (zh) 2018-03-09 2018-03-09 一种数据存储系统、方法及装置
CN201810193323.5 2018-03-09

Publications (1)

Publication Number Publication Date
WO2019170004A1 true WO2019170004A1 (fr) 2019-09-12

Family

ID=67845508

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/075696 WO2019170004A1 (fr) 2018-03-09 2019-02-21 Système, procédé et dispositif de stockage de données

Country Status (2)

Country Link
CN (1) CN110244904B (fr)
WO (1) WO2019170004A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111404980B (zh) * 2019-09-29 2023-04-18 杭州海康威视系统技术有限公司 一种数据存储方法及一种对象存储系统
CN111399761B (zh) * 2019-11-19 2023-06-30 杭州海康威视系统技术有限公司 存储资源分配方法、装置及设备、存储介质
CN112486421B (zh) * 2020-12-16 2024-05-28 上海达梦数据库有限公司 一种数据存储方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080201335A1 (en) * 2007-02-20 2008-08-21 Nec Laboratories America, Inc. Method and Apparatus for Storing Data in a Peer to Peer Network
CN102752402A (zh) * 2012-07-20 2012-10-24 广东威创视讯科技股份有限公司 一种云存储方法及系统
CN105025053A (zh) * 2014-04-24 2015-11-04 苏宁云商集团股份有限公司 基于云存储技术的分布式文件的上传方法及其系统
CN105872031A (zh) * 2016-03-26 2016-08-17 天津书生云科技有限公司 存储系统
CN106657356A (zh) * 2016-12-29 2017-05-10 郑州云海信息技术有限公司 一种云存储系统的数据写入方法、装置及云存储系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7356728B2 (en) * 2004-06-24 2008-04-08 Dell Products L.P. Redundant cluster network
KR100623553B1 (ko) * 2004-12-15 2006-10-16 한국정보보호진흥원 분산 저장장치 기반 침입 탐지 시스템 및 방법
CN102279777B (zh) * 2011-08-18 2014-09-03 华为数字技术（成都）有限公司 数据冗余处理方法、装置和分布式存储系统
CN102663005B (zh) * 2012-03-19 2014-03-26 杭州海康威视数字技术股份有限公司 基于云计算的海量视频文件存储系统、分析方法及其系统
EP2953025B1 (fr) * 2013-10-18 2017-01-25 Hitachi Ltd. Intégrité des données indépendante en fonction de la cible et récupération de la redondance dans un système de stockage distribué « shared-nothing »
CN105138280B (zh) * 2015-07-31 2018-10-19 成都华为技术有限公司 数据写入方法、装置及系统
CN206178763U (zh) * 2016-08-31 2017-05-17 金友兵 一种具有高可用性的数据存储装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080201335A1 (en) * 2007-02-20 2008-08-21 Nec Laboratories America, Inc. Method and Apparatus for Storing Data in a Peer to Peer Network
CN102752402A (zh) * 2012-07-20 2012-10-24 广东威创视讯科技股份有限公司 一种云存储方法及系统
CN105025053A (zh) * 2014-04-24 2015-11-04 苏宁云商集团股份有限公司 基于云存储技术的分布式文件的上传方法及其系统
CN105872031A (zh) * 2016-03-26 2016-08-17 天津书生云科技有限公司 存储系统
CN106657356A (zh) * 2016-12-29 2017-05-10 郑州云海信息技术有限公司 一种云存储系统的数据写入方法、装置及云存储系统

Also Published As

Publication number Publication date
CN110244904A (zh) 2019-09-17
CN110244904B (zh) 2020-08-28

Similar Documents

Publication Publication Date Title
US10942828B2 (en) Method for storing data shards, apparatus, and system
CN108139941B (zh) 基于网络流控制的动态资源分配
WO2019170004A1 (fr) Système, procédé et dispositif de stockage de données
JP4433967B2 (ja) マルチサイト上の遠隔二重化リンクを経由するハートビート装置、及びその使用方法
CN106210147B (zh) 一种基于轮询的负载均衡方法和装置
US11262916B2 (en) Distributed storage system, data processing method, and storage node
US20190220379A1 (en) Troubleshooting Method, Apparatus, and Device
US20210075665A1 (en) Implementing switchover operations between computing nodes
US20060129615A1 (en) Performing scheduled backups of a backup node associated with a plurality of agent nodes
US20090024823A1 (en) Overlayed separate dma mapping of adapters
WO2014000430A1 (fr) Procédé et appareil de réalisation de services battement de cœur en grappe à haute disponibilité
US20150288753A1 (en) Remote monitoring pool management
CN109873714B (zh) 云计算节点配置更新方法及终端设备
US11886309B2 (en) Cell-based storage system with failure isolation
CN108512753B (zh) 一种集群文件系统中消息传输的方法及装置
CN117632808B (zh) 多控存储阵列、存储系统、数据处理方法以及存储介质
US9952951B2 (en) Preserving coredump data during switchover operation
US7904752B2 (en) Synchronizing device error information among nodes
US20180131758A1 (en) Memory system including plurality of server nodes sharing storage array and operating method of memory system
EP3188029A1 (fr) Appareil de distribution et de gestion, système et procédé sur la base de sas
US20080250421A1 (en) Data Processing System And Method
CN115378853B (zh) 一种网络监控方法、装置和设备
CN107491270B (zh) 一种多控存储系统的资源访问方法及装置
CN115550287A (zh) 一种远程复制关系的建立方法及相关装置
WO2019170133A1 (fr) Procédé et dispositif de stockage de données

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19764270

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19764270

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19764270

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.03.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19764270

Country of ref document: EP

Kind code of ref document: A1