RU2777270C1 - Method and system for distributed storage of recovered data which ensures integrity and confidentiality of information - Google Patents

Abstract

FIELD: data storage.

SUBSTANCE: invention relates to a system for distributed storage of recoverable data and a method for distributed storage of recoverable data. The method for distributed storage of recoverable data with information integrity is performed by at least one system, which contains a plurality of storage nodes connected by a network and storing data in the form of extents, data service (DS) agents for managing extents, metadata service (MDS) agents for managing metadata , related to nodes and extents, and cluster manager (CM) agents. The method consists of the following steps: generating a checksum for each data extent; detection of a node failure in the system by one of the CM agents or a disk failure in the system by the DS agent; notifying DS or MDS agents of a failure; formation by independently notified DS or MDS agents of a recovery plan for data extents affected by the failure; and collectively restoring the affected extents based on the generated plan in the previous step using checksums, with each node including a plurality of disks; and a disk failure is detected by the DS agent based on the disk error rate.

EFFECT: improving the reliability of data recovery.

17 cl, 9 dwg

Description

FIELD OF TECHNOLOGY

[001] This technical solution generally relates to computer data storage systems and, in particular, to a distributed storage system for recoverable data and a method for recovering data after a failure of a storage node in a system or a disk in one of the nodes.

BACKGROUND OF THE INVENTION

[002] Currently, a large amount of computer data is usually stored in a distributed storage system. Distributed storage systems offer several benefits, such as the ability to increase storage capacity as user requirements grow, data reliability based on data redundancy, and the flexibility to maintain and replace failed components. Distributed storage systems have been implemented in various forms, such as Redundant Arrays of Independent Disks (RAID) systems, which are widely known in the art.

[003] For example, "TickerTAIP Parallel RAID Architecture" (by Cao et al.) describes a disk array (RAID) system that includes a number of worker nodes and source nodes. Each worker node has multiple disks connected via a bus. Source nodes provide connections to client computers. When a disk or node fails, the system recovers the lost data using the partial redundancy provided in the system. The described method only applies to RAID storage systems. Also, this method does not solve the problem of a distributed storage system consisting of a very large number of independent storage arrays.

[004] Also known from the prior art is US patent US 6438661 "Method, system, and program for managing meta data in a storage system and rebuilding lost meta data in cache" (right holder: International Business Machines Corp), which describes a method and system for recovering lost metadata in the cache. Metadata provides information about user data stored on a storage device. The method determines if the metadata tracks in the cache have changed, indicates in non-volatile memory that the metadata tracks have changed, and restores the metadata tracks. Data recovery includes accessing the data tracks associated with the metadata tracks, placing the accessed data tracks in a cache, and processing the data tracks to recover the metadata tracks. The described method does not recover data lost due to a node failure in a distributed storage system.

[005] Patent application US20020062422A1 "Method for rebuilding meta-data in a data storage system and a data storage system" (copyright holder: International Business Machines Corp) describes another method for restoring metadata in a storage system in which data streams are written to the system as segments. The method scans the metadata in each segment to determine the last segment written from each stream. It then reconstructs the metadata using the metadata in the segments, excluding the metadata for the last segments identified. The described method is not applicable to a distributed multi-node storage system and does not solve the problem of data recovery after a node failure.

SUMMARY OF THE INVENTION

[006] The technical problem or technical problem solved in this technical solution is the implementation of a method and system for distributed storage of recoverable data while ensuring the integrity and confidentiality of information.

[007] The technical result achieved by solving the above technical problem is to increase the reliability of data recovery.

[008] The specified technical result is achieved by implementing a method for distributed storage of recoverable data while ensuring the integrity and confidentiality of information, which is performed by at least one system that contains a plurality of storage nodes connected by a network and storing data in the form of extents, data service agents (DS ) for managing extents, metadata service (MDS) agents for managing metadata related to nodes and extents, and cluster manager (CM) agents, the method consisting of the following steps: detection of a disk failure in the system by one of the CM agents; notifying DS or MDS agents of a failure; formation by independently notified DS or MDS agents of a recovery plan for data extents affected by the failure; and collectively restoring the affected extents based on the generated plan in the previous step, each node including a plurality of disks; and a disk failure is detected by the DS agent based on the disk error rate.

[009] Also, the specified technical result is achieved through the implementation of a distributed storage system for recoverable data ensuring the integrity and confidentiality of information, which contains a plurality of storage nodes connected by a network, each node storing data in the form of extents; a data service (DS) agent in each node to manage the extents in the node; a plurality of metadata service (MDS) agents for managing metadata related to nodes and extents, the MDS agents operating on a subset of the nodes; a Cluster Manager (CM) agent on each node to detect a system failure and notify a subset of DS or MDS agents of the failure, whereby upon notification of a failure, a subset of DS or MDS agents independently generates a plan to restore the extents affected by the failure and collectively restore the affected extents based on the plan; and a persistent map that correlates data extents with nodes, with each MDS agent managing a subset of the map.

[0010] In some implementations of the technical solution, the system further comprises an interface that allows the application to access data stored in the system.

[0011] In some implementations of the technical solution, the system further comprises a means for determining a subset of nodes in which MDS agents operate.

[0012] In some implementations of the technical solution, each CM agent maintains an ordered list of nodes that are currently running in the system.

[0013] In some embodiments of the technical solution, each CM agent includes a means for detecting node failure.

[0014] In some embodiments of the technical solution, each CM agent includes a means for discovering a new node in the system.

[0015] In some embodiments of the technical solution, said means for discovering a new node includes means for monitoring network messages from a new node.

[0016] In some implementations of the technical solution, each DS agent distributes updates to extents in the associated node to other DS agents in the system.

[0017] In some implementations of the technical solution, each DS agent controls the caching of data in the associated node.

[0018] In some implementations of the technical solution, each node contains a plurality of data disks, and each DS agent includes a means for detecting disk failure in an associated node.

[0019] In some implementations of the technical solution, the means for detecting a disk failure is based on the error rate of the disks.

[0020] In some implementations of the technical solution, the recovery plan includes a list of extents that must be restored to restore data redundancy in the system.

[0021] In some implementations of the technical solution, the extents to be restored are jointly restored by the DS agents.

[0022] In some embodiments, space is allocated in nodes that are still running to replace extents affected by the failure.

[0023] In some implementations of the technical solution, the data in the affected extents is determined and transferred to the allocated space.

[0024] In some implementations of the technical solution, the DS agents are notified of the failure and the notified DS agents determine those extents that have data on the failed node.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG. 1 is a block diagram showing a typical configuration of a shared storage system according to the invention.

[0026] FIG. 2 illustrates a block diagram showing a typical physical configuration of a storage node of a shared storage system.

[0027] FIG. 3 illustrates an example of a data extent and its fragments that reside on different storage nodes.

[0028] FIG. 4 illustrates a block diagram of a typical configuration of a shared storage system with agents to support data recovery operations in accordance with the invention.

[0029] FIG. 5 illustrates an example of a Fragment_To_Disk card in accordance with the invention.

[0030] FIG. 6 illustrates an example of an Extent_To_Node map in accordance with the invention.

[0031] FIG. 7 illustrates a flowchart showing a general data recovery process after a node failure or disk failure in a shared storage system.

[0032] FIG. 8 is a flowchart showing the preferred process for creating a data recovery plan after a node failure.

[0033] FIG. 9 is a flowchart showing the preferred process for creating a data recovery plan after a disk failure.

DETAILED DESCRIPTION OF THE INVENTION

[0034] Below will be discussed in detail the terms and their definitions used in the description of the technical solution.

[0035] In this invention, the system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations (actions, instructions), centralized and distributed databases, smart contracts.

[0036] A command processing device refers to an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs), a smart contract, an Ethereum virtual machine (EVM), or the like. An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. The role of a storage device can be, but not limited to, hard disk drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

[0037] A program is a sequence of instructions intended to be executed by a computer control device or command processing device.

[0038] The Cluster Manager is typically a back end graphical user interface (GUI) or command line software that runs on one or all of the cluster nodes (in some cases it runs on a different server or cluster of management servers). The Cluster Manager works in conjunction with the Cluster Management Agent. These agents run on each cluster node to manage and configure services, a set of services, or to manage and configure the entire cluster server itself (see supercomputing). In some cases, the cluster manager is mainly used to dispatch the work that the cluster (or cloud) must do. In the latter case, a subset of the cluster manager could be a remote desktop application that is used not for configuration, but simply for submitting work and receiving work results from the cluster. In other cases, the cluster has more to do with availability and load balancing than with compute or specific service clusters.

[0039] The invention will be described primarily as a system for distributed storage of recoverable data while ensuring the integrity and confidentiality of information. However, those skilled in the art will appreciate that a system, which may be a data processing means, including a central processing unit, memory, an I/O unit, a program store, a backplane, and other relevant components, may be programmed or otherwise designed to facilitate implementation of the invention. Such a system would include appropriate software to carry out the operations of the invention.

[0040] In addition, a technical solution, such as a pre-recorded disc or other similar computer program product, for use with a data processing system, may include a storage medium and software stored thereon for controlling the data processing system to facilitate the implementation of the invention. . Such devices and technical solutions also fall within the essence and scope of the invention.

[0041] FIG. 1 is a high-level block diagram of a data integrity and confidentiality distributed storage system 100 for recoverable data in accordance with the invention. System 100 includes a large number of storage nodes 101, which are typically connected to node structure 102. Each node 101 includes one or more privately attached drives 103 that contain customer data. The node structure 102 is preferably a Fiber Channel network (ultra-high speed (up to 1 Gb/s and higher) full duplex data transfer scheme), an iSCSI network, or another type of network. The storage nodes 101 are described in more detail below with reference to FIG. 2–6. The integrity of the data stored in the storage devices depends on the physical state of these devices, namely whether they are functioning properly or, for example, at least to some extent faulty.

[0042] The data clients supported by the shared storage system 100 are called hosts. Hosts can be connected to the storage system 100 using two types of interfaces: a host driver or a gateway. On FIG. 1, two hosts 107 and 108 are shown. In the case of host 107, its application 109 accesses data in storage nodes 101 using host driver 110. In the case of the host 108, its application 111 accesses the client's data using the gateway agent 112, which resides in the gateway node 113. More detailed information about the functions of host driver 110 and gateway agent 112 is described below with other agents on system 100. Applications 109 and 111 are typically file systems or database systems.

[0043] FIG. 2 illustrates a preferred embodiment of storage node 101 in terms of its physical components. The node 101 has a fabric interface 201 for connecting to the node fabric 102, RAM 202 and NVRAM 203 for storing data while the node is processing data, a processor 204 for managing and processing data, and a disk interface 205. The node 101 further includes a set of storage disks 207 data of clients that are connected to the disk interface 205 via the disk array 206 .

[0044] The storage nodes 101 in the shared storage system 100 are not redundant, that is, the hardware components in the nodes, such as the processor 204, memory 202-203, and disks 207, are not redundant. Thus, the failure of one or more components in the storage node 101 can disable the entire node and prevent access to the disks 207 on that node. This lack of redundancy, combined with the higher failure rate resulting from the large number of disks 207 and nodes 101, means that the software in system 100 must fail over and make customer data redundant enough to allow data to be accessed despite failures. .

[0045] In the shared storage system 100, the availability of client data can be ensured using a technique based on redundancy such as RAID data partitioning, simple replication. In each of these methods, customer data is stored in a set of blocks or chunks of physical memory, with each block of data located on a separate node.

[0046] The distribution of client data across nodes 101 and disks 103 is performed by the distribution process to achieve one or more system goals. The desired data distribution process should have two key goals: availability and load balancing. With respect to availability, if node 101 or disk 103 fails, all storage chunks on that node 101 or disk 103 may be lost. Therefore, there must be enough redundant data on the other nodes to recover the lost pieces of data. As far as load balancing is concerned, the data read/write load should be distributed across 101 nodes to avoid bottlenecks.

[0047] The organization of the implementation data is as follows, as disclosed below.

[0048] Client data in system 100 is organized into fixed-size blocks that are linearly addressed in SCSI logical devices (ie, disks). Each LUN has a number called a LUN (or LUN). The system looks like a SCSI block device on a single bus with multiple LUNs (although this is flexible for host operating systems that have limited LUN capabilities). Internally, a logical unit is mapped to a container. A container is a collection of objects called extents. The container attributes include an identifier (container ID) that uniquely identifies the container in the storage system 100. The host LUN mapping or mapping function between LUN and container ID is explicitly stored and managed by the system 100.

[0049] Extent - in file systems, a contiguous area of a storage medium. Typically, in extent-aware filesystems, large files consist of several extents that are not directly related to each other. The extent is stored as a group of fragments, where each fragment is located on a different node. A fragment is uniquely identified by an extent ID and a fragment ID. The group of fragments that make up an extent is called an extent group (or fragment relationship group). Fig. 3 illustrates an extent 300 that contains three fragments 301, 302, and 303. Fragments 301, 302, and 303 are located at node 1, node 2, and node 3, respectively.

[0050] An extent refers to a number of contiguous blocks on storage media, and may vary depending on the application. For example, one application may partition a disk into extents of the same size, while another application may partition a disk into extents of a different size.

[0051] If a certain block on the disk is changed after the shadow copy is made, then before the block is changed, the extent containing that block is copied to a storage location located in the differential region. For a particular "shadow" copy, the extent is only copied the first time any block within the extent is modified. When a request is received for information contained in a shadow copy, a check is first made to determine whether the block has changed in the original volume (eg, by checking whether the extent containing the block exists in the differential area). If the block has not changed, then the data from the original volume is retrieved and returned. If the block has changed, then the data from the differential area is retrieved and returned. Note that if a block is overwritten with the same data, then the extent containing that block is not written to the differential area.

[0052] The extent attributes include an extent identifier (extent ID), a redundancy level, and a list of nodes 101 that have data dependencies (replicas, erasure codes, etc.) on client data stored in that extent. The extent ID is unique within the container. The reliability and availability of the extent 300 is achieved by distributing its fragments 301-303 into separate nodes 101.

[0053] In some system implementations, various cryptographic functions can be implemented to ensure data integrity, such as SFLA-1 (secure hash algorithm), MD5, and RSA, and several encryption options are provided, including DES, 3DES, and AES. In some technical solutions, cyclic redundancy check (CRC) is one of the checksum finding algorithms designed to check data integrity. In accordance with non-limiting embodiments of the present technology, a cyclic redundancy code may be implemented with a finite field polynomial division operation.

[0054] In some embodiments, to check the integrity of each extent, its checksum is determined. After calculating the checksum from each extent, for example, using the MD5 algorithm, a set of values is obtained: 1bc29b36f623ba82aaf6724fd3bl6718,…, 026f8e459c8f89ef75fa7a78265a0025. Checksum - some value calculated from a set of data by applying a computational algorithm and used to verify the integrity of data during transmission or storage.

[0055] In a preferred embodiment of the method, the data node is divided into three areas, while each of the three areas is divided into a replacement sub-area and a data sub-area, while the sub-areas are divided into fixed-length blocks, each of which is protected by a CRC checksum (cyclic redundancy check - cyclic redundancy code) to check the integrity of the data in the block, and, if an incorrect checksum is detected in a block from the data subarea, the corresponding block from the replacement subarea is used instead of this block from the data subarea.

[0056] When restoring data in data nodes, checksums of extents are used when checking integrity.

[0057] Data privacy in the current system is addressed by applying additional data encryption. Encryption is the traditional way to ensure the confidentiality of data while it is stored. This solution uses an encryption algorithm based on the use of block ciphers. Such ciphers operate with data fragments of a fixed length - blocks, and combine durability and high speed.

[0058] The trend towards an increase in the amount of stored information and the speed of transmitted data requires high performance from the used block ciphers. An effective method of increasing the speed of the encryption algorithm is the use of parallel computing. One of the ways to organize parallel computing in the case of a software implementation of an algorithm is the use of SIMD technologies, which are based on the use of one processor instruction for the simultaneous processing of several data fragments previously placed on one register. SIMD technologies have become widespread and are supported on most modern computing platforms, including general-purpose processors from Intel and AMD. This solution can use several typical sets of SIMD instructions, each of which is designed to work with registers of a certain length, widely known from the prior art.

[0059] In this technical solution, in block cipher algorithms, SIMD technologies are used to efficiently process several blocks of data at once. Using SIMD technologies, an encryption algorithm designed to process one block is performed simultaneously for several blocks. The effectiveness of this approach directly depends on the possibility of parallel execution of the transformations and operations used in the encryption algorithm, which, in turn, is determined by the presence in the computing platform of the corresponding SIMD instructions. If it is possible to parallelize each of the operations, this approach allows processing several data blocks in the time required to process one data block, that is, the performance of the algorithm grows in proportion to the number of simultaneously processed blocks. Since the number of simultaneously processed blocks is determined by the length of the used registers, the performance in this case increases in proportion to the increase in the length of the used registers. In this technical solution, the Grasshopper encryption algorithm can be used. Grasshopper (English Kuznyechik or English Kuznechik) is a symmetric block cipher algorithm with a block size of 128 bits and a key length of 256 bits, which uses an SP network to generate round keys.

[0060] The system configuration is detailed below.

[0061] FIG. 4 is a block diagram showing a typical configuration of a shared storage system 400 with the necessary agents to support data recovery operations. Each node 401 of the collective storage system 400 simultaneously executes one or more software agents, also called modules. There are four types of agents in the shared storage system 400: cluster manager (CM) agents 410, data server (DS) agents 420, metadata server (MDS) agents 430, and host driver (HD) agent 440. Not all four agent types work on every storage node 401 . In addition, in preferred embodiments of the invention, no more than one instance of an agent type can run on node 401.

[0062] The Cluster Manager (CM) agent works as follows.

[0063] The Cluster Manager (CM) agent 410 is responsible for maintaining an ordered list of nodes 401 that are currently operating in the shared storage system 400, detecting node failures, and discovering new nodes in the system 400. In preferred embodiments of the invention, only one CM agent 410 runs on each node 401. Each CM agent 410 in system 400 has a unique name, which is preferably a unique identifier for the corresponding node. The ordered lists of current functional nodes 401 maintained by CM agents 410 in system 400 are identical to each other. Thus, all nodes 401 in system 400 have the same information about which nodes 401 are currently running on the system, i.e., an identical view of the system.

[0064] The cluster manager agent 410 detects a node failure when it does not receive the expected response within the failure detection interval of the failed node. Fault detection is well known in the art.

[0065] When multiple nodes 401 attempt to simultaneously join or leave the shared storage system 400, each CM agent 410 ensures that the join and leave notifications are delivered sequentially in the same order to all nodes 401 in the system 400.

[0066] The Data Service (DS) agent works as follows.

[0067] The data service (DS) agent 420 is responsible for managing the raw data in the storage node 401 in the form of data fragments residing on that node. This includes data caching and data redundancy. Data caching is related to write caching or read ahead. To provide data redundancy for an extent, updating any if its data chunks is propagated to nodes that contain related chunks in the same extent group. In addition, the DS agent 420 maintains a persistent map called a Fragment_To_Disk map. The Fragment_To_Disk map contains information that correlates data fragments in the node 101 on which the DS agent is running with logical block addresses on disks 103 in that node. On FIG. 5 shows an example Fragment_To_Disk map 400 that indicates that fragment 0001 of extent ABC0052 is located on disk 01 at disk address 1001.

[0068] The DS agent 420 also manages a local copy of a second permanent map, called the Extent_To_Node map. This map includes information that correlates the extent with the work nodes that contain the fragments that make up that extent. The DS agent 420 that contains the first fragment in the extent group is called the leader for that extent. DS agents 420 containing remaining fragments of this size are called successors. The extent group leader is responsible for restoring, restoring, and updating data for successors. Leadership is transferred when a disk or node fails, resulting in the loss of a leader. The consistency control mechanism for reading/writing to fragment mapping is controlled by the leader to the same extent. That is, all I/Os are sent to the leader of the extent group, and the leader acts as a serialization point for reads/writes to that extent.

[0069] Extent group leaders in system 400 are distributed among nodes 401 using a distribution algorithm to ensure that no node becomes a bottleneck for being a leader. You can relax the requirement that all data reads go through the leader. You can also ease the need for all writes to be passed to the leader by carefully synchronizing clocks for nodes and using timestamps. On FIG. Figure 6 shows an example Extent_To_Node map for extent ABC0052.

[0070] All DS agents 420 in storage system 400 collectively manage extent groups in the system during read and write client data, disk or node failure recovery, reorganization, and snapshot operations.

[0071] In preferred embodiments of the invention, there is one instance of the DS agent running on each node in the shared storage system 400 . Data extents are completely disk-based, meaning that an extent does not span multiple disks. Therefore, with respect to extents, the failure of a disk containing an extent fragment is akin to the failure of node 401 containing the same fragment.

[0072] The data service agent 420 detects a disk failure using regular ping request timeouts. For example, when a disk is not in use, the DS agent 420 polls the disk at regular intervals to determine the status of the disk. In the event of request timeouts, the DS agent 420 may periodically check the disk error rate to determine its status.

[0073] The Metadata Service (MDS) agent is detailed below.

[0074] The Metadata Service (MDS) agent 430 is responsible for managing the Extent_To_Node map, a persistent map that correlates all data extents in the system with the nodes that contain the fragments that make up the extents. Each DS agent 420 (described above) has an Extent_To_Node map that contains entries only for those extents whose fragments it manages. On the other hand, the MDS agent 430 has an Extent_To_Node map for all extents in the storage system 400 . The MDS map is indexed with the extent ID as the primary key. A secondary index on nodes is also supported, which is useful when creating a repair plan in response to a node failure. Also, unlike the DS agent 420, the MDS agent 430 does not run on every node 401. Instead, an adaptive method is used to determine the set of nodes 401 that will run the MDS agents 430. The MDS agent 430 also stores and manages extent metadata and performs extent creation and deletion.

[0075] Converting an extent in a container to its extent group is performed by the MDS agent 430. In addition, extent allocation is provided by the MDS agent 430, which generates an extent group for the new extent. The MDS agent 430 further manages the list of configured containers and their attributes.

[0076] The Host Driver Agent (HD) is discussed below.

[0077] The host driver (HD) agent 440 is an interface through which the client application 402 can access data in the shared storage system 400. The HD agent 440 communicates with the application 402 in terms of logical units (LUNs) and with the rest of the storage system 400 in terms of containers and data extents. The HD agent 440 is typically located in the host 403 where the application 402 is running. However, its interface functions may be provided in the storage node 404 in the form of a gateway agent 405 . The host application then accesses the data in the storage system 400 through a gateway agent 405 that resides in the gateway node 404 .

[0078] To access client data in system 400, HD agent 440 or gateway agent 405 determines an extent and a leader for that extent. The HD agent 440 or gateway agent 405 then accesses the data from the storage node that contains the extent leader. To access a data extent, the client must first obtain the location of the extent. This function is provided by the agent 430 MDS. Given an extent identifier, the MDS agent 430 returns a list of nodes where fragments of that extent can be found.

[0079] Data recovery is discussed in detail below.

[0080] A process for restoring data to a shared storage system 400 after a node failure or disk failure is now described. Recovery is the process of recreating lost data due to a failed disk or node on new nodes or disks to insure against data loss due to subsequent failures. The desired recovery process must have the following properties:

[0081] a) Reliability: All affected customer data is eventually recovered.

[0082] b) fault tolerance: if a second failure occurs during data recovery, then property (a) is preserved.

[0083] c) efficiency: the recovery process requires as few message exchanges and data movements between nodes as possible.

[0084] d) Balanced: Data recovery work is distributed across nodes to minimize the impact on concurrent access to client data.

[0085] e) scalability: The time required to recover from a failed disk or node should scale inversely with system size, ie the larger the system, the shorter the recovery time. This is important because the system has to deal with higher failure rates as the system grows.

[0086] FIG. 7 is a flowchart showing a general process for recovering lost data after a node failure or disk failure in the shared storage system 400. At block 701, a failure is detected by an agent in system 400. As described above for the CM and DS agents, a node failure is detected by the CM agent and a disk failure is detected by the DS agent. At block 702, agents in system 400 responsible for coordinating data recovery are notified of the failure. In one preferred embodiment in accordance with the invention, the restoration is coordinated by one of the MDS agents in the system. In another embodiment of the invention, recovery is coordinated by one of the DS agents in the system. At step 703, the coordinator agent generates a data recovery plan, as described below with reference to FIG. 8–9. The plan determines the data extents in the system that are affected by the failure. At block 704, the agents in the system collectively restore the affected extents based on the restoration plan.

[0087] In FIG. 8 is a flow diagram showing the preferred process for creating a data recovery plan after a node failure in accordance with the invention. At block 801, the MDS agent determines the data extents affected by the node failure using the Extent_To_Node map. Entries in this map map the extents in the storage system to the nodes in the system. When this map is scanned, affected extents are identified based on the identity of the failed node. At 802, the method determines if leaders of any degree are lost. If the leader is lost due to a failure, then a new leader for that extent is selected from the remaining fragments in the extent group at step 803. A simple choice here is to have the DS agent hold the first of the remaining fragments in the extent as the new leader, if not was the previous leader. Then, at block 804, the MDS agent allocates new space on the still running nodes to accommodate the affected extent data.

[0088] FIG. 9 is a flowchart showing the preferred process for creating a data recovery plan after a disk failure in accordance with the invention. At block 901, the DS agent associated with the failed disk determines the data extents affected by the disk failure based on the Fragment_To_Disk mappings in the DS Agent. This map specifies the fragment IDs for the fragments on each disk. Using the ID of the failed drive, fragments on the failed drive can be identified by scanning the map. At 902, the method determines if leaders of any degree are lost. If the leader is lost due to a failure, then a new leader for that extent is selected from the remaining fragments in the extent group at step 903. A simple choice here is to have the DS agent hold the first of the remaining fragments in the extent as the new leader, if not was the previous leader. At block 904, new space is allocated in the still-functioning nodes to accommodate the affected extent data. In one preferred embodiment of the invention, the DS agents start a restore task and thus allocate new space. In another preferred embodiment of the invention, the MDS agent is responsible for coordinating the restoration work and handling the allocation task.

[0089] The above example showed that the claimed method and system for distributed storage of recoverable data with ensuring the integrity and confidentiality of information functions correctly, is technically feasible (a) and allows solving the problem.

[0090] The elements of the proposed technical solution are in a functional relationship, and their joint use leads to the creation of a new and unique technical solution. Thus, all blocks are functionally connected.

[0091] All blocks used in the system can be implemented using electronic components used to create digital integrated circuits, which is obvious to a person skilled in the art. Not limited to, microcircuits can be used, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGA), the logic of which is set by programming. Programmers and debugging environments are used for programming, allowing you to set the desired structure of a digital device in the form of a circuit diagram or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs can be programmable logic controllers (PLCs), basic matrix crystals ( BMK), requiring a factory production process for programming; ASIC - specialized custom-made large integrated circuits (LSI), which are significantly more expensive for small-scale and single-piece production.

[0092] Typically, the FPGA chip itself consists of the following components:

• configurable logical blocks that implement the required logical function;

• programmable electronic links between configurable logic blocks;

• programmable input/output blocks that provide communication between the external output of the microcircuit and the internal logic.

[0093] Blocks can also be implemented using read-only memories.

[0094] Thus, the implementation of all used blocks is achieved by standard means based on the classical principles of implementing the fundamentals of computer technology.

[0095] As will be appreciated by one of skill in the art, aspects of the present technical solution may be implemented as a system, method, or computer program product. Accordingly, various aspects of the present technical solution may be implemented solely as hardware, as software (including application software, etc.), or as an embodiment combining software and hardware aspects, which may be generally referred to as a "module" , "system" or "architecture". In addition, aspects of the present technical solution may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code embodied thereon.

[0096] Any combination of one or more computer-readable media can also be used. The computer-readable storage medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specifically, examples (non-exhaustive list) of a computer-readable storage medium include: an electrical connection using one or more wires, a portable computer diskette; hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), fiber optic connection, compact disc read only memory (CD-ROM), optical storage device, magnetic storage device, or any combination of the above. As used herein, a computer-readable storage medium can be any flexible storage medium that can contain or store a program for use by or in connection with a system, device, apparatus.

[0097] The program code embedded in a computer-readable medium may be transmitted using any medium, including, without limitation, wireless, wired, fiber optic, infrared, and any other suitable network, or a combination of the foregoing.

[0098] The computer program code for performing the operations for the steps of the present technical solution may be written in any programming language or combinations of programming languages, including an object-oriented programming language such as Python, R, Java, Smalltalk, C++, and so on, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may be executed in whole, in part, on the user's computer, or as a separate software package, in part on the user's computer and in part on a remote computer, or entirely on a remote computer. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN), a wide area network (WAN), or a connection to an external computer (eg, via the Internet via ISPs).

[0099] Aspects of the present technical solution have been described in detail with reference to block diagrams, circuit diagrams and/or diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the present technical solution. It should be appreciated that each block from the block diagram and/or diagrams, as well as combinations of blocks from the block diagram and/or diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, a special purpose computer, or other data processing device to create a procedure, such that the instructions executed by the computer processor or other programmable data processing device create the means to implement the functions/actions specified in block or blocks of a flowchart and/or diagram.

[00100] These computer program instructions may also be stored on a computer-readable medium that can control a computer other than a programmable data processing device or other than devices that operate in a particular manner such that the instructions stored on the computer-readable medium create a device including instructions that perform the functions/actions indicated in the block diagram and/or diagram.

Claims (28)

1. A method for distributed storage of recoverable data with information integrity, performed by at least one system that contains a plurality of storage nodes connected by a network and storing data in the form of extents, data service (DS) agents for managing extents, metadata service (MDS) agents to manage metadata related to nodes and extents and cluster manager (CM) agents, the method consisting of the following steps: • generating a checksum for each data extent; • detection of a node failure in the system by one of the CM agents or a disk failure in the system by the DS agent; • notifying DS or MDS agents of the failure; • independently notified DS or MDS agents form a recovery plan for data extents affected by the failure; and • collectively recovering the affected extents based on the plan generated in the previous step using checksums, with each node including multiple disks; and disk failure is detected by the DS agent based on the disk error rate. 2. A system for distributed storage of recoverable data with information integrity, containing • a plurality of storage nodes connected by a network, each node storing data in the form of extents over which checksums are calculated; • a data service (DS) agent in each node to manage the extents in the node; • a plurality of Metadata Service (MDS) agents for managing metadata related to nodes and extents, with MDS agents operating on a subset of nodes; • Cluster Manager (CM) agent in each node to detect node failure in the system, • whereby, upon notification of a failure, a subset of the DS or MDS Agents independently generates a plan to restore the extents using the checksums affected by the failure and collectively restore the affected extents based on the plan; and • a persistent map that correlates data extents with nodes, with each MDS agent managing a subset of the map. 3. The storage system of claim 2, further comprising an interface allowing an application to access data stored on the system. 4. The storage system of claim 2, further comprising means for determining a subset of nodes running MDS agents. 5. The storage system of claim 2, wherein each CM agent maintains an ordered list of nodes that are currently running on the system. 6. The storage system of claim 2, wherein each CM agent includes means for detecting node failure. 7. The storage system of claim 2, wherein each CM agent includes means for discovering a new node in the system. 8. The storage system of claim 7, wherein said means for discovering a new node includes means for monitoring network communications from the new node. 9. The storage system of claim 2, wherein each DS agent distributes updates to extents in an associated node to other DS agents in the system. 10. The storage system of claim 2, wherein each DS agent manages data caching at an associated node. 11. The storage system of claim 2, wherein each node includes a plurality of data disks and each DS agent includes means for detecting disk failure in an associated node. 12. The storage system of claim 11, wherein said drive failure detection means is based on the error rate of the drives. 13. The storage system of claim 2, wherein the recovery plan includes a list of extents that must be recovered to restore data redundancy to the system. 14. The storage system of claim 13, wherein the extents to be restored are jointly restored by the DS agents. 15. The storage system of claim 13, wherein space is allocated in nodes that are still running to replace extents affected by the failure. 16. The storage system of claim 15, wherein the data in the affected extents is determined and transferred to the allocated space. 17. The storage system of claim 2, wherein the DS agents are notified of the failure and the notified DS agents determine those extents that have data on the failed node.

Images