CN114443431A - Distributed storage system fault recovery method, device, equipment and medium - Google Patents
Distributed storage system fault recovery method, device, equipment and medium Download PDFInfo
- Publication number
- CN114443431A CN114443431A CN202210087994.XA CN202210087994A CN114443431A CN 114443431 A CN114443431 A CN 114443431A CN 202210087994 A CN202210087994 A CN 202210087994A CN 114443431 A CN114443431 A CN 114443431A
- Authority
- CN
- China
- Prior art keywords
- monitored
- storage
- storage node
- recovered
- object storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000011084 recovery Methods 0.000 title claims abstract description 33
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 12
- 230000001360 synchronised effect Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a distributed storage system fault recovery method, which comprises the steps of marking storage nodes to be monitored and faulty object storage equipment in the storage nodes to be monitored when a first preset quantity requirement is met; after the storage node to be monitored is recovered from the fault, if the second preset quantity requirement is met in the preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling the fault mark of the recovered object storage devices in the storage node to be monitored and the fault mark of the storage node to be monitored; otherwise, after waiting for a preset time period, uniformly changing the state of the restored object storage devices in the storage nodes to be monitored, and canceling the fault marks of the restored object storage devices in the storage nodes to be monitored.
Description
Technical Field
The present invention relates to the field of distributed system storage, and in particular, to a method, an apparatus, a device, and a medium for recovering a failure in a distributed storage system.
Background
In order to ensure the security of user data, the distributed storage system copies multiple copies (replication) of the same data and stores the copies on different storage nodes. In order to manage data distribution and recovery, a Storage pool in a general Storage system is divided into multiple PGs (place Group, a logical unit for data distribution), and each PG includes a plurality of OSD (Object Storage devices) according to a certain duplication or erasure rule. For example, a three copy pool is created, and one copy of data is stored in three copies, each on three OSDs, and each PG contains three OSDs, and each OSD has one copy of data. The data recovery among the multiple copies is completed through PG, and the multiple copies of data are synchronized according to the states of the OSD. In a normal scenario, each OSD has 4 states: UP, DOWN, IN, OUT. The UP and IN states are typically simultaneous, representing that the OSD is providing read and write services normally. The DOWN state identifies that the OSD has a fault and cannot provide service, when an OSD in the DOWN state exists among copies in the PG, all objects written in the PG are subjected to data degradation writing, that is, few copies are written (in the case of one OSD in the DOWN state in the three-copy scene, data is written in only two copies), so that the DOWN state is a temporary state, and when a certain time (for example, 10min) elapses, the OSD cannot provide service, the OSD state is marked as OUT by the distributed cluster, that is, the PG member is kicked OUT, at this time, the PG reselects an OSD as a member from the OSDs remaining in accordance with the fault domain (a waiting process, that is, the OSD marked as the OUT state in the PG is kicked OUT, and the OSD as a member from the OSD remaining in accordance with the fault domain in the PG) and synchronizes all data maintained by the PG to the new OSD, that is, a data recovery process.
In the above process, the peering process and the data recovery process of the PG require calculation and a large number of temporary PGs (members who once belong to the PG may generate a corresponding PG for receiving read-write services in the data recovery process and delete the PG after the data recovery is completed) are generated, which consumes system CPU and memory resources. When a large number of OSDs on a certain storage node are all in the DOWN state, the system resource occupation of the storage node is very serious, and even when a swap partition is used (the memory is released very slowly), other service processes such as OSD and MON (Monitor daemon of a cluster) on the node are slow to process, and serious consequences such as service waiting, interruption, even cluster abnormality and the like are caused. Therefore, it is desirable to avoid having a large number of OSDs in the storage node in the DOWN state, i.e., the temporary state.
In the prior art, in order to prevent the cluster from performing unnecessary data reconstruction, when a certain node fails (network failure and power failure), all OSDs of the corresponding node are marked as DOWN, and then the OUT time is set to be one hour (the node fails and does not kick the disk for a short time), during which time, because the OSDs cannot be OUT, the PG cannot be elected, and data reconstruction cannot be performed. However, after the fault is recovered, because the OSDs are not started at the same time, when a certain OSD start flag of the node is UP (that is, the OSD is recovered), the condition is not met by the policy of not kicking the disk when the previous node fails, and at this time, the rest of the non-UP (recovered) OSDs on the node become OUT, so that almost all PGs on the node are in a waiting state, a large amount of data recovery is generated, and a large amount of temporary PGs are generated at the same time, which further aggravates the above-mentioned system resource occupation condition and is not favorable for the stability of the distributed storage cluster.
Disclosure of Invention
The invention provides a distributed storage system fault recovery method for solving the problems in the prior art, and the method can reduce the frequency of OSD state change in storage nodes as much as possible when the storage nodes are recovered from faults, thereby reducing PG waiting and temporary PG generation, greatly reducing the occupation of system resources and improving the stability of clusters.
The first aspect of the present invention provides a distributed storage system fault recovery method, including:
monitoring whether a storage node has a fault object storage device, and marking the storage node to be monitored and the fault object storage device in the storage node to be monitored if the quantity of the fault object storage devices in the storage node to be monitored meets a first preset quantity requirement;
after the storage node to be monitored is recovered from a fault, acquiring the recovered number of the object storage devices in the storage node to be monitored, if the recovered number of the object storage devices in the storage node to be monitored meets a second preset number requirement in a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling a fault mark of the recovered object storage devices in the storage node to be monitored and a fault mark of the storage node to be monitored;
and if the recovered number of the object storage devices in the storage node to be monitored does not meet the second preset number requirement, after waiting for a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, canceling the fault marks of the recovered object storage devices in the storage node to be monitored, and updating the number of the unrecovered object storage devices in the storage node to be monitored.
Optionally, the first preset number is required to be greater than a first preset ratio threshold value when the ratio of the number of failed object storage devices in the storage node to be monitored to the number of all object storage devices in the storage node to be monitored is greater than the first preset ratio threshold value.
Optionally, the second preset number requires that a ratio of the number of the recovered object storage devices in the storage node to be monitored to the number of the object storage devices marked with the fault in the storage node to be monitored is greater than a second preset ratio threshold.
Optionally, the method further comprises:
and after the state of the recovered object storage devices in the storage nodes to be monitored is uniformly changed, emptying the number of the recovered object storage devices of the currently counted storage nodes to be monitored.
Further, still include:
when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, and whether the recovered number of the object storage devices in the storage nodes to be monitored meets a second preset number requirement or not is judged within a preset time period.
Optionally, the method further comprises:
when it is monitored that new fault object storage equipment exists in the storage node to be monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement.
Optionally, the method further comprises:
when monitoring that the storage node to be monitored has new fault object storage equipment in a preset time period, no longer continuing to wait, and immediately and uniformly changing the state of the recovered object storage equipment in the storage node to be monitored.
A second aspect of the present invention provides a failure recovery apparatus for a distributed storage system, including:
the monitoring marking module is used for monitoring whether the storage nodes have the failed object storage equipment or not, and marking the storage nodes to be monitored and the failed object storage equipment in the storage nodes to be monitored if the number of the failed object storage equipment in the storage nodes to be monitored meets a first preset number requirement;
the first state changing module is used for acquiring the number of the restored object storage devices in the storage node to be monitored after the storage node to be monitored is recovered from a fault, and if the number of the restored object storage devices in the storage node to be monitored meets the requirement of a second preset number within a preset time period, uniformly changing the state of the restored object storage devices in the storage node to be monitored, and canceling the fault mark of the restored object storage devices in the storage node to be monitored and the fault mark of the storage node to be monitored;
and the second state changing module is used for uniformly changing the states of the restored object storage devices in the storage node to be monitored after waiting for a preset time period if the restored number of the object storage devices in the storage node to be monitored does not meet the second preset number requirement, canceling the fault mark of the restored object storage devices in the storage node to be monitored, and updating the number of the unrecovered object storage devices in the storage node to be monitored.
A third aspect of the present invention provides an electronic device comprising: a memory for storing a computer program; a processor for implementing the steps of a distributed storage system failure recovery method according to the first aspect of the present invention when executing the computer program.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a distributed storage system failure recovery method according to the first aspect of the present invention.
The technical scheme adopted by the invention comprises the following technical effects:
1. according to the technical scheme, when the storage node is recovered from the fault, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG forwarding and temporary PG generation can be reduced, occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
2. According to the technical scheme, after the recovered object storage devices in the storage nodes to be monitored are subjected to state change uniformly, the number of the recovered object storage devices of the currently counted storage nodes to be monitored is emptied, so that the counting of the next preset time period is facilitated.
3. According to the technical scheme, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, whether the recovered number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number or not in the preset time period is judged, and timely adjustment is facilitated according to the recovery conditions of the object storage devices in the storage nodes to be monitored.
4. According to the technical scheme, when the storage node to be monitored is monitored to have new fault object storage equipment, the number of the fault object storage equipment in the storage node to be monitored is obtained again, the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, and timely adjustment is facilitated according to the fault condition of the object storage equipment in the storage node to be monitored.
5. According to the technical scheme, when the existence of new fault object storage equipment in the storage node to be monitored is monitored within a preset time period, the state of the recovered object storage equipment in the storage node to be monitored is not continuously waited, and the state of the recovered object storage equipment in the storage node to be monitored is immediately changed in a unified mode, so that the influence of the existence of the new fault object storage equipment in the storage node to be monitored on front-end services is avoided.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without any creative effort.
FIG. 1 is a schematic flow diagram of a process according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 3 is a schematic flow diagram of a process according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 5 is a schematic flow diagram of a process according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of an apparatus according to a second embodiment of the present invention;
fig. 7 is a schematic structural diagram of a third apparatus in an embodiment of the present invention.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
Example one
As shown in fig. 1, the present invention provides a method for recovering from a failure of a distributed storage system, including:
s101, monitoring whether a storage node has a fault object storage device;
s102, judging that the number of the failed object storage devices in the storage node to be monitored meets a first preset number requirement, and if the judgment result is yes, executing the step S103; if the judgment result is no, continuing to execute the step S102;
s103, marking the storage nodes to be monitored and the object storage equipment with faults in the storage nodes to be monitored;
s104, when the fault of the storage node to be monitored is recovered, acquiring the recovered number of the object storage devices in the storage node to be monitored;
s105, judging that the recovered number of the object storage devices in the storage node to be monitored in the preset time period meets a second preset number requirement; if the judgment result is yes, executing step S106, and if the judgment result is no, executing step S107;
s106, uniformly changing the state of the recovered object storage devices in the storage nodes to be monitored, and canceling fault marks of the recovered object storage devices in the storage nodes to be monitored and fault marks of the storage nodes to be monitored;
s107, after waiting for a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, canceling the fault marks of the recovered object storage devices in the storage node to be monitored, and updating the quantity of the unrecovered object storage devices in the storage node to be monitored.
In steps S101 to S102, the object storage device that monitors whether there is a failure in the storage node specifically is: monitoring whether fault information sent by object storage equipment in the storage nodes is received, and if fault information sent by one object storage equipment in the storage nodes to be monitored is received, the object storage equipment has faults. And counting the number of the failed object storage devices in the storage node to be monitored by counting the number of the object storage devices which receive the failure message sent by the node to be monitored. The first preset number requirement is specifically as follows: the ratio of the number of the failed object storage devices in the storage node to be monitored to the number of all the object storage devices in the storage node to be monitored is greater than a first preset ratio threshold. The first preset proportion threshold may be 70%, or may be flexibly adjusted according to the actual situation, and the present invention is not limited herein.
In step S103, if the number of failed object storage devices in the storage node to be monitored meets the first preset number requirement, marking the storage node to be monitored as a failed storage node, and marking the failed object storage device in the storage node to be monitored as a failed object storage device; otherwise, no fault marking is performed.
In step S104, after the failure of the storage node to be monitored is recovered, the specific step of acquiring the recovered number of the object storage devices in the storage node to be monitored is: after the fault of the storage node to be monitored is recovered, whether recovery information (boot message) sent by an object storage device in the storage node is received or not is monitored, and if a recovered message sent by one object storage device in the storage node to be monitored is received, the object storage device is recovered. And counting the number of the restored object storage devices in the storage node to be monitored by counting the number of the object storage devices which have sent the restoration message in the received node to be monitored.
In step S105, the second preset number requirement specifically includes: the ratio of the number of the recovered object storage devices in the storage node to be monitored to the number of the object storage devices marked with the faults in the storage node to be monitored is greater than a second preset ratio threshold. The second preset proportion threshold may be 75%, and may also be flexibly adjusted according to the actual situation, and the present invention is not limited herein.
In step S106, if the number of the object storage devices restored in the storage node to be monitored meets the second preset number requirement within the preset time period, the object storage devices restored in the storage node to be monitored are uniformly subjected to state change (OSDMonitor protocol), and the fault flag of the object storage device restored in the storage node to be monitored and the fault flag of the storage node to be monitored are cancelled.
S107, if the number of the restored object storage devices in the storage node to be monitored does not meet the second preset number requirement within the preset time period, after waiting for the preset time period, uniformly changing the state of the restored object storage devices in the storage node to be monitored, canceling the fault mark of the restored object storage devices in the storage node to be monitored, updating the number of the object storage devices which are not restored in the storage node to be monitored, but not canceling the fault mark of the storage node to be monitored until the number of the restored object storage devices in the storage node to be monitored meets the second preset number requirement.
When a large number of OSD faults occur to a storage node to be monitored in a certain time period, fault marking is carried out on the storage node after a set threshold value condition (a first preset quantity requirement) is met; after the OSD fault on the storage node is recovered, even if the OSD is not recovered at the same time, the state of the OSD is not changed immediately after the OSD state recovery message is received, but the number of the state recovery messages is counted first, so that the number of the recovered OSD is judged, and if the safety threshold condition evaluated in advance is met (the second preset number requirement, PG waiting and the temporary PG number caused by the state change are within a controllable range after a certain condition is met), the state is changed once (namely, OSDMonitor protocol). Otherwise, the method continues to wait for the sending of the rest OSD recovery messages. Meanwhile, giving a timeout time limit (preset time period) to the waiting process, after the timeout, not continuing to wait for the message, firstly carrying out a protocol, and updating the number of the OSD (on screen display) with the residual fault to be recovered. Waiting for the next state change.
Further, as shown in fig. 2, the technical solution of the present invention further provides a method for recovering from a failure of a distributed storage system, further comprising:
and S108, when it is monitored that the storage node to be monitored has new fault object storage equipment in a preset time period, no longer continuing to wait, and immediately carrying out unified state change on the recovered object storage equipment in the storage node to be monitored.
In step S108, if the OSD state is DOWN, the fault is not marked immediately, that is, the OSD state is DOWN, and the service allocated to the OSD by the front end is blocked or interrupted, which affects the front end service.
Preferably, as shown in fig. 3, the technical solution of the present invention further provides a distributed storage system fault recovery method, further including:
and S109, after the state of the recovered object storage devices in the storage node to be monitored is changed uniformly, emptying the number of the recovered object storage devices of the currently counted storage node to be monitored.
Further, as shown in fig. 4, the technical solution of the present invention further provides a method for recovering from a failure of a distributed storage system, further comprising:
s110, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, and whether the recovered number of the object storage devices in the storage nodes to be monitored meets a second preset number requirement or not is judged within a preset time period. I.e. steps S104-S107 are re-executed.
Further, as shown in fig. 5, the technical solution of the present invention further provides a method for recovering from a failure of a distributed storage system, further comprising:
and S111, when it is monitored that the storage node to be monitored has new fault object storage equipment, re-acquiring the number of the faults of the object storage equipment in the storage node to be monitored, and judging that the number of the fault object storage equipment in the storage node to be monitored meets a first preset number requirement. Namely, steps S101 (or S102) -S107 are re-executed.
It should be noted that the technical solution of the present invention can be implemented by hardware or software programming, and the implementation idea corresponds to the steps, and can also be implemented by other manners, which is not limited herein.
According to the technical scheme, when the storage node is recovered from the fault, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG forwarding and temporary PG generation can be reduced, occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the state of the recovered object storage devices in the storage nodes to be monitored is changed uniformly, the number of the recovered object storage devices of the storage nodes to be monitored, which are counted currently, is emptied, so that the counting of the next preset time period is facilitated.
According to the technical scheme, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, whether the recovered number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number or not in the preset time period is judged, and timely adjustment is facilitated according to the recovery conditions of the object storage devices in the storage nodes to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have new fault object storage equipment, the number of the fault object storage equipment in the storage node to be monitored is obtained again, the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, and timely adjustment is facilitated according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have the new storage device with the fault object in the preset time period, the storage node to be monitored does not continue to wait any more, and the restored storage devices in the storage node to be monitored are immediately subjected to unified state change, so that the influence of the storage device with the new fault object in the storage node to be monitored on the front-end service is avoided.
Example two
As shown in fig. 6, the technical solution of the present invention further provides a distributed storage system failure recovery apparatus, including:
the monitoring marking module 101 is used for monitoring whether the storage nodes have the failed object storage devices, and marking the storage nodes to be monitored and the failed object storage devices in the storage nodes to be monitored if the number of the failed object storage devices in the storage nodes to be monitored meets a first preset number requirement;
the first state changing module 102 is configured to, after the failure of the storage node to be monitored is recovered, obtain the recovered number of the object storage devices in the storage node to be monitored, and if the recovered number of the object storage devices in the storage node to be monitored meets a second preset number requirement within a preset time period, uniformly change the state of the recovered object storage devices in the storage node to be monitored, and cancel the failure flag of the recovered object storage devices in the storage node to be monitored and the failure flag of the storage node to be monitored;
and the second state changing module 103, if the number of the restored object storage devices in the storage node to be monitored does not meet the second preset number requirement, after waiting for a preset time period, uniformly changing the state of the restored object storage devices in the storage node to be monitored, canceling the fault flag of the restored object storage devices in the storage node to be monitored, and updating the number of the unrecovered object storage devices in the storage node to be monitored.
According to the technical scheme, when the storage node is recovered from the fault, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG forwarding and temporary PG generation can be reduced, occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the recovered object storage devices in the storage nodes to be monitored are subjected to state change uniformly, the number of the recovered object storage devices of the currently counted storage nodes to be monitored is emptied, so that the counting of the next preset time period is facilitated.
According to the technical scheme, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, whether the recovered number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number or not in the preset time period is judged, and timely adjustment is facilitated according to the recovery conditions of the object storage devices in the storage nodes to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have new fault object storage equipment, the number of the fault object storage equipment in the storage node to be monitored is obtained again, the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, and timely adjustment is facilitated according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have the new storage device with the fault object in the preset time period, the storage node to be monitored does not continue to wait any more, and the restored storage devices in the storage node to be monitored are immediately subjected to unified state change, so that the influence of the storage device with the new fault object in the storage node to be monitored on the front-end service is avoided.
EXAMPLE III
As shown in fig. 7, the present invention further provides an electronic device, including: a memory 201 for storing a computer program; the processor 202 is configured to implement the steps of the failure recovery method of the distributed storage system in the first embodiment when executing the computer program.
The memory 201 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device. It will be appreciated that the memory 201 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memory 201 described in embodiments herein is intended to comprise, without being limited to, these and any other suitable types of memory.
The method disclosed in the embodiments of the present application may be applied to the processor 202, or implemented by the processor 202. The processor 202 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 202. The processor 202 may be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing Digital Signal Processing), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. Processor 202 may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in the memory 201, and the processor 202 reads the program in the memory 201 and performs the steps of the foregoing method in combination with its hardware. When the processor 202 executes the program, the corresponding processes in the methods according to the embodiments of the present application are implemented, and for brevity, are not described herein again.
According to the technical scheme, when the storage node is recovered from the fault, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG forwarding and temporary PG generation can be reduced, occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the recovered object storage devices in the storage nodes to be monitored are subjected to state change uniformly, the number of the recovered object storage devices of the currently counted storage nodes to be monitored is emptied, so that the counting of the next preset time period is facilitated.
According to the technical scheme, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, whether the recovered number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number or not in the preset time period is judged, and timely adjustment is facilitated according to the recovery conditions of the object storage devices in the storage nodes to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have new fault object storage equipment, the number of the fault object storage equipment in the storage node to be monitored is obtained again, the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, and timely adjustment is facilitated according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have the new storage device with the fault object in the preset time period, the storage node to be monitored does not continue to wait any more, and the restored storage devices in the storage node to be monitored are immediately subjected to unified state change, so that the influence of the storage device with the new fault object in the storage node to be monitored on the front-end service is avoided.
Example four
The technical solution of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for recovering from a failure in a distributed storage system in the first embodiment are implemented.
For example, comprising a memory 201 storing a computer program executable by a processor 202 for performing the steps of the method as described above. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code. Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
According to the technical scheme, when the storage node is recovered from the fault, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG forwarding and temporary PG generation can be reduced, occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the recovered object storage devices in the storage nodes to be monitored are subjected to state change uniformly, the number of the recovered object storage devices of the currently counted storage nodes to be monitored is emptied, so that the counting of the next preset time period is facilitated.
According to the technical scheme, when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, whether the recovered number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number or not in the preset time period is judged, and timely adjustment is facilitated according to the recovery conditions of the object storage devices in the storage nodes to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have new fault object storage equipment, the number of the fault object storage equipment in the storage node to be monitored is obtained again, the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, and timely adjustment is facilitated according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the storage node to be monitored is monitored to have the new storage device with the fault object in the preset time period, the storage node to be monitored does not continue to wait any more, and the restored storage devices in the storage node to be monitored are immediately subjected to unified state change, so that the influence of the storage device with the new fault object in the storage node to be monitored on the front-end service is avoided.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.
Claims (10)
1. A distributed storage system fault recovery method is characterized by comprising the following steps:
monitoring whether a storage node has a fault object storage device, and marking the storage node to be monitored and the fault object storage device in the storage node to be monitored if the quantity of the fault object storage devices in the storage node to be monitored meets a first preset quantity requirement;
after the storage node to be monitored is recovered from a fault, acquiring the recovered number of the object storage devices in the storage node to be monitored, if the recovered number of the object storage devices in the storage node to be monitored meets a second preset number requirement in a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling a fault mark of the recovered object storage devices in the storage node to be monitored and a fault mark of the storage node to be monitored;
and if the recovered number of the object storage devices in the storage node to be monitored does not meet the second preset number requirement, after waiting for a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, canceling the fault mark of the recovered object storage devices in the storage node to be monitored, and updating the number of the unrecovered object storage devices in the storage node to be monitored.
2. The method according to claim 1, wherein the first preset number is required to be greater than a first preset ratio threshold value, where a ratio of the number of failed object storage devices in the storage node to be monitored to the number of all object storage devices in the storage node to be monitored is greater than the first preset ratio threshold value.
3. The method according to claim 1, wherein the second predetermined number is required to be greater than a second predetermined ratio threshold value, where a ratio of the number of restored object storage devices in the storage node to be monitored to the number of object storage devices marked with a fault in the storage node to be monitored is greater than the second predetermined ratio threshold value.
4. A method for failure recovery of a distributed storage system according to any one of claims 1 to 3, further comprising:
and after the state of the recovered object storage devices in the storage nodes to be monitored is uniformly changed, emptying the number of the recovered object storage devices of the currently counted storage nodes to be monitored.
5. The method of claim 4, further comprising:
when the object storage devices of the storage nodes to be monitored are monitored to be recovered again, the recovered number of the object storage devices in the storage nodes to be monitored is obtained again, and whether the recovered number of the object storage devices in the storage nodes to be monitored meets a second preset number requirement or not within a preset time is judged.
6. The method of claim 4, further comprising:
when it is monitored that new fault object storage equipment exists in the storage node to be monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement.
7. A method for failure recovery of a distributed storage system according to any one of claims 1 to 3, further comprising:
when monitoring that the storage node to be monitored has new fault object storage equipment in a preset time period, no longer continuing to wait, and immediately and uniformly changing the state of the recovered object storage equipment in the storage node to be monitored.
8. A distributed storage system fault recovery device, characterized by includes:
the monitoring marking module is used for monitoring whether the storage nodes have the failed object storage equipment or not, and marking the storage nodes to be monitored and the failed object storage equipment in the storage nodes to be monitored if the number of the failed object storage equipment in the storage nodes to be monitored meets a first preset number requirement;
the first state changing module is used for acquiring the number of the restored object storage devices in the storage node to be monitored after the storage node to be monitored is recovered from a fault, and if the number of the restored object storage devices in the storage node to be monitored meets the requirement of a second preset number within a preset time period, uniformly changing the state of the restored object storage devices in the storage node to be monitored, and canceling the fault mark of the restored object storage devices in the storage node to be monitored and the fault mark of the storage node to be monitored;
and the second state changing module is used for uniformly changing the states of the object storage devices recovered in the storage node to be monitored after waiting for a preset time period if the recovered number of the object storage devices in the storage node to be monitored does not meet the second preset number requirement, canceling the fault marks of the recovered object storage devices in the storage node to be monitored, and updating the number of the object storage devices which are not recovered in the storage node to be monitored.
9. An electronic device, comprising: a memory for storing a computer program; a processor for implementing the steps of a distributed storage system fault recovery method as claimed in any one of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of a method of fault recovery for a distributed storage system as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210087994.XA CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210087994.XA CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114443431A true CN114443431A (en) | 2022-05-06 |
| CN114443431B CN114443431B (en) | 2024-05-10 |
Family
ID=81369664
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210087994.XA Active CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443431B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN109213637A (en) * | 2018-11-09 | 2019-01-15 | 浪潮电子信息产业股份有限公司 | Data recovery method, device and medium for cluster nodes of distributed file system |
| CN110659148A (en) * | 2019-09-12 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Distributed cluster data recovery method, device, equipment and storage medium |
-
2022
- 2022-01-25 CN CN202210087994.XA patent/CN114443431B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN109213637A (en) * | 2018-11-09 | 2019-01-15 | 浪潮电子信息产业股份有限公司 | Data recovery method, device and medium for cluster nodes of distributed file system |
| CN110659148A (en) * | 2019-09-12 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Distributed cluster data recovery method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114443431B (en) | 2024-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2751551C1 (en) | Method and apparatus for restoring disrupted operating ability of a unit, electronic apparatus and data storage medium | |
| CN110535692B (en) | Fault processing method and device, computer equipment, storage medium and storage system | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| CN107817950B (en) | Data processing method and device | |
| CN110389858B (en) | Method and device for recovering faults of storage device | |
| CN106933843B (en) | Database heartbeat detection method and device | |
| CN112506710B (en) | Distributed file system data restoration method, device, equipment and storage medium | |
| JPH06290092A (en) | File storing device to storage device | |
| CN107729185B (en) | Fault processing method and device | |
| CN109491609B (en) | Cache data processing method, device and equipment and readable storage medium | |
| US20070011575A1 (en) | Autonomous method and apparatus for mitigating soft-errors in integrated circuit memory storage devices at run-time | |
| CN109656742A (en) | Node exception handling method and device and storage medium | |
| CN114625325B (en) | Distributed storage system and storage node offline processing method thereof | |
| CN109582459A (en) | The method and device that the trustship process of application is migrated | |
| US8621276B2 (en) | File system resiliency management | |
| CN110941591A (en) | File deletion method, device and equipment and readable storage medium | |
| CN114443431B (en) | Distributed storage system fault recovery method, device, equipment and medium | |
| CN116401122A (en) | Method, system, device and medium for processing node faults of distributed storage system | |
| US20070180312A1 (en) | Software duplication | |
| CN109032854B (en) | Election request processing method and device, management node and storage medium | |
| CN112596758B (en) | Version updating method, device, equipment and medium of OSDMap | |
| CN114003612A (en) | Processing method and processing system for abnormal conditions of database | |
| JP2009086758A (en) | Computer system and system management program | |
| US10007563B2 (en) | Event chain detection | |
| CN113162781B (en) | Service recovery method, device, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |