CN114443431B - Distributed storage system fault recovery method, device, equipment and medium - Google Patents
Distributed storage system fault recovery method, device, equipment and medium Download PDFInfo
- Publication number
- CN114443431B CN114443431B CN202210087994.XA CN202210087994A CN114443431B CN 114443431 B CN114443431 B CN 114443431B CN 202210087994 A CN202210087994 A CN 202210087994A CN 114443431 B CN114443431 B CN 114443431B
- Authority
- CN
- China
- Prior art keywords
- monitored
- storage
- object storage
- node
- storage devices
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000011084 recovery Methods 0.000 title claims abstract description 48
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012544 monitoring process Methods 0.000 claims description 11
- 230000001360 synchronised effect Effects 0.000 description 9
- 230000005291 magnetic effect Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a fault recovery method of a distributed storage system, which is characterized in that when a first preset number of requirements are met, storage nodes to be monitored and object storage equipment which has faults in the storage nodes to be monitored are marked; after the storage node to be monitored is recovered, if the second preset number of requirements are met in a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling the fault marks of the recovered object storage devices in the storage node to be monitored and the fault marks of the storage node to be monitored; otherwise, after waiting for a preset time period, uniformly changing the state of the restored object storage devices in the storage nodes to be monitored, and canceling the fault marks of the restored object storage devices in the storage nodes to be monitored.
Description
Technical Field
The present invention relates to the field of distributed system storage, and in particular, to a method, an apparatus, a device, and a medium for recovering a failure of a distributed storage system.
Background
In order to ensure the security of user data, a distributed storage system copies multiple copies (copies) of the same data and stores the copies on different storage nodes. In order to facilitate data distribution and recovery, a storage pool in a general storage system is divided into a plurality of PGs (PLACEMENT GROUP, i.e., a logical unit of data distribution, i.e., a homing group), where each PG includes a plurality of OSDs (Object Storage Device, object storage devices) according to a certain copy or erasure rule. For example, a triple copy storage pool is created, and a piece of data is stored in three parts and placed on three OSDs, respectively, where each PG contains three OSDs, and there is a piece of data on each OSD. The data recovery among the multiple copies is completed through PG, and the multiple copies of the data are synchronized according to the OSD states of the multiple copies. In a typical scenario, each OSD has 4 states: UP, DOWN, IN, OUT. The UP and IN states typically occur simultaneously, representing that the OSD is normally providing read and write services. The DOWN state identifies that the OSD fails and cannot provide service, when there is an OSD in the DOWN state between copies in the PG, all objects written in the PG are data-downgraded written, i.e., few copies are written (one OSD in the three copies of the scene, the data is written only two copies), so the DOWN is a temporary state, when a certain time (e.g., 10 min) has elapsed, the OSD cannot provide service yet, the distributed cluster marks the OSD state as OUT, i.e., kicks OUT the PG member, at this time, the PG reselects an OSD as a member thereof on the remaining OSD conforming to the failure domain (peering process, i.e., kicks OUT the OSD marked as OUT state in the PG), reselects an OSD as a member thereof on the remaining OSD conforming to the failure domain in the PG), and synchronizes all data maintained by the PG to a new one, i.e., data recovery process.
In the above process, the peering process of PG and the data recovery process are generated due to the need of calculation and a large number of temporary PGs (the members belonging to the PG may generate a corresponding PG for receiving the read-write service in the data recovery process, and then delete after the data recovery is completed), which can consume the system CPU and the memory resources. When a large number of OSDs on a certain storage node are in a DOWN state, the occupation of system resources of the storage node is very serious, even if memory is used in a swap partition (release is very slow), so that other service processes such as OSDs, MON (Monitor daemon) on the node are slow to process, and serious consequences such as service waiting, interruption, cluster abnormality and the like are caused. Therefore, it is desirable to avoid a large number of OSDs in the storage node being in the DOWN state, i.e., temporary state.
In the prior art, in order to prevent unnecessary data reconstruction of a cluster, when a node fails (network failure or power failure), after all OSDs of the corresponding node are marked as DOWN, the time of OUT is set to one hour (the node fails and does not kick a disk for a short time), during which no data reconstruction will occur because OSDs will not OUT and PGs will not elect. However, after the fault is recovered, because the OSD is not started at the same time, when a certain OSD of the node is started and marked as UP (i.e. the OSD is recovered), the strategy of not kicking the disk when the node fails before is not satisfied, at this time, the remaining OSDs which are not UP (recovered) on the node will become OUT, so that almost all PGs on the node are in peering states, a large amount of data recovery is generated, and a large amount of temporary PGs are generated at the same time, which further aggravates the above-mentioned system resource occupation situation and is unfavorable for the stability of the distributed storage cluster.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a fault recovery method of a distributed storage system, which reduces the frequency of OSD state change in a storage node as much as possible when the storage node is in fault recovery, thereby reducing PG PEERING and temporary PG, greatly reducing the occupation of system resources and improving the stability of a cluster.
The first aspect of the present invention provides a distributed storage system fault recovery method, including:
monitoring whether a faulty object storage device exists in the storage node, and if the number of the faulty object storage devices in the storage node to be monitored meets the first preset number requirement, marking the storage node to be monitored and the faulty object storage devices in the storage node to be monitored;
When the storage node to be monitored is recovered from faults, acquiring the recovered quantity of the object storage devices in the storage node to be monitored, and if the recovered quantity of the object storage devices in the storage node to be monitored meets the requirement of a second preset quantity in a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling the fault marks of the recovered object storage devices in the storage node to be monitored and the fault marks of the storage node to be monitored;
And if the restored number of the object storage devices in the storage nodes to be monitored does not meet the second preset number requirement, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored after waiting for a preset time period, canceling fault marks of the restored object storage devices in the storage nodes to be monitored, and updating the unrecovered number of the object storage devices in the storage nodes to be monitored.
Optionally, the first preset number is required that a ratio of the number of failed object storage devices in the storage node to be monitored to the number of all object storage devices in the storage node to be monitored is greater than a first preset ratio threshold.
Optionally, the second preset number is required that a ratio of the number of recovered object storage devices in the storage node to be monitored to the number of object storage devices marked with the failure in the storage node to be monitored is greater than a second preset ratio threshold.
Optionally, the method further comprises:
And after unified state change is carried out on the restored object storage devices in the storage nodes to be monitored, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared.
Further, the method further comprises the following steps:
And when the object storage equipment of the storage node to be monitored is monitored again to be recovered, the recovered quantity of the object storage equipment in the storage node to be monitored is obtained again, and whether the recovered quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged in a preset time period.
Optionally, the method further comprises:
When the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the object storage equipment in the storage node to be monitored, which is faulty, is obtained again, and the number of the object storage equipment in the storage node to be monitored, which is faulty, is judged to meet the first preset number requirement.
Optionally, the method further comprises:
And when the storage node to be monitored is monitored to have new fault object storage equipment in the preset time period, the waiting is not continued, and the restored object storage equipment in the storage node to be monitored is uniformly changed in state.
A second aspect of the present invention provides a distributed storage system failure recovery apparatus, including:
The monitoring marking module is used for monitoring whether the object storage equipment with faults exists in the storage nodes, and marking the storage nodes to be monitored and the object storage equipment with faults in the storage nodes to be monitored if the number of the object storage equipment with faults in the storage nodes to be monitored meets the first preset number requirement;
the first state change module is used for obtaining the restored number of the object storage devices in the storage nodes to be monitored after the storage nodes to be monitored are restored in faults, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored and canceling fault marks of the restored object storage devices in the storage nodes to be monitored and fault marks of the storage nodes to be monitored if the restored number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number in a preset time period;
and the second state change module is used for uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored after waiting for a preset time period if the restored number of the object storage devices in the storage nodes to be monitored does not meet the second preset number requirement, canceling the fault marks of the restored object storage devices in the storage nodes to be monitored and updating the number of the unrecovered object storage devices in the storage nodes to be monitored.
A third aspect of the present invention provides an electronic device, comprising: a memory for storing a computer program; a processor for implementing the steps of a distributed storage system failure recovery method according to the first aspect of the present invention when executing the computer program.
A fourth aspect of the present invention provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the steps of a distributed storage system failure recovery method according to the first aspect of the present invention.
The technical scheme adopted by the invention comprises the following technical effects:
1. According to the technical scheme, when the storage node is recovered from faults, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG PEERING and temporary PG generation can be reduced, the occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
2. According to the technical scheme, after the restored object storage devices in the storage nodes to be monitored are subjected to state change in a unified manner, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared, and the statistics of the next preset time period is facilitated.
3. According to the technical scheme, when the object storage equipment of the storage node to be monitored is monitored again to recover, the recovery quantity of the object storage equipment in the storage node to be monitored is obtained again, whether the recovery quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged, and timely adjustment is facilitated according to the recovery condition of the object storage equipment in the storage node to be monitored.
4. According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, so that timely adjustment is conducted according to the fault condition of the object storage equipment in the storage node to be monitored.
5. According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored within the preset time period, waiting is not continued, and state change is uniformly performed on the restored object storage equipment in the storage node to be monitored immediately, so that the influence of the new fault object storage equipment on front-end business of the storage node to be monitored is avoided.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed.
Drawings
For a clearer description of embodiments of the invention or of the solutions of the prior art, reference will be made to the accompanying drawings, which are used in the description of the embodiments or of the prior art, and it will be obvious to those skilled in the art that other drawings can be obtained from these without inventive labour.
FIG. 1 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart (III) of a method according to an embodiment of the present invention;
FIG. 4 is a flow chart of a method according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart (V) of a method according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a second embodiment of the present invention;
Fig. 7 is a schematic structural diagram of a third device according to an embodiment of the present invention.
Detailed Description
In order to clearly illustrate the technical features of the present solution, the present invention will be described in detail below with reference to the following detailed description and the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different structures of the invention. In order to simplify the present disclosure, components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and processes are omitted so as to not unnecessarily obscure the present invention.
Example 1
As shown in fig. 1, the present invention provides a distributed storage system fault recovery method, which includes:
S101, monitoring whether a faulty object storage device exists in a storage node;
S102, judging that the number of the failed object storage devices in the storage node to be monitored meets a first preset number requirement, and if the judgment result is yes, executing a step S103; if not, continuing to execute the step S102;
s103, marking the storage nodes to be monitored and the object storage equipment which has faults in the storage nodes to be monitored;
S104, after the storage node to be monitored is recovered from faults, obtaining the recovered quantity of the object storage devices in the storage node to be monitored;
S105, judging that the restored quantity of the object storage devices in the storage nodes to be monitored meets the second preset quantity requirement in a preset time period; if the judgment result is yes, executing step S106, and if the judgment result is no, executing step S107;
s106, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored, and canceling fault marks of the restored object storage devices in the storage nodes to be monitored and fault marks of the storage nodes to be monitored;
and S107, after waiting for a preset time period, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored, canceling the fault marks of the restored object storage devices in the storage nodes to be monitored, and updating the number of the unrecovered object storage devices in the storage nodes to be monitored.
In steps S101-S102, the object storage device that monitors whether there is a failure in the storage node is specifically: and monitoring whether fault information sent by the object storage equipment in the storage node is received, and if the fault information sent by one of the object storage equipment in the storage node to be monitored is received, determining that the object storage equipment has faults. The number of the object storage devices which have faults in the storage nodes to be monitored is counted by counting the number of the object storage devices which receive the fault messages which have sent in the nodes to be monitored. The first preset number of requirements is specifically: the ratio of the number of failed object storage devices in the storage node to be monitored to the number of all object storage devices in the storage node to be monitored is greater than a first preset ratio threshold. The first preset proportional threshold may be 70%, or may be flexibly adjusted according to practical situations, which is not limited herein.
In step S103, if the number of failed object storage devices in the storage nodes to be monitored meets the first preset number requirement, marking the storage nodes to be monitored as failed storage nodes, and marking the failed object storage devices in the storage nodes to be monitored as failed object storage devices; otherwise, no fault marking is performed.
In step S104, after the failure recovery of the storage node to be monitored, the number of recovered object storage devices in the storage node to be monitored is specifically: and after the storage node to be monitored is recovered due to faults, monitoring whether recovery information (boot information) sent by the object storage equipment in the storage node is received, and if the recovery information sent by one object storage equipment in the storage node to be monitored is received, recovering the object storage equipment. The number of the restored object storage devices in the node to be monitored is counted by counting the number of the object storage devices which receive the restoration message sent in the node to be monitored.
In step S105, the second preset number of requirements is specifically: the ratio of the number of restored object storage devices in the storage node to be monitored to the number of object storage devices marked with faults in the storage node to be monitored is greater than a second preset ratio threshold. The second preset proportional threshold may be 75%, or may be flexibly adjusted according to practical situations, which is not limited in this disclosure.
In step S106, if the number of restored object storage devices in the storage node to be monitored meets the second preset number requirement within the preset time period, the state of the restored object storage devices in the storage node to be monitored is uniformly changed (OSDMonitor protocol), and the fault marks of the restored object storage devices in the storage node to be monitored and the fault marks of the storage node to be monitored are cancelled.
S107, if the number of restored object storage devices in the storage nodes to be monitored does not meet the second preset number requirement in the preset time period, after waiting for the preset time period, uniformly changing the state of the restored object storage devices in the storage nodes to be monitored, canceling the fault marks of the restored object storage devices in the storage nodes to be monitored, updating the number of unrecovered object storage devices in the storage nodes to be monitored, and not canceling the fault marks of the storage nodes to be monitored until the number of restored object storage devices in the storage nodes to be monitored meets the second preset number requirement.
When a large number of OSD failures occur in a storage node to be monitored within a certain time period, after a set threshold condition (a first preset number requirement) is reached, the storage node is subjected to failure marking; after the OSD fault on the storage node is recovered, even if the OSD is not recovered at the same time, the state of the OSD is not changed immediately after the OSD state recovery message is received, and the number of messages recovered by the state is counted first, so as to judge the number of the recovered OSDs, if the safety threshold condition (the second preset number requirement is met, and the number of PG PEERING and temporary PGs caused by the state change after reaching a certain condition is within a controllable range) is met, the state change is uniformly performed once (namely, OSDMonitor proposal). Otherwise, continuing to wait for the other OSD restoration messages to be sent. Meanwhile, a timeout time limit (preset time period) is given to the waiting process, after timeout, the message is not continued to wait, the proposal is firstly conducted once, and the number of the OSD to be recovered from the residual faults is updated. Waiting for the next state change.
Further, as shown in fig. 2, the technical solution of the present invention further provides a distributed storage system fault recovery method, which further includes:
and S108, when the fact that the storage node to be monitored has new fault object storage equipment is monitored in the preset time period, the storage node does not continue to wait, and the restored object storage equipment in the storage node to be monitored is subjected to state change uniformly.
In step S108, if the OSD status is DOWN, the fault flag is not immediately set, that is, the OSD is marked DOWN, and at this time, the front-end service is affected by the blocking or interruption of the OSD service allocated to the front-end.
Preferably, as shown in fig. 3, the technical solution of the present invention further provides a distributed storage system fault recovery method, which further includes:
S109, after the restored object storage devices in the storage nodes to be monitored are subjected to state change uniformly, the number of the restored object storage devices of the storage nodes to be monitored, which are counted currently, is cleared.
Further, as shown in fig. 4, the technical solution of the present invention further provides a distributed storage system fault recovery method, which further includes:
s110, when the object storage equipment of the storage node to be monitored is monitored again to be recovered, the recovered quantity of the object storage equipment in the storage node to be monitored is obtained again, and whether the recovered quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged in a preset time period. I.e. re-executing steps S104-S107.
Further, as shown in fig. 5, the technical solution of the present invention further provides a distributed storage system fault recovery method, which further includes:
S111, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement. I.e. re-executing steps S101 (or S102) -S107.
It should be noted that, the technical solution of the present invention may be implemented by hardware or software programming, where the implementing thought corresponds to the steps, and may also be implemented by other modes, where the present invention is not limited herein.
According to the technical scheme, when the storage node is recovered from faults, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG PEERING and temporary PG generation can be reduced, the occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the restored object storage devices in the storage nodes to be monitored are subjected to state change in a unified manner, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared, and the statistics of the next preset time period is facilitated.
According to the technical scheme, when the object storage equipment of the storage node to be monitored is monitored again to recover, the recovery quantity of the object storage equipment in the storage node to be monitored is obtained again, whether the recovery quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged, and timely adjustment is facilitated according to the recovery condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, so that timely adjustment is conducted according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored within the preset time period, waiting is not continued, and state change is uniformly performed on the restored object storage equipment in the storage node to be monitored immediately, so that the influence of the new fault object storage equipment on front-end business of the storage node to be monitored is avoided.
Example two
As shown in fig. 6, the technical solution of the present invention further provides a distributed storage system fault recovery device, including:
The monitoring marking module 101 is used for monitoring whether the object storage devices with faults exist in the storage nodes, and marking the storage nodes to be monitored and the object storage devices with faults in the storage nodes to be monitored if the number of the object storage devices with faults in the storage nodes to be monitored meets a first preset number requirement;
the first state change module 102 is used for obtaining the recovered number of the object storage devices in the storage nodes to be monitored after the storage nodes to be monitored are recovered from faults, and uniformly carrying out state change on the recovered object storage devices in the storage nodes to be monitored and canceling the fault marks of the recovered object storage devices in the storage nodes to be monitored and the fault marks of the storage nodes to be monitored if the recovered number of the object storage devices in the storage nodes to be monitored meets the second preset number requirement in a preset time period;
And the second state change module 103 uniformly changes the state of the restored object storage devices in the storage nodes to be monitored after waiting for a preset time period if the restored number of the object storage devices in the storage nodes to be monitored does not meet the second preset number requirement, cancels the fault marks of the restored object storage devices in the storage nodes to be monitored, and updates the number of the unrecovered object storage devices in the storage nodes to be monitored.
According to the technical scheme, when the storage node is recovered from faults, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG PEERING and temporary PG generation can be reduced, the occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the restored object storage devices in the storage nodes to be monitored are subjected to state change in a unified manner, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared, and the statistics of the next preset time period is facilitated.
According to the technical scheme, when the object storage equipment of the storage node to be monitored is monitored again to recover, the recovery quantity of the object storage equipment in the storage node to be monitored is obtained again, whether the recovery quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged, and timely adjustment is facilitated according to the recovery condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, so that timely adjustment is conducted according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored within the preset time period, waiting is not continued, and state change is uniformly performed on the restored object storage equipment in the storage node to be monitored immediately, so that the influence of the new fault object storage equipment on front-end business of the storage node to be monitored is avoided.
Example III
As shown in fig. 7, the technical solution of the present invention further provides an electronic device, including: a memory 201 for storing a computer program; a processor 202 is configured to implement the steps of a distributed storage system failure recovery method as in the first embodiment when executing the computer program.
The memory 201 in embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device. It will be appreciated that memory 201 may be volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. Wherein the nonvolatile Memory may be Read Only Memory (ROM), programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk-Only Memory (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random AccessMemory) which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronousDynamic Random Access Memory), double data rate synchronous dynamic random access memory (ddr sdram, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory 201 described in embodiments of the present application is intended to comprise, without being limited to, these and any other suitable types of memory.
The method disclosed in the above embodiment of the present application may be applied to the processor 202 or implemented by the processor 202. The processor 202 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuitry in hardware in the processor 202 or by instructions in software. The processor 202 may be a general purpose processor, a DSP (DIGITAL SIGNAL Processing, i.e., a chip capable of implementing digital signal Processing techniques), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. Processor 202 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the application can be directly embodied in the hardware of the decoding processor or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium in the memory 201 and the processor 202 reads the program in the memory 201 to perform the steps of the method described above in connection with its hardware. The corresponding flow in each method of the embodiments of the present application is implemented when the processor 202 executes the program, and for brevity, will not be described in detail herein.
According to the technical scheme, when the storage node is recovered from faults, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG PEERING and temporary PG generation can be reduced, the occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the restored object storage devices in the storage nodes to be monitored are subjected to state change in a unified manner, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared, and the statistics of the next preset time period is facilitated.
According to the technical scheme, when the object storage equipment of the storage node to be monitored is monitored again to recover, the recovery quantity of the object storage equipment in the storage node to be monitored is obtained again, whether the recovery quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged, and timely adjustment is facilitated according to the recovery condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, so that timely adjustment is conducted according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored within the preset time period, waiting is not continued, and state change is uniformly performed on the restored object storage equipment in the storage node to be monitored immediately, so that the influence of the new fault object storage equipment on front-end business of the storage node to be monitored is avoided.
Example IV
The technical solution of the present invention further provides a computer readable storage medium, where a computer program is stored, where the computer program, when executed by a processor, implements the steps of a distributed storage system failure recovery method as in the first embodiment.
For example comprising a memory 201 storing a computer program executable by a processor 202 to perform the steps of the method described above. The computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code. Or the above-described integrated units of the application may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
According to the technical scheme, when the storage node is recovered from faults, the frequency of OSD state change in the storage node is reduced as much as possible, so that PG PEERING and temporary PG generation can be reduced, the occupation of system resources is greatly reduced, and the stability of the distributed storage cluster is improved.
According to the technical scheme, after the restored object storage devices in the storage nodes to be monitored are subjected to state change in a unified manner, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared, and the statistics of the next preset time period is facilitated.
According to the technical scheme, when the object storage equipment of the storage node to be monitored is monitored again to recover, the recovery quantity of the object storage equipment in the storage node to be monitored is obtained again, whether the recovery quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged, and timely adjustment is facilitated according to the recovery condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the fault object storage equipment in the storage node to be monitored is obtained again, and the number of the fault object storage equipment in the storage node to be monitored is judged to meet the first preset number requirement, so that timely adjustment is conducted according to the fault condition of the object storage equipment in the storage node to be monitored.
According to the technical scheme, when the fact that the storage node to be monitored has new fault object storage equipment is monitored within the preset time period, waiting is not continued, and state change is uniformly performed on the restored object storage equipment in the storage node to be monitored immediately, so that the influence of the new fault object storage equipment on front-end business of the storage node to be monitored is avoided.
While the foregoing description of the embodiments of the present invention has been presented in conjunction with the drawings, it should be understood that it is not intended to limit the scope of the invention, but rather, it is intended to cover all modifications or variations within the scope of the invention as defined by the claims of the present invention.
Claims (10)
1. A method for recovering a failure of a distributed storage system, comprising:
monitoring whether a faulty object storage device exists in the storage node, and if the number of the faulty object storage devices in the storage node to be monitored meets the first preset number requirement, marking the storage node to be monitored and the faulty object storage devices in the storage node to be monitored;
When the storage node to be monitored is recovered from faults, acquiring the recovered quantity of the object storage devices in the storage node to be monitored, and if the recovered quantity of the object storage devices in the storage node to be monitored meets the requirement of a second preset quantity in a preset time period, uniformly changing the state of the recovered object storage devices in the storage node to be monitored, and canceling the fault marks of the recovered object storage devices in the storage node to be monitored and the fault marks of the storage node to be monitored;
And if the restored number of the object storage devices in the storage nodes to be monitored does not meet the second preset number requirement, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored after waiting for a preset time period, canceling fault marks of the restored object storage devices in the storage nodes to be monitored, and updating the unrecovered number of the object storage devices in the storage nodes to be monitored.
2. The method of claim 1, wherein the first predetermined number of object storage devices is greater than a first predetermined proportion threshold for the number of failed object storage devices in the storage node to be monitored to be greater than the number of all object storage devices in the storage node to be monitored.
3. The method of claim 1, wherein the second predetermined number of requirements is that a ratio of a number of recovered object storage devices in the storage node to be monitored to a number of object storage devices marked for failure in the storage node to be monitored is greater than a second predetermined ratio threshold.
4. A distributed storage system failure recovery method according to any of claims 1-3, further comprising:
And after unified state change is carried out on the restored object storage devices in the storage nodes to be monitored, the number of the restored object storage devices of the storage nodes to be monitored, which are counted at present, is cleared.
5. The method for recovering from a failure of a distributed storage system of claim 4, further comprising:
and when the object storage equipment of the storage node to be monitored is monitored again to be recovered, the recovered quantity of the object storage equipment in the storage node to be monitored is obtained again, and whether the recovered quantity of the object storage equipment in the storage node to be monitored meets the second preset quantity requirement or not is judged in the preset time.
6. The method for recovering from a failure of a distributed storage system of claim 4, further comprising:
When the fact that the storage node to be monitored has new fault object storage equipment is monitored, the number of the object storage equipment in the storage node to be monitored, which is faulty, is obtained again, and the number of the object storage equipment in the storage node to be monitored, which is faulty, is judged to meet the first preset number requirement.
7. A distributed storage system failure recovery method according to any of claims 1-3, further comprising:
And when the storage node to be monitored is monitored to have new fault object storage equipment in the preset time period, the waiting is not continued, and the restored object storage equipment in the storage node to be monitored is uniformly changed in state.
8. A distributed storage system failure recovery apparatus, comprising:
The monitoring marking module is used for monitoring whether the object storage equipment with faults exists in the storage nodes, and marking the storage nodes to be monitored and the object storage equipment with faults in the storage nodes to be monitored if the number of the object storage equipment with faults in the storage nodes to be monitored meets the first preset number requirement;
the first state change module is used for obtaining the restored number of the object storage devices in the storage nodes to be monitored after the storage nodes to be monitored are restored in faults, uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored and canceling fault marks of the restored object storage devices in the storage nodes to be monitored and fault marks of the storage nodes to be monitored if the restored number of the object storage devices in the storage nodes to be monitored meets the requirement of the second preset number in a preset time period;
and the second state change module is used for uniformly carrying out state change on the restored object storage devices in the storage nodes to be monitored after waiting for a preset time period if the restored number of the object storage devices in the storage nodes to be monitored does not meet the second preset number requirement, canceling the fault marks of the restored object storage devices in the storage nodes to be monitored and updating the number of the unrecovered object storage devices in the storage nodes to be monitored.
9. An electronic device, comprising: a memory for storing a computer program; a processor for implementing the steps of a distributed storage system failure recovery method according to any of claims 1 to 7 when executing said computer program.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of a distributed storage system failure recovery method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210087994.XA CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210087994.XA CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114443431A CN114443431A (en) | 2022-05-06 |
| CN114443431B true CN114443431B (en) | 2024-05-10 |
Family
ID=81369664
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210087994.XA Active CN114443431B (en) | 2022-01-25 | 2022-01-25 | Distributed storage system fault recovery method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114443431B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN109213637A (en) * | 2018-11-09 | 2019-01-15 | 浪潮电子信息产业股份有限公司 | Data reconstruction method, device and the medium of distributed file system clustered node |
| CN110659148A (en) * | 2019-09-12 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Distributed cluster data recovery method, device, equipment and storage medium |
-
2022
- 2022-01-25 CN CN202210087994.XA patent/CN114443431B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109144787A (en) * | 2018-09-03 | 2019-01-04 | 郑州云海信息技术有限公司 | A kind of data reconstruction method, device, equipment and readable storage medium storing program for executing |
| CN109213637A (en) * | 2018-11-09 | 2019-01-15 | 浪潮电子信息产业股份有限公司 | Data reconstruction method, device and the medium of distributed file system clustered node |
| CN110659148A (en) * | 2019-09-12 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Distributed cluster data recovery method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114443431A (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110300984B (en) | Changing smart contracts recorded in a blockchain | |
| US10509585B2 (en) | Data synchronization method, apparatus, and system | |
| US10884623B2 (en) | Method and apparatus for upgrading a distributed storage system | |
| CN107817950B (en) | Data processing method and device | |
| CN111475340A (en) | Method, apparatus and computer program product for creating a replica | |
| CN110389858B (en) | Method and device for recovering faults of storage device | |
| US9355005B2 (en) | Detection apparatus and detection method | |
| WO2021208184A1 (en) | Method and system for calling-in and recovery of node traffic and central server | |
| EP1550036A2 (en) | Method of solving a split-brain condition | |
| US20140089732A1 (en) | Thread sparing between cores in a multi-threaded processor | |
| CN112506710B (en) | Distributed file system data restoration method, device, equipment and storage medium | |
| CN107729185B (en) | Fault processing method and device | |
| CN104424186A (en) | Method and device for realizing persistence in flow calculation application | |
| CN112468361A (en) | Network connection state monitoring method and device, electronic equipment and storage medium | |
| US20160277152A1 (en) | Method and system for robust message retransmission | |
| CN113625945A (en) | Distributed storage slow disk processing method, system, terminal and storage medium | |
| CN110764953A (en) | Data recovery method, device and equipment and readable storage medium | |
| CN111880956A (en) | Data synchronization method and device | |
| US20180129668A1 (en) | Intermediate file processing method, client, server, and system | |
| US8719622B2 (en) | Recording and preventing crash in an appliance | |
| CN114443431B (en) | Distributed storage system fault recovery method, device, equipment and medium | |
| CN109117263A (en) | A kind of method and device of user terminal memory release | |
| CN115629901A (en) | Log playback method and device, data recovery method and device and electronic equipment | |
| CN110399098B (en) | Data processing method and electronic equipment | |
| CN114003612A (en) | Processing method and processing system for abnormal conditions of database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |