CN116974808A - Fault recovery method and device during copy synchronization, electronic equipment and medium - Google Patents
Fault recovery method and device during copy synchronization, electronic equipment and medium Download PDFInfo
- Publication number
- CN116974808A CN116974808A CN202311095872.6A CN202311095872A CN116974808A CN 116974808 A CN116974808 A CN 116974808A CN 202311095872 A CN202311095872 A CN 202311095872A CN 116974808 A CN116974808 A CN 116974808A
- Authority
- CN
- China
- Prior art keywords
- peering
- delay
- storage system
- reaches
- abnormal state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 96
- 238000011084 recovery Methods 0.000 title claims abstract description 36
- 230000002159 abnormal effect Effects 0.000 claims abstract description 88
- 238000012545 processing Methods 0.000 claims abstract description 82
- 230000005540 biological transmission Effects 0.000 claims description 58
- 241000607479 Yersinia pestis Species 0.000 claims description 31
- 238000013500 data storage Methods 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 3
- 238000007689 inspection Methods 0.000 claims description 3
- 230000000737 periodic effect Effects 0.000 claims description 3
- 230000007547 defect Effects 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 32
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000005856 abnormality Effects 0.000 description 5
- 235000012174 carbonated soft drink Nutrition 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 230000032683 aging Effects 0.000 description 4
- 238000005457 optimization Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000003213 activating effect Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0727—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a storage system, e.g. in a DASD or network based storage system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The application discloses a fault recovery method, a fault recovery device, electronic equipment and a medium during copy synchronization. By applying the technical scheme of the application, after the existence of the Peer i ng flow in the storage system is detected, whether the PG reaches an abnormal state or not can be determined pertinently by comparing the time delay state index generated in the period of the PG with the standard time delay index, which is obtained in real time or periodically, so that the quick recovery of the PG can be realized by restarting the Peer i ng when the PG is determined to be abnormal subsequently. Therefore, the technical scheme that the failure of the Peer i ng can be automatically solved to achieve the purpose of automatically recovering the service state of the PE as soon as possible without manual intervention is realized, and the failure self-processing capacity of the distributed storage system is further improved. The defect that the recovery time of the storage system is slower because the mode of restarting the network manually can only be adopted to recover when the storage system has the Peer i ng fault is avoided.
Description
Technical Field
The present application relates to data processing technologies, and in particular, to a method, an apparatus, an electronic device, and a medium for recovering a failure during synchronization of copies.
Background
When the distributed storage system faces the system initialization or the newly built placement group (PG placement_group), the copy synchronization process for PG is triggered.
Further, the PG engineering process is a process of bringing each cache pool data storage device (CSD cache storage device) in one PG to a data storage consistent state. It can be understood that when the master-slave CSD of the PG reaches a state where data storage is consistent, the PG is in a service complete state, so as to determine that the scheduling process is finished.
However, in the PG engineering process of the distributed storage system, an engineering failure event often occurs due to poor network status and the like. In the related art, once a failure event of the peer occurs, the service personnel can only restart the network manually to achieve the purpose of failure recovery. However, this approach is inefficient, resulting in a slow recovery time for the storage system.
Disclosure of Invention
The embodiment of the application provides a fault recovery method, a device, electronic equipment and a medium during copy synchronization. Thereby solving the problem of how to reduce the recovery time of the pest failure event in the related technology.
According to an aspect of the embodiment of the present application, a method for recovering a failure during synchronization of copies is provided, which is applied to a distributed storage system including a placement group PG, and the method includes:
when detecting that a target PG in the distributed storage system is located in a replica synchronization period, acquiring a time delay state index for reflecting the target PG in the period;
determining whether the pecking reaches an abnormal state or not based on the time delay state index;
if the Peering reaches the abnormal state, stopping the Peering currently in progress, and restarting the Peering flow.
Optionally, in another embodiment of the above method according to the present application, the determining, based on the delay status indicator, whether the peer reaches an abnormal status includes:
determining whether the peer reaches an abnormal state based on a transmission delay index contained in the delay state index, wherein the transmission delay index is used for reflecting the delay degree of message transmission of the target PG during the peer; the method comprises the steps of,
determining whether the peer reaches an abnormal state based on a processing delay index contained in the delay state index, wherein the processing delay index is used for reflecting the delay degree of message processing of the target PG during the peer; the method comprises the steps of,
And determining whether the pecking reaches an abnormal state or not based on a continuous time delay index contained in the time delay state index, wherein the continuous time delay index is used for reflecting the duration of the pecking.
Optionally, in another embodiment of the above method according to the present application, the determining, based on a transmission delay indicator included in the delay status indicator, whether the peer reaches an abnormal status includes:
acquiring all the Peering messages generated by the target PG in the Peering period, and counting the transmission time of each Peering message in a network layer in the distributed storage system;
and if detecting that at least one Peering message with the transmission duration not meeting the first delay standard exists, determining that the Peering reaches an abnormal state.
Optionally, in another embodiment of the above method according to the present application, the counting the transmission duration of each of the peer messages at the network layer in the distributed storage system includes:
recording a receiving time stamp of each of the Peering messages through a message module in the distributed storage system; and, retrieving the transmission time stamp recorded by the corresponding buffer pool data storage device CSD in each of the ping messages;
And determining the transmission duration corresponding to each of the PERING messages based on the difference between the receiving time stamp and the sending time stamp of each of the PERING messages.
Optionally, in another embodiment of the above method according to the present application, the determining, based on a processing delay indicator included in the delay status indicator, whether the peer reaches an abnormal status includes:
acquiring all the pest events generated by the target PG during the pest period, and counting the processing time of each pest event in the distributed storage system;
and if detecting that at least one Peering message with the processing time length not meeting the second time delay standard exists, determining that the Peering reaches an abnormal state.
Optionally, in another embodiment of the above method according to the present application, the counting the processing duration of each of the events in the distributed storage system includes:
acquiring message processing queues corresponding to each CSD contained in the target PG, wherein each message processing queue comprises at least one Peering event responsible for the corresponding CSD, and each Peering event is marked with an enqueuing time point added into the message processing queue;
And determining the processing duration corresponding to each of the events based on the difference between the current time point and the enqueuing time point corresponding to each of the events.
Optionally, in another embodiment of the above method according to the present application, the determining whether the pecing reaches an abnormal state based on a duration delay indicator included in the delay state indicator includes:
acquiring the duration of the early;
and if the duration is detected not to meet the third time delay standard, determining that the Peering reaches an abnormal state.
Optionally, in another embodiment of the above method according to the present application, the acquiring the duration of the early includes:
the duration of the pen is obtained by periodic inspection of the pen by the primary CSD associated with the target PG.
Optionally, in another embodiment of the above method according to the present application, after the detecting that the target PG in the distributed storage system is located during copy synchronization period, the method further includes:
detecting that the duration of the early exceeds a preset duration; and/or detecting that the current network state of the distributed storage system exceeds a preset congestion threshold;
The delay state index used for reflecting the target PG during the period of the Peering is acquired.
Optionally, in another embodiment of the above method according to the present application, the determining that the peer reaches an abnormal state, stopping the peer currently in progress, and restarting the peer process includes:
determining that the early reaches an abnormal state;
and restarting the pest flow by updating the data osdmap of the object storage device currently set by the distributed storage system.
In accordance with another aspect of the embodiments of the present application, there is provided a failure recovery apparatus during copy synchronization, applied to a distributed storage system including a placement group PG, the apparatus comprising:
the detection module is configured to acquire a time delay state index for reflecting the target PG in the replica synchronization period when the target PG in the distributed storage system is detected to be positioned in the replica synchronization period;
a determining module configured to determine, based on the delay state indicator, whether the pecking reaches an abnormal state;
and the starting module is configured to stop the current on-going Peering and restart the Peering flow if the Peering is determined to reach an abnormal state.
According to still another aspect of an embodiment of the present application, there is provided an electronic apparatus including:
a memory for storing executable instructions; and
and a display for executing the executable instructions with the memory to perform the operations of the fault recovery method during any one of the above-described replica syncs.
According to yet another aspect of an embodiment of the present application, there is provided a computer-readable storage medium storing computer-readable instructions that, when executed, perform the operations of any of the above-described failure recovery methods during replica synchronization.
In the application, when the target PG in the distributed storage system is detected to be positioned in the replica synchronization period, the delay state index for reflecting the target PG in the replica period can be acquired; determining whether the early reaches an abnormal state or not based on the time delay state index; if it is determined that the Peering reaches the abnormal state, stopping the current Peering and restarting the Peering flow.
By applying the technical scheme of the application, after the fact that the Peering flow exists in the storage system is detected, whether the PG reaches an abnormal state during the period of Peering or not can be determined pertinently through the comparison of the time delay state index generated during the period of Peering and the standard time delay index obtained in real time or periodically, so that when the PG is determined to be abnormal in the follow-up process, the quick recovery of the PG can be completed by restarting the Peering. Therefore, the technical scheme that the failure of the engineering can be automatically solved to recover the service state of the PG as soon as possible without manual intervention is realized, and the failure self-processing capacity of the distributed storage system is further improved. The defect that the recovery time of the storage system is slow because the network can only be restarted manually to recover when the storage system has a pest failure is avoided.
The technical scheme of the application is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.
The application may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a schematic diagram of a method for recovering from failures during synchronization of replicas according to the present application;
fig. 2 is a schematic diagram of acquiring a transmission delay indicator according to the present application;
fig. 3 is a schematic flow chart of determining whether the early reaches an abnormal state based on a transmission delay index;
FIG. 4 is a schematic flow chart of determining whether the early reaches an abnormal state based on the processing delay index;
FIG. 5 is a schematic flow chart for determining whether the early reaches an abnormal state based on a continuous time delay index;
FIG. 6 is a schematic overall flow chart of a method for recovering from failures during synchronization of replicas according to the present application;
fig. 7 is a schematic structural diagram of an electronic device according to the present application;
fig. 8 is a schematic structural diagram of an electronic device according to the present application.
Detailed Description
Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present application unless it is specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.
The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the application, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
In addition, the technical solutions of the embodiments of the present application may be combined with each other, but it is necessary to be based on the fact that those skilled in the art can implement the technical solutions, and when the technical solutions are contradictory or cannot be implemented, the combination of the technical solutions should be considered as not existing, and not falling within the scope of protection claimed by the present application.
It should be noted that all directional indicators (such as up, down, left, right, front, and rear … …) in the embodiments of the present application are merely used to explain the relative positional relationship, movement conditions, etc. between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indicators are correspondingly changed.
A fault recovery method for use during copy synchronization according to an exemplary embodiment of the present application is described below in conjunction with fig. 1-6. It should be noted that the following application scenarios are only shown for facilitating understanding of the spirit and principles of the present application, and embodiments of the present application are not limited in this respect. Rather, embodiments of the application may be applied to any scenario where applicable.
The application also provides a fault recovery method, a fault recovery device, electronic equipment and a medium during copy synchronization.
In the related art, a distributed storage system generally supports three storage services, namely, block storage, object storage and file storage. During the operation of the service, PG faults may occur on probability, some are caused by software, and some are external or hardware problems.
It will be appreciated that since in a distributed system, it is the placement group (PG placement_group) that is directly responsible for traffic handling. Therefore, once the PG failure is detected, it is most important to wait for all PGs to recover to the service-complete Active state, and the distributed storage system can continue to carry the service.
The present application describes the PG engineering flow of a distributed storage system:
in one manner, the process of caching pools PG in a distributed storage system is a process of enabling CSDs in a PG to achieve a consistent state of data. That is, after the master-slave CSD of the PG completes the copy synchronization process, the PG will be in an Active state, and the peer process is determined to end.
As an example, the PG of a distributed storage system in the related art triggers the following three cases:
first case:
when the distributed storage system is initialized, CSD (cache pool data storage cache storage device) under the PG will cause the PG to be restarted to reload. Thereby putting PG into the engineering process.
Second case:
the PG may also occur once during the pearing process when the distributed storage system newly creates one or more PGs.
Third case:
when the mapping of the corresponding PG changes due to a CSD failure, addition or deletion of the CSD, etc., in the distributed storage system, the PG also undergoes a pearing process.
As another example, in the process of performing the peer discovery in the PG of the distributed storage system in the related art, the peer discovery fault condition (for example, a long-time unresponsive stuck condition, which further causes the system to fail to perform the service processing) occurs in the following three states:
first state:
the PG communicates with several other replica CSDs over the network during the CSD GetInfo process. During this time, when an abnormality occurs in the network of the distributed system, the transmitted or received message is lost, which will lead to PG engineering failure.
In one mode, the CSD GetInfo process of PG is:
after the master CSD of the PG sends the MCSDPGQuery message to the slave CSD, the slave CSD replies the MCSDPGNotification to the master CSD to acquire the objects on each copy, thereby acquiring the corresponding authority object.
Second state:
the PG also communicates with several other replica CSDs over the network during CSD activation. Similarly, when the network of the distributed system is abnormal, the message is very easy to be lost, and the PG engineering fault problem is caused.
In one mode, the CSD Activating process of PG is:
the process of activating the master CSD and the slave CSD.
Third state:
PG failure problems may also arise when messages are received from the CSD, when the CSD does not process or reply to the message due to the slow processing of messages inside the CSD or other software BUGs.
However, in the related art, when detecting that the distributed storage system has a PG failure, in order to enable the PG to recover to the aactive state as soon as possible, so as to reduce the influence on the service, the following two human intervention methods are generally adopted to achieve the purpose of the PG:
the first way is:
and (5) manually restarting the network to recover: when the distributed storage system is blocked in the PG (Peering) state, the network of the cluster is manually restarted, so that Peering of all PGs can be retried, and the PGs can be recovered as soon as possible.
The second way is:
the primary CSD of the PG is manually restarted: when the distributed storage system is blocked in the PG scheduling state, the main CSD mapped by the current PG is queried, the CSD is restarted, and the scheduling of the PG is triggered again, so that the PG is restored.
It will be appreciated that this manual handling in the related art is inefficient, and may also lead to problems of slow recovery time of the storage system.
A flow diagram of a method for failure recovery during synchronization of replicas according to an embodiment of the present application.
As shown in fig. 1, the method is applied to a distributed storage system including a placement group PG, and includes:
s101, when detecting that the target PG in the distributed storage system is located in the replica synchronization period, acquiring a time delay state index for reflecting the target PG in the replica period.
In one form, the storage system of the present application may be a distributed storage system.
Among them, a distributed storage system (Distributed Storage System) is a storage system that achieves high availability and reliability of data by storing a larger amount of data on a plurality of computers in a distributed manner.
Furthermore, the data can be stored and managed by utilizing a plurality of servers, and the data can be stored on the servers through data interaction of a network, so that multiple backups of the data are realized. And can meet the economic storage scheme of high-speed read-write access with larger storage capacity, has low space use cost and higher availability and reliability.
In one mode, in order to avoid the problem that the failure recovery efficiency of the distributed storage system is low because the distributed storage system can only rely on a manual recovery mode under the condition of facing PG research failure. The embodiment of the application adopts the purpose of acquiring the time delay state index of the PG in the period of the Peering to automatically monitor whether the PG has the Peering fault currently.
S102, determining whether the early reaches an abnormal state or not based on the time delay state index.
In one manner, the delay state index of the target PG during the period of the aging may be used to reflect the delay condition of the target PG in different scenes.
As an example, the latency status indicators may include, for example, the following three indicators:
first index, transmission delay index:
as an example, a transmission delay index may be used to reflect the delay degree of message transmission during the period of the peer PG.
Specifically, the process of the target PG is a process of copy synchronization from the master CSD to the slave CSD corresponding to the target PG. Therefore, the embodiment of the application can realize the acquisition of the transmission delay index by detecting the network layer delay in the copy synchronization process.
As an example, the embodiment of the present application may use a message module to detect and monitor the time stamp of each message, and further count the time stamps of actually sent and received messages in the PG searching process, and determine that there is a case where the searching reaches an abnormal state currently when detecting that there are one or more searching messages whose processing duration does not meet the first latency criterion.
The second index, the processing delay index:
As an example, a processing delay indicator may be used to reflect the degree of delay of message processing during the period of the target PG.
Specifically, after receiving the ping message, each CSD of the target PG converts the ping message into a message event and records the message event in its own message processing queue. Therefore, the embodiment of the application can realize the acquisition of the processing time delay index by detecting the enqueuing duration of each of the events in the message processing queue.
By way of example, the embodiment of the application can acquire all the events of the target PG generated during the period of the Peering and count the processing time of each event of the Peering in the distributed storage system; and when detecting that one or more Peering messages with processing time length not meeting the second time delay standard exist, determining that the situation that the Peering reaches an abnormal state exists currently.
Third index, duration index:
as an example, a duration indicator may be used to reflect the duration of the pen.
Specifically, since all of the paging messages are sent only once during the PG paging of the distributed storage system. Thus, in the case when a network abnormality, a software BUG, or a hardware abnormality of the distributed system occurs, it is extremely easy to cause a loss of a message. And loss of messages can cause PG engineering failure problems.
Therefore, the embodiment of the application can realize the acquisition of the continuous time delay index by detecting the continuous time length of the early. And when the duration is detected not to meet the third time delay standard, determining that the condition that the Peering reaches the abnormal state exists currently.
S103, if determining that the pest reaches an abnormal state, stopping the current pest and restarting the pest flow.
In order to avoid the problem that in the related art, once a failure event of the peer is generated, the failure recovery can only be realized by manual operation of a service staff, so that the failure recovery time of the storage system is slower. The embodiment of the application adopts a mode of automatically restarting the pest process when the distributed storage system has faults such as PG pest jam and the like, so that the fault PG can be automatically restored to a complete service state as soon as possible. It can be appreciated that because it does not require human intervention, the reliability and self-healing capability of the distributed storage system can be improved, and the impact of PG faults on the service can be reduced.
In one mode, after the distributed storage system determines that the pest process in which a certain PG exists currently is abnormal, the embodiment of the present application can implement the problem of repairing the pest failure of the PG (i.e., replace the old pest process currently blocked due to the failure with a new pest process) by starting the update process of the self-osdmap data so that the pest process will be triggered again when the PG receives the new osdmap.
In the application, when the target PG in the distributed storage system is detected to be positioned in the replica synchronization period, the delay state index for reflecting the target PG in the replica period can be acquired; determining whether the early reaches an abnormal state or not based on the time delay state index; if it is determined that the Peering reaches the abnormal state, stopping the current Peering and restarting the Peering flow.
By applying the technical scheme of the application, after the fact that the Peering flow exists in the storage system is detected, whether the PG reaches an abnormal state during the period of Peering or not can be determined pertinently through the comparison of the time delay state index generated during the period of Peering and the standard time delay index obtained in real time or periodically, so that when the PG is determined to be abnormal in the follow-up process, the quick recovery of the PG can be completed by restarting the Peering. Therefore, the technical scheme that the failure of the engineering can be automatically solved to recover the service state of the PG as soon as possible without manual intervention is realized, and the failure self-processing capacity of the distributed storage system is further improved. The defect that the recovery time of the storage system is slow because the network can only be restarted manually to recover when the storage system has a pest failure is avoided.
Optionally, in another embodiment of the above method according to the present application, determining whether the peer reaches the abnormal state based on the delay state index includes:
determining whether the delay reaches an abnormal state based on a transmission delay index contained in the delay state index, wherein the transmission delay index is used for reflecting the delay degree of message transmission of the target PG during the delay period of the delay; determining whether the target PG reaches an abnormal state or not based on a processing time delay index contained in the time delay state index, wherein the processing time delay index is used for reflecting the delay degree of message processing of the target PG in the period of the target PG; and determining whether the pecking reaches the abnormal state based on a duration time delay index contained in the time delay state index, wherein the duration time delay index is used for reflecting the duration time of the pecking.
In one manner, for the first index (i.e., the transmission delay index), the manner of determining whether the peer reaches the abnormal state in the embodiment of the present application may be:
acquiring all the PERING messages generated by the target PG during the PERING period;
recording a receiving time stamp of each of the Peering messages through a message module in the distributed storage system; and, retrieving the transmission time stamp recorded by the corresponding buffer pool data storage device CSD in each of the ping messages;
And determining the corresponding transmission duration of each of the PERING messages based on the difference between the receiving time stamp and the sending time stamp of each of the PERING messages.
And if detecting that at least one Peering message with the transmission duration not meeting the first delay standard exists, determining that the Peering reaches an abnormal state.
In one mode, the embodiment of the application can utilize a plurality of message modules in the distributed storage system to respectively count the sending and receiving time stamps of each of the paging messages interacted between the master CSD and the slave CSD in the PGing process.
For example, taking fig. 2 as an example, the embodiment of the present application may use the message module 1 (i.e. msg 1) to count the time stamp of the first time of transmission of the paging message for the first time interaction between the master CSD and the slave CSD, and calculate the first time of transmission (reflected by the round trip delay) according to this time stamp. Meanwhile, the message module 2 (i.e. msg 2) can be used to count the transceiving time stamp of the second interactive peer message between the master CSD and the slave CSD, and calculate the second transmission time length according to the transceiving time stamp. Until the transmission time length corresponding to each of the paging messages is calculated.
In one mode, the transmission time length is calculated by recording a transmission time stamp (recorded on the paging message by the CSD of the messaging body) when the paging message is sent, and comparing the transmission time stamp with the transmission time stamp when the paging message is received by the messaging module.
As an example, referring to fig. 3, in an embodiment of the present application, the step of determining whether the early reaches the abnormal state with respect to the delay indicator may include:
in step a1, counting the transmission time length of all the current paging messages in the network layer in the target PG.
And a2, calculating the proportion of the transmission duration in the timeout interval [ min, max ] in all the paging messages.
It can be appreciated that if the preset ratio is exceeded, it is determined that the current PG optimization reaches an abnormal state.
Step a3, checking whether the transmission time length of the paging message is larger than the maximum value in the timeout interval [ min, max ].
It will be appreciated that if there is a maximum value that exceeds the timeout period, then it is determined that the current PG marine reaches an abnormal state.
In one mode, after detecting that the current PG has reached an abnormal state, the embodiment of the present application may further enter a reset frequency limiting determination, and if a frequency limiting condition is met (for example, only one connection can be reset within 5 minutes), the communication link between the master CSD and the slave CSD corresponding to the target PG is reset, and the peer message is resent, so as to repair the peer seizing problem.
In another manner, for the second index (i.e. the processing delay index), the manner of determining whether the peer reaches the abnormal state in the embodiment of the present application may be:
All of the events generated by the target PG during the period of PERING are acquired.
Acquiring message processing queues corresponding to the CSDs contained in the target PG, wherein each message processing queue comprises at least one Peering event responsible for the corresponding CSD, and each Peering event is marked with an enqueuing time point for joining the message processing queue;
and determining the processing duration corresponding to each of the events based on the difference between the current time point and the enqueuing time point corresponding to each of the events.
And if detecting that at least one Peering message with the processing time length not meeting the second time delay standard exists, determining that the Peering reaches an abnormal state.
In one mode, the embodiment of the application can respectively count the processing time of each of the events in the PG engineering process by utilizing the message processing queues corresponding to each CSD contained in the target PG.
After receiving the paging message, the CSD generally includes receiving, converting, and processing three flows, and in the embodiment of the present application, after each paging message is converted into a corresponding paging event, it is determined, according to a processing duration of each paging event, whether the paging of the current target PG reaches an abnormal state.
As an example, referring to fig. 4, in an embodiment of the present application, the step of determining whether the peer reaches an abnormal state with respect to the delay indicator may include:
Step b1, firstly, after each CSD (including both the master CSD and the slave CSD) of the target PG converts the scheduling message into a message event, the message event is recorded in a message processing queue of the target PG, and the current time is recorded as an enqueuing time point of the scheduling event.
And b2, ordering all the various events in the message processing queue based on the enqueuing time point of the event by using a binary ordering algorithm.
It can be understood that, in the ordered message queue, the first scheduling event is the message event with the longest processing duration in the CSD.
And b3, regularly taking out the time stamp of the closest preceding events in each message queue, and calculating the message event with the longest current processing time length in each message queue according to the time stamp.
It can be appreciated that when a message event exists in which the processing duration exceeds the preset duration, it is determined that the current pgpen reaches the abnormal state.
And b4, calculating the average processing time length of all the processing time lengths of the events of the PEring in each message queue at the same time.
It can be appreciated that when the processing duration exceeds the preset average duration, it is determined that the current pgpen reaches the abnormal state.
In one mode, after determining that the current PG optimization reaches an abnormal state, the embodiment of the present application may report the corresponding CSD optimization exception to a MON (a Monitor component of monitoring and management of the distributed storage system) of the distributed storage system.
In another manner, for the third indicator (i.e., the duration indicator), the manner of determining whether the early reaches the abnormal state may be:
periodically checking the Peering by the main CSD associated with the target PG to obtain the duration of the Peering;
and if the duration is detected not to meet the third time delay standard, determining that the Peering reaches the abnormal state.
In one mode, since in the distributed storage system, the PG scheduling process message is only sent once, the message is very easy to be lost in case of network abnormality or software BUG and hardware abnormality. And the loss of the paging message can cause the failure problems such as PG paging jamming and the like.
As an example, as shown in connection with fig. 5, embodiments of the present application may periodically check for periodicity of the Peering by the primary CSD of the target PG. It can be understood that if the target PG is detected to be in the engineering state for a long time and the duration exceeds the set time, reporting the CSD exception to the Monitor of the distributed storage system, downloading the CSD flag, and re-triggering the engineering after updating the new osdmap.
Optionally, in another embodiment of the above method according to the present application, after detecting that the target PG in the distributed storage system is located during copy synchronization of the peer PG, the method further includes:
Detecting that the duration of the American exceeds a preset duration; and/or detecting that the network state of the current distributed storage system exceeds a preset congestion threshold;
a delay state index reflecting the target PG during the period of aging is acquired.
In one mode, since the running stability of the early warning is relatively good, the embodiment of the application aims to avoid the problem of consuming system running resources caused by meaningless monitoring of the time delay state index of the early warning during the initial period. The distributed system can choose to only if the events satisfy certain conditions the acquisition step of the delay state index of the target PG during the period of the adaption starts.
As an example, the preset duration may be 1 minute or 30 seconds, or the like. The preset congestion threshold may be a preset network limit value, etc.
Fig. 6 is a schematic flow chart of a fault recovery method during copy synchronization according to an embodiment of the present application, where the fault recovery method includes:
step 1, detecting the duration of the target PG in the distributed storage system during the period of the Peering and/or the current network state of the distributed storage system.
And step 2, acquiring a transmission delay index, a processing delay index and a duration delay index after determining that the duration exceeds a preset duration and/or the network state exceeds a preset congestion threshold. Then, step 3a, step 3b and step 3c are entered simultaneously or in any order.
The transmission delay index is used for reflecting the delay degree of message transmission of the target PG in the period of the aging.
The processing delay index is used to reflect the delay degree of the message processing of the target PG during the period of the aging.
The duration indicator is used to reflect the duration of the adaption.
And 3a, acquiring all the PERING messages generated by the target PG in the PERING period, and counting the transmission time of each PERING message in a network layer in the distributed storage system.
In one manner, the transmission duration may be obtained by:
recording a receiving time stamp of each of the Peering messages through a message module in the distributed storage system; and, retrieving the transmission time stamp recorded by the corresponding buffer pool data storage device CSD in each of the ping messages;
and determining the corresponding transmission duration of each of the PERING messages based on the difference between the receiving time stamp and the sending time stamp of each of the PERING messages.
And 4a, if detecting that at least one Peering message with the transmission duration not meeting the first time delay standard exists, determining that the Peering reaches an abnormal state. Step 5 is then entered.
And 3b, acquiring all the pest events generated by the target PG in the pest period, and counting the processing time of each pest event in the distributed storage system.
In one mode, the method for obtaining the processing duration may be:
acquiring message processing queues corresponding to the CSDs contained in the target PG, wherein each message processing queue comprises at least one Peering event responsible for the corresponding CSD, and each Peering event is marked with an enqueuing time point for joining the message processing queue;
and determining the processing duration corresponding to each of the events based on the difference between the current time point and the enqueuing time point corresponding to each of the events.
And step 4b, if detecting that at least one of the Peering messages with the processing time length not meeting the second time delay standard exists, determining that the Peering reaches an abnormal state. Step 5 is then entered.
And step 3c, acquiring the duration of the early, and determining that the early reaches an abnormal state if the duration is detected to not meet the third time delay standard. Step 5 is then entered.
And 5, determining that the Peering reaches an abnormal state, and stopping the current on-going Peering.
And step 6, restarting the pest flow by updating the data osdmap of the object storage device currently set by the distributed storage system.
By applying the technical scheme of the application, after the fact that the Peering flow exists in the storage system is detected, whether the PG reaches an abnormal state during the period of Peering or not can be determined pertinently through the comparison of the time delay state index generated during the period of Peering and the standard time delay index obtained in real time or periodically, so that when the PG is determined to be abnormal in the follow-up process, the PG can be recovered as soon as possible by restarting the Peering mode. Therefore, the technical scheme that the PE service state can be automatically recovered as soon as possible by automatically solving the failure of the PE, and manual intervention is not needed is realized, and the failure self-processing capacity of the distributed storage system is further improved. The defect that the recovery time of the storage system is slow because the network can only be restarted manually to recover when the storage system has a pest failure is avoided.
Optionally, in another embodiment of the present application, as shown in fig. 7, the present application further provides a fault recovery apparatus during copy synchronization. In a distributed storage system including a placement group PG, comprising:
a detection module 201, configured to obtain a delay state index for reflecting a target PG in the distributed storage system during replica synchronization period when detecting that the target PG is located during the replica synchronization period;
a determining module 202 configured to determine, based on the delay status indicator, whether the pecking reaches an abnormal status;
and a starting module 203, configured to stop the current running pepering and restart the pepering procedure if it is determined that the pepering reaches an abnormal state.
By applying the technical scheme of the application, after the fact that the Peering flow exists in the storage system is detected, whether the PG reaches an abnormal state during the period of Peering or not can be determined pertinently through the comparison of the time delay state index generated during the period of Peering and the standard time delay index obtained in real time or periodically, so that when the PG is determined to be abnormal in the follow-up process, the quick recovery of the PG can be completed by restarting the Peering. Therefore, the technical scheme that the failure of the engineering can be automatically solved to recover the service state of the PG as soon as possible without manual intervention is realized, and the failure self-processing capacity of the distributed storage system is further improved. The defect that the recovery time of the storage system is slow because the network can only be restarted manually to recover when the storage system has a pest failure is avoided.
In another embodiment of the present application, the determining module 202 is configured to:
determining whether the peer reaches an abnormal state based on a transmission delay index contained in the delay state index, wherein the transmission delay index is used for reflecting the delay degree of message transmission of the target PG during the peer; the method comprises the steps of,
determining whether the peer reaches an abnormal state based on a processing delay index contained in the delay state index, wherein the processing delay index is used for reflecting the delay degree of message processing of the target PG during the peer; the method comprises the steps of,
and determining whether the pecking reaches an abnormal state or not based on a continuous time delay index contained in the time delay state index, wherein the continuous time delay index is used for reflecting the duration of the pecking.
In another embodiment of the present application, the determining module 202 is configured to:
acquiring all the Peering messages generated by the target PG in the Peering period, and counting the transmission time of each Peering message in a network layer in the distributed storage system;
and if detecting that at least one Peering message with the transmission duration not meeting the first delay standard exists, determining that the Peering reaches an abnormal state.
In another embodiment of the present application, the determining module 202 is configured to:
recording a receiving time stamp of each of the Peering messages through a message module in the distributed storage system; and, retrieving the transmission time stamp recorded by the corresponding buffer pool data storage device CSD in each of the ping messages;
and determining the transmission duration corresponding to each of the PERING messages based on the difference between the receiving time stamp and the sending time stamp of each of the PERING messages.
In another embodiment of the present application, the determining module 202 is configured to:
acquiring all the pest events generated by the target PG during the pest period, and counting the processing time of each pest event in the distributed storage system;
and if detecting that at least one Peering message with the processing time length not meeting the second time delay standard exists, determining that the Peering reaches an abnormal state.
In another embodiment of the present application, the determining module 202 is configured to:
acquiring message processing queues corresponding to each CSD contained in the target PG, wherein each message processing queue comprises at least one Peering event responsible for the corresponding CSD, and each Peering event is marked with an enqueuing time point added into the message processing queue;
And determining the processing duration corresponding to each of the events based on the difference between the current time point and the enqueuing time point corresponding to each of the events.
In another embodiment of the present application, the determining module 202 is configured to:
acquiring the duration of the early;
and if the duration is detected not to meet the third time delay standard, determining that the Peering reaches an abnormal state.
In another embodiment of the present application, the determining module 202 is configured to:
the duration of the pen is obtained by periodic inspection of the pen by the primary CSD associated with the target PG.
In another embodiment of the present application, the determining module 202 is configured to:
detecting that the duration of the early exceeds a preset duration; and/or detecting that the current network state of the distributed storage system exceeds a preset congestion threshold;
the delay state index used for reflecting the target PG during the period of the Peering is acquired.
In another embodiment of the present application, the determining module 202 is configured to:
determining that the early reaches an abnormal state;
and restarting the pest flow by updating the data osdmap of the object storage device currently set by the distributed storage system.
The embodiment of the application also provides the electronic equipment for executing the fault recovery method during the copy synchronization. Referring to fig. 8, a schematic diagram of an electronic device according to some embodiments of the present application is shown. As shown in fig. 8, the electronic apparatus 3 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, the processor 300, the communication interface 303 and the memory 301 being connected by the bus 302; the memory 301 stores a computer program executable on the processor 300, and the processor 300 executes the fault recovery method during copy synchronization provided by any of the foregoing embodiments of the present application when the computer program is executed.
The memory 301 may include a high-speed random access memory (RAM: random Access Memory), and may further include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. The communication connection between the device network element and at least one other network element is achieved through at least one communication interface 303 (which may be wired or wireless), the internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
Bus 302 may be an ISA bus, a PCI bus, an EISA bus, or the like. The buses may be classified as address buses, data buses, control buses, etc. The memory 301 is configured to store a program, and the processor 300 executes the program after receiving an execution instruction, and the video transmission method disclosed in any of the foregoing embodiments of the present application may be applied to the processor 300 or implemented by the processor 300.
The processor 300 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 300 or by instructions in the form of software. The processor 300 may be a general-purpose processor, including a processor (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but may also be a Digital Signal Processor (DSP), application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301, and in combination with its hardware, performs the steps of the above method.
The fault recovery method during the synchronization of the electronic device and the copy provided by the embodiment of the application has the same beneficial effects as the method adopted, operated or realized by the same inventive concept.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (13)
1. A method for recovering from failures during synchronization of replicas, applied to a distributed storage system comprising a placement group PG, comprising:
when detecting that a target PG in the distributed storage system is located in a replica synchronization period, acquiring a time delay state index for reflecting the target PG in the period;
determining whether the pecking reaches an abnormal state or not based on the time delay state index;
if the Peering reaches the abnormal state, stopping the Peering currently in progress, and restarting the Peering flow.
2. The method of claim 1, wherein the determining whether the pecking reaches an abnormal state based on the latency state indicator comprises:
Determining whether the peer reaches an abnormal state based on a transmission delay index contained in the delay state index, wherein the transmission delay index is used for reflecting the delay degree of message transmission of the target PG during the peer; the method comprises the steps of,
determining whether the peer reaches an abnormal state based on a processing delay index contained in the delay state index, wherein the processing delay index is used for reflecting the delay degree of message processing of the target PG during the peer; the method comprises the steps of,
and determining whether the pecking reaches an abnormal state or not based on a continuous time delay index contained in the time delay state index, wherein the continuous time delay index is used for reflecting the duration of the pecking.
3. The method of claim 2, wherein the determining whether the pecing reaches an abnormal state based on a transmission delay indicator included in the delay state indicator comprises:
acquiring all the Peering messages generated by the target PG in the Peering period, and counting the transmission time of each Peering message in a network layer in the distributed storage system;
and if detecting that at least one Peering message with the transmission duration not meeting the first delay standard exists, determining that the Peering reaches an abnormal state.
4. The method of claim 3, wherein said counting the transmission time of each of the Peering messages at the network layer in the distributed storage system comprises:
recording a receiving time stamp of each of the Peering messages through a message module in the distributed storage system; and, retrieving the transmission time stamp recorded by the corresponding buffer pool data storage device CSD in each of the ping messages;
and determining the transmission duration corresponding to each of the PERING messages based on the difference between the receiving time stamp and the sending time stamp of each of the PERING messages.
5. The method of claim 2, wherein the determining whether the pecing reaches an abnormal state based on a processing latency indicator included in the latency state indicator comprises:
acquiring all the pest events generated by the target PG during the pest period, and counting the processing time of each pest event in the distributed storage system;
and if detecting that at least one Peering message with the processing time length not meeting the second time delay standard exists, determining that the Peering reaches an abnormal state.
6. The method of claim 5, wherein said counting the processing time of each of the events in the distributed storage system comprises:
Acquiring message processing queues corresponding to each CSD contained in the target PG, wherein each message processing queue comprises at least one Peering event responsible for the corresponding CSD, and each Peering event is marked with an enqueuing time point added into the message processing queue;
and determining the processing duration corresponding to each of the events based on the difference between the current time point and the enqueuing time point corresponding to each of the events.
7. The method of claim 2, wherein the determining whether the pecing reaches an abnormal state based on a duration delay indicator included in the delay state indicator comprises:
acquiring the duration of the early;
and if the duration is detected not to meet the third time delay standard, determining that the Peering reaches an abnormal state.
8. The method of claim 7, wherein the obtaining the duration of the pecking comprises:
the duration of the pen is obtained by periodic inspection of the pen by the primary CSD associated with the target PG.
9. The method of claim 1, wherein after the detecting that the target PG in the distributed storage system is located during replica synchronization peak, further comprising:
Detecting that the duration of the early exceeds a preset duration; and/or detecting that the current network state of the distributed storage system exceeds a preset congestion threshold;
the delay state index used for reflecting the target PG during the period of the Peering is acquired.
10. The method of claim 1, wherein said determining that said pecking reaches an abnormal state, stopping said pecking currently in progress, and restarting a pecking procedure comprises:
determining that the early reaches an abnormal state;
and restarting the pest flow by updating the data osdmap of the object storage device currently set by the distributed storage system.
11. A failure recovery apparatus during synchronization of replicas, for use in a distributed storage system comprising a placement group PG, comprising:
the detection module is configured to acquire a time delay state index for reflecting the target PG in the replica synchronization period when the target PG in the distributed storage system is detected to be positioned in the replica synchronization period;
a determining module configured to determine, based on the delay state indicator, whether the pecking reaches an abnormal state;
and the starting module is configured to stop the current on-going Peering and restart the Peering flow if the Peering is determined to reach an abnormal state.
12. An electronic device, comprising:
a memory for storing executable instructions; the method comprises the steps of,
a processor for executing the executable instructions with the memory to perform the operations of the failback method during copy synchronization of any one of claims 1-10.
13. A computer readable storage medium storing computer readable instructions which, when executed, perform the operations of the method of recovering from failure during synchronization of copies of any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311095872.6A CN116974808A (en) | 2023-08-28 | 2023-08-28 | Fault recovery method and device during copy synchronization, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311095872.6A CN116974808A (en) | 2023-08-28 | 2023-08-28 | Fault recovery method and device during copy synchronization, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116974808A true CN116974808A (en) | 2023-10-31 |
Family
ID=88481601
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311095872.6A Pending CN116974808A (en) | 2023-08-28 | 2023-08-28 | Fault recovery method and device during copy synchronization, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116974808A (en) |
-
2023
- 2023-08-28 CN CN202311095872.6A patent/CN116974808A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11907561B2 (en) | Data backup method and apparatus | |
| CN112506710B (en) | Distributed file system data restoration method, device, equipment and storage medium | |
| JP2017517060A (en) | Fault processing method, related apparatus, and computer | |
| CN109656895B (en) | Distributed storage system, data writing method, device and storage medium | |
| US20160321127A1 (en) | Determine when an error log was created | |
| CN111221775B (en) | Processor, cache processing method and electronic equipment | |
| WO2022033586A1 (en) | Message sending method and device | |
| TWI518680B (en) | Method for maintaining file system of computer system | |
| CN111211993B (en) | Incremental persistence method, device and storage medium for stream computation | |
| CN111880947B (en) | Data transmission method and device | |
| CN117762652A (en) | Distributed transaction processing method and device based on message middleware | |
| CN116974808A (en) | Fault recovery method and device during copy synchronization, electronic equipment and medium | |
| WO2020094063A1 (en) | Data storage method and device, storage medium and electronic device | |
| CN116541461A (en) | Data processing method, device, equipment and storage medium applied to database | |
| CN114116282B (en) | Method and device for reporting and repairing network additional storage faults | |
| CN113268395B (en) | Service data processing method, processing device and terminal | |
| CN113542001B (en) | OSD (on-screen display) fault heartbeat detection method, device, equipment and storage medium | |
| CN113850664A (en) | Data anomaly detection method and data reporting service | |
| CN114722029B (en) | Method, system, equipment and storage medium for repairing monitor database | |
| CN113176967B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN118445126A (en) | Data processing method, device, equipment, readable storage medium and product | |
| JP6674916B2 (en) | Communication fault management device and communication system | |
| CN109815064B (en) | Node isolation method, node isolation device, node equipment and computer readable storage medium | |
| CN118277336A (en) | Metadata management method, metadata management device, and computer storage medium | |
| CN116263721A (en) | System fault recovery method and equipment for data backup |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |