CN107181608B - Method for recovering service and improving performance and operation and maintenance management system - Google Patents
Method for recovering service and improving performance and operation and maintenance management system Download PDFInfo
- Publication number
- CN107181608B CN107181608B CN201610140348.XA CN201610140348A CN107181608B CN 107181608 B CN107181608 B CN 107181608B CN 201610140348 A CN201610140348 A CN 201610140348A CN 107181608 B CN107181608 B CN 107181608B
- Authority
- CN
- China
- Prior art keywords
- node
- nodes
- normal
- parameter
- node cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0659—Management of faults, events, alarms or notifications using network fault recovery by isolating or reconfiguring faulty entities
- H04L41/0661—Management of faults, events, alarms or notifications using network fault recovery by isolating or reconfiguring faulty entities by reconfiguring faulty entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0681—Configuration of triggering conditions
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
- Telephonic Communication Services (AREA)
Abstract
A method for recovering service and improving performance and an operation and maintenance management system are disclosed, wherein the operation and maintenance management system detects the state change of nodes in a node cluster of an operation state synchronization protocol and determines the number NN of normal nodes, wherein NN is an integer; and if the NN is changed from being greater than or equal to S0 to being smaller than S0, performing emergency processing to restore normal service, wherein when the normal node comprises the main node, the emergency processing comprises: modifying the values of the parameters S stored by the configuration center and the normal nodes into positive integer values smaller than or equal to NN; wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol. The method and the device can effectively solve the problem that hardware errors occur simultaneously in a plurality of nodes and bring unavailability.
Description
Technical Field
The present invention relates to a distributed system, and more particularly, to a method for recovering services and improving performance of a distributed system and an operation and maintenance management system.
Background
Currently, in a large-scale distributed storage system, in order to implement centralized authority authentication and quota control, a centralized metadata management method is mostly adopted, that is, metadata of all data in the whole storage system is stored in a plurality of nodes in a centralized manner for storage. The availability of metadata nodes (also called metadata servers and the like) in the architecture is directly related to the availability of the whole system, and the availability of metadata services is increased in a redundant mode in various distributed systems.
The redundant approach introduces multiple nodes, and a state synchronization protocol must be used between the nodes to ensure that decisions made at any time are correct and undeniable. In a distributed system, if the initial states of the nodes are consistent and each node executes the same command sequence, they can eventually get a consistent state. To ensure that each node executes the same sequence of commands, a "consensus algorithm" needs to be executed on each instruction to ensure that the instructions seen by each node are consistent.
The Paxos protocol is known as one of the most widely used protocols in the state synchronization protocol, and solves the problem of how a distributed system agrees about a certain value (resolution). When the Paxos protocol has modification operation, all modification states are numbered monotonically and incrementally, and a decision is made on a plurality of nodes, and if most nodes agree to accept the decision, the modification is persisted to the plurality of nodes respectively. Such a protocol design can ensure that each resolution is agreed to be made by most nodes, ensuring that the resolution is correct, whereas if a few nodes can make a resolution, this can result in two resolutions being made at the same protocol number, either as a false resolution or as a non-consistent resolution from the user's perspective. At the same time, each time the resolution number and the resolution per se are persistent, the previous resolution is still reserved as long as most of the data of the nodes are not lost, and any subsequent resolution can be continued on the basis of a correct resolution, so that the data consistency, namely the resolution is correct at any time.
In a distributed storage system using a plurality of metadata nodes as backups, if the Paxos protocol is used as a theoretical basis for election and journal backup, a normal metadata service cannot be provided in the case where a few metadata nodes remain. In a production system, because the hardware configuration of the machine where the metadata node is located is basically consistent, for example, all the machines use Solid State Drives (SSDs) of the same manufacturer, the erase and write lifetimes are not very different, so that the probability that multiple machines go wrong at the same time is increased. Once the majority of machines have disk read-only mode, service will be stopped. When most metadata nodes are down, if the main nodes are still available, the service of reading the metadata can be provided for the outside, but the operation of modifying the metadata cannot be successful.
When a simplified mode of a Paxos protocol is used in a distributed storage system, a plurality of metadata nodes are elected through the Paxos protocol, and a Primary node (Primary) is generated to provide metadata service; the other nodes, as Slave nodes (Slave), only accept log synchronization of the master node. The logs generated by the master node are sent to all the slave nodes, if the slave nodes agree and accept log synchronization, the master node receives feedback agreed by the slave nodes, when most nodes are successfully synchronized (including the master node), the master node returns success to a Client (Client) sending a service request, otherwise, the Client requests are suspended, the Client receives a timeout message, and the service is stopped at the moment. That is, when the Paxos protocol is used to provide redundancy capability for metadata services, if most metadata nodes are out of service, the entire service will be stopped even if there are normal nodes in it. In addition, if at least half of the nodes are degraded, the performance of the entire service is degraded. Because most nodes return consent to complete the operation of the client only after log synchronization, the operation performance depends on the performance of the slowest node in most nodes.
Similar situations exist for other node clusters running state synchronization protocols.
Disclosure of Invention
In view of this, the present invention provides the following.
A method for recovering service is applied to an operation and maintenance management system and comprises the following steps:
detecting the state change of nodes in a node cluster of an operating state synchronization protocol, and determining the number NN of normal nodes, wherein NN is an integer;
and if the NN is changed from being greater than or equal to S0 to being smaller than S0, performing emergency processing to restore normal service, wherein when the normal node comprises the main node, the emergency processing comprises: modifying the values of the parameters S stored by the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol.
An operation and maintenance management system comprises a state detection module, a control module and an emergency processing module, wherein:
the state detection module is used for detecting the state change of the nodes in the node cluster of the running state synchronization protocol, determining the number NN of normal nodes and informing the control module, wherein NN is an integer;
the control module is used for calling the emergency processing module to perform emergency processing after the NN is changed from being greater than or equal to S0 to being smaller than S0 so as to recover normal service;
the emergency processing module is configured to, when the normal node includes a master node, perform the following emergency processing: modifying the values of the parameters S stored by the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol.
According to the scheme, when the number of the normal nodes does not reach the minimum number of the synchronization successful nodes required by the normal service, the service can be immediately recovered through parameter change, the service can be conveniently recovered after the fault is eliminated, and data inconsistency and data loss are avoided in the period. The processing method can effectively solve the problem of unavailability caused by hardware errors occurring in a plurality of nodes at the same time, and reduces economic loss.
A method of performance enhancement, comprising:
determining low-performance nodes in a node cluster running a state synchronization protocol;
and when at least one low-performance node is successfully synchronized and the synchronization process of the node cluster can be successful, performing performance improvement processing, so that the synchronization process can be successful when the low-performance node is not successfully synchronized.
An operation and maintenance management system comprising a performance management module, wherein:
the performance management module is used for performing performance improvement processing on the node cluster of the running state synchronization protocol, and comprises the following steps:
the first processing unit is used for modifying the value of the parameter S stored in the configuration center into T-SN;
the second processing unit is used for modifying the value of the parameter S stored by the nodes in the node cluster into T-SN;
wherein T is the number of nodes of the node cluster, T is more than or equal to 2, SN is the number of low-performance nodes in the node cluster, and parameter S represents the minimum number of successful synchronization nodes required by the node cluster for providing normal service.
According to the scheme, when the number of low-performance nodes is increased so that the synchronization speed is reduced, the performance of the node cluster is always matched with that of the high-performance nodes through reasonable parameter configuration.
Drawings
FIG. 1 is a diagram of a network architecture according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of recovering service in accordance with an embodiment of the present invention;
FIG. 3 is a block diagram of an operation and maintenance management system according to an embodiment of the present invention;
FIG. 4 is a flow chart of a method for performance enhancement according to a second embodiment of the present invention;
fig. 5 is a block diagram of a second operation and maintenance management system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the present application, the embodiments and features of the embodiments may be arbitrarily combined with each other without conflict.
Example one
In this embodiment, a metadata node cluster running the Paxos protocol or a derivative protocol thereof in the distributed storage system is taken as an example, and for convenience of description, the metadata node in the embodiment is also referred to as a node for short. Fig. 1 shows a network architecture related to the present embodiment, which includes: the system comprises a metadata node cluster, a configuration center, an operation and maintenance management system and a client, wherein 3 nodes in the metadata node cluster are taken as an example in a graph, and a plurality of metadata nodes are divided into a master node and a slave node.
A master node: the method comprises the steps that a node providing read and write services to the outside converts write operation into a modification log and synchronizes the modification log to all slave nodes;
the slave node: and receiving the logs synchronized by the main node, judging whether the logs can be received or not by protocol agreement, if so, returning to receive success and applying the logs to the memory, otherwise, returning to fail in receiving.
The configuration center is a storage system with persistent node startup parameters, and when each node is started, the configuration parameters used in the election stage and the service stage are acquired from the configuration center, stored in the memory and applied in the corresponding stage.
The operation and maintenance management system comprises: also referred to as an operation and maintenance tool, sends a request to modify parameters to any node when necessary. The operation of extracting the logs from the logs recorded to the local by any node and synchronizing the logs to other normal nodes can be executed on the machine where the node is located; the state of the nodes in the current cluster can be checked, and the number of normal nodes and the number of abnormal nodes are determined; judging whether emergency processing is needed to recover normal service; and, determining which emergency treatment scheme should be used, etc. The operation and maintenance management system comprises various functional modules for completing corresponding operation, maintenance and management functions. In terms of entities, the present invention may be deployed on a metadata node, or may be deployed as other devices, and may be implemented by a single device, or may be implemented by multiple devices, which is not limited to the present invention.
Client (Client): and initiating processes of reading and writing metadata requests, wherein the requests of reading and writing the metadata are all sent to the current host node, and if the write operation is accepted, a modification log is generated at the host node.
And setting the parameters in the current normal metadata node memory as the new parameters configured in the step 2 by using the operation and maintenance tool.
The related processing of the metadata node cluster is divided into an election stage and a service stage, wherein:
and (3) an election stage:
this phase allows multiple nodes to participate in the election process and decide on the master node that provides the service externally, and other nodes become slaves. In Paxos protocol, two parameters T and E are needed to complete elections. . And T is EN + NN, EN is the number of abnormal nodes in the T nodes, and NN is the number of normal nodes in the T nodes. E represents the minimum number of normal nodes required for successful election, the value of E determined according to the state synchronization protocol is denoted as E0, in Paxos protocol, E0 is floor (T/2) +1, and floor () represents rounding down. That is, when most nodes, i.e., more than half of the nodes, are normal, a new master node can be successfully elected.
And (3) a service stage:
after the master node is selected to be generated, all read-write requests are processed through the master node, the write operation can generate a modification log, and the master node synchronizes the log to all slave nodes through a network. The synchronization stage needs two parameters T and S to determine whether synchronization is complete. T has the meaning as defined above. S represents the minimum number of the nodes which are required by the node cluster to provide normal service and are successfully synchronized. The value of S determined according to the state synchronization protocol is denoted as S0, and in the Paxos protocol, S0 is floor (T/2) +1, e.g., S0 is 3 when T is 5 and S0 is 5 when T is 9. In one operation of modifying the metadata, only if the number of nodes successfully written into the corresponding modification log is larger than or equal to S, the operation of modifying the metadata is successful, and the main node is counted in S. That is, in addition to writing the log into the local, the master node needs to synchronize the modification log to S-1 slave nodes successfully to complete the metadata modification operation, and if the metadata modification operation cannot be completed, the client request is suspended, which appears as a service stop.
Obviously, the parameters representing the number of nodes are integers.
As described above, when the number of normal nodes is lower than the parameter S0 required by the Paxos protocol, the abnormal nodes cannot successfully write the modification log, and thus the node cluster cannot provide normal service. Therefore, the operation and maintenance management system makes a decision according to the node state, and when the normal service cannot be provided, the service is immediately recovered through parameter change.
As shown in fig. 2, the method for recovering service in this embodiment is applied to an operation and maintenance management system, and includes:
step 110, detecting the state change of nodes in a node cluster of the running state synchronization protocol, and determining the number NN of normal nodes, wherein NN is an integer;
in this embodiment, the node cluster is a metadata node cluster running a Paxos protocol or a protocol derived from the Paxos protocol in a distributed storage system. The synchronization refers to log synchronization. However, the present invention is not limited to this, and may also be used in other node clusters running a state synchronization protocol, because this embodiment enables the nodes to successfully synchronize with each other by modifying the protocol parameters, so as to recover the service, and the service provided by the node clusters is different in type and does not affect the recovery of the service. The invention can also be used for nodes running other state synchronization protocols, such as the HA protocol.
Step 120, if NN is changed from S0 or more to S0 or less, performing an emergency process to recover the normal service, where the emergency process includes, when the normal node includes a master node: modifying the values of the parameters S stored by the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the meaning of the parameter S is as described above.
Under the condition that the main node is a normal node, the main node is still in a service state, so that the recovery service does not need to pass through an election stage at the moment, and the recovery can be finished only by changing the synchronous log parameter S of the service stage. In this embodiment, after the value of the parameter S is modified to NN (changed value), because NN normal nodes exist in the cluster, NN nodes can be successfully synchronized, and the requirement for the minimum number of nodes that are successfully synchronized is met, so that normal service can be restored. In another embodiment, the value of the parameter S may also be modified to be smaller than the value of the NN, and at this time, the normal service may also be recovered, and when a small number of normal nodes become abnormal nodes, the S does not need to be modified, but when the embodiment is modified to be NN, the number of nodes whose log synchronization is successfully modified is relatively large, and the data security is better.
In this embodiment, when the normal node includes a master node, the emergency processing further includes: and modifying the values of a parameter E stored in the configuration center and the normal nodes to be T-NN '+ 1, wherein the parameter E represents the minimum number of the normal nodes required by the successful election of the node cluster, and NN' is the positive integer value which is smaller than or equal to NN and is modified by the parameter S. Here, the value E is modified to T-NN +1, that is, FN +1, so that to succeed in election, a normal node must participate, and the log stored by the normal node is complete, thereby avoiding inconsistency and loss of log data caused by election of a new master node between abnormal nodes after the original abnormal node is restored to the normal node. Here, it is a convenient way to ensure the integrity of the log by modifying the E value, and other interfaces cannot be added. But not exclusively, for example, the current normal node may be recorded, one of the normal nodes must be referenced in the subsequent election to succeed in the election, and so on.
The operation and maintenance management system may initiate the above process when an emergency recovery metadata service is required (e.g., receiving an instruction from an administrator or triggered according to a corresponding policy).
The embodiment also provides an emergency processing scheme when the normal node does not include the master node after the emergency processing is triggered, at this time, the master node needs to be generated in an election stage because the master node does not exist, and then the processing flow is the same as the emergency processing when the master node exists in the normal node. Specifically, the emergency processing scheme when the normal node does not include the master node includes:
synchronizing local logs of at least FN-floor (T/2) abnormal nodes to a normal node, and modifying values of a parameter E stored in the configuration center and the normal node into NN after synchronization is successful;
when the master node is abnormal, the logs saved by the normal nodes may have inconsistency. So here the synchronization of the logs is performed first. After local logs of at least FN-floor (T/2) abnormal nodes are synchronized to a normal node, the normal node and other normal nodes have logs of (FN-floor (T/2)) + NN (T-floor (T/2)) nodes together, and the nodes can determine the latest state before the abnormality occurs together. Certainly, the precondition of successful synchronization is that at least FN-floor (T/2) machines where abnormal nodes are located can log in, where the abnormal nodes need not complete log synchronization, but it is still possible to log in the machines where the abnormal nodes are located through the operation and maintenance management tool and copy the local logs out.
In this step, the value of the parameter E is modified to NN so that NN normal nodes can successfully host a new master node to restore service.
And secondly, after the NN normal nodes select a new main node, stopping service, executing emergency processing when the normal nodes comprise the main node, and recovering normal service after execution is finished.
After a new master node is elected, the value of S may be modified in accordance with the emergency processing manner in step 120 to restore normal service, and the value of E may also be modified. In order to avoid the generation of new modification logs when modifying the parameters, at which point the metadata service should be stopped, a simple way is to modify the values of the parameters S stored by the configuration center and the normal nodes to values greater than or equal to FN +1 before electing a new master node, so that the synchronization cannot be successful and the service is stopped. And normal service can be resumed after the completion of the emergency process, i.e., the modification of the S value in step 120. It is also possible to disable the service in other ways, such as adding steps where additional configuration items are added.
After emergency processing is carried out, the working state of the node cluster is switched into an emergency state, and the capability that a few metadata nodes can provide services to the outside is realized. In an emergency state, the state (normal or abnormal) of the node may change, for example, a normal node becomes an abnormal node, and the abnormal node becomes a normal node after being restarted after troubleshooting, so this embodiment further provides a processing scheme when the state of the node changes, as follows:
after the NN is changed from being greater than or equal to S0 to being less than S0, and emergency processing is performed and normal service is resumed, if the state of a node in the node cluster is changed again, the NN is also changed, and at this time, the changed NN is compared with S0:
if NN < S0, the emergency process is resumed to resume normal service;
if NN is not less than S0, modifying the values of the parameter S and the parameter E stored by the configuration center and the normal node into a value S0 and a value E0 respectively;
where E0 is the value of parameter E determined according to the state synchronization protocol. .
That is, if there are already a plurality of nodes in a normal state, the S, E values are modified to the respective values determined according to the synchronization status protocol to restore the original operating state. In Paxos protocol, S0 ═ E0 ═ floor (T/2) + 1. If the state of most nodes is abnormal, the emergency processing is required to be carried out again to recover the normal service, and the redundancy characteristic realized by the Paxos protocol is recovered.
In the above solution, the operation and maintenance management system needs to modify the parameters stored in the metadata node, which can be accomplished by one of the following ways:
when the main node is normal or after the main node is reselected, the operation and maintenance management system sends a parameter modification command to the main node; the master node directly modifies the parameters in the memory of the master node, generates a modification log record to the local and sends the modification log record to all slave nodes; the slave node receives the log synchronization, judges whether the log can be received according to protocol agreement, records the log to the local, modifies parameters in the memory and returns the success of the master node if the log can be received.
When the main node is abnormal, the operation and maintenance management system acquires all normal metadata nodes and sends a parameter modification request to each metadata node; the slave node receiving the parameter modification request directly modifies the configuration parameters in the own memory, but does not generate any modification log.
In this embodiment, in the emergency processing schemes in the two cases, in the process of modifying the values of the parameter S and/or the parameter E stored in the configuration center and the normal node, the starting of the abnormal node is prohibited, or the connection between the configuration center and the abnormal node is disconnected, so as to prevent the abnormal node from starting the configuration parameter before the abnormal node is applied in the configuration process.
The embodiment further provides an operation and maintenance management system, as shown in fig. 3, including a status detection module 10, a control module 20, and an emergency processing module 30, where:
the state detection module 10 is configured to detect a state change of a node in a node cluster running a state synchronization protocol, determine the number NN of normal nodes, and notify the control module of the number NN, where NN is an integer;
the control module 20 is configured to invoke the emergency processing module to perform emergency processing to recover normal service after the NN is changed from being greater than or equal to S0 to being smaller than S0;
the emergency processing module 30 is configured to, when the normal node includes a master node, perform the following emergency processing: modifying the values of the parameters S stored by the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol.
Alternatively,
when the normal node includes the master node, the emergency processing module executes emergency processing further including: and modifying the values of a parameter E stored by the configuration center and the normal nodes to be T-NN '+ 1, wherein the parameter E represents the minimum number of the normal nodes required by the successful election of the node cluster, and NN' is the positive integer value which is modified by the parameter S and is less than or equal to NN.
Alternatively,
the emergency processing module is further configured to, when the normal node does not include the master node, perform the following emergency processing: synchronizing local logs of at least FN-floor (T/2) abnormal nodes to a normal node, and modifying the values of parameters E stored in the configuration center and the normal node into NN after synchronization is successful; and after the NN normal nodes select a new master node, stopping service, executing emergency processing when the normal nodes comprise the master node, and recovering normal service after execution is finished, wherein T is the number of the nodes in the node cluster, T is more than or equal to 2, FN is the number of abnormal nodes in the node cluster, FN is T-NN, and floor () represents rounding-down.
Alternatively,
the emergency processing module stops the service after the NN normal nodes select a new main node, and the method is realized by the following steps: before a new master node is elected, the values of the parameters S stored in the configuration center and the normal nodes are modified to be larger than or equal to the value of FN + 1.
Alternatively,
the control module is further configured to compare the changed NN with S0 when the NN notified by the state detection module is changed again:
if NN < S0, calling an emergency processing module to perform emergency processing again to restore normal service;
if NN is not less than S0, modifying the values of the parameter S and the parameter E stored by the configuration center and the normal node into S0 and E0 respectively;
where E0 is the value of parameter E determined according to the state synchronization protocol.
Alternatively,
the node cluster is a metadata node cluster running a Paxos protocol or a derived protocol thereof in the distributed storage system, the synchronization refers to log synchronization, S0 ═ E0 ═ floor (T/2) +1, and floor () represents rounding down.
Alternatively,
and in the process that the emergency processing module modifies the values of the parameters S and/or the parameters E stored in the configuration center and the normal nodes, the emergency processing module prohibits the starting of the abnormal nodes or disconnects the connection between the configuration center and the abnormal nodes.
The above solution of this embodiment is based on the Paxos state synchronization protocol, and by configuring relevant parameters of election and log synchronization among multiple metadata servers, metadata read-write service can still be provided under the condition that a few metadata nodes in a cluster are normal, and the service can be conveniently recovered after troubleshooting, and data inconsistency and loss are not caused in the period.
Example two
This embodiment also relates to a node cluster running a synchronization state protocol, and takes a metadata node cluster running a Paxos protocol or a protocol derived from the Paxos protocol in a distributed storage system as an example. The network architecture is shown in fig. 1 and is not described again.
The embodiment is a method for improving performance, which is provided for a node cluster with many low-performance nodes and thus causing the service performance of the whole cluster to be deteriorated, and as shown in fig. 4, the method includes:
step 210, determining low-performance nodes in a node cluster running a state synchronization protocol;
in this step, the low-performance node may be determined according to a set index such as a node response speed, for example, may be determined by an administrator.
Step 220, when at least one low-performance node is successfully synchronized and the synchronization process of the node cluster can be successful, performing performance improvement processing to ensure that the synchronization process can be successful without the successful synchronization of the low-performance node.
In this embodiment, if the number SN of the low-performance nodes in the node cluster is greater than or equal to T-S0+1, that means that at least one low-performance node is successfully synchronized, the synchronization process of the node cluster can be successful. The performance enhancement process includes: and the operation and maintenance management system modifies the values of the parameter S stored in the configuration center and the nodes in the node cluster into T-SN. Wherein T is the number of nodes of the node cluster, T is greater than or equal to 2, parameter S represents the minimum number of synchronization successful nodes required by the node cluster for providing normal service, and S0 is the value of parameter S determined according to the state synchronization protocol. The present embodiment considers a scenario in which the number of normal nodes is greater than S0.
After the value of the parameter S is modified to T-SN, the synchronization is successful as long as T-SN nodes return success during synchronization, and the T-SN nodes can not comprise low-performance nodes, so that the performance of the node cluster can be matched with the high-performance nodes and cannot be reduced by the low-performance nodes.
In this embodiment, the performance improvement processing may further include at least one of the following processing:
processing one, modifying the value of a parameter E stored by the configuration center and the nodes in the node cluster into SN + 1;
taking Paxos protocol as an example, and S0 ═ floor (T/2) +1 as an example, if more than half of nodes are low-performance nodes, performance improvement processing needs to be performed, the value of S is modified to T-SN and is smaller than the value S0 specified by the protocol, and in order to avoid generating inconsistency and loss of log data in extreme cases, the parameter E is modified to SN +1, so as to ensure that at least one node participating in the previous synchronization participates in election.
The modification of the value of the parameter S, E stored by the node in the node cluster can be realized by sending a modification request of the value of the parameter S, E to the master node in the node cluster, wherein the request carries the values T-SN and SN +1 to be modified.
And processing II, if the current master node is a low-performance node, switching the master node to another node except the low-performance node in the node cluster.
In this embodiment, the node cluster is a metadata node cluster running a Paxos protocol or a protocol derived from the Paxos protocol in a distributed storage system, where the synchronization refers to log synchronization, S0 ═ floor (T/2) +1, and floor () represents rounding down. However, the solution of this embodiment may also be used for node clusters providing other services.
This embodiment also provides an operation and maintenance management system, including a performance management module, where the performance management module is configured to perform performance improvement processing on a node cluster running a state synchronization protocol, as shown in fig. 5, including:
the first processing unit 50 is used for modifying the value of the parameter S stored in the configuration center into T-SN;
a second processing unit 60, configured to modify a value of the parameter S stored in the node cluster to T-SN;
wherein T is the number of nodes of the node cluster, T is more than or equal to 2, SN is the number of low-performance nodes in the node cluster, and parameter S represents the minimum number of successful synchronization nodes required by the node cluster for providing normal service.
Alternatively,
the performance management module further comprises at least one of the following elements:
the third processing unit is used for modifying the values of a parameter E stored by the configuration center and the nodes in the node cluster into SN +1, wherein the parameter E represents the minimum number of normal nodes required by successful election of the node cluster;
and the fourth processing unit is configured to switch the master node to another node in the node cluster except the low-performance node when the current master node is the low-performance node.
Alternatively,
the node cluster is a metadata node cluster running a Paxos protocol or a derivative protocol thereof in the distributed storage system, and the synchronization refers to log synchronization.
In this embodiment, when the performance of the cluster is reduced due to the performance reduction of more metadata nodes, the service performance of the cluster can be improved by modifying the parameters.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the embodiments of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (21)
1. A method for recovering service is applied to an operation and maintenance management system and comprises the following steps:
detecting the state change of nodes in a node cluster of an operating state synchronization protocol, and determining the number NN of normal nodes, wherein NN is an integer;
and if the NN is changed from being greater than or equal to S0 to being smaller than S0, performing emergency processing to restore normal service, wherein when the normal node comprises the main node, the emergency processing comprises: modifying the values of the parameters S stored in the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol.
2. The method of claim 1, wherein:
when the normal node includes a master node, the emergency processing further includes: and modifying the values of a parameter E stored in the configuration center and the normal nodes to be T-NN '+ 1, wherein the parameter E represents the minimum number of the normal nodes required by the successful election of the node cluster, and NN' is the positive integer value which is modified by the parameter S and is less than or equal to NN.
3. The method of claim 2, wherein:
the performing emergency processing to recover normal service, wherein when the normal node does not include a master node, the emergency processing includes:
synchronizing local logs of at least FN-floor (T/2) abnormal nodes to a normal node, and modifying the values of parameters E stored in the configuration center and the normal node into NN after synchronization is successful; and
after the NN normal nodes elect a new main node, stopping service, executing emergency processing when the normal nodes comprise the main node, and recovering normal service after execution is completed;
wherein, T is the node number of the node cluster, T is more than or equal to 2, FN is the number of abnormal nodes in the node cluster, FN is T-NN, and floor () represents rounding-down.
4. The method of claim 3, wherein:
stopping service after the NN normal nodes elect a new master node, and the method is realized by the following steps: before a new master node is elected, the values of the parameters S stored in the configuration center and the normal nodes are modified to be larger than or equal to the value of FN + 1.
5. The method of any of claims 2-4, wherein:
after the NN is changed from S0 or more to S0 or less, the emergency processing is performed and the normal service is resumed, the method further includes: when the NN is changed again, the changed NN is compared with S0:
if NN < S0, the emergency process is resumed to resume normal service;
if NN is not less than S0, modifying the values of the parameter S and the parameter E stored by the configuration center and the normal node into S0 and E0 respectively;
where E0 is the value of parameter E determined according to the state synchronization protocol.
6. The method of claim 5, wherein:
the node cluster is a metadata node cluster running a Paxos protocol or a derivative protocol thereof in the distributed storage system, the synchronization refers to log synchronization, S0 ═ E0 ═ floor (T/2) +1, and floor () represents rounding down.
7. The method of any of claims 2-4, 6, wherein:
and in the process of modifying the values of the parameters S and/or E stored in the configuration center and the normal nodes, prohibiting abnormal nodes from starting or prohibiting the configuration center from disconnecting from the abnormal nodes.
8. The operation and maintenance management system is characterized by comprising a state detection module, a control module and an emergency processing module, wherein:
the state detection module is used for detecting the state change of the nodes in the node cluster of the running state synchronization protocol, determining the number NN of normal nodes and informing the control module, wherein NN is an integer;
the control module is used for calling the emergency processing module to perform emergency processing after the NN is changed from being greater than or equal to S0 to being smaller than S0 so as to recover normal service;
the emergency processing module is configured to, when the normal node includes a master node, perform the following emergency processing: modifying the values of the parameters S stored in the configuration center and the normal nodes into positive integer values smaller than or equal to NN;
wherein the parameter S represents the minimum number of synchronization successful nodes required by the node cluster to provide normal service, and S0 is the value of the parameter S determined according to the state synchronization protocol.
9. The operation and maintenance management system according to claim 8, wherein:
when the normal node includes the master node, the emergency processing module executes emergency processing further including: and modifying the values of a parameter E stored in the configuration center and the normal nodes to be T-NN '+ 1, wherein the parameter E represents the minimum number of the normal nodes required by the successful election of the node cluster, and NN' is the positive integer value which is modified by the parameter S and is less than or equal to NN.
10. The operation and maintenance management system according to claim 9, wherein:
the emergency processing module is further configured to, when the normal node does not include the master node, perform the following emergency processing: synchronizing local logs of at least FN-floor (T/2) abnormal nodes to a normal node, and modifying the values of parameters E stored in the configuration center and the normal node into NN after synchronization is successful; and after the NN normal nodes select new master nodes, stopping service, executing emergency processing when the normal nodes comprise the master nodes, and recovering normal service after execution is finished, wherein T is the number of the nodes in the node cluster, T is not less than 2, FN is the number of abnormal nodes in the node cluster, FN is T-NN, and floor () represents rounding-down.
11. The operation and maintenance management system according to claim 10, wherein:
the emergency processing module stops service after the NN normal nodes select a new main node, and the method is realized by the following steps: before a new master node is elected, the values of the parameters S stored in the configuration center and the normal nodes are modified to be larger than or equal to the value of FN + 1.
12. The operation and maintenance management system according to any one of claims 8-11, wherein:
the control module is further configured to compare the changed NN with S0 when the NN notified by the state detection module is changed again:
if NN < S0, calling an emergency processing module to perform emergency processing again to restore normal service;
if NN is not less than S0, modifying the values of the parameter S and the parameter E stored by the configuration center and the normal node into S0 and E0 respectively;
where E0 is the value of parameter E determined according to the state synchronization protocol.
13. The operation and maintenance management system according to claim 12, wherein:
the node cluster is a metadata node cluster running a Paxos protocol or a derivative protocol thereof in the distributed storage system, the synchronization refers to log synchronization, S0 ═ E0 ═ floor (T/2) +1, and floor () represents rounding down.
14. The operation and maintenance management system according to any one of claims 8-11, 13, wherein:
and in the process that the emergency processing module modifies the values of the parameters S and/or E stored in the configuration center and the normal nodes, the emergency processing module prohibits the abnormal nodes from starting or disconnecting the configuration center and the abnormal nodes.
15. A method of performance enhancement, comprising:
determining low-performance nodes in a node cluster running a state synchronization protocol;
when at least one low-performance node is successfully synchronized and the synchronization process of the node cluster can be successful, performing performance improvement processing, including: modifying values of a parameter S stored by nodes in a configuration center and a node cluster, wherein the modified value of S is not more than the number of other nodes except for low-performance nodes in the node cluster, so that the synchronization process can be successful when the low-performance nodes are not successfully synchronized,wherein S represents a synchronization success node required for the node cluster to provide normal service Minimum number of。
16. The method of claim 15, wherein:
when at least one low-performance node is successfully synchronized and the synchronization process of the node cluster can be successful, performing performance improvement processing, including: when the number SN of the low-performance nodes in the node cluster is more than or equal to T-S0+1, performing performance improvement processing;
the modifying of the values of the parameter S stored by the configuration center and the nodes in the node cluster includes: the operation and maintenance management system modifies the values of the parameters S stored in the configuration center and the nodes in the node cluster into T-SN;
wherein T is the number of nodes of the node cluster, T is more than or equal to 2, and S0 is the value of the parameter S determined according to the state synchronization protocol.
17. The method of claim 16, wherein:
the performance enhancement process further comprises at least one of:
modifying the values of the parameter E stored in the configuration center and the nodes in the node cluster into SN + 1; and
if the current master node is a low-performance node, switching the master node to another node except the low-performance node in the node cluster;
wherein the parameter E represents the minimum number of normal nodes required for successful election of the node cluster.
18. The method of claim 16 or 17, wherein:
the node cluster is a metadata node cluster running a Paxos protocol or a derivative protocol thereof in the distributed storage system, the synchronization refers to log synchronization, S0 is floor (T/2) +1, and floor () represents rounding-down.
19. An operation and maintenance management system, comprising a performance management module, wherein:
the performance management module is used for performing performance improvement processing on the node cluster of the running state synchronization protocol, and comprises the following steps:
the first processing unit is used for modifying the value of the parameter S stored in the configuration center into T-SN;
the second processing unit is used for modifying the value of the parameter S stored by the nodes in the node cluster into T-SN;
wherein T is the number of nodes of the node cluster, T is more than or equal to 2, SN is the number of low-performance nodes in the node cluster, and parameter S represents the minimum number of successful synchronization nodes required by the node cluster for providing normal service.
20. The operation and maintenance management system according to claim 19, wherein:
the performance management module further comprises at least one of the following elements:
the third processing unit is used for modifying the values of a parameter E stored by the configuration center and the nodes in the node cluster into SN +1, wherein the parameter E represents the minimum number of normal nodes required by successful election of the node cluster;
and the fourth processing unit is configured to switch the master node to another node in the node cluster except the low-performance node when the current master node is the low-performance node.
21. The operation and maintenance management system according to claim 19 or 20, wherein:
the node cluster is a metadata node cluster running a Paxos protocol or a derivative protocol thereof in the distributed storage system, and the synchronization refers to log synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610140348.XA CN107181608B (en) | 2016-03-11 | 2016-03-11 | Method for recovering service and improving performance and operation and maintenance management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610140348.XA CN107181608B (en) | 2016-03-11 | 2016-03-11 | Method for recovering service and improving performance and operation and maintenance management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107181608A CN107181608A (en) | 2017-09-19 |
| CN107181608B true CN107181608B (en) | 2020-06-09 |
Family
ID=59830377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610140348.XA Active CN107181608B (en) | 2016-03-11 | 2016-03-11 | Method for recovering service and improving performance and operation and maintenance management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107181608B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108984635A (en) * | 2018-06-21 | 2018-12-11 | 郑州云海信息技术有限公司 | A kind of HDFS storage system and date storage method |
| CN109167690A (en) * | 2018-09-25 | 2019-01-08 | 郑州云海信息技术有限公司 | A kind of restoration methods, device and the relevant device of the service of distributed system interior joint |
| CN114328098B (en) * | 2021-12-23 | 2023-04-18 | 北京百度网讯科技有限公司 | Slow node detection method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1107119A2 (en) * | 1999-12-02 | 2001-06-13 | Sun Microsystems, Inc. | Extending cluster membership and quorum determinations to intelligent storage systems |
| US6526432B1 (en) * | 1999-08-31 | 2003-02-25 | International Business Machines Corporation | Relaxed quorum determination for a quorum based operation of a distributed computing system |
| CN1568467A (en) * | 2001-09-06 | 2005-01-19 | Bea系统公司 | Exactly once cache framework |
| CN1201245C (en) * | 1999-08-31 | 2005-05-11 | 国际商业机器公司 | Non-strict legal number decision based on legal number operation |
| US7120821B1 (en) * | 2003-07-24 | 2006-10-10 | Unisys Corporation | Method to revive and reconstitute majority node set clusters |
| WO2011134053A1 (en) * | 2010-04-26 | 2011-11-03 | Locationary, Inc. | Method and system for distributed data verification |
| CN104077181A (en) * | 2014-06-26 | 2014-10-01 | 国电南瑞科技股份有限公司 | Status consistent maintaining method applicable to distributed task management system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6904448B2 (en) * | 2001-12-20 | 2005-06-07 | International Business Machines Corporation | Dynamic quorum adjustment |
-
2016
- 2016-03-11 CN CN201610140348.XA patent/CN107181608B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6526432B1 (en) * | 1999-08-31 | 2003-02-25 | International Business Machines Corporation | Relaxed quorum determination for a quorum based operation of a distributed computing system |
| CN1201245C (en) * | 1999-08-31 | 2005-05-11 | 国际商业机器公司 | Non-strict legal number decision based on legal number operation |
| EP1107119A2 (en) * | 1999-12-02 | 2001-06-13 | Sun Microsystems, Inc. | Extending cluster membership and quorum determinations to intelligent storage systems |
| CN1568467A (en) * | 2001-09-06 | 2005-01-19 | Bea系统公司 | Exactly once cache framework |
| US7120821B1 (en) * | 2003-07-24 | 2006-10-10 | Unisys Corporation | Method to revive and reconstitute majority node set clusters |
| WO2011134053A1 (en) * | 2010-04-26 | 2011-11-03 | Locationary, Inc. | Method and system for distributed data verification |
| CN104077181A (en) * | 2014-06-26 | 2014-10-01 | 国电南瑞科技股份有限公司 | Status consistent maintaining method applicable to distributed task management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107181608A (en) | 2017-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106331098B (en) | Server cluster system | |
| CN110807064B (en) | Data recovery device in RAC distributed database cluster system | |
| US20210320977A1 (en) | Method and apparatus for implementing data consistency, server, and terminal | |
| CN107329859B (en) | Data protection method and storage device | |
| CN113965578B (en) | Election method, device, equipment and storage medium of master node in cluster | |
| CN102394914A (en) | Cluster brain-split processing method and device | |
| CN107181608B (en) | Method for recovering service and improving performance and operation and maintenance management system | |
| JP3808874B2 (en) | Distributed system and multiplexing control method | |
| JP7532524B2 (en) | Method for storing data in a storage system, method for reading data, data storage device, data reading device, storage device, and system | |
| CN110635941A (en) | Database node cluster fault migration method and device | |
| CN113438111A (en) | Method for restoring RabbitMQ network partition based on Raft distribution and application | |
| WO2024008156A1 (en) | Database system, and master database election method and apparatus | |
| CN114138732A (en) | Data processing method and device | |
| CN111752488A (en) | Management method and device of storage cluster, management node and storage medium | |
| CN113986450A (en) | Virtual machine backup method and device | |
| CN118018463A (en) | Fault processing method, device, equipment and readable storage medium | |
| CN113794765A (en) | Gate load balancing method and device based on file transmission | |
| WO2018076696A1 (en) | Data synchronization method and out-of-band management device | |
| CN108984602B (en) | Database control method and database system | |
| CN111367998A (en) | Database cluster recovery method based on Galera and terminal equipment | |
| CN114124803B (en) | Device management method and device, electronic device and storage medium | |
| CN116346582A (en) | Method, device, equipment and storage medium for realizing redundancy of main network and standby network | |
| CN113064768B (en) | Method and device for switching fragment nodes in block chain system | |
| CN112202601B (en) | Application method of two physical node mongo clusters operated in duplicate set mode | |
| CN113961398A (en) | Business processing method, device, system, equipment, storage medium and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210402 Address after: Room 508, floor 5, building 4, No. 699, Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Patentee after: Alibaba (China) Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Patentee before: Alibaba Group Holding Ltd. |
|
| TR01 | Transfer of patent right |