CN110351122B

CN110351122B - Disaster recovery method, device, system and electronic equipment

Info

Publication number: CN110351122B
Application number: CN201910521769.0A
Authority: CN
Inventors: 侯焯明; 刘林
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-06-17
Filing date: 2019-06-17
Publication date: 2022-02-25
Anticipated expiration: 2039-06-17
Also published as: CN110351122A

Abstract

The disclosure provides a disaster recovery method, a disaster recovery device and electronic equipment, and relates to the technical field of computers. The disaster recovery method comprises the following steps: sending first information from a client to N peer service nodes, wherein N is an integer greater than or equal to 2; receiving M second information from the N peer service nodes, wherein M is a positive integer less than or equal to N; and sending one of the M pieces of second information to the client. The disaster recovery method provided by the disclosure can improve the reliability of data transmission between the client and the server and improve the disaster recovery capability.

Description

Disaster recovery method, device, system and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a disaster recovery method, apparatus, system, and electronic device.

Background

Disaster recovery is an operation scheme designed to ensure normal operation of an information system and implement service continuity when a disaster occurs. In the related art, a main/standby service node switching method is usually used to implement disaster recovery, that is, when a main service node fails, a backup service node is upgraded to a new main service node to take over the work of the original main service node. The primary/standby switching method is generally implemented by manual or third-party services.

In the process of manually switching the main service node and the backup service node, each service node is in an unavailable state, and the service node is in the unavailable state for a long time due to low manual efficiency, so that the defects are large.

The method for switching the main service node and the standby service node based on the third-party service needs to register the service node into the third-party service, the main service node is selected by the third-party service or a new main service node is selected when the main service node fails, the service node monitors the state change information of the main service node and restores the corresponding state information when the backup service node is upgraded to the main service node. The method needs to add monitoring logic of a main service node switching event and recovery logic of a service state during main service node switching on a code of a service node, so that the code invasion of the original service is large, and the operation and maintenance risk is increased; when a new service is added to a service node, a state recovery code of the new service in the process of recovering the state of the service node needs to be added, which is not beneficial to maintaining the stability of the system. Meanwhile, the method needs to maintain a third-party service, and increases the operation and maintenance cost. Finally, in the same way as manual method, there is a service unavailable state in the process of switching between the main and standby nodes, and for a service node with a state, when switching the service node, the state information of the main service node needs to be synchronized to the backup service node, so that the reliability of data is low.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The embodiment of the disclosure provides a disaster recovery method, a disaster recovery device, a disaster recovery system and an electronic device, which are used for overcoming the problems of unavailable service state, low reliability and the like of a disaster recovery scheme caused by the limitations and the defects of the related art at least to a certain extent.

According to an aspect of the present disclosure, a disaster recovery method is provided, which is performed by a service cluster disposed in a server, where the service cluster includes a primary management node and one or more backup management nodes, the primary management node is configured to perform the disaster recovery method, and the backup management nodes are configured to serve as new candidate objects of the primary management node when a failure occurs in the primary management node, where the disaster recovery method includes:

sending first information from a client to N peer service nodes, wherein N is an integer greater than or equal to 2;

receiving M second information from the N peer service nodes, wherein M is an integer less than or equal to N;

and sending one of the M pieces of second information to the client.

In an exemplary embodiment of the present disclosure, the sending one of the M second information to the client includes:

and sending the second information with the earliest arrival time to the client.

In an exemplary embodiment of the present disclosure, the sending one of the M second information to the client further includes:

if the second information with the later arrival time is the same as the second information with the earliest arrival time, discarding the second information with the later arrival time;

and if the second information with the later arrival time is different from the second information with the earliest arrival time, informing the service node corresponding to the second information with the later arrival time to close.

In an exemplary embodiment of the disclosure, the sending the first information from the client to the N peer serving nodes comprises: sending the first information to all the backup management nodes; the sending one of the M second information to the client further comprises: and sending one of the M pieces of second information to all the backup management nodes.

In an exemplary embodiment of the present disclosure, further comprising:

determining an identifier of a new service node to be registered in response to a service node registration request, wherein the service node registration request comprises an execution file encryption value of the new service node;

if the registered service node corresponding to the identifier does not exist, starting an isolated node registration process for the new service node;

if the registered service node corresponding to the identifier exists, comparing whether the execution file encryption value of the new service node is consistent with the execution file encryption value of the registered service node;

if the service nodes are consistent, starting a redundant node registration process for the new service node;

and if the service node registration request is inconsistent, rejecting the service node registration request.

In an exemplary embodiment of the present disclosure, the orphan node registration procedure includes:

determining an absolute value x of a sequence number difference between the maximum executed instruction sequence number n1 of the new serving node and the maximum received instruction sequence number n2 in the cache corresponding to the identifier of the new serving node;

if x is equal to zero, registering the new serving node;

if x is smaller than or equal to a first preset value, calling x first information with sequence numbers of n 1-n 2 from a cache, sending the x first information to the new service node, and after the new service node executes the x first information, registering the new service node;

and if x is larger than the first preset value, rejecting the service node registration request, wherein n1 and n2 are positive integers.

In an exemplary embodiment of the present disclosure, the redundant node registration procedure includes:

determining an absolute value y of a sequence number difference between the maximum executed instruction sequence number n1 of the new serving node and the maximum executed instruction sequence number n3 of the registered serving node;

if y is equal to zero, registering the new serving node;

if y is smaller than or equal to a second preset value, calling y first information with sequence numbers of n 1-n 3 from a cache, sending the y first information to the new service node, and after the new service node executes the y first information, registering the new service node;

and if y is larger than the second preset value, copying state information from the registered service node, and then sending the copied state information to the new service node, and registering the new service node, wherein n1 and n3 are positive integers.

In an exemplary embodiment of the present disclosure, the registering the new serving node includes:

and issuing initialization information to the new service node, wherein the initialization information comprises a random seed and self-driven logic, and the self-driven logic comprises a timestamp of the management node.

In an exemplary embodiment of the disclosure, the sending the first information from the client to the N peer serving nodes comprises:

determining an identifier of a service node corresponding to the first information;

and sending the first information to the N peer service nodes according to the identifier.

In an exemplary embodiment of the disclosure, the second information includes a sequence number of the second information sent by the serving node, and the receiving M second information from the N peer serving nodes includes:

and determining second information corresponding to the first information according to the sequence number of the second information sent by the service node in a plurality of information from the service node.

In an exemplary embodiment of the disclosure, the N peer service nodes use a shared memory, and each of the service nodes has an independent storage space in the shared memory.

In an exemplary embodiment of the present disclosure, when a new service node is a service node that is restarted after being turned off, the new service node continues to use a storage space used in the shared memory before being turned off when being turned on again.

According to another aspect of the present disclosure, comprising:

the information distribution module is arranged for sending first information from the client to N peer service nodes, wherein N is an integer greater than or equal to 2;

an information receiving module configured to receive M second information from the N peer service nodes, where M is a positive integer less than or equal to N;

and the information sending module is configured to send one of the M pieces of second information to the client.

According to still another aspect of the present disclosure, there is provided a disaster recovery system including:

at least one client;

and the server cluster is coupled to the client and is provided with a service cluster, the service cluster comprises a main management node and more than one backup management node, the backup management node is used as a candidate object of a new main management node when the main management node fails, and the main management node is used for executing the disaster recovery method.

According to yet another aspect of the present disclosure, there is provided an electronic device including:

a memory; and

a processor coupled to the memory, the processor configured to perform the disaster recovery method as recited in any of the above, based on instructions stored in the memory.

The embodiment of the disclosure deploys a plurality of peer-to-peer service nodes, simultaneously distributes the first information to be processed to the plurality of peer-to-peer service nodes, and determines the fed-back second information according to the processing result of each service node, so that when one or more service nodes fail, the normal service nodes can be used for guaranteeing the normal processing of the service, and the service unavailable state existing in the related main/standby switching disaster recovery method is eliminated.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

Fig. 1 is a schematic diagram of a disaster recovery system in an exemplary embodiment of the present disclosure.

Fig. 2 is a flowchart of a disaster recovery method in an exemplary embodiment of the present disclosure.

Fig. 3 is a flow chart of determining feedback information for the first information in one embodiment of the present disclosure.

Fig. 4 is a schematic diagram of a method for implementing disaster recovery by a service cluster in an embodiment of the present disclosure.

Fig. 5 is a schematic diagram of a service cluster implemented based on Paxos protocol in one embodiment of the present disclosure.

Fig. 6 is a flow chart of registration of a service node in one embodiment of the present disclosure.

Fig. 7A is a flow diagram of an orphan node registration process when a serving node has a state in one embodiment of the disclosure.

Fig. 7B is a flow diagram of a redundant node registration procedure when a serving node has a status in one embodiment of the present disclosure.

Fig. 8 is a schematic diagram of interaction in a service node registration process in an embodiment of the present disclosure.

Fig. 9 is a schematic diagram of node registration in one embodiment of the present disclosure.

Fig. 10 is a schematic diagram of initializing a service node in one embodiment of the disclosure.

Fig. 11 is a schematic diagram of a Master management node (Master Gdriver) managing an uplink data packet according to an embodiment of the disclosure.

Fig. 12 is a schematic diagram of a Master management node (Master Gdriver) managing a downstream data packet according to an embodiment of the disclosure.

Fig. 13 is a schematic diagram of an application scenario of the present disclosure.

Fig. 14 is a block diagram of a disaster recovery device in an exemplary embodiment of the present disclosure.

FIG. 15 is a block diagram of an electronic device in an exemplary embodiment of the disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Further, the drawings are merely schematic illustrations of the present disclosure, and the same or similar parts are denoted by the same reference numerals in the drawings, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

Fig. 1 is a schematic diagram of a disaster recovery system 100 according to an embodiment of the present disclosure. The disaster recovery method or the disaster recovery apparatus according to the embodiment of the present disclosure may be applied to the disaster recovery system 100.

Referring to fig. 1, a disaster recovery system 100 may include:

at least one client A;

and the server cluster B is coupled to the client A and is provided with a service cluster, the service cluster comprises a main management node and more than one backup management node, the backup management node is used as a candidate object of a new main management node when the main management node fails, and the main management node is used for executing the disaster recovery method of the following embodiment.

As shown in fig. 1, the client a may be, for example, various electronic devices having a display screen, including but not limited to a smart phone, a tablet computer, a portable computer, a desktop computer, and the like. The client a and the server cluster B may be coupled by a medium providing a communication link using a network, so that the client a can send the first information to the server cluster B or receive the second information sent by the server cluster B. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

It should be understood that the number of clients a, networks, and server clusters B in fig. 1 is merely illustrative. There may be any number of terminals, networks, and servers, as desired for an implementation.

A plurality of peer service nodes, which may be processes of an application, for example, may be provided in the server cluster B, and a plurality of service nodes are provided in one or more servers.

The disaster recovery method provided by the embodiment of the present disclosure may be executed by the server cluster B, and accordingly, the disaster recovery device may be disposed in the server cluster B.

The following detailed description of exemplary embodiments of the disclosure refers to the accompanying drawings.

Fig. 2 is a flowchart of a disaster recovery method in an exemplary embodiment of the present disclosure. Referring to fig. 2, a disaster recovery method 200 may include:

step S1, sending the first information from the client to N peer service nodes, where N is an integer greater than or equal to 2.

Step S2, receiving M second messages from the N peer service nodes, where M is a positive integer less than or equal to N.

Step S3, sending one of the M pieces of second information to the client.

In the embodiment of the present disclosure, the form of the first information and the second information may be, for example, a message/data packet, and when the first information and the second information are data packets, the first information may be referred to as an uplink data packet, and the second information may be referred to as a downlink data packet.

The method comprises the steps that a plurality of peer-to-peer service nodes are deployed, information to be processed is simultaneously distributed to the peer-to-peer service nodes, the information sent to a client side is determined according to redundant information sent by the peer-to-peer service nodes, and when a part of service nodes are in failure, normal processing of services can be guaranteed through other normal service nodes, and the service unavailable state existing in a related main/standby switching disaster recovery method is eliminated; moreover, because only the input and the output of the information are managed, the intrusion of the service code is less; because the information processed by the service nodes is the same, the service processing and the state updating can be kept when one or more service nodes are in failure, and the system unavailable disaster caused by the failure of the service nodes and the loss of the service state can be avoided.

The steps of the embodiment shown in FIG. 2 will be described in detail below.

In step S1, the first information from the client is sent to N peer service nodes, where N is an integer greater than or equal to 2.

In the embodiment of the present disclosure, a peer service node refers to a service node having consistent execution logic and consistent initialization conditions. In some embodiments, the N peer-to-peer service nodes may be deployed on one or more servers.

When the service node is a process of an application program, starting the application program correspondingly starts one service node. There may be multiple service nodes on the server corresponding to multiple applications at the same time. At this time, an identifier (for example, a number or a name of the application program) of the service node corresponding to the first information may be determined first, and then the first information may be sent to the N peer service nodes corresponding to the identifier.

In step S2, M second messages from the N peer serving nodes are received, where M is a positive integer less than or equal to N.

In this embodiment of the present disclosure, the second information may be information that is actively sent to the client by the service node, or may be feedback information that is sent to the client by the service node in response to the first information, which is not limited in this disclosure.

In an exemplary embodiment, the second information includes an identifier of a service node that transmits the second information and a sequence number of the second information transmitted by the service node, and the second information corresponding to the first information may be determined according to the sequence number of the second information transmitted by the service node among the plurality of information from the service node.

For example, the second information sent by the

service nodes

1, 2, 3, … …, and N respectively includes sequence numbers 99, … …, and 99 of the second information sent by the service node, where 99 indicates that the second information is the 99 th information sent by the service node 1, and other service nodes are the same. The above numerical values are merely examples, and the present disclosure is not limited thereto.

In step S3, one of the M pieces of second information is sent to the client.

In the embodiment of the present disclosure, the second information that arrives earliest may be sent to the client, if the second information that arrives later is the same as the second information that arrives earliest, the second information that arrives later is discarded, otherwise, the service node corresponding to the second information that arrives later is notified to close.

Because the service nodes are equal, if all the service nodes normally operate, the second information (downlink data packets) should be all the same, at the moment, redundancy is eliminated, and the first arrived second information is sent to the client; if part of the service nodes have faults, for example, the second information with errors is sent or the second information is not sent, the second information with the later arrival time is checked by taking the first second information as a standard, and as long as the second information with the later arrival time is different from the second information with the earliest arrival time, the second information with the later arrival time is judged to be the error information, and the service node corresponding to the second information with the later arrival time is informed to be closed.

Referring to fig. 3, step S3 may include:

in step S31, the second information with the earliest arrival time is sent to the client.

Step S32, judging whether the second information with the later arrival time is the same as the second information with the earliest arrival time, if so, entering the step S33; otherwise, the process proceeds to step S34.

In step S33, the second information having the later arrival time is discarded.

Step S34, the service node corresponding to the second information with the later arrival time is notified to close.

The service node corresponding to the second information with the later arrival time may be notified to close according to the identifier of the service node included in the second information.

If the second information is the feedback information of the service node to the first information, the second information output by each peer service node should be the same because the peer and input information of the service node is the same; if the second information is the information which is actively sent to the client by the service node, the second information output by each peer service node should be the same.

However, in some cases, there may be various errors such as an information reception error, an information processing error, and an information transmission error due to a service node failure, and further, the second information transmitted by the failed service node is different from the second information transmitted by other normal service nodes, and at this time, it is necessary to identify the failed service node by the erroneous second information and notify the failed service node of shutdown.

Notifying the failure service node to close can improve the efficiency of identifying the second information next time, and the notifying manner can be, for example, notifying the service node to be offline, or clearing the registration information of the service node, and the like. In other embodiments, the service node corresponding to the notification error information may be restarted, and the like, and a person skilled in the art may set the service node according to an actual situation.

In the method provided in the foregoing embodiment, because the second information is redundant, as long as not all service nodes have a failure, normal transmission of the feedback information is not affected. Even if each service node is a stateful node, since the peer service nodes have the same input, the state change process and the latest state of each peer service node are the same, and even if part of the service nodes fail, the state of other service nodes is not lost or not synchronized. Therefore, compared with the defect of unavailable service caused by switching of the main service node and the standby service node in the related technology, or the condition that the service nodes cannot be accurately synchronized when the service nodes have faults, the technical scheme provided by the embodiment of the disclosure has higher reliability and better user experience.

In one embodiment of the present disclosure, the disaster recovery method 200 may be performed by a service cluster disposed in a server.

Fig. 4 is a schematic diagram of implementing the disaster recovery method 200 by the service cluster 400 in one embodiment of the present disclosure.

Referring to fig. 4, the service cluster 400 includes a primary management node 41 and one or more backup management nodes 42, where the primary management node 41 is configured to perform the disaster recovery method 200, and the backup management node 42 is configured to be a candidate of a new primary management node when the primary management node 41 fails.

The service cluster 400 may be located on one or more servers and the management nodes may be located on the same or different servers. When the management nodes are located on different servers, the service cluster 400 may set a communication channel between the servers so as to implement data synchronization of the management nodes.

In operation, the service cluster 400 first selects one management node from the plurality of management nodes as the primary management node 41, and sets the other management nodes as the backup management nodes 42. In order that the backup management node 42 can take over working and synchronizing data in time when the primary management node 41 fails, the primary management node 41 may synchronize all backup management nodes through a service cluster protocol when receiving information sent by the client 1 or the service node 2.

As shown in fig. 4, the client 1 and the service node 2 interact only with the master management node 41. The master management node 41 outputs information to the client 1 and the service node 2 and receives information transmitted by the client 1 and the service node 2. Meanwhile, the master management node 41 submits the input information and the output information of the client 1 and the service node 2 to the service cluster 400 through the service cluster protocol, for example, the master management node 41 may submit the input information and the output information of the client 1 and the service node 2 to the service cluster 400 through the Paxos protocol so as to synchronize the first information or the second information to all backup management nodes. In some embodiments, when receiving the first information, the primary management node 41 may first synchronize the first information to all backup management nodes, and after receiving the response of the backup management nodes, distribute the first information to the N peer service nodes 2; when receiving the second information, the primary management node 41 may synchronize the second information to all backup management nodes, and after receiving a response from the backup management node, send the second information to the client 1. Alternatively, the process of sending data to the client/service node and the process of synchronizing data to the backup management node may be performed simultaneously.

The service cluster 400 monitors the working state of the primary management node 41 in real time, and when the primary management node 41 fails, immediately selects a new primary management node from the plurality of backup management nodes 42, and automatically switches the routing addresses of the client 1 and the service node 2 to the new primary management node.

Since the service cluster 400 itself also supports disaster recovery, further guarantees are provided for the reliability of the system. In addition, since the service cluster 400 is added only by modifying the information transceiving logic, the code modification amount is small, the original service logic is less invaded, and the problems of large code modification amount, increased system fault risk and the like in the related technology can be effectively avoided.

In some embodiments, the service cluster 400 may be implemented based on Paxos protocol, and the management nodes may be implemented by, for example, gdiver nodes.

FIG. 5 is a schematic diagram of a service cluster implemented based on the Paxos protocol in one embodiment.

Referring to fig. 5, the Paxos cluster 500 elects a gdrver node as the Master management node 51(Master gdrver), and sets the routing addresses of the client and the service process as the address of the Master gdrver. The master management node 51 receives the first information (Log, which is both the binLog of Paxos and the traffic message) from the client 1, distributes the first information to a plurality of peer service nodes 20 (service nodes), and transmits one of a plurality of second information (Log) transmitted by the service node 20 to the client 10. The primary management node 51 and the backup management node 52 synchronize data through Paxos protocol, and the primary management node 51 synchronizes one of the first information received to the client 10 and the second information received to the service node 20 to all the backup management nodes 52.

Each management node has a local Log pool, the information sequence number in the Log pool is strictly increased, and the Paxos cluster ensures that the Log pools of all the management nodes are completely consistent.

In order to ensure that a plurality of peer service nodes are identical, in the embodiment of the present disclosure, the service cluster is further configured to manage synchronization, registration, and driving of each service node.

After the service node is started, the service node may register with the service cluster, so that the service cluster may send the first information to all registered service nodes when receiving the first information corresponding to the identifier of the service node. In order to ensure that registered service nodes corresponding to the same identifier are completely consistent, the service nodes may be verified during the registration process of the service nodes.

Referring to fig. 6, the registration procedure of the service node may include:

step S61, responding to the service node registration request to determine the identifier of the new service node to be registered, wherein the service node registration request comprises the execution file encryption value of the new service node;

step S62, judging whether there is registered service node corresponding to the identifier, if not, entering step S63 to start the isolated node registration process for the new service node, otherwise, entering step S64;

step S64, comparing whether the encryption value of the new service node execution file is consistent with the encryption value of the registered service node execution file, if so, going to step S65, otherwise, going to step S66;

step S65, starting a redundant node registration process for the new service node;

step S66, the service node registration request is rejected.

The execution file encryption value may be, for example, an MD5(Message-Digest Algorithm) value, which is not particularly limited by the present disclosure.

Executing the flow shown in fig. 6 can eliminate all nodes with execution files different from the registered nodes, and preliminarily ensures the consistency of the new service node and the registered service nodes. The isolated node registration procedure in step S63 and the redundant node registration procedure in step S65 may be to directly register a new serving node when the serving node does not have a status, but also to adjust the status of the new serving node when the serving node has a status.

Fig. 7A is a flow diagram of an orphan node registration flow when a serving node has a state.

Referring to fig. 7A, when the service node has a state, the orphan node registration procedure may include:

step S71, determining an absolute value x of a sequence number difference between the maximum executed command sequence number n1 of the new service node and the maximum received command sequence number n2 in the cache corresponding to the identifier of the new service node;

step S72, if x is equal to zero, registering the new serving node;

step S73, if x is less than or equal to a first preset value, calling x first information with sequence numbers of n 1-n 2 from a cache and sending the x first information to the new service node, and after the new service node executes the x first information, registering the new service node;

step S74, if x is greater than the first preset value, rejecting the service node registration request.

No matter whether the new service node is cold-started (the executed instruction record is zero) or hot-started (the executed instruction record before closing is restarted and loaded after closing), before the new service node is registered, the state of the new service node needs to be adjusted according to the first information (instruction) corresponding to the new service node, which is received by the main management node, so that the state of the new service node after successful registration is ensured to be normal, and correct feedback can be given to the new first information.

In order to facilitate management and recording of states of each service node before and after the restart, in an embodiment of the present disclosure, a shared memory used by multiple service nodes may be set, and an independent storage space is set for each service node in the shared memory. Therefore, when the service node is restarted, the storage space used in the shared memory before being closed can be continuously used, and the restarting service node can timely recover the state.

The main management node stores the received first information number in a cache (Log cache pool), the service node stores the processed first information number in a shared memory, reads the maximum executed instruction sequence number when restarting, and restores the state to the state before closing according to the record in the shared memory.

If the maximum executed instruction sequence number of the new service node is the same as the maximum received instruction sequence number in the cache, it indicates that the master management node does not receive the new first information in the process of closing and restarting the new service node or before the cold start of the new service node, and the new service node can be directly registered at this time.

If the difference between the maximum executed instruction sequence number of the new service node and the maximum received instruction sequence number in the cache is smaller (x is smaller than or equal to a first preset value), it indicates that the master management node receives a small amount of new first information in the closing and restarting process of the new service node or before the cold start of the new service node, at this time, the new first information (x) can be read from the cache and sent to the new service node in sequence, so that the new service node obtains the latest state after sequential processing, and then registers the new service node. The first preset value may be set by a person skilled in the art, for example, the first preset value is a maximum value of the first information amount that can be stored in the cache.

If the difference between the maximum executed instruction sequence number of the new service node and the maximum received instruction sequence number in the cache is larger (x is larger than a first preset value), it indicates that a large amount of new first information is received by the master management node in the closing and restarting process of the new service node or before the cold start of the new service node, and because the storage capacity of the cache is limited, all the new first information cannot be obtained, that is, the state of the new service node cannot be restored to the latest state, and at this time, the registration of the new service node can only be rejected.

Fig. 7B is a flow chart of a redundant node registration flow when the serving node has a status.

Referring to fig. 7B, when the serving node has a status, the redundant node registration procedure may include:

step S75, determining the absolute value y of the sequence number difference between the maximum executed command sequence number n1 of the new service node and the maximum executed command sequence number n3 of the registered service node;

step S76, if y is equal to zero, registering the new serving node;

step S77, if y is less than or equal to a second preset value, calling y first information with sequence numbers of n1 to n3 from a cache and sending the y first information to the new service node, and after the new service node executes the y first information, registering the new service node;

and step S78, if y is larger than the second preset value, copying state information from the registered service node and sending the state information to the new service node, and then registering the new service node.

When the registered service node has processed some first information, the state of each registered service node has changed, and at this time, the new service node needs to execute the first information processed by the registered service node as well, adjust the state of the new service node to be consistent with the registered service node in time, and register the new service node, so as to keep all the registered service nodes completely identical.

In some cases, the new service node is a service node that is restarted after being shut down, and at this time, the new service node may read data such as a node state before being shut down, and restore the state to the state before being shut down according to the data. In other cases, the start state of the new serving node may be a cold start with no information to perform the recording (i.e., n1 ═ 0).

When calculating the absolute value y of the sequence number difference between the maximum executed instruction sequence number n1 of the new service node and the maximum executed instruction sequence number n3 of the registered service node, if the new service node does not process new first information during the shutdown and restart process or during the cold start process, the maximum executed instruction sequence number n1 processed by the new service node may be the same as the maximum executed instruction sequence number n3 of the registered service node, that is, y is 0. The new serving node may be registered directly at this point.

In other cases, if the new service node processes only a small amount of new first information, for example, less than or equal to a second preset value, during the shutdown and restart process or during the cold start, the new first information (y) may be read from the cache and sequentially sent to the new service node, so that the new service node obtains a state consistent with the processed node after sequential processing, and then registers the new service node.

Since the amount of data that can be stored in the cache is small, when the new service node processes more new first information, for example, more than a second preset value, during the shutdown and restart process or during the cold start, other peer service nodes may copy the state information from the registered service node and copy the state information to the new service node, so as to achieve fast synchronization of the states of the new service node and the registered service node.

In some embodiments, the second preset value may be the same as or different from the first preset value. The second preset value may be set by a person skilled in the art, for example, the second preset value may be a maximum value of the number of first information that can be stored in the cache, or may be determined according to a threshold value of the maximum value of the number of first information that can be stored in the cache, a time T1 when the first information is read from the cache and synchronized to the new service node, and a time T2 when the state information is copied from the registered service node and copied to the new service node.

Referring to fig. 8, in the service node registration process, a service node registration request is first sent by the new service node to the master management node, where the service node registration request includes an Identifier (ID) of the new service node and an encrypted value of the execution file (MD 5).

The main management node judges whether the ID corresponds to a registered node, if not, judges whether the absolute value x of the sequence number difference between the maximum executed instruction sequence number n1 of the new service node and the maximum received instruction sequence number n2 of the identifier corresponding to the new service node in the cache is equal to zero, and if the absolute value x is equal to zero, registers the new service node and sends a service node registration message to the new service node; if not, judging whether x is smaller than or equal to a first preset value, if not, sending a service node registration rejection message to the service node, if so, reading the latest x pieces of first information from the cache and sending the latest x pieces of first information to the new service node, and after the new service node finishes executing the first information and sends a first information execution finishing message, registering the new service node and sending a service node registration message to the new service node.

If the ID corresponds to the registered node, judging whether the MD5 values of the new service node and the registered service node are consistent, if not, sending a service node registration rejection message to the new service node; if the absolute value y of the sequence number difference between the maximum executed instruction sequence number n1 of the new service node and the maximum executed instruction sequence number n3 of the registered service node is equal to zero, if the absolute value y is equal to zero, the new service node is registered and a service node registration message is sent to the new service node, and if the absolute value y is not equal to zero, whether y is smaller than a second preset value is judged. If y is smaller than or equal to a second preset value, reading latest y pieces of first information from the cache and sending the latest y pieces of first information to the new service node, and after the new service node finishes executing the first information and sends a first information execution finishing message, registering the new service node and sending a service node registration message to the new service node; and if y is larger than a second preset value, sending a state information copying instruction to the registered service node, and after the registered service node copies the state information and sends the state information to the new service node, the new service node fills the state information and sends feedback of the completion of the state information filling, registering the new service node and sending a service node registration message to the new service node.

Referring to fig. 9, the service node 92 during node registration may transmit an encrypted executable file value (MD5 value) and a maximum sequence number of executed instructions to the master management node 91 (gdiver) for the master management node 91 to verify whether the service node is identical to the registered node.

After the new service node is determined to be registered, in order to keep the service nodes which are equal to the identifier completely consistent, the service cluster can issue initialization information to the new service node, wherein the initialization information comprises a random seed and self-driven logic, and the self-driven logic comprises a timestamp of a management node.

For example, a service Node (Application Node) is used as a service Node, and a self-driven logic (Tick operation) and some time judgment logics are provided inside the service Node, and in order to ensure that the states of each service Node are consistent, it is necessary to unify the time and Tick logic of each service Node.

Referring to fig. 10, a Master management node 101(Master gdiver) sends Tick requests to a service node 102 through an uplink Log circular buffer pool, where each Tick request includes a timestamp of a service cluster. The time interval of the Tick request may be configured to be 10ms once, for example. The service node 102 replaces the system time function (hook time/gettimeoffday) with the time sent by the Master management node 101(Master Gdriver), and the "management setting time" in the figure is the time (accurate to millisecond) issued by the Master management node 101.

After the initialization is completed, all the service nodes are completely the same. With the above embodiment, each service node becomes an independent "state machine", each state machine has consistent initialization and consistent execution logic, and as long as consistent input is ensured, consistent response and data can be obtained, thereby realizing disaster recovery of stateful services.

Referring to fig. 11, the client 111 may set a smaller Log circular buffer pool locally (mainly for retransmission, which is typically used when switching management nodes). After a Log (a first message corresponds to a Log) generated by a client 111 is sent to a server, an increasing first sequence number is marked by a Master management node 112(Master gdrver), so that the Master management node 112(Master gdrver) can record the maximum first sequence number of the message sent by each client 111, detect whether a packet is missed according to whether the first sequence number of the Log sent by one client and the recorded maximum first sequence number are continuous, and judge that the packet is missed if the first sequence number is discontinuous, and notify the client to resend the Log. In some embodiments, the master management node may be skipped for missing packets for clients, so the capacity of the Log queue inside the client may be small and unnecessary.

It should be noted that, in the cache of the master management node, each client has a first sequence number, the identifier of each service node has a maximum sequence number of the received instruction, and the two sequence numbers correspond to different objects.

The Master management node 112(Master Gdriver) collects the logs sent by all the clients into an uplink Log queue for sending, and marks a new incremental second sequence number on each Log. Service node 113 records the maximum second sequence number of the executed Log, checks whether the Log sent by main management node 112 is missing, and if missing, notifies main management node 112 to resend the packet. The service node 113 has zero tolerance to missing packets, and cannot skip any data packet, and since the latest request packets of all the clients are cached, and the Log buffer pools of the management nodes and the main management node have the same capacity and content, the Log buffer pool of each management node can be set to be larger.

Referring to fig. 12, each service node 121 is provided with a local Log circular buffer pool (for retransmitting the downlink Log, the capacity may be set to be small, but not necessary). Each service node 121 locally adds a third sequence number that is incremented to a downlink data packet (second information), and a Master management node 122(Master Gdriver) determines that the downlink data packets are responses to the same uplink data packet (first information) or actively sent from the service nodes 121 in the same state according to the fact that the third sequence numbers are the same.

For a plurality of downlink data packets with the same third sequence number, the Master management node 122(Master Gdriver) may only take the first arriving downlink data packet to issue to the client 123, compare the later arriving downlink data packet with the first data packet with the same sequence number (for example, by using a memcmp function), and if the later arriving downlink data packet is not consistent with the first downlink data packet with the same sequence number, consider that the state of the service node corresponding to the later downlink data packet is in a problem, and may notify the service node corresponding to the later downlink data packet to go offline. The client 123 may locally record the maximum third sequence number of the received downlink data packet, further detect a missing packet according to whether the third sequence number is continuous, and notify a Master driver (Master gdrver) to retransmit (skip) the missing packet.

Referring to fig. 13, in one application scenario, the disclosed embodiments may be applied to a service cluster 11 of a mobile game. The service cluster 11 is implemented by a plurality of servers 13, the service cluster 11 is connected to a plurality of clients 12 through the servers 13, and the clients 12 may be mobile terminals such as mobile phones and tablet computers. In the embodiment shown in fig. 12, each server 13 is provided with, for example, one management node (a primary management node 111 or a backup management node 112) and one service node 131 in the service cluster 11.

A master management node 111 in the service cluster 11 receives first information sent by a client 12 and distributes the first information to a plurality of peer service nodes 131 in a plurality of servers 13; the master management node 111 sends one of the second information sent by the peer service nodes 131 to the client 12, so that when a part of servers or service nodes fail, it can be guaranteed that the instructions of the client 12 can be correctly executed and correct feedback can be received, or that the service nodes can normally execute the message transmitted by the client 12.

In addition, when receiving the first information or the second information, the primary management node 111 synchronizes the first information and the second information to all backup management nodes 112, so that the service cluster 11 can select a new primary management node from the plurality of backup management nodes 112 when monitoring that the primary management node 111 fails, so that the backup management nodes can enter a working state in time. By the dual redundancy design of the service node and the management node, the disaster tolerance capability is improved, and the reliability of the system is improved.

In the design of a mobile game architecture, partial processes of a cell are often stateful and cannot be removed, in the operation process of the whole system, the stateful processes exist as single points, when one process fails, partial services are unavailable, and in the serious case, the cell is out of service.

When the disaster recovery method provided by the embodiment of the present disclosure is used to manage communication between a client and a server, the service cluster 11 distributes a message sent by the client 12 to a plurality of completely peer active service nodes 131, and sends the message to the client 12 according to a plurality of messages sent by the service nodes 131, even if a part of the service nodes or servers fail, the server cluster 11 can also select correct feedback information through feedback of the remaining active service nodes, which does not cause unavailability of services or loss of states of the service nodes, does not affect normal game experience of players, i.e., can implement disaster recovery of stateful service processes without removing states, and improves availability of the system. In addition, the method is also suitable for other businesses with the problem that the stateful service is a single point.

In summary, the method provided by the embodiment of the present disclosure can implement disaster recovery with stateful services, solve the problem of system failure caused by a single point failure with stateful services, and improve the availability of the entire system.

Corresponding to the above method embodiment, the present disclosure also provides a disaster recovery device, which can be used to execute the above method embodiment.

Referring to fig. 14, a disaster recovery device 1400 provided in an embodiment of the present disclosure may include: an information distribution module 1402, which may be configured to send a first information from a client to N peer service nodes, where N is an integer greater than or equal to 2; an information receiving module 1404, which may be configured to receive M second information from the N peer serving nodes, M being a positive integer less than or equal to N; the information sending module 1406 may be configured to send one of the M second information to the client.

In an exemplary embodiment, the disaster recovery apparatus 1400 is disposed in a service cluster, where the service cluster includes a primary management node for executing the disaster recovery method and one or more backup management nodes for serving as a candidate object of a new primary management node when the primary management node fails.

In an exemplary embodiment, the information distribution module 1402 is configured to: and determining an identifier of a service node corresponding to the first information, and sending the first information to the N peer service nodes according to the identifier.

In an exemplary embodiment, the second information includes a sequence number of the second information sent by the serving node, and the information receiving module 1404 is configured to: and determining second information corresponding to the first information according to the sequence number of the second information sent by the service node in a plurality of information from the service node.

In an exemplary embodiment, the information sending module 1406 is configured to: and sending the second information with the earliest arrival time to the client.

In an exemplary embodiment, the information sending module 1406 is configured to: and if the second information with the later arrival time is the same as the second information with the earliest arrival time, discarding the second information with the later arrival time, otherwise, informing the service node corresponding to the second information with the later arrival time to close.

In an exemplary embodiment, the disaster recovery apparatus 1400 further includes a node registration module 1408, and the node registration module 1408 is configured to: determining an identifier of a new service node to be registered in response to a service node registration request, wherein the service node registration request comprises an execution file encryption value of the new service node; if the registered service node corresponding to the identifier does not exist, starting an isolated node registration process for the new service node; otherwise, comparing whether the execution file encryption value of the new service node is consistent with the execution file encryption value of the registered service node; if the service nodes are consistent, starting a redundant node registration process for the new service node; and if the service node registration request is inconsistent, rejecting the service node registration request.

In an exemplary embodiment, the orphaned node registration process includes: determining an absolute value x of a sequence number difference between the maximum executed instruction sequence number n1 of the new serving node and the maximum received instruction sequence number n2 in the cache corresponding to the identifier of the new serving node; if x is equal to zero, registering a new service node; if x is smaller than or equal to a preset value, calling x first information with sequence numbers of n 1-n 2 from the cache and sending the first information to the new service node, and after the new service node executes the x first information, registering the new service node; and if x is larger than the preset value, rejecting the service node registration request.

In an exemplary embodiment, the redundant node registration procedure includes: determining an absolute value y of a sequence number difference between the maximum executed instruction sequence number n1 of the new service node and the maximum executed instruction sequence number n3 of the registered service node; if y is equal to zero, registering a new service node; if y is smaller than or equal to the preset value, calling y first information with sequence numbers of n 1-n 3 from the cache and sending the y first information to the new service node, and after the new service node executes the y first information, registering the new service node; and if y is larger than the preset value, copying the state information from the registered service node and sending the state information to the new service node, and then registering the new service node.

In an exemplary embodiment, the node registration module 1408 is configured to: and issuing initialization information to the new service node, wherein the initialization information comprises a random seed and self-driven logic, and the self-driven logic comprises a timestamp of the management node.

In an exemplary embodiment, the information distribution module 1402 is configured to: sending the first information to all the backup management nodes; the information sending module 1406 is configured to: and sending one of the M pieces of second information to all the backup management nodes.

In an exemplary embodiment, the N peer service nodes use a shared memory, and each of the service nodes has an independent storage space in the shared memory.

In an exemplary embodiment, when a new service node is a service node that is restarted after being shut down, the new service node continues to use the storage space used in the shared memory before being shut down when being restarted.

Since the functions of the apparatus 1400 have been described in detail in the corresponding method embodiments, the disclosure is not repeated herein.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

An electronic device 1500 according to this embodiment of the invention is described below with reference to fig. 15. The electronic device 1500 shown in fig. 15 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 15, electronic device 1500 is in the form of a general purpose computing device. Components of electronic device 1500 may include, but are not limited to: the at least one processing unit 1510, the at least one memory unit 1520, and the bus 1530 that connects the various system components (including the memory unit 1520 and the processing unit 1510).

Wherein the memory unit stores program code that is executable by the processing unit 1510 to cause the processing unit 1510 to perform steps according to various exemplary embodiments of the present invention as described in the above section "exemplary methods" of the present specification. For example, the processing unit 1510 may perform the steps as shown in fig. 2.

The storage unit 1520 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)15201 and/or a cache memory unit 15202, and may further include a read only memory unit (ROM) 15203.

Storage unit 1520 may also include a program/utility 15204 having a set (at least one) of program modules 15205, such program modules 15205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 1530 may be any bus representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 1500 can also communicate with one or more external devices 1600 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 1500, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 1500 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 1550. Also, the electronic device 1500 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 1560. As shown, the network adapter 1560 communicates with the other modules of the electronic device 1500 over the bus 1530. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

The above-described figures are merely schematic illustrations of the processes involved in the method according to an exemplary embodiment of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A disaster recovery method is characterized in that the disaster recovery method is executed by a service cluster arranged in a server, the service cluster comprises a main management node and more than one backup management node, the main management node is used for executing the disaster recovery method, the backup management node is used as a candidate object of a new main management node when the main management node fails, and the disaster recovery method comprises the following steps:

receiving M second information from the N peer service nodes, wherein M is a positive integer less than or equal to N;

sending one of the M pieces of second information to the client;

wherein the sending one of the M second information to the client comprises:

sending the second information with the earliest arrival time to the client;

2. The disaster recovery method as claimed in claim 1, wherein said sending the first information from the client to the N peer service nodes comprises: sending the first information to all the backup management nodes; the sending one of the M second information to the client further comprises: and sending one of the M pieces of second information to all the backup management nodes.

3. The disaster recovery method of claim 1, further comprising:

4. The disaster recovery method according to claim 3, wherein said orphan node registration procedure comprises:

if x is equal to zero, registering the new serving node;

5. The disaster recovery method as claimed in claim 3, wherein said redundant node registration procedure comprises:

if y is equal to zero, registering the new serving node;

6. The disaster recovery method of claim 4 or 5, wherein said registering said new serving node comprises:

7. The disaster recovery method as claimed in claim 1, wherein said sending the first information from the client to the N peer service nodes comprises:

8. The disaster recovery method as claimed in claim 1, wherein said second information comprises a sequence number of second information sent by said service node, and said receiving M second information from said N peer service nodes comprises:

9. The disaster recovery method as claimed in claim 1, wherein said N peer-to-peer service nodes use a shared memory, and each of said service nodes has an independent storage space in said shared memory.

10. The disaster recovery method according to claim 9, wherein when a new service node is a service node that is restarted after being shutdown, the new service node continues to use the storage space used in the shared memory before being shutdown when being restarted.

11. A disaster recovery device, comprising:

an information sending module configured to send one of the M second information to the client;

wherein the information sending module is configured to:

sending the second information with the earliest arrival time to the client;

12. A disaster recovery system, comprising:

at least one client;

a server cluster coupled to the client, wherein the server cluster is provided with a service cluster, the service cluster includes a primary management node and one or more backup management nodes, the backup management nodes are used as candidate objects of a new primary management node when the primary management node fails, and the primary management node is used for executing the disaster recovery method according to any one of claims 1 to 10.

13. An electronic device, comprising:

a memory; and

a processor coupled to the memory, the processor configured to perform the disaster recovery method of any of claims 1-10 based on instructions stored in the memory.