CN116684256B - Node fault monitoring method, device, system, electronic equipment and storage medium - Google Patents

Abstract

The present invention provides a node fault monitoring method, device, system, electronic equipment and storage medium, and relates to the field of computer technology. The method includes: sending a first heartbeat message to a second node in the distributed cluster system; receiving the first heartbeat message; According to the second heartbeat message returned by the second node, obtain the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node; The second heartbeat message is a response message to the first heartbeat message; and the fault monitoring result of the second node is obtained according to the current number of heartbeat timeouts and the current network connectivity status table. The present invention enables accurate analysis of faulty nodes in a sub-healthy network state, preventing misjudgment from causing normal nodes to perform failover and fault recovery actions, thereby improving the stability and reliability of node detection, and thereby improving the stability of the cluster. , safety and reliability.

Description

Node fault monitoring method, device, system, electronic equipment and storage medium

Technical field

The present invention relates to the field of computer technology, and in particular to a node fault monitoring method, device, system, electronic equipment and storage medium.

Background technique

A distributed cluster system is a cluster composed of multiple node servers. Each node runs a processing program. When the network status of one or several nodes is in a fault state, it will affect the performance of the entire distributed cluster system. Therefore, how to efficiently and accurately monitor failed nodes is an important issue that needs to be solved urgently in the industry.

In related technologies, PING (Packet Internet Groper, Internet packet explorer) or heartbeat monitoring is usually used to determine whether other nodes send response information to this node within a preset time period to determine whether other network nodes are abnormal nodes. In a sub-healthy network, due to unstable network connection status, the CTDB (Cluster Trivial Database) in the node with network abnormality will be mistakenly considered as if the response information transmitted by other nodes through PING or heartbeat monitoring is lost. There are faults in other nodes, which makes the node fault detection accuracy low, thus affecting the stability and reliability of the cluster system.

Contents of the invention

The present invention provides a node fault monitoring method, device, system, electronic equipment and storage medium to solve the shortcomings in the prior art of low node fault detection accuracy, thereby affecting the stability and reliability of the cluster, and to improve node fault detection. accuracy, thus improving the stability and reliability of the cluster system.

The present invention provides a node fault monitoring method, which is applied to the first node in a distributed cluster system, including:

Send the first heartbeat message to the second node in the distributed cluster system;

Receive the second heartbeat message returned by the second node, and obtain the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node according to the second heartbeat message. ;The second heartbeat message is a response message to the first heartbeat message;

Obtain the fault monitoring result of the second node according to the current number of heartbeat timeouts and the current network connectivity status table.

According to a node fault monitoring method provided by the present invention, obtaining the fault monitoring result of the second node based on the current number of heartbeat timeouts and the current network connectivity status table includes:

Compare the current number of heartbeat timeouts with a number threshold to obtain a first comparison result;

When it is determined that the current number of heartbeat timeouts is greater than the number threshold according to the first comparison result, it is determined according to the current network connectivity status table whether there is at least one third node in the distributed cluster system. The network connectivity state between the second nodes is a normal state;

According to the judgment result, obtain the fault monitoring result of the second node;

Wherein, the third node is a network node other than the first node and the second node in the distributed cluster system.

According to a node fault monitoring method provided by the present invention, obtaining the fault monitoring result of the second node according to the judgment result includes:

When it is determined according to the judgment result that there is no at least one network connection state between the third node and the second node in the distributed cluster system and the network connectivity state between the second node and the third node is in a normal state, it is determined that the second node The fault monitoring result is fault status.

According to a node fault monitoring method provided by the present invention, obtaining the fault monitoring result of the second node according to the judgment result includes:

When it is determined that there is at least one third node in the distributed cluster system and the network connectivity state between the third node and the second node is normal according to the judgment result, obtain the data corresponding to the second node. The number of reference nodes;

Obtain the fault monitoring result of the second node according to the number of reference nodes;

Wherein, the reference node is used to provide a response message for updating the current network connectivity status table of the second node within a preset period.

According to a node fault monitoring method provided by the present invention, obtaining the fault monitoring result of the second node according to the number of reference nodes includes:

Compare the number of reference nodes with a quantity threshold to obtain a second comparison result;

If it is determined that the number of reference nodes is greater than the number threshold according to the second comparison result, it is determined that the fault monitoring result of the second node is in a normal state.

According to a node fault monitoring method provided by the present invention, the method further includes:

If, according to the second comparison result, it is determined that the number of reference nodes is greater than the number threshold, trigger an isolation action;

Wherein, the isolation action is used to isolate the first node from other network nodes in the distributed cluster system except the first node, or to isolate the network port of the first node from the The network ports of other network nodes are isolated.

According to a node fault monitoring method provided by the present invention, the method further includes:

If it is determined that the number of reference nodes is less than or equal to the number threshold according to the second comparison result, it is determined that the fault monitoring result of the second node is in a fault state.

According to a node fault monitoring method provided by the present invention, the method further includes:

When it is determined that the fault monitoring result of the second node is in a fault state, a fourth node is obtained in the distributed cluster system; the fourth node has a fault monitoring result in a normal state and is in a normal state with the second node. Nodes are network nodes with the same service functions;

Migrate the pending tasks of the second node to the fourth node;

When the fault monitoring result of the second node switches from a fault state to a normal state, the pending task is restored to the second node.

According to a node fault monitoring method provided by the present invention, the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node are obtained according to the second heartbeat message. ,include:

Parse the second heartbeat message to obtain the current network connectivity status table;

Determine the current network connectivity state with the second node according to the current network connectivity state table;

Update the count value of the heartbeat timeout counter according to the current network connectivity status;

According to the updated count value, obtain the current number of heartbeat timeouts.

According to a node fault monitoring method provided by the present invention, updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

When it is determined that the current network connectivity state is an abnormal connectivity state, the count value of the heartbeat timeout counter is cumulatively increased by 1.

According to a node fault monitoring method provided by the present invention, updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

When it is determined that the current network connectivity state is a normal connectivity state, the count value of the heartbeat timeout counter is kept unchanged.

According to a node fault monitoring method provided by the present invention, determining the current network connectivity status with the second node based on the current network connectivity status table includes:

In the current network connectivity status table, search for connectivity information with the second node;

When the search result is empty, it is determined that the current network connectivity state is an abnormal connectivity state.

According to a node fault monitoring method provided by the present invention, the method further includes:

When the search result is that the connectivity information is found, determine whether to disconnect from the second node based on the connectivity information;

When it is determined that the connection with the second node is disconnected, it is determined that the current network connectivity state is an abnormal connectivity state.

According to a node fault monitoring method provided by the present invention, the method further includes:

When it is determined that there is a normal connection with the second node, it is determined that the current network connection state is a normal connection state.

According to a node fault monitoring method provided by the present invention, sending a first heartbeat message to a second node in the distributed cluster system includes:

Generate a target network connectivity state table according to the network connectivity state with each network node in the distributed cluster system;

Generate the first heartbeat message according to the target network connectivity status table;

If the time interval between the current time and the last sending time meets the time interval threshold, the first heartbeat message is sent to the second node.

According to a node fault monitoring method provided by the present invention, the method further includes:

Update the target network connectivity status table according to the current network connectivity status table;

According to the updated target network connectivity status table and the current number of heartbeat timeouts, a target fault monitoring result is obtained; the target fault monitoring result is a fault monitoring result of the first node.

The present invention also provides a node fault monitoring device, which is applied to the first node in the distributed cluster system, including:

A sending thread, used to send the first heartbeat message to the second node in the distributed cluster system;

A receiving thread, configured to receive the second heartbeat message returned by the second node, and obtain the current number of heartbeat timeouts with the second node and the number of heartbeat timeouts between the second node and the second node according to the second heartbeat message. Current network connectivity status table; the second heartbeat message is a response message to the first heartbeat message;

A detection thread, configured to obtain the fault monitoring result of the second node according to the current number of heartbeat timeouts and the current network connectivity status table.

The invention also provides a node fault monitoring system, including a distributed cluster system;

The distributed cluster system includes a first node, a plurality of second nodes, and a cluster trivial database;

The cluster trivial database is used to provide network connectivity status detection services for the first node and the second node;

The first node is used to execute the node fault monitoring method as described in any one of the above.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any one of the above node fault monitoring methods. method.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any one of the above node fault monitoring methods.

The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements any one of the above node fault monitoring methods.

The node fault monitoring method, device, system, electronic equipment and storage medium provided by the present invention send a first heartbeat message to a second node through a first node and receive a second heartbeat message sent by the second node to synchronize the second heartbeat message. The current network connectivity status table between the node and each network node, and then obtain the current heartbeat timeout number based on the current network connectivity status table, and jointly obtain the fault monitoring results of the second node based on the current network connectivity status table and the current heartbeat timeout number, so that CTDB accurately analyzes faulty nodes when the network is in a sub-healthy state, preventing misjudgment from causing normal nodes to perform failover and fault recovery actions, thus improving the stability and reliability of node detection, thereby improving the stability and security of the cluster. performance and reliability.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the invention, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is one of the flow diagrams of the node fault monitoring method provided by the present invention;

Figure 2 is the second schematic flow chart of the node fault monitoring method provided by the present invention;

Figure 3 is the third schematic flow chart of the node fault monitoring method provided by the present invention;

Figure 4 is the fourth schematic flow chart of the node fault monitoring method provided by the present invention;

Figure 5 is a timing diagram of interactions between network nodes provided by the present invention;

Figure 6 is a schematic structural diagram of the node fault monitoring device provided by the present invention;

Figure 7 is a schematic structural diagram of the electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

A state that usually allows the network to operate normally and recover quickly after being impacted by the outside world is called a network health state; while a state in which the network is paralyzed and cannot operate normally is called an unhealthy state. The network can operate normally under normal circumstances, but its ability to resist risks is extremely low. It is easy to be paralyzed when exposed to sudden network risks, and a state that is difficult to recover for a long time is called a sub-health state of the network. Usually the networks of many large and medium-sized enterprises are in a sub-healthy network state. Therefore, how to efficiently and accurately monitor network nodes in a sub-healthy network state is an important issue that needs to be solved urgently in the industry.

In related technologies, the CTDB service is usually used to determine whether other network nodes are abnormal nodes by point-to-point judging whether other nodes send response information to this node through PING or heartbeat monitoring. However, since CTDB lacks the logic to judge the sub-health status of the network port, when the network is in a sub-health state, the CTDB in the node with network abnormalities will lose the response information transmitted by other nodes through PING or heartbeat monitoring, and will mistakenly think that other nodes are in a sub-health state. All nodes are faulty, which triggers failover and recovery actions on other nodes, affecting the stability and reliability of the cluster. Therefore, when the network is in a sub-healthy state, CTDB may misjudge faulty nodes, resulting in low node fault detection accuracy, thus affecting the stability and reliability of the cluster system.

Based on the above problems, embodiments of the present application provide a node fault monitoring method, device, system, electronic equipment and storage medium for CTDB in a distributed cluster system when the network is in a sub-healthy state. Heartbeat messages are used to synchronize network connectivity status records between nodes, and the network connectivity status table and heartbeat timeout times are accurately obtained based on heartbeat message analysis, and then multiple re-judgments are made based on the network connectivity status table and heartbeat timeout times, so that CTDB accurately analyzes faulty nodes when the network is in a sub-healthy state, preventing misjudgments that would cause normal nodes to perform failover and fault recovery actions, thus improving the stability and reliability of node detection, thereby improving the stability of the cluster. , safety and reliability.

The node fault monitoring method of this application is described below in conjunction with Figures 1-5.

Figure 1 is one of the flow diagrams of a node fault monitoring method provided by an embodiment of the present application. This method can be applied to a distributed cluster system including a first node and multiple second nodes. The connection between the nodes in the distributed cluster system can be a wireless connection established based on Wireless Fidelity (WIFI), Bluetooth and other technologies, or a wired connection established through Universal Serial Bus and other technologies. , the embodiments of this application do not limit the connection methods between nodes.

The same CTDB service is running on each node, that is, the CTDB service is a distributed service to enable each node to detect the network status of each other. The specific detection method is by sending heartbeat messages or network sub-health detection messages.

Each node includes at least three threads, namely a sending thread, a receiving thread and a detection thread; among them, the sending thread is used to send information to the peer, such as heartbeat messages, etc.; the receiving thread is used to receive information transmitted by the peer, such as Heartbeat messages, etc., the detection thread is used to detect faults on the local node or the opposite node. It should be noted that the at least three threads mentioned above can run synchronously or asynchronously, which is not specifically limited in this embodiment.

The so-called nodes may be electronic devices. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (personal digital assistant). assistant, PDA), etc., the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., the present invention does not Specific limitations.

It can be understood that based on the connections between the nodes, communication can be carried out between the nodes. Specifically, heartbeat packets can be transmitted between the nodes for node fault monitoring, or business data can be transmitted between the nodes for corresponding processing. Business processing, etc., this embodiment does not specifically limit this.

Among them, the first node is the network node currently performing node failure monitoring, which can be randomly or determined in the distributed cluster system according to preset rules; the second node is required in the distributed cluster system in addition to the first node. One or more network nodes for fault monitoring. The following takes one second node as an example to describe the node fault monitoring method provided by this implementation. When there are multiple second nodes, you can refer to this method to perform fault monitoring on other second nodes.

The execution subject of this embodiment is the first node, as shown in Figure 1. The method specifically includes the following steps:

Step 101: Send the first heartbeat message to the second node in the distributed cluster system;

Optionally, the first node generates the first heartbeat message in real time and sends the first heartbeat message to the second node in the distributed cluster system.

The first heartbeat message may be generated based on the node connectivity status test instruction, or may be generated based on the network connectivity status table between each network node and the first node in the distributed cluster system. In this embodiment, the first heartbeat message The message generation method is not specifically limited.

It should be noted that the network connectivity status table of each node is detected and obtained by its internal CTDB service. The specific detection method can be to determine whether the heartbeat response time between each node and other network nodes exceeds the corresponding response of each other network node. The timeout duration is used to determine the network connectivity status; or it is further determined whether the number of times the response timeout duration is exceeded is greater than the threshold corresponding to each other network node to determine the network connectivity status, thereby avoiding network connectivity problems caused by different response performance of different network nodes. Status acquisition errors enable accurate acquisition of network connectivity status tables, thereby improving the accuracy of node fault detection.

The sending method may be real-time sending, that is, the first heartbeat message is sent as soon as it is generated, or it may be sent periodically. This embodiment does not specifically limit the sending method of the first heartbeat message.

In some embodiments, sending the first heartbeat message to the second node in the distributed cluster system includes:

Generate a target network connectivity state table according to the network connectivity state with each network node in the distributed cluster system;

Generate the first heartbeat message according to the target network connectivity status table;

If the time interval between the current time and the last sending time meets the time interval threshold, the first heartbeat message is sent to the second node.

Optionally, when performing node monitoring, the first node starts at least three threads, namely a sending thread, a receiving thread and a detection thread, and collects data between the first node and each network node in the distributed cluster system through the sending thread. generate a target network connectivity status table based on the network connectivity status; and write the target network connectivity status table into the heartbeat message to generate the first heartbeat message.

Next, it is determined whether the time interval between the current time and the last sending time meets the time interval threshold. If the time interval threshold is met, the first heartbeat message is sent to the second node. Among them, the time interval threshold can be set according to actual needs, such as 1 second.

The method provided by this embodiment is to send the first heartbeat message carrying the target network connectivity status table formed by the network connectivity status between the first node and each network node in the distributed cluster system to the second node, so as to realize each node's The network connection status between them is shared and synchronized in real time, so that the second node can update the current network connection status table of the second node in real time according to the target network connection status table, and write the current network connection status table into the second heartbeat message. And the second heartbeat message is returned to the first node in the form of response information to achieve timely sending of more reliable response messages, thereby improving the accuracy of node failure monitoring and further improving the stability, security and safety of the cluster system. reliability.

Step 102: Receive the second heartbeat message returned by the second node, and obtain the current number of heartbeat timeouts with the second node and the current network of the second node according to the second heartbeat message. Connectivity status table; the second heartbeat message is a response message to the first heartbeat message;

Optionally, when sending the first heartbeat message to the second node, the second node can parse the target network connection status table of the first node from the first heartbeat packet to determine the target network connection status table according to the target network connection status table. Update the network connection status table of the second node, obtain the second heartbeat message, and respond to the second heartbeat message to the first node.

The receiving thread of the first node can monitor each network node in the distributed cluster system in real time, so that when the second node returns the second heartbeat message based on the first heartbeat message, it can receive the second heartbeat message in real time. After receiving the second heartbeat message, the second heartbeat message can be parsed to obtain the current network connectivity status table of the second node, and the current network connectivity status table is transmitted to the detection thread, which uses the current The network connectivity status table obtains the current number of heartbeat timeouts.

The method of obtaining the current number of heartbeat timeouts may be to input the current network connectivity status table and the previous number of heartbeat timeouts between the first node and the second node into a pre-trained update model, and output the current number of heartbeat timeouts; or The current network connectivity status table is analyzed using preset update rules to update the previous heartbeat timeout number based on the analysis results to obtain the current heartbeat timeout number, which is not specifically limited in this embodiment.

Step 103: Obtain the fault monitoring result of the second node according to the current number of heartbeat timeouts and the current network connectivity status table.

The so-called fault monitoring results include normal status or fault status.

Optionally, after obtaining the current number of heartbeat timeouts and the current network connectivity status table, the detection thread can combine the current number of heartbeat timeouts and the current network connectivity status table to obtain the fault monitoring results of the second node.

Here, the method of obtaining the fault monitoring results of the second node includes: inputting the current number of heartbeat timeouts and the current network connectivity status table into a pre-trained detection model, and having the detection model output the fault monitoring results of the second node; or, using One or multiple fault monitoring judgment conditions include judging and analyzing the current number of heartbeat timeouts and the current network connectivity status table to obtain the fault monitoring results of the second node, which is not specifically limited in this embodiment.

The node fault monitoring method provided by the embodiment of the present application uses the first node to send a first heartbeat message to the second node and receive the second heartbeat message sent by the second node to synchronize the current status between the second node and each network node. Network connectivity status table, and then obtain the current number of heartbeat timeouts based on the current network connectivity status table, and jointly obtain the fault monitoring results of the second node based on the current network connectivity status table and the current number of heartbeat timeouts, so that CTDB can accurately detect the sub-health status of the network It analyzes faulty nodes to prevent misjudgment from causing normal nodes to perform failover and fault recovery actions, thus improving the stability and reliability of node detection, thereby improving the stability, security and reliability of the cluster.

In some embodiments, obtaining the fault monitoring result of the second node based on the current number of heartbeat timeouts and the current network connectivity status table includes:

Compare the current number of heartbeat timeouts with a number threshold to obtain a first comparison result;

When it is determined that the current number of heartbeat timeouts is greater than the number threshold according to the first comparison result, it is determined according to the current network connectivity status table whether there is at least one third node in the distributed cluster system. The network connectivity state between the second nodes is a normal state;

According to the judgment result, obtain the fault monitoring result of the second node;

Wherein, the third node is a network node other than the first node and the second node in the distributed cluster system.

Optionally, the step of obtaining the fault monitoring result of the second node in step 103 specifically includes:

After obtaining the current number of heartbeat timeouts and the current network connectivity status table, the detection thread compares the current number of heartbeat timeouts with the number threshold to determine whether the current number of heartbeat timeouts is greater than the number threshold. The times threshold here can be set according to the fault tolerance performance of the first node.

When it is determined that the current number of heartbeat timeouts is greater than the number threshold, it indicates that there is an abnormality in the communication between the second node and the first node, and it is necessary to further determine whether the second node is a faulty node according to the current network connectivity status table.

Optionally, according to the current network connectivity status table, obtain the network connectivity status between the second node and the network nodes in the distributed cluster system other than the first node and the second node, that is, the third node. , to determine whether the network connectivity state between the second node and at least one third node is a normal state according to the network connectivity state, and to further determine whether the second node is a fault node based on the determination result.

The method provided in this embodiment uses multiple re-judgments to monitor node faults by judging the current number of heartbeat timeouts and the current network connectivity status table, so that the CTDB service can accurately analyze and detect faults in a sub-healthy network state. Whether the second node is a faulty node can prevent misjudgment and cause normal nodes to perform failover and fault recovery actions, thereby improving the accuracy of node failure detection, thus improving the stability and reliability of the cluster system.

In some embodiments, obtaining the fault monitoring result of the second node according to the judgment result includes:

When it is determined according to the judgment result that there is no at least one network connection state between the third node and the second node in the distributed cluster system and the network connectivity state between the second node and the third node is in a normal state, it is determined that the second node The fault monitoring result is fault status.

Optionally, when it is determined that there is no network connection status between at least one third node and a second node in the distributed cluster system and the network connection status is normal, that is, all third nodes and second nodes in the distributed cluster system The network connectivity status between them is all abnormal. At this time, it indicates that there is abnormal communication between the second node and any network node in the distributed cluster system. At this time, the second node is determined to be a fault node.

The method provided in this embodiment uses multiple re-judgments to determine the current number of heartbeat timeouts and the current network connectivity status table, so that the second node is determined only after multiple re-judgments determine that the second node meets the fault condition. For faulty nodes, the accuracy of node fault detection is higher, thereby improving the stability and reliability of the cluster system.

In some embodiments, obtaining the fault monitoring result of the second node according to the judgment result includes:

When it is determined that there is at least one third node in the distributed cluster system and the network connectivity state between the third node and the second node is normal according to the judgment result, obtain the data corresponding to the second node. The number of reference nodes;

Obtain the fault monitoring result of the second node according to the number of reference nodes;

Wherein, the reference node is used to provide a response message for updating the current network connectivity status table of the second node within a preset period.

Optionally, when it is determined that the network connectivity status between at least one third node and the second node in the distributed cluster system is normal, that is, not all third nodes and second nodes in the distributed cluster system The network connectivity status between them is all abnormal, which indicates that there is normal communication between the second node and some network nodes in the distributed cluster system. In order to more accurately determine whether the second node is a faulty node, further troubleshooting is required. The detection conditions are re-judged.

Optionally, obtain the network node in the distributed cluster system that provides a response message for updating the current network connectivity status table of the second node within a preset period, that is, maintaining a normal network with the second node within the preset period. Communication connected network nodes to obtain reference nodes. The preset period here may include the current period, or include the current period and multiple periods before the current period. Specifically, it may be determined based on the fault tolerance performance of the first node, which is not specifically limited in this embodiment.

Next, the number of reference nodes is counted, and the number of reference nodes is compared with the quantity threshold to determine whether the number of reference nodes is greater than the quantity threshold. If the number of reference nodes is greater than the quantity threshold, the second node is determined. The fault monitoring result is normal. The quantity threshold can be set according to actual needs, such as 0.

In some embodiments, the method further includes:

If it is determined that the number of reference nodes is less than or equal to the number threshold according to the second comparison result, it is determined that the fault monitoring result of the second node is in a fault state.

Optionally, when the number of reference nodes is less than or equal to the number threshold, it is further characterized that the communication of the second node is unstable, and the second node is determined to be a fault node.

The method provided by this embodiment further determines that the network connection status between at least one third node and the second node in the distributed cluster system is normal, and further determines that the network connection status with the second node remains normal within a preset period. Whether the number of reference nodes connected by network communication is greater than the number threshold is used to further determine the communication stability of the second node, thereby accurately determining whether the second node is a fault node, thereby improving the stability and reliability of the cluster system.

In some embodiments, the method further includes:

If, according to the second comparison result, it is determined that the number of reference nodes is greater than the number threshold, trigger an isolation action;

Wherein, the isolation action is used to isolate the first node from other network nodes in the distributed cluster system except the first node, or to isolate the network port of the first node from the The network ports of other network nodes are isolated.

Optionally, when it is determined that the number of reference nodes is greater than the number threshold, it can be further determined that the first node is in an unknown state. In order to improve the accuracy of subsequent node fault detection, an isolation action can be triggered at this time to remove the first node. Isolate the node from other network nodes in the distributed cluster system except the first node, or isolate the network port of the first node from the network ports of other network nodes, that is, set the first node to Ban (banned) status so that the first node does not participate in the fault detection logic of other network nodes.

In some embodiments, the method further includes:

When it is determined that the fault monitoring result of the second node is in a fault state, a fourth node is obtained in the distributed cluster system; the fourth node has a fault monitoring result in a normal state and is in a normal state with the second node. Nodes are network nodes with the same service functions;

Migrate the pending tasks of the second node to the fourth node;

When the fault monitoring result of the second node switches from a fault state to a normal state, the pending task is restored to the second node.

The so-called service functions include one or more combinations of monitoring functions, data processing functions, data storage functions, and data forwarding functions, which are not specifically limited in this embodiment.

Optionally, when it is determined that the second node is a faulty node according to the fault monitoring result, that is, it is in a faulty state, the fault monitoring result can be obtained in the distributed cluster system to be in a normal state and have the same node as the second node. The network node of the service function serves as the fourth node to failover the second node based on the fourth node, that is, to migrate the pending tasks of the second node to the fourth node so that the fourth node can perform the pending tasks of the second node. Tasks perform business processing, maintain the normal operation of the distributed cluster system, and ensure the stability, security and reliability of the distributed cluster system.

When the fault monitoring result of the second node switches from the fault state to the normal state, the second node can be fault restored, that is, the pending tasks can be restored to the second node to continue business processing.

The method provided in this embodiment can quickly migrate and restore the tasks to be processed on the second node when the second node is a faulty node, thereby improving the stability, security, and reliability of the distributed cluster system.

In some embodiments, obtaining the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node according to the second heartbeat message includes:

Parse the second heartbeat message to obtain the current network connectivity status table;

Determine the current network connectivity state with the second node according to the current network connectivity state table;

Update the count value of the heartbeat timeout counter according to the current network connectivity status;

According to the updated count value, obtain the current number of heartbeat timeouts.

Figure 2 is a flow diagram of the second node fault monitoring method provided by the embodiment of the present application; as shown in Figure 2, the steps of obtaining the current number of heartbeat timeouts and the current network connectivity status table in step 102 include:

Step 1021, parse the second heartbeat message to obtain the current network connectivity status table; obtain the current network connectivity status between the first node and the second node from the current network connectivity status table;

The current network connectivity status here can be determined and obtained based on the storage status and/or content of the connectivity information between the first node and the second node included in the current network connectivity status table.

In some embodiments, the step of determining the current network connectivity status includes:

In the current network connectivity status table, search for connectivity information with the second node;

When the search result is empty, it is determined that the current network connectivity state is an abnormal connectivity state.

Optionally, obtain the connection information set between the second node and each network node in the current network connection status table, and then search the connection information set between the first node and the first node according to the identification of the first node and the identification of the second node. The connection information between the second nodes is determined, and whether the search result is empty, that is, whether the connection information between the first node and the second node is deleted. The connectivity information here is a combined identification formed by the identification of the first node and the identification of the second node (direct splicing of the identification of the first node and the identification of the second node) or a coded identification (the identification of the first node and the identification of the second node). There is a pre-established mapping relationship between the two node identifiers).

When it is determined that the connectivity information between the first node and the second node is deleted, it indicates that the current network connectivity state between the first node and the second node is an abnormal connectivity state.

In some embodiments, the method further includes:

When the search result is that the connectivity information is found, determine whether to disconnect from the second node based on the connectivity information;

When it is determined that the connection with the second node is disconnected, it is determined that the current network connectivity state is an abnormal connectivity state.

Optionally, when it is determined that the search result is that connectivity information is found, it is necessary to further determine whether the first node and the second node are disconnected based on the content of the connectivity information. In the case of intermittent disconnection, it is determined that the current network connectivity state of the first node and the second node is an abnormal connectivity state.

In some embodiments, the method further includes:

When it is determined that there is a normal connection with the second node, it is determined that the current network connection state is a normal connection state.

Optionally, when it is determined that there is a normal connection between the first node and the second node, it is indicated that the current network connectivity state of the first node and the second node is the normal connectivity state.

Step 1022: Update the count value of the heartbeat timeout counter according to the current network connectivity status, and use the updated count value as the current number of heartbeat timeouts. The so-called heartbeat timeout counter is a counter used to record the number of heartbeat timeouts.

It should be noted that every time it is detected that the second node is a faulty node, the count value of the heartbeat timeout counter needs to be reset to zero in order to continue to provide fault detection parameters in the future.

The so-called update method can be to input the current network connectivity status and the count value of the heartbeat timeout counter into the pre-trained update model and output the current number of heartbeat timeouts; or, use pre-judgment rules to judge and analyze the current network connectivity status, Determine how to update the count value of the heartbeat timeout counter based on the judgment and analysis results to obtain the current number of heartbeat timeouts.

In some embodiments, updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

When it is determined that the current network connectivity state is an abnormal connectivity state, the count value of the heartbeat timeout counter is cumulatively increased by 1.

Optionally, when the current network connectivity state is an abnormal connectivity state, it can be determined that the heartbeat message transmission duration between the first node and the second node has timed out, and at this time, the count value of the heartbeat timeout counter is cumulatively increased by 1, so as to Get the current number of heartbeat timeouts corresponding to the second node.

In some embodiments, updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

When it is determined that the current network connectivity state is a normal connectivity state, the count value of the heartbeat timeout counter is kept unchanged.

Optionally, when the current network connectivity state is a normal connectivity state, it can be determined that the heartbeat message transmission duration between the first node and the second node has not timed out, and at this time, the count value of the heartbeat timeout counter remains unchanged, so as to Get the current number of heartbeat timeouts corresponding to the second node.

The method provided in this embodiment can accurately obtain the second current heartbeat timeout number in real time by judging the current network connectivity status between the first node and the second node, and combine the heartbeat timeout number to perform multiple repetitions on the node. It can realize node fault monitoring, thereby improving the accuracy of fault monitoring and improving the stability and reliability of the cluster system.

In some embodiments, the method further includes:

Update the target network connectivity status table according to the current network connectivity status table;

According to the updated target network connectivity status table and the current number of heartbeat timeouts, a target fault monitoring result is obtained; the target fault monitoring result is a fault monitoring result of the first node.

Figure 3 is the third flow diagram of the node fault monitoring method provided by the embodiment of the present application; as shown in Figure 3, the steps of performing fault monitoring on this node (that is, the first node) include:

Step 301: Obtain the network connectivity status between the first node and each network node, and obtain the target network connectivity status table of the first node;

Step 302: Send the first heartbeat message carrying the target network connectivity status table to the second node;

Step 303: Receive the second heartbeat message returned by the second node, obtain the network connectivity status table of the second node, update the network connectivity status table of the first node according to the current network connectivity status table, and obtain the updated target network connectivity status table; and obtain the current number of heartbeat timeouts with the second node according to the current network connectivity status table;

Step 304: Perform fault monitoring on the current node (that is, the first node) based on the updated target network connectivity status table and the current number of heartbeat timeouts.

Optionally, when it is determined that the current number of heartbeat timeouts is greater than the number threshold, it is determined that the heartbeat message transmitted between the second node and the current node has timed out. At this time, it is necessary to determine whether the network connectivity status between at least one third node and the first node in the distributed cluster system is normal based on the updated target network connectivity status table. If not, determine that the node is faulty. Trigger the failover of this node; if it exists, further obtain the number of reference nodes corresponding to the first node, and determine whether the number of reference nodes is greater than the quantity threshold. If it is greater than the quantity threshold, it is determined that this node is in an unknown state, and the This node is set to the Ban state and no longer participates in the detection logic of other network nodes; if it is not greater than the number threshold, it is determined that this node is a faulty node and the failover of this node is triggered.

The method provided in this embodiment can perform real-time fault monitoring on this node through multiple re-judgments based on the updated target network connectivity status table of the first node and the current number of heartbeat timeouts, so that CTDB can accurately analyze the network in a sub-healthy state. It detects whether the first node is a faulty node and prevents misjudgment from causing the normal node to perform failover and fault recovery actions, thereby improving the stability and reliability of node detection and thereby improving the stability, security and reliability of the cluster.

Figure 4 is a schematic flow chart of the node fault monitoring method provided by the embodiment of the present application. Figure 5 is a timing diagram of the interaction between network nodes provided by the embodiment of the present application. As shown in Figures 4 and 5, node 1 is the first node, node 2 is the second node, and node 3 is a node other than the first node and the second node. As an example, the node fault monitoring method provided by this implementation is described;

Optionally, start the sending thread, receiving thread and detection thread deployed in the cluster trivial database of the first node (hereinafter also referred to as this node or node 1) to perform fault monitoring on each second node (hereinafter also referred to as node 2) , specific steps include:

Step 401: The sending thread generates the first heartbeat message according to the network connection status of the node and each network node;

Step 402: The sending thread sends the first heartbeat message;

Step 403: The receiving thread receives the second heartbeat message returned by the second node; the second heartbeat message is sent by the second node to each other network node (such as node 3) when receiving the first heartbeat message. The heartbeat message is generated by updating the network connection state table of the second node based on the heartbeat message carrying the network connection state table returned by each other network node.

Step 404: The receiving thread parses the second heartbeat message, obtains the current network connectivity status table of the second node, and updates the target network connectivity status table of the current node according to the current network connectivity status table;

Step 405: The detection thread cyclically determines the heartbeat timeout between the current node and the second node based on the current network connectivity status table, which is specifically implemented through steps 406 to 408;

Step 406: The detection thread determines whether the connection information between the current node and the second node is deleted according to the current network connection status table, or whether the connection between the current node and the second node is disconnected. If the connection is deleted or disconnected, execute Step 408, if not, execute step 407;

Step 407: The detection thread refreshes the network connection status between the current node and the second node to the normal state, and the count value of the heartbeat timeout counter remains unchanged. Step 409 is executed;

Step 408: The detection thread refreshes the network connection status between this node and the second node to an abnormal state, and the count value of the heartbeat timeout counter is cumulatively increased by 1, and step 409 is executed;

Step 409: Determine whether the current number of heartbeat timeouts between the current node and the second node is greater than the number threshold. If it is greater than the number threshold, execute step 410; if not, determine that the second node is a normal node and exit;

Step 410: Determine whether the network connectivity status between at least one network node and the second node is normal. If it exists, perform step 411. If it does not exist, determine whether the second node is a fault node and perform step 413;

Step 411, determine the number of reference nodes. If the number is greater than 0, determine that the second node is a normal node, and execute step 412. If not, determine that the second node is a fault node, and execute step 413;

Step 412: No node failover is performed on the second node, the current node is updated to the BAN state, and exits;

Step 413: Perform node failover on the second node.

It should be noted that if it is determined that this node is a faulty node, network port isolation or node isolation can be performed on this node to ensure the stability of the distributed cluster system.

In addition, when the first node discovers that there is a heartbeat timeout between the first node and the second node, the first node can also notify other network nodes in real time that it has discovered a node that has problems with its network communication, and triggers a communication between this node and the problem node. Network sub-health detection; after other nodes receive the message to turn on network sub-health detection, they will also trigger the network sub-health detection between their own nodes and the problem node, and summarize the network sub-health detection of other nodes and the second node. , when it is found that more than half of the network nodes detect that the second node has network sub-health, the second node's network port isolation or node isolation action is triggered to ensure the stability of the distributed cluster system. The node fault monitoring device provided by the present invention is described below. The node fault monitoring device described below and the node fault monitoring method described above can be mutually referenced.

Figure 6 is a schematic structural diagram of a node fault monitoring device provided by an embodiment of the present application. As shown in Figure 6, the device includes:

The sending thread 601 is used to send the first heartbeat message to the second node in the distributed cluster system;

The receiving thread 602 is used to receive the second heartbeat message returned by the second node, and obtain the current heartbeat timeout number between the second node and the second node according to the second heartbeat message. Current network connectivity status table; the second heartbeat message is a response message to the first heartbeat message;

The detection thread 603 is used to obtain the fault monitoring result of the second node according to the current number of heartbeat timeouts and the current network connectivity status table.

The node fault monitoring device provided by the embodiment of the present application uses the first node to send a first heartbeat message to the second node and receives the second heartbeat message sent by the second node to synchronize the current status between the second node and each network node. Network connectivity status table, and then obtain the current number of heartbeat timeouts based on the current network connectivity status table, and jointly obtain the fault monitoring results of the second node based on the current network connectivity status table and the current number of heartbeat timeouts, so that CTDB can accurately detect the sub-health status of the network It analyzes faulty nodes to prevent misjudgment from causing normal nodes to perform failover and fault recovery actions, thus improving the stability and reliability of node detection, thereby improving the stability, security and reliability of the cluster.

Embodiments of the present application also provide a node fault monitoring system, which system includes: a distributed cluster system; the distributed cluster system includes a first node, a plurality of second nodes, and a cluster trivial database;

The cluster trivial database is used to provide network connectivity status detection services for the first node and the second node;

The first node is used to perform a node fault monitoring method, which method includes sending a first heartbeat message to a second node in the distributed cluster system; receiving a second heartbeat message returned by the second node, and according to The second heartbeat message obtains the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node; the second heartbeat message is the first heartbeat message. a response message of the message; and obtain the fault monitoring result of the second node according to the current heartbeat timeout number and the current network connectivity status table.

The node fault monitoring system provided by the embodiment of the present application uses the first node to send a first heartbeat message to the second node and receive the second heartbeat message sent by the second node to synchronize the current status between the second node and each network node. Network connectivity status table, and then obtain the current number of heartbeat timeouts based on the current network connectivity status table, and jointly obtain the fault monitoring results of the second node based on the current network connectivity status table and the current number of heartbeat timeouts, so that CTDB can accurately detect the sub-health status of the network It analyzes faulty nodes to prevent misjudgment from causing normal nodes to perform failover and fault recovery actions, thus improving the stability and reliability of node detection, thereby improving the stability, security and reliability of the cluster.

Figure 7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 7, the electronic device may include: a processor (processor) 701, a communications interface (Communications Interface) 702, a memory (memory) 703, and a communication bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 complete communication with each other through the communication bus 704. The processor 701 can call logical instructions in the memory 703 to execute a node fault monitoring method. The method includes: sending a first heartbeat message to a second node in the distributed cluster system; receiving a first heartbeat message returned by the second node. The second heartbeat message is used to obtain the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node according to the second heartbeat message; the second heartbeat message is the response message of the first heartbeat message; and obtains the fault monitoring result of the second node according to the current number of heartbeat timeouts and the current network connectivity status table.

In addition, the above-mentioned logical instructions in the memory 703 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Executing the node fault monitoring method provided by the above methods, the method includes: sending a first heartbeat message to the second node in the distributed cluster system; receiving the second heartbeat message returned by the second node, according to The second heartbeat message obtains the current number of heartbeat timeouts with the second node and the current network connectivity status table of the second node; the second heartbeat message is the first heartbeat message. a response message of the message; and obtain the fault monitoring result of the second node according to the current heartbeat timeout number and the current network connectivity status table.

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by a processor to perform the node fault monitoring method provided by each of the above methods. The method includes : Send a first heartbeat message to the second node in the distributed cluster system; receive the second heartbeat message returned by the second node, and obtain the information related to the second node according to the second heartbeat message. The current number of heartbeat timeouts and the current network connectivity status table of the second node; the second heartbeat message is a response message of the first heartbeat message; according to the current number of heartbeat timeouts and the current network connectivity status table of the second node; The current network connectivity status table is described to obtain the fault monitoring result of the second node.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disc, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be used Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (18)

1. A method for monitoring node failure, applied to a first node in a distributed cluster system, comprising:

sending a first heartbeat message to a second node in the distributed cluster system;

receiving a second heartbeat message returned by the second node, and acquiring the current heartbeat timeout times between the second node and a current network communication state table of the second node according to the second heartbeat message; the second heartbeat message is a response message of the first heartbeat message;

acquiring a fault monitoring result of the second node according to the current heartbeat timeout times and the current network connection state table;

The obtaining the fault monitoring result of the second node according to the current heartbeat timeout times and the current network connection state table includes:

comparing the current heartbeat timeout times with a frequency threshold value to obtain a first comparison result;

judging whether a network communication state between at least one third node and the second node in the distributed cluster system is a normal state according to the current network communication state table under the condition that the current heartbeat timeout times are larger than the times threshold according to the first comparison result;

obtaining a fault monitoring result of the second node according to the judging result;

the third node is a network node except the first node and the second node in the distributed cluster system;

and acquiring a fault monitoring result of the second node according to the judging result, wherein the fault monitoring result comprises:

and under the condition that the network communication state between at least one third node and the second node is not normal in the distributed cluster system according to the judging result, determining that the fault monitoring result of the second node is a fault state.

2. The method for monitoring a node failure according to claim 1, wherein the obtaining the failure monitoring result of the second node according to the determination result includes:

acquiring the number of referenceable nodes corresponding to the second node under the condition that the network communication state between at least one third node and the second node in the distributed cluster system is determined to be in a normal state according to the judging result;

obtaining a fault monitoring result of the second node according to the number of the referenceable nodes;

the referenceable node is configured to provide a response message for updating the current network connection status table of the second node in a preset period.

3. The method for monitoring node failure according to claim 2, wherein the obtaining the failure monitoring result of the second node according to the number of referenceable nodes includes:

comparing the number of the referenceable nodes with a number threshold to obtain a second comparison result;

and under the condition that the number of the referenceable nodes is larger than the number threshold according to the second comparison result, determining that the fault monitoring result of the second node is in a normal state.

4. A node failure monitoring method according to claim 3, characterized in that the method further comprises:

triggering an isolation action if it is determined that the number of referenceable nodes is greater than the number threshold based on the second comparison result;

the isolation action is used for isolating the first node from other network nodes except the first node in the distributed cluster system or isolating the network port of the first node from the network ports of other network nodes.

5. A node failure monitoring method according to claim 3, characterized in that the method further comprises:

and determining that the fault monitoring result of the second node is a fault state under the condition that the number of the referenceable nodes is less than or equal to the number threshold according to the second comparison result.

6. The node failure monitoring method of any of claims 1-5, further comprising:

under the condition that the fault monitoring result of the second node is determined to be in a fault state, a fourth node is obtained in the distributed cluster system; the fourth node is a network node with the same service function as the second node and the fault monitoring result is in a normal state;

Migrating the task to be processed of the second node to the fourth node;

and under the condition that the fault monitoring result of the second node is switched from the fault state to the normal state, recovering the task to be processed to the second node.

7. The method for monitoring node failure according to any one of claims 1-5, wherein the obtaining, according to the second heartbeat message, a current heartbeat timeout number with the second node, and a current network connection state table of the second node includes:

analyzing the second heartbeat message to obtain the current network communication state table;

determining a current network communication state between the second node and the current network communication state table according to the current network communication state table;

updating the count value of the heartbeat timeout counter according to the current network connection state;

and acquiring the current heartbeat timeout times according to the updated count value.

8. The method for monitoring node failure according to claim 7, wherein updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

and under the condition that the current network communication state is determined to be an abnormal communication state, adding 1 to the count value of the heartbeat timeout counter.

9. The method for monitoring node failure according to claim 7, wherein updating the count value of the heartbeat timeout counter according to the current network connectivity status includes:

and under the condition that the current network communication state is determined to be a normal communication state, keeping the count value of the heartbeat timeout counter unchanged.

10. The method of claim 7, wherein determining the current network connectivity status with the second node based on the current network connectivity status table comprises:

searching the communication information between the current network communication state table and the second node;

and under the condition that the searching result is empty, determining that the current network communication state is an abnormal communication state.

11. The node failure monitoring method of claim 10, further comprising:

if the connection information is found according to the search result, determining whether the connection with the second node is disconnected according to the connection information;

and under the condition that disconnection with the second node is determined, determining that the current network communication state is an abnormal communication state.

12. The node failure monitoring method of claim 11, further comprising:

and under the condition of determining normal connection with the second node, determining that the current network communication state is a normal communication state.

13. The method for monitoring node failure according to any of claims 1-5, wherein the sending a first heartbeat message to a second node in the distributed cluster system includes:

generating a target network communication state table according to the network communication state between the target network communication state table and each network node in the distributed cluster system;

generating the first heartbeat message according to the target network communication state table;

and sending the first heartbeat message to the second node under the condition that the time interval between the current time and the last sending time meets a time interval threshold value.

14. The node failure monitoring method of claim 13, characterized in that the method further comprises:

updating the target network communication state table according to the current network communication state table;

acquiring a target fault monitoring result according to the updated target network communication state table and the current heartbeat timeout times; and the target fault monitoring result is the fault monitoring result of the first node.

15. A node failure monitoring apparatus, for use with a first node in a distributed cluster system, comprising:

the sending module is used for sending a first heartbeat message to a second node in the distributed cluster system;

the receiving module is used for receiving a second heartbeat message returned by the second node, and acquiring the current heartbeat timeout times between the second node and the current network communication state table of the second node according to the second heartbeat message; the second heartbeat message is a response message of the first heartbeat message;

the detection module is used for acquiring a fault monitoring result of the second node according to the current heartbeat timeout times and the current network connection state table;

the detection module is specifically configured to:

comparing the current heartbeat timeout times with a frequency threshold value to obtain a first comparison result;

judging whether a network communication state between at least one third node and the second node in the distributed cluster system is a normal state according to the current network communication state table under the condition that the current heartbeat timeout times are larger than the times threshold according to the first comparison result;

Obtaining a fault monitoring result of the second node according to the judging result;

the third node is a network node except the first node and the second node in the distributed cluster system;

the detection module is further used for:

and under the condition that the network communication state between at least one third node and the second node is not normal in the distributed cluster system according to the judging result, determining that the fault monitoring result of the second node is a fault state.

16. A node fault monitoring system, comprising a distributed cluster system;

the distributed cluster system comprises a first node, a plurality of second nodes and a cluster trivial database;

the cluster trivial database is used for providing network communication state detection service for the first node and the second node;

the first node is configured to perform the node failure monitoring method of any of claims 1-14.

17. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the node fault monitoring method of any of claims 1 to 14 when the program is executed by the processor.

18. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the node fault monitoring method of any of claims 1 to 14.