WO2021159897A1 - 故障通知方法及相关设备 - Google Patents
故障通知方法及相关设备 Download PDFInfo
- Publication number
- WO2021159897A1 WO2021159897A1 PCT/CN2021/071042 CN2021071042W WO2021159897A1 WO 2021159897 A1 WO2021159897 A1 WO 2021159897A1 CN 2021071042 W CN2021071042 W CN 2021071042W WO 2021159897 A1 WO2021159897 A1 WO 2021159897A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- fault
- failure
- notification
- broadcast message
- detected
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Definitions
- the present invention relates to the field of computer technology, in particular to a fault notification method and related equipment.
- fault detection is mainly to find faulty equipment through similar detection techniques such as heartbeat detection.
- the detecting device will send a heartbeat request to the detected device in the distributed cluster at regular intervals (for example, 3 seconds), and the detected device will respond to the detecting device in time after receiving the heartbeat request to indicate that it can provide business services normally. If there is no response from the detected device within a period of time (for example, 10s, that is, 3 heartbeat requests are sent), the detecting device considers the detected device to be faulty.
- the embodiment of the application discloses a fault notification method and related equipment, which can greatly reduce the time for the detection equipment in a distributed cluster to detect the failure of the detected equipment, thereby avoiding the long service delay caused by the detection equipment sensing the failure time. Even the problem of interruption.
- this application discloses a fault notification method, which is applied to devices in a distributed cluster, and these devices include detecting devices and detected devices.
- the method includes:
- the detected device In the case where the detected device detects that it has a failure, the detected device sends a broadcast message to the detection device; wherein the broadcast message is used to indicate that the detected device has a failure; the broadcast message is unreliable Transmission protocol message.
- the detected device actively detects its own failure, and once a failure is found, it immediately sends a failure notification to the detection device.
- this application can make the detection device quickly perceive the existing failure and respond to the failure, thereby To avoid the problem of long service delay or even interruption caused by the detection device's long time to detect faults.
- the above fault is a fault that the operating system of the detected device cannot perceive;
- the detected device includes a motherboard and a network card; and the above detected device detects a failure of itself when the detected device detects a failure.
- the detection device sends a broadcast message to the aforementioned detection device, including:
- the detected device detects the failure of the detected device through the motherboard; the detected device sends a notification signal to the network card through the motherboard; the notification signal is generated by the motherboard based on the failure; the detected device uses the network card according to The notification signal sends the broadcast message to the detection device.
- the fault is sensed by the mainboard of the detected device, and the network card of the detected device actively sends a fault notification, that is, the above-mentioned broadcast message, to the detecting device, thereby solving the problem of notification of faults that the operating system cannot perceive in the detected device. .
- the above-mentioned broadcast message is registered in the network card driver of the above-mentioned network card.
- the failure notification when a failure occurs is registered in the network card driver in advance. Once a failure occurs, the failure notification message can be sent to the detection device immediately, which is convenient and fast.
- the above-mentioned fault is a fault that can be sensed by the operating system of the above-mentioned detected device; the above-mentioned detected device sends a broadcast report to the above-mentioned detecting device when the above-mentioned detected device detects that it has a fault.
- Text including:
- the detected device detects the failure of the detected device through the operating system
- the detected device sends the broadcast message to the detecting device through the kernel notification chain of the operating system.
- This application uses the kernel notification chain to actively notify the detection device of the failure of the detected device, which is convenient and quick.
- the above-mentioned broadcast message is registered in the callback function included in the above-mentioned kernel notification chain; or, the callback function included in the above-mentioned kernel notification chain is used to generate the above-mentioned broadcast message.
- the above-mentioned broadcast message includes the unique identifier of the above-mentioned detected device in the above-mentioned distributed cluster; wherein, the above-mentioned unique identifier is an identifier regenerated when the above-mentioned detected device recently joined the above-mentioned distributed cluster.
- the unique identification of the detected device in the distributed cluster is added to the failure notification, and the detection device can directly discard the repeatedly received failure notification according to the unique identification, thereby saving computing resources.
- the above-mentioned unreliable transmission protocol is a user datagram protocol (UDP);
- the above-mentioned broadcast message sent by the above-mentioned detected device to the above-mentioned detecting device includes a plurality of the above-mentioned broadcast messages.
- This application uses UDP broadcast messages to carry fault notification information, and can use the UDP connectionless protocol feature to complete the fault notification in time even when the device is powered off, the operating system is down, or the host or program stops working.
- the use of UDP broadcast messages can also realize failure notification in the case of a cluster main failure or multi-point failure.
- the problem of packet loss caused by unreliable UDP transmission can be avoided by sending multiple broadcast packets to the detection device.
- this application discloses a fault notification device, the above-mentioned fault notification device belongs to a detected device in a distributed cluster, the above-mentioned distributed cluster further includes a detection device; the above-mentioned fault notification device includes:
- the first sending unit is configured to send a broadcast message to the detection device when the failure notification device detects that it has a failure; wherein the broadcast message is used to indicate that the failure notification device has a failure; the broadcast message It is an unreliable transmission protocol packet.
- the above-mentioned fault is a fault that the operating system of the above-mentioned fault notification device cannot perceive;
- the above-mentioned fault notification device includes a motherboard and a network card;
- the above-mentioned fault notification device further includes a first detection unit and a second sending unit;
- the above-mentioned first detection unit is configured to detect the above-mentioned fault notification device through the above-mentioned motherboard;
- the second sending unit is configured to send a notification signal to the network card through the main board; the notification signal is generated by the main board according to the failure;
- the first sending unit is specifically configured to send the broadcast message to the detection device according to the notification signal through the network card.
- the above-mentioned broadcast message is registered in the network card driver of the above-mentioned network card.
- the above-mentioned fault is a fault that can be sensed by the operating system of the above-mentioned fault notification device; the above-mentioned fault notification device further includes a second detection unit;
- the second detection unit is configured to detect that the failure notification device has the failure through the operating system
- the first sending unit is specifically configured to send the broadcast message to the detection device through the kernel notification chain of the operating system.
- the above-mentioned broadcast message is registered in the callback function included in the above-mentioned kernel notification chain; or, the callback function included in the above-mentioned kernel notification chain is used to generate the above-mentioned broadcast message.
- the above-mentioned broadcast message includes the unique identifier of the above-mentioned fault notification device in the above-mentioned distributed cluster; wherein, the above-mentioned unique identifier is an identifier regenerated when the above-mentioned fault notification device recently joined the above-mentioned distributed cluster.
- the aforementioned unreliable transmission protocol is the User Datagram Protocol UDP;
- the aforementioned broadcast message sent by the aforementioned detected device to the aforementioned detection device includes a plurality of aforementioned broadcast messages.
- the present application discloses a fault notification device.
- the above-mentioned fault notification device belongs to a detected device in a distributed cluster, and the above-mentioned distributed cluster further includes a detection device.
- the failure notification device includes a processor, a memory, and a communication interface; the memory, the communication interface are coupled to the processor, and the memory stores a computer program. When the processor executes the computer program, the failure notification device performs the following operations:
- the detected device In the case where the detected device detects that it has a failure, the detected device sends a broadcast message to the detection device; wherein the broadcast message is used to indicate that the detected device has a failure; the broadcast message is unreliable Transmission protocol message.
- the above-mentioned fault is a fault that the operating system of the detected device cannot perceive;
- the above-mentioned fault notification device includes a motherboard and a network card; and the above-mentioned fault is detected when the detected device detects its own failure.
- the detection device sends a broadcast message to the aforementioned detection device, including:
- the detected device detects the failure of the detected device through the motherboard; the detected device sends a notification signal to the network card through the motherboard; the notification signal is generated by the motherboard based on the failure; the detected device uses the network card according to The notification signal sends the broadcast message to the detection device.
- the above-mentioned broadcast message is registered in the network card driver of the above-mentioned network card.
- the above-mentioned fault is a fault that can be sensed by the operating system of the above-mentioned detected device; the above-mentioned detected device sends a broadcast report to the above-mentioned detecting device when the above-mentioned detected device detects that it has a fault.
- Text including:
- the detected device detects the failure of the detected device through the operating system
- the detected device sends the broadcast message to the detecting device through the kernel notification chain of the operating system.
- the above-mentioned broadcast message is registered in the callback function included in the above-mentioned kernel notification chain; or, the callback function included in the above-mentioned kernel notification chain is used to generate the above-mentioned broadcast message.
- the above-mentioned broadcast message includes the unique identifier of the above-mentioned detected device in the above-mentioned distributed cluster; wherein, the above-mentioned unique identifier is an identifier regenerated when the above-mentioned detected device recently joined the above-mentioned distributed cluster.
- the aforementioned unreliable transmission protocol is the User Datagram Protocol UDP;
- the aforementioned broadcast message sent by the aforementioned detected device to the aforementioned detection device includes a plurality of aforementioned broadcast messages.
- the present application discloses a computer-readable storage medium that stores a computer program, and the computer program is executed by a processor to implement the method described in any one of the above-mentioned first aspects.
- the present application provides a computer program product.
- the computer program in the computer program product is read and executed by a computer, the method described in any one of the above-mentioned first aspects will be executed.
- the detected device actively detects its own failure, and once a failure is found, it immediately sends a failure notification to the detection device.
- this application can enable the detection device to quickly perceive the existing failure and respond to the failure. Respond, so as to avoid the problem of long service delay or even interruption caused by the long time of detecting failure of the detection equipment.
- FIG. 1 is a schematic diagram of a system architecture to which the fault notification method provided by an embodiment of the application is applicable;
- FIG. 2 is a schematic flowchart of a fault notification method provided by an embodiment of the application
- FIG. 3 is a schematic diagram of a process of implementing failure notification through a notification chain provided by an embodiment of the application
- FIG. 4 is a schematic diagram of the process of implementing failure notification through a network card according to an embodiment of the application
- FIG. 5 is a schematic diagram of the logical structure of a fault notification device provided by an embodiment of the application.
- FIG. 6 is a schematic diagram of the hardware structure of a fault notification device provided by an embodiment of the application.
- FIG. 1 is a schematic diagram of a system architecture to which the fault notification method provided in an embodiment of the present application is applicable.
- the system architecture may include one or more detection devices 101 and one or more detected devices 102.
- the detecting device 101 and the detected device 102 may be devices belonging to the same distributed cluster.
- the detection device 101 may be used to detect whether the detected device 102 fails, so as to ensure a timely response when the detected device 102 fails, thereby reducing the impact on service processing.
- the detected device 102 when the detected device 102 fails, the detected device 102 can actively notify the detection device 101 of the occurrence of the failure event, thereby greatly reducing the time for the detection device 101 to detect the failure, and thereby It avoids the problem of long service delay or even interruption caused by too long fault detection time.
- Each detection device 101 can be used to detect whether one or more detected devices 102 have failed, and each detected device 102 can also be detected by one or more detection devices for failure.
- the specific detection devices and detected devices are based on The actual situation is determined, and this plan does not impose restrictions on this.
- the detection device 101 may be a device used for distributing service processing tasks to the detected device 102 in a distributed cluster, and the detected device 102 may be a device used for executing tasks in a distributed cluster.
- the detection device 101 needs to know whether the detected device 102 is faulty, so as to ensure that the task performed by the detected device 102 can be assigned to other normal detected devices 102 for execution in the event that the detected device 102 fails, so as to ensure the task performance Perform normally.
- the detection device 101 and the detected device 102 are respectively a slave device and a master device for distributing service processing tasks in a distributed cluster.
- the slave device needs to know whether the master device fails, so as to ensure that the task executed by the master device can be switched to the slave device for execution when the master device fails, so as to ensure the normal execution of the task.
- the same device can be either the detecting device or the detected device.
- the main device used to distribute business processing tasks in a distributed cluster can be used as a detection device to detect whether the device used to perform tasks in the distributed cluster appears.
- the fault can also be whether the detected device is detected by its slave device to monitor whether a fault occurs.
- system architecture to which the fault notification method provided in the embodiment of this application is applicable is not limited to the architecture shown in FIG. I won't repeat it here.
- the following provides a fault notification method, which can be applied to the system architecture shown in FIG. 1 above.
- the method includes but is not limited to the following steps:
- Step 201 The detected device detects that it has a failure.
- the faults involved in the embodiments of the present application include two types.
- the first type is a fault that can be sensed by the operating system (OS) of the detected device
- the second type is a fault that the OS cannot sense, that is, the OS cannot. normal work.
- failures involved in the embodiments of the present application include situations where the detected device cannot normally perform business processing tasks.
- the first type of fault can include active reset situations such as reboot, shutdown, and initialization, as well as out of memory (oom), emergent, and watchdog ( The passive reset of the operating system initiated by watchdog) and unpredictable events (panic).
- the first type of failure can also include process failures caused by ending the process with the Kill Kill command and process Crash.
- the process crash here refers to a situation where the system crashes due to some reason, or the host or program stops working during the normal operation of the device system.
- the second type of failure can include abnormal resetting of the OS of the detected device or direct power failure, such as system crash, power failure, long press of the shutdown button, and intelligent platform management interface (IPMI) mandatory Power off, etc.
- IPMI is a new generation of universal interface standard that makes hardware management "intelligent”. Users can use IPMI to monitor the physical characteristics of the device, such as temperature, voltage, fan working status, power supply, and chassis intrusion.
- the detected device uses two different methods to detect faults respectively. Specifically, for the first type of failure, the detected device uses its own operating system to detect the failure. For the second type of failure, the detected device detects the failure through its own motherboard.
- Step 202 The detected device sends a broadcast message to the detection device; where the broadcast message is used to indicate that the detected device has a fault; the broadcast message is a message of an unreliable transmission protocol.
- the above-mentioned broadcast message includes a destination port, the destination port is a preset port, and the detection device is pre-configured to listen to the preset destination port.
- the detection device can receive the broadcast message.
- the destination ports of the broadcast packets sent by different detected devices can be different. These destination ports can be mapped to the detected devices one by one.
- the detecting device can determine which is detected by the port number of the received broadcast packet. The device has malfunctioned.
- the above-mentioned broadcast message includes the above-mentioned unique identifier of the detected device, and the unique identifier may be a serial number or identification code that uniquely identifies the detected device in a distributed cluster.
- the unique identifier may be session id, etc.
- the detection device after the detection device receives the broadcast message, it can determine which detected device is malfunctioning according to the unique identifier in the broadcast message.
- the above-mentioned broadcast message includes the unique identification of the detected device so that when the detection device receives the same broadcast message again, it can learn from the unique identification that the received message is a duplicate, and it can be directly discarded, thereby saving duplication. Processing computing resources.
- the detected device if the detected device returns to normal and can process business normally, the detected device will regenerate its unique identifier in the distributed cluster, and notify the regenerated unique identifier to the distributed cluster. other devices. Therefore, the unique identifier included in the broadcast message sent at the time of failure is the identifier regenerated when the detected device recently joined the distributed cluster.
- the detecting device receives the broadcast message sent by the detected device, if the unique identifier in the broadcast message is not a newly generated unique identifier of the detected device, it can be directly discarded, thereby avoiding the problem of misjudgment of faults.
- Example 1 If the detection device is a device for allocating service processing tasks in a distributed cluster, then after receiving the broadcast message, the detection device learns that the detected device cannot normally perform service processing tasks. In order not to affect the normal processing of the business, the detection device can kick the failed detected device out of the cluster, that is, the detection device will no longer assign business processing tasks to the detected device for processing, and will find other available devices To handle the corresponding business. Until the detected device returns to normal and applies for rejoining the cluster, the detected device will regenerate its unique identification number in the cluster at this time.
- Example 2 If the detection device is a slave device that distributes business processing tasks in a distributed cluster, and the detected device is a master device that distributes business processing tasks in a distributed cluster, then after the detection device receives the failure notification message, Knowing that the detected device cannot normally perform the distribution work of the business processing task, in order not to affect the normal processing of the business, the detection device will take over the distribution work of the business processing task. And you can kick the failed detected device out of the cluster until the detected device returns to normal and apply to rejoin the cluster. At this time, the detected device will regenerate its unique identification number in the cluster.
- the above-mentioned unreliable transmission protocol is used to realize the active fault notification of the detected device to the detecting device, which can ensure that when the detected device fails, the failure notification, that is, the above-mentioned broadcast message, can also be sent to the detecting device, thereby realizing the detection device to the detection device.
- the fault of the detected equipment is quickly sensed.
- the foregoing unreliable transmission protocol may be a user datagram protocol (UDP), that is, the foregoing broadcast message is a UDP broadcast message.
- UDP user datagram protocol
- failure notification can be completed in time even when the device is powered off, the operating system is down, or the host or program stops working.
- the use of UDP broadcast messages can realize failure notification in the case of a cluster main failure or multi-point failure.
- the detected device may continuously send the broadcast message to the detection device multiple times to ensure that the detection and the device successfully receive the message.
- the broadcast message is a UDP broadcast message. Since UDP is a feature of best effort delivery and does not guarantee reliable delivery, the broadcast message can be sent several times to solve the problem of message loss caused by UDP packet loss.
- the detected device respectively uses two different methods to send broadcast messages to the detecting device. Specifically, for the first type of fault, the detected device sends a broadcast message to the detecting device through the kernel notification chain of the device's operating system. For the second type of failure, the detected device sends a broadcast message to the detecting device through the network card.
- two embodiments are used to respectively introduce the specific process of implementing active fault notification when the detected device has the above-mentioned two types of faults.
- Embodiment 1 Active fault notification to the detecting device is realized when the detected device has the above-mentioned first type of fault.
- the notification chain of the failure is registered in the OS kernel of the detected device.
- the notification chain can be registered when the service of the detection device is started.
- the notification chain may include a callback function.
- the callback function When the operating system of the detected device senses the occurrence of a failure, the callback function will be called to send the aforementioned broadcast message to the detecting device.
- the callback function registers the above-mentioned broadcast message, then the pre-registered broadcast message can be directly sent to the detection device when the callback function is called.
- the callback function may be used to generate the above-mentioned broadcast message, that is, only the information included in the above-mentioned broadcast message, such as the unique identification and destination port number of the detected device, are registered in the callback function.
- the callback function When the callback function is called, the broadcast message needs to be generated according to the pre-registered information, and then the generated broadcast message is sent to the detection device.
- Figure 3 includes the user space and operating system space of the detected device.
- the operating system kernel registers the fault notification chain.
- the notification is called The chain realizes proactive notification of failures.
- the chain realizes proactive notification of failures.
- Embodiment 2 When the detected device has the above-mentioned second type of fault, the active fault notification to the detecting device is realized.
- the operating system of the detected device because the operating system of the detected device has stopped working when the second type of failure occurs, it cannot be sensed, but the motherboard of the detected device can sense the second type of failure. In addition, because the operating system of the detected device stops working, it is also impossible to notify the detecting device that the detected device is malfunctioning through a normal communication method.
- the fault notification that is, the above-mentioned broadcast message
- the network card driver of the network card of the detected device registers the above-mentioned broadcast message, and the network card driver adds the processing logic for sending the broadcast message to the detection device when the detected device has the second type of failure. That is, a computer program.
- the detected device can detect the failure through the motherboard. Then, the main board generates a notification signal according to the fault, and sends the notification signal to the network card of the detected device to trigger the network card to execute the above-mentioned processing logic. That is, the network card sends the broadcast message pre-registered in the network card driver to the detection device according to the notification signal.
- the above notification signal may be a hardware signal, for example, it may be an AC_LOST signal.
- the notification signal can also be other self-defined signals. This solution does not restrict which signal is used.
- Figure 4 includes the main board and network card of the detected device.
- the network card driver of the network card registers the broadcast message and the corresponding calculation program.
- a notification is sent to trigger the network card to send the broadcast message to the detection device to realize the active notification of the failure.
- the above-mentioned broadcast message may include the cause of the specific failure, for example, whether it is a power failure or a restart failure.
- the broadcast message sent to the detection device after the failure may be the same or different.
- the main purpose is to inform the detection device that a detected device has a fault and cannot process services.
- the detected device may first send a broadcast message to the detecting device using the method of the first embodiment, and then the second type of fault is caused by the first type of fault The detected device is triggered to send a broadcast message to the detecting device in the manner described in the second embodiment.
- the broadcast messages sent in these two times may be the same or different, but both indicate that the detected device has malfunctioned and cannot work normally. In order to facilitate understanding, the following examples illustrate.
- Example 3 Assuming that the detected device needs to be shut down first, the operating system will call the callback function of the notification chain to send broadcast messages when executing the shutdown process. Then, the detected device is shut down, and the operating system cannot work after shutdown. At this time, the motherboard senses this situation and triggers the network card to send a broadcast message to the detection device.
- the method of the above-mentioned embodiment 2 may no longer be used to send a broadcast message to the detecting device.
- the detection device sends a broadcast message. That is, for the second type of fault caused by the first type of fault, the mainboard no longer triggers the network card to send a broadcast message to the detection device after sensing the second type of fault.
- the detected device actively detects its own failure, and once a failure is found, it immediately sends a failure notification to the detecting device.
- a heartbeat detection type detection technology is used to detect whether a device is malfunctioning, and the entire process takes 10 seconds or even tens of seconds, and this process is likely to cause the problem of large service delay or even interruption.
- the embodiment of the present application can shorten the detection time of the detection device to the detected device's fault to the millisecond level, thereby avoiding the problem of large service delay or even interruption caused by the detection device's long detection device failure time.
- the fault perception time is shortened to the millisecond level, the problem of detected devices being kicked out of the cluster due to problems such as network delay/disorder can also be avoided.
- each device includes a corresponding hardware structure and/or software module for performing each function.
- the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
- the embodiments of the present application can divide the detection device and the detected device into functional modules based on the foregoing method examples.
- each functional module can be divided corresponding to each function, or two or more functions can be integrated into one module.
- the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
- FIG. 5 shows a schematic diagram of the logical structure of a fault notification device provided in an embodiment of the present application.
- the fault notification device may be the detected device in the foregoing method embodiment.
- the failure notification device 500 may include:
- the first sending unit 501 is configured to send a broadcast message to the detection device when the failure notification device 500 detects that it has a failure; wherein the broadcast message is used to indicate that the failure notification device 500 has a failure; the broadcast message
- the text is an unreliable transmission protocol packet.
- the above-mentioned fault is a fault that the operating system of the fault notification device 500 cannot perceive;
- the fault notification device 500 includes a motherboard and a network card;
- the fault notification device 500 further includes a first detection unit and a second sending unit;
- the above-mentioned first detection unit is configured to detect that the failure notification device 500 has the above-mentioned failure through the above-mentioned motherboard;
- the second sending unit is configured to send a notification signal to the network card through the main board; the notification signal is generated by the main board according to the failure;
- the first sending unit is specifically configured to send the broadcast message to the detection device according to the notification signal through the network card.
- the above-mentioned broadcast message is registered in the network card driver of the above-mentioned network card.
- the above-mentioned fault is a fault that can be sensed by the operating system of the fault notification device 500; the fault notification device 500 further includes a second detection unit;
- the above-mentioned second detection unit is configured to detect the above-mentioned fault in the fault notification device 500 through the above-mentioned operating system;
- the first sending unit is specifically configured to send the broadcast message to the detection device through the kernel notification chain of the operating system.
- the above-mentioned broadcast message is registered in the callback function included in the above-mentioned kernel notification chain; or, the callback function included in the above-mentioned kernel notification chain is used to generate the above-mentioned broadcast message.
- the above-mentioned broadcast message includes the unique identifier of the failure notification device 500 in the above-mentioned distributed cluster; wherein, the above-mentioned unique identifier is an identifier regenerated when the failure notification device 500 recently joined the above-mentioned distributed cluster.
- the above-mentioned unreliable transmission protocol is a user datagram protocol
- the above-mentioned broadcast message sent by the above-mentioned detected device to the above-mentioned detecting device includes a plurality of the above-mentioned broadcast messages.
- FIG. 6 shows a schematic diagram of a possible hardware structure of a fault notification device provided by an embodiment of this application.
- the fault notification device 600 includes a processor 601, a memory 602, and a communication interface 603.
- the processor 601, the communication interface 603, and the memory 602 may be connected to each other or connected to each other through a bus 604.
- the memory 602 is used to store computer programs and data of the first vehicle 600.
- the memory 602 may include, but is not limited to, random access memory (RAM) and read-only memory (ROM). , Erasable programmable read-only memory (erasable programmable read-only memory, EPROM) or portable read-only memory (compact disc read-only memory, CD-ROM), etc.
- the communication interface 603 is used to support the device 600 to communicate, for example, to receive or send data.
- the processor 601 may be a central processing unit, a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
- the processor may also be a combination of computing functions, for example, a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on.
- the processor 601 may be used to read the program stored in the memory 602, and execute the operations performed by the detected device in the method described in FIG. 2 and possible implementation manners.
- the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the detection device in the method described in FIG. 2 and possible implementations. Do the operation.
- the embodiments of the present application also provide a computer program product.
- the computer program in the computer program product is read and executed by a computer, the method described in FIG. 2 and possible implementations will be executed.
- the detected device actively detects its own failure, and once a failure is found, it immediately sends a failure notification to the detection device.
- this application can enable the detection device to quickly perceive the existing failure and respond to the failure. Respond, so as to avoid the problem of long service delay or even interruption caused by the long time of detecting failure of the detection equipment.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
本申请实施例提供了一种故障通知方法及相关设备,该方法应用于分布式集群中的设备,该设备包括探测设备和被探测设备; 该方法包括: 在该被探测设备检测到自身出现故障的情况下,该被探测设备向该探测设备发送广播报文; 其中,该广播报文用于指示该被探测设备出现故障; 该广播报文为不可靠传输协议的报文。采用本申请实施例,能够极大减少分布式集群中探测设备感知被探测设备出现故障的时间,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。
Description
本申请要求于2020年02月10日提交中国专利局、申请号为202010084819.6、申请名称为“故障通知方法及相关设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本发明涉及计算机技术领域,尤其涉及一种故障通知方法及相关设备。
在分布式系统中,故障探测主要是通过心跳探测等类似的探测技术发现故障设备。具体的,探测设备每隔一段时间(如3秒)会向分布式集群中的被探测设备发送一次心跳请求,被探测设备收到心跳请求后及时应答探测设备以表明自己可以正常提供业务服务。如果一段时间内(如10s,即发送了3个心跳请求)一直未收到被探测设备的应答,则探测设备认为该被探测设备出现故障。
但是心跳类探测技术探测时间过长,容易造成故障期间业务时延大甚至中断的问题。综上所述,如何解决故障探测时间过长导致故障期间业务时延大甚至中断的问题是本领域技术人员急需解决的技术问题。
发明内容
本申请实施例公开了一种故障通知方法及相关设备,能够极大减少分布式集群中探测设备感知被探测设备出现故障的时间,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。
第一方面,本申请公开了一种故障通知方法,该方法应用于分布式集群中的设备,这些设备包括探测设备和被探测设备。该方法包括:
在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文;其中,上述广播报文用于指示上述被探测设备出现故障;该广播报文为不可靠传输协议的报文。
本申请中被探测设备主动检测自身的故障,一旦发现故障立即向探测设备发送故障通知,相比于现有技术,本申请可以使得探测设备快速感知到存在的故障并对故障做出响应,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。
在一种可能的实施方式中,上述故障为上述被探测设备的操作系统无法感知的故障;上述被探测设备包括主板和网卡;上述在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文,包括:
上述被探测设备通过上述主板检测到上述被探测设备出现上述故障;上述被探测设备通过上述主板向上述网卡发送通知信号;上述通知信号为上述主板根据上述故障生成;上述被探测设备通过上述网卡根据上述通知信号向上述探测设备发送上述广播报文。
在本申请中,通过被探测设备的主板感知故障,并通过被探测设备的网卡主动向探测设备发送故障通知即上述广播报文,从而解决了被探测设备中操作系统无法感知的故障的 通知问题。
在一种可能的实施方式中,上述广播报文注册在上述网卡的网卡驱动中。
在本申请中,预先将出现故障时的故障通知注册在网卡驱动程序中,一旦出现故障可以立即将故障通知消息发送给探测设备,便捷快速。
在一种可能的实施方式中,上述故障为上述被探测设备的操作系统能够感知的故障;上述在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文,包括:
上述被探测设备通过上述操作系统检测到上述被探测设备出现上述故障;
上述被探测设备通过上述操作系统的内核通知链向上述探测设备发送上述广播报文。
本申请通过内核通知链向探测设备主动通知被探测设备出现故障,方便快捷。
在一种可能的实施方式中,上述内核通知链包括的回调函数中注册了上述广播报文;或者,上述内核通知链包括的回调函数用于生成上述广播报文。
在一种可能的实施方式中,上述广播报文包括上述被探测设备在上述分布式集群中的唯一标识;其中,上述唯一标识为上述被探测设备最近一次加入上述分布式集群时重新生成的标识。
本申请将被探测设备在分布式集群中的唯一标识添加到故障通知中,探测设备可以根据该唯一标识对重复接收到的故障通知直接丢弃,从而节约了计算资源。
在一种可能的实施方式中,上述不可靠传输协议为用户数据报协议(user datagram protocol,UDP);上述被探测设备向上述探测设备发送的上述广播报文包括多个上述广播报文。
本申请采用UDP广播报文来承载故障通知的信息,能够利用UDP无连接协议特性,即使在设备掉电,操作系统宕机或主机、程序停止工作等情况下也能及时完成故障通知。采用UDP广播报文还可以在集群主故障或多点故障的情况下都能够实现故障通知。此外,通过向探测设备发送多个该广播报文可以避免UDP不可靠传输导致报文丢失的问题。
第二方面,本申请公开了一种故障通知设备,上述故障通知设备属于分布式集群中的被探测设备,上述分布式集群还包括探测设备;上述故障通知设备包括:
第一发送单元,用于在上述故障通知设备检测到自身出现故障的情况下,向上述探测设备发送广播报文;其中,上述广播报文用于指示上述故障通知设备出现故障;该广播报文为不可靠传输协议的报文。
在一种可能的实施方式中,上述故障为上述故障通知设备的操作系统无法感知的故障;上述故障通知设备包括主板和网卡;上述故障通知设备还包括第一检测单元和第二发送单元;
上述第一检测单元,用于通过上述主板检测到上述故障通知设备出现上述故障;
上述第二发送单元,用于通过上述主板向上述网卡发送通知信号;上述通知信号为上述主板根据上述故障生成;
上述第一发送单元,具体用于通过上述网卡根据上述通知信号向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述广播报文注册在上述网卡的网卡驱动中。
在一种可能的实施方式中,上述故障为上述故障通知设备的操作系统能够感知的故障;上述故障通知设备还包括第二检测单元;
上述第二检测单元,用于通过上述操作系统检测到上述故障通知设备出现上述故障;
上述第一发送单元,具体用于通过上述操作系统的内核通知链向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述内核通知链包括的回调函数中注册了上述广播报文;或者,上述内核通知链包括的回调函数用于生成上述广播报文。
在一种可能的实施方式中,上述广播报文包括上述故障通知设备在上述分布式集群中的唯一标识;其中,上述唯一标识为上述故障通知设备最近一次加入上述分布式集群时重新生成的标识。
在一种可能的实施方式中,上述不可靠传输协议为用户数据报协议UDP;上述被探测设备向上述探测设备发送的上述广播报文包括多个上述广播报文。
第三方面,本申请公开了一种故障通知设备,上述故障通知设备属于分布式集群中的被探测设备,上述分布式集群还包括探测设备。上述故障通知设备包括处理器、存储器以及通信接口;上述存储器、上述通信接口与上述处理器耦合,上述存储器存储有计算机程序,上述处理器执行上述计算机程序时,上述故障通知设备执行如下操作:
在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文;其中,上述广播报文用于指示上述被探测设备出现故障;该广播报文为不可靠传输协议的报文。
在一种可能的实施方式中,上述故障为上述被探测设备的操作系统无法感知的故障;上述故障通知设备包括主板和网卡;上述在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文,包括:
上述被探测设备通过上述主板检测到上述被探测设备出现上述故障;上述被探测设备通过上述主板向上述网卡发送通知信号;上述通知信号为上述主板根据上述故障生成;上述被探测设备通过上述网卡根据上述通知信号向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述广播报文注册在上述网卡的网卡驱动中。
在一种可能的实施方式中,上述故障为上述被探测设备的操作系统能够感知的故障;上述在上述被探测设备检测到自身出现故障的情况下,上述被探测设备向上述探测设备发送广播报文,包括:
上述被探测设备通过上述操作系统检测到上述被探测设备出现上述故障;
上述被探测设备通过上述操作系统的内核通知链向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述内核通知链包括的回调函数中注册了上述广播报文;或者,上述内核通知链包括的回调函数用于生成上述广播报文。
在一种可能的实施方式中,上述广播报文包括上述被探测设备在上述分布式集群中的唯一标识;其中,上述唯一标识为上述被探测设备最近一次加入上述分布式集群时重新生成的标识。
在一种可能的实施方式中,上述不可靠传输协议为用户数据报协议UDP;上述被探测设备向上述探测设备发送的上述广播报文包括多个上述广播报文。
第四方面,本申请公开了一种计算机可读存储介质,该计算机可读存储介质存储有计算机程序,该计算机程序被处理器执行以实现上述第一方面任意一项所述的方法。
第五方面,本申请提供一种计算机程序产品,当该计算机程序产品中的计算机程序被计算机读取并执行时,上述第一方面任意一项所述的方法将被执行。
综上所述,本申请中被探测设备主动检测自身的故障,一旦发现故障立即向探测设备发送故障通知,相比于现有技术,本申请可以使得探测设备快速感知到存在的故障并对故障做出响应,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。
图1为本申请实施例提供的故障通知方法适用的系统架构示意图;
图2为本申请实施例提供的一种故障通知方法的流程示意图;
图3为本申请实施例提供的通过通知链实现故障通知的流程示意图;
图4为本申请实施例提供的通过网卡实现故障通知的流程示意图;
图5为本申请实施例提供的一种故障通知设备的逻辑结构示意图;
图6为本申请实施例提供的一种故障通知设备的硬件结构示意图。
下面结合附图对本申请实施例中的技术方案进行描述。
为了更好的理解本申请实施例提供的一种故障通知方法,下面先对本申请实施例适用的场景进行示例性地描述。参阅图1,图1是本申请实施例提供的故障通知方法适用的系统构架示意图。如图1所示,系统构架可以包括一个或多个探测设备101和一个或多个被探测设备102。探测设备101和被探测设备102可以是属于同一个分布式集群中的设备。
探测设备101可以用于探测被探测设备102是否出现故障,以保证在被探测设备102出现故障的情况下及时响应,从而减少对业务处理的影响。在本申请实施例中,当被探测设备102出现故障时,该被探测设备102可以主动将出现的故障事件通知给探测设备101,从而极大减少了探测设备101感知到该故障的时间,进而避免了故障探测时间过长导致的业务时延大甚至中断的问题。
每一个探测设备101可以用于探测一个或多个被探测设备102是否出现故障,每一个被探测设备102也可以被一个或多个探测设备探测是否出现故障,具体的探测设备和被探测设备根据实际情况确定,本方案对此不做限制。
可选的,探测设备101可以是分布式集群中用于分发业务处理任务给被探测设备102的设备,被探测设备102可以是分布式集群中用于执行任务的设备。探测设备101需要知道被探测设备102是否出现故障,以保证在被探测设备102出现故障的情况下可以将由该被探测设备102执行的任务分配给其它正常的被探测设备102执行,从而保证任务的正常执行。
可选的,探测设备101和被探测设备102分别是分布式集群中用于分发业务处理任务的从设备和主设备。从设备需要知道主设备是否出现故障,以保证在主设备出现故障的情况下可以将由该主设备执行的任务切换到该从设备来执行,从而保证任务的正常执行。
可选的,同一个设备既可以是探测设备也可以是被探测设备。例如,从上述两个可选的实施例中可以看到,分布式集群中用于分发业务处理任务的主设备既可以作为探测设备用于探测该分布式集群中用于执行任务的设备是否出现故障,又可以是被探测设备被其从设备监控探测是否出现故障。
需要说明的是,本申请实施例提供的故障通知方法适用的系统构架不限于图1所示架构,分布式系统中的集群架构都属于本申请实施例提供的故障通知方法适用的系统构架,此处不再赘述。
下面提供一种故障通知方法,该方法可以适用于上述图1所示的系统架构。参见图2,该方法包括但不限于如下步骤:
步骤201、被探测设备检测到自身出现故障。
在具体实施例中,本申请实施例涉及的故障包括两类,第一类是被探测设备的操作系统(operating system,OS)可以感知的故障,第二类是OS无法感知的故障即OS无法正常工作。
需要说明的是,本申请实施例所涉及的故障包括被探测设备无法正常执行业务处理任务的情况。
那么,第一类故障可以包括重启(reboot)、关机(shutdown)和初始化(init)等主动复位的情况以及包括内存溢出(out of memory,oom)、急关断(emerge)、看门狗(watchdog)和不可预知的事情(panic)等发起的操作系统被动复位的情况。第一类故障还可以包括用杀死Kill命令结束进程和进程Crash等引起的进程故障。这里的进程Crash是指在正常设备系统运行过程中,因某种原因宕机,或者主机或程序停止工作等情况。
第二类故障可以包括被探测设备的OS异常复位或者直接掉电的情况,例如包括系统崩溃(system crash)、掉电、长按关机按钮和智能平台管理接口(intelligent platform management interface,IPMI)强制下电等情况。IPMI是使硬件管理具备“智能化”的新一代通用接口标准。用户可以利用IPMI监视设备的物理特征,如温度、电压、电扇工作状态、电源供应以及机箱入侵等。
在本申请实施例中,针对上述两类故障,上述被探测设备分别采用两种不同的方式检测故障。具体的,对于第一类故障,被探测设备通过自身的操作系统来检测故障。对于第二类故障,被探测设备通过自身的主板来检测故障。
步骤202、该被探测设备向探测设备发送广播报文;其中,该广播报文用于指示该被探测设备出现故障;该广播报文为不可靠传输协议的报文。
在具体实施例中,上述广播报文包括目的端口,该目的端口为预设的端口,探测设备预先配置监听该预设的目的端口。当上述被探测设备将上述广播报文发送到该目的端口时,该探测设备可以接收到该广播报文。
可选的,不同的被探测设备发送的广播报文的目的端口可以不同,这些目的端口可以与被探测设备一一映射,探测设备可以通过接收到广播报文的端口号来确定是哪个被探测设备出现了故障。
可选的,上述广播报文中包括上述被探测设备的唯一标识,该唯一标识可以是在分布 式集群中唯一标识该被探测设备的序列号或者标识码等。例如该唯一标识可以是会话标识session id等。在具体实施例中,探测设备接收到该广播报文后,可以根据该广播报文中的唯一标识确定是哪个被探测设备出现了故障。
此外,上述广播报文中包括被探测设备的唯一标识可以使得探测设备再次接收到相同的广播报文时,可以根据该唯一标识获知接收的为重复的报文,可以直接丢弃,从而节省了重复处理的计算资源。
需要说明的是,如果被探测设备恢复正常,可以正常处理业务后,该被探测设备会重新生成自身在分布式集群中的唯一标识,并将该重新生成的唯一标识告知该分布式集群中的其他设备。因此,故障时发送的广播报文中包括的唯一标识为该被探测设备最近一次加入分布式集群时重新生成的标识。当探测设备接收到该被探测设备发送的广播报文时,如果该广播报文中的唯一标识不是该被探测设备新生成的唯一标识,那么可以直接丢弃,从而避免了误判故障的问题。
为了便于理解,举例说明:
例一、如果该探测设备是分布式集群中用于分配业务处理任务的设备,那么,该探测设备接收到该广播报文之后,获知该被探测设备无法正常执行业务处理任务。为了不影响业务的正常处理,该探测设备可以将该故障的被探测设备踢出该集群,即该探测设备不会再将业务处理任务分配给该被探测设备进行处理,会找其它可用的设备来处理相应地业务。直到该被探测设备恢复正常,并申请重新加入集群,此时该被探测设备会重新生成在该集群中的唯一标识号。
例二、如果该探测设备是分布式集群中分发业务处理任务的从设备,该被探测设备是分布式集群中分发业务处理任务的主设备,那么,该探测设备接收到该故障通知消息之后,获知该被探测设备无法正常执行业务处理任务的分发工作,为了不影响业务的正常处理,该探测设备会接管业务处理任务的分发工作。并可以将该故障的被探测设备踢出该集群,直到该被探测设备恢复正常,并申请重新加入集群,此时该被探测设备会重新生成在该集群中的唯一标识号。
上述采用不可靠传输协议来实现被探测设备对探测设备的主动故障通知,能够保证被探测设备出现故障的情况下,也可以向探测设备发送故障通知即上述广播报文,从而实现了探测设备对被探测设备的故障快速感知。
可选的,上述不可靠传输协议可以是用户数据报协议(user datagram protocol,UDP),即上述广播报文为UDP广播报文。利用UDP无连接协议特性,即使在设备掉电,操作系统宕机或主机、程序停止工作等情况下也能及时完成故障通知。此外,采用UDP广播报文可以在集群主故障或多点故障的情况下都能够实现故障通知。
可选的,上述被探测设备可以连续发送多次上述广播报文给探测设备以保证探测和设备成功接收到该报文。例如,假设该广播报文为UDP广播报文,由于UDP的特性是尽最大努力交付,不保证可靠交付,因此可以多发几次广播报文以解决UDP丢包导致报文丢失的问题。
在具体实施例中,针对上述两类故障,被探测设备分别采用两种不同的方式向探测设备发送广播报文。具体的,对于第一类故障,被探测设备通过该设备操作系统的内核通知 链向探测设备发送广播报文。对于第二类故障,被探测设备通过所述网卡向探测设备发送广播报文。
下面通过两个实施例来分别介绍上述被探测设备出现上述两类故障时实现故障主动通知的具体过程。
实施例一、被探测设备出现上述第一类故障时实现对探测设备的主动故障通知。
对于该第一类故障,被探测设备的OS内核中注册了故障的通知链,可选的,可以在该探测设备的业务启动时注册该通知链。
具体的,该通知链可以包括回调函数,当上述被探测设备的操作系统感知到故障的发生时,会调用该回调函数向探测设备发送上述广播报文。
可选的,该回调函数注册了上述广播报文,那么在该回调函数被调用时可以直接将该预先注册的广播报文发送给探测设备。
可选的,该回调函数可以用于生成上述广播报文,即该回调函数中只注册了上述广播报文中包括的信息例如被探测设备的唯一标识和目的端口号等。当该回调函数被调用时,需要先根据预先注册的信息生成该广播报文,再将生成的广播报文发送给探测设备。
为了便于理解实施例一,可以参见图3。图3中包括被探测设备的用户空间和操作系统空间,当用户空间的业务启动时,操作系统的内核中注册了故障的通知链,当操作系统感知到第一类故障出现时,调用该通知链实现故障的主动通知。具体的实现过程参见上述实施例一的描述,此处不再赘述。
实施例二、被探测设备出现上述第二类故障时实现对探测设备的主动故障通知。
在具体实施例中,由于出现上述第二类故障时,被探测设备的操作系统已经停止工作,无法感知,但是被探测设备的主板可以感知该第二类故障。此外,由于被探测设备的操作系统停止工作,因此也无法通过正常的通信方式通知探测设备该被探测设备出现了故障。
基于此,在本申请实施例中,可以通过被探测设备的网卡向探测设备发送故障通知即上述广播报文。具体的,被探测设备网卡的网卡驱动中注册了上述广播报文,并且在该网卡驱动中添加了当该被探测设备出现该第二类故障时,向探测设备发送该广播报文的处理逻辑即计算机程序。
在出现第二类故障的情况下,被探测设备可通过主板检测到该故障。然后,该主板根据该故障生成一个通知信号,并将该通知信号发送给被探测设备的网卡,触发该网卡执行上述处理逻辑。即该网卡根据该通知信号将预先在网卡驱动中注册的广播报文发送给探测设备。
可选的,上述通知信号可以是硬件信号,例如可以是AC_LOST信号等。该通知信号也可以是其它自定义的信号,具体采用哪种信号,本方案对此不做限制。
为了便于理解实施例二,可以参见图4。图4中包括被探测设备的主板和网卡,当被探测设备的业务启动时,网卡的网卡驱动中注册了广播报文和相应的计算程序,当主板感知到第二类故障出现时,向网卡发出通知,触发网卡向探测设备发送该广播报文实现故障的主动通知。具体的实现过程参见上述实施例二的描述,此处不再赘述。
可选的,上述广播报文中可以包括具体故障的原因,例如是掉电故障还是重启故障等等。
可选的,不管是什么故障,出现故障后向探测设备发送的广播报文可以相同也可以不同,主要目的是告知探测设备某个被探测设备出现了故障,无法处理业务即可。
可选的,当出现第一类故障后引发第二类故障时,被探测设备可以先采用上述实施例一的方式向探测设备发送广播报文,然后由第一类故障引发的第二类故障触发被探测设备采用上述实施例二的方式向探测设备发送广播报文。这两次发送的广播报文可以相同也可以不相同,但都指示该被探测设备出现了故障,无法正常工作。为了便于理解,下面举例说明。
例三、假设被探测设备首先出现了需要关机的情况,那么操作系统在执行关机流程的时候会调用通知链的回调函数实现广播报文的发送。然后,被探测设备关机,关机之后操作系统无法工作,此时主板感知到这个情况,会触发网卡向探测设备发送广播报文。
可选的,当出现第一类故障后引发第二类故障时,在被探测设备已经采用上述实施例一的方式向探测设备发送广播报文后,可以不再采用上述实施例二的方式向探测设备发送广播报文。即对于由第一类故障引发的第二类故障,主板感知到该第二类故障后不再触发网卡向探测设备发送广播报文。
综上所述,本申请中被探测设备主动检测自身的故障,一旦发现故障立即向探测设备发送故障通知。现有技术中采用心跳探测类探测技术探测设备是否出现故障,整个过程需要10秒甚至几十秒,在这个过程中容易导致业务时延大甚至中断的问题。而本申请实施例能够将探测设备对被探测设备的故障的感知时间缩短至毫秒级别,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。此外,由于故障的感知时间缩短至毫秒级别,还可以避免因为网络时延/乱序等问题导致的被探测设备被误踢出集群的问题。
上述主要从探测设备和被探测设备之间的交互对故障通知方法进行了介绍。可以理解的是,各个设备为了实现上述对应的功能,其包括了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到,结合本文中所公开的实施例描述的各示例的单元及算法步骤,本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请的范围。
本申请实施例可以根据上述方法示例对探测设备和被探测设备等进行功能模块的划分,例如,可以对应各个功能划分各个功能模块,也可以将两个或两个以上的功能集成在一个模块中。上述集成的模块既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。需要说明的是,本申请实施例中对模块的划分是示意性的,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下,图5示出了本申请实施例提供的一种故障通知设备的逻辑结构示意图,该故障通知设备可以是上述方法实施例中的被探测设备。该故障通知设备500可以包括:
第一发送单元501,用于在故障通知设备500检测到自身出现故障的情况下,向上述探测设备发送广播报文;其中,上述广播报文用于指示故障通知设备500出现故障;该广 播报文为不可靠传输协议的报文。
在一种可能的实施方式中,上述故障为故障通知设备500的操作系统无法感知的故障;故障通知设备500包括主板和网卡;故障通知设备500还包括第一检测单元和第二发送单元;
上述第一检测单元,用于通过上述主板检测到故障通知设备500出现上述故障;
上述第二发送单元,用于通过上述主板向上述网卡发送通知信号;上述通知信号为上述主板根据上述故障生成;
上述第一发送单元,具体用于通过上述网卡根据上述通知信号向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述广播报文注册在上述网卡的网卡驱动中。
在一种可能的实施方式中,上述故障为故障通知设备500的操作系统能够感知的故障;故障通知设备500还包括第二检测单元;
上述第二检测单元,用于通过上述操作系统检测到故障通知设备500出现上述故障;
上述第一发送单元,具体用于通过上述操作系统的内核通知链向上述探测设备发送上述广播报文。
在一种可能的实施方式中,上述内核通知链包括的回调函数中注册了上述广播报文;或者,上述内核通知链包括的回调函数用于生成上述广播报文。
在一种可能的实施方式中,上述广播报文包括故障通知设备500在上述分布式集群中的唯一标识;其中,上述唯一标识为故障通知设备500最近一次加入上述分布式集群时重新生成的标识。
在一种可能的实施方式中,上述不可靠传输协议为用户数据报协议;上述被探测设备向上述探测设备发送的上述广播报文包括多个上述广播报文。
图6所示,为本申请实施例提供的一种故障通知设备的可能的硬件结构示意图。故障通知设备600包括:处理器601、存储器602和通信接口603。处理器601、通信接口603以及存储器602可以相互连接或者通过总线604相互连接。
示例性的,存储器602用于存储第一车辆600的计算机程序和数据,存储器602可以包括但不限于是随机存储记忆体(random access memory,RAM)、只读存储器(read-only memory,ROM)、可擦除可编程只读存储器(erasable programmable read only memory,EPROM)或便携式只读存储器(compact disc read-only memory,CD-ROM)等。通信接口603用于支持设备600进行通信,例如接收或发送数据。
示例性的,处理器601可以是中央处理器单元、通用处理器、数字信号处理器、专用集成电路、现场可编程门阵列或者其他可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。处理器也可以是实现计算功能的组合,例如包含一个或多个微处理器组合,数字信号处理器和微处理器的组合等等。处理器601可以用于读取上述存储器602中存储的程序,执行上述图2以及可能的实施方式所述方法中被探测设备所做的操作。
本申请实施例还提供一种计算机可读存储介质,该计算机可读存储介质存储有计算机程序,该计算机程序被处理器执行以实现上述图2以及可能的实施方式所述方法中被探测 设备所做的操作。
本申请实施例还提供一种计算机程序产品,当该计算机程序产品中的计算机程序被计算机读取并执行时,上述图2以及可能的实施方式所述的方法将被执行。
综上所述,本申请中被探测设备主动检测自身的故障,一旦发现故障立即向探测设备发送故障通知,相比于现有技术,本申请可以使得探测设备快速感知到存在的故障并对故障做出响应,从而避免因探测设备感知故障时间过长导致的业务时延大甚至中断的问题。
最后应说明的是:以上各实施例仅用以说明本发明的技术方案,而非对其限制;尽管参照前述各实施例对本发明进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分或者全部技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。
Claims (16)
- 一种故障通知方法,其特征在于,所述方法应用于分布式集群中的设备,所述设备包括探测设备和被探测设备;所述方法包括:在所述被探测设备检测到自身出现故障的情况下,所述被探测设备向所述探测设备发送广播报文;其中,所述广播报文用于指示所述被探测设备出现故障;所述广播报文为不可靠传输协议的报文。
- 根据权利要求1所述的方法,其特征在于,所述故障为所述被探测设备的操作系统无法感知的故障;所述被探测设备包括主板和网卡;所述在所述被探测设备检测到自身出现故障的情况下,所述被探测设备向所述探测设备发送广播报文,包括:所述被探测设备通过所述主板检测到所述被探测设备出现所述故障;所述被探测设备通过所述主板向所述网卡发送通知信号;所述通知信号为所述主板根据所述故障生成;所述被探测设备通过所述网卡根据所述通知信号向所述探测设备发送所述广播报文。
- 根据权利要求2所述的方法,其特征在于,所述广播报文注册在所述网卡的网卡驱动中。
- 根据权利要求1所述的方法,其特征在于,所述故障为所述被探测设备的操作系统能够感知的故障;所述在所述被探测设备检测到自身出现故障的情况下,所述被探测设备向所述探测设备发送广播报文,包括:所述被探测设备通过所述操作系统检测到所述被探测设备出现所述故障;所述被探测设备通过所述操作系统的内核通知链向所述探测设备发送所述广播报文。
- 根据权利要求4所述的方法,其特征在于,所述内核通知链包括的回调函数中注册了所述广播报文;或者,所述内核通知链包括的回调函数用于生成所述广播报文。
- 根据权利要求1至5任一项所述的方法,其特征在于,所述广播报文包括所述被探测设备在所述分布式集群中的唯一标识;其中,所述唯一标识为所述被探测设备最近一次加入所述分布式集群时重新生成的标识。
- 根据权利要求1至6任一项所述的方法,其特征在于,所述不可靠传输协议为用户数据报协议UDP;所述被探测设备向所述探测设备发送的所述广播报文包括多个所述广播报文。
- 一种故障通知设备,其特征在于,所述故障通知设备属于分布式集群中的被探测设备,所述分布式集群还包括探测设备;所述故障通知设备包括:第一发送单元,用于在所述故障通知设备检测到自身出现故障的情况下,向所述探测设备发送广播报文;其中,所述广播报文用于指示所述故障通知设备出现故障;所述广播报文为不可靠传输协议的报文。
- 根据权利要求8所述的故障通知设备,其特征在于,所述故障为所述故障通知设备 的操作系统无法感知的故障;所述故障通知设备包括主板和网卡;所述故障通知设备还包括第一检测单元和第二发送单元;所述第一检测单元,用于通过所述主板检测到所述故障通知设备出现所述故障;所述第二发送单元,用于通过所述主板向所述网卡发送通知信号;所述通知信号为所述主板根据所述故障生成;所述第一发送单元,具体用于通过所述网卡根据所述通知信号向所述探测设备发送所述广播报文。
- 根据权利要求9所述的故障通知设备,其特征在于,所述广播报文注册在所述网卡的网卡驱动中。
- 根据权利要求8所述的故障通知设备,其特征在于,所述故障为所述故障通知设备的操作系统能够感知的故障;所述故障通知设备还包括第二检测单元;所述第二检测单元,用于通过所述操作系统检测到所述故障通知设备出现所述故障;所述第一发送单元,具体用于通过所述操作系统的内核通知链向所述探测设备发送所述广播报文。
- 根据权利要求11所述的故障通知设备,其特征在于,所述内核通知链包括的回调函数中注册了所述广播报文;或者,所述内核通知链包括的回调函数用于生成所述广播报文。
- 根据权利要求8至12任一项所述的故障通知设备,其特征在于,所述广播报文包括所述故障通知设备在所述分布式集群中的唯一标识;其中,所述唯一标识为所述故障通知设备最近一次加入所述分布式集群时重新生成的标识。
- 根据权利要求8至13任一项所述的故障通知设备,其特征在于,所述不可靠传输协议为用户数据报协议UDP;所述被探测设备向所述探测设备发送的所述广播报文包括多个所述广播报文。
- 一种故障通知设备,其特征在于,所述故障通知设备包括处理器、存储器以及通信接口;所述存储器、所述通信接口与所述处理器耦合,所述存储器存储有计算机程序,所述处理器执行所述计算机程序时,所述故障通知设备执行如权利要求1至7任一项所述的方法。
- 一种计算机程序产品,其特征在于,所述计算机程序产品存储有计算机程序,所述计算机程序被处理器执行以实现权利要求1至7任意一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010084819.6A CN111338914A (zh) | 2020-02-10 | 2020-02-10 | 故障通知方法及相关设备 |
CN202010084819.6 | 2020-02-10 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021159897A1 true WO2021159897A1 (zh) | 2021-08-19 |
Family
ID=71183398
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/071042 WO2021159897A1 (zh) | 2020-02-10 | 2021-01-11 | 故障通知方法及相关设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111338914A (zh) |
WO (1) | WO2021159897A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111338914A (zh) * | 2020-02-10 | 2020-06-26 | 华为技术有限公司 | 故障通知方法及相关设备 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102970167A (zh) * | 2012-11-26 | 2013-03-13 | 华为技术有限公司 | 集群系统中网络节点的故障检测方法、网络节点和系统 |
US20140108533A1 (en) * | 2012-10-15 | 2014-04-17 | Oracle International Corporation | System and method for supporting out-of-order message processing in a distributed data grid |
CN105204977A (zh) * | 2014-06-30 | 2015-12-30 | 中兴通讯股份有限公司 | 一种系统异常的捕获方法、主系统、影子系统及智能设备 |
CN107908537A (zh) * | 2017-11-27 | 2018-04-13 | 郑州云海信息技术有限公司 | 一种基于内核模块异常信息处理的系统及方法 |
CN109831350A (zh) * | 2018-11-01 | 2019-05-31 | 华为技术有限公司 | 设备信息发送的方法、计算机设备和分布式计算机设备系统 |
CN111338914A (zh) * | 2020-02-10 | 2020-06-26 | 华为技术有限公司 | 故障通知方法及相关设备 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10210038B2 (en) * | 2015-10-08 | 2019-02-19 | Lightbend, Inc. | Tuning context-aware rule engine for anomaly detection |
CN106330531B (zh) * | 2016-08-15 | 2019-05-03 | 东软集团股份有限公司 | 节点故障记录和处理的方法以及装置 |
-
2020
- 2020-02-10 CN CN202010084819.6A patent/CN111338914A/zh active Pending
-
2021
- 2021-01-11 WO PCT/CN2021/071042 patent/WO2021159897A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140108533A1 (en) * | 2012-10-15 | 2014-04-17 | Oracle International Corporation | System and method for supporting out-of-order message processing in a distributed data grid |
CN102970167A (zh) * | 2012-11-26 | 2013-03-13 | 华为技术有限公司 | 集群系统中网络节点的故障检测方法、网络节点和系统 |
CN105204977A (zh) * | 2014-06-30 | 2015-12-30 | 中兴通讯股份有限公司 | 一种系统异常的捕获方法、主系统、影子系统及智能设备 |
CN107908537A (zh) * | 2017-11-27 | 2018-04-13 | 郑州云海信息技术有限公司 | 一种基于内核模块异常信息处理的系统及方法 |
CN109831350A (zh) * | 2018-11-01 | 2019-05-31 | 华为技术有限公司 | 设备信息发送的方法、计算机设备和分布式计算机设备系统 |
CN111338914A (zh) * | 2020-02-10 | 2020-06-26 | 华为技术有限公司 | 故障通知方法及相关设备 |
Also Published As
Publication number | Publication date |
---|---|
CN111338914A (zh) | 2020-06-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11994940B2 (en) | Fault processing method, related device, and computer storage medium | |
US9026860B2 (en) | Securing crash dump files | |
US8713350B2 (en) | Handling errors in a data processing system | |
US8839032B2 (en) | Managing errors in a data processing system | |
US9021317B2 (en) | Reporting and processing computer operation failure alerts | |
US9189316B2 (en) | Managing failover in clustered systems, after determining that a node has authority to make a decision on behalf of a sub-cluster | |
WO2021027481A1 (zh) | 故障处理方法、装置、计算机设备、存储介质及存储系统 | |
US11330071B2 (en) | Inter-process communication fault detection and recovery system | |
US20130179566A1 (en) | Native bi-directional communication for hardware management | |
US7788520B2 (en) | Administering a system dump on a redundant node controller in a computer system | |
US8984266B2 (en) | Techniques for stopping rolling reboots | |
US7434085B2 (en) | Architecture for high availability using system management mode driven monitoring and communications | |
WO2017215441A1 (zh) | 一种分布式系统中单板配置自恢复方法及装置 | |
US10353786B2 (en) | Virtualization substrate management device, virtualization substrate management system, virtualization substrate management method, and recording medium for recording virtualization substrate management program | |
WO2015058711A1 (zh) | 故障快速检测方法及装置 | |
US10530634B1 (en) | Two-channel-based high-availability | |
WO2021159897A1 (zh) | 故障通知方法及相关设备 | |
WO2020088351A1 (zh) | 设备信息发送的方法、计算机设备和分布式计算机设备系统 | |
US8392751B2 (en) | System and method for recovery from uncorrectable bus errors in a teamed NIC configuration | |
US8036105B2 (en) | Monitoring a problem condition in a communications system | |
CN115033428A (zh) | 分布式数据库的管理方法、系统及管理服务器 | |
CN113760459A (zh) | 虚拟机故障检测方法、存储介质和虚拟化集群 | |
CN110752939B (zh) | 一种业务进程故障处理方法、通知方法和装置 | |
US11797368B2 (en) | Attributing errors to input/output peripheral drivers | |
US10599510B2 (en) | Computer system and error isolation method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21754295 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21754295 Country of ref document: EP Kind code of ref document: A1 |