JP7498128B2 - Monitoring device, fault detection method and fault detection program - Google Patents

Description

The present invention relates to a monitoring device, etc.

As cloud computing becomes more widespread, higher quality than ever is required of data centers (DC) networks, which are the foundation of DCs. In DC networks, silent failures can occur, where network devices such as switches malfunction without issuing a failure alert. Because silent failures are difficult to detect, recovery from the failure can be delayed, potentially affecting many services.

Figure 9 is a diagram for explaining an example of a silent failure. In the example shown in Figure 9, switches 4 and 5 are connected to a monitoring device 6. Switch 4 has a control plane 4a and a data plane 4b. Control plane 4a is a control unit that controls the entire switch 4. Data plane 4b is an ASIC (Application Specific Integrated Circuit) that actually manages data communication. Switch 5, like switch 4, includes a control plane 5a and a data plane 5b.

For example, if an abnormality occurs in the data plane 4b of the switch 4, causing a disruption in communication, but the control plane 4a is normal, this is a silent failure. Here, if the control plane 4a is operating normally, even if the monitoring device 6 sends an SNMP request to the switch 4, an alert indicating the abnormality is not sent to the monitoring device 6, and the monitoring device 6 cannot detect the failure in the data plane 4b through the SNMP request.

Conventional techniques for detecting the above-mentioned silent failures include Conventional Techniques 1 and 2. In Conventional Technique 1, a monitoring device periodically transmits test data to a monitored device, and detects an abnormality (such as a silent failure) based on the presence or absence of a response.

In conventional technology 2, a monitoring device periodically collects information from each monitored device, and a system administrator defines normal network behavior based on the collected information, and detects abnormalities (such as silent failures) based on differences from normal behavior and symptoms.

JP 2020-88786 A JP 2011-211350 A

The above-mentioned conventional technology has the problem that it is not possible to efficiently detect silent failures.

For example, if conventional technology 1 is applied as is to a large-scale network, there is a problem that the amount of traffic increases due to the test data. Furthermore, conventional technology 2 places a heavy burden on the administrator who defines the network behavior under normal circumstances, and it also incurs operational costs.

In one aspect, the present invention aims to provide a monitoring device, a fault detection method, and a fault detection program that can efficiently detect silent faults.

In the first proposal, the monitoring device has an acquisition unit and a detection unit. The acquisition unit acquires a first communication status with the first switch, a second communication status with the second switch, and a third communication status with the other monitoring switches from a monitoring switch that connects a first switch, a second switch, and other monitoring switches to be monitored by a virtual network among multiple switches included in the network. The detection unit detects a failed switch from the first switch and the second switch based on the first communication status, the second communication status, and the third communication status.

It can efficiently detect silent failures.

FIG. 1 is a diagram showing a monitoring system according to the present embodiment. FIG. 2 is a diagram for explaining the IP SLA function. FIG. 3 is a functional block diagram showing the configuration of the monitoring device according to the present embodiment. FIG. 4 is a diagram illustrating an example of the data structure of the pattern table. FIG. 5 illustrates an example of the data structure of the judgment policy table. FIG. 6 is a diagram for explaining an example of route switching by message transmission. FIG. 7 is a flowchart showing a processing procedure of the monitoring device according to the present embodiment. FIG. 8 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the monitoring device of the embodiment. FIG. 9 is a diagram illustrating an example of a silent failure.

Below, examples of the monitoring device, fault detection method, and fault detection program disclosed in this application are described in detail with reference to the drawings. Note that the present invention is not limited to these examples.

Figure 1 is a diagram showing an example of a monitoring system according to the present embodiment. As shown in Figure 1, the monitoring system 1 includes core switches 10A and 10B, a floor switch 20, monitoring switches 30A and 30B, and a monitoring device 100.

The core switches 10A and 10B, the floor switch 20, and the monitoring switches 30A and 30B are connected to each other by a wireless LAN (Local Area Network) or a wired LAN. Although not shown, the core switches 10A and 10B, the floor switch 20, and the monitoring switches 30A and 30B are also connected to other switches and terminal devices in the network by a wireless LAN or a wired LAN.

The core switch 10A is a network switch that forwards and relays packets within the network. For example, the core switch 10A holds a routing table, and when it receives a packet from the core switch 10B, another switch, or a terminal device, it forwards and relays the data based on the routing table. The core switch 10A also has a switching function.

The core switch 10B is a network switch that forwards and relays packets within the network. For example, the core switch 10B holds a routing table, and when it receives a packet from the core switch 10A, another switch, or a terminal device, it forwards and relays the data based on the routing table. The core switch 10B also has a switching function.

The floor switch 20 is a network switch that acts as a bridge between the central and peripheral parts of the network.

The monitoring switch 30A has an IP SLA function, creates a VLAN (Virtual Local Area Network) that reaches the floor switch 20 via the core switches 10A and 10B, and monitors the core switches 10A and 10B, the floor switch 20, and the monitoring switch 30B.

The monitoring switch 30B has an IP SLA function, creates a VLAN that reaches the floor switch 20 via the core switches 10A and 10B, and monitors the core switches 10A and 10B, the floor switch 20, and the monitoring switch 30A.

Figure 2 is a diagram for explaining the IP SLA function. As an example, the explanation will be given using a monitoring switch 30A and a core switch 10A as a monitoring target. The monitoring switch 30A sends a monitoring packet to the core switch 10A, and determines whether an alert has occurred in the core switch 10A based on the response from the core switch 10A. Although not explained below, the monitoring switch 30A and the core switch 10A exchange information related to the monitoring packet via a VLAN.

When the monitoring switch 30A sends a monitoring packet and receives a response from the core switch 10A, it determines that no alert has occurred in the core switch 10A.

On the other hand, the monitoring switch 30A sends a monitoring packet to the core switch 10A, and if it does not receive a response from the core switch 10A, it determines that an alert has occurred in the core switch 10A and sends alert information to the monitoring device 100. A protocol such as SYSLOG/SNMP trap is used to communicate the alert information.

The monitoring switch 30A also determines whether an alert has occurred for the other monitored switches, the core switch 10B, the floor switch 20, and the monitoring switch 30B, by exchanging information about monitoring packets via the VLAN, and if an alert has occurred, it sends the alert information to the monitoring device 100.

The alert information contains information about the sending monitoring switch 30A and information about the monitoring target for which the alert occurred. The monitoring switch 30A sends the alert information to the monitoring device 100 each time it detects a monitoring target for which an alert has occurred.

Similar to monitoring switch 30A, monitoring switch 30B sends monitoring packets to the monitored objects (core switches 10A and 10B, floor switch 20, and monitoring switch 30A) and determines whether an alert has occurred for the monitored object based on the response from the monitored object. If monitoring switch 30B determines that an alert has occurred for the monitored object, it sends alert information to monitoring device 100.

When the monitoring device 100 receives alert information from the monitoring switches 30A and 30B, it detects a monitored switch in which a silent failure has occurred based on the alert information. When the monitoring device 100 detects a monitored switch in which a silent failure has occurred, it blocks the monitored port by sending a message to the detected switch. For example, if the network is made redundant, it is possible to automatically detect the switch in which the silent failure has occurred and recover the network from the failure by executing this process.

Next, an example of the configuration of the monitoring device 100 will be described. FIG. 3 is a functional block diagram showing the configuration of a monitoring device according to this embodiment. As shown in FIG. 3, the monitoring device 100 has a communication unit 110, an input unit 120, a display unit 130, a memory unit 140, and a control unit 150.

The communication unit 110 transmits and receives information between the monitoring switches 30A and 30B via the network. For example, the communication unit 110 is realized by a network interface card (NIC) or the like.

The input unit 120 is an input device for inputting various types of information. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc.

The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc.

The storage unit 140 has a registration table 141, a pattern table 142, and a judgment policy table 143. The storage unit 140 is realized, for example, by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

Registration table 141 is a table that holds alert information sent from monitoring switches 30A and 30B. The alert information includes identification information (IP <Internet Protocol> address, MAC <Media Access Control> address, etc.) of the monitoring switch that is the sender of this alert information, and identification information (IP address, MAC address, etc.) of the monitored switch in which the alert occurred.

Pattern table 142 is a table that defines patterns corresponding to combinations of monitoring targets for which an alert has occurred and monitoring targets for which no alert has occurred. FIG. 4 is a diagram showing an example of the data structure of a pattern table. As shown in FIG. 4, this pattern table 142 associates the location where an alert occurred with a pattern. The location where an alert occurred indicates the switch where an alert was detected by a monitoring packet. Here, the explanation will be given using the monitoring switches (monitoring switches 30A, 30B), core switch 10A, and floor switch 20 as the location where an alert occurred.

For example, if the alert information sent from monitoring switch 30A indicates that no alert has occurred in monitoring switch 30B, and the alert information sent from monitoring switch 30B indicates that no alert has occurred in monitoring switch 30A, the monitoring switch in pattern table 142 will be judged as "○".

On the other hand, if the alert information sent from monitoring switch 30A indicates that an alert has occurred in monitoring switch 30B, or if the alert information sent from monitoring switch 30B indicates that an alert has occurred in monitoring switch 30A, the judgment of the monitoring switch in pattern table 142 will be "X".

If the alert information sent from the monitoring switch 30A indicates that no alert has occurred in the core switch 10A, and the alert information sent from the monitoring switch 30B indicates that no alert has occurred in the core switch 10A, the core switch is judged as "○" in the pattern table 142.

If the alert information sent from the monitoring switch 30A indicates that an alert has occurred in the core switch 10A, or if the alert information sent from the monitoring switch 30B indicates that an alert has occurred in the core switch 10A, the core switch is judged as "X" in the pattern table 142.

If the alert information sent from the monitoring switch 30A indicates that no alert has occurred in the floor switch 20, and the alert information sent from the monitoring switch 30B indicates that no alert has occurred in the floor switch 20, the floor switch determination in the pattern table 142 will be "○".

If the alert information sent from the monitoring switch 30A indicates that an alert has occurred in the floor switch 20, or if the alert information sent from the monitoring switch 30B indicates that an alert has occurred in the floor switch 20, the core switch in the pattern table 142 is determined to be "X".

As shown in FIG. 4, if the monitoring switch judges "○", the core switch 10A judges "○", and the floor switch 20 judges "×", the pattern is "A". If the monitoring switch judges "○", the core switch 10A judges "×", and the floor switch 20 judges "○", the pattern is "B".

If the monitoring switch judges "X", the core switch 10A judges "O", and the floor switch 20 judges "O", the pattern is "C". If the monitoring switch judges "X", the core switch 10A judges "X", and the floor switch 20 judges "O", the pattern is "D".

If the monitoring switch judges as "○", the core switch 10A judges as "×", and the floor switch 20 judges as "×", the pattern is "E". If the monitoring switch judges as "×", the core switch 10A judges as "○", and the floor switch 20 judges as "×", the pattern is "F". If the monitoring switch judges as "×", the core switch 10A judges as "×", and the floor switch 20 judges as "×", the pattern is "G".

The pattern table 142 described in FIG. 4 is a pattern table corresponding to core switch 10A, but the pattern table corresponding to core switch 10B is similar. For ease of explanation, a partial explanation will be given. If no alert has occurred in core switch 10B in the alert information sent from monitoring switch 30A, and no alert has occurred in core switch 10B in the alert information sent from monitoring switch 30B, the core switch judgment in the pattern table (pattern table corresponding to core switch 10B) will be "○".

If the alert information sent from the monitoring switch 30A indicates that an alert has occurred in the core switch 10B, or if the alert information sent from the monitoring switch 30B indicates that an alert has occurred in the core switch 10B, the core switch is judged as "X" in the pattern table (pattern table corresponding to the core switch 10B).

Then, a pattern related to the core switch 10B is identified by the combination of "○" and "×" for the monitoring switch, core switch 10A, and floor switch 20.

The judgment policy table 143 holds information for determining the cause of a silent failure according to a pattern. FIG. 5 is a diagram showing an example of the data structure of the judgment policy table. As shown in FIG. 5, this judgment policy table 143 associates patterns with causes. The patterns correspond to patterns A to G described in FIG. 4. The causes indicate the causes of silent failures. Here, an explanation will be given using a pattern related to the core switch 10A as an example.

For example, the cause of pattern A is "a failure occurs in the floor switch 20 or the core switch 10A (the routing function of the core switch 10A)." The cause of pattern B is "a failure occurs in the core switch 10A."

The cause of pattern C is "a failure occurs in core switch 10A (the switching function of core switch 10A)." The cause of pattern D is "a failure occurs in core switch 10A."

The cause of pattern E is "a failure occurs in the core switch 10A (the routing function of the core switch 10A)." The cause of pattern F is "a failure occurs in the core switch 10A (the routing function, switching function of the core switch 10A)." The cause of pattern G is "a failure occurs in the core switch 10A."

In Figure 5, the explanation was given using a pattern related to core switch 10A. Although not shown in the figure, the factors corresponding to the pattern of core switch 10B are obtained by replacing core switch 10A in the above explanation with core switch 10B.

Returning to the explanation of FIG. 3, the control unit 150 has an acquisition unit 151, a detection unit 152, and a transmission unit 153. The control unit 150 is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 150 may also be executed by an integrated circuit, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The acquisition unit 151 acquires alert information from the monitoring switches 30A and 30B. The acquisition unit 151 registers the acquired alert information in the registration table 141. The acquisition unit 151 repeats the above process each time it acquires alert information.

The detection unit 152 identifies a pattern based on a combination of alert information registered in the registration table 141 and the pattern table 142. The detection unit 152 detects a location that is the cause of a silent failure based on the identified pattern and the judgment policy table 143, and outputs the detection result to the transmission unit 153. The detection unit 152 may output the detection result to the display unit 130 for display.

For example, the detection unit 152 refers to each piece of alert information registered in the registration table 141, and executes a judgment as to whether the monitoring switch (30A, 30B), the core switch 10A, the core switch 10B, and the floor switch are "○" or "×". The process by which the detection unit 152 judges whether the switch is "○" or "×" corresponds to the method described in FIG. 4.

The detection unit 152 identifies a pattern based on a combination of the judgment results of "○" and "×" and the pattern table 142. The process by which the detection unit 152 identifies a pattern corresponds to the method described in FIG. 4. If all judgment results are "○", the detection unit 152 assumes that a silent failure has not occurred and repeats the above process until any judgment result becomes "×".

When the detection unit 152 identifies a pattern (any of patterns A to G described in FIG. 4), it detects the location that is the cause of the silent failure based on the identified pattern and the judgment policy table 143, and outputs the detection result to the transmission unit 153. In addition to the location that is the cause of the silent failure, the detection unit 152 may also output whether there is a failure in the routing function or the switching function.

The sending unit 153 sends a message to the switch that is the cause of the silent failure based on the detection result of the detection unit 152. The message is set with the identification information of the destination switch.

The switch that receives the message from the transmission unit 153 performs processing to stop communication with other switches. For example, the transmission unit 153 transmits a message to the relevant switch via the monitoring switches 30A and 30B. Execution of this processing causes a route switch by the core switches 10A and 10B.

Figure 6 is a diagram for explaining an example of path switching by message transmission. For example, a case will be explained in which the monitoring device 100 detects that a silent failure has occurred in the core switch 10A, and the transmission unit 153 transmits a message to the core switch 10A.

When the monitoring switch 30A receives a message from the transmitter 153 of the monitoring device 100, it transfers the message to the core switch 10A. When the core switch 10A receives the message, it executes a specified script and shuts down the port of the core switch 10A. When the port of the core switch 10A goes down, packets that had been routed through the core switch 10A until then are now routed through the core switch 10B, causing a switch in the path. This allows the network to be automatically restored even if a silent failure occurs in some of the core switches.

Next, an example of the processing procedure of the monitoring device 100 according to this embodiment will be described. FIG. 7 is a flowchart showing the processing procedure of the monitoring device according to this embodiment. As shown in FIG. 7, when the acquisition unit 151 of the monitoring device 100 receives alert information from the monitoring switches 30A and 30B, the acquisition unit 151 registers the alert information in the registration table 141 (step S101).

The detection unit 152 of the monitoring device 100 compares each piece of alert information in the registration table 141 with the pattern table 142 to identify a pattern (step S102). The detection unit 152 detects a switch in which a silent failure has occurred based on the pattern and the judgment policy table 143 (step S103).

The transmitter of the monitoring device 100 sends a message to the switch in which the silent failure occurred, and blocks the port of the destination switch (step S104).

The monitoring device 100 determines whether or not to continue the process (step S105). If the monitoring device 100 continues the process (step S105, Yes), the monitoring device 100 proceeds to step S101. If the monitoring device 100 does not continue the process (step S105, No), the monitoring device 100 ends the process.

Next, the effects of the monitoring device 100 according to this embodiment will be described. The monitoring device 100 acquires alert information from the monitoring switches 30A and 30B that monitor the switches to be monitored, and detects the switch in which a silent failure has occurred based on the combination of switches in which an alert has occurred. This makes it possible to efficiently detect silent failures in the switches to be monitored.

For example, the monitoring device 100 classifies the combination of switches in which an alert has occurred into one of patterns A to G, and detects the switch in which the silent failure has occurred based on the classified pattern and the judgment policy table 143. This makes it possible to pinpoint the location corresponding to the silent failure with high accuracy.

When the monitoring device 100 detects a switch in which a silent failure has occurred, it sends a message to the detected switch and blocks the switch's port. By executing such processing in a redundant network, the network can be automatically restored even if a silent failure occurs.

Next, an example of the hardware configuration of a computer that realizes the same functions as the monitoring device 100 shown in the above embodiment will be described. Figure 8 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the monitoring device of the embodiment.

As shown in FIG. 8, computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. Computer 200 also has a communication device 204 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 205. Computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk device 207. Each of devices 201 to 207 is connected to a bus 208.

The hard disk device 207 has an acquisition program 207a, a detection program 207b, and a transmission program 207c. The CPU 201 also reads out each of the programs 207a to 207c and expands them in the RAM 206.

The acquisition program 207a functions as an acquisition process 206a. The detection program 207b functions as a detection process 206b. The transmission program 207c functions as a transmission process 206c.

The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the detection process 206b corresponds to the processing of the detection unit 152. The processing of the transmission process 206c corresponds to the processing of the transmission unit 153.

Note that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

REFERENCE SIGNS LIST 100 Monitoring device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Registration table 142 Pattern table 143 Judgment policy table 150 Control unit 151 Acquisition unit 152 Detection unit 153 Transmission unit

Claims (11)

an acquisition unit that acquires, from a second monitoring switch that connects a first switch, a second switch, and a first monitoring switch to be monitored by a virtual network among a plurality of switches included in the network, a first communication status with the first switch, a second communication status with the second switch, and a third communication status with the first monitoring switch and the second monitoring switch ;
a detection unit that detects a switch in which a failure has occurred from the first switch and the second switch based on the first communication status, the second communication status, and the third communication status;
A monitoring device comprising: The monitoring device according to claim 1, further comprising a transmission unit that performs control to stop communication by sending a message to a switch in which a fault has occurred and is detected by the detection unit. The monitoring device according to claim 1 or 2, characterized in that the detection unit detects a failure in the first switch or the second switch when no alert has occurred in the first communication state and the third communication state, and an alert has occurred only in the second communication state. The monitoring device according to claim 1, 2 or 3, characterized in that the detection unit detects a failure in the first switch when no alert has occurred in the second communication state and the third communication state, and an alert has occurred only in the first communication state. The monitoring device according to any one of claims 1 to 4, characterized in that the detection unit detects a failure in the first switch when no alert has occurred in the first communication state and the second communication state, and an alert has occurred only in the third communication state. The monitoring device according to any one of claims 1 to 5, characterized in that the detection unit detects a failure in the first switch when no alert has occurred in the second communication state and an alert has occurred only in the first communication state and the third communication state. The monitoring device according to any one of claims 1 to 6, characterized in that the detection unit detects a failure in the first switch when no alert has occurred in the third communication state and an alert has occurred only in the first communication state and the second communication state. The monitoring device according to any one of claims 1 to 7, characterized in that the detection unit detects an abnormality in the first switch when no alert has occurred in the first communication state and only alerts have occurred in the second communication state and the third communication state. The monitoring device according to any one of claims 1 to 8, characterized in that the detection unit detects a failure in the first switch when an alert occurs in the first communication status, the second communication status, and the third communication status. 1. A computer-implemented method for fault detection, comprising:
obtain, from a second monitoring switch that connects a first switch, a second switch, and a first monitoring switch to be monitored by a virtual network among a plurality of switches included in the network, a first communication status with the first switch, a second communication status with the second switch, and a third communication status with the first monitoring switch and the second monitoring switch ;
a process of detecting a switch in which a fault has occurred from the first switch and the second switch based on the first communication status, the second communication status, and the third communication status. On the computer,
obtain, from a second monitoring switch that connects a first switch, a second switch, and a first monitoring switch to be monitored by a virtual network among a plurality of switches included in the network, a first communication status with the first switch, a second communication status with the second switch, and a third communication status with the first monitoring switch and the second monitoring switch ;
a fault detection program that executes a process of detecting a switch in which a fault has occurred from the first switch and the second switch based on the first communication status, the second communication status, and the third communication status.

Images