TW202026879A - Method for remotely clearing abnormal status of racks applied in data center - Google Patents

Abstract

A method for remotely clearing abnormal status of racks is disclosed and includes following steps: obtaining each information of a rack management controller (RMC) and multiple baseboard management controllers (BMCs) of a rack regularly by a management system; recording each operating action performed by manager through the management system; analyzing the information and the operating action by the management system for determining whether any RMC or BMC is under one of multiple default attention-statuses; and, automatically performing a remotely rescuing procedure to one of the RMC and the BMCs for eliminating an abnormal connection status from the RMC or the BMC when the RMC or the BMC is determined losing a connection with the management system.

Description

Remote troubleshooting method for abnormal state of cabinets used in data center (3)

The present invention relates to a data center, and in particular to a method for analyzing and eliminating abnormal conditions of cabinets in the data center.

Generally speaking, a data center usually uses the Intelligent Platform Management Interface (IPMI) to control the rack management controller (RMC) and baseboard of the racks, endpoint servers and other equipment in the data center. The management controller (Baseboard Management Controller, BMC) performs remote management.

No matter what method is used for remote management, as long as the RMC or BMC of any cabinet or endpoint server is abnormal, the administrator will receive many warning letters. However, it is generally difficult for managers to know the real problems of the state in the first time through these warning letters. They often need to keep advancing over time until they receive hundreds of warning letters and lose the connection with the device before they can be sure. The RMC and BMC are abnormal.

What's more, even if part of the management platform collects error messages from different monitoring channels and submits the fault assessment report to the manager after compiling, such a monitoring method still requires the manager to make the final judgment and decide the processing method. . However, as long as human factors are involved, the possibility of misjudgment cannot be completely avoided.

In view of this, the field does need to develop a set of novel systems and methods that can automatically implement a remote repair mechanism for abnormal RMC and BMC, thereby strengthening the monitoring capabilities of the data center, making cabinet management highly automated, and at the same time Reduce the time lost indirectly due to human judgment and avoid human misjudgment.

The main purpose of the present invention is to provide a remote elimination method for the abnormal state of the cabinet applied to the data center, which can directly eliminate the substrate remotely when it is determined that the baseboard management controller has lost the connection with the cabinet server management system The abnormal state of the management controller.

In order to achieve the above objectives, the method of the present invention is that a rack server management system periodically obtains information about a rack management controller and a plurality of baseboard management controllers in a rack at a remote location, and records the information passed by a manager Various operations performed by the rack server management system. The rack server management system analyzes the above-mentioned information and operation behaviors to determine whether the rack management controller or each baseboard management controller in the rack is in one of the preset attention states.

If it is determined that any of the baseboard management controllers has lost the connection with the rack server management system, the rack server management system automatically implements a remote rescue mechanism to eliminate the abnormal connection of the baseboard management controller status.

Compared with related technologies, the method of the present invention analyzes and automatically implements the remote rescue mechanism by the cabinet server management system connected to the cabinet, without waiting for the manager's artificial judgment on the abnormal state, which can greatly reduce the management cost, and also make The monitoring of the cabinet does not require human intervention, nor is it affected by distance and time.

With regard to a preferred embodiment of the present invention, detailed description is given below in conjunction with the drawings.

The present invention discloses a remote elimination method for cabinet abnormal state (hereinafter referred to as the elimination method in the specification). The elimination method is mainly used in the data center to assist managers in automatically monitoring, analyzing and eliminating the data center. Abnormal state.

Refer to Figure 1, which is a schematic diagram of the data center of the present invention. As shown in Fig. 1, the data center 1 of the present invention mainly has a plurality of cabinets 2, and a cabinet server management system 3 (hereinafter referred to as the management system 3) connected to the plurality of cabinets 2 by a remote end. The management system 3 can be installed inside or outside the data center 1 and connected to a public network switch 4 via a network, and then connected to a plurality of cabinets 2 in the data center 1 via the public network switch 4.

The management system 3 of the present invention can monitor the multiple cabinets 2 in the data center 1 in real time, obtain various information of the multiple cabinets 2, and analyze the information. When an abnormal state or an abnormal state is about to occur in any cabinet 2, the management system 3 of the present invention can automatically implement a corresponding processing mechanism to eliminate the situation. Thereby, the present invention can eliminate the abnormal state that has occurred in the cabinet 2 or prevent the abnormal state that may be about to occur on the premise that no human intervention is required, human misjudgment is greatly reduced, and the processing speed is increased.

In one embodiment, the management system 3 can be a personal computer or a cloud server, with one or more central processing units (not shown). After the management system 3 is activated, it can be connected to the multiple cabinets 2 in the data center 1 through the public network switch 4, and can be implemented by one or more central processing units to execute specific applications and algorithms. Cabinet 2 monitoring, data analysis, and abnormal state elimination.

The management system 3 also has a database 31 for temporarily storing or permanently storing various information obtained from the plurality of cabinets 2 in the data center 1. In the embodiment of FIG. 1, the database 31 is built in the management system 3. In other embodiments, the management system 3 can also connect to one or more databases 31 from the outside, without limitation.

Refer to Figure 2, which is a first specific embodiment of the block diagram of the cabinet of the present invention. In the embodiment of FIG. 2, a single cabinet 2 in the data center 1 is connected to the management system 3 as an example for description. However, the data center 1 can be equipped with multiple cabinets 2 according to actual needs. Those shown in 2 are limited.

As shown in Figure 2, the cabinet 2 of the present invention mainly includes at least one Rack Management Controller (Rack Management Controller, RMC) 21, and multiple endpoint servers 220 connected to the RMC 21, wherein each endpoint server 220 has At least one baseboard management controller (Baseboard Management Controller, BMC) 22.

The RMC21 is an embedded system, which is set in the cabinet 2 and assists in processing all external hardware equipment (cooling fans, various sensors or power supplies, etc.) of the cabinet 2 through various hardware circuits. And communicate with the BMC 22 of all the endpoint servers 220 in the cabinet 2. The BMC 22 is also an embedded system, which is set in the endpoint server 220 and assists in processing all external communications of the internal hardware devices (various sensors, etc.) of the endpoint server 220.

In this embodiment, the RMC 21 is connected to the BMC 22 of all the endpoint servers 220 in the cabinet 2 through the internal hardware circuit 24, and communicates with each BMC 22 to control each endpoint server 220 and obtain required information. In this embodiment, the endpoint server may be, for example, a tower model server (Tower Model Server) or a blade server (Blade Server), but it is not limited.

As shown in Figure 2, each endpoint server 220 installed in the cabinet 2 has a fixed location number (#1, #2, #n, etc. in Figure 2). When the endpoint server 220 or When the external network function of BMC22 fails, RMC21 can be connected to the designated position in cabinet 2 through internal hardware line 24 (such as #1, #2, #n above), and then serve with the endpoint in the designated position The device 220 communicates with the BMC22. In this way, even if the endpoint server 220 or the BMC 22 loses network connection, the cabinet 2 can still use the RMC 21 to monitor, manage each BMC 22 and eliminate the abnormal condition of each BMC 22.

In addition, the RMC 21 of the present invention is provided with a network interface controller (NIC) 211, and each BMC 22 is also provided with a network interface controller 221. The RMC 21 is connected to the internal network switch 23 inside the cabinet 2 through the NIC 211, and each BMC 22 is connected to the internal network switch 23 through its respective NIC 221. The cabinet 2 is connected to a public network switch 4 through an internal network switch 23, and a network connection is established with the management system 3 through the public network switch 4. In this way, the management system 3 can remotely access the cabinet 2 in the data center 1 via the network, thereby querying and obtaining various information of all RMC 21 and BMC 22 in the cabinet 2 and storing the information in the database 31.

The main technical feature of the present invention is that the management system 3 can periodically access the cabinet 2 via the network, and obtain various information of all RMC21 and BMC22 in the cabinet 2 (such as status data, event log, system resource utilization rate, The sensing values of the internal sensors of the endpoint server 220, etc.), use these information to actively analyze whether the RMC21 and BMC22 have abnormal states or are about to occur. When the management system 3 determines that it is necessary after analysis, it can actively implement corresponding mechanisms at the remote end to directly eliminate the abnormal state of RMC21 and/or BMC22 at the remote end, or prevent RMC21 and/or BMC22 from entering the Abnormal state.

The technical solution of the present invention can handle abnormal conditions without human intervention, greatly reduces the possibility of human misjudgment, and can make the monitoring of the cabinet 2 highly automated.

Please continue to refer to FIG. 3A, which is a first specific embodiment of the data collection flowchart of the present invention.

As shown in FIG. 3A, if the manager wants to monitor the cabinet 2 in the data center 1, the manager can directly activate the remote management system 3 (step S11). When the management system 3 is activated, it will actively remotely access the RMC 21 and all the BMC 22 in the cabinet 2 in the data center 1 (take the single cabinet 2 in FIG. 2 as an example) (step S12). In addition, the management system 3 obtains various information of the RMC 21 and all the BMC 22 in the cabinet 2 through remote access (step S13), and then stores the obtained information in the local data 31 (step S14).

Specifically, in this embodiment, the management system 3 actively accesses the cabinet 2 periodically after activation, that is, the access actions, information acquisition actions, and storage actions of steps S12, S13, and S14 are regarded as routines after activation. ). When the above routine is executed, it is continuously determined whether the management system 3 is shut down (step S15), and the above steps S12 to S14 are continuously executed before the management system 3 is shut down to continuously monitor the RMC21 and BMC22 in the cabinet 2.

Refer to FIG. 3B, which is a second specific embodiment of the data collection flowchart of the present invention.

In this embodiment, when the manager starts the management system 3 (step S21), the management system 3 can provide an operation interface (step S22). Through this operation interface, the administrator can log in to the management system 3, and through the management system 3 to remotely monitor and control the information of each cabinet 2 in the data center 1. In this embodiment, the operating interface can be a physical interface or a web interface, and is not limited.

After the operation interface is provided, the management system 3 continuously judges whether the operation performed by the manager has been accepted through the operation interface (step S23). If the manager's operation is indeed accepted, the management system 3 implements corresponding remote management of the cabinet 2 and the RMC 21 and BMC 22 in the cabinet 2 from the remote end according to the manager's operation behavior (step S24). Then, the management system 3 can record the above-mentioned operation behavior of the administrator (step S25), and can also obtain and record the management system 3, the cabinet 2, each endpoint server 220, and RMC21, BMC22 generated by the remote management Feedback information such as the feedback, system parameters, and execution data (step S26). Finally, the management system 3 also stores the operation behavior and feedback information in the database 31 (step S27), so as to facilitate the subsequent analysis of the abnormal state.

Similarly, the management system 3 of this embodiment regards the actions from step S22 to step S27 as a routine after activation. When the above routine is executed, it is continuously judged whether the management system 3 is shut down (step S28), and the above steps S22 to S27 are continuously executed before the management system 3 shuts down, so as to continuously monitor and analyze the operation behavior implemented by the manager on the cabinet 2 The impact of RMC21 and BMC22.

Please continue to refer to FIG. 4, which is a first specific embodiment of the analysis and elimination flowchart of the present invention.

As shown in FIG. 4, the management system 3 in this embodiment will periodically access the database 31 (step S31), and obtain various information of RMC21 and BMC22 in the cabinet 2 from the database 31, the operation behavior of the manager, and Various feedback information (step S32), and analyze it. Based on the above data, the management system 3 can analyze whether the RMC 21 and each BMC 22 in the cabinet 2 are in one of the preset multiple attention states (step S33).

In one embodiment, the management system 3 can obtain various information of the RMC21 and BMC22 in the cabinet 2 in real time, and obtain the operation behavior of the manager from the operation interface in real time, and analyze it accordingly. In another embodiment, the management system 3 can periodically store the above-mentioned data in the database 31 through step S14 of FIG. 3A and step S27 of FIG. 3B, and periodically read the above-mentioned data from the database 31 for analysis. Not limited.

In one embodiment, the various information of RMC21 and BMC22 mentioned above can be, for example, status data (such as current working mode or update mode, IP address, MAC address, subnet mask, gateway IP address) , IPMI session number, etc.), event log, etc., and the above operation behavior can be, for example, the data query operation, update operation, and reset performed by the administrator for a specific cabinet 2, endpoint server 220 or RMC21, BMC22 Homework, etc., but not limited. Based on the above data, the management system 3 can analyze whether there is an RMC21 or BMC22 in the cabinet 2 that needs immediate rescue by executing the corresponding algorithm.

In the embodiment of FIG. 4, the management system 3 can preset at least three types of attention states, including the first type of attention state, the second type of attention state, and the third type of attention state. The three types of attention states are respectively Corresponding to different abnormal conditions of RMC21/BMC22, and respectively need to be eliminated or prevented by the management system 3 directly implementing different mechanisms at the remote end.

As shown in Figure 4, if the management system 3 analyzes based on the above data (mainly based on the status data, event logs and manager’s operation behavior) and finds that any RMC21 or BMC22 has been in an abnormal state, but has not lost contact with the management system 3. Connected, it will be determined that this RMC21 or BMC22 is in the first type of attention state (step S34). When any RMC21 or BMC22 is found to be in the first type of attention state, the management system 3 can automatically implement a remote recovery mechanism for the RMC21 and BMC22 in the first type of attention state to remotely release the abnormal state of RMC21 or BMC22 (step S37) .

If the management system 3 analyzes based on the above data (mainly based on the status data of RMC21 and BMC22) and finds that any RMC21 or BMC22 is connected to the management system 3 normally, but it is judged that an abnormal state may be about to occur, the RMC21 or BMC22 will be recognized It is in the second type of attention state (step S35). When any RMC21 or BMC22 is found to be in the second type of attention state, the management system 3 can automatically implement a remote service restart mechanism for the RMC21 and BMC22 in the second type of attention state to remotely prevent RMC21 or BMC22 from entering a possible abnormal state ( Step S38).

If the management system 3 analyzes the above data (mainly based on the status data, the manager’s operation behavior and various feedback information) and finds that any BMC22 has lost its network connection (that is, the management system 3 cannot directly access this remotely). BMC22), it will be determined that this BMC22 is in the third type of attention state (step S36). When it is found that any BMC22 is in the third type of attention state, the management system 3 can automatically implement a remote rescue mechanism for the BMC22 in the third type of attention state, so as to remotely eliminate the state of BMC22 losing connection, and make the BMC22 network connection The line returns to normal (step S39).

The following paragraphs discuss the first type of attention state.

Because some RMC21/BMC22 do not have a basic input/output system (Basic Input/Output System, BIOS), they need to use the Network Time Protocol (NTP) service provided by an external server, or a hardware clock chip The Real-time Clock (RTC) service is provided to set the time to synchronize with other devices.

As mentioned above, if a system event occurs before the time synchronization program of RMC21 or BMC22 is completed, although the system event will still be recorded in the event log of RMC21, BMC22, the time field of the system event will not be recorded The correct time of occurrence of the event, and only words such as "Pre-init" will be recorded. If there is no correct event occurrence time, the administrator cannot use the event log as a reference indicator of the system event, which will lead to judgment errors. In addition, if the RMC21 and BMC22 need to perform a Reset operation, it may also cause an error or abnormality in the event occurrence time recording of the above system event.

Refer to FIG. 5, which is a first specific embodiment of the first type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 will periodically access the database 31 (step S41) to obtain the status data and event logs of the RMC21 and BMC22 in the cabinet 2 from the database 31, and determine the status of the RMC21 and BMC22 Change (step S42).

In this embodiment, the management system 3 mainly judges whether there is any system event in the obtained event log with an unknown or wrong event occurrence time (step S43). If all system events in the event log record the correct event occurrence time, the management system 3 does not take the initiative to implement any actions.

If after analysis, the management system 3 finds that any RMC21 or BMC22 has an unknown or wrong system event, the management system 3 will regard the RMC21 or BMC22 as being in the first type of attention state (step S44), that is, It is determined that the RMC21 or BMC22 is in an abnormal state, but the network connection has not been lost.

In one embodiment, the management system 3 can mainly be used when the event occurrence time of any system event in the event log is recorded as "Pre-init" or similar (ie, the occurrence time of the system event cannot be correctly stated) , The event occurrence time of the system event is unknown or wrong. In another embodiment, the management system 3 can mainly find from the event log that any RMC21 or BMC22 has a system event with an unknown time of the event, and find from the status data that this RMC21 or BMC22 has not completed the time synchronization process or is required When the reset operation is performed, it is determined that the event occurrence time of the system event is unknown or wrong.

When the management system 3 determines in step S44 that an RMC21 or BMC22 is in the first type of attention state, the management system 3 first obtains the time stamp of this access event log (step S45), and uses this time stamp as the system event The identification information of the preparation time is stored in the database 31 (step S46). In one embodiment, the management system 3 uses the time when the database 31 is accessed this time to read the event log as the time stamp. In another embodiment, the management system 3 uses the time when the cabinet 2 is remotely accessed this time and the event log is obtained from the RMC 21 and the BMC 22 as the above-mentioned time stamp, but is not limited.

For example, the original content of the event log may be as shown in the following table:

If the management system 3 accesses the event log at 11:32:23 PM on December 22, 2018, and finds that the time of the event 2 is wrong, the management system 3 can take the initiative to generate the event 2 State the identification information of the standby time, and modify the content of the event log or generate a new event log. The new event log can be as shown in the table below:

When the manager logs into the management system 3 through the operation interface and queries the event log in the management system 3, the management system 3 can display the identification information of the preparation time as shown in the above table as the event two The time the incident occurred. In this way, even if a system event occurs in RMC21 or BMC22 before the time synchronization is completed, the management system 3 can still set a recognizable standby time for the system event, so that the management system 3 and the administrator can use the system Interpretation of the event, and to strengthen the effect of remote recovery.

After step S46, the management system 3 may further send a control command (such as a first control command) via the network to the RMC21 or BMC22 in the first type of attention state, so as to execute the time correction procedure for the RMC21 or BMC22 in the abnormal state with a time error (Step S47). In one embodiment, the time correction procedure is to control the RMC21 or BMC22 to perform time correction through the NTP service. In another embodiment, the time correction procedure is to force the RMC21 or BMC22 to perform a reset operation, but is not limited.

The following paragraphs continue to explain other possible first-type concerns.

Due to the large number of cabinets 2 in the data center 1, it is difficult to manually update them one by one when the manager has a need for updating. Therefore, when the administrator wants to update the RMC21 and BMC22 in the cabinet 2 (such as firmware update), the management system 3 can be operated to send the update command and the latest version through the relevant code of the management system 3. The firmware of, to update the RMC21 and BMC22 of multiple cabinets 2 in the data center 1 at the same time remotely.

If during the update process, network congestion or unstable network signal causes network connection interruption, etc., some RMC21 and BMC22 cannot follow the normal update process to complete the update operation, which may cause the update operation to fail. However, some RMC21 and BMC22 will only cause the system to fail to operate normally after the update operation fails, but the network connection is not lost (for example, the update mode cannot be restored to the working mode). At this time, the management system 3 Intervene to eliminate abnormal conditions.

Refer to FIG. 6, which is a second specific embodiment of the first type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 also regularly accesses the database 31 (step S51) to obtain the status data and event logs of the RMC21 and BMC22 in the cabinet 2 from the database 31, and obtain the information implemented by the manager through the operation interface. And determine the status change of RMC21 and BMC22 (step S52).

In this embodiment, the management system 3 can first analyze the status data and event logs of RMC21 and BMC22 to determine whether any update operation of RMC21 or BMC22 has timed out or an error has occurred (step S54), and determines the Whether the network connection of the RMC21 or BMC22 where the update operation timed out or the error occurred is normal (step S55). If the management system 3 finds after analysis that any RMC21 or BMC22 update operation is timed out or has an error but the network connection is still normal, the RMC21 or BMC22 can be regarded as being in the first type of concern state (step S56 ), that is, in an abnormal state, but the connection has not been lost.

More specifically, after the above step S52, the management system 3 can first determine whether the administrator has performed an update operation on the RMC21 and/or BMC22 in the cabinet 2 according to the operation behavior (step S53). Moreover, after confirming that the administrator has performed the update operation, the management system 3 continues to perform step S54 and step S55 to determine whether the update operations of the RMC21 and BMC22 are timed out or have errors, and whether the network connection is normal.

The RMC21 and BMC22 will automatically enter the update mode after accepting the update operation implemented by the administrator. At this time, RMC21 and BMC22 will set a flag (flag) that has entered the update mode in the status data. When the peripheral device communicates with RMC21, BMC22 and reads the mark of the update mode, it will automatically stop the interaction with this RMC21, BMC22. Therefore, as long as the update operation of RMC21 and BMC22 fails and cannot leave the update mode, the RMC21 and BMC22 cannot operate normally. When the management system 3 finds that any RMC21, BMC22 has accepted the update operation, the update operation has timed out or an error has occurred, but the network connection has not been lost, it can determine that this RMC21, BMC22 is in the first attention state.

After step S56, the management system 3 may further send a control command (such as a second control command) to the RMC21 or BMC22 in the first type of attention state through the network, to force the RMC21 or BMC22 that failed the update operation to leave the update mode (step S57).

As mentioned above, when the update operation referred to in this embodiment fails (that is, cannot leave the update mode), the RMC21 and BMC22 can still receive and process related instructions, but the peripheral device is reading the flag of the update mode (flag ) Will automatically stop interacting with RMC21 and BMC22. In this embodiment, the management system 3 has determined that the RMC21 and BMC22 have abnormal states, and therefore ignores the above flags, and forces the RMC21 and BMC22 to leave the update mode by issuing control commands.

After step S57, the management system 3 can further send another control command (for example, a third control command) to the RMC21 or BMC22 that has left the update mode through the network, so as to force the RMC21 or BMC22 to perform the reset operation, or to perform the reset operation again. The update operation is described (step S58). In this way, the management system 3 can ensure that the RMC21 and BMC22 have resumed normal operation, and the firmware or software is in the latest version after the update.

The following paragraphs then discuss the second type of attention state.

The RMC21 and BMC22 in the present invention are an embedded system (Embbeded System). Therefore, even if the endpoint server 220 in the cabinet 2 is not powered on, the management system 3 can still realize remote booting and remote booting by communicating with RMC21 and BMC22. Remote management functions such as shutdown and viewing device status.

Generally speaking, when a manager implements remote management procedures, he can use the Intelligent Platform Management Interface (IPMI) tool program on the management system 3 to send IPMI commands through the network to communicate with the RMC21 in the cabinet 2. , BMC22 communication. In the case of using the IPMI tool program, each command transmission needs to establish an IPMI session with the destination RMC21 and BMC22, so as to communicate with the destination RMC21 and BMC22. Specifically, after the establishment of the IPMI session is completed, the management system 3 can communicate with the underlying hardware devices of the RMC21, BMC22, and the cabinet 2, the endpoint server 220 through the network, and then obtain the execution result of the command (for example, obtain the firmware Version, sensing values of all sensors in the endpoint server 220, etc.).

However, the computing resources of the embedded system itself are quite limited. In addition to the basic resource consumption required for operation, communication with RMC21, communication with BMC22, and replying to various monitoring systems in data center 1 will all be further consumed. Computing resources for embedded systems.

Furthermore, when the administrator implements the remote management program for each RMC21 and BMC22 through the management system 3, it also consumes the computing resources of RMC21 and BMC22. The most obvious is to increase the number of IPMI sessions of RMC21 and BMC22 significantly, making RMC21, BMC22 appears to respond late or request timeout (timeout). At this time, although the abnormal state of the RMC21 and BMC22 has not yet occurred, the management system 3 may need to be remotely intervened to prevent the abnormal state of the RMC21 and BMC22 from affecting the operation of the cabinet 2 in the future.

Refer to FIG. 7, which is a first specific embodiment of the second type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 will also periodically access the database 31 (step S61) to obtain the status data of the RMC21 and BMC22 in the cabinet 2 from the database 31, and determine the status change of the RMC21 and BMC22 ( Step S62). In one embodiment, the management system 3 mainly obtains the total number of IPMI sessions of the RMC 21 and each BMC 22 in step S62. In another embodiment, the management system 3 simultaneously obtains the current system resource utilization rates of the RMC 21 and each BMC 22 in step S62.

After step S63, the management system 3 determines whether the total number of IPMI sessions of any RMC21, BMC22 is higher than the first threshold (step S63), and when the total number of IPMI sessions of any RMC21, BMC22 is higher than the first threshold, it is determined The RMC21 and BMC22 are in the second attention state (step S65), that is, the connection of the RMC21 or BMC22 is normal, but it is judged that an abnormal state is about to occur.

It is worth mentioning that if the management system 3 obtains the system resource utilization rate of RMC21 and each BMC22 at the same time in step S62, the management system 3 can simultaneously determine whether any RMC21, BMC22 system resource utilization rate is higher than the second Threshold value (step S64). In this situation, the management system 3 will determine that the current total number of IPMI sessions is higher than the first threshold, and the RMC21 or BMC22 whose system resource usage rate is higher than the second threshold is in the second attention state.

In one embodiment, the system resource utilization rate is the utilization rate of the central processing unit or memory of the RMC21 and BMC22. In another embodiment, the system resource utilization rate is the percentage of system resources used by RMC21 and BMC22 to provide various services (such as HyperText Transfer Protocol (HTTP) service or IPMI service, etc.) Usage rate, but not limited.

When the management system 3 determines that an RMC21 or BMC22 is in the second type of attention state, the management system 3 can further send a control command (such as a fourth control command) through the network to the RMC21 or BMC22 in the second type of attention state to make all The RMC21 or BMC22 restarts the IPMI service (step S66). In this way, RMC21 and BMC22 can clear the current accumulated IPMI session to avoid abnormal status.

In one embodiment, the fourth control command is a reset command, and the management system 3 sends a reset command to the RMC21 or BMC22 in the second type of attention state through the network to force the RMC21 or BMC22 to perform the reset operation. In this way, the reset RMC21 and BMC22 can directly restart the IPMI service. However, the above is only one specific embodiment of the present invention, but it is not limited to the above.

Through the above technical solutions, the management system 3 can find early through analysis that an abnormal state of the RMC21 or BMC22 may be about to occur, and therefore can actively implement a service restart mechanism remotely to prevent the abnormal state of the RMC21 or BMC22 from actually occurring and affecting the operation of the cabinet 2.

The following paragraphs then discuss the third type of attention state.

As mentioned above, the management system 3 of the present invention mainly communicates with the RMC21 and BMC22 in the cabinet 2 in the data center 1 through the network, and the administrator also implements remote management procedures for these RMC21 and BMC22 through the network. Therefore, when the BMC22 in the cabinet 2 loses network connection, the management system 3 will not be able to communicate with the BMC22, and the administrator will not be able to manage the BMC22. In this embodiment, the abnormal condition that the BMC 22 loses network connection may be caused by an incorrect IP address setting.

Generally speaking, the BMC22 in the cabinet 2 may be set to use a dynamic IP address (that is, the network mode of BMC22 is set to dynamic IP mode) or a static IP address (that is, the network mode of BMC22 is set to static IP mode). If the network mode of BMC22 is dynamic IP mode, the Dynamic Host Configuration Protocol (DHCP) server (not shown in the figure) in Data Center 1 can actively allocate a dynamic IP address to BMC22. . If the network mode of the BMC22 is a static IP mode, the administrator can set a static IP address for the BMC22 through the operation interface of the management system 3.

To implement network configuration operations on BMC22 to set a set of available static IP addresses, the administrator needs to issue at least four commands to BMC22 via management system 3 (ie, four IPMI sessions need to be established), including: (1) Setting The network mode of BMC22 is static IP mode; (2) Set static IP address; (3) Set subnet mask (netmask); (4) Set gateway (Gateway) IP address.

As mentioned above, if the static IP address set by the administrator is wrong (for example, it is repeated with one of the multiple dynamic IP addresses allocated by the DHCP server), or the gateway IP address is set incorrectly, then In an environment where multiple sub-domains coexist, or an environment where communication is required through a gateway, the BMC 22 cannot connect to the management system 3. For the management system 3, although the endpoint server 220 to which this BMC22 belongs still exists, because the management system 3 has lost the connection with the BMC22, it will not be able to communicate with the BMC22 (and the endpoint server to which it belongs). 220) Manage. At this time, the management system 3 may need to intervene remotely to enable the BMC 22 to restore the network connection.

Refer to FIG. 8, which is a first specific embodiment of the third type of attention state elimination flowchart of the present invention. In this embodiment, the management system 3 will periodically access the database 31 (step S71) to obtain the status data of each BMC 22 in the cabinet 2 from the database 31, the operation behaviors implemented by the manager through the management system 3, And the management system 3 obtains various feedback information based on the operation behavior, and judges the status change of the BMC 22 (step S72).

In one embodiment, the status data obtained by the management system 3 in step S72 includes at least the network mode (static IP mode or dynamic IP mode) of each BMC 22, the currently used static IP address, subnet mask, and gate. The IP address of the router is not limited. In addition, the feedback information obtained by the management system 3 in step S72 mainly includes the feedback generated by the management system 3, the cabinet 2 and each endpoint server 220 (and each BMC 22) based on this operation behavior when the operation behavior is implemented. Data such as parameters and execution data, but not limited.

After step S72, the management system 3 first judges whether any BMC 22 in the cabinet 2 has lost the connection with the management system 3 based on the status data and feedback information (step S73), and judges the manager based on the operation behavior Whether the network setting operation has just been implemented for any BMC 22 in the cabinet 2 (step S74). If it is found after analysis that the administrator has just implemented a network configuration operation on a certain BMC22, and the BMC22 loses connection after the network configuration operation, the management system 3 can regard the BMC22 as being in the third category Attention state (step S75), that is, BMC 22 has lost connection.

It is worth mentioning that in the aforementioned step S73, the management system 3 can be set to the static IP mode in any BMC22 network mode, and the static IP address of the BMC22 and the multiple groups allocated by the DHCP server When one of the dynamic IP addresses is repeated, it is judged that the BMC22 loses network connection (the connection has been lost, or the connection may be lost).

In addition, in the aforementioned step S73, the management system 3 can also set the network mode of any BMC22 to static IP mode, and when the IP address of the gateway of this BMC22 is set incorrectly, it is judged that the BMC22 loses network connection (The connection has been lost, or the connection may be lost). However, the foregoing are only part of the specific implementation examples of the present invention, but should not be limited to the foregoing.

After step S75, the management system 3 can determine that a certain BMC22 is in the third type of attention state. Then, the management system 3 determines which RMC21 is mainly responsible for this BMC22 in the data center 1 (step S76), and controls this RMC21 The endpoint server 220 to which the BMC 22 belongs is checked through the internal hardware circuit 24 of the cabinet 2 (step S77) to confirm whether the endpoint server 220 exists (step S78).

As shown in Figure 2, the RMC21 in a cabinet 2 can mainly physically connect the BMC22 in all the endpoint servers 220 in the cabinet 2 through the internal hardware line 24. Therefore, even if the BMC22 loses network connection, the same cabinet 2 The internal RMC21 can still communicate with the BMC22 through the internal hardware circuit 24.

If it is determined in the above step S78 that the endpoint server 220 does not exist (for example, it has been removed from the cabinet 2 or has been damaged), the management system 3 correspondingly sends a warning signal (step S79). In one embodiment, the management system 3 can issue a warning signal (such as text, light, or sound) through the operating interface to alert the manager. In another embodiment, the management system 3 can send a warning signal (such as a text message, email or communication software) to the manager via the network to achieve the warning effect.

If it is determined in the above step S78 that the endpoint server 220 still exists, the management system 3 controls the RMC 21 to send a set of IPMI commands to the BMC 22 through the internal hardware circuit 24, so that the BMC 22 can restore the network connection ( Step S80). In one embodiment, the management system 3 can send an IPMI command to the BMC22 through the RMC21 to reset the static IP address of the BMC22 or reset the gateway IP address of the BMC22, thereby Make BMC22 restore the connection with the management system 3.

Through the above technical solution, the management system 3 can actively implement a rescue mechanism to the BMC 22 remotely after the BMC 22 loses the connection, so that the BMC 22 can restore the network connection.

In the method of the present invention, the management system 3 can automatically collect the required information and analyze the status of all RMC21 and BMC22. At the same time, when any RMC21 and BMC22 are in one of multiple attention states, the corresponding mechanism can be automatically implemented to eliminate abnormal conditions. In this way, the technical solution of the present invention can greatly reduce the management cost, and the monitoring of the data center 1 does not require human intervention, nor is it affected by distance and time.

The above are only preferred specific examples of the present invention, and are not limited to the scope of the patent of the present invention. Therefore, all equivalent changes made by using the content of the present invention are included in the scope of the present invention in the same way, and they are all included in the present invention. Bright.

1: Data Center

2: Cabinet

21: Cabinet Management Controller

211, 221: network interface controller

22: baseboard management controller

23: Internal network switch

24: Internal hardware circuit

3: Cabinet server management system

31: Database

4: Public network switch

S11~S15, S21~S28: Collection steps

S31~S39: Analysis and elimination steps

S41~S47, S51~S58, S61~S66, S71~S80: Elimination steps

Figure 1 is a schematic diagram of the data center of the present invention.

Fig. 2 is a first specific embodiment of the block diagram of the cabinet of the present invention.

Fig. 3A is a first specific embodiment of the data collection flowchart of the present invention.

Fig. 3B is a second specific embodiment of the data collection flowchart of the present invention.

Fig. 4 is a first specific embodiment of the analysis and elimination flowchart of the present invention.

FIG. 5 is a first specific embodiment of the first type of attention state elimination flowchart of the present invention.

Fig. 6 is a second specific embodiment of the first type of attention state elimination flowchart of the present invention.

FIG. 7 is a first specific embodiment of the second type of attention state elimination flowchart of the present invention.

FIG. 8 is a first specific embodiment of the third type of attention state elimination flowchart of the present invention.

S31~S39: Analysis and elimination steps

Claims (10)

A remote troubleshooting method for cabinet abnormal state is applied to a data center having a cabinet and a cabinet server management system connected to the cabinet by a remote end, wherein the cabinet has a Rack Management Controller (RMC) ) And a plurality of endpoint servers, each of which has a baseboard management controller (BMC), and the remote elimination method includes: 　　a) The rack server management system regularly accesses a database In order to obtain the status data of each BMC, the operation behavior implemented by a manager through the rack server management system on the rack, and the feedback information obtained corresponding to the operation behavior; 　　b) According to the status data, the operation behavior and the The feedback information determines whether one of the BMCs is in one of the preset multiple attention states; and 　　 c) When determining that any BMC is in a third type of attention state among the multiple attention states, the rack server The management system automatically implements a remote rescue mechanism for the BMC in the third type of attention state to eliminate the abnormal state of the BMC losing network connection, where the third type of attention state means that the BMC has lost the server server with the cabinet. It manages the connection between the systems. The remote troubleshooting method for the abnormal state of the cabinet as described in claim 1, which further includes the following steps: 　　a01) the cabinet server management system is started; 　　a02) after this step a01), the cabinet server management system periodically and actively remote access The RMC and each BMC in the cabinet; 　　a03) Obtain the status data of the RMC and each BMC; 　　a04) Save the status data to the database; and 　　a05) Continue until the cabinet server management system is shut down Perform this step a02) to this step a04). For example, the remote troubleshooting method for the abnormal state of the cabinet as described in claim 1, which further includes the following steps: 　　a11) the cabinet server management system is activated; 　　a12) after the step a11), the cabinet server management system provides an operating interface　　A13) When accepting the operation behavior of the manager through the operation interface, implement a remote management program on the RMC and each BMC according to the content of the operation behavior; 　　a14) Obtain the feedback corresponding to the remote management program Information; 　　a15) store the operation behavior and the feedback information in the database; and 　　a16) continue to execute the step a12) to the step a15) before the cabinet server management system is closed. The method for remotely eliminating the abnormal state of the cabinet according to claim 1, wherein the state data includes at least the network mode, IP address, subnet mask, and gateway IP address of each BMC. The remote elimination method of the abnormal state of the cabinet according to claim 1, wherein the feedback information includes the cabinet server management system, the cabinet, each endpoint server, the RMC, and each BMC when the operation behavior is performed Respectively generated feedback, system parameters and execution data. For the remote elimination method of the abnormal state of the cabinet as described in claim 1, the step b) includes the following steps: 　　b1) Determine whether one of the BMCs has lost contact with the cabinet server based on the status data and the feedback information Manage the connection between systems; 　　b2) Determine whether one of the BMCs has just implemented a network configuration operation based on the operation behavior; and 　　b3) Have just implemented the network configuration operation on any BMC, and connected to the network When the connection is lost after the route setting operation, the BMC is deemed to be in the third type of attention state. The remote troubleshooting method for the abnormal state of the cabinet as described in claim 6, wherein the step b1) is to set the network mode of any BMC to a static IP mode, and the static IP address of the BMC and the data center When one of the multiple sets of dynamic IP addresses allocated by a Dynamic Host Configuration Protocol (DHCP) server in the server is repeated, it is determined that the BMC has lost connection. The remote troubleshooting method for the abnormal state of the cabinet as described in claim 6, wherein the step b1) is when the network mode of any BMC is set to a static IP mode, and the IP address of the BMC gateway is set incorrectly , It is judged that the BMC has lost connection. For the remote elimination method of the abnormal state of the cabinet as described in claim 6, the step c includes the following steps: 　　c1) When determining that any BMC is in the third type of concern state, determine and connect to the BMC RMC; 　　c2) Control the RMC to check the endpoint server to which the BMC belongs through an internal hardware circuit of the cabinet, where the RMC is physically connected to all the BMCs in the cabinet through the internal hardware circuit; 　　c3) When the endpoint server does not exist, a warning signal is issued; and 　　c4) When the endpoint server exists, control the RMC to send an Intelligent Platform Management Interface (IPMI) command to the endpoint server through the internal hardware line BMC to restore the connection between the BMC and the server management system of the rack. According to the remote elimination method of the abnormal state of the cabinet as described in claim 9, the step c4) is to reset the static IP address of the BMC through the IPMI instruction, or reset the gateway IP address of the BMC.

Images