CN114138536A

CN114138536A - Server fault positioning method, system, terminal and storage medium

Info

Publication number: CN114138536A
Application number: CN202111471091.3A
Authority: CN
Inventors: 刘玉磊
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2021-12-03
Filing date: 2021-12-03
Publication date: 2022-03-04

Abstract

The invention relates to the technical field of servers, in particular to a server fault positioning method, a system, a terminal and a storage medium, which comprise the following steps: monitoring the test log, and disconnecting the test log from all external PCIe devices after monitoring that the test log has error information; restarting the server, re-collecting the test log and judging whether the new test log has error reporting information: if yes, judging that the PCIe port has no fault; if not, the PCIe ports are judged to have faults, and the PCIe ports are checked one by one. The invention can quickly and effectively position PCIe interruption and error diagnosis generated by conflict. The method has the characteristics of accurate positioning and quick aging.

Description

Server fault positioning method, system, terminal and storage medium

Technical Field

The invention relates to the technical field of servers, in particular to a server fault positioning method, a server fault positioning system, a server fault positioning terminal and a storage medium.

Background

In the existing server fault diagnosis method, a server needs to be linked to a BMC web to check related bad logs or a system down-call black box log is analyzed. This approach suffers from two disadvantages: first, the parsing of the log requires the help of professional developers, is not time-efficient, and sometimes affects the delivery of the product. Secondly, the amount of information acquired by the system bmc sel is limited, and even a detailed position cannot be located. This causes much inconvenience and interference in maintenance.

Disclosure of Invention

In view of the above-mentioned deficiencies of the prior art, the present invention provides a method, a system, a terminal and a storage medium for server fault location, so as to solve the above-mentioned technical problems.

In a first aspect, the present invention provides a server fault location method, including:

monitoring the test log, and disconnecting the test log from all external PCIe devices after monitoring that the test log has error information;

restarting the server, re-collecting the test log and judging whether the new test log has error reporting information:

if yes, judging that the PCIe port has no fault;

if not, the PCIe ports are judged to have faults, and the PCIe ports are checked one by one.

Further, monitoring the test log includes:

and creating a monitoring process, wherein the monitoring process screens error information from the test logs by utilizing a keyword screening technology.

Further, after determining that the PCIe port is not faulty, the method further includes:

restarting the server, and monitoring the state of a system health indicator lamp in the process of restarting the server;

and if the system health indicator light is converted into a fault state from the monitoring state when the basic input and output system finishes all startup skip network startup, judging that the network module has a fault.

Further, the PCIe ports are checked one by one, including:

reconnecting all PCIe ports to corresponding external PCIe devices;

randomly selecting a PCIe port as a target port, and unloading the drive of the target port;

restarting a server, collecting a test log again, monitoring whether the test log has error reporting information, storing a monitoring result and installing a driver for the target port;

and traversing all PCIe ports, and outputting the PCIe port with the monitoring result of no error reporting information as a fault port.

In a second aspect, the present invention provides a server fault location system, including:

the error reporting monitoring unit is used for monitoring the test log and disconnecting the test log from all external PCIe devices after monitoring that the test log has error reporting information;

the error reporting judgment unit is used for restarting the server, re-collecting the test logs and judging whether the new test logs have error reporting information or not;

the first judging unit is used for judging that the PCIe port has no fault if the new test log has error information;

and the second judging unit is used for judging that the PCIe port has faults if the new test log does not have error information, and checking the PCIe port one by one.

Further, the error monitoring unit includes:

and the process creation module is used for creating a monitoring process, and the monitoring process screens the error information from the test log by using a keyword screening technology.

Further, the system further comprises:

the indication monitoring unit is used for restarting the server and monitoring the state of the system health indicator lamp in the process of restarting the server;

and the network judging unit is used for judging that the network module has a fault if the system health indicator lamp is converted into a fault state from a monitoring state when the basic input and output system finishes all startup skip network starting.

Further, the second determination unit includes:

the device connection module is used for reconnecting all the PCIe ports with corresponding external PCIe devices;

the driving unloading module is used for randomly selecting one PCIe port as a target port and unloading the driving of the target port;

the result acquisition module is used for restarting the server, re-collecting the test log, monitoring whether the test log has error reporting information, storing the monitoring result and installing a driver for the target port;

and the result output module is used for traversing all PCIe ports and outputting the PCIe port with the monitoring result of no error reporting information as a fault port.

In a third aspect, a terminal is provided, including:

a processor, a memory, wherein,

the memory is used for storing a computer program which,

the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.

In a fourth aspect, a computer storage medium is provided having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the above aspects.

The server fault positioning method, the server fault positioning system, the server fault positioning terminal and the storage medium have the advantages that all peripheral equipment are disconnected after error information is acquired by monitoring the error information. The minimum configuration of the machine during starting is met. And successively electrifying to observe the state of the machine, and reading whether error information still exists or not by the system until the error information disappears. To the module that last disconnected the component. Then, the modules are planed out in sequence. The invention can quickly and effectively position PCIe interruption and error diagnosis generated by conflict. The method has the characteristics of accurate positioning and quick aging.

In addition, the invention has reliable design principle, simple structure and very wide application prospect.

Drawings

In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.

FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.

FIG. 2 is a schematic block diagram of a system of one embodiment of the present invention.

Fig. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The following explains key terms appearing in the present invention.

The BMC executes a server remote Management controller, which is called Basebard Management controller in English. The method can perform operations such as firmware upgrading, machine equipment checking and the like on the machine in a state that the machine is not started. Fully implementing IPMI functionality in a BMC requires a powerful 16-bit or 32-bit microcontroller as well as RAM for data storage, flash memory and firmware for non-volatile data storage, providing basic remote manageability in terms of secure remote restart, secure re-power-up, LAN alerts and system health monitoring. In addition to the basic IPMI function and system operation monitoring function, the mBMC enables selection and protection of BIOS flash devices by storing the previous BIOS using one of 2 flash memories. For example, when the system cannot be started after the remote BIOS is upgraded, the remote administrator can switch back to the BIOS image that worked before to start the system. Once BIOS is upgraded, BIOS image can be locked to prevent virus from invading it.

BIOS is an abbreviation of English "Basic Input Output System", and the name of Chinese after translation is "Basic Input Output System". It is a standard firmware interface in the industry for IBM PC compatible systems. It is a set of programs solidified on a ROM chip on the internal mainboard of computer, it stores the most important basic input and output program of computer, self-checking program after starting up and system self-starting program, it can read and write the specific information set by system from CMOS. Its primary function is to provide the lowest level, most direct hardware setup and control for the computer. In addition, the BIOS provides some system parameters to the operating system. The change of system hardware is hidden by BIOS, and programs use BIOS functions rather than directly control the hardware. Modern operating systems ignore the abstraction layer provided by the BIOS and directly control the hardware components.

FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention. The execution subject in fig. 1 may be a server fault location system.

As shown in fig. 1, the method includes:

step 110, monitoring the test log, and disconnecting the test log from all external PCIe devices after monitoring that the test log has error information;

step 120, restarting the server, re-collecting the test log and judging whether the new test log has error reporting information:

step 130, if yes, judging that the PCIe port has no fault;

and 140, if not, judging that the PCIe ports have faults, and checking the PCIe ports one by one.

In order to facilitate understanding of the present invention, the server fault location method provided by the present invention is further described below with reference to the principle of the server fault location method of the present invention and the process of locating a server fault in the embodiments.

Specifically, the server fault positioning method includes:

and S1, monitoring the test log, and disconnecting all external PCIe devices after monitoring that the test log has error information.

And creating a monitoring process, wherein the monitoring process screens error information from the test logs by utilizing a keyword screening technology. The keyword may be an error report type such as critical interrupt PCIE _ STATUS | Bus Degraded | Asserted.

S2, restarting the server, re-collecting the test log and judging whether the new test log has error information: if yes, judging that the PCIe port has no fault; if not, the PCIe ports are judged to have faults, and the PCIe ports are checked one by one.

After all external PCIe devices are disconnected, the machine cover is assembled, the machine is powered on and started, and a bmc sel log is extracted from the system. It is found that the error message of the clinical interrupt PCI _ status disappears. Meanwhile, the system health lamp returns to normal.

After the peripheral module is disconnected, the machine is recovered to be normal, and the Critical interrupt PCI _ status disappears, so that the PCI interrupt caused by the error report of the external PCI can be judged. If there are multiple peripheral modules, further fault location is required:

1) reconnecting all PCIe ports to corresponding external PCIe devices;

2) randomly selecting a PCIe port as a target port, and unloading the drive of the target port;

3) restarting a server, collecting a test log again, monitoring whether the test log has error reporting information, storing a monitoring result and installing a driver for the target port;

4) and traversing all PCIe ports, and outputting the PCIe port with the monitoring result of no error reporting information as a fault port.

The above embodiment is an automatic troubleshooting embodiment, and may be executed by a script. In other embodiments of the present invention, the external devices may be manually disconnected one by one.

If all the peripheral modules are disconnected, the error information still exists. And restarting the server, and finding that the system health lamp keeps a normally bright green lamp in the whole startup process of the BIOS in the startup process, and when the startup is completed at the moment, jumping to the network for startup, the system health lamp is suddenly changed into a red lamp. Thereby determining that the network is out of order with a high probability. The analysis network module comprises five parts: 25G network card, adapter card 1, 10G network card, adapter card 2 and slim OCPlink line. The processing method, for example, verifies the network modules one by one, and finally finds that there is a problem in the line linking the slim sas interface of the riser card 2. As long as the line of the adapter card 2 is accessed, the machine immediately reports that the Critical interrupt PCI _ status disappears. After the plug is pulled out, the machine returns to normal.

As shown in fig. 2, the system 200 includes:

an error monitoring unit 210, configured to monitor the test log, and disconnect all external PCIe devices after monitoring that the test log has error information;

an error reporting determination unit 220, configured to restart the server, re-collect the test log, and determine whether the new test log has error reporting information;

a first determining unit 230, configured to determine that there is no failure in the PCIe port if there is error information in the new test log;

the second determining unit 240 is configured to determine that the PCIe ports have a failure if the new test log does not have the error information, and perform one-by-one troubleshooting on the PCIe ports.

Optionally, as an embodiment of the present invention, the error monitoring unit includes:

Optionally, as an embodiment of the present invention, the system further includes:

Optionally, as an embodiment of the present invention, the second determination unit includes:

Fig. 3 is a schematic structural diagram of a terminal 300 according to an embodiment of the present invention, where the terminal 300 may be used to execute the server fault location method according to the embodiment of the present invention.

Among them, the terminal 300 may include: a processor 310, a memory 320, and a communication unit 330. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.

The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.

The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.

A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.

The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).

Therefore, the invention can disconnect all peripheral equipment by monitoring the error information and acquiring the error information. The minimum configuration of the machine during starting is met. And successively electrifying to observe the state of the machine, and reading whether error information still exists or not by the system until the error information disappears. To the module that last disconnected the component. Then, the modules are planed out in sequence. The invention can quickly and effectively position PCIe interruption and error diagnosis generated by conflict. The method has the characteristics of accurate positioning and fast aging, and the technical effects achieved by the embodiment can be seen from the description above, which is not repeated herein.

Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.

The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.

In the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, systems or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A server fault location method is characterized by comprising the following steps:

if yes, judging that the PCIe port has no fault;

2. The method of claim 1, wherein monitoring the test log comprises:

3. The method of claim 1, wherein after determining that the PCIe port is not faulty, the method further comprises:

4. The method of claim 1, wherein the checking PCIe ports one by one comprises:

reconnecting all PCIe ports to corresponding external PCIe devices;

5. A server fault location system, comprising:

6. The system of claim 5, wherein the error monitoring unit comprises:

7. The system of claim 5, further comprising:

8. The system of claim 5, wherein the second decision unit comprises:

9. A terminal, comprising:

a processor;

a memory for storing instructions for execution by the processor;

wherein the processor is configured to perform the method of any one of claims 1-4.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-4.