CN114911578A - Storage system monitoring and fault collection method, device, terminal and storage medium - Google Patents

Abstract

The invention relates to the field of storage system monitoring, and specifically discloses a storage system monitoring and fault collection method, device, terminal and storage medium. A monitoring server is built so that the monitoring server communicates with the storage system; Storage system status; when the storage system status is abnormal, dump file collection is triggered; based on the data in the collected dump file, the storage system fault cause analysis and fault cause location are performed. The present invention builds a monitoring server, collects dump information or OSES information in time when a fault occurs, and the collected information includes fault information, so as to analyze and locate the fault information, and avoid the dilemma that the fault problem cannot be reproduced or is difficult to reproduce.

Description

Storage system monitoring and fault collection method, device, terminal and storage medium

technical field

The invention relates to the field of storage system monitoring, in particular to a storage system monitoring and fault collection method, device, terminal and storage medium.

Background technique

During the test, the tester cannot keep staring at the operation of the storage system, or some fault injection or repeated tests that need to be performed for a long time need to be scheduled through scripts, so the storage system status cannot be checked frequently, while the storage system logs It will grow over time, and the log may be overwritten. When an exception is found, the log when the problem occurs cannot be viewed. Some faults occur probabilistically and cannot be reproduced every time. Therefore, once some faults miss the information at that time, it will cost a lot of labor and time to reproduce again.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention provides a storage system monitoring and fault collection method, device, terminal and storage medium, which can collect all required information after a fault occurs, and perform fault analysis and location.

In a first aspect, the technical solution of the present invention provides a storage system monitoring and fault collection method, comprising the following steps:

Build a monitoring server so that the monitoring server communicates with the storage system;

Log in to the storage system, access the storage system periodically, and query the storage system status;

When the storage system status is abnormal, dump file collection is triggered;

Based on the collected data in the dump file, analyze the fault cause of the storage system and locate the fault cause.

Further, the monitoring server is connected to each controller of the storage system through a serial port;

The method also includes the following steps:

If the storage system cannot be logged in, a login attempt will be made every preset time;

If you still cannot log in to the storage system after trying to log in a preset number of times, enter the chassis management service of each controller;

Under the chassis management service, record the specified information by querying the command;

Perform fault analysis and fault location of the storage system based on the specified information recorded.

Further, when logging into the storage system normally, the cycle of accessing the storage system is the same as the failure injection cycle of the storage system.

Further, the queried storage system status includes cluster status and alarm events;

Abnormal storage system status includes that the cluster status is not as expected or an unexpected alarm event is generated.

In a second aspect, the technical solution of the present invention provides a storage system monitoring and fault collection device, which is characterized in that a monitoring server is built so that the monitoring server communicates with the storage system;

The device includes,

Login module: log in to the storage system;

Status query module: Periodically access the storage system to query the storage system status;

File collection trigger module: when the storage system status is abnormal, trigger dump file collection;

The first fault analysis and localization module: analyzes the fault cause of the storage system and locates the fault cause according to the collected data in the dump file.

Further, the monitoring server is connected to each controller of the storage system through a serial port;

If the login module cannot log in to the storage system, a login attempt will be made every preset time;

The device also includes,

Chassis management service entry module: If you still cannot log in to the storage system after trying to log in a preset number of times, enter the chassis management service of each controller;

Specified information query and record module: under the chassis management service, query specified information to record through commands;

The second fault analysis and localization module: analyzes the fault cause of the storage system and locates the fault cause according to the recorded specified information.

Further, the cycle of accessing the storage system by the status query module is the same as the failure injection cycle of the storage system.

Further, the storage system status queried by the fault query module includes cluster status and alarm events;

Abnormal storage system status includes that the cluster status is not as expected or an unexpected alarm event is generated.

In a third aspect, the technical solution of the present invention provides a terminal, including:

Memory, used to store storage system monitoring and fault collection procedures;

The processor is configured to implement the steps of the storage storage system monitoring and fault collection method described in any one of the above when executing the storage storage system monitoring and fault collection program.

In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which a storage storage system monitoring and fault collection program is stored, and the storage storage system monitoring and fault collection program is executed by a processor When implementing the steps of the storage storage system monitoring and fault collection method described in any of the above.

Compared with the prior art, the method, device, terminal and storage medium for monitoring and collecting faults of a storage storage system provided by the present invention have the following beneficial effects: a monitoring server is built, dump information or OSES information is collected in time when a fault occurs, and the collected The information contains fault information, so as to analyze and locate the fault information, so as to avoid the dilemma that the fault problem cannot be reproduced or is difficult to reproduce.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application or the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only For some embodiments of the present application, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a schematic flowchart of a method for monitoring and collecting faults in a storage system according to an embodiment of the present invention.

FIG. 2 is a schematic block diagram of the structure of a storage system monitoring and fault collecting apparatus according to an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.

Detailed ways

The English terms involved in the present invention are explained below.

Dump: The dump file is the memory image of the process, and the execution state of the program can be saved to the dump file through the debugger.

OSES: is the abbreviation of Organic SAS Enclosure Service. The full name in Chinese is Unified SAS Chassis Service. OSES, as the whole chassis management module of storage devices, has powerful functions. It can not only monitor the running status of devices in real time, but also realize and store various systems. Interaction and management between modules; SAS is the abbreviation of Serial Attached SCSI, and the full name in Chinese is serial port connection interface.

LDBE: is a tool that can view dump information.

In order to make those skilled in the art better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The core of the present invention is to create a monitoring server to monitor the status of the storage system for the problem that the log of the storage system may be overwritten, resulting in inability to obtain fault information. Perform fault analysis and location to prevent logs from being overwritten and failure to analyze faults.

FIG. 1 is a schematic flowchart of a method for monitoring and collecting faults in a storage system according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps.

S101, build a monitoring server, so that the monitoring server communicates with the storage system.

A monitoring server is built in advance to monitor the storage system and issue instructions. It can be understood that the communication between the monitoring server and the storage system not only realizes the monitoring of the storage system by the monitoring server, but also enables the monitoring server to collect data from the storage system.

The built monitoring server can be a Linux server.

S102, log in to the storage system, periodically access the storage system, and query the storage system status.

The monitoring server first logs in to the storage system, and password-free login from the monitoring server to the storage system can be configured. Of course, password login can also be set according to specific conditions and user needs. The choice of specific login method does not affect the implementation of the embodiments of this application.

The monitoring server periodically accesses the storage system, visits the storage system at regular intervals, queries the storage system status, and determines whether the storage system is abnormal.

S103, when the storage system state is abnormal, trigger the dump file collection.

The storage system provides CLI commands for collecting dump files. When the monitoring server finds that the storage system is abnormal, it triggers the storage system to issue CLI commands to collect dump files.

S104 , analyzing the failure cause of the storage system and locating the failure cause according to the collected data in the dump file.

It is understandable that the collected dump file is transmitted to the monitoring server, and the monitoring server analyzes the storage system failure cause and locates the failure cause for the data in the dump file.

When an unexpected failure or other problem occurs in the storage system, the dump file is used to analyze the running state of the process to analyze the cause of the problem. Dump can be triggered manually or collected automatically by the system. The storage system triggers dump log collection when it encounters unexpected restarts or some other business failures. You can use the LDBE tool to view the dump, so that maintenance or R&D personnel can analyze the cause of the problem.

The embodiment of the present invention provides a storage system monitoring and fault collection method. A monitoring server is built to collect dump information or OSES information in time when a fault occurs. The collected information includes fault information, so as to analyze and locate fault information and avoid fault problems. Difficult dilemmas cannot be reproduced or reproduced.

On the basis of the above embodiment, as a preferred implementation, the method records the relevant information through the chassis management server (OSES) to analyze and locate the fault when encountering the situation that the storage system cannot be logged in and the dump file cannot be performed. Specifically, the method includes: :

If the storage system cannot be logged in, a login attempt will be made every preset time;

If you still cannot log in to the storage system after trying to log in a preset number of times, enter the chassis management service of each controller;

Under the chassis management service, record the specified information by querying the command;

Perform fault analysis and fault location of the storage system based on the specified information recorded.

It should be noted that after the monitoring server is created, connect the monitoring server to each controller of the storage system through the serial port, so that when the storage system cannot be logged in later, it can enter the chassis management service of the controller to record relevant information. The storage system manages the chassis through the chassis management service. During the use or testing of the storage system, some faults may cause the storage system to fail to start, so that the dump file cannot be collected and the problem cannot be located. At this time, you need to log in to OSES to check the status of the hardware or underlying software, so as to analyze the reason why the storage system cannot be started.

When you cannot log in to the storage system, first try to log in at a certain interval. If the storage system still fails after a certain number of attempts, enter the chassis management service of each controller, and then log in to the chassis management system. Under the service, the specified information is queried through the command to record. For example, a login attempt is made at an interval of 30s, and a timeout occurs after 20 attempts. Of course, the specific time interval and number of times can be specifically set according to the actual situation and user needs, and the set value does not affect the implementation of the embodiments of the present application.

On the basis of the above embodiment, as a preferred embodiment, the method further includes:

When logging in to the storage system normally, the cycle of accessing the storage system is the same as the failure injection cycle of the storage system.

When the monitoring server can log in to the storage system normally, if the storage system is performing a fault injection test, the monitoring server accesses the storage system at the same cycle as the storage system's fault injection cycle. For example, the current fault injection cycle is 30 minutes. The same frequency is also maintained to ensure that the storage state is checked after each fault injection. Of course, the period of the monitoring server accessing the storage system is determined according to the actual test. For example, if a regular stress test is not performed for periodic fault injection tests, you can set a monitoring period for monitoring. It can be understood that the specific set value of the monitoring period does not affect the implementation of the embodiments of the present invention.

On the basis of the above-mentioned embodiment, as a preferred embodiment, when the monitoring server queries the storage system status, the content to be checked can also be determined by the test content. For example, the storage system status to be queried includes cluster status and alarm events. The abnormal state of the storage system includes that the cluster state does not meet expectations or an unexpected alarm event is generated. When the cluster state does not meet expectations or an unexpected alarm event occurs, the storage system is triggered to collect dump files.

In order to further understand the present invention, a specific embodiment is provided below to further illustrate the present invention, and the specific embodiment includes the following steps.

Step 1, build a linux server to monitor the storage system and issue instructions. You need to configure the password-free login from the Linux server to the storage.

Step 2: Connect each controller in the storage with a serial cable through the serial port server, which is used to collect logs when the storage system fails to start.

Step 3: The monitoring script in the Linux server periodically accesses the storage system to query the status of the storage system.

The access period is determined according to the actual test. For example, if the current fault injection is performed for a period of 30 minutes, the storage status query will also maintain the same frequency to ensure that the storage status check is performed after each fault injection; the storage status check can also be checked according to the test. It depends on the content, such as general cluster status, alarm events, etc.

If you do not perform periodic fault injection tests, but perform regular stress tests, etc., you can set a monitoring cycle for monitoring.

Step 4: If it is found that the storage status to be queried is not in line with expectations, or an unexpected alarm event is generated, the CLI command is used to trigger the liveump collection.

Step 5: If you cannot log in to the storage system during status query, try once every 30s (the time can be set by yourself), and the timeout will expire after 20 attempts.

Step 6: After the timeout, use the serial port to connect to each controller through the serial port server, and then enter OSES through the command.

Step 7: Under the OSES command line, query the specified information through the command to record.

Through the information collected in steps 4 and 7, the root cause of the fault can be located.

Embodiments of a storage system monitoring and fault collection method are described in detail above. Based on the storage system monitoring and fault collection methods described in the above embodiments, an embodiment of the present invention further provides a storage system monitoring and fault collection apparatus, It is used to implement storage system monitoring and fault collection methods.

FIG. 2 is a schematic block diagram of the structure of a storage system monitoring and fault collection device provided by an embodiment of the present invention. As shown in FIG. 2, the device includes: a login module 101, a status query module 102, a file collection trigger module 103, a first fault An analysis and location module 104 , a chassis management service entry module 105 , a specified information query and record module 106 , and a second fault analysis and location module 107 .

Build a monitoring server so that the monitoring server communicates with the storage system. The monitoring server is connected to each controller of the storage system through the serial port.

Login module 101: log in to the storage system.

Status query module 102: Periodically access the storage system to query the storage system status.

File collection triggering module 103 : triggering dump file collection when the storage system status is abnormal.

The first fault analysis and location module 104 : analyzes the fault cause of the storage system and locates the fault cause according to the collected data in the dump file.

Wherein, if the login module 101 fails to log in to the storage system, a login attempt is made every preset time. The storage system status queried by the fault query module 102 includes the cluster status and alarm events; the abnormal storage system status includes that the cluster status is inconsistent with expectations or an unexpected alarm event is generated.

Chassis management service entry module 105: If the storage system cannot be logged in after a preset number of attempts, enter the chassis management service of each controller.

Specified information query and record module 106: Under the chassis management service, query specified information through an instruction to record.

The second fault analysis and location module 107 : analyzes the fault cause of the storage system and locates the fault cause according to the recorded specified information.

The storage and storage system monitoring and fault collection apparatus in this embodiment is used to implement the aforementioned storage and storage system monitoring and fault collection method. Therefore, the specific implementation of the apparatus can be found in the embodiment section of the storage and storage system monitoring and fault collection method above. Therefore, reference may be made to the descriptions of the corresponding partial embodiments for specific implementations thereof, which will not be described herein again.

In addition, since the storage storage system monitoring and fault collecting apparatus in this embodiment is used to implement the aforementioned storage storage system monitoring and fault collecting method, its function corresponds to the function of the above method, and will not be repeated here.

FIG. 3 is a schematic structural diagram of a terminal device 300 according to an embodiment of the present invention, including: a processor 310 , a memory 320 , and a communication unit 330 . The processor 310 implements the following steps when implementing the storage storage system monitoring and fault collection program stored in the memory 320:

Log in to the storage system, access the storage system periodically, and query the storage system status;

When the storage system status is abnormal, dump file collection is triggered;

Based on the collected data in the dump file, analyze the fault cause of the storage system and locate the fault cause.

The present invention: builds a monitoring server, collects dump information or OSES information in time when a fault occurs, and the collected information includes fault information, so as to analyze and locate the fault information, and avoid the dilemma that the fault problem cannot be reproduced or is difficult to reproduce.

In some specific embodiments, when the processor 310 executes the storage system monitoring and fault collection subprograms stored in the memory 320, it can be specifically implemented that: if the storage system cannot be logged in, a login attempt is made every preset time; After trying to log in to the storage system for a preset number of times, if you still cannot log in to the storage system, enter the chassis management service of each controller; under the chassis management service, query the specified information through commands to record; analyze the storage system failure cause and analyze the specified information according to the recorded specified information. Locating the cause of the fault.

In some specific embodiments, when the processor 310 executes the storage system monitoring and fault collection subprograms stored in the memory 320, it can be specifically implemented: when logging into the storage system normally, the period of accessing the storage system and the period of fault injection of the storage system same.

In some specific embodiments, when the processor 310 executes the storage system monitoring and fault collection subprograms stored in the memory 320, it can be specifically implemented that: the queried storage system status includes the cluster status and alarm events; the abnormal storage system status includes the cluster status The status is not as expected or an unexpected alarm event is generated.

The terminal device 300 includes a processor 310 , a memory 320 and a communication unit 330 . These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation of the present invention. It can be either a bus structure, a star structure, or a More or fewer components than shown may be included, or some components may be combined, or a different arrangement of components.

Wherein, the memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or their combination, such as static random access memory (SRAM), electrical Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk . When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is enabled to execute some or all of the steps in the following method embodiments.

The processor 310 is the control center of the storage terminal, using various interfaces and lines to connect various parts of the entire electronic terminal, by running or executing the software programs and/or modules stored in the memory 320, and calling the data stored in the memory, To perform various functions of the electronic terminal and/or process data. The processor may be composed of an integrated circuit (Integrated Circuit, IC for short), for example, may be composed of a single packaged IC, or may be composed of a plurality of packaged ICs connected with the same function or different functions. For example, the processor 310 may only include a central processing unit (Central Processing Unit, CPU for short). In the embodiment of the present invention, the CPU may be a single computing core, or may include multiple computing cores.

The communication unit 330 is used for establishing a communication channel, so that the storage terminal can communicate with other terminals. Receive user data sent by other terminals or send user data to other terminals.

The present invention also provides a computer storage medium, which can be a magnetic disk, an optical disk, a read-only memory (English: read-only memory, ROM for short) or a random access memory (English: random access memory). , referred to as: RAM) and so on.

The computer storage medium stores a storage system monitoring and fault collection program, and when the storage system monitoring and fault collection program is executed by the processor, the following steps are implemented:

Log in to the storage system, access the storage system periodically, and query the storage system status;

When the storage system status is abnormal, dump file collection is triggered;

Based on the collected data in the dump file, analyze the fault cause of the storage system and locate the fault cause.

The present invention: builds a monitoring server, collects dump information or OSES information in time when a fault occurs, and the collected information includes fault information, so as to analyze and locate the fault information, and avoid the dilemma that the fault problem cannot be reproduced or is difficult to reproduce.

In some specific embodiments, when the storage system monitoring and fault collection subprograms stored in the readable storage medium are executed by the processor, it can be specifically implemented that: if the storage system cannot be logged in, a login attempt is made every preset time. ;If you still cannot log in to the storage system after trying to log in for a preset number of times, enter the chassis management service of each controller; under the chassis management service, query the specified information through commands to record; according to the specified information recorded, determine the cause of the storage system failure Analysis and fault location.

In some specific embodiments, when the storage system monitoring and fault collection subprograms stored in the readable storage medium are executed by the processor, it can be specifically implemented that: when logging into the storage system normally, the period of accessing the storage system and the failure of the storage system The injection cycle is the same.

In some specific embodiments, when the storage system monitoring and fault collection subprograms stored in the readable storage medium are executed by the processor, it can be specifically implemented that: the queried storage system status includes cluster status and alarm events; the storage system status is abnormal Including that the cluster state is not as expected or an unexpected alarm event is generated.

Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention can be embodied in the form of software products in essence or in the parts that make contributions to the prior art. The computer software products are stored in a storage medium such as a USB flash drive, a mobile Hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes, including several instructions to make a computer terminal (It may be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The above disclosure is only the preferred embodiment of the present invention, but the present invention is not limited thereto, any non-creative changes that can be conceived by those skilled in the art, and some improvements and modifications made without departing from the principles of the present invention , all should fall within the protection scope of the present invention.

Claims (10)

1. a storage system monitoring and fault collection method, is characterized in that, comprises the following steps: Build a monitoring server so that the monitoring server communicates with the storage system; Log in to the storage system, access the storage system periodically, and query the storage system status; When the storage system status is abnormal, dump file collection is triggered; Based on the collected data in the dump file, analyze the fault cause of the storage system and locate the fault cause. 2. The storage system monitoring and fault collection method according to claim 1, wherein the monitoring server is connected to each controller of the storage system through a serial port; The method also includes the following steps: If the storage system cannot be logged in, a login attempt will be made every preset time; If you still cannot log in to the storage system after trying to log in a preset number of times, enter the chassis management service of each controller; Under the chassis management service, record the specified information by querying the command; Perform fault analysis and fault location of the storage system based on the specified information recorded. 3 . The storage system monitoring and fault collection method according to claim 2 , wherein when logging into the storage system normally, the period of accessing the storage system is the same as the period of fault injection of the storage system. 4 . 4. The storage system monitoring and fault collection method according to claim 3, wherein the queried storage system status includes a cluster status and an alarm event; Abnormal storage system status includes that the cluster status is not as expected or an unexpected alarm event is generated. 5. A storage system monitoring and fault collection device, characterized in that a monitoring server is set up so that the monitoring server communicates with the storage system; The device includes, Login module: log in to the storage system; Status query module: Periodically access the storage system to query the storage system status; File collection trigger module: when the storage system status is abnormal, trigger dump file collection; The first fault analysis and localization module: analyzes the fault cause of the storage system and locates the fault cause according to the collected data in the dump file. 6. The storage system monitoring and fault collection device according to claim 5, wherein the monitoring server is connected to each controller of the storage system through a serial port; If the login module cannot log in to the storage system, a login attempt will be made every preset time; The device also includes, Chassis management service entry module: If you still cannot log in to the storage system after trying to log in a preset number of times, enter the chassis management service of each controller; Specified information query and record module: under the chassis management service, query specified information to record through commands; The second fault analysis and localization module: analyzes the fault cause of the storage system and locates the fault cause according to the recorded specified information. 7 . The storage system monitoring and fault collection device according to claim 6 , wherein the cycle of accessing the storage system by the status query module is the same as the failure injection cycle of the storage system. 8 . 8 . The storage system monitoring and fault collection device according to claim 7 , wherein the storage system status queried by the fault query module includes a cluster status and an alarm event; 8 . Abnormal storage system status includes that the cluster status is not as expected or an unexpected alarm event is generated. 9. A terminal, characterized in that, comprising: Memory, used to store storage system monitoring and fault collection procedures; The processor is configured to implement the steps of the storage system monitoring and fault collecting method according to any one of claims 1-4 when executing the storage system monitoring and fault collecting program. 10. A computer-readable storage medium, characterized in that, a storage system monitoring and fault collection program is stored on the readable storage medium, and the storage system monitoring and fault collection program is implemented as claimed in claim 1 when the storage system monitoring and fault collection program is executed by a processor. -4 The steps of any one of the storage system monitoring and fault collection methods.

Images