WO2021129367A1

WO2021129367A1 - Method and apparatus for monitoring distributed storage system

Info

Publication number: WO2021129367A1
Application number: PCT/CN2020/134339
Authority: WO
Inventors: 龚治文; 饶俊明; 卢道和; 郑晓腾; 龚洵峰; 刘生庆; 吴立; 吴传民
Original assignee: 深圳前海微众银行股份有限公司
Priority date: 2019-12-23
Filing date: 2020-12-07
Publication date: 2021-07-01
Also published as: CN111049705A; CN111049705B

Abstract

The present invention provides a method and an apparatus for monitoring a distributed storage system. The method comprises: a monitoring server sending acquisition instructions to clusters in a distributed storage system; the monitoring server acquiring monitoring data fed back by the clusters on the basis of the acquisition instructions, the monitoring data comprising health data of the clusters and state data of clients connected to the the clusters; and for at least one cluster, the monitoring server determining alarm information from the monitoring data of the clusters according to a preset alarm rule, and reporting the alarm information to an alarm platform. In the solution, the monitoring server issues the acquisition instructions to the clusters in the distributed storage system, so that the monitoring server can monitor a plurality of clusters at the same time; in addition, the monitoring data fed back by the clusters comprises state data of the clients connected to the clusters, facilitating the monitoring server determining alarm information by analyzing the state data of the clients connected to the clusters, thereby achieving the purpose of the monitoring server monitoring the clients connected to the clusters.

Description

Method and device for monitoring distributed storage system

Cross references to related applications

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201911336662.5, and the application name is "a method and device for monitoring a distributed storage system" on December 23, 2019, the entire content of which is incorporated by reference In this application.

Technical field

The present invention relates to the field of financial technology (Fintech), in particular to a method and device for monitoring a distributed storage system.

Background technique

With the development of computer technology, more and more technologies (such as cloud computing and big data) are applied in the financial field. The traditional financial industry is gradually transforming to financial technology, and big data technology is no exception. However, due to the security and real-time requirements of the financial and payment industries, higher requirements are also placed on big data technology.

In consideration of factors such as scalability and high availability required for massive data, the banking industry generally chooses a distributed storage system such as CephFS (Ceph File System) as a technical solution for shared storage. Among them, the Ceph Fuse client (user space file system client of the Ceph file system) is connected to CephFS; at the same time, those skilled in the art usually use the open source Prometheus (Prometheus) to monitor CephFS. To monitor. Among them, Prometheus is mainly composed of Exporter (client for Prometheus monitoring data collection) and Prometheus Sever (server for Prometheus monitoring); CephFS is mainly composed of monitor (Monitor, abbreviated as MON), target storage device (Object Storage Device, It is abbreviated as OSD) and metadata server (MetaData Sever, abbreviated as MDS) and other components. In addition, the CephFS OSD component also has placement groups (Placement Groups, abbreviated as PG).

Regarding the technical solution of Prometheus monitoring CephFS in the prior art, there are the following two problems:

First, Prometheus's monitoring of CephFS is mainly manifested in Prometheus's data collection of CephFS OSD component status and CephFS PG status, but Prometheus does not implement the monitoring of Ceph Fuse client.

Second, Prometheus's monitoring architecture for CephFS is very bloated, which is manifested in the need to deploy a set of Prometheus for each CephFS; in addition, due to the different versions of CephFS, different Exporters need to be deployed for different versions of CephFS. As shown in Figure 1, it is a diagram of the monitoring architecture of CephFS by Prometheus in the prior art. Referring to Figure 1, the M-numbered Prometheus monitoring data collection client collects the M-numbered Ceph file system monitoring data. If the collected monitoring data meets the rules for generating alarm information, it will report the generated alarm information to The M-numbered Prometheus server, in the same way, the N-numbered Prometheus monitoring data collection client collects the monitoring data of the N-numbered Ceph file system. If the collected monitoring data meets the rules for generating alarm information, The generated alarm information is reported to the N-numbered Prometheus server; however, the M-numbered Prometheus monitoring data collection client does not match the N-numbered Ceph file system version, so the M-numbered The client of Prometheus monitoring data collection is used to collect the monitoring data of the N-numbered Ceph file system to report the alarm information of the N-numbered Ceph file system. That is, Prometheus Sever, Exporter, and CephFS did not achieve high availability among the three, resulting in failure to report monitoring information in a timely manner under abnormal conditions.

In summary, the existing technology has problems that Prometheus cannot monitor the Ceph Fuse client and Prometheus has low monitoring efficiency for CephFS.

Summary of the invention

The present invention provides a method and device for monitoring a distributed storage system, which are used to solve the problems that Prometheus cannot monitor Ceph Fuse clients and Prometheus has low monitoring efficiency for CephFS.

In the first aspect, an embodiment of the present invention provides a method for monitoring a distributed storage system. The method includes: a monitoring server sends collection instructions to each cluster in the distributed storage system; and the monitoring server obtains that each cluster is based on The monitoring data fed back by the collection instruction includes the health data of the cluster itself and the status data of the client connected to the cluster; for at least one cluster, the monitoring server obtains data from the cluster according to preset alarm rules. Determine the alarm information in the monitoring data and report the alarm information to the alarm platform.

Based on this solution, the monitoring server can monitor multiple clusters at the same time by issuing collection instructions to each cluster in the distributed storage system, thus avoiding the ineffectiveness of the monitoring server when the cluster and the Exporter version do not match. Monitor each cluster locally; in addition, the monitoring data that each cluster feeds back to the monitoring server also includes the status data of the client connected to the cluster, which is beneficial for the monitoring server to determine the alarm information by analyzing the status data of the client connected to the cluster. The purpose of the monitoring server to monitor the clients connected to the cluster is realized.

As a possible implementation method, there are multiple monitoring servers; any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; the monitoring server is distributed to the Each cluster in the storage system sends collection instructions, including: for any monitoring server, the monitoring server issues collection instructions to at least two node servers in any cluster.

Based on this solution, by setting up multiple monitoring servers for the distributed storage system, on the one hand, by frequently obtaining the monitoring data of each cluster from each cluster in the distributed storage system, it is possible to achieve a full range of the distributed storage system. , And even real-time monitoring goals; on the other hand, by setting up multiple monitoring servers, you can also ensure that when one or more of the monitoring servers is down, there are other available monitoring servers for the distributed monitoring. The storage system is monitored. For any one of the multiple monitoring servers, the monitoring server sends collection instructions to at least two node servers in each cluster to help ensure that the monitoring server is down when one of the node servers is down. The monitoring data of the cluster where the node server is located can also be obtained from other available node servers, so as to realize the effective monitoring of each cluster by the monitoring server.

As a possible implementation method, the alarm rule includes an alarm generation rule; the monitoring server determines the alarm information from the monitoring data according to the preset alarm rule, including: the monitoring server obtains the alarm information from the monitoring data Determine the first client whose connection status with the cluster has changed; the monitoring server determines the second client whose connection status with the cluster has changed according to the service change of the cluster; The client in one client but not included in the second client and the alarm generation rule generate the alarm information of the client.

Based on this solution, through the analysis of monitoring data, the first client whose connection status with the cluster has changed is determined, and through the analysis of known business changes, it is determined that the connection status with the cluster has changed. The second client, by comparing the first client with the second client, can generate alarm information generated due to the abnormality of the client.

As a possible implementation method, the alarm rule further includes an alarm suppression rule; the monitoring server determines the change duration of the service change of the cluster; the monitoring server sets the alarm suppression rule for the alarm information of the client, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.

Based on this solution, after determining the necessary time required by the cluster for business needs, the monitoring server will not report the alarm information of the client to the alarm platform during this necessary time, which can be effective To avoid generating known but useless alarms.

As a possible implementation method, the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself; the monitoring server reports the alarm information to the alarm according to a preset alarm rule The platform includes: the monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and then reports the alarm information of the MDS component to the alarm platform.

Based on this solution, when the monitoring server simultaneously obtains the alarm information of the MDS component of the cluster and the alarm information of the client connected to the cluster, it is considered that the abnormal event of the client connected to the cluster may be caused by the abnormality of the MDS component of the cluster. The monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and reports the alarm information of the MDS component to the alarm platform, automatically shielding the alarm information of the low-level client.

As a possible implementation method, after the monitoring server obtains the monitoring data fed back by each cluster based on the collection instruction, the method further includes: the monitoring server sets a cluster identifier corresponding to each monitoring data.

Based on this solution, the monitoring server marks each acquired monitoring data with the corresponding cluster identification, which helps the monitoring server to quickly make corresponding alarm operations when receiving the same monitoring data of the same cluster in the future.

As a possible implementation method, the alarm rules further include alarm convergence rules; the monitoring server reports the alarm information to the alarm platform according to preset alarm rules, including: the monitoring server determines the alarm information Is the same alarm information that does not appear for the first time in the cluster, then according to the comparison relationship between the alarm level in the alarm convergence rule and the alarm delay, the alarm information is reported to the alarm platform after the delay is set; Among them, the lower the alarm level, the longer the corresponding alarm delay.

Based on this solution, after the monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in a certain cluster, it reports the same alarm that does not appear for the first time to the alarm platform according to the alarm convergence rules and after a set time delay, which can effectively prevent The cluster continues to send out the same alarm repeatedly, resulting in a waste of resources.

In a second aspect, an embodiment of the present invention provides a device for monitoring a distributed storage system, the device includes: a sending unit, configured to send collection instructions to each cluster in the distributed storage system; The monitoring data of each cluster is based on the collection instruction feedback, and the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster; the determining unit, for at least one cluster, is used for according to preset alarm rules, Determine alarm information from the monitoring data of the cluster and report the alarm information to the alarm platform.

As a possible implementation method, there are multiple monitoring servers; any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; for any monitoring server, all The sending unit is specifically configured to issue collection instructions to at least two node servers in any cluster.

Based on this solution, by setting up multiple monitoring servers for the distributed storage system, on the one hand, by frequently obtaining the monitoring data of each cluster from each cluster in the distributed storage system, it is possible to achieve a full range of the distributed storage system. , And even real-time monitoring goals; on the other hand, by setting up multiple monitoring servers, you can also ensure that when one or more of the monitoring servers is down, there are other available monitoring servers for the distributed The storage system is monitored. For any one of the multiple monitoring servers, the monitoring server sends collection instructions to at least two node servers in each cluster to help ensure that the monitoring server is down when one of the node servers is down. The monitoring data of the cluster where the node server is located can also be obtained from other available node servers, so as to realize the effective monitoring of each cluster by the monitoring server.

As a possible implementation method, the alarm rule includes an alarm generation rule; the determining unit is specifically configured to determine from the monitoring data the first client whose connection status with the cluster has changed; The service change of the cluster determines the second client whose connection state with the cluster has changed; and the alarm is generated according to the client included in the first client but not included in the second client and the alarm Rules to generate alarm information for the client.

As a possible implementation method, the alarm rule further includes an alarm suppression rule; the determining unit is specifically configured to determine the change duration of the service change of the cluster; and the alarm suppression rule for the alarm information of the client is set, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.

As a possible implementation method, the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself; the determining unit is specifically configured to determine the alarm level of the alarm information of the MDS component If the alarm information is higher than the alarm information of the client, the alarm information of the MDS component is reported to the alarm platform.

As a possible implementation method, after the monitoring server obtains the monitoring data fed back by each cluster based on the collection instruction, the determining unit is further configured to set a cluster identification corresponding to each monitoring data.

Based on this solution, the monitoring server marks each acquired monitoring data with its corresponding cluster identification, which helps the monitoring server to quickly make corresponding alarm operations when receiving the same monitoring data of the same cluster in the future.

As a possible implementation method, the alarm rule further includes an alarm convergence rule; the determining unit is specifically configured to determine that the alarm information is the same alarm information that does not appear for the first time in the cluster, and then converge according to the alarm The control relationship between the alarm level and the alarm delay in the rule, the alarm information is reported to the alarm platform after the delay is set; wherein, the lower the alarm level is, the longer the corresponding alarm delay is .

In the third aspect, an embodiment of the present invention provides a computing device, including:

Memory, used to store program instructions;

The processor is configured to call the program instructions stored in the memory, and execute the method according to any one of the first aspects according to the obtained program.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute any of the operations described in the first aspect method.

Description of the drawings

In order to explain the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative labor.

Figure 1 is a monitoring architecture diagram of CephFS by Prometheus in the prior art;

Figure 2 is a method for monitoring a distributed storage system provided by the present invention;

Figure 3 is a diagram of the monitoring architecture of Prometheus for CephFS provided by the present invention;

Figure 4 is a device for monitoring a distributed storage system provided by the present invention;

Fig. 5 is a schematic diagram of a computing device provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

As shown in FIG. 2, it is a method for monitoring a distributed storage system provided by an embodiment of the present invention, and the method includes:

Step 201: The monitoring server sends a collection instruction to each cluster in the distributed storage system.

Step 202: The monitoring server obtains monitoring data fed back by each cluster based on the collection instruction, the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster.

Step 203: For at least one cluster, the monitoring server determines alarm information from the monitoring data of the cluster according to a preset alarm rule, and reports the alarm information to an alarm platform.

In the above step 201, the monitoring server sends collection instructions to each cluster in the distributed storage system.

Suppose that a distributed storage system such as CephFS (Ceph File System, Ceph file system) has multiple clusters, such as 3, which are Ceph file system clusters numbered A, Ceph file system clusters numbered B, and number C Ceph file system cluster; as a monitoring server for CephFS, Prometheus (Prometheus), through its internal Prometheus Sever (Prometheus server) to issue collection instructions to CephFS, the specific performance is Prometheus Sever to A number Prometheus Sever issues collection instruction I to the Ceph file system cluster numbered B, Prometheus Sever issues collection instruction I to the Ceph file system cluster number C.

In the above step 202, the monitoring server obtains the monitoring data fed back by the clusters based on the collection instruction, and the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster.

When Prometheus Sever issues the collection command I to the Ceph file system cluster numbered A, the Ceph file system cluster number A will respond to the collection command I and get the monitoring data of the Ceph file system cluster number A. This Prometheus Sever obtains the monitoring data on the Ceph file system cluster with the A number; in the same way, Prometheus Sever can obtain the monitoring data on the Ceph file system cluster with the B number and the monitoring data on the Ceph file system cluster with the C number. .

Regarding the monitoring data of the A-numbered Ceph file system cluster, it can be specifically expressed as the health data of the A-numbered Ceph file system cluster itself (such as the operating status of the OSD component, the status data of the PG), and the Ceph file system cluster with the A number The status data of the user space file system client of the connected A-numbered Ceph file system (such as whether the user space file system client of the A-numbered Ceph file system is connected to the A-numbered Ceph file system). For example, there are 100 user space file system clients of the A-numbered Ceph file system connected to the A-numbered Ceph file system cluster, and the monitoring data components of the A-numbered Ceph file system cluster include the A-numbered Ceph file system cluster itself The health data also includes the status data of the user space file system client of the 100 A-numbered Ceph file system connected to the A-numbered Ceph file system cluster; the monitoring data about the B-numbered Ceph file system cluster and the C-number The monitoring data of the Ceph file system cluster can refer to the monitoring data of the Ceph file system cluster with the A number, which will not be repeated here.

In the above step 203, for at least one cluster, the monitoring server determines alarm information from the monitoring data of the cluster according to a preset alarm rule, and reports the alarm information to the alarm platform.

Suppose that for the Ceph file system cluster with A number, Prometheus analyzes the monitoring data obtained from the Ceph file system cluster with A number, and the analysis is based on the preset alarm rules, so as to determine the Ceph file system with A number The alarm information of the cluster; further, Prometheus will obtain the alarm information about the Ceph file system cluster with the A number and report it to the alarm platform, and the report is still based on the preset alarm rules. The alarm platform may be an IMS system (Information Management System, information management system), or other alarm platforms, which is not limited in the present invention. In the same way, the alarm process of Prometheus for the Ceph file system cluster of B number and Ceph file system cluster of C number can refer to the alarm process of Ceph file system cluster of A number, which will not be repeated here.

As shown in FIG. 3, it is a diagram of a Prometheus monitoring architecture for CephFS provided by an embodiment of the present invention. Referring to Figure 3, two monitoring servers are deployed, namely the X-numbered Prometheus server and the Y-numbered Prometheus server, the X-numbered Prometheus server and the Y-numbered Prometheus server are both It is used to monitor distributed storage systems. The system has deployed A-numbered Ceph file system clusters, B-numbered Ceph file system clusters, and C-numbered Ceph file system clusters; for A-numbered Ceph file system clusters, the cluster includes For the convenience of description, the Ceph file system cluster with A number includes 4 node servers, which are the node server numbered A1, the node server numbered A2, the node server numbered A3, and the node server numbered A4. Node server; similarly, for the B-numbered Ceph file system cluster, the cluster includes multiple node servers. For the convenience of description, the B-numbered Ceph file system cluster includes 4 node servers, which are respectively numbered B1 The node server numbered B2, the node server numbered B3, and the node server numbered B4; similarly, for the Ceph file system cluster numbered C, the cluster includes multiple node servers. For the convenience of description, set The Ceph file system cluster with number C includes 4 node servers, which are the node server numbered C1, the node server numbered C2, the node server numbered C3, and the node server numbered C4.

For the A-numbered Ceph file system cluster, a user space file system client with 100 A-numbered Ceph file system is connected to the node server configured with MDS components in the cluster, and the A-numbered Ceph file system cluster is set If there are 3 node servers configured with MDS components, the user space file system clients of these 100 A-numbered Ceph file systems are all connected to these 3 node servers configured with MDS components (not shown in the figure); Similarly, for a B-numbered Ceph file system cluster, a user space file system client with 200 B-numbered Ceph file systems is connected to a node server configured with MDS components in the cluster. Suppose there are 3 in the CephFS_B cluster. If each node server is configured with MDS components, the user space file system clients of the 200 B-numbered Ceph file systems are all connected to these three node servers (not shown in the figure) configured with MDS components; the same is true For a Ceph file system cluster with a C number, a user space file system client with 300 C-number Ceph file systems is connected to a node server configured with MDS components in the cluster, and a Ceph file system cluster with a C number is set There are 3 node servers configured with MDS components, then the user space file system clients of the 300 C-numbered Ceph file systems are all connected to these 3 node servers configured with MDS components (not shown in the figure) .

Suppose that for the X-numbered Prometheus server, the monitoring server sends at least two nodes in any of the above-mentioned A-numbered Ceph file system cluster, B-numbered Ceph file system cluster, and C-numbered Ceph file system cluster. The server issues collection instructions, specifically as follows:

Set at 8:00 am, the X-numbered Prometheus server sends the A1 numbered node server, A2 numbered node server, and A4 numbered node server in the A-numbered Ceph file system cluster to the three node servers Issue collection instruction I; at the same time, the X-numbered Prometheus server delivers to the three node servers of B1 numbered node server, B3 numbered node server, and B4 numbered node server in the B-numbered Ceph file system cluster Acquisition instruction I; At the same time, the X-numbered Prometheus server issues acquisition instructions to the C-numbered Ceph file system cluster C1 numbered node server, C2 numbered node server, and C4 numbered node server. I.

It should be noted that when the X-numbered Prometheus server sends collection commands to at least two node servers in the A-numbered Ceph file system cluster, it is randomly sent to any of the A-numbered Ceph file system clusters. At least two node servers issue collection instructions. For example, the X-numbered Prometheus server can be distributed to the three node servers of the A1 numbered node server, the A2 numbered node server, and the A4 numbered node server in the A-numbered Ceph file system cluster. Instruction I can also be issued to the three node servers of A2 numbered node server, A3 numbered node server and A4 numbered node server in the A numbered Ceph file system cluster, or it can be numbered A The three node servers of the A1 numbered node server, the A2 numbered node server, and the A3 numbered node server in the Ceph file system cluster in the Ceph file system cluster issue a collection instruction I, which is not limited by the present invention. Similarly, when the X-numbered Prometheus server sends collection instructions to at least two node servers in the B-numbered Ceph file system cluster, it randomly sends at least two of the B-numbered Ceph file system clusters. Node servers issue collection instructions; similarly, when the Prometheus server with X number sends collection instructions to at least two node servers in the Ceph file system cluster with C number, it sends collection instructions to Ceph with C number in a random manner. Any at least two node servers in the file system cluster issue collection instructions.

As a possible implementation manner, the alarm rule includes an alarm generation rule; the monitoring server determines the alarm information from the monitoring data according to a preset alarm rule, including: the monitoring server determines the alarm information from the monitoring data Determine the first client whose connection status with the cluster has changed; the monitoring server determines the second client whose connection status with the cluster has changed according to the service change of the cluster; The client in one client but not included in the second client and the alarm generation rule generate the alarm information of the client.

For example, for the CephFS_A cluster, for the convenience of description, there are 10 Ceph Fuse_A clients connected to the cluster: W1, W2, W3, W4, W5, W6, W7, W8, W9, and W10. The node server of the MDS component; Prometheus Sever_X issued the collection command I to the three node servers A1, A2, and A4 in the CephFS_A cluster. Suppose Prometheus Sever_X first obtains the monitoring data on the A1 node server. Analysis of the monitoring data of, it is determined that the 10 Ceph Fuse_A clients, W1, W2, W3, W4, W5, W6, W7, W8, W9, and W10 are all connected to the CephFS_A cluster; then, Prometheus Sever_X then obtains The monitoring data on the A2 node server, through the analysis of the monitoring data on the A2 node server, it is determined that only three Ceph Fuse_A clients, W8, W9, and W10, are still connected to the CephFS_A cluster, while W1, W2, W3, The 7 Ceph Fuse_A clients W4, W5, W6 and W7 have been offline from the CephFS_A cluster. That is, the first clients whose connection status with the cluster changes are W1, W2, W3, W4, W5, W6, and W7, respectively, seven Ceph Fuse_A clients.

For this abnormal event in the Ceph Fuse_A client, it is necessary to further determine the reason why the seven Ceph Fuse_A clients W1, W2, W3, W4, W5, W6 and W7 are offline from the CephFS_A cluster, that is, the Ceph Fuse_A client Is it uninstalled from the CephFS_A cluster normally, or is the Ceph Fuse_A client passively uninstalled due to the CephFS_A cluster itself?

For the business running on the CephFS_A cluster, for the purpose of business needs, some clients connected to the CephFS_A cluster will be uninstalled daily. For example, for the purpose of business needs, business personnel will uninstall the three Ceph Fuse_A clients, W5, W6, and W7 in the CephFS_A cluster. That is, the second clients whose connection status with the cluster have changed are three Ceph Fuse_A clients, W5, W6, and W7.

Through the first client (there are 7 Ceph Fuse_A clients W1, W2, W3, W4, W5, W6, and W7) and the second client (3 Ceph Fuse_A clients W5, W6, and W7) By comparison, it can be found that the uninstallation of the three Ceph Fuse_A clients, W5, W6, and W7, is a normal uninstall event of the Ceph Fuse_A client. Therefore, the offline of the three Ceph Fuse_A clients of W5, W6 and W7 in the monitoring data does not need to be reported to IMS system; and the uninstallation of the four Ceph Fuse_A clients, W1, W2, W3, and W4, belongs to the abnormal uninstallation event of the Ceph Fuse_A client, and the alarm information of the client is generated according to the alarm generation rules.

As a possible implementation, the alarm rules also include alarm suppression rules; the monitoring server determines the change duration of the cluster's business changes; the monitoring server sets the alarm suppression rules for the alarm information of the client, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.

As in the previous example, for the purpose of business needs, the three Ceph Fuse_A clients connected to the CephFS_A cluster, W5, W6, and W7, are normally uninstalled. Suppose the three Ceph Fuse_A, W5, W6, and W7, are uninstalled. The time required by the client is 3h, then Prometheus Sever_X will not connect to W5, W6, and W7 on the CephFS_A cluster for the entire time period of 3h in the future after Prometheus Sever_X obtains the monitoring data on the A2 node server. The offline events of the three Ceph Fuse_A clients are reported to the IMS system. That is, Prometheus Sever_X writes the offline events of the three Ceph Fuse_A clients W5, W6 and W7 from the CephFS_A cluster into the alarm suppression rules.

As in the foregoing example, Prometheus Sever_X's monitoring data for the CephFS_A cluster includes the health data of the CephFS_A cluster itself (such as the operating status of the OSD component, the status data of the PG), and the status data of the Ceph Fuse_A client connected to the CephFS_A cluster (such as Whether the Ceph Fuse_A client is connected to the CephFS_A cluster). Suppose at time T, Prometheus Sever_X obtained monitoring data about the CephFS_A cluster. This monitoring data showed that the MDS component in the CephFS_A cluster was abnormal during operation. At the same time, the CephFuse_A client W1 connected to the CephFS_A cluster was also If an abnormal uninstall event occurs, Prometheus Sever_X defines the alarm level of the abnormal event that occurs during the operation of the MDS component in the CephFS_A cluster as high, and defines the alarm level of the abnormal uninstall event that occurs on the 1 Ceph Fuse_A client W1 as low. Level; then Prometheus Sever_X will report high-level alarm events to the IMS system, that is, Prometheus Sever_X will report the abnormal events that occur during the operation of the MDS component in the CephFS_A cluster to the IMS system, instead of the W1 Ceph Fuse_A The abnormal uninstall event that occurs on the client is reported to the IMS system.

It should be noted that the reason why the monitoring server can set the alarm level of the alarm information of the MDS component in the cluster to be higher than the alarm level of the alarm information of the client is that the abnormality of the MDS component in the cluster will cause a problem with the cluster. The abnormal event of the connected client, therefore, after reporting the alarm information of the MDS component in the cluster to the IMS system, and after the operation and maintenance personnel conduct the operation and maintenance investigation, not only can the MDS component be restored to the normal operating state, but also can be connected to the cluster. The connected client also returns to its normal state.

As in the foregoing example, referring to Figure 3, Prometheus Sever_X sends collection instructions I to the three node servers A1, A2, and A4 in the CephFS_A cluster, and sends collection instructions to the three node servers B1, B3, and B4 in the CephFS_B cluster. I, and send collection command I to the three node servers C1, C2, and C4 in the CephFS_C cluster at the same time; when the collection command I is responded to in the above three clusters of CephFS_A cluster, CephFS_B cluster and CephFS_C cluster, Prometheus Sever_X will The monitoring data of each of the above-mentioned clusters will be obtained. Among them, the monitoring data can be expressed as the cluster identifier. For example, the first one obtained by Prometheus Sever_X is the monitoring data on the A1 node server of the CephFS_A cluster, the second is the monitoring data on the B3 node server of the CephFS_B cluster, and the third is It is the monitoring data on the C4 node server of the CephFS_C cluster, and so on.

As a possible implementation method, the alarm rule further includes an alarm convergence rule; the monitoring server reports the alarm information to the alarm platform according to a preset alarm rule, including: the monitoring server determines the alarm information Is the same alarm information that does not appear for the first time in the cluster, then according to the comparison relationship between the alarm level in the alarm convergence rule and the alarm delay, the alarm information is reported to the alarm platform after the delay is set; Among them, the lower the alarm level, the longer the corresponding alarm delay.

As in the foregoing example, suppose that the first piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_A cluster. After analyzing the first piece of monitoring data according to the preset alarm rules, it is determined that the first piece of monitoring data can be reported as alarm information. To the IMS system, the alarm information generated according to the first piece of monitoring data is set to Info_1, and the alarm level of Infro_1 is set to level 1. Assume that the sixth piece of monitoring data obtained by Prometheus Sever_X is still related to the CephFS_A cluster After analyzing the sixth monitoring data according to the preset alarm rules, it is found that the alarm information generated according to the sixth monitoring data conforms to Info_1, then Prometheus Sever_X needs to further determine when to use the alarm level of Infro_1. The sixth monitoring data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 1 is 1h, then Prometheus Sever_X will not report the Infro_1 corresponding to the sixth monitoring data within the next 1h. Report to the IMS system.

Suppose that the second piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_B cluster. After analyzing the second piece of monitoring data according to the preset alarm rules, it is determined that the second piece of monitoring data can be reported to the IMS system as alarm information. The alarm information order generated according to the second monitoring data is Info_2, and the alarm level of Infro_2 is set to level 2. Assume that the ninth monitoring data obtained by Prometheus Sever_X is still related to the CephFS_B cluster, according to the preset After analyzing the alarm rules and the monitoring data of Article 9, it is found that the alarm information generated according to the monitoring data of Article 9 conforms to Info_2, then Prometheus Sever_X needs to further determine when to monitor the ninth item according to the alarm level of Infro_2. The data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 2 is set to 2h, then Prometheus Sever_X will not report the Infro_2 corresponding to the ninth monitoring data to the IMS system within the next 2h.

Suppose that the third piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_C cluster. After analyzing the third piece of monitoring data according to the preset alarm rules, it is determined that the third piece of monitoring data can be reported to the IMS system as alarm information. The alarm information order generated according to the third monitoring data is Info_3, and the alarm level of Infro_3 is set to level 3; suppose that the tenth monitoring data obtained by Prometheus Sever_X is still related to the CephFS_C cluster, according to the preset After analyzing the alarm rules and the monitoring data of Article 10, it is found that the alarm information generated according to the monitoring data of Article 10 conforms to Info_3. Then Prometheus Sever_X needs to further determine when to monitor the monitoring data of Article 10 according to the alarm level of Infro_3. The data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 3 is set to 3h, then Prometheus Sever_X will not report the Infro_3 corresponding to the tenth monitoring data to the IMS system in the next 3h.

It should be noted that, in the above example, as the alarm levels of level 1, level 2, and level 3 decrease, the delay of the corresponding alarm delay is longer, corresponding to 1h, 2h, and 3h, respectively.

Based on this solution, after the monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in a certain cluster, it reports the same alarm that does not appear for the first time to the alarm platform according to the alarm convergence rules and after a set delay. The cluster continues to send out the same alarm repeatedly, resulting in a waste of resources.

Based on the same concept, an embodiment of the present invention also provides a device for monitoring a distributed storage system. As shown in FIG. 4, the device includes:

The sending unit 401 is configured to send collection instructions to each cluster in the distributed storage system;

The obtaining unit 402 is configured to obtain monitoring data fed back by each cluster based on the collection instruction, the monitoring data including the health data of the cluster itself and the status data of the client connected to the cluster;

The determining unit 403, for at least one cluster, is configured to determine alarm information from the monitoring data of the cluster according to preset alarm rules and report the alarm information to the alarm platform.

Further, for the device, there are multiple monitoring servers; any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; for any monitoring server, The sending unit 401 is specifically configured to issue collection instructions to at least two node servers in any cluster.

Further, for the device, the alarm rule includes an alarm generation rule; the determining unit 403 is specifically configured to determine from the monitoring data the first client whose connection status with the cluster has changed; according to The service change of the cluster determines the second client whose connection state with the cluster has changed; according to the client included in the first client but not included in the second client and the alarm Generate rules to generate alarm information for the client.

Further, for the device, the alarm rule also includes an alarm suppression rule; the determining unit 403 is specifically configured to determine the change duration of the service change of the cluster; set the alarm suppression rule for the alarm information of the client, The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.

Further, for the device, the monitoring server generates the alarm information of the MDS component of the cluster according to the health data of the cluster itself; the determining unit 403 is specifically configured to determine the alarm of the alarm information of the MDS component If the level is higher than the alarm information of the client, the alarm information of the MDS component is reported to the alarm platform.

Further, for the device, after the monitoring server obtains the monitoring data fed back by each cluster based on the collection instruction, the determining unit 403 is further configured to set a cluster identifier corresponding to each monitoring data.

Further, for the device, the alarm rule also includes an alarm convergence rule; the determining unit 403 is specifically configured to determine that the alarm information is the same alarm information that does not appear for the first time in the cluster, and then according to the alarm The contrast relationship between the alarm level and the alarm delay in the convergence rule, the alarm information is reported to the alarm platform after the delay is set; wherein, the lower the alarm level is, the longer the corresponding alarm delay is long.

The embodiment of the present invention provides a computing device, and the computing device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), etc. The computing device may include a central processing unit (CPU), a memory, an input/output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc., and an output device may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), Cathode Ray Tube (CRT), etc.

The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiment of the present invention, the memory may be used to store the program instructions of the method for monitoring the distributed storage system;

The processor is configured to call the program instructions stored in the memory, and execute the method of monitoring the distributed storage system according to the obtained program.

As shown in FIG. 5, it is a schematic diagram of a computing device provided by an embodiment of this application, and the computing device includes:

A processor 501, a memory 502, a transceiver 503, and a bus interface 504; among them, the processor 501, the memory 502, and the transceiver 503 are connected by a bus 505;

The processor 501 is configured to read a program in the memory 502, and execute the foregoing method for monitoring a distributed storage system;

The processor 501 may be a central processing unit (central processing unit, CPU for short), a network processor (NP for short), or a combination of CPU and NP. It can also be a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC for short), a programmable logic device (PLD for short), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (generic array logic, GAL), or any of them combination.

The memory 502 is configured to store one or more executable programs, and can store data used by the processor 501 when performing operations.

Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 502 may include a volatile memory (volatile memory), such as random-access memory (RAM for short); the memory 502 may also include a non-volatile memory (non-volatile memory), such as flash memory ( flash memory), hard disk drive (HDD for short) or solid-state drive (SSD for short); the memory 502 may also include a combination of the foregoing types of memories.

The memory 502 stores the following elements, executable modules or data structures, or their subsets, or their extended sets:

Operating instructions: including various operating instructions, used to implement various operations.

Operating system: Including various system programs, used to implement various basic services and process hardware-based tasks.

The bus 505 may be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used to represent in FIG. 5, but it does not mean that there is only one bus or one type of bus.

The bus interface 504 may be a wired communication access port, a wireless bus interface or a combination thereof, where the wired bus interface may be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface may be a WLAN interface.

The embodiment of the present invention provides a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute a method for monitoring a distributed storage system.

Those skilled in the art should understand that the embodiments of the present invention can be provided as a method or a computer program product. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are used to generate It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

A method for monitoring a distributed storage system is characterized in that it includes:

The monitoring server sends collection instructions to each cluster in the distributed storage system;

The monitoring server obtains monitoring data fed back by each cluster based on the collection instruction, the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster;

For at least one cluster, the monitoring server determines alarm information from the monitoring data of the cluster according to preset alarm rules and reports the alarm information to the alarm platform.
The method according to claim 1, wherein there are multiple monitoring servers; any cluster includes multiple node servers, and each node server connected to the client is connected to the same client;

The monitoring server sending collection instructions to each cluster in the distributed storage system includes:

For any monitoring server, the monitoring server issues collection instructions to at least two node servers in any cluster.
The method according to claim 1, wherein the alarm rule comprises an alarm generation rule;

The monitoring server determines alarm information from the monitoring data according to preset alarm rules, including:

The monitoring server determines from the monitoring data the first client whose connection status with the cluster has changed;

Determining, by the monitoring server, a second client whose connection status with the cluster has changed according to the business change of the cluster;

The alarm information of the client is generated according to the client included in the first client but not included in the second client and the alarm generation rule.
The method of claim 3, wherein the alarm rule further comprises an alarm suppression rule;

The monitoring server determines the change duration of the service change of the cluster;

The monitoring server sets an alarm suppression rule for the alarm information of the client, and the alarm suppression rule of the client is used to not report the alarm information of the client generated within the change period.
The method according to claim 3, wherein the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself;

The monitoring server reports the alarm information to the alarm platform according to preset alarm rules, including:

The monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and then reports the alarm information of the MDS component to the alarm platform.
The method according to claim 1, wherein after the monitoring server obtains the monitoring data fed back by the clusters based on the collection instruction, the method further comprises:

The monitoring server sets a cluster identifier corresponding to each monitoring data.
The method according to any one of claims 1-6, wherein the alarm rule further comprises an alarm convergence rule;

The monitoring server reports the alarm information to the alarm platform according to preset alarm rules, including:

The monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in the cluster, and then, according to the comparison relationship between the alarm level and the alarm delay in the alarm convergence rule, the The alarm information is reported to the alarm platform; wherein, the lower the alarm level, the longer the corresponding alarm delay.
A device for monitoring a distributed storage system is characterized in that it comprises:

A sending unit, configured to send collection instructions to each cluster in the distributed storage system;

An obtaining unit, configured to obtain monitoring data fed back by each cluster based on the collection instruction, the monitoring data including the health data of the cluster itself and the status data of the client connected to the cluster;

The determining unit, for at least one cluster, is configured to determine alarm information from the monitoring data of the cluster according to preset alarm rules and report the alarm information to the alarm platform.
A computing device, characterized in that it comprises:

Memory, used to store program instructions;

The processor is configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1-7 according to the obtained program.
A computer-readable storage medium, wherein the storage medium stores computer-executable instructions, and the computer-executable instructions are used to make a computer execute the method according to any one of claims 1-7.