WO2021129367A1 - 一种监控分布式存储系统的方法及装置 - Google Patents

一种监控分布式存储系统的方法及装置 Download PDF

Info

Publication number
WO2021129367A1
WO2021129367A1 PCT/CN2020/134339 CN2020134339W WO2021129367A1 WO 2021129367 A1 WO2021129367 A1 WO 2021129367A1 CN 2020134339 W CN2020134339 W CN 2020134339W WO 2021129367 A1 WO2021129367 A1 WO 2021129367A1
Authority
WO
WIPO (PCT)
Prior art keywords
cluster
alarm
monitoring
client
alarm information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/134339
Other languages
English (en)
French (fr)
Inventor
龚治文
饶俊明
卢道和
郑晓腾
龚洵峰
刘生庆
吴立
吴传民
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WeBank Co Ltd
Original Assignee
WeBank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by WeBank Co Ltd filed Critical WeBank Co Ltd
Publication of WO2021129367A1 publication Critical patent/WO2021129367A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
    • H04L41/0609—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time based on severity or priority
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/06—Generation of reports
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present invention relates to the field of financial technology (Fintech), in particular to a method and device for monitoring a distributed storage system.
  • CephFS Ceph File System
  • the Ceph Fuse client user space file system client of the Ceph file system
  • CephFS open source Prometheus
  • Prometheus is mainly composed of Exporter (client for Prometheus monitoring data collection) and Prometheus Sever (server for Prometheus monitoring); CephFS is mainly composed of monitor (Monitor, abbreviated as MON), target storage device (Object Storage Device, It is abbreviated as OSD) and metadata server (MetaData Sever, abbreviated as MDS) and other components.
  • the CephFS OSD component also has placement groups (Placement Groups, abbreviated as PG).
  • Prometheus's monitoring of CephFS is mainly manifested in Prometheus's data collection of CephFS OSD component status and CephFS PG status, but Prometheus does not implement the monitoring of Ceph Fuse client.
  • Prometheus's monitoring architecture for CephFS is very bloated, which is manifested in the need to deploy a set of Prometheus for each CephFS; in addition, due to the different versions of CephFS, different Exporters need to be deployed for different versions of CephFS.
  • Figure 1 it is a diagram of the monitoring architecture of CephFS by Prometheus in the prior art.
  • the M-numbered Prometheus monitoring data collection client collects the M-numbered Ceph file system monitoring data.
  • the collected monitoring data meets the rules for generating alarm information, it will report the generated alarm information to The M-numbered Prometheus server, in the same way, the N-numbered Prometheus monitoring data collection client collects the monitoring data of the N-numbered Ceph file system. If the collected monitoring data meets the rules for generating alarm information, The generated alarm information is reported to the N-numbered Prometheus server; however, the M-numbered Prometheus monitoring data collection client does not match the N-numbered Ceph file system version, so the M-numbered The client of Prometheus monitoring data collection is used to collect the monitoring data of the N-numbered Ceph file system to report the alarm information of the N-numbered Ceph file system. That is, Prometheus Sever, Exporter, and CephFS did not achieve high availability among the three, resulting in failure to report monitoring information in a timely manner under abnormal conditions.
  • the existing technology has problems that Prometheus cannot monitor the Ceph Fuse client and Prometheus has low monitoring efficiency for CephFS.
  • the present invention provides a method and device for monitoring a distributed storage system, which are used to solve the problems that Prometheus cannot monitor Ceph Fuse clients and Prometheus has low monitoring efficiency for CephFS.
  • an embodiment of the present invention provides a method for monitoring a distributed storage system.
  • the method includes: a monitoring server sends collection instructions to each cluster in the distributed storage system; and the monitoring server obtains that each cluster is based on
  • the monitoring data fed back by the collection instruction includes the health data of the cluster itself and the status data of the client connected to the cluster; for at least one cluster, the monitoring server obtains data from the cluster according to preset alarm rules. Determine the alarm information in the monitoring data and report the alarm information to the alarm platform.
  • the monitoring server can monitor multiple clusters at the same time by issuing collection instructions to each cluster in the distributed storage system, thus avoiding the ineffectiveness of the monitoring server when the cluster and the Exporter version do not match.
  • the purpose of the monitoring server to monitor the clients connected to the cluster is realized.
  • any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; the monitoring server is distributed to the Each cluster in the storage system sends collection instructions, including: for any monitoring server, the monitoring server issues collection instructions to at least two node servers in any cluster.
  • the monitoring server sends collection instructions to at least two node servers in each cluster to help ensure that the monitoring server is down when one of the node servers is down.
  • the monitoring data of the cluster where the node server is located can also be obtained from other available node servers, so as to realize the effective monitoring of each cluster by the monitoring server.
  • the alarm rule includes an alarm generation rule; the monitoring server determines the alarm information from the monitoring data according to the preset alarm rule, including: the monitoring server obtains the alarm information from the monitoring data Determine the first client whose connection status with the cluster has changed; the monitoring server determines the second client whose connection status with the cluster has changed according to the service change of the cluster; The client in one client but not included in the second client and the alarm generation rule generate the alarm information of the client.
  • the first client whose connection status with the cluster has changed is determined, and through the analysis of known business changes, it is determined that the connection status with the cluster has changed.
  • the second client by comparing the first client with the second client, can generate alarm information generated due to the abnormality of the client.
  • the alarm rule further includes an alarm suppression rule; the monitoring server determines the change duration of the service change of the cluster; the monitoring server sets the alarm suppression rule for the alarm information of the client, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.
  • the monitoring server will not report the alarm information of the client to the alarm platform during this necessary time, which can be effective To avoid generating known but useless alarms.
  • the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself; the monitoring server reports the alarm information to the alarm according to a preset alarm rule
  • the platform includes: the monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and then reports the alarm information of the MDS component to the alarm platform.
  • the monitoring server when the monitoring server simultaneously obtains the alarm information of the MDS component of the cluster and the alarm information of the client connected to the cluster, it is considered that the abnormal event of the client connected to the cluster may be caused by the abnormality of the MDS component of the cluster.
  • the monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and reports the alarm information of the MDS component to the alarm platform, automatically shielding the alarm information of the low-level client.
  • the method further includes: the monitoring server sets a cluster identifier corresponding to each monitoring data.
  • the monitoring server marks each acquired monitoring data with the corresponding cluster identification, which helps the monitoring server to quickly make corresponding alarm operations when receiving the same monitoring data of the same cluster in the future.
  • the alarm rules further include alarm convergence rules; the monitoring server reports the alarm information to the alarm platform according to preset alarm rules, including: the monitoring server determines the alarm information Is the same alarm information that does not appear for the first time in the cluster, then according to the comparison relationship between the alarm level in the alarm convergence rule and the alarm delay, the alarm information is reported to the alarm platform after the delay is set; Among them, the lower the alarm level, the longer the corresponding alarm delay.
  • the monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in a certain cluster, it reports the same alarm that does not appear for the first time to the alarm platform according to the alarm convergence rules and after a set time delay, which can effectively prevent The cluster continues to send out the same alarm repeatedly, resulting in a waste of resources.
  • an embodiment of the present invention provides a device for monitoring a distributed storage system, the device includes: a sending unit, configured to send collection instructions to each cluster in the distributed storage system; The monitoring data of each cluster is based on the collection instruction feedback, and the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster; the determining unit, for at least one cluster, is used for according to preset alarm rules, Determine alarm information from the monitoring data of the cluster and report the alarm information to the alarm platform.
  • the monitoring server can monitor multiple clusters at the same time by issuing collection instructions to each cluster in the distributed storage system, thus avoiding the ineffectiveness of the monitoring server when the cluster and the Exporter version do not match.
  • the purpose of the monitoring server to monitor the clients connected to the cluster is realized.
  • any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; for any monitoring server, all The sending unit is specifically configured to issue collection instructions to at least two node servers in any cluster.
  • the monitoring server sends collection instructions to at least two node servers in each cluster to help ensure that the monitoring server is down when one of the node servers is down.
  • the monitoring data of the cluster where the node server is located can also be obtained from other available node servers, so as to realize the effective monitoring of each cluster by the monitoring server.
  • the alarm rule includes an alarm generation rule; the determining unit is specifically configured to determine from the monitoring data the first client whose connection status with the cluster has changed; The service change of the cluster determines the second client whose connection state with the cluster has changed; and the alarm is generated according to the client included in the first client but not included in the second client and the alarm Rules to generate alarm information for the client.
  • the first client whose connection status with the cluster has changed is determined, and through the analysis of known business changes, it is determined that the connection status with the cluster has changed.
  • the second client by comparing the first client with the second client, can generate alarm information generated due to the abnormality of the client.
  • the alarm rule further includes an alarm suppression rule; the determining unit is specifically configured to determine the change duration of the service change of the cluster; and the alarm suppression rule for the alarm information of the client is set, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.
  • the monitoring server will not report the alarm information of the client to the alarm platform during this necessary time, which can be effective To avoid generating known but useless alarms.
  • the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself; the determining unit is specifically configured to determine the alarm level of the alarm information of the MDS component If the alarm information is higher than the alarm information of the client, the alarm information of the MDS component is reported to the alarm platform.
  • the monitoring server when the monitoring server simultaneously obtains the alarm information of the MDS component of the cluster and the alarm information of the client connected to the cluster, it is considered that the abnormal event of the client connected to the cluster may be caused by the abnormality of the MDS component of the cluster.
  • the monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and reports the alarm information of the MDS component to the alarm platform, automatically shielding the alarm information of the low-level client.
  • the determining unit is further configured to set a cluster identification corresponding to each monitoring data.
  • the monitoring server marks each acquired monitoring data with its corresponding cluster identification, which helps the monitoring server to quickly make corresponding alarm operations when receiving the same monitoring data of the same cluster in the future.
  • the alarm rule further includes an alarm convergence rule; the determining unit is specifically configured to determine that the alarm information is the same alarm information that does not appear for the first time in the cluster, and then converge according to the alarm The control relationship between the alarm level and the alarm delay in the rule, the alarm information is reported to the alarm platform after the delay is set; wherein, the lower the alarm level is, the longer the corresponding alarm delay is .
  • the monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in a certain cluster, it reports the same alarm that does not appear for the first time to the alarm platform according to the alarm convergence rules and after a set time delay, which can effectively prevent The cluster continues to send out the same alarm repeatedly, resulting in a waste of resources.
  • an embodiment of the present invention provides a computing device, including:
  • Memory used to store program instructions
  • the processor is configured to call the program instructions stored in the memory, and execute the method according to any one of the first aspects according to the obtained program.
  • an embodiment of the present invention provides a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute any of the operations described in the first aspect method.
  • Figure 1 is a monitoring architecture diagram of CephFS by Prometheus in the prior art
  • Figure 2 is a method for monitoring a distributed storage system provided by the present invention
  • Figure 3 is a diagram of the monitoring architecture of Prometheus for CephFS provided by the present invention.
  • Figure 4 is a device for monitoring a distributed storage system provided by the present invention.
  • Fig. 5 is a schematic diagram of a computing device provided by the present invention.
  • FIG. 2 it is a method for monitoring a distributed storage system provided by an embodiment of the present invention, and the method includes:
  • Step 201 The monitoring server sends a collection instruction to each cluster in the distributed storage system.
  • Step 202 The monitoring server obtains monitoring data fed back by each cluster based on the collection instruction, the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster.
  • Step 203 For at least one cluster, the monitoring server determines alarm information from the monitoring data of the cluster according to a preset alarm rule, and reports the alarm information to an alarm platform.
  • the monitoring server can monitor multiple clusters at the same time by issuing collection instructions to each cluster in the distributed storage system, thus avoiding the ineffectiveness of the monitoring server when the cluster and the Exporter version do not match.
  • the purpose of the monitoring server to monitor the clients connected to the cluster is realized.
  • the monitoring server sends collection instructions to each cluster in the distributed storage system.
  • CephFS Ceph File System, Ceph file system
  • Ceph file system has multiple clusters, such as 3, which are Ceph file system clusters numbered A, Ceph file system clusters numbered B, and number C Ceph file system cluster
  • a monitoring server for CephFS Prometheus (Prometheus), through its internal Prometheus Sever (Prometheus server) to issue collection instructions to CephFS
  • the specific performance is Prometheus Sever to A number
  • Prometheus Sever issues collection instruction I to the Ceph file system cluster numbered B
  • Prometheus Sever issues collection instruction I to the Ceph file system cluster number C.
  • the monitoring server obtains the monitoring data fed back by the clusters based on the collection instruction, and the monitoring data includes the health data of the cluster itself and the status data of the client connected to the cluster.
  • Prometheus Sever issues the collection command I to the Ceph file system cluster numbered A
  • the Ceph file system cluster number A will respond to the collection command I and get the monitoring data of the Ceph file system cluster number A.
  • This Prometheus Sever obtains the monitoring data on the Ceph file system cluster with the A number; in the same way, Prometheus Sever can obtain the monitoring data on the Ceph file system cluster with the B number and the monitoring data on the Ceph file system cluster with the C number. .
  • the monitoring data of the A-numbered Ceph file system cluster can be specifically expressed as the health data of the A-numbered Ceph file system cluster itself (such as the operating status of the OSD component, the status data of the PG), and the Ceph file system cluster with the A number The status data of the user space file system client of the connected A-numbered Ceph file system (such as whether the user space file system client of the A-numbered Ceph file system is connected to the A-numbered Ceph file system).
  • the monitoring data components of the A-numbered Ceph file system cluster include the A-numbered Ceph file system cluster itself
  • the health data also includes the status data of the user space file system client of the 100 A-numbered Ceph file system connected to the A-numbered Ceph file system cluster; the monitoring data about the B-numbered Ceph file system cluster and the C-number
  • the monitoring data of the Ceph file system cluster can refer to the monitoring data of the Ceph file system cluster with the A number, which will not be repeated here.
  • the monitoring server determines alarm information from the monitoring data of the cluster according to a preset alarm rule, and reports the alarm information to the alarm platform.
  • Prometheus analyzes the monitoring data obtained from the Ceph file system cluster with A number, and the analysis is based on the preset alarm rules, so as to determine the Ceph file system with A number
  • the alarm information of the cluster further, Prometheus will obtain the alarm information about the Ceph file system cluster with the A number and report it to the alarm platform, and the report is still based on the preset alarm rules.
  • the alarm platform may be an IMS system (Information Management System, information management system), or other alarm platforms, which is not limited in the present invention.
  • the alarm process of Prometheus for the Ceph file system cluster of B number and Ceph file system cluster of C number can refer to the alarm process of Ceph file system cluster of A number, which will not be repeated here.
  • any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; the monitoring server is distributed to the Each cluster in the storage system sends collection instructions, including: for any monitoring server, the monitoring server issues collection instructions to at least two node servers in any cluster.
  • FIG. 3 it is a diagram of a Prometheus monitoring architecture for CephFS provided by an embodiment of the present invention.
  • two monitoring servers are deployed, namely the X-numbered Prometheus server and the Y-numbered Prometheus server, the X-numbered Prometheus server and the Y-numbered Prometheus server are both It is used to monitor distributed storage systems.
  • the system has deployed A-numbered Ceph file system clusters, B-numbered Ceph file system clusters, and C-numbered Ceph file system clusters; for A-numbered Ceph file system clusters, the cluster includes For the convenience of description, the Ceph file system cluster with A number includes 4 node servers, which are the node server numbered A1, the node server numbered A2, the node server numbered A3, and the node server numbered A4. Node server; similarly, for the B-numbered Ceph file system cluster, the cluster includes multiple node servers.
  • the B-numbered Ceph file system cluster includes 4 node servers, which are respectively numbered B1 The node server numbered B2, the node server numbered B3, and the node server numbered B4; similarly, for the Ceph file system cluster numbered C, the cluster includes multiple node servers.
  • set The Ceph file system cluster with number C includes 4 node servers, which are the node server numbered C1, the node server numbered C2, the node server numbered C3, and the node server numbered C4.
  • a user space file system client with 100 A-numbered Ceph file system is connected to the node server configured with MDS components in the cluster, and the A-numbered Ceph file system cluster is set If there are 3 node servers configured with MDS components, the user space file system clients of these 100 A-numbered Ceph file systems are all connected to these 3 node servers configured with MDS components (not shown in the figure); Similarly, for a B-numbered Ceph file system cluster, a user space file system client with 200 B-numbered Ceph file systems is connected to a node server configured with MDS components in the cluster. Suppose there are 3 in the CephFS_B cluster.
  • each node server is configured with MDS components
  • the user space file system clients of the 200 B-numbered Ceph file systems are all connected to these three node servers (not shown in the figure) configured with MDS components; the same is true
  • a Ceph file system cluster with a C number a user space file system client with 300 C-number Ceph file systems is connected to a node server configured with MDS components in the cluster, and a Ceph file system cluster with a C number is set
  • There are 3 node servers configured with MDS components then the user space file system clients of the 300 C-numbered Ceph file systems are all connected to these 3 node servers configured with MDS components (not shown in the figure) .
  • the monitoring server sends at least two nodes in any of the above-mentioned A-numbered Ceph file system cluster, B-numbered Ceph file system cluster, and C-numbered Ceph file system cluster.
  • the server issues collection instructions, specifically as follows:
  • the X-numbered Prometheus server sends the A1 numbered node server, A2 numbered node server, and A4 numbered node server in the A-numbered Ceph file system cluster to the three node servers Issue collection instruction I; at the same time, the X-numbered Prometheus server delivers to the three node servers of B1 numbered node server, B3 numbered node server, and B4 numbered node server in the B-numbered Ceph file system cluster Acquisition instruction I; At the same time, the X-numbered Prometheus server issues acquisition instructions to the C-numbered Ceph file system cluster C1 numbered node server, C2 numbered node server, and C4 numbered node server. I.
  • the X-numbered Prometheus server when the X-numbered Prometheus server sends collection commands to at least two node servers in the A-numbered Ceph file system cluster, it is randomly sent to any of the A-numbered Ceph file system clusters. At least two node servers issue collection instructions.
  • the X-numbered Prometheus server can be distributed to the three node servers of the A1 numbered node server, the A2 numbered node server, and the A4 numbered node server in the A-numbered Ceph file system cluster.
  • Instruction I can also be issued to the three node servers of A2 numbered node server, A3 numbered node server and A4 numbered node server in the A numbered Ceph file system cluster, or it can be numbered A
  • the three node servers of the A1 numbered node server, the A2 numbered node server, and the A3 numbered node server in the Ceph file system cluster in the Ceph file system cluster issue a collection instruction I, which is not limited by the present invention.
  • the X-numbered Prometheus server sends collection instructions to at least two node servers in the B-numbered Ceph file system cluster, it randomly sends at least two of the B-numbered Ceph file system clusters.
  • Node servers issue collection instructions; similarly, when the Prometheus server with X number sends collection instructions to at least two node servers in the Ceph file system cluster with C number, it sends collection instructions to Ceph with C number in a random manner. Any at least two node servers in the file system cluster issue collection instructions.
  • the alarm rule includes an alarm generation rule; the monitoring server determines the alarm information from the monitoring data according to a preset alarm rule, including: the monitoring server determines the alarm information from the monitoring data Determine the first client whose connection status with the cluster has changed; the monitoring server determines the second client whose connection status with the cluster has changed according to the service change of the cluster; The client in one client but not included in the second client and the alarm generation rule generate the alarm information of the client.
  • CephFS_A cluster for the convenience of description, there are 10 Ceph Fuse_A clients connected to the cluster: W1, W2, W3, W4, W5, W6, W7, W8, W9, and W10.
  • the node server of the MDS component Prometheus Sever_X issued the collection command I to the three node servers A1, A2, and A4 in the CephFS_A cluster.
  • Prometheus Sever_X first obtains the monitoring data on the A1 node server.
  • CephFS_A cluster For the business running on the CephFS_A cluster, for the purpose of business needs, some clients connected to the CephFS_A cluster will be uninstalled daily. For example, for the purpose of business needs, business personnel will uninstall the three Ceph Fuse_A clients, W5, W6, and W7 in the CephFS_A cluster. That is, the second clients whose connection status with the cluster have changed are three Ceph Fuse_A clients, W5, W6, and W7.
  • the offline of the three Ceph Fuse_A clients of W5, W6 and W7 in the monitoring data does not need to be reported to IMS system; and the uninstallation of the four Ceph Fuse_A clients, W1, W2, W3, and W4, belongs to the abnormal uninstallation event of the Ceph Fuse_A client, and the alarm information of the client is generated according to the alarm generation rules.
  • the alarm rules also include alarm suppression rules; the monitoring server determines the change duration of the cluster's business changes; the monitoring server sets the alarm suppression rules for the alarm information of the client, so The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.
  • the three Ceph Fuse_A clients connected to the CephFS_A cluster, W5, W6, and W7, are normally uninstalled.
  • the three Ceph Fuse_A, W5, W6, and W7, are uninstalled.
  • the time required by the client is 3h, then Prometheus Sever_X will not connect to W5, W6, and W7 on the CephFS_A cluster for the entire time period of 3h in the future after Prometheus Sever_X obtains the monitoring data on the A2 node server.
  • the offline events of the three Ceph Fuse_A clients are reported to the IMS system. That is, Prometheus Sever_X writes the offline events of the three Ceph Fuse_A clients W5, W6 and W7 from the CephFS_A cluster into the alarm suppression rules.
  • the monitoring server generates alarm information of the MDS component of the cluster according to the health data of the cluster itself; the monitoring server reports the alarm information to the alarm according to a preset alarm rule
  • the platform includes: the monitoring server determines that the alarm level of the alarm information of the MDS component is higher than the alarm information of the client, and then reports the alarm information of the MDS component to the alarm platform.
  • Prometheus Sever_X's monitoring data for the CephFS_A cluster includes the health data of the CephFS_A cluster itself (such as the operating status of the OSD component, the status data of the PG), and the status data of the Ceph Fuse_A client connected to the CephFS_A cluster (such as Whether the Ceph Fuse_A client is connected to the CephFS_A cluster).
  • the status data of the Ceph Fuse_A client connected to the CephFS_A cluster such as Whether the Ceph Fuse_A client is connected to the CephFS_A cluster.
  • Prometheus Sever_X defines the alarm level of the abnormal event that occurs during the operation of the MDS component in the CephFS_A cluster as high, and defines the alarm level of the abnormal uninstall event that occurs on the 1 Ceph Fuse_A client W1 as low.
  • Prometheus Sever_X will report high-level alarm events to the IMS system, that is, Prometheus Sever_X will report the abnormal events that occur during the operation of the MDS component in the CephFS_A cluster to the IMS system, instead of the W1 Ceph Fuse_A
  • the abnormal uninstall event that occurs on the client is reported to the IMS system.
  • the reason why the monitoring server can set the alarm level of the alarm information of the MDS component in the cluster to be higher than the alarm level of the alarm information of the client is that the abnormality of the MDS component in the cluster will cause a problem with the cluster.
  • the abnormal event of the connected client therefore, after reporting the alarm information of the MDS component in the cluster to the IMS system, and after the operation and maintenance personnel conduct the operation and maintenance investigation, not only can the MDS component be restored to the normal operating state, but also can be connected to the cluster.
  • the connected client also returns to its normal state.
  • the method further includes: the monitoring server sets a cluster identifier corresponding to each monitoring data.
  • Prometheus Sever_X sends collection instructions I to the three node servers A1, A2, and A4 in the CephFS_A cluster, and sends collection instructions to the three node servers B1, B3, and B4 in the CephFS_B cluster.
  • I and send collection command I to the three node servers C1, C2, and C4 in the CephFS_C cluster at the same time; when the collection command I is responded to in the above three clusters of CephFS_A cluster, CephFS_B cluster and CephFS_C cluster, Prometheus Sever_X will The monitoring data of each of the above-mentioned clusters will be obtained.
  • the monitoring data can be expressed as the cluster identifier.
  • the first one obtained by Prometheus Sever_X is the monitoring data on the A1 node server of the CephFS_A cluster
  • the second is the monitoring data on the B3 node server of the CephFS_B cluster
  • the third is It is the monitoring data on the C4 node server of the CephFS_C cluster, and so on.
  • the alarm rule further includes an alarm convergence rule;
  • the monitoring server reports the alarm information to the alarm platform according to a preset alarm rule, including: the monitoring server determines the alarm information Is the same alarm information that does not appear for the first time in the cluster, then according to the comparison relationship between the alarm level in the alarm convergence rule and the alarm delay, the alarm information is reported to the alarm platform after the delay is set; Among them, the lower the alarm level, the longer the corresponding alarm delay.
  • the first piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_A cluster. After analyzing the first piece of monitoring data according to the preset alarm rules, it is determined that the first piece of monitoring data can be reported as alarm information.
  • the alarm information generated according to the first piece of monitoring data is set to Info_1, and the alarm level of Infro_1 is set to level 1.
  • the sixth piece of monitoring data obtained by Prometheus Sever_X is still related to the CephFS_A cluster
  • the alarm information generated according to the sixth monitoring data conforms to Info_1
  • Prometheus Sever_X needs to further determine when to use the alarm level of Infro_1.
  • the sixth monitoring data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 1 is 1h, then Prometheus Sever_X will not report the Infro_1 corresponding to the sixth monitoring data within the next 1h. Report to the IMS system.
  • the second piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_B cluster. After analyzing the second piece of monitoring data according to the preset alarm rules, it is determined that the second piece of monitoring data can be reported to the IMS system as alarm information.
  • the alarm information order generated according to the second monitoring data is Info_2, and the alarm level of Infro_2 is set to level 2.
  • the ninth monitoring data obtained by Prometheus Sever_X is still related to the CephFS_B cluster, according to the preset
  • the alarm information generated according to the monitoring data of Article 9 conforms to Info_2
  • Prometheus Sever_X needs to further determine when to monitor the ninth item according to the alarm level of Infro_2.
  • the data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 2 is set to 2h, then Prometheus Sever_X will not report the Infro_2 corresponding to the ninth monitoring data to the IMS system within the next 2h.
  • the third piece of monitoring data obtained by Prometheus Sever_X comes from the CephFS_C cluster. After analyzing the third piece of monitoring data according to the preset alarm rules, it is determined that the third piece of monitoring data can be reported to the IMS system as alarm information.
  • the alarm information order generated according to the third monitoring data is Info_3, and the alarm level of Infro_3 is set to level 3; suppose that the tenth monitoring data obtained by Prometheus Sever_X is still related to the CephFS_C cluster, according to the preset After analyzing the alarm rules and the monitoring data of Article 10, it is found that the alarm information generated according to the monitoring data of Article 10 conforms to Info_3.
  • Prometheus Sever_X needs to further determine when to monitor the monitoring data of Article 10 according to the alarm level of Infro_3. The data is reported to the IMS system; if the alarm delay corresponding to the alarm information with the alarm level of level 3 is set to 3h, then Prometheus Sever_X will not report the Infro_3 corresponding to the tenth monitoring data to the IMS system in the next 3h.
  • the monitoring server determines that the alarm information is the same alarm information that does not appear for the first time in a certain cluster, it reports the same alarm that does not appear for the first time to the alarm platform according to the alarm convergence rules and after a set delay.
  • the cluster continues to send out the same alarm repeatedly, resulting in a waste of resources.
  • an embodiment of the present invention also provides a device for monitoring a distributed storage system. As shown in FIG. 4, the device includes:
  • the sending unit 401 is configured to send collection instructions to each cluster in the distributed storage system
  • the obtaining unit 402 is configured to obtain monitoring data fed back by each cluster based on the collection instruction, the monitoring data including the health data of the cluster itself and the status data of the client connected to the cluster;
  • the determining unit 403, for at least one cluster is configured to determine alarm information from the monitoring data of the cluster according to preset alarm rules and report the alarm information to the alarm platform.
  • any cluster includes multiple node servers, and each node server connected to the client is connected to the same client; for any monitoring server, The sending unit 401 is specifically configured to issue collection instructions to at least two node servers in any cluster.
  • the alarm rule includes an alarm generation rule;
  • the determining unit 403 is specifically configured to determine from the monitoring data the first client whose connection status with the cluster has changed; according to The service change of the cluster determines the second client whose connection state with the cluster has changed; according to the client included in the first client but not included in the second client and the alarm Generate rules to generate alarm information for the client.
  • the alarm rule also includes an alarm suppression rule;
  • the determining unit 403 is specifically configured to determine the change duration of the service change of the cluster; set the alarm suppression rule for the alarm information of the client, The alarm suppression rule of the client is used to not report the alarm information of the client generated within the change duration.
  • the monitoring server generates the alarm information of the MDS component of the cluster according to the health data of the cluster itself; the determining unit 403 is specifically configured to determine the alarm of the alarm information of the MDS component If the level is higher than the alarm information of the client, the alarm information of the MDS component is reported to the alarm platform.
  • the determining unit 403 is further configured to set a cluster identifier corresponding to each monitoring data.
  • the alarm rule also includes an alarm convergence rule; the determining unit 403 is specifically configured to determine that the alarm information is the same alarm information that does not appear for the first time in the cluster, and then according to the alarm The contrast relationship between the alarm level and the alarm delay in the convergence rule, the alarm information is reported to the alarm platform after the delay is set; wherein, the lower the alarm level is, the longer the corresponding alarm delay is long.
  • the embodiment of the present invention provides a computing device, and the computing device may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), etc.
  • the computing device may include a central processing unit (CPU), a memory, an input/output device, etc.
  • the input device may include a keyboard, a mouse, a touch screen, etc.
  • an output device may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), Cathode Ray Tube (CRT), etc.
  • the memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory.
  • the memory may be used to store the program instructions of the method for monitoring the distributed storage system;
  • the processor is configured to call the program instructions stored in the memory, and execute the method of monitoring the distributed storage system according to the obtained program.
  • FIG. 5 it is a schematic diagram of a computing device provided by an embodiment of this application, and the computing device includes:
  • the processor 501 is configured to read a program in the memory 502, and execute the foregoing method for monitoring a distributed storage system;
  • the processor 501 may be a central processing unit (central processing unit, CPU for short), a network processor (NP for short), or a combination of CPU and NP. It can also be a hardware chip.
  • the aforementioned hardware chip may be an application-specific integrated circuit (ASIC for short), a programmable logic device (PLD for short), or a combination thereof.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (generic array logic, GAL), or any of them combination.
  • the memory 502 is configured to store one or more executable programs, and can store data used by the processor 501 when performing operations.
  • the program may include program code, and the program code includes computer operation instructions.
  • the memory 502 may include a volatile memory (volatile memory), such as random-access memory (RAM for short); the memory 502 may also include a non-volatile memory (non-volatile memory), such as flash memory ( flash memory), hard disk drive (HDD for short) or solid-state drive (SSD for short); the memory 502 may also include a combination of the foregoing types of memories.
  • volatile memory volatile memory
  • RAM random-access memory
  • non-volatile memory non-volatile memory
  • flash memory flash memory
  • HDD hard disk drive
  • SSD solid-state drive
  • the memory 502 stores the following elements, executable modules or data structures, or their subsets, or their extended sets:
  • Operating instructions including various operating instructions, used to implement various operations.
  • Operating system Including various system programs, used to implement various basic services and process hardware-based tasks.
  • the bus 505 may be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • PCI peripheral component interconnect standard
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used to represent in FIG. 5, but it does not mean that there is only one bus or one type of bus.
  • the bus interface 504 may be a wired communication access port, a wireless bus interface or a combination thereof, where the wired bus interface may be, for example, an Ethernet interface.
  • the Ethernet interface can be an optical interface, an electrical interface, or a combination thereof.
  • the wireless bus interface may be a WLAN interface.
  • the embodiment of the present invention provides a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute a method for monitoring a distributed storage system.
  • the embodiments of the present invention can be provided as a method or a computer program product. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Environmental & Geological Engineering (AREA)
  • Debugging And Monitoring (AREA)
  • Hardware Redundancy (AREA)

Abstract

本发明提供一种监控分布式存储系统的方法及装置，监控服务器向分布式存储系统中的各集群发送采集指令；监控服务器获取各集群基于采集指令反馈的监控数据，监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；针对至少一个集群，监控服务器根据预设的告警规则，从集群的监控数据中确定告警信息并将告警信息上报至告警平台。该方案，监控服务器将采集指令下发至分布式存储系统中的各集群，使得监控服务器可同时监控多个集群；此外，各集群所反馈的监控数据中包括与集群相连的客户端的状态数据，有利于监控服务器通过对与集群相连的客户端的状态数据的分析来确定告警信息，实现了监控服务器对与集群相连的客户端进行监控的目的。

Description

一种监控分布式存储系统的方法及装置
相关申请的交叉引用
本申请要求在2019年12月23日提交中国专利局、申请号为201911336662.5、申请名称为“一种监控分布式存储系统的方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及金融科技(Fintech)领域，尤其涉及一种监控分布式存储系统的方法及装置。
背景技术
随着计算机技术的发展，越来越多的技术(例如云计算、大数据)应用在金融领域，传统金融业正在逐步向金融科技转变，大数据技术也不例外。但由于金融、支付行业的安全性、实时性要求，也对大数据技术提出了更高的要求。
出于对海量数据所要求的可扩展性以及高可用性等因素的考虑，银行业一般选择将CephFS(Ceph File System，Ceph文件系统)这样一种分布式存储系统作为共享存储的技术方案。其中，CephFS下连接有Ceph Fuse客户端(Ceph文件系统的用户空间文件系统客户端)；与此同时，本领域的技术人员通常采用开源的Prometheus(普罗米修斯)这样一种监控系统对CephFS进行监控。其中，Prometheus主要由Exporter(Prometheus监控数据采集的客户端)和Prometheus Sever(Prometheus监控的服务器端)等部分组成；CephFS主要由监视器(Monitor，简写成MON)、目标存储设备(Object Storage Device，简写成OSD)以及元数据服务器(MetaData Sever，简写成MDS)等各类组件组成，此外，CephFS OSD组件上还分布有归置小组(Placement Groups，简写成PG)。
针对现有技术中的Prometheus对于CephFS进行监控的技术方案，存在以下两方面问题：
第一，Prometheus对于CephFS的监控主要表现为Prometheus对CephFS OSD组件状态以及CephFS PG状态的数据采集，但Prometheus并没有实现对Ceph Fuse客户端的监控。
第二，Prometheus对于CephFS的监控架构非常臃肿，表现为需要给每个CephFS部署一套Prometheus；此外，由于CephFS版本的不同，还需要为不同版本的CephFS部署不同的Exporter。如图1所示，为现有技术的Prometheus对于CephFS的监控架构图。参考图1，M编号的普罗米修斯监控数据采集的客户端采集M编号的Ceph文件系统的监控数据，若所采集到的监控数据满足生成告警信息的规则，则将生成的告警信息上报至M编号的普罗米修斯服务器，同理，N编号的普罗米修斯监控数据采集的客户端采集N编号的Ceph文件系统的监控数据，若所采集到的监控数据满足生成告警信息的规则，则将生成的告警信息上报至N编号的普罗米修斯服务器；但由于M编号的普罗米修斯监控数据采集的客户端与N编号的Ceph文件系统版本的不匹配，从而不能将M编号的普罗米修斯监控数据采集的客户端用于采集N编号的Ceph文件系统的监控数据，以实现对N编号的Ceph文件系统的告警信息的上报。也即，Prometheus Sever、Exporter和CephFS这三者之间没有实 现高可用，导致在异常情况下无法及时上报监控信息。
综上，现有技术存在Prometheus无法监控Ceph Fuse客户端以及Prometheus对于CephFS的监控效率低下的问题。
发明内容
本发明提供一种监控分布式存储系统的方法及装置，用以解决Prometheus无法监控Ceph Fuse客户端以及Prometheus对于CephFS的监控效率低下的问题。
第一方面，本发明实施例提供一种监控分布式存储系统的方法，该方法包括：监控服务器向所述分布式存储系统中的各集群发送采集指令；所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；针对至少一个集群，所述监控服务器根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
基于该方案，监控服务器通过将采集指令下发至分布式存储系统中的各集群的方式，使得监控服务器可以同时监控多个集群，从而避免了由于集群与Exporter版本不匹配时、监控服务器无法有效地监控各集群；此外，各集群反馈给监控服务器的监控数据中还包括与集群相连的客户端的状态数据，有利于监控服务器通过对与集群相连的客户端的状态数据的分析来确定告警信息，从而实现了监控服务器对与集群相连的客户端进行监控的目的。
作为一种可能实现的方法，所述监控服务器为多台；任一集群中包括多台节点服务器，且连接有客户端的各节点服务器所连接的客户端均相同；所述监控服务器向所述分布式存储系统中的各集群发送采集指令，包括：针对任一台监控服务器，所述监控服务器向任一集群中的至少两台节点服务器下发采集指令。
基于该方案，通过为分布式存储系统设置多台监控服务器，一方面，通过频繁地从分布式存储系统中的各集群中获取各集群的监控数据，可以实现对于该分布式存储系统的全方位、甚至实时监控的目标；另一方面，通过设置多台监控服务器的方式，还可以确保在其中一台或几台监控服务器宕机的情况下，还有其他可用的监控服务器来对该分布式存储系统进行监控。对于多台监控服务器中的任一台监控服务器，该监控服务器通过向各集群中的至少两台节点服务器下发采集指令，有利于确保在其中一台节点服务器宕机的情况下，该监控服务器还可以从其他可用的节点服务器上来获取该节点服务器所在集群的监控数据，从而实现监控服务器对各集群的有效监控。
作为一种可能实现的方法，所述告警规则包括告警生成规则；所述监控服务器根据预设的告警规则，从所述监控数据中确定告警信息，包括：所述监控服务器从所述监控数据中确定出与所述集群的连接状态发生变化的第一客户端；所述监控服务器根据所述集群的业务变化确定与所述集群的连接状态发生变化的第二客户端；根据包含在所述第一客户端中却不包含在所述第二客户端中的客户端及所述告警生成规则，生成客户端的告警信息。
基于该方案，通过对监控数据的分析，确定出与所述集群的连接状态发生变化的第一客户端，以及通过对已知业务变化的分析，确定出与所述集群的连接状态发生变化的第二客户端，通过将第一客户端与第二客户端进行对比，即可生成由于客户端的异常而产生的告警信息。
作为一种可能实现的方法，所述告警规则还包括告警抑制规则；所述监控服务器确定所述集群的业务变化的变化时长；所述监控服务器设置所述客户端的告警信息的告警抑制 规则，所述客户端的告警抑制规则用于将在所述变化时长内产生的所述客户端的告警信息不进行上报。
基于该方案，在确定出集群出于业务需要的目的而要求的必要的时长后，监控服务器并不会在这段必要的时长的过程中、将客户端的告警信息上报至告警平台，从而可以有效地避免产生已知而无用的告警。
作为一种可能实现的方法，所述监控服务器根据所述集群自身的健康数据生成所述集群的MDS组件的告警信息；所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：所述监控服务器确定所述MDS组件的告警信息的告警级别高于所述客户端的告警信息，则将所述MDS组件的告警信息上报至告警平台。
基于该方案，当监控服务器同时获取到集群的MDS组件的告警信息和与集群相连的客户端的告警信息时，考虑到可能是集群的MDS组件的异常造成了与集群相连的客户端的异常事件，因此监控服务器确定MDS组件的告警信息的告警级别高于客户端的告警信息，并将MDS组件的告警信息上报至告警平台，自动屏蔽低级别的客户端的告警信息。
作为一种可能实现的方法，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据之后，还包括：所述监控服务器设置各监控数据对应的集群标识。
基于该方案，监控服务器通过为获取到的各监控数据打上与其对应的集群的标识，有助于监控服务器后期对于接收到同一集群的相同监控数据时、迅速地做出相应的告警操作。
作为一种可能实现的方法，所述告警规则还包括告警收敛规则；所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：所述监控服务器确定所述告警信息为所述集群中非首次出现的同一告警信息，则根据所述告警收敛规则中的告警级别与告警时延的对照关系，在设定时延后将所述告警信息上报至所述告警平台；其中，告警级别的级别越低，相应的告警时延的时延越长。
基于该方案，在监控服务器确定告警信息为某集群非首次出现的相同告警信息后，根据告警收敛规则、在设定时延后将非首次出现的相同告警上报至所述告警平台，可以有效防止该集群持续重复发出相同告警，而造成的资源浪费现象。
第二方面，本发明实施例提供一种监控分布式存储系统的装置，该装置包括：发送单元，用于向所述分布式存储系统中的各集群发送采集指令；获取单元，用于获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；确定单元，针对至少一个集群，用于根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
基于该方案，监控服务器通过将采集指令下发至分布式存储系统中的各集群的方式，使得监控服务器可以同时监控多个集群，从而避免了由于集群与Exporter版本不匹配时、监控服务器无法有效地监控各集群；此外，各集群反馈给监控服务器的监控数据中还包括与集群相连的客户端的状态数据，有利于监控服务器通过对与集群相连的客户端的状态数据的分析来确定告警信息，从而实现了监控服务器对与集群相连的客户端进行监控的目的。
作为一种可能实现的方法，所述监控服务器为多台；任一集群中包括多台节点服务器，且连接有客户端的各节点服务器所连接的客户端均相同；针对任一台监控服务器，所述发送单元，具体用于向任一集群中的至少两台节点服务器下发采集指令。
基于该方案，通过为分布式存储系统设置多台监控服务器，一方面，通过频繁地从分布式存储系统中的各集群中获取各集群的监控数据，可以实现对于该分布式存储系统的全 方位、甚至实时监控的目标；另一方面，通过设置多台监控服务器的方式，还可以确保在其中一台或几台监控服务器宕机的情况下，还有其他可用的监控服务器来对该分布式存储系统进行监控。对于多台监控服务器中的任一台监控服务器，该监控服务器通过向各集群中的至少两台节点服务器下发采集指令，有利于确保在其中一台节点服务器宕机的情况下，该监控服务器还可以从其他可用的节点服务器上来获取该节点服务器所在集群的监控数据，从而实现监控服务器对各集群的有效监控。
作为一种可能实现的方法，所述告警规则包括告警生成规则；所述确定单元，具体用于从所述监控数据中确定出与所述集群的连接状态发生变化的第一客户端；根据所述集群的业务变化确定与所述集群的连接状态发生变化的第二客户端；根据包含在所述第一客户端中却不包含在所述第二客户端中的客户端及所述告警生成规则，生成客户端的告警信息。
基于该方案，通过对监控数据的分析，确定出与所述集群的连接状态发生变化的第一客户端，以及通过对已知业务变化的分析，确定出与所述集群的连接状态发生变化的第二客户端，通过将第一客户端与第二客户端进行对比，即可生成由于客户端的异常而产生的告警信息。
作为一种可能实现的方法，所述告警规则还包括告警抑制规则；所述确定单元，具体用于确定所述集群的业务变化的变化时长；设置所述客户端的告警信息的告警抑制规则，所述客户端的告警抑制规则用于将在所述变化时长内产生的所述客户端的告警信息不进行上报。
基于该方案，在确定出集群出于业务需要的目的而要求的必要的时长后，监控服务器并不会在这段必要的时长的过程中、将客户端的告警信息上报至告警平台，从而可以有效地避免产生已知而无用的告警。
作为一种可能实现的方法，所述监控服务器根据所述集群自身的健康数据生成所述集群的MDS组件的告警信息；所述确定单元，具体用于确定所述MDS组件的告警信息的告警级别高于所述客户端的告警信息，则将所述MDS组件的告警信息上报至告警平台。
基于该方案，当监控服务器同时获取到集群的MDS组件的告警信息和与集群相连的客户端的告警信息时，考虑到可能是集群的MDS组件的异常造成了与集群相连的客户端的异常事件，因此监控服务器确定MDS组件的告警信息的告警级别高于客户端的告警信息，并将MDS组件的告警信息上报至告警平台，自动屏蔽低级别的客户端的告警信息。
作为一种可能实现的方法，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据之后，所述确定单元，还用于设置各监控数据对应的集群标识。
基于该方案，监控服务器通过为获取到的各监控数据打上与其对应的集群的标识，有助于监控服务器后期对于接收到同一集群的相同监控数据时、迅速地做出相应的告警操作。
作为一种可能实现的方法，所述告警规则还包括告警收敛规则；所述确定单元，具体用于确定所述告警信息为所述集群中非首次出现的同一告警信息，则根据所述告警收敛规则中的告警级别与告警时延的对照关系，在设定时延后将所述告警信息上报至所述告警平台；其中，告警级别的级别越低，相应的告警时延的时延越长。
基于该方案，在监控服务器确定告警信息为某集群非首次出现的相同告警信息后，根据告警收敛规则、在设定时延后将非首次出现的相同告警上报至所述告警平台，可以有效防止该集群持续重复发出相同告警，而造成的资源浪费现象。
第三方面，本发明实施例提供了一种计算设备，包括：
存储器，用于存储程序指令；
处理器，用于调用所述存储器中存储的程序指令，按照获得的程序执行如第一方面任一所述的方法。
第四方面，本发明实施例提供了一种计算机可读存储介质，所述计算机存储介质存储有计算机可执行指令，所述计算机可执行指令用于使计算机执行如第一方面任一所述的方法。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简要介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域的普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为现有技术的Prometheus对于CephFS的监控架构图；
图2为本发明提供的一种监控分布式存储系统的方法；
图3为本发明提供的一种Prometheus对于CephFS的监控架构图；
图4为本发明提供的一种监控分布式存储系统的装置；
图5为本发明提供的一种计算设备的示意图。
具体实施方式
为了使本发明的目的、技术方案和优点更加清楚，下面将结合附图对本发明作进一步地详细描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本发明保护的范围。
如图2所示，为本发明实施例提供的一种监控分布式存储系统的方法，该方法包括：
步骤201，监控服务器向所述分布式存储系统中的各集群发送采集指令。
步骤202，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据。
步骤203，针对至少一个集群，所述监控服务器根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
基于该方案，监控服务器通过将采集指令下发至分布式存储系统中的各集群的方式，使得监控服务器可以同时监控多个集群，从而避免了由于集群与Exporter版本不匹配时、监控服务器无法有效地监控各集群；此外，各集群反馈给监控服务器的监控数据中还包括与集群相连的客户端的状态数据，有利于监控服务器通过对与集群相连的客户端的状态数据的分析来确定告警信息，从而实现了监控服务器对与集群相连的客户端进行监控的目的。
在上述步骤201中，监控服务器向所述分布式存储系统中的各集群发送采集指令。
设CephFS(Ceph File System，Ceph文件系统)这样一种分布式存储系统中设置有多个集群，如3个，分别令为A编号的Ceph文件系统集群，B编号的Ceph文件系统集群，C编号的Ceph文件系统集群；作为监控CephFS的一种监控服务器Prometheus(普罗米修斯)，通过其内部的Prometheus Sever(普罗米修斯服务器)向CephFS下发采集指令，具体表现为Prometheus Sever向A编号的Ceph文件系统集群下发采集指令I，Prometheus  Sever向B编号的Ceph文件系统集群下发采集指令I，Prometheus Sever向C编号的Ceph文件系统集群下发采集指令I。
在上述步骤202中，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据。
当Prometheus Sever向A编号的Ceph文件系统集群下发采集指令I后，A编号的Ceph文件系统集群会对该采集指令I作出相应的响应，得到关于A编号的Ceph文件系统集群的监控数据，由此Prometheus Sever获取到关于A编号的Ceph文件系统集群的监控数据；同理，Prometheus Sever可以获取到关于B编号的Ceph文件系统集群的监控数据以及获取到关于C编号的Ceph文件系统集群的监控数据。
关于A编号的Ceph文件系统集群的监控数据，具体可以表现为A编号的Ceph文件系统集群自身的健康数据(如OSD组件的运行状态、PG的状态数据)，以及与A编号的Ceph文件系统集群相连的A编号的Ceph文件系统的用户空间文件系统客户端的状态数据(如A编号的Ceph文件系统的用户空间文件系统客户端是否接入A编号的Ceph文件系统)。比如与A编号的Ceph文件系统集群相连的A编号的Ceph文件系统的用户空间文件系统客户端有100个，则关于A编号的Ceph文件系统集群的监控数据部件包括A编号的Ceph文件系统集群自身的健康数据，还包括与A编号的Ceph文件系统集群相连的100个A编号的Ceph文件系统的用户空间文件系统客户端的状态数据；关于B编号的Ceph文件系统集群的监控数据、关于C编号的Ceph文件系统集群的监控数据可以参考关于A编号的Ceph文件系统集群的监控数据的情形，在此不赘述。
在上述步骤203中，针对至少一个集群，所述监控服务器根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
设对于A编号的Ceph文件系统集群，Prometheus通过对获取的来自于A编号的Ceph文件系统集群的监控数据的分析，分析的依据是预设的告警规则，从而确定出关于A编号的Ceph文件系统集群的告警信息；进一步地，Prometheus将得到的关于A编号的Ceph文件系统集群的告警信息后上报至告警平台，上报的依据仍然是预设的告警规则。其中，告警平台可以为IMS系统(Information Management System，信息管理系统)，还可以是其他告警平台，对此，本发明不做限定。同理，Prometheus对于B编号的Ceph文件系统集群、C编号的Ceph文件系统集群的告警过程可以参考A编号的Ceph文件系统集群的告警过程，在此不赘述。
作为一种可能实现的方法，所述监控服务器为多台；任一集群中包括多台节点服务器，且连接有客户端的各节点服务器所连接的客户端均相同；所述监控服务器向所述分布式存储系统中的各集群发送采集指令，包括：针对任一台监控服务器，所述监控服务器向任一集群中的至少两台节点服务器下发采集指令。
如图3所示，为本发明实施例提供的一种Prometheus对于CephFS的监控架构图。参考图3，部署了两台监控服务器，分别令为X编号的普罗米修斯服务器和Y编号的普罗米修斯服务器，X编号的普罗米修斯服务器和Y编号的普罗米修斯服务器均用于监控分布式存储系统，该系统中部署有A编号的Ceph文件系统集群、B编号的Ceph文件系统集群和C编号的Ceph文件系统集群；对于A编号的Ceph文件系统集群，该集群中包括了多台节点服务器，为了叙述的方便，设A编号的Ceph文件系统集群包括了4台节点服务器，分别令为A1编号的节点服务器、A2编号的节点服务器、A3编号的节点服务器和A4编号的 节点服务器；同样的，对于B编号的Ceph文件系统集群，该集群中包括了多台节点服务器，为了叙述的方便，设B编号的Ceph文件系统集群包括了4台节点服务器，分别令为B1编号的节点服务器、B2编号的节点服务器、B3编号的节点服务器和B4编号的节点服务器；同样的，对于C编号的Ceph文件系统集群，该集群中包括了多台节点服务器，为了叙述的方便，设C编号的Ceph文件系统集群包括了4台节点服务器，分别令为C1编号的节点服务器、C2编号的节点服务器、C3编号的节点服务器和C4编号的节点服务器。
对于A编号的Ceph文件系统集群，设有100台A编号的Ceph文件系统的用户空间文件系统客户端连接于该集群中的被配置有MDS组件的节点服务器，设A编号的Ceph文件系统集群中有3台节点服务器被配置有MDS组件，则这100台A编号的Ceph文件系统的用户空间文件系统客户端均连接于这3台被配置有MDS组件的节点服务器(图中未示出)；同理，对于B编号的Ceph文件系统集群，设有200台B编号的Ceph文件系统的用户空间文件系统客户端连接于该集群中的被配置有MDS组件的节点服务器，设CephFS_B集群中有3台节点服务器被配置有MDS组件，则这200台B编号的Ceph文件系统的用户空间文件系统客户端均连接于这3台被配置有MDS组件的节点服务器(图中未示出)；同理，对于C编号的Ceph文件系统集群，设有300台C编号的Ceph文件系统的用户空间文件系统客户端连接于该集群中的被配置有MDS组件的节点服务器，设C编号的Ceph文件系统集群中有3台节点服务器被配置有MDS组件，则这300台C编号的Ceph文件系统的用户空间文件系统客户端均连接于这3台被配置有MDS组件的节点服务器(图中未示出)。
设对于X编号的普罗米修斯服务器，该监控服务器向上述A编号的Ceph文件系统集群、B编号的Ceph文件系统集群和C编号的Ceph文件系统集群中的任一集群中的至少两台节点服务器下发采集指令，具体表现为：
设在上午8:00这一时刻，X编号的普罗米修斯服务器向A编号的Ceph文件系统集群中的A1编号的节点服务器、A2编号的节点服务器和A4编号的节点服务器这3台节点服务器下发采集指令I；同时，X编号的普罗米修斯服务器向B编号的Ceph文件系统集群中的B1编号的节点服务器、B3编号的节点服务器和B4编号的节点服务器这3台节点服务器下发采集指令I；同时，X编号的普罗米修斯服务器向C编号的Ceph文件系统集群中的C1编号的节点服务器、C2编号的节点服务器和C4编号的节点服务器这3台节点服务器下发采集指令I。
需要说明的是，X编号的普罗米修斯服务器向A编号的Ceph文件系统集群中的至少两台节点服务器下发采集指令时，是通过随机的方式向A编号的Ceph文件系统集群中的任意至少两台节点服务器下发采集指令。举个例子，上述X编号的普罗米修斯服务器可以是向A编号的Ceph文件系统集群中的A1编号的节点服务器、A2编号的节点服务器和A4编号的节点服务器这3台节点服务器下发采集指令I，也可以是向A编号的Ceph文件系统集群中的A2编号的节点服务器、A3编号的节点服务器和A4编号的节点服务器这3台节点服务器下发采集指令I，也可以是向A编号的Ceph文件系统集群中的A1编号的节点服务器、A2编号的节点服务器和A3编号的节点服务器这3台节点服务器下发采集指令I，对此本发明不做限定。同样的，X编号的普罗米修斯服务器向B编号的Ceph文件系统集群中的至少两台节点服务器下发采集指令时，是通过随机的方式向B编号的Ceph文件系统集群中的任意至少两台节点服务器下发采集指令；同样的，X编号的普罗米修斯服务 器向C编号的Ceph文件系统集群中的至少两台节点服务器下发采集指令时，是通过随机的方式向C编号的Ceph文件系统集群中的任意至少两台节点服务器下发采集指令。
作为一种可能实现的方式，所述告警规则包括告警生成规则；所述监控服务器根据预设的告警规则，从所述监控数据中确定告警信息，包括：所述监控服务器从所述监控数据中确定出与所述集群的连接状态发生变化的第一客户端；所述监控服务器根据所述集群的业务变化确定与所述集群的连接状态发生变化的第二客户端；根据包含在所述第一客户端中却不包含在所述第二客户端中的客户端及所述告警生成规则，生成客户端的告警信息。
举个例子，对于CephFS_A集群，为了叙述的方便，设有W1、W2、W3、W4、W5、W6、W7、W8、W9和W10这10台Ceph Fuse_A客户端连接于该集群中的被配置有MDS组件的节点服务器；Prometheus Sever_X向CephFS_A集群中的A1、A2和A4这3台节点服务器下发了采集指令I，设Prometheus Sever_X首先获取到A1节点服务器上的监控数据，通过对A1节点服务器上的监控数据的分析，确定出其中的W1、W2、W3、W4、W5、W6、W7、W8、W9和W10这10台Ceph Fuse_A客户端均连接于该CephFS_A集群；随后，Prometheus Sever_X接着获取到A2节点服务器上的监控数据，通过对A2节点服务器上的监控数据的分析，确定出其中仅有W8、W9和W10这3台Ceph Fuse_A客户端仍然连接于CephFS_A集群，而W1、W2、W3、W4、W5、W6和W7这7台Ceph Fuse_A客户端已经从CephFS_A集群上离线。也即，与所述集群的连接状态发生变化的第一客户端分别为W1、W2、W3、W4、W5、W6和W7这7台Ceph Fuse_A客户端。
对于Ceph Fuse_A客户端出现的这种异常事件，则需要进一步判断W1、W2、W3、W4、W5、W6和W7这7台Ceph Fuse_A客户端从CephFS_A集群上离线的原因，即在于Ceph Fuse_A客户端是正常地从CephFS_A集群中卸载，还是由于CephFS_A集群自身的原因而导致的Ceph Fuse_A客户端被动卸载。
运行于CephFS_A集群上的业务，出于业务需要的目的，会对连接于CephFS_A集群上的部分客户端进行日常的卸载工作。比如，出于业务需要的目的，业务人员会对CephFS_A集群中的W5、W6和W7这3台Ceph Fuse_A客户端进行卸载。也即与所述集群的连接状态发生变化的第二客户端分别为W5、W6和W7这3台Ceph Fuse_A客户端。
通过对第一客户端(分别有W1、W2、W3、W4、W5、W6和W7这7台Ceph Fuse_A客户端)和第二客户端(W5、W6和W7这3台Ceph Fuse_A客户端)的比较，可以发现W5、W6和W7这3台Ceph Fuse_A客户端的卸载是属于Ceph Fuse_A客户端的正常卸载事件，从而对于监控数据中的W5、W6和W7这3台Ceph Fuse_A客户端的离线不需要上报至IMS系统；而对于W1、W2、W3和W4这4台Ceph Fuse_A客户端的卸载属于Ceph Fuse_A客户端的异常卸载事件，则根据告警生成规则，生成客户端的告警信息。
作为一种可能实现的方式，所述告警规则还包括告警抑制规则；所述监控服务器确定所述集群的业务变化的变化时长；所述监控服务器设置所述客户端的告警信息的告警抑制规则，所述客户端的告警抑制规则用于将在所述变化时长内产生的所述客户端的告警信息不进行上报。
如前述的例子，设出于业务需要的目的，对连接于CephFS_A集群上的W5、W6和W7这3台Ceph Fuse_A客户端进行正常的卸载操作，设卸载W5、W6和W7这3台Ceph Fuse_A客户端所要求的时长为3h，则Prometheus Sever_X在获取到A2节点服务器上的监控数据后的未来3h的整个时间段内，Prometheus Sever_X并不会将连接于CephFS_A集群 上的W5、W6和W7这3台Ceph Fuse_A客户端的离线事件上报至IMS系统。也即，Prometheus Sever_X将W5、W6和W7这3台Ceph Fuse_A客户端从CephFS_A集群上离线的事件写入了告警抑制规则中。
作为一种可能实现的方法，所述监控服务器根据所述集群自身的健康数据生成所述集群的MDS组件的告警信息；所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：所述监控服务器确定所述MDS组件的告警信息的告警级别高于所述客户端的告警信息，则将所述MDS组件的告警信息上报至告警平台。
如前述的例子中，Prometheus Sever_X对于CephFS_A集群的监控数据，包括CephFS_A集群自身的健康数据(如OSD组件的运行状态、PG的状态数据)，以及与CephFS_A集群相连的Ceph Fuse_A客户端的状态数据(如Ceph Fuse_A客户端是否接入CephFS_A集群)。设在T时刻，Prometheus Sever_X获取到的有关于CephFS_A集群的监控数据，该条监控数据显示CephFS_A集群中的MDS组件在运行时出现异常，同时与CephFS_A集群相连的W1这1台Ceph Fuse_A客户端也出现异常卸载事件，则Prometheus Sever_X将CephFS_A集群中的MDS组件在运行时出现的异常事件的告警级别定义为高级别，将W1这1台Ceph Fuse_A客户端出现的异常卸载事件的告警级别定义为低级别；随后Prometheus Sever_X将高级别的告警事件上报至IMS系统，也即Prometheus Sever_X会将CephFS_A集群中的MDS组件在运行时出现的异常事件上报至IMS系统，而不会将W1这1台Ceph Fuse_A客户端出现的异常卸载事件上报至IMS系统。
需要说明的是，监控服务器之所以可以将集群中的MDS组件的告警信息的告警级别设置的比所述客户端的告警信息的告警级别高，原因在于由于集群中的MDS组件的异常会造成与集群相连的客户端的异常事件，因而在将集群中的MDS组件的告警信息上报至IMS系统、在运维人员进行运维排查后，不仅可以将MDS组件恢复至正常的运行状态，同时可以让与集群相连的客户端也恢复至正常状态。
作为一种可能实现的方法，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据之后，还包括：所述监控服务器设置各监控数据对应的集群标识。
如前述的例子，参考图3，Prometheus Sever_X向CephFS_A集群中的A1、A2和A4这三台节点服务器发送采集指令I，同时向CephFS_B集群中的B1、B3和B4这三台节点服务器发送采集指令I，以及同时向CephFS_C集群中的C1、C2和C4这三台节点服务器发送采集指令I；当采集指令I在上述CephFS_A集群、CephFS_B集群和CephFS_C集群这三个集群中被响应后，Prometheus Sever_X会将获取上述各个集群的监控数据。其中，监控数据可以表现为集群的标识，比如Prometheus Sever_X获取到的第一条是CephFS_A集群的A1节点服务器上的监控数据，第二条是CephFS_B集群的B3节点服务器上的监控数据，第三条是CephFS_C集群的C4节点服务器上的监控数据，等等。
作为一种可能实现的方法，所述告警规则还包括告警收敛规则；所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：所述监控服务器确定所述告警信息为所述集群中非首次出现的同一告警信息，则根据所述告警收敛规则中的告警级别与告警时延的对照关系，在设定时延后将所述告警信息上报至所述告警平台；其中，告警级别的级别越低，相应的告警时延的时延越长。
如前述的例子，设Prometheus Sever_X获取到的第一条监控数据是来自于CephFS_A集群，根据预设的告警规则、对第一条监控数据进行分析后，确定第一条监控数据可以作 为告警信息上报至IMS系统，将根据第一条监控数据所生成的告警信息令为Info_1，且将Infro_1的告警级别为令为级别1；设Prometheus Sever_X获取到的第六条监控数据仍然是有关于CephFS_A集群的，根据预设的告警规则、对第六条监控数据进行分析后，发现根据这第六条监控数据所生成的告警信息符合Info_1，则Prometheus Sever_X需要进一步的根据Infro_1的告警级别来确定何时将这第六条监控数据上报至IMS系统；设告警级别为级别1的告警信息所对应的告警时延为1h，则Prometheus Sever_X在接下来的1h内并不会将第六条监控数据对应的Infro_1上报至IMS系统。
设Prometheus Sever_X获取到的第二条监控数据是来自于CephFS_B集群，根据预设的告警规则、对第二条监控数据进行分析后，确定第二条监控数据可以作为告警信息上报至IMS系统，将根据第二条监控数据所生成的告警信息令为Info_2，且将Infro_2的告警级别为令为级别2；设Prometheus Sever_X获取到的第九条监控数据仍然是有关于CephFS_B集群的，根据预设的告警规则、对第九条监控数据进行分析后，发现根据这第九条监控数据所生成的告警信息符合Info_2，则Prometheus Sever_X需要进一步的根据Infro_2的告警级别来确定何时将这第九条监控数据上报至IMS系统；设告警级别为级别2的告警信息所对应的告警时延为2h，则Prometheus Sever_X在接下来的2h内并不会将第九条监控数据对应的Infro_2上报至IMS系统。
设Prometheus Sever_X获取到的第三条监控数据是来自于CephFS_C集群，根据预设的告警规则、对第三条监控数据进行分析后，确定第三条监控数据可以作为告警信息上报至IMS系统，将根据第三条监控数据所生成的告警信息令为Info_3，且将Infro_3的告警级别为令为级别3；设Prometheus Sever_X获取到的第十条监控数据仍然是有关于CephFS_C集群的，根据预设的告警规则、对第十条监控数据进行分析后，发现根据这第十条监控数据所生成的告警信息符合Info_3，则Prometheus Sever_X需要进一步的根据Infro_3的告警级别来确定何时将这第十条监控数据上报至IMS系统；设告警级别为级别3的告警信息所对应的告警时延为3h，则Prometheus Sever_X在接下来的3h内并不会将第十条监控数据对应的Infro_3上报至IMS系统。
需要说明的是，上述例子中，随着级别1、级别2、级别3的告警级别的降低，相应的告警时延的时延越长，分别对应1h、2h、3h。
基于该方案，在监控服务器确定告警信息为某集群非首次出现的相同告警信息后，根据告警收敛规则、在设定时延后将非首次出现的相同告警上报至所述告警平台，可以有效防止该集群持续重复发出相同告警，而造成的资源浪费现象。
基于同样的构思，本发明实施例还提供一种监控分布式存储系统的装置，如图4所示，该装置包括：
发送单元401，用于向所述分布式存储系统中的各集群发送采集指令；
获取单元402，用于获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；
确定单元403，针对至少一个集群，用于根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
进一步地，对于所述装置，所述监控服务器为多台；任一集群中包括多台节点服务器，且连接有客户端的各节点服务器所连接的客户端均相同；针对任一台监控服务器，所述发 送单元401，具体用于向任一集群中的至少两台节点服务器下发采集指令。
进一步地，对于所述装置，所述告警规则包括告警生成规则；所述确定单元403，具体用于从所述监控数据中确定出与所述集群的连接状态发生变化的第一客户端；根据所述集群的业务变化确定与所述集群的连接状态发生变化的第二客户端；根据包含在所述第一客户端中却不包含在所述第二客户端中的客户端及所述告警生成规则，生成客户端的告警信息。
进一步地，对于所述装置，所述告警规则还包括告警抑制规则；所述确定单元403，具体用于确定所述集群的业务变化的变化时长；设置所述客户端的告警信息的告警抑制规则，所述客户端的告警抑制规则用于将在所述变化时长内产生的所述客户端的告警信息不进行上报。
进一步地，对于所述装置，所述监控服务器根据所述集群自身的健康数据生成所述集群的MDS组件的告警信息；所述确定单元403，具体用于确定所述MDS组件的告警信息的告警级别高于所述客户端的告警信息，则将所述MDS组件的告警信息上报至告警平台。
进一步地，对于所述装置，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据之后，所述确定单元403，还用于设置各监控数据对应的集群标识。
进一步地，对于所述装置，所述告警规则还包括告警收敛规则；所述确定单元403，具体用于确定所述告警信息为所述集群中非首次出现的同一告警信息，则根据所述告警收敛规则中的告警级别与告警时延的对照关系，在设定时延后将所述告警信息上报至所述告警平台；其中，告警级别的级别越低，相应的告警时延的时延越长。
本发明实施例提供了一种计算设备，该计算设备具体可以为桌面计算机、便携式计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)等。该计算设备可以包括中央处理器(Center Processing Unit，CPU)、存储器、输入/输出设备等，输入设备可以包括键盘、鼠标、触摸屏等，输出设备可以包括显示设备，如液晶显示器(Liquid Crystal Display，LCD)、阴极射线管(Cathode Ray Tube，CRT)等。
存储器，可以包括只读存储器(ROM)和随机存取存储器(RAM)，并向处理器提供存储器中存储的程序指令和数据。在本发明实施例中，存储器可以用于存储监控分布式存储系统的方法的程序指令；
处理器，用于调用所述存储器中存储的程序指令，按照获得的程序执行监控分布式存储系统的方法。
如图5所示，为本申请实施例提供的一种计算设备的示意图，该计算设备包括：
处理器501、存储器502、收发器503、总线接口504；其中，处理器501、存储器502与收发器503之间通过总线505连接；
所述处理器501，用于读取所述存储器502中的程序，执行上述监控分布式存储系统的方法；
处理器501可以是中央处理器(central processing unit，简称CPU)，网络处理器(network processor，简称NP)或者CPU和NP的组合。还可以是硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，简称ASIC)，可编程逻辑器件(programmable logic device，简称PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，简称CPLD)，现场可编程逻辑门阵列(field-programmable gate array，简称FPGA)，通用阵列逻辑(generic array logic,简称GAL) 或其任意组合。
所述存储器502，用于存储一个或多个可执行程序，可以存储所述处理器501在执行操作时所使用的数据。
具体地，程序可以包括程序代码，程序代码包括计算机操作指令。存储器502可以包括易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，简称RAM)；存储器502也可以包括非易失性存储器(non-volatile memory)，例如快闪存储器(flash memory)，硬盘(hard disk drive，简称HDD)或固态硬盘(solid-state drive，简称SSD)；存储器502还可以包括上述种类的存储器的组合。
存储器502存储了如下的元素，可执行模块或者数据结构，或者它们的子集，或者它们的扩展集：
操作指令：包括各种操作指令，用于实现各种操作。
操作系统：包括各种系统程序，用于实现各种基础业务以及处理基于硬件的任务。
总线505可以是外设部件互连标准(peripheral component interconnect，简称PCI)总线或扩展工业标准结构(extended industry standard architecture，简称EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图5中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
总线接口504可以为有线通信接入口，无线总线接口或其组合，其中，有线总线接口例如可以为以太网接口。以太网接口可以是光接口，电接口或其组合。无线总线接口可以为WLAN接口。
本发明实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使计算机执行监控分布式存储系统的方法。
本领域内的技术人员应明白，本发明的实施例可提供为方法、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个 方框或多个方框中指定的功能的步骤。
尽管已描述了本发明的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。

Claims (10)

  1. 一种监控分布式存储系统的方法，其特征在于，包括：
    监控服务器向所述分布式存储系统中的各集群发送采集指令；
    所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；
    针对至少一个集群，所述监控服务器根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
  2. 如权利要求1所述的方法，其特征在于，所述监控服务器为多台；任一集群中包括多台节点服务器，且连接有客户端的各节点服务器所连接的客户端均相同；
    所述监控服务器向所述分布式存储系统中的各集群发送采集指令，包括：
    针对任一台监控服务器，所述监控服务器向任一集群中的至少两台节点服务器下发采集指令。
  3. 如权利要求1所述的方法，其特征在于，所述告警规则包括告警生成规则；
    所述监控服务器根据预设的告警规则，从所述监控数据中确定告警信息，包括：
    所述监控服务器从所述监控数据中确定出与所述集群的连接状态发生变化的第一客户端；
    所述监控服务器根据所述集群的业务变化确定与所述集群的连接状态发生变化的第二客户端；
    根据包含在所述第一客户端中却不包含在所述第二客户端中的客户端及所述告警生成规则，生成客户端的告警信息。
  4. 如权利要求3所述的方法，其特征在于，所述告警规则还包括告警抑制规则；
    所述监控服务器确定所述集群的业务变化的变化时长；
    所述监控服务器设置所述客户端的告警信息的告警抑制规则，所述客户端的告警抑制规则用于将在所述变化时长内产生的所述客户端的告警信息不进行上报。
  5. 如权利要求3所述的方法，其特征在于，所述监控服务器根据所述集群自身的健康数据生成所述集群的MDS组件的告警信息；
    所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：
    所述监控服务器确定所述MDS组件的告警信息的告警级别高于所述客户端的告警信息，则将所述MDS组件的告警信息上报至告警平台。
  6. 如权利要求1所述的方法，其特征在于，所述监控服务器获取所述各集群基于所述采集指令反馈的监控数据之后，还包括：
    所述监控服务器设置各监控数据对应的集群标识。
  7. 如权利要求1-6任一项所述的方法，其特征在于，所述告警规则还包括告警收敛规则；
    所述监控服务器根据预设的告警规则，将所述告警信息上报至告警平台，包括：
    所述监控服务器确定所述告警信息为所述集群中非首次出现的同一告警信息，则根据所述告警收敛规则中的告警级别与告警时延的对照关系，在设定时延后将所述告警信息上报至所述告警平台；其中，告警级别的级别越低，相应的告警时延的时延越长。
  8. 一种监控分布式存储系统的装置，其特征在于，包括：
    发送单元，用于向所述分布式存储系统中的各集群发送采集指令；
    获取单元，用于获取所述各集群基于所述采集指令反馈的监控数据，所述监控数据包括集群自身的健康数据以及与集群相连的客户端的状态数据；
    确定单元，针对至少一个集群，用于根据预设的告警规则，从所述集群的监控数据中确定告警信息并将所述告警信息上报至告警平台。
  9. 一种计算设备，其特征在于，包括：
    存储器，用于存储程序指令；
    处理器，用于调用所述存储器中存储的程序指令，按照获得的程序执行如权利要求1-7任一项所述的方法。
  10. 一种计算机可读存储介质，其特征在于，所述存储介质存储有计算机可执行指令，所述计算机可执行指令用于使计算机执行如权利要求1-7任一项所述的方法。
PCT/CN2020/134339 2019-12-23 2020-12-07 一种监控分布式存储系统的方法及装置 Ceased WO2021129367A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911336662.5 2019-12-23
CN201911336662.5A CN111049705B (zh) 2019-12-23 2019-12-23 一种监控分布式存储系统的方法及装置

Publications (1)

Publication Number Publication Date
WO2021129367A1 true WO2021129367A1 (zh) 2021-07-01

Family

ID=70238567

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/134339 Ceased WO2021129367A1 (zh) 2019-12-23 2020-12-07 一种监控分布式存储系统的方法及装置

Country Status (2)

Country Link
CN (1) CN111049705B (zh)
WO (1) WO2021129367A1 (zh)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113900898A (zh) * 2021-10-19 2022-01-07 北京金山云网络技术有限公司 一种数据处理系统、设备及介质
CN114115718A (zh) * 2021-08-31 2022-03-01 济南浪潮数据技术有限公司 分布式块存储系统服务质量控制方法、装置、设备及介质
CN114595126A (zh) * 2022-03-24 2022-06-07 山西合力创新科技股份有限公司 一种基于规则引擎的告警方法、系统、存储设备及终端
CN114760221A (zh) * 2022-03-31 2022-07-15 深信服科技股份有限公司 一种业务监控方法、系统和存储介质
CN115118630A (zh) * 2022-07-29 2022-09-27 济南浪潮数据技术有限公司 一种硬盘健康信息的获取方法及相关组件
CN115567526A (zh) * 2022-09-21 2023-01-03 中国平安人寿保险股份有限公司 数据监控方法、装置、设备及介质
CN115686450A (zh) * 2022-10-08 2023-02-03 浪潮软件股份有限公司 一种服务大厅系统监控预警的方法及监控预警系统
CN115766770A (zh) * 2022-10-26 2023-03-07 中银金融科技有限公司 Gbase数据库的事前预警系统
CN115878274A (zh) * 2021-09-27 2023-03-31 北京字跳网络技术有限公司 告警任务调度方法、装置、电子设备及计算机可读存储介质
US20230108213A1 (en) * 2021-10-05 2023-04-06 Softiron Limited Ceph Failure and Verification
CN115934464A (zh) * 2022-12-13 2023-04-07 浪潮云信息技术股份公司 一种信息化平台监控采集系统
CN115996180A (zh) * 2022-12-01 2023-04-21 深圳前海环融联易信息科技服务有限公司 监控告警系统、方法、设备和计算机存储介质
CN116302838A (zh) * 2023-03-31 2023-06-23 北京青云科技股份有限公司 一种系统监控方法、装置、电子设备和存储介质
CN116302788A (zh) * 2023-01-06 2023-06-23 浪潮软件集团有限公司 一种自动监控Ceph PG实时分布情况的方法及系统
CN116599992A (zh) * 2023-05-16 2023-08-15 中国工商银行股份有限公司 对象的监控方法、装置、处理器以及电子设备
CN116701115A (zh) * 2023-06-06 2023-09-05 中数海量(中山)科技投资有限公司 国资业务数据上报监控系统和方法
CN118642908A (zh) * 2023-03-13 2024-09-13 赛诺青石(广州)营销科技有限公司 一种基于Spring的自定义监控服务健康状态的方法与系统
CN119094445A (zh) * 2024-08-30 2024-12-06 苏州元脑智能科技有限公司 一种节点流量调整方法、装置及设备
WO2025057233A1 (en) * 2023-09-13 2025-03-20 Jio Platforms Limited System and method for centralized monitoring of database ecosystems in a network

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111049705B (zh) * 2019-12-23 2023-09-12 深圳前海微众银行股份有限公司 一种监控分布式存储系统的方法及装置
CN111597091A (zh) * 2020-05-20 2020-08-28 北京金山云网络技术有限公司 数据监控方法及系统、电子设备、计算机存储介质
CN111625421B (zh) * 2020-05-26 2021-07-16 云和恩墨(北京)信息技术有限公司 监控分布式存储系统的方法及装置、存储介质和处理器
CN111988165B (zh) * 2020-07-09 2023-01-24 云知声智能科技股份有限公司 一种分布式存储系统使用情况的监控方法及系统
CN112084098A (zh) * 2020-10-21 2020-12-15 中国银行股份有限公司 资源监控系统及工作方法
CN112650642B (zh) * 2020-12-07 2024-08-20 深圳前海微众银行股份有限公司 一种告警处理方法及装置、设备、存储介质
CN112751726B (zh) * 2020-12-17 2022-09-09 北京达佳互联信息技术有限公司 一种数据处理方法、装置、电子设备和存储介质
CN112783745A (zh) * 2021-02-02 2021-05-11 无锡车联天下信息技术有限公司 一种集群数据的监控方法、装置、系统及存储介质
CN113688149A (zh) * 2021-07-20 2021-11-23 青岛海尔科技有限公司 监控方法和装置
CN113641558A (zh) * 2021-08-31 2021-11-12 合众人寿保险股份有限公司 一种健康检查方法、装置及电子设备
CN114090644B (zh) * 2022-01-20 2022-04-26 飞狐信息技术（天津）有限公司 一种数据处理方法及装置
CN115714711A (zh) * 2022-11-02 2023-02-24 中国建设银行股份有限公司 一种基于容器云平台的告警信息的处理方法及装置
CN115865756A (zh) * 2022-11-29 2023-03-28 中国建设银行股份有限公司 服务器集群的监测方法、相关装置及计算机存储介质
CN116170284B (zh) * 2023-02-24 2025-09-09 济南浪潮数据技术有限公司 一种客户端的故障优化方法、系统、设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107291594A (zh) * 2017-06-30 2017-10-24 上海白虹软件科技股份有限公司 openstack平台对ceph进行监控和管理的装置及方法
US20180341682A1 (en) * 2017-05-26 2018-11-29 Nutanix, Inc. System and method for generating rules from search queries
CN109298945A (zh) * 2018-10-17 2019-02-01 北京京航计算通讯研究所 面向大数据平台的Ceph分布式存储监控与调优管理方法
CN109522287A (zh) * 2018-09-18 2019-03-26 平安科技（深圳）有限公司 分布式文件存储集群的监控方法、系统、设备及介质
CN111049705A (zh) * 2019-12-23 2020-04-21 深圳前海微众银行股份有限公司 一种监控分布式存储系统的方法及装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202212A (zh) * 2014-08-28 2014-12-10 浪潮(北京)电子信息产业有限公司 一种获取分布式集群系统告警的系统和方法
CN107864063B (zh) * 2017-12-12 2021-09-17 北京奇艺世纪科技有限公司 一种异常监控方法、装置及电子设备
US11102174B2 (en) * 2017-12-26 2021-08-24 Palo Alto Networks, Inc. Autonomous alerting based on defined categorizations for network space and network boundary changes

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180341682A1 (en) * 2017-05-26 2018-11-29 Nutanix, Inc. System and method for generating rules from search queries
CN107291594A (zh) * 2017-06-30 2017-10-24 上海白虹软件科技股份有限公司 openstack平台对ceph进行监控和管理的装置及方法
CN109522287A (zh) * 2018-09-18 2019-03-26 平安科技（深圳）有限公司 分布式文件存储集群的监控方法、系统、设备及介质
CN109298945A (zh) * 2018-10-17 2019-02-01 北京京航计算通讯研究所 面向大数据平台的Ceph分布式存储监控与调优管理方法
CN111049705A (zh) * 2019-12-23 2020-04-21 深圳前海微众银行股份有限公司 一种监控分布式存储系统的方法及装置

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114115718A (zh) * 2021-08-31 2022-03-01 济南浪潮数据技术有限公司 分布式块存储系统服务质量控制方法、装置、设备及介质
CN114115718B (zh) * 2021-08-31 2024-03-29 济南浪潮数据技术有限公司 分布式块存储系统服务质量控制方法、装置、设备及介质
CN115878274A (zh) * 2021-09-27 2023-03-31 北京字跳网络技术有限公司 告警任务调度方法、装置、电子设备及计算机可读存储介质
US12174713B2 (en) * 2021-10-05 2024-12-24 Softiron Limited Ceph failure and verification
US20230108213A1 (en) * 2021-10-05 2023-04-06 Softiron Limited Ceph Failure and Verification
CN113900898A (zh) * 2021-10-19 2022-01-07 北京金山云网络技术有限公司 一种数据处理系统、设备及介质
CN114595126A (zh) * 2022-03-24 2022-06-07 山西合力创新科技股份有限公司 一种基于规则引擎的告警方法、系统、存储设备及终端
CN114760221A (zh) * 2022-03-31 2022-07-15 深信服科技股份有限公司 一种业务监控方法、系统和存储介质
CN114760221B (zh) * 2022-03-31 2024-02-23 深信服科技股份有限公司 一种业务监控方法、系统和存储介质
CN115118630A (zh) * 2022-07-29 2022-09-27 济南浪潮数据技术有限公司 一种硬盘健康信息的获取方法及相关组件
CN115567526A (zh) * 2022-09-21 2023-01-03 中国平安人寿保险股份有限公司 数据监控方法、装置、设备及介质
CN115567526B (zh) * 2022-09-21 2024-05-14 中国平安人寿保险股份有限公司 数据监控方法、装置、设备及介质
CN115686450A (zh) * 2022-10-08 2023-02-03 浪潮软件股份有限公司 一种服务大厅系统监控预警的方法及监控预警系统
CN115766770A (zh) * 2022-10-26 2023-03-07 中银金融科技有限公司 Gbase数据库的事前预警系统
CN115996180A (zh) * 2022-12-01 2023-04-21 深圳前海环融联易信息科技服务有限公司 监控告警系统、方法、设备和计算机存储介质
CN115934464A (zh) * 2022-12-13 2023-04-07 浪潮云信息技术股份公司 一种信息化平台监控采集系统
CN116302788A (zh) * 2023-01-06 2023-06-23 浪潮软件集团有限公司 一种自动监控Ceph PG实时分布情况的方法及系统
CN118642908A (zh) * 2023-03-13 2024-09-13 赛诺青石(广州)营销科技有限公司 一种基于Spring的自定义监控服务健康状态的方法与系统
CN116302838A (zh) * 2023-03-31 2023-06-23 北京青云科技股份有限公司 一种系统监控方法、装置、电子设备和存储介质
CN116599992A (zh) * 2023-05-16 2023-08-15 中国工商银行股份有限公司 对象的监控方法、装置、处理器以及电子设备
CN116701115A (zh) * 2023-06-06 2023-09-05 中数海量(中山)科技投资有限公司 国资业务数据上报监控系统和方法
WO2025057233A1 (en) * 2023-09-13 2025-03-20 Jio Platforms Limited System and method for centralized monitoring of database ecosystems in a network
CN119094445A (zh) * 2024-08-30 2024-12-06 苏州元脑智能科技有限公司 一种节点流量调整方法、装置及设备

Also Published As

Publication number Publication date
CN111049705B (zh) 2023-09-12
CN111049705A (zh) 2020-04-21

Similar Documents

Publication Publication Date Title
WO2021129367A1 (zh) 一种监控分布式存储系统的方法及装置
US10365915B2 (en) Systems and methods of monitoring a network topology
CN114756301B (zh) 日志处理方法、装置和系统
US11329869B2 (en) Self-monitoring
CN110532322B (zh) 运维交互方法、系统、计算机可读存储介质及设备
CN111078695B (zh) 计算企业内元数据关联关系的方法及装置
CN113626869B (zh) 数据处理方法、系统、电子设备以及存储介质
WO2017080161A1 (zh) 云计算中报警信息的处理方法及装置
WO2019169765A1 (zh) 电子装置、集群环境下状态信息的获取方法、系统及存储介质
CN105162632A (zh) 一种服务器集群故障自动处理系统
CN116028577A (zh) 数据同步方法及装置、电子设备和存储介质
CN120892295B (zh) 服务器硬件故障诊断方法、电子设备和存储介质
CN111917812B (zh) 数据的传输控制方法、装置、设备以及存储介质
CN109766238B (zh) 基于session数的运维平台性能监控方法、装置及相关设备
JP2011180805A (ja) 運用管理装置、運用管理方法、運用管理プログラム
CN114490003A (zh) 大规模数据的分布式作业调度方法及相关设备
CN109302723B (zh) 一种基于互联网的多节点实时无线电监测控制方法
CN110502424A (zh) 一种应用软件的性能数据处理方法、装置、系统及终端
CN113590424B (zh) 一种故障监控方法、装置、设备及存储介质
CN115834353A (zh) 故障自愈方法、装置、设备、存储介质和计算机程序产品
CN116260703A (zh) 分布式消息服务节点cpu性能故障自恢复方法及装置
US10949232B2 (en) Managing virtualized computing resources in a cloud computing environment
US20250217177A1 (en) System and method for correlating virtual machine interruptions and node characteristics
CN118170617B (zh) 集群异常的检测方法及装置、电子设备和存储介质
CN114048068B (zh) Mysql数据库的运维方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20905266

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 02-11-22)

122 Ep: pct application non-entry in european phase

Ref document number: 20905266

Country of ref document: EP

Kind code of ref document: A1