CN112965879A - Data processing method and device, electronic equipment and readable storage medium - Google Patents
Data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112965879A CN112965879A CN202110286932.7A CN202110286932A CN112965879A CN 112965879 A CN112965879 A CN 112965879A CN 202110286932 A CN202110286932 A CN 202110286932A CN 112965879 A CN112965879 A CN 112965879A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- service
- monitored
- monitoring system
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a data processing method, a data processing device, electronic equipment and a readable storage medium, and belongs to the technical field of networks. In the method, a monitoring system is stripped from a service cluster, and information to be monitored issued by the service cluster is acquired based on a first monitoring service in the monitoring system; the information to be monitored comprises indexes to be monitored, target operation data matched with the indexes to be monitored are obtained from operation data of the service cluster based on the monitoring examples established in the monitoring system, the service cluster is monitored according to the target operation data, and the monitoring examples established in the monitoring system are adjusted based on the current load related information of the monitoring system. Therefore, the monitoring system and the service are decoupled, so that monitoring personnel can conveniently and independently manage the monitoring cluster, and the management difficulty of the monitoring system can be reduced to a certain extent, so that the service operation and maintenance pressure can be reduced.
Description
Technical Field
The present invention relates to the field of network technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
Currently, in order to ensure normal operation of a service cluster, a monitoring system is used to monitor the service cluster. In the prior art, in order to implement monitoring, a set of corresponding monitoring system is often deployed in each service cluster, and monitoring is performed by using the monitoring system deployed in each service cluster.
In this way, the management difficulty of the monitoring system is high, and the pressure of business operation and maintenance is high.
Disclosure of Invention
The invention provides a data processing method, a data processing device, electronic equipment and a readable storage medium, which are used for solving the problems of high management difficulty and high service operation and maintenance pressure of a monitoring system.
According to a first aspect of the present invention, there is provided a data processing method, where the method is applied to a monitoring system, the monitoring system is deployed in a monitoring cluster, and the monitoring cluster and a service cluster are deployed in different server clusters, and the method includes:
acquiring information to be monitored issued by the service cluster based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
acquiring target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring example established in the monitoring system, and monitoring the service cluster according to the target operation data;
and adjusting the created monitoring examples in the monitoring system based on the current load related information of the monitoring system.
According to a second aspect of the present invention, there is provided a data processing apparatus, which is applied to a monitoring system, wherein the monitoring system is deployed in a monitoring cluster, and the monitoring cluster and a service cluster are deployed in different server clusters, the apparatus comprising:
a first obtaining module, configured to obtain information to be monitored issued by the service cluster based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
a second obtaining module, configured to obtain target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring instance created in the monitoring system, and monitor the service cluster according to the target operation data;
and the first adjusting module is used for adjusting the created monitoring instance in the monitoring system based on the current load related information of the monitoring system.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to execute any of the data processing methods described above.
In yet another aspect of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the data processing methods described above.
Aiming at the prior art, the invention has the following advantages:
the monitoring system is stripped from the service cluster, and information to be monitored issued by the service cluster is acquired based on a first monitoring service in the monitoring system; the information to be monitored comprises indexes to be monitored, target operation data matched with the indexes to be monitored are obtained from operation data of the service cluster based on the monitoring examples established in the monitoring system, the service cluster is monitored according to the target operation data, and the monitoring examples established in the monitoring system are adjusted based on the current load related information of the monitoring system. Therefore, the monitoring system and the service are decoupled, so that monitoring personnel can conveniently and independently manage the monitoring cluster, and the management difficulty of the monitoring system can be reduced to a certain extent, so that the service operation and maintenance pressure can be reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow chart of steps of a data processing method according to an embodiment of the present invention;
FIG. 2 is a diagram of an implementation architecture provided by an embodiment of the present invention;
FIG. 3 is a system block diagram provided by an embodiment of the present invention;
FIG. 4 is a schematic diagram of a scenario provided by an embodiment of the present invention;
fig. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is a block diagram of another data processing apparatus provided by an embodiment of the present invention;
FIG. 7 is a block diagram of another data processing apparatus provided by an embodiment of the present invention;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
First, a part of the concept related to the embodiment of the present invention is explained:
and the service cluster refers to a server cluster formed by servers for realizing various functions of the application program/webpage.
The monitoring cluster refers to a server cluster formed by servers with deployed monitoring systems.
The monitoring system is used for monitoring specific indexes of the service cluster and giving an alarm when the specific indexes are abnormal.
Fig. 1 is a flowchart of steps of a data processing method provided in an embodiment of the present invention, where the method may be applied to a monitoring system, the monitoring system may be deployed in a monitoring cluster, and the monitoring cluster and a service cluster are deployed in different server clusters, as shown in fig. 1, the method may include:
In the embodiment of the present invention, the first monitoring service may be pre-registered by the monitoring system, and the first monitoring service may be a process in nature. The information to be monitored can be used for representing which indexes of the service cluster need to be monitored, and the specific content of the information to be monitored can be set by service personnel according to actual conditions. For example, the index to be monitored may include Central Processing Unit (CPU) occupancy, error rate, Processing speed, and the like.
Further, a service person may issue information to be monitored in a service cluster according to an actual requirement, for example, in an application scenario where a deployment management tool (e.g., kubernets) of a portable container is deployed in the service cluster, the service cluster may issue the information to be monitored through a preset component (e.g., ServiceMonitor) in the kubernets.
The Kubernetes is an important project of a cloud native technology and is a management platform for running containerized applications, accordingly, the monitoring system in the embodiment of the present invention may be implemented based on a monitoring system "Prometheus" matched with the monitoring system, and the monitoring system in the embodiment of the present invention may be implemented based on a preset development tool (for example, Java, gold, Python, and other tools) in a preset development environment (for example, a Windows development environment, a Linux development environment, and other development environments).
Further, different servicemonitors can interface different services, and different services can represent different functions of an application or a web page, and taking a video application as an example, the services can include a video playing service, a video comment service, a video mall service, and the like.
The service personnel can determine the actual requirement based on the data concurrency quantity of each service and input the corresponding information to be monitored, and the corresponding ServiceMonitor can issue the information to be monitored input by the service personnel. Correspondingly, the monitoring system can actively monitor the information to be monitored issued in the service cluster based on the first monitoring service, so as to obtain the information to be monitored. The first monitoring service may be a process in nature, and when the service cluster issues the information to be monitored, the service cluster may issue the information to be monitored to a specific interface of the monitoring cluster. The monitoring system may monitor the designated interface based on the process, and when detecting that the information is transmitted through the designated interface, acquire the information transmitted through the designated interface as information to be monitored.
102, obtaining target operation data matched with the index to be monitored from the operation data of the service cluster based on the created monitoring example in the monitoring system, and monitoring the service cluster according to the target operation data.
In the embodiment of the present invention, the monitoring instance may be a process generated by the monitoring system using the computing resource of the monitoring cluster and used for monitoring. The operation data of the service cluster can be various data of the service cluster in the operation process. For example, the operation data may include CPU usage, processing speed, error rate, latency, and the like at various times. The target operation data may be operation data corresponding to an index to be monitored.
Further, the service cluster is monitored according to the target operation data, and whether the index to be monitored is abnormal or not can be monitored based on the target operation data, so that an alarm is given under the abnormal condition. For example, whether the target operation data hits the preset alarm condition may be detected, and if the target operation data hits the preset alarm condition, the monitoring alarm information may be output. For example, the alarm information is distributed to corresponding business operation and maintenance personnel.
In the embodiment of the present invention, the load-related information may be information that may affect the load size of the monitoring instance in the monitoring system, and the specific type of the load-related information may be set according to actual requirements. If the load of the monitoring instance is too large, the monitoring operation may not be performed normally. If the load of the monitoring instance is too small, the monitoring resource may not be fully utilized, and the problem of resource waste occurs. Therefore, the monitoring examples in the monitoring system can be adaptively adjusted based on the load related information, so that the problem of resource waste caused by oversaturation of the processing amount of the monitoring examples or low utilization rate of the monitoring examples is avoided, and the resource utilization rate is further improved.
It should be noted that the execution sequence between the above steps is not exclusive. For example, step 103 may be performed before step 102, or may be performed before step 101, synchronously with step 101, and the like, which is not limited by the embodiment of the present invention.
In summary, in the data processing method provided in the embodiment of the present invention, the monitoring system is separated from the service cluster, and information to be monitored issued by the service cluster is acquired based on the first monitoring service in the monitoring system; the information to be monitored comprises indexes to be monitored, target operation data matched with the indexes to be monitored are obtained from operation data of the service cluster based on the monitoring examples established in the monitoring system, the service cluster is monitored according to the target operation data, and the monitoring examples established in the monitoring system are adjusted based on the current load related information of the monitoring system. Therefore, the monitoring system and the service are decoupled, so that monitoring personnel can conveniently and independently manage the monitoring cluster, and the management difficulty of the monitoring system can be reduced to a certain extent, so that the service operation and maintenance pressure can be reduced.
Meanwhile, based on the current load related information of the monitoring system, the created monitoring instance in the monitoring system is adaptively adjusted, so that the monitoring instance can be ensured to adapt to the current requirement, and the monitoring effect is further improved to a certain extent.
Optionally, the step 103 of adjusting the created monitoring instance in the monitoring system based on the current load related information of the monitoring system may include the following steps:
step 1031, obtaining the load related information; the load related information comprises the number of nodes of the service cluster; the larger the number of nodes, the larger the load of each monitoring instance in the monitoring system.
In the embodiment of the present invention, the number of the nodes of the service cluster may be the total number of the servers included in the service cluster docked by the monitoring system. If the number of the nodes is larger, the processing capacity of the monitoring system is correspondingly larger, and further the load of each monitoring instance in the monitoring system is larger. Conversely, if the number of nodes is smaller, the processing amount of the monitoring system is correspondingly smaller, and thus the load of each monitoring instance in the monitoring system is smaller. Therefore, in the embodiment of the invention, the number of the nodes of the service cluster is set as the load related information, so that the accuracy of subsequent adjustment operation can be ensured to a certain extent.
Further, the load related information may also include other information, for example, the load related information may include load conditions of a Central Processing Unit (CPU), a memory, a network, a disk, etc. of the monitored object, and information of a request amount, a corresponding time, a request error condition, etc. of the service. The load related information may further include the traffic volume of each service cluster and/or the number of types of the index to be monitored.
The number of adjusted monitoring instances may be matched to the amount of traffic and the number of categories. Wherein, the traffic volume may represent the number of services interfaced by the service cluster. If the traffic volume is more, the service cluster is more active correspondingly, the generated operation data is more, and the load of each monitoring instance in the monitoring system is larger. The number of monitoring instances can be correspondingly reduced. Conversely, if the amount of traffic is smaller, the generated operation data will be correspondingly smaller, and the load of each monitoring instance in the monitoring system will be smaller. The number of monitoring instances can be correspondingly increased. If the number of the types of the indexes to be monitored is larger, the processing capacity of the monitoring system is correspondingly larger, and the load of each monitoring instance in the monitoring system is larger. The number of monitoring instances can be correspondingly reduced. Conversely, if the number of the types of the indexes to be monitored is smaller, the processing amount of the monitoring system is correspondingly smaller, and thus the load of each monitoring instance in the monitoring system is smaller. The number of monitoring instances can be correspondingly increased. That is, in the embodiment of the present invention, when the number of the types of the traffic volume and/or the monitoring index of the traffic cluster is larger, the number of the adjusted monitoring instances may be larger. According to the embodiment of the invention, the operation precision of the subsequently executed adjustment operation can be further improved by further combining the service volume and the type number of the indexes to be monitored, and the monitoring resources are more reasonably distributed to the service. Furthermore, the monitoring cluster in the embodiment of the invention can be connected with a plurality of service clusters, so that the available monitoring resources corresponding to the service clusters can be conveniently integrated, multi-service monitoring can be conveniently realized, and the monitoring management efficiency is improved. And the subsequent scheduling of the monitoring resources is facilitated. Wherein the monitoring resource can be characterized by a monitoring instance.
Optionally, when the load related information is acquired in the embodiment of the present invention, the load related information input by the user may be received. Therefore, the load related information can be acquired only by executing the receiving operation, and the acquisition efficiency can be improved to a certain extent. The monitoring system may provide an input interface to the user, which the user may access through the terminal used and enter load related information in the interface.
Or, the method may also be to send an acquisition request to the service cluster, and then receive load-related information returned by the service cluster based on the acquisition request. The service cluster can confirm that the monitoring system needs to acquire the load related information currently under the condition of receiving the acquisition request sent by the monitoring system. Accordingly, the service cluster may read the preconfigured load-related information from the configuration information and return the preconfigured load-related information to the monitoring system. Therefore, manual operation can be simplified by sending an acquisition request to acquire load related information from the service cluster in real time without manual participation.
Step 1032, adjusting the created monitoring instances in the monitoring system according to the load related information, so that the number of the adjusted monitoring instances is matched with the load related information.
For example, the amount of adaptation to the current load-related information may be determined first according to the current load-related information. The number of monitoring instances is then adjusted to a number that is adapted to the current load-related information. The number of adjusted monitoring instances may be larger in case the load amount of the monitoring system represented by the load related information is larger.
Specifically, when the adaptive quantity is determined according to the load-related information, the load-related information may be used as an input of a preset determination function, and then an output of the function is used as the adaptive quantity, where the preset load determination function is a function in which the dependent variable and the independent variable are positively correlated. Or, the number matched with the current load related information may be searched according to the corresponding relationship between the preset number and the load related information, so as to obtain the adapted number. Of course, other determination manners may be adopted, and the embodiment of the present invention is not limited thereto.
In the embodiment of the invention, load related information is obtained; and the load related information comprises the number of the nodes of the service cluster, and the monitoring examples established in the monitoring system are adjusted according to the load related information so as to enable the number of the adjusted monitoring examples to be matched with the load related information. Therefore, when the nodes of the service cluster change, the monitoring party can automatically adjust the adaptability, and further the problems that monitoring operation cannot be normally performed due to too large load of a monitoring instance or monitoring resources cannot be fully utilized due to too small load can be avoided. Meanwhile, through self-adaptive adjustment, manual adjustment is not needed, and operation and maintenance cost can be reduced to a certain extent.
Optionally, in the embodiment of the present invention, after the step 102, the following steps a to B may be further performed:
step A, acquiring monitoring resource information issued by a management node in the monitoring cluster based on a second monitoring service; wherein the monitoring resource information is used for indicating the target number of the monitoring instances.
In the embodiment of the present invention, the management node may be a node that is pre-designated in the monitoring cluster according to actual requirements. The second listening service may be pre-registered by the monitoring system and the second listening service may be a process in nature. The second monitoring service and the first monitoring service may be two independent processes, and the second monitoring service may be configured to perform internal monitoring on the monitoring cluster. The monitoring system can actively monitor the monitoring resource information issued inside the monitoring cluster based on the second monitoring service so as to obtain the information to be monitored. The target number may be set according to actual requirements, for example, the target number may be set by the monitoring operation and maintenance staff according to actual changes of the service party.
For example, in one implementation, a controller may be included in the monitoring system, and the controller may register, as the first monitoring service, a monitoring service whose monitoring type is remote monitoring with the service cluster, and register, as the second monitoring service, a monitoring service whose monitoring type is local monitoring with the monitoring cluster. The controller may be a control node that is pre-designated in the monitoring system, for example, the controller may be "proxy-Operator", and in the embodiment of the present invention, development and improvement may be performed on a processing logic of the controller in advance, so that the controller has an ability to perform a registration operation, and further the monitoring system may subsequently monitor the service cluster and the monitoring cluster at the same time, for example, the controller in the monitoring system may monitor information that is issued to the monitoring cluster by the service cluster based on remote monitoring of registration. The registration monitoring may be performed in a preset manner, for example, an anonymous manner may be used to register the remote/local monitoring, or an interface manner may be used to register the remote/local monitoring, and certainly, other manners may also be used to perform registration, which is not limited in the embodiment of the present invention. In the embodiment of the invention, the remote monitoring and the local monitoring are registered in advance, so that the monitoring system can conveniently acquire the issuing of the service cluster/monitoring cluster, and further the information acquisition efficiency is ensured.
And B, adjusting the created monitoring examples in the monitoring system according to the monitoring resource information so as to enable the number of the adjusted monitoring examples to be matched with the target number.
In this step, the monitoring resource information may be the target quantity itself or information carrying the target quantity. Further, it may be determined whether the number of current monitoring instances is the same as the target number. If not the same and the number of current monitoring instances is less than the target number, a new monitoring instance may be created until the number of current monitoring instances is the same as the target number. If the number of the current monitoring instances is not the same as the target number, part of the monitoring instances can be deleted until the number of the current monitoring instances is the same as the target number. Of course, if the number of the current monitoring instances is the same as the target number, the adjustment operation may not be performed to avoid performing an unnecessary operation, resulting in a problem that resources are wasted.
In the embodiment of the invention, the monitoring resource information is monitored through the second monitoring service, and the number of the monitoring instances in the monitoring system is adjusted to the target number of the monitoring instances indicated by the monitoring resource information, so that a user can conveniently control the number of the monitoring instances by only internally issuing from the monitoring cluster. Compared with a mode of coupling and deploying the monitoring system and the service cluster and performing issuing control from the interior of the service cluster, the method and the system can improve the safety to a certain extent and increase the independence between the monitoring system and the service cluster.
Optionally, in the embodiment of the present invention, the information to be monitored may further include data acquisition interface information corresponding to each index to be monitored. Correspondingly, the data processing method may further include: and determining the indexes to be monitored corresponding to the monitoring examples.
The data acquisition interfaces corresponding to the indexes to be monitored can be set according to actual requirements, and operation data corresponding to different indexes to be monitored can be stored in different data sources in an actual application scene, so that the data acquisition interfaces corresponding to the indexes to be monitored can be set in the information to be monitored, and the operation data can be acquired subsequently. The data acquisition interface corresponding to the index to be monitored can be an interface of a data source for storing the operating data corresponding to the index to be monitored.
Further, the to-be-monitored indexes corresponding to the monitoring instances may be divided for the monitoring instances in advance, for example, the controller may randomly allocate the corresponding to-be-monitored indexes to the monitoring instances, and then record the to-be-monitored indexes allocated to the monitoring instances. And when the indexes to be monitored corresponding to the monitoring examples are determined, the recorded indexes to be monitored corresponding to the monitoring examples can be directly read.
Accordingly, the step 102 of obtaining the target operation data matched with the index to be monitored from the operation data of the service cluster based on the monitoring instance created in the monitoring system may specifically include:
step 1021, for any monitoring instance, pulling operation data related to the index to be monitored from a target data acquisition interface corresponding to the monitoring instance to obtain the target operation data; the target data acquisition interface is the data acquisition interface indicated by the data acquisition interface information of the index to be monitored.
Illustratively, the monitoring instance may access the target data collection interface to pull operational data. In the embodiment of the invention, the target operation data can be obtained by defining the data acquisition interface for the index to be monitored, determining the index to be monitored corresponding to each monitoring example, and directly pulling the operation data related to the index to be monitored from the target data acquisition interface corresponding to the monitoring example when the target operation data is obtained. Thus, the data acquisition efficiency can be improved to a certain extent.
It should be noted that, in the embodiment of the present invention, a related promemeus resource may also be generated in the monitoring cluster based on an Operator according to a load condition, where the related promemeus resource is used to describe the number of instances adapted to the current load, and the monitoring instances are automatically deployed and managed according to the number of instances. Or, the current load capacity of the monitoring instance may be detected in real time, if the current load capacity is lower than the preset range, part of the monitoring instance is deleted, and if the current load capacity is higher than the preset range, a new monitoring instance is created, so that the current load capacity is within the preset range. Thus, the problems of resource waste or overload of the monitoring instance can be avoided to a greater extent. Further, after a new monitoring instance is created, a part of indexes to be monitored corresponding to the original monitoring instance may be allocated to the new monitoring instance, and after a part of monitoring instances is deleted, the indexes to be monitored corresponding to the deleted monitoring instances may be allocated to the monitoring instances that are not deleted, so as to ensure the comprehensiveness of the monitoring.
In an existing implementation manner, different service clusters apply for different monitoring resources, and a conventional monitoring scheme is to deploy one monitoring system in each service cluster. This results in the coupling of the service cluster with the monitoring cluster. With the increase of the number of the service clusters, the management difficulty of the monitoring team is increased, and meanwhile, due to the coupling deployment of the service clusters and the monitoring team, the configuration of the service is exposed to the monitoring team, and the safety is low.
Fig. 2 is an implementation architecture diagram provided in an embodiment of the present invention, and as shown in fig. 2, a data storage party 01 may be configured to store operation data and monitoring results generated by service clusters, and data of each service cluster may be stored for a long time in a shared storage manner. The monitoring system 021 in the monitoring cluster 02 includes a controller 0211 and a monitoring instance 0212, wherein RWX is used for indicating read-write right information, and 2Gi/4Gi is used for indicating the amount of space allocated by the monitoring instance. The monitoring cluster 02 may be configured to monitor the service cluster 03 and the service cluster 04, push alarm information when an abnormal condition is detected, and display and output the alarm information through the alarm display party 05.
Further, fig. 3 is a system structure diagram provided in the embodiment of the present invention, and as shown in fig. 3, services 1 to 5 represent services that are interfaced by preset components (servicemonitors). The controller in the monitoring cluster monitors the service cluster and the monitoring cluster, and the controller can deploy and manage a monitoring instance (e.g., a Promultimedia Server) according to the monitored information. The monitoring resource information in fig. 3 may be denoted as "Prometheus". It should be noted that, in the embodiment of the present invention, the issued monitoring content, that is, which services the indexes to be monitored specifically correspond to, may be further determined to the preset component, so as to guide the management and deployment of the monitoring instance according to the preset corresponding relationship between the services and the monitoring instance. For example, the number of monitoring instances corresponding to each service may be determined based on the correspondence, and when the number of existing monitoring instances corresponding to the service does not conform to the number of corresponding monitoring instances, the existing monitoring instances may be adjusted.
Further, fig. 4 is a schematic view of a scenario provided in the embodiment of the present invention, as shown in fig. 4, a service operation and maintenance worker may issue a monitoring policy to a service cluster, so that the service cluster performs monitoring and issuing according to the monitoring policy, that is, issues information to be monitored. Accordingly, the controller of the monitoring cluster may listen to a calling interface service (API Server) of the service cluster to listen to the service cluster. Furthermore, monitoring deployment or capacity expansion can be performed according to the monitored information, that is, deployment and capacity expansion are performed on the monitoring instance, meanwhile, service data can be collected from an exposed interface of the service cluster, and when the collected data is abnormal, an alarm is given. And finally, sending an alarm notice to the service operation and maintenance personnel.
In the embodiment of the invention, the monitoring cluster is separated from the service cluster, and the monitoring cluster is independently deployed outside the service cluster, so that the monitoring cluster of each service cluster can be conveniently managed by a unified monitoring team, and correspondingly, the service team only needs to operate and maintain the own service cluster, so that the infrastructure (monitoring cluster) can be charged by the unified team, and the service team only charges the own service operation and maintenance, thereby realizing the foundation operation and maintenance sinking. Meanwhile, one monitoring cluster is adopted to be connected with a plurality of service clusters in a butt joint mode, so that the monitoring cluster can reasonably distribute monitoring resources according to the actual load condition.
Fig. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention, where the apparatus is applied to a monitoring system, the monitoring system is deployed in a monitoring cluster, and the monitoring cluster and a service cluster are deployed in different server clusters, where the apparatus 50 includes:
a first obtaining module 501, configured to obtain information to be monitored, which is issued by the service cluster, based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
a second obtaining module 502, configured to obtain target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring instance created in the monitoring system, and monitor the service cluster according to the target operation data;
a first adjusting module 503, configured to adjust a monitoring instance created in the monitoring system based on the current load related information of the monitoring system.
In summary, in the data processing apparatus provided in the embodiment of the present invention, by detaching the monitoring system from the service cluster, the information to be monitored issued by the service cluster is obtained based on the first monitoring service in the monitoring system; the information to be monitored comprises indexes to be monitored, target operation data matched with the indexes to be monitored are obtained from operation data of the service cluster based on the monitoring examples established in the monitoring system, the service cluster is monitored according to the target operation data, and the monitoring examples established in the monitoring system are adjusted based on the current load related information of the monitoring system. Therefore, the monitoring system and the service are decoupled, so that monitoring personnel can conveniently and independently manage the monitoring cluster, and the management difficulty of the monitoring system can be reduced to a certain extent, so that the service operation and maintenance pressure can be reduced.
Optionally, the first adjusting module 503 is specifically configured to:
acquiring the load related information; the load related information comprises the number of nodes of the service cluster; the more the number of the nodes is, the larger the load of each monitoring instance in the monitoring system is;
and adjusting the created monitoring examples in the monitoring system according to the load related information so as to match the number of the adjusted monitoring examples with the load related information.
Optionally, fig. 6 is a block diagram of another data processing apparatus provided in an embodiment of the present invention, and as shown in fig. 6, the apparatus 50 further includes:
a third obtaining module 504, configured to obtain, based on the second monitoring service, monitoring resource information issued by a management node in the monitoring cluster; wherein the monitoring resource information is used for indicating the target number of the monitoring instances;
a second adjusting module 505, configured to adjust the monitoring instances created in the monitoring system according to the monitoring resource information, so that the number of the adjusted monitoring instances matches the target number.
Optionally, the apparatus 50 further comprises a controller;
the controller registers, as the first monitoring service, a monitoring service whose monitoring type is remote monitoring to the service cluster, and registers, as the second monitoring service, a monitoring service whose monitoring type is local monitoring to a management node in the monitoring cluster.
Optionally, the first adjusting module 503 is further specifically configured to:
receiving the load-related information input by a user; alternatively, the first and second electrodes may be,
sending an acquisition request to the service cluster; and receiving the load related information returned by the service cluster based on the acquisition request.
Optionally, the load related information further includes a traffic volume of each service cluster and/or a type number of the to-be-monitored indicator; the number of the adjusted monitoring instances is matched with the traffic volume and the number of the categories.
Optionally, the information to be monitored further includes data acquisition interface information corresponding to each index to be monitored; fig. 7 is a block diagram of another data processing apparatus according to an embodiment of the present invention, and as shown in fig. 7, the apparatus 50 further includes: and the determining module 507 is configured to determine the to-be-monitored index corresponding to each monitoring instance. Optionally, the second obtaining module 502 is specifically configured to: for any monitoring example, pulling operation data related to the index to be monitored from a target data acquisition interface corresponding to the monitoring example to obtain the target operation data; the target data acquisition interface is the data acquisition interface indicated by the data acquisition interface information of the index to be monitored.
For the above device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant points, refer to the partial description of the method embodiment.
The embodiment of the present invention further provides an electronic device, as shown in fig. 8, which includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904.
A memory 903 for storing computer programs;
the processor 901 is configured to implement the following steps when executing the program stored in the memory 903:
acquiring information to be monitored issued by the service cluster based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
acquiring target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring example established in the monitoring system, and monitoring the service cluster according to the target operation data;
and adjusting the created monitoring examples in the monitoring system based on the current load related information of the monitoring system.
The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the terminal and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the instructions are executed on a computer, the computer is caused to execute the data processing method described in any one of the above embodiments.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the data processing method of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A data processing method is applied to a monitoring system, the monitoring system is deployed in a monitoring cluster, the monitoring cluster and a service cluster are deployed in different server clusters, and the method comprises the following steps:
acquiring information to be monitored issued by the service cluster based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
acquiring target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring example established in the monitoring system, and monitoring the service cluster according to the target operation data;
and adjusting the created monitoring examples in the monitoring system based on the current load related information of the monitoring system.
2. The method of claim 1, wherein the adjusting the created monitoring instances in the monitoring system based on the current load related information of the monitoring system comprises:
acquiring the load related information; the load related information comprises the number of nodes of the service cluster; the more the number of the nodes is, the larger the load of each monitoring instance in the monitoring system is;
and adjusting the created monitoring examples in the monitoring system according to the load related information so as to match the number of the adjusted monitoring examples with the load related information.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
acquiring monitoring resource information issued by a management node in the monitoring cluster based on a second monitoring service; wherein the monitoring resource information is used for indicating the target number of the monitoring instances;
and adjusting the created monitoring examples in the monitoring system according to the monitoring resource information so as to enable the number of the adjusted monitoring examples to be matched with the target number.
4. The method of claim 3, wherein the monitoring system includes a controller; the method further comprises the following steps:
the controller registers, as the first monitoring service, a monitoring service whose monitoring type is remote monitoring to the service cluster, and registers, as the second monitoring service, a monitoring service whose monitoring type is local monitoring to a management node in the monitoring cluster.
5. The method of claim 2, wherein the obtaining the load-related information comprises:
receiving the load-related information input by a user; alternatively, the first and second electrodes may be,
sending an acquisition request to the service cluster; and receiving the load related information returned by the service cluster based on the acquisition request.
6. The method according to claim 5, wherein the load-related information further includes a traffic volume of each service cluster and/or a type number of the index to be monitored;
the number of the adjusted monitoring instances is matched with the traffic volume and the number of the categories.
7. The method according to claim 1, wherein the information to be monitored further includes data acquisition interface information corresponding to each index to be monitored; the method further comprises the following steps: determining indexes to be monitored corresponding to the monitoring examples;
the acquiring, from the operation data of the service cluster, target operation data matched with the index to be monitored based on the monitoring instance created in the monitoring system includes:
for any monitoring example, pulling operation data related to the index to be monitored from a target data acquisition interface corresponding to the monitoring example to obtain the target operation data; the target data acquisition interface is the data acquisition interface indicated by the data acquisition interface information of the index to be monitored.
8. A data processing device is applied to a monitoring system, the monitoring system is deployed in a monitoring cluster, the monitoring cluster and a service cluster are deployed in different server clusters, and the device comprises:
a first obtaining module, configured to obtain information to be monitored issued by the service cluster based on a first monitoring service; the information to be monitored comprises indexes to be monitored;
a second obtaining module, configured to obtain target operation data matched with the index to be monitored from operation data of the service cluster based on a monitoring instance created in the monitoring system, and monitor the service cluster according to the target operation data;
and the first adjusting module is used for adjusting the created monitoring instance in the monitoring system based on the current load related information of the monitoring system.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110286932.7A CN112965879A (en) | 2021-03-17 | 2021-03-17 | Data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110286932.7A CN112965879A (en) | 2021-03-17 | 2021-03-17 | Data processing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112965879A true CN112965879A (en) | 2021-06-15 |
Family
ID=76279050
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110286932.7A Pending CN112965879A (en) | 2021-03-17 | 2021-03-17 | Data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112965879A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114760221A (en) * | 2022-03-31 | 2022-07-15 | 深信服科技股份有限公司 | Service monitoring method, system and storage medium |
| CN114866551A (en) * | 2022-06-06 | 2022-08-05 | 中国工商银行股份有限公司 | Prometous cluster processing method, device, equipment and storage medium |
| CN114900444A (en) * | 2022-04-25 | 2022-08-12 | 北京奇艺世纪科技有限公司 | Data monitoring method and device, electronic equipment and storage medium |
| CN115499299A (en) * | 2022-09-13 | 2022-12-20 | 航天信息股份有限公司 | Cluster equipment monitoring method and device |
| CN115865644A (en) * | 2022-11-09 | 2023-03-28 | 浪潮通信信息系统有限公司 | Service fault processing method, device, equipment and storage medium |
| CN117014284A (en) * | 2023-10-08 | 2023-11-07 | 北京凌云雀科技有限公司 | Inspection system, inspection method, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876932A (en) * | 2009-11-30 | 2010-11-03 | 中国移动通信集团浙江有限公司 | Method, system and device for monitoring internal storage database |
| US20120221690A1 (en) * | 2011-02-25 | 2012-08-30 | International Business Machines Corporation | Data Processing Environment Monitoring |
| US20120317274A1 (en) * | 2011-06-13 | 2012-12-13 | Richter Owen E | Distributed metering and monitoring system |
| CN109189643A (en) * | 2018-09-11 | 2019-01-11 | 五八有限公司 | Monitoring method, device, equipment and the storage medium of request interface response data |
| CN111522723A (en) * | 2020-04-17 | 2020-08-11 | 杭州威佩网络科技有限公司 | Data acquisition method and device |
| CN111708676A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Example cluster monitoring method and device and server |
| CN112437145A (en) * | 2020-11-18 | 2021-03-02 | 北京浪潮数据技术有限公司 | Server cluster management method and device and related components |
-
2021
- 2021-03-17 CN CN202110286932.7A patent/CN112965879A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876932A (en) * | 2009-11-30 | 2010-11-03 | 中国移动通信集团浙江有限公司 | Method, system and device for monitoring internal storage database |
| US20120221690A1 (en) * | 2011-02-25 | 2012-08-30 | International Business Machines Corporation | Data Processing Environment Monitoring |
| US20120317274A1 (en) * | 2011-06-13 | 2012-12-13 | Richter Owen E | Distributed metering and monitoring system |
| CN109189643A (en) * | 2018-09-11 | 2019-01-11 | 五八有限公司 | Monitoring method, device, equipment and the storage medium of request interface response data |
| CN111522723A (en) * | 2020-04-17 | 2020-08-11 | 杭州威佩网络科技有限公司 | Data acquisition method and device |
| CN111708676A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Example cluster monitoring method and device and server |
| CN112437145A (en) * | 2020-11-18 | 2021-03-02 | 北京浪潮数据技术有限公司 | Server cluster management method and device and related components |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114760221A (en) * | 2022-03-31 | 2022-07-15 | 深信服科技股份有限公司 | Service monitoring method, system and storage medium |
| CN114760221B (en) * | 2022-03-31 | 2024-02-23 | 深信服科技股份有限公司 | Service monitoring method, system and storage medium |
| CN114900444A (en) * | 2022-04-25 | 2022-08-12 | 北京奇艺世纪科技有限公司 | Data monitoring method and device, electronic equipment and storage medium |
| CN114866551A (en) * | 2022-06-06 | 2022-08-05 | 中国工商银行股份有限公司 | Prometous cluster processing method, device, equipment and storage medium |
| CN115499299A (en) * | 2022-09-13 | 2022-12-20 | 航天信息股份有限公司 | Cluster equipment monitoring method and device |
| CN115865644A (en) * | 2022-11-09 | 2023-03-28 | 浪潮通信信息系统有限公司 | Service fault processing method, device, equipment and storage medium |
| CN117014284A (en) * | 2023-10-08 | 2023-11-07 | 北京凌云雀科技有限公司 | Inspection system, inspection method, electronic equipment and storage medium |
| CN117014284B (en) * | 2023-10-08 | 2023-12-19 | 北京凌云雀科技有限公司 | Inspection system, inspection method, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112965879A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN106489251B (en) | The methods, devices and systems of applied topology relationship discovery | |
| JP2020173778A (en) | Method, apparatus, electronic facility, computer readable medium, and computer program for allocating resource | |
| CN110708212B (en) | Method and device for tracking call link in distributed system | |
| CN110727560A (en) | Cloud service alarm method and device | |
| CN112506755B (en) | Log acquisition method, device, computer equipment and storage medium | |
| CN111405052A (en) | P2P penetration method based on port prediction, electronic device and medium | |
| CN112100034A (en) | Service monitoring method and device | |
| CN114760233B (en) | Service processing method, device, electronic equipment and storage medium | |
| CN110706148B (en) | Face image processing method, device, equipment and storage medium | |
| CN110430070B (en) | Service state analysis method, device, server, data analysis equipment and medium | |
| CN116627333A (en) | Log caching method and device, electronic equipment and computer readable storage medium | |
| CN107046581A (en) | A kind of monitoring method, device and the server of service operation state | |
| CN114465919B (en) | Network service testing method, system, electronic equipment and storage medium | |
| KR20170122874A (en) | Apparatus for managing log of application based on data distribution service | |
| CN114090268B (en) | Container management method and container management system | |
| CN115083030A (en) | Service inspection method and device and electronic equipment | |
| CN114070889A (en) | Configuration method, traffic forwarding method, device, storage medium, and program product | |
| CN112714161A (en) | Video access method, device and system | |
| CN113761433A (en) | Service processing method and device | |
| KR20170131007A (en) | Apparatus for monitoring communication based on data distribution service | |
| CN111163088B (en) | Message processing method, system and device and electronic equipment | |
| CN113407493B (en) | Operation method, data reading and writing method, device, electronic equipment and medium | |
| EP4163863A1 (en) | Program, information processing method, and information processing device | |
| CN113395291B (en) | Flow control method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |