CN116594835A - Application health monitoring method and device, storage medium and computer equipment - Google Patents
Application health monitoring method and device, storage medium and computer equipment Download PDFInfo
- Publication number
- CN116594835A CN116594835A CN202310526793.XA CN202310526793A CN116594835A CN 116594835 A CN116594835 A CN 116594835A CN 202310526793 A CN202310526793 A CN 202310526793A CN 116594835 A CN116594835 A CN 116594835A
- Authority
- CN
- China
- Prior art keywords
- data
- application
- monitored
- abnormal
- early warning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 148
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000036541 health Effects 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 61
- 230000002159 abnormal effect Effects 0.000 claims description 155
- 230000005856 abnormality Effects 0.000 claims description 77
- 238000012549 training Methods 0.000 claims description 57
- 238000004458 analytical method Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000013500 data storage Methods 0.000 claims description 7
- 238000012806 monitoring device Methods 0.000 claims description 4
- 230000002547 anomalous effect Effects 0.000 claims 2
- 238000013480 data collection Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the technical field of digital medical treatment, and discloses an application health monitoring method and device, a storage medium and computer equipment, wherein the method comprises the following steps: determining respective monitoring indexes of a plurality of monitored applications, and performing de-duplication statistics on all the monitoring indexes of the monitored applications to obtain target monitoring indexes; for each monitored application, based on an application architecture of the monitored application, invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to a monitoring index of the monitored application, wherein the data to be monitored carries an application identifier; after standardized processing is carried out on the collected data to be monitored, the data to be monitored is written into a preset data warehouse; based on the index anomaly rule of the target monitoring index, identifying anomaly data in the data warehouse, determining anomaly application according to the application identification of the anomaly data, and outputting the anomaly data and anomaly application information corresponding to the anomaly application.
Description
Technical Field
The present application relates to the field of computer and digital medical technology, and in particular, to an application health monitoring method and apparatus, a storage medium, and a computer device.
Background
At present, various enterprises realize unification of service models and data models through intensive management of application systems, but various problems which cause abnormal systems are easy to occur due to uneven system construction, operation and maintenance and application levels in various places, and stable use of the systems is not facilitated. At present, the health monitoring mode of an enterprise on an application system depends on the monitoring function provided by the application system, which has higher requirements on the research and development capability of the enterprise, and for some enterprises using open source applications, the technical level of monitoring function development of the open source applications is difficult to achieve. The problem of the application system cannot be found in time, and the enterprise is difficult to ensure that stable and reliable service is provided, and especially for some medical services, the result of lack of stable and reliable service guarantee is more serious.
Disclosure of Invention
In view of the above, the application provides an application health monitoring method and device, a storage medium and a computer device, which are beneficial to expanding the data acquisition dimension and improving the timeliness and accuracy of anomaly identification, and simultaneously realize unified processing and identification of data, avoid multiple identification of data with the same index and improve the data monitoring efficiency.
According to one aspect of the present application, there is provided an application health monitoring method, the method comprising:
determining respective monitoring indexes of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes;
for each monitored application, based on an application architecture of the monitored application, invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to a monitoring index of the monitored application, wherein the data to be monitored carries an application identifier;
after standardized processing is carried out on the collected data to be monitored, the data to be monitored is written into a preset data warehouse;
based on the index anomaly rule of the target monitoring index, identifying anomaly data in the data warehouse, determining anomaly application according to the application identification of the anomaly data, and outputting the anomaly data and anomaly application information corresponding to the anomaly application.
Optionally, after the writing the data to be monitored into a preset data warehouse, the method further includes:
performing data early warning analysis on data to be monitored in the data warehouse based on a data early warning model to generate early warning information;
Accordingly, the method further comprises:
based on the identified historical abnormal data, acquiring an abnormal time stamp and an abnormal application identifier of the historical abnormal data;
taking the abnormal time stamp and a time period of a preset early warning time period before the abnormal time stamp as early warning time periods, acquiring data to be monitored, which corresponds to the abnormal application identifier and is applied in the early warning time periods, from the data warehouse, and constructing an early warning model training sample;
and training the data early warning model based on the early warning model training sample.
Optionally, the training the data early warning model based on the early warning model training sample includes:
training a model copy of the data early warning model based on the early warning model training sample, wherein the model copy has the same model structure as the data early warning model;
updating model parameters of a currently used data early warning model based on model parameters of the model copy after every preset time period or after the preset number of samples are trained, and continuously acquiring early warning model training samples to train the model copy.
Optionally, invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to the monitoring index of the monitored application, and writing the acquired data to be monitored into a preset data warehouse after standardized processing of the acquired data to be monitored, including:
Invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to the monitoring index of the monitored application, an application running log of the monitored application and server environment information of the monitored application;
respectively carrying out standardized processing on the data to be monitored, the application running log and the server environment information, and writing the standardized processing into the data warehouse;
correspondingly, the obtaining the data to be monitored of the monitored application corresponding to the abnormal application identifier in the data warehouse in the early warning period, and constructing an early warning model training sample, including:
acquiring data to be monitored, application running logs and server environment information of the monitored application corresponding to the abnormal application identifier in the early warning period from the data warehouse, and constructing an early warning model training sample;
correspondingly, the data pre-warning analysis is performed on the data to be monitored in the data warehouse based on the data pre-warning model, and pre-warning information is generated, including:
and carrying out data early warning analysis on the data to be monitored, the application running log and the server environment information in the data warehouse based on the data early warning model, and generating early warning information.
Optionally, after determining the abnormal application based on the application identifier of the abnormal data, the method further includes:
writing the abnormal data into an abnormal data queue;
and polling the abnormal data queue through a preset abnormal monitoring process, and sending the alarm information of the abnormal data to a preset alarm information receiving terminal in a notification mode corresponding to the monitoring index priority of the abnormal data.
Optionally, before the writing the abnormal data into the abnormal data queue, the method further includes:
inquiring whether an abnormality processing script corresponding to the abnormality monitoring index exists in a preset abnormality processing script according to the abnormality monitoring index corresponding to the abnormality data;
if so, calling an abnormality processing script corresponding to the abnormality monitoring index, automatically processing the abnormality application through the abnormality processing script, identifying that new abnormality data corresponding to the abnormality monitoring index exists in the application again within a preset monitoring duration, and writing the new abnormality data into an abnormality data queue;
and if the abnormal data does not exist, executing the writing of the abnormal data into an abnormal data queue.
Optionally, the outputting the abnormal data and the abnormal application information corresponding to the abnormal application includes:
counting the data to be monitored in the data warehouse according to a preset data display rule to obtain the statistical data to be displayed;
and carrying out data display based on the statistical data, the abnormal data and the abnormal application.
According to another aspect of the present application, there is provided an application health monitoring apparatus, the apparatus comprising:
the index determining module is used for determining the monitoring indexes of each of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes;
the data acquisition module is used for aiming at each monitored application, calling a data acquisition method matched with the application architecture based on the application architecture of the monitored application to acquire data to be monitored corresponding to the monitoring index of the monitored application, wherein the data to be monitored carries an application identifier;
the data storage module is used for writing the data to be monitored into a preset data warehouse after carrying out standardized processing on the collected data to be monitored;
the abnormal identification module is used for identifying abnormal data in the data warehouse based on the index abnormal rule of the target monitoring index, determining abnormal application according to the application identification of the abnormal data, and outputting the abnormal data and abnormal application information corresponding to the abnormal application.
Optionally, the apparatus further comprises:
the early warning module is used for carrying out data early warning analysis on the data to be monitored in the data warehouse based on a data early warning model, and generating early warning information;
correspondingly, the device further comprises: the early warning model training module is used for:
based on the identified historical abnormal data, acquiring an abnormal time stamp and an abnormal application identifier of the historical abnormal data;
taking the abnormal time stamp and a time period of a preset early warning time period before the abnormal time stamp as early warning time periods, acquiring data to be monitored, which corresponds to the abnormal application identifier and is applied in the early warning time periods, from the data warehouse, and constructing an early warning model training sample;
and training the data early warning model based on the early warning model training sample.
Optionally, the early warning model training module is further configured to:
training a model copy of the data early warning model based on the early warning model training sample, wherein the model copy has the same model structure as the data early warning model;
updating model parameters of a currently used data early warning model based on model parameters of the model copy after every preset time period or after the preset number of samples are trained, and continuously acquiring early warning model training samples to train the model copy.
Optionally, the data acquisition module is further configured to: invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to the monitoring index of the monitored application, an application running log of the monitored application and server environment information of the monitored application;
the data storage module is further configured to: respectively carrying out standardized processing on the data to be monitored, the application running log and the server environment information, and writing the standardized processing into the data warehouse;
the early warning model training module is further used for: acquiring data to be monitored, application running logs and server environment information of the monitored application corresponding to the abnormal application identifier in the early warning period from the data warehouse, and constructing an early warning model training sample;
the early warning module is further used for: and carrying out data early warning analysis on the data to be monitored, the application running log and the server environment information in the data warehouse based on the data early warning model, and generating early warning information.
Optionally, the apparatus further comprises: an alarm module for:
writing the abnormal data into an abnormal data queue;
And polling the abnormal data queue through a preset abnormal monitoring process, and sending the alarm information of the abnormal data to a preset alarm information receiving terminal in a notification mode corresponding to the monitoring index priority of the abnormal data.
Optionally, the apparatus further comprises:
the script inquiry module is used for inquiring whether an abnormality processing script corresponding to the abnormality monitoring index exists in a preset abnormality processing script according to the abnormality monitoring index corresponding to the abnormality data;
the abnormality processing module is used for calling an abnormality processing script corresponding to the abnormality monitoring index if the abnormality processing script exists, automatically processing the abnormality application through the abnormality processing script, identifying that new abnormality data corresponding to the abnormality monitoring index exists in the application again within a preset monitoring duration, and writing the new abnormality data into an abnormality data queue;
and the alarm module is further used for executing the writing of the abnormal data into an abnormal data queue if the abnormal data does not exist.
Optionally, the anomaly identification module is further configured to: counting the data to be monitored in the data warehouse according to a preset data display rule to obtain the statistical data to be displayed; and carrying out data display based on the statistical data, the abnormal data and the abnormal application.
According to yet another aspect of the present application, there is provided a storage medium having stored thereon a computer program which when executed by a processor implements the above-described application health monitoring method.
According to still another aspect of the present application, there is provided a computer device including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, the processor implementing the above-mentioned application health monitoring method when executing the program.
By means of the technical scheme, the application health monitoring method, the device, the storage medium and the computer equipment provided by the application are characterized in that the monitoring system is utilized to count the monitoring indexes of different monitored applications, the target monitoring indexes to be monitored are determined, the data acquisition method matched with the application architecture of the different monitored applications is called to acquire the data to be monitored matched with the monitoring indexes from each application, the data to be monitored is placed into a data warehouse after standardized processing, and the data in the data warehouse are subjected to abnormal recognition and output based on the index abnormal rules of the target monitoring indexes constructed in advance. According to the application, the built monitoring system utilizes the pre-packaged data acquisition methods of different application architecture types to acquire data of the monitored application based on different architectures, so that unified data acquisition is realized, the monitoring function of the open source application is not relied on, the data acquisition dimension is expanded, the timeliness and accuracy of anomaly identification are improved, the acquired data is standardized and then placed in a unified data warehouse to carry out anomaly identification, unified data processing and identification are realized, the repeated identification of the data with the same index is avoided, and the data monitoring efficiency is improved.
The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present application more readily apparent.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 shows a schematic flow chart of an application health monitoring method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of another method for monitoring health of an application according to an embodiment of the present application;
fig. 3 shows a schematic structural diagram of an application health monitoring device according to an embodiment of the present application;
fig. 4 shows a schematic device structure of a computer device according to an embodiment of the present application.
Detailed Description
The application will be described in detail hereinafter with reference to the drawings in conjunction with embodiments. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
In this embodiment, an application health monitoring method is provided, as shown in fig. 1, and the method includes:
step 101, determining respective monitoring indexes of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes.
When an enterprise uses an open source application, monitoring is often realized based on a monitoring function provided by the open source application, and the enterprise has difficulty in developing the monitoring function of the open source application. Aiming at the problem, the open source application used by the enterprise can be monitored through a monitoring system built by a third party.
The embodiment of the application is applied to the built monitoring system, the health monitoring is carried out on a plurality of open source applications deployed on the server by the monitoring system, the same, partial same or different monitoring indexes can be set for different applications, the monitoring system can monitor all the data to be monitored of the applications in the same data warehouse, so that the monitoring efficiency is improved, and the deduplication statistics is carried out based on the monitoring indexes of all the applications, so that the target monitoring indexes for the data warehouse are obtained.
The monitored application can be a medical health service application created by combining the medical technology and using cloud computing on the basis of new technologies such as cloud computing, mobile technology, multimedia, 4G communication, big data, internet of things and the like, so that the sharing of medical resources and the expansion of medical scope are realized. Because the cloud computing technology is applied to combination, the medical application improves the efficiency of medical institutions, and residents can conveniently seek medical advice. Like reservation registration, electronic medical records, medical insurance and the like of the traditional hospital are products of combination of cloud computing and medical field, and medical application also has the advantages of data security, information sharing, dynamic expansion and overall layout. For example, the monitored application is a reservation registration application, and the monitoring indicator includes the number of reservation registrations per minute.
Step 102, for each monitored application, based on the application architecture of the monitored application, invoking a data acquisition method matched with the application architecture to acquire to-be-monitored data corresponding to the monitoring index of the monitored application, wherein the to-be-monitored data carries an application identifier.
Different monitored applications may be built based on different application architectures, and the data acquisition modes corresponding to the applications based on the different application architectures are different. In addition, the data to be monitored collected from different monitored applications carries an application identifier which is written into the monitored application and is used for representing the source of the data.
And 103, after standardized processing is carried out on the collected data to be monitored, writing the data to be monitored into a preset data warehouse.
Because the data formats collected from the applications under different application architectures are different, in order to realize unified processing of the data, the data to be monitored is standardized, and the data sources are rewritten into the unified standard format and then stored in the data warehouse.
Step 104, based on the index anomaly rule of the target monitoring index, identifying anomaly data in the data warehouse, determining anomaly application according to the application identification of the anomaly data, and outputting the anomaly data and anomaly application information corresponding to the anomaly application.
The monitoring system monitors the health of the data warehouse, and can specifically judge whether the data to be monitored in the data warehouse is abnormal or not by utilizing index abnormality rules which are constructed in advance and aim at different target monitoring indexes, and if the abnormal data exist in the data warehouse, the abnormal application can be positioned based on the application identification carried by the abnormal data. And then outputting the abnormal data and the abnormal application to realize the health monitoring of the monitored application.
According to the technical scheme, the monitoring system is utilized to count the monitoring indexes of different monitored applications, the target monitoring indexes to be monitored are determined, the data acquisition method matched with the application architecture of the different monitored applications is called to acquire the data to be monitored matched with the monitoring indexes of the data to be monitored from each application, the data to be monitored is standardized and then is placed into a data warehouse, and based on the index abnormality rule of the target monitoring indexes constructed in advance, the data in the data warehouse are abnormally identified and output. According to the application, the built monitoring system utilizes the pre-packaged data acquisition methods of different application architecture types to acquire data of the monitored application based on different architectures, so that unified data acquisition is realized, the monitoring function of the open source application is not relied on, the data acquisition dimension is expanded, the timeliness and accuracy of anomaly identification are improved, the acquired data is standardized and then placed in a unified data warehouse to carry out anomaly identification, unified data processing and identification are realized, the repeated identification of the data with the same index is avoided, and the data monitoring efficiency is improved.
Further, as a refinement and extension of the foregoing embodiment, in order to fully describe the implementation procedure of this embodiment, another application health monitoring method is provided, as shown in fig. 2, where the method includes:
step 201, determining respective monitoring indexes of a plurality of monitored applications, and performing deduplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes.
Step 202, for each monitored application, based on the application architecture of the monitored application, invoking a data acquisition method matched with the application architecture to acquire to-be-monitored data corresponding to the monitoring index of the monitored application, an application running log of the monitored application and server environment information of the monitored application, wherein the to-be-monitored data carries an application identifier.
And 203, writing the data to be monitored, the application running log and the server environment information into the data warehouse after respectively carrying out standardized processing.
The monitored application in the embodiment of the application can be a plurality of open source applications deployed on the same physical server, different open source applications have respective concerned indexes, the monitoring indexes can be the indexes concerned by the application, or the indexes which are manually set or are determined by big data analysis, after the respective monitoring indexes of different monitored applications are determined, all the monitoring indexes are subjected to de-duplication analysis to obtain target monitoring indexes, so that data anomaly monitoring is performed based on the target monitoring indexes. The data acquisition method of the multiple application architectures is pre-packaged in the monitoring system, the data acquisition method corresponding to each application architecture can be called for data acquisition aiming at different monitored applications to obtain data to be monitored, meanwhile, the application running logs of each monitored application can be pulled, the server environment information of a server where the application is located is obtained, and the acquired various data are unified and standardized and then are put into a data warehouse.
In a specific application scene, because the acquisition modes of the open source service index are inconsistent, the acquired index data formats are also inconsistent, the acquired index data formats are difficult to unify to one acquisition system for processing, and the acquisition efficiency is far from being comparable with that of big data real-time distributed computing engines such as Flink, spark and the like. The Prairie utilizes the Flink real-time engine to define the acquisition methods of different acquisition modes by rewriting data sources, namely, the mode of self-defining Source components, not only can acquire indexes provided by an application system, but also can acquire application text logs, server environment information and the like, improves data diversity, can dynamically configure the indexes required to be acquired by utilizing the Flink CDC, is equivalent to making an adapter to adapt to different open Source applications even if the indexes are effective, and is finally integrated into one application. And integrating the data of different systems into unified standard data through a Flink operator, writing the unified standard data into a real-time digital bin, and modeling the real-time digital bin to further avoid subsequent repeated calculation and reuse the data. The large data HDFS is used as a bottom file system, the reliability provided by an ultra-large data volume and a copy mechanism can be accommodated, and storage services such as HBase built on the HDFS can be suitable for hundred million-level concurrent read-write, so that large-scale read-write requests of an acquisition system can be handled from the beginning.
And 204, performing data early warning analysis on the data to be monitored, the application running log and the server environment information in the data warehouse based on the data early warning model, and generating early warning information.
Step 205, based on the index anomaly rule of the target monitoring index, identifying anomaly data in the data warehouse, determining anomaly application according to the application identification of the anomaly data, and outputting the anomaly data and anomaly application information corresponding to the anomaly application.
In this embodiment, in addition to alarming the situation that the abnormality has occurred when the abnormal data is identified, early warning may be performed on the situation that the abnormality may occur, specifically, the data to be monitored, the application running log and the server environment information may be input into a trained data early warning model, and data early warning analysis is performed through the data early warning model to obtain early warning information about whether an early warning event exists.
Step 206, based on the identified historical abnormal data, acquiring an abnormal time stamp and an abnormal application identifier of the historical abnormal data; taking the abnormal time stamp and a time period of a preset early warning time period before the abnormal time stamp as early warning time periods, acquiring data to be monitored, application running logs and server environment information, which correspond to the abnormal application identification and are applied to the early warning time periods, in the data warehouse, and constructing an early warning model training sample; and training the data early warning model based on the early warning model training sample.
In this embodiment, the data early warning model is obtained by performing model training based on the identified historical abnormal data. Specifically, each time the monitoring system identifies the abnormal data, the time of generating the abnormal data is determined based on the timestamp of the abnormal data, and a period of time (i.e. a preset early warning time period) before the time of generating the abnormal data is taken as an early warning period. And acquiring data to be monitored, application running logs and server environment information in the early warning period from a data warehouse, constructing an early warning model training sample, and performing data early warning model training. The model can quickly judge whether the abnormality is likely to occur or not based on the real-time acquired data to be monitored, the application running log and the server environment through the data early warning model by learning the data feature to be monitored, the application running log feature and the server environment feature for a period of time before the data abnormality occurs, so that early warning can be carried out.
In the embodiment of the present application, optionally, in step 206, "training the data early-warning model based on the early-warning model training sample", includes: training a model copy of the data early warning model based on the early warning model training sample, wherein the model copy has the same model structure as the data early warning model; updating model parameters of a currently used data early warning model based on model parameters of the model copy after every preset time period or after the preset number of samples are trained, and continuously acquiring early warning model training samples to train the model copy.
In this embodiment, the data pre-warning model used may be updated at intervals (i.e., a preset time period), or may be updated every time a certain number of training times (i.e., a preset number) is reached. Specifically, two data early-warning models with the same model structure and model parameters can be preset, one is used as a data early-warning model for carrying out data early-warning, the other is used as a model copy for carrying out parameter training, the model copy is trained by using an early-warning model training sample, and the model parameters of the model for data early-warning are updated once at intervals or under the condition of reaching a certain training amount.
Step 207, inquiring whether an abnormality processing script corresponding to the abnormality monitoring index exists in a preset abnormality processing script according to the abnormality monitoring index corresponding to the abnormality data.
And step 208, if the application exists, calling an abnormality processing script corresponding to the abnormality monitoring index, automatically processing the abnormality application through the abnormality processing script, identifying that new abnormality data corresponding to the abnormality monitoring index exists in the application again within a preset monitoring duration, and writing the new abnormality data into an abnormality data queue.
Step 209, if not, writing the abnormal data into an abnormal data queue.
Step 210, polling the abnormal data queue through a preset abnormal monitoring process, and sending the alarm information of the abnormal data to a preset alarm information receiving terminal in a notification mode corresponding to the monitoring index priority of the abnormal data.
In the embodiment of the application, the exception handling script can be constructed in advance for some common exception problems, and the exception handling is automatically carried out through the script, so that the exception handling efficiency is improved. When abnormal data is monitored, corresponding abnormal monitoring indexes are determined, and whether an abnormal processing script corresponding to the abnormal monitoring indexes is contained in a preset abnormal processing script constructed in advance is inquired. If the corresponding exception handling script is included, the exception handling script is directly called to carry out automatic processing, the data to be monitored in the data warehouse is continuously monitored, and if the data exception problem occurs again in the same monitoring index of the same application within the preset monitoring duration, the problem is described as not being solved through script automatic processing, and the exception data is written into an exception data queue. If the preset exception handling script does not contain the script corresponding to the exception monitoring index, the exception data is directly written into the exception data queue. Further, abnormal data in the abnormal data queue is polled through a preset abnormal monitoring process, and multi-type alarms such as mails, short messages, telephones and the like are carried out according to the priority of monitoring indexes corresponding to the abnormal data, for example, the alarms of the telephones, the short messages and the mails are respectively carried out from high priority to low priority.
Step 211, counting the data to be monitored in the data warehouse according to preset data display rules to obtain the statistical data to be displayed; and carrying out data display based on the statistical data, the abnormal data and the abnormal application.
The monitoring system provided by the embodiment of the application also provides a data display function, wherein after the data to be monitored in the data warehouse are processed according to the preset data display rule, the data display is performed on the display screen, and in addition, abnormal data, abnormal application and early warning information can be displayed, so that a worker can observe the running condition of the monitored application in real time through the display screen and find the abnormality in time.
By applying the technical scheme of the embodiment, the acquisition system is unified, the data is more various, the data format is unified, the concurrency efficiency is good, repeated calculation is reduced, large data quantity is stored, high concurrency read-write data is stored, the data can be used for multiple times, whether the data is abnormal or not can be judged in advance according to the self-defined rule before the data is acquired into the data storage, the rule is dynamically configured, the data can be used as an algorithm model for data prejudgement, and the data in the data storage can be used as a training set continuous training model, so that the abnormal prejudgement precision is improved. The displayed data can be classified alarming after deep processing of preset rules, and is used for automatically processing and perceiving the abnormality in advance.
Further, as a specific implementation of the method of fig. 1, an embodiment of the present application provides an application health monitoring device, as shown in fig. 3, where the device includes:
the index determining module is used for determining the monitoring indexes of each of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes;
the data acquisition module is used for aiming at each monitored application, calling a data acquisition method matched with the application architecture based on the application architecture of the monitored application to acquire data to be monitored corresponding to the monitoring index of the monitored application, wherein the data to be monitored carries an application identifier;
the data storage module is used for writing the data to be monitored into a preset data warehouse after carrying out standardized processing on the collected data to be monitored;
the abnormal identification module is used for identifying abnormal data in the data warehouse based on the index abnormal rule of the target monitoring index, determining abnormal application according to the application identification of the abnormal data, and outputting the abnormal data and abnormal application information corresponding to the abnormal application.
Optionally, the apparatus further comprises:
The early warning module is used for carrying out data early warning analysis on the data to be monitored in the data warehouse based on a data early warning model, and generating early warning information;
correspondingly, the device further comprises: the early warning model training module is used for:
based on the identified historical abnormal data, acquiring an abnormal time stamp and an abnormal application identifier of the historical abnormal data;
taking the abnormal time stamp and a time period of a preset early warning time period before the abnormal time stamp as early warning time periods, acquiring data to be monitored, which corresponds to the abnormal application identifier and is applied in the early warning time periods, from the data warehouse, and constructing an early warning model training sample;
and training the data early warning model based on the early warning model training sample.
Optionally, the early warning model training module is further configured to:
training a model copy of the data early warning model based on the early warning model training sample, wherein the model copy has the same model structure as the data early warning model;
updating model parameters of a currently used data early warning model based on model parameters of the model copy after every preset time period or after the preset number of samples are trained, and continuously acquiring early warning model training samples to train the model copy.
Optionally, the data acquisition module is further configured to: invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to the monitoring index of the monitored application, an application running log of the monitored application and server environment information of the monitored application;
the data storage module is further configured to: respectively carrying out standardized processing on the data to be monitored, the application running log and the server environment information, and writing the standardized processing into the data warehouse;
the early warning model training module is further used for: acquiring data to be monitored, application running logs and server environment information of the monitored application corresponding to the abnormal application identifier in the early warning period from the data warehouse, and constructing an early warning model training sample;
the early warning module is further used for: and carrying out data early warning analysis on the data to be monitored, the application running log and the server environment information in the data warehouse based on the data early warning model, and generating early warning information.
Optionally, the apparatus further comprises: an alarm module for:
writing the abnormal data into an abnormal data queue;
And polling the abnormal data queue through a preset abnormal monitoring process, and sending the alarm information of the abnormal data to a preset alarm information receiving terminal in a notification mode corresponding to the monitoring index priority of the abnormal data.
Optionally, the apparatus further comprises:
the script inquiry module is used for inquiring whether an abnormality processing script corresponding to the abnormality monitoring index exists in a preset abnormality processing script according to the abnormality monitoring index corresponding to the abnormality data;
the abnormality processing module is used for calling an abnormality processing script corresponding to the abnormality monitoring index if the abnormality processing script exists, automatically processing the abnormality application through the abnormality processing script, identifying that new abnormality data corresponding to the abnormality monitoring index exists in the application again within a preset monitoring duration, and writing the new abnormality data into an abnormality data queue;
and the alarm module is further used for executing the writing of the abnormal data into an abnormal data queue if the abnormal data does not exist.
Optionally, the anomaly identification module is further configured to: counting the data to be monitored in the data warehouse according to a preset data display rule to obtain the statistical data to be displayed; and carrying out data display based on the statistical data, the abnormal data and the abnormal application.
It should be noted that, for other corresponding descriptions of each functional unit related to the application health monitoring device provided by the embodiment of the present application, reference may be made to corresponding descriptions in the methods of fig. 1 to fig. 2, and no further description is given here.
The embodiment of the application also provides a computer device, which can be a personal computer, a server, a network device and the like, and as shown in fig. 4, the computer device comprises a bus, a processor, a memory and a communication interface, and can also comprise an input/output interface and a display device. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing location information. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement the steps in the method embodiments.
It will be appreciated by persons skilled in the art that the architecture shown in fig. 4 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer readable storage medium is provided, which may be non-volatile or volatile, and on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (10)
1. An application health monitoring method, the method comprising:
determining respective monitoring indexes of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes;
for each monitored application, based on an application architecture of the monitored application, invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to a monitoring index of the monitored application, wherein the data to be monitored carries an application identifier;
After standardized processing is carried out on the collected data to be monitored, the data to be monitored is written into a preset data warehouse;
based on the index anomaly rule of the target monitoring index, identifying anomaly data in the data warehouse, determining anomaly application according to the application identification of the anomaly data, and outputting the anomaly data and anomaly application information corresponding to the anomaly application.
2. The method of claim 1, wherein after the writing of the data to be monitored into a preset data warehouse, the method further comprises:
performing data early warning analysis on data to be monitored in the data warehouse based on a data early warning model to generate early warning information;
accordingly, the method further comprises:
based on the identified historical abnormal data, acquiring an abnormal time stamp and an abnormal application identifier of the historical abnormal data;
taking the abnormal time stamp and a time period of a preset early warning time period before the abnormal time stamp as early warning time periods, acquiring data to be monitored, which corresponds to the abnormal application identifier and is applied in the early warning time periods, from the data warehouse, and constructing an early warning model training sample;
And training the data early warning model based on the early warning model training sample.
3. The method of claim 2, wherein training the data pre-warning model based on the pre-warning model training samples comprises:
training a model copy of the data early warning model based on the early warning model training sample, wherein the model copy has the same model structure as the data early warning model;
updating model parameters of a currently used data early warning model based on model parameters of the model copy after every preset time period or after the preset number of samples are trained, and continuously acquiring early warning model training samples to train the model copy.
4. The method according to claim 2, wherein invoking a data collection method matched with the application architecture to collect data to be monitored corresponding to a monitoring index of the monitored application, and writing the data to be monitored into a preset data warehouse after performing standardization processing on the collected data to be monitored, comprises:
invoking a data acquisition method matched with the application architecture to acquire data to be monitored corresponding to the monitoring index of the monitored application, an application running log of the monitored application and server environment information of the monitored application;
Respectively carrying out standardized processing on the data to be monitored, the application running log and the server environment information, and writing the standardized processing into the data warehouse;
correspondingly, the obtaining the data to be monitored of the monitored application corresponding to the abnormal application identifier in the data warehouse in the early warning period, and constructing an early warning model training sample, including:
acquiring data to be monitored, application running logs and server environment information of the monitored application corresponding to the abnormal application identifier in the early warning period from the data warehouse, and constructing an early warning model training sample;
correspondingly, the data pre-warning analysis is performed on the data to be monitored in the data warehouse based on the data pre-warning model, and pre-warning information is generated, including:
and carrying out data early warning analysis on the data to be monitored, the application running log and the server environment information in the data warehouse based on the data early warning model, and generating early warning information.
5. The method according to any one of claims 1 to 4, wherein after the determining of an anomalous application based on the application identification of anomalous data, the method further comprises:
writing the abnormal data into an abnormal data queue;
And polling the abnormal data queue through a preset abnormal monitoring process, and sending the alarm information of the abnormal data to a preset alarm information receiving terminal in a notification mode corresponding to the monitoring index priority of the abnormal data.
6. The method of claim 5, wherein prior to writing the exception data to an exception data queue, the method further comprises:
inquiring whether an abnormality processing script corresponding to the abnormality monitoring index exists in a preset abnormality processing script according to the abnormality monitoring index corresponding to the abnormality data;
if so, calling an abnormality processing script corresponding to the abnormality monitoring index, automatically processing the abnormality application through the abnormality processing script, identifying that new abnormality data corresponding to the abnormality monitoring index exists in the application again within a preset monitoring duration, and writing the new abnormality data into an abnormality data queue;
and if the abnormal data does not exist, executing the writing of the abnormal data into an abnormal data queue.
7. The method according to any one of claims 1 to 4, wherein the outputting of the abnormal data and the abnormal application information corresponding to the abnormal application includes:
Counting the data to be monitored in the data warehouse according to a preset data display rule to obtain the statistical data to be displayed;
and carrying out data display based on the statistical data, the abnormal data and the abnormal application.
8. An application health monitoring device, the device comprising:
the index determining module is used for determining the monitoring indexes of each of a plurality of monitored applications, and performing de-duplication statistics on the monitoring indexes of all the monitored applications to obtain target monitoring indexes;
the data acquisition module is used for aiming at each monitored application, calling a data acquisition method matched with the application architecture based on the application architecture of the monitored application to acquire data to be monitored corresponding to the monitoring index of the monitored application, wherein the data to be monitored carries an application identifier;
the data storage module is used for writing the data to be monitored into a preset data warehouse after carrying out standardized processing on the collected data to be monitored;
the abnormal identification module is used for identifying abnormal data in the data warehouse based on the index abnormal rule of the target monitoring index, determining abnormal application according to the application identification of the abnormal data, and outputting the abnormal data and abnormal application information corresponding to the abnormal application.
9. A storage medium having stored thereon a computer program, which when executed by a processor, implements the method of any of claims 1 to 7.
10. A computer device comprising a storage medium, a processor and a computer program stored on the storage medium and executable on the processor, characterized in that the processor implements the method of any one of claims 1 to 7 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310526793.XA CN116594835A (en) | 2023-05-10 | 2023-05-10 | Application health monitoring method and device, storage medium and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310526793.XA CN116594835A (en) | 2023-05-10 | 2023-05-10 | Application health monitoring method and device, storage medium and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116594835A true CN116594835A (en) | 2023-08-15 |
Family
ID=87607377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310526793.XA Pending CN116594835A (en) | 2023-05-10 | 2023-05-10 | Application health monitoring method and device, storage medium and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116594835A (en) |
-
2023
- 2023-05-10 CN CN202310526793.XA patent/CN116594835A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111581045B (en) | Database anomaly monitoring method, device, computer device and storage medium | |
| US10545967B2 (en) | Control apparatus of an automation system | |
| CN111917877A (en) | Data processing method and device for Internet of things equipment, electronic equipment and storage medium | |
| CN106815125A (en) | A kind of log audit method and platform | |
| CN112580914A (en) | Method and device for realizing enterprise-level data middling platform system for collecting multi-source data | |
| CN113010374B (en) | Quantum device monitoring method and system based on monitoring platform | |
| US20180341369A1 (en) | System and method for bulding customized web applications within a domain | |
| CN109684320B (en) | Method and equipment for online cleaning of monitoring data | |
| CN111984495A (en) | Big data monitoring method and device and storage medium | |
| CN113485988A (en) | Data quality monitoring method and device and computer readable storage medium | |
| CN114443437A (en) | Alarm root cause output method, apparatus, device, medium, and program product | |
| CN110611715A (en) | System and method for collecting cloud monitoring information by service link | |
| CN112818003B (en) | Execution risk estimation method and device for query task | |
| CN114003568A (en) | Data processing method and related device | |
| CN113239327A (en) | Method, apparatus, computer device and storage medium for monitoring software licenses | |
| CN116594835A (en) | Application health monitoring method and device, storage medium and computer equipment | |
| CN115225470B (en) | Business abnormality monitoring method and device, electronic equipment and storage medium | |
| CN108052637B (en) | Real-time image-text monitoring method for power grid time scale measurement data access full life cycle | |
| CN108334524A (en) | A kind of storm daily records error analysis methodology and device | |
| CN105607983A (en) | Data exception monitoring method and apparatus | |
| CN117557951A (en) | Power grid abnormal event monitoring method, device, computer equipment and storage medium | |
| CN116610664A (en) | Data monitoring method, device, computer equipment, storage medium and product | |
| CN117453693A (en) | MYSQL data model management method and device based on DORIS database | |
| CN117591585A (en) | Target data acquisition method and device and computer equipment | |
| CN115311070A (en) | Management method and device for bank wind control, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |