CN115168139A

CN115168139A - Early warning method and system based on monitoring prediction

Info

Publication number: CN115168139A
Application number: CN202210743757.4A
Authority: CN
Inventors: 刘浩
Original assignee: CITIC Aibank Corp Ltd
Current assignee: CITIC Aibank Corp Ltd
Priority date: 2022-06-28
Filing date: 2022-06-28
Publication date: 2022-10-11

Abstract

The invention relates to a monitoring prediction based early warning method and a system, comprising the following steps: acquiring training data; inputting the training data serving as a training set into a Prophet algorithm model for training to obtain prediction data; configuring an early warning rule; carrying out early warning judgment on the prediction data according to the configured early warning rule to obtain an early warning judgment result; sending an early warning event notification according to an early warning judgment result; and processing the early warning event according to the early warning event notification. The method and the system realize the prediction and early warning of the monitoring trend, train the prediction model by taking the actual monitoring data as a training set, and predict the monitoring trend in a period of time in the future by using the model, thereby achieving the purpose of perceiving and finding problems in advance. And the disposal platform is utilized to self-heal the problems found in advance, thereby ensuring the stability and continuity of the service system.

Description

Early warning method and system based on monitoring prediction

Technical Field

The invention relates to the technical field of event early warning, in particular to an early warning method and system based on monitoring prediction.

Background

In the existing technical solutions for monitoring trend prediction and event early warning, a threshold value is mostly set on a monitoring platform, and an event alarm notification is triggered when a monitoring condition is reached; the common mode is to use zabbix or propamix monitoring platform to alarm directly.

In the above solution, there are three disadvantages:

1. monitoring information trend, namely when monitoring information is checked, only the historical trend of the monitoring index can be checked, and the trend of the future index trend cannot be given.

2. And (3) periodic alarm, namely, the monitoring indexes are periodically alarmed, and the monitoring system cannot discover and sense alarm events in advance.

3. And (4) alarm processing, namely triggering a disk capacity alarm event in the early morning or in non-working time and synchronously processing the disk capacity alarm event to a relevant responsible person. The processing time and the processing efficiency are not fast in working time, and the situations such as misoperation are easy to occur.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides an early warning method and system based on monitoring prediction, which can sense an early warning event in advance, carry out early warning notification and automatically dispose. By the method and the system, the frequency of the occurrence of the events can be reduced, and the stability and the continuity of the service are ensured to the maximum extent by processing the early warning events in advance.

In order to achieve the above purpose, the technical scheme adopted by the invention comprises the following steps:

a monitoring prediction based early warning method is characterized by comprising the following steps:

s1, acquiring training data;

s2, inputting the training data serving as a training set into a Prophet algorithm model for training to obtain prediction data;

s3, configuring an early warning rule;

s4, performing early warning judgment on the prediction data according to the configured early warning rule to obtain an early warning judgment result;

s5, sending an early warning event notification according to an early warning judgment result;

and S6, processing the early warning event according to the early warning event notification.

Further, before the step S1, the method includes:

sa, acquiring zabbix basic monitoring data;

sb, acquiring transaction monitoring data;

and Sc, aggregating the zabbix basic monitoring data and the transaction monitoring data into training data in a unified format.

Further, zabbix basis monitoring data, including: host information and monitoring item information.

Further, the early warning rule comprises: the method comprises the following steps of starting time, ending time, applicable date, early warning days, early warning level, early warning value, early warning mode, total number of sending mechanisms and sending mechanism average value point.

Further, the early warning judgment comprises:

judging whether the prediction data has an associated early warning rule or not;

and judging whether the predicted data accords with an early warning rule or not.

And further, delaying the sending of the early warning event notification according to the early warning start time and the early warning end time configured by the early warning rule.

The invention also relates to a monitoring prediction-based early warning system, which is characterized by comprising the following components:

the data acquisition module is used for acquiring training data;

the model training module is used for inputting training data serving as a training set into a Prophet algorithm model for training to obtain prediction data;

the early warning rule configuration module is used for configuring early warning rules;

the early warning judgment module is used for carrying out early warning judgment on the prediction data according to the configured early warning rule to obtain an early warning judgment result;

the early warning event notification module is used for sending an early warning event notification according to the early warning judgment result;

and the early warning event processing module is used for processing the early warning event according to the early warning event notification.

The invention also relates to a computer-readable storage medium, which is characterized in that the storage medium stores a computer program, and the computer program is executed by a processor to execute the early warning method based on monitoring prediction.

The invention also relates to an electronic device, which is characterized by comprising a processor and a memory;

the memory is used for storing training data and early warning rules;

the processor is used for executing the early warning method based on monitoring prediction by calling the training data and the early warning rule.

The invention also relates to a computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the above-mentioned monitoring prediction based warning method.

The beneficial effects of the invention are as follows:

by adopting the early warning method and the early warning system based on monitoring prediction, the monitoring data of the Zabbix host computer and the transaction monitoring data are collected, processed and aggregated into the monitoring data with a unified standard format, and the trend of a period of time in the future is predicted by utilizing an algorithm and historical data of monitoring indexes. Early warning rules are set to discover early warning events in advance, early warning is carried out in advance, and predicted events are notified to relevant personnel through an event platform in working hours. And performing some prefabricated disposal means by using the event platform and the disposal platform according to the early warning event to achieve the self-healing effect of the event.

Drawings

Fig. 1 is a schematic flow chart of an early warning method based on monitoring prediction according to the present invention.

Fig. 2 is a schematic diagram of a monitoring prediction-based early warning system structure according to the present invention.

Detailed Description

For a clearer understanding of the contents of the present invention, reference will be made to the accompanying drawings and examples.

In order to improve the time for discovering an event in advance, the first aspect of the present invention relates to a monitoring and predicting early warning method, which can sense an early warning event in advance, perform early warning notification, and automatically handle, and the flow of steps of the monitoring and predicting early warning method is as shown in fig. 1. The method can reduce the frequency of the occurrence of the event and ensure the stability and continuity of the service to the maximum extent by processing the early warning event in advance.

Celery is an open-source Python-based development distributed asynchronous message task pair, and can realize asynchronous processing of tasks through the request.

Facebook open source time series prediction algorithm in 2017, which is based on fitting of time series decomposition and machine learning, wherein a pyStan open source tool is used in fitting a model, so that a needed prediction result can be obtained in a relatively fast time.

Redis is an open source log-type and Key-Value database written by using ANSI C language, complying with BSD protocol, supporting network, being based on memory and being persistent, and providing API of multiple languages

MySQL is a relational database management system developed by the MySQLAB company, sweden, and belongs to the product under Oracle flag. MySQL is one of the most popular Relational Database Management systems, and in terms of WEB applications, mySQL is one of the best RDBMS (Relational Database Management System) application software.

The method comprises the following steps:

the trend prediction early warning specification:

distributed task processing program

And starting a cell distributed task processing program, and starting a plurality of working nodes and a task trigger node for ensuring the processing efficiency. Redis is used as a task storage medium. And receiving and processing the task by the working node after the task enters the execution queue. And updating the processed intermediate state change into redis, and returning the processed result data to mysql for storage. If a large number of tasks enter, the tasks are distributed to a plurality of working nodes for task processing.

Acquiring zabbix basic monitoring data:

1. all host information and monitoring item information in the synchronous zabbix database are stored in the Mysql database.

2. And starting an asynchronous task of synchronous monitoring value information according to the host information and the associated monitoring item information, and handing the asynchronous task to a distributed task processing program for execution. Here, up to 10 tasks are scheduled to be performed simultaneously, and the configuration is performed with reference to the number of working nodes.

3. The asynchronous task acquires monitoring information in a recent period of time through parameters of the host and the monitoring index, and converts the monitoring information into data aggregated in 10 minutes by data processing. And handles the anomalous data. And storing the data after the actual monitoring values are aggregated into a Mysql database for subsequent prediction models.

Acquiring transaction monitoring data:

1. and connecting a transaction monitoring ES storage medium, and storing the relation information of the synchronous system code, the service name, the initiator and the receiver to a Mysql database.

2. And calling asynchronous tasks of the synchronous transaction monitoring data according to the information of the service name, the initiator, the requester and the like, and handing the asynchronous tasks to a distributed task processing program for execution. Here, up to 10 tasks are scheduled to be performed simultaneously, and the configuration is performed with reference to the number of working nodes.

3. The asynchronous task obtains transaction monitoring information in a recent period of time through parameters such as a service name, an initiator and a requester, converts the transaction monitoring information into data aggregated in 10 minutes, and stores the information such as the transaction amount, the timeout time, the transaction success rate and the service success rate until mysql data is used by a subsequent prediction model.

And aggregating the zabbix basic monitoring data and the transaction monitoring data into training data with a uniform format.

Acquiring training data;

judging whether to start a prediction task (the maximum number of tasks is limited to be simultaneously carried out by 20) according to whether a prediction flag bit is started or not by utilizing the saved zabbix host information and monitoring item information;

inputting the training data serving as a training set into a Prophet algorithm model for training to obtain prediction data;

and acquiring historical data of the last 10 days which are locally stored, performing model training by using the historical data as a training set and using a Prophet algorithm, and calculating prediction data of the future 3 days by using the trained model. The prediction data is saved to the Mysql database.

Configuring an early warning rule:

starting time: early warning rule value start time (working time)

End time early warning rule value end time (working time)

Applicable date-the date on which the warning rule takes effect (more selections from Monday to Sunday)

Early warning days, using prediction data of several days

Early warning level early warning event notification level (primary, secondary, notification, information)

Prediction data value determination

Early warning mode max/min judgment mode (higher than early warning value notice or lower than early warning value notice)

Total number of transmission mechanisms, using predicted value point position (judging predicted data trend) for judging early warning value

Sending mechanism average value points, wherein the mechanism average value points are used together with the total value, and the average value (judging the trend of the predicted data) of the point positions is used for carrying out early warning judgment on the predicted data according to the configured early warning rule to obtain an early warning judgment result;

sending an early warning event notification according to an early warning judgment result;

and after the trend prediction task is completed, triggering early warning judgment logic, judging whether the predicted information has an associated early warning rule, judging whether the predicted information meets early warning conditions, and delaying to send an early warning event notice according to the early warning starting time and ending time information.

Processing the early warning event according to the early warning event notification:

and configuring a matching rule by the event platform, taking the disk early warning event as a matching basis, and calling the disposal system by the event platform to perform host disk space cleaning scripts if the matching is successful. And the self-healing effect is realized.

The embodiment of the early warning method based on monitoring and prediction is further explained according to the embodiment:

a Mysql database is required for storing information such as actual aggregated data, trend prediction data, early warning notifications, etc. A Redis is required for storing asynchronous tasks as well as task states. 5 servers are needed to deploy the distributed task processing program, and the number of required CPU cores is more than 8C because the training model of the method needs a large amount of calculation.

Configuring a timed-triggered task in a situation awareness system:

triggering to acquire zabbix host and monitoring item information at 12 points every day.

And triggering and acquiring relationship information such as a transaction monitoring application service name and the like at 12 points every day.

And triggering once per hour to acquire transaction monitoring actual data, and aggregating to 10-minute granularity data to be stored locally.

Obtaining zabbix actual monitoring data is carried out at 20 points every day, and aggregation is carried out to obtain granularity data of 10 minutes and the granularity data is stored to the local.

And triggering a prediction task at 0 point every day, and performing model training and predicting the trend. Triggering early warning judgment logic.

And triggering a cleaning task at 4 points every day, and cleaning historical data and predicted trend data for more than 30 days.

Configuring an early warning rule:

rule name: monday to friday early warning disk space

Daily early warning start time: 17:00

The early warning end time every day: 10:00

The applicable date of the rule is Monday to Friday

Number of days of early warning 3

Level of early warning secondary

Early warning value: 10

The early warning value max/min: min

Sending mechanism total value point number: 30

Mean point number of sending mechanism: 5

When the data of the predicted monitoring item meets the relation between the monitoring item and the host, acquiring the data from the current time to the 3 days in the future, calculating whether the predicted value meets the condition of being lower than 10 according to the total value and the mean value of the sending mechanism, and if so, sending a secondary early warning event which is only sent from Monday to Friday.

Triggering an early warning event:

alarm warning notification _ mount point/Cloudapp available space is lower than 10.0% _ host bz1 × 1006%

[ CURRENT value ] 7.32%

Predicted at 2022-05-2317, 10, the host monitor term vfs.fs.size [/Cloudapp, pfree ] is predicted to be 0%, which will be below the forewarning value 10.00%, please process in time after receiving the forewarning |! ::

(host computer) bz1 × 1006

【IP】172.xx.xx.xx

Early warning triggering self-healing:

[ ALARM ] DISC CLEANING NOICE _ OBject _ SELF-HEALING

[ CURRENT value ] 0

The file is permanently deleted 7 days after the file is transferred, if the executable recovery script/compress/file/bz 1 x 1006/202205231000/back _ up is needed to be restored.

(host computer) bz1 × 1006

【IP】172.xx.xx.xx

Another aspect of the present invention also relates to a monitoring and prediction based early warning system, which is structurally shown in fig. 2 and includes:

the data acquisition module is used for acquiring training data;

By using this system, the above-described arithmetic processing method can be executed, and a corresponding technical effect can be achieved.

An embodiment of the present invention further provides a computer-readable storage medium capable of implementing all the steps of the method in the above embodiment, where the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements all the steps of the monitoring prediction based early warning method in the above embodiment.

The embodiment of the present invention further provides an electronic device for executing the method, as an implementation apparatus of the method, the electronic device at least has a processor and a memory, and particularly, the memory stores data required for executing the method and related computer programs, such as training data, warning rules, and the like, and all steps of implementing the warning method based on the monitoring prediction are executed by calling the data in the memory and executing the program by the processor, and corresponding technical effects are obtained.

Preferably, the electronic device may comprise a bus architecture, which may include any number of interconnected buses and bridges linking together various circuits including one or more processors and memory. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor in performing operations.

Additionally, the electronic device may further include a communication module, an input unit, an audio processor, a display, a power source, and the like. The processor (or controller, operational controls) employed may include a microprocessor or other processor device and/or logic device that receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the above related data information, and may further store a program for executing the related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used for providing input to the processor, and can be a key or a touch input device; the power supply is used for supplying power to the electronic equipment; the display is used for displaying display objects such as images and characters, and may be an LCD display, for example. A communication module is a transmitter/receiver that sends and receives signals via an antenna. The communication module (transmitter/receiver) is coupled to the processor to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal. Based on different communication technologies, a plurality of communication modules, such as a cellular network module, a bluetooth module and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone to implement the usual telecommunication functions. The audio processor may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor is also coupled to the central processor, so that recording on the local machine can be performed through the microphone, and sound stored on the local machine can be played through the loudspeaker.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are also within the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A monitoring prediction based early warning method is characterized by comprising the following steps:

s1, acquiring training data;

s3, configuring an early warning rule;

2. The method of claim 1, wherein step S1 is preceded by:

sa, acquiring zabbix basic monitoring data;

sb, acquiring transaction monitoring data;

3. The method of claim 2, wherein zabbix-based monitoring data comprises: host information and monitoring item information.

4. The method of claim 1, wherein the early warning rules comprise: the method comprises the following steps of starting time, ending time, applicable date, early warning days, early warning level, early warning value, early warning mode, total number of sending mechanisms and sending mechanism average value point.

5. The method of claim 1, wherein the early warning determination comprises:

6. The method of claim 4, wherein delaying sending of the warning event notification is performed according to a warning start time and a warning end time configured by the warning rule.

7. An early warning system based on monitoring prediction, comprising:

the data acquisition module is used for acquiring training data;

8. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, implements the monitoring prediction based alert method of any one of claims 1 to 6.

9. An electronic device comprising a processor and a memory;

the memory is used for storing training data and early warning rules;

the processor is used for executing the early warning method based on monitoring prediction in any one of claims 1 to 6 by calling training data and early warning rules.

10. A computer program product comprising a computer program and/or instructions, wherein the computer program and/or instructions, when executed by a processor, implement the steps of the monitoring prediction based alert method of any one of claims 1 to 6.