CN114881349A - Big data analysis-based early warning method and storage medium - Google Patents
Big data analysis-based early warning method and storage medium Download PDFInfo
- Publication number
- CN114881349A CN114881349A CN202210581629.4A CN202210581629A CN114881349A CN 114881349 A CN114881349 A CN 114881349A CN 202210581629 A CN202210581629 A CN 202210581629A CN 114881349 A CN114881349 A CN 114881349A
- Authority
- CN
- China
- Prior art keywords
- information
- data
- alarm
- early warning
- original data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/03—Credit; Loans; Processing thereof
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/18—Status alarms
- G08B21/185—Electrical failure alarms
Abstract
The invention relates to artificial intelligence technology, and provides an early warning method based on big data analysis, which trains the data characteristics of first original data by using a preset algorithm, constructs a prediction model to realize accurate prediction of service data and is used for predicting the trend of the data, utilizes the prediction model to perform trend prediction on second original data to obtain the predicted value of the second original data, outputs warning information if the difference set between the predicted value and the second original data is greater than a first dynamic threshold value set, constructs a warning strategy chain based on the incidence relation between warning items in the warning information and the abnormal reason determined by the warning information, matches the real-time warning information in the real-time service data with the target node information in the warning strategy chain, sends the warning information to an early warning terminal according to the target node information, and can quickly locate a fault position according to the warning information, the usability of the system is improved.
Description
Technical Field
The invention relates to the field of artificial rules, in particular to an early warning method and device based on big data analysis, computer equipment and a storage medium.
Background
With the development of computer technology, a large amount of data is generated, including data under different business scenes, and the growth of data in the present society is multi-directional, including the growth of data volume, the diversity of data types or data sources, the acceleration of data growth rate, and the data is a continuously growing and complex data set, and the traditional data management system is continuously challenged at present. In the face of huge data volume, when data is abnormal, immeasurable loss may be generated, so that early warning detection on abnormal data is more and more important. The early warning strategy in the prior art is generally a simple value method, a mobile average value method and the like, has few applicable scenes, low prediction accuracy, no universality, and can not adapt to indexes of different types, different sampling intervals and different curve waveforms, and the early warning aim can not be achieved when the missed report is too high in a complex scene, and the interference to operation and maintenance personnel during processing is too large when the false report is too high. Therefore, the method for early warning the data in the prior art has the problems of small application range and low abnormal point detection precision.
Disclosure of Invention
Therefore, it is necessary to provide an early warning method, an early warning device, a computer device and a storage medium based on big data analysis to solve the problems of a small application range and low abnormal point detection accuracy of the method for early warning data.
A first aspect of an embodiment of the present application provides an early warning method based on big data analysis, including:
training the data characteristics of the first original data by using a preset algorithm to construct a prediction model; the preset algorithm is selected from a preset database according to the data characteristics;
performing trend prediction on second original data through the prediction model to obtain a predicted value of the second original data;
if the difference set between the predicted value and the second original data is larger than a first dynamic threshold value set, outputting alarm information; the alarm information is used for representing the abnormal condition in the second original data;
constructing an alarm strategy chain based on the incidence relation among alarm items in the alarm information and the abnormal reason of the alarm information; the incidence relation between alarm items in the alarm information is obtained by analyzing the alarm information by using an Apriori algorithm and an FP-Growth algorithm; the abnormal reason of the alarm information is obtained through a decision tree algorithm;
and if the real-time alarm information in the real-time service data is matched with the target node information in the alarm strategy chain, sending early warning information to an early warning terminal according to the target node information.
A second aspect of the embodiments of the present application provides an early warning method and apparatus based on big data analysis, including:
a training unit: training the data characteristics of the first original data by using a preset algorithm to construct a prediction model; the preset algorithm is obtained by selecting from a preset database according to the data characteristics;
a prediction unit: performing trend prediction on second original data through the prediction model to obtain a predicted value of the second original data;
an alarm unit: if the difference set between the predicted value and the second original data is larger than a first dynamic threshold value set, outputting alarm information; the alarm information is used for representing the abnormal condition in the second original data;
a construction unit: constructing an alarm strategy chain based on the incidence relation among alarm items in the alarm information and the abnormal reason of the alarm information; the incidence relation between alarm items in the alarm information is obtained by analyzing the alarm information by using an Apriori algorithm and an FP-Growth algorithm; the abnormal reason of the alarm information is obtained through a decision tree algorithm;
a transmission unit: and if the real-time alarm information in the real-time service data is matched with the target node information in the alarm strategy chain, sending early warning information to an early warning terminal according to the target node information.
A third aspect of embodiments of the present application provides a computer device, including: the system includes a memory, a processor, and computer readable instructions stored in the memory and executable on the processor for causing the computer to perform the steps of a big data analysis based early warning method.
A fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing a computer program, where the computer program is executed by a processor to perform steps of a big data analysis-based early warning method.
The early warning method based on big data analysis provided by the embodiment of the application has the following beneficial effects:
the invention relates to artificial intelligence technology, and provides an early warning method based on big data analysis, which trains the data characteristics of first original data by using a preset algorithm, constructs a prediction model to realize accurate prediction of service data, is used for predicting the trend of the data, constructs the prediction model by using the data characteristics, is suitable for any complex scene, utilizes the prediction model to perform trend prediction on second original data to obtain the predicted value of the second original data, compares the predicted value with the second original data, outputs warning information if the difference set between the predicted value and the second original data is greater than a first dynamic threshold set, looks up a warning strategy chain according to the warning information to check the abnormal condition of the data, constructs a warning strategy chain based on the incidence relation between the warning items in the warning information and the abnormal reason of the warning information, and matches the real-time warning information in the real-time service data with the target node information in the warning strategy chain, early warning information is sent to the early warning terminal according to the target node information, the fault position can be quickly and accurately positioned according to the early warning information, and the usability of the system is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
Fig. 1 is a flowchart illustrating an implementation of an early warning method based on big data analysis according to an embodiment of the present application;
fig. 2 is a block diagram of a structure of an early warning method device based on big data analysis according to an embodiment of the present disclosure;
fig. 3 is a block diagram of a server-side device according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The big data analysis-based early warning method is applied to the field of artificial intelligence and can be executed by a server side.
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of an early warning method based on big data analysis according to an embodiment of the present disclosure.
As shown in fig. 1, an early warning method based on big data analysis includes:
s11: training the data characteristics of the first original data by using a preset algorithm to construct a prediction model; and the preset algorithm is selected from a preset database according to the data characteristics.
In step S11, the first raw data is part of historical application data in the service scenario, and the first raw data is preprocessed and analyzed to obtain data features through a correlation technique. The preset algorithm is obtained by selecting from a preset database according to the data characteristics, the preset database comprises a plurality of preset algorithms, such as a radial basis function neural network algorithm, an LSTM network algorithm, an RNN algorithm and the like, and the data characteristics of the first original data are trained by using the preset algorithm to construct a prediction model.
In this embodiment, the first original data is historical data in a bank transaction process, for example, when a client applies for a loan, a client manager of a bank needs to collect and input a large amount of data information such as enterprise or personal basic information, financial statements, guarantee information, and mortgage information of the client. In the process of integrating the information related to the credit business, commercial banks are mainly faced with the risk of information asymmetry and loan fraud caused by the information asymmetry. Different service scenes comprise different data, the data are massive and diverse, and the data are processed to obtain the data characteristic of the first original data, which is a time sequence characteristic.
In the embodiment, the preset algorithm selects the LSTM network algorithm, the LSTM network can learn long-term dependence information, the LSTM network model has good generalization when a training sequence of a regular language and a simple irregular language is used, and the LSTM has certain capability and success in the aspect of processing complex problems with long-term characteristics and is widely applied. LSTM can learn long-term information, is suitable for processing time-series data, and can easily extract abstract, high-level information or metrics from image or sequential data. It has an internal feedback link that can be used to classify, process and predict time series data. Each cell in the LSTM network consists of three message flow regulators, a forgetting gate, an input gate, and an output gate. After training, the input gate controls which inputs are important enough. The forget gate determines how long and past state memory should be kept, the input gate controls new information from the current input to the state of the cell and how to add, and the output gate determines how much memory is used to produce the output. Together, they allow the network to remember information from the past while discarding unnecessary information.
It should be noted that, according to different service scenarios, corresponding data characteristics are different, and the prediction model may be classified into a stationary type, a periodic type and a non-periodic type. The stable prediction model has no systematic change in mean value and variance in a time sequence, and strictly eliminates periodic change, the periodic prediction model has periodic change in mean value and variance in a time sequence, and the non-periodic prediction model has irregular change in mean value and variance in a time sequence.
In this embodiment, a user savings card transaction is taken as an example, and a typical savings card transaction periodically changes seasonally, and in a busy season, the savings card transaction amount is large, and in a slack season, the savings card transaction amount is relatively reduced, so a prediction model obtained through LSTM network training is a periodic prediction model.
As an embodiment of the present application, step S11 specifically includes:
processing the first original data by utilizing a principal component analysis technology to obtain data characteristics of the first original data; and training the data characteristics of the first original data through a preset algorithm to construct a prediction model.
In this embodiment, a principal component analysis technique is used to extract data features of the first original data, the principal component analysis technique can try to recombine the original variables into a new set of several comprehensive variables that are independent of each other, and meanwhile, according to actual needs, a few sum variables can be extracted from the set of comprehensive variables to reflect information of the original variables as much as possible. For example, each dimension of a set of n-dimensional data has a data feature, but in the actual application process, the data features of the dimensions having small influence on the data features can be ignored, and only the data features of the dimensions having large influence on the data features are calculated, so that the dimensions of the data are reduced, and the main data features are obtained.
It should be noted that, before the feature extraction process is performed, the first raw data needs to be preprocessed and analyzed, the data preprocessing is to supplement, clean, calculate, and the like the raw data, and the data analysis uses a distributed database or a distributed calculation cluster to perform common analysis, classification, summarization, and the like on the mass data stored therein, so as to meet most common analysis requirements. The data cleaning is to clean the error data in the database, because the data in the database is a collection of data oriented to a certain subject, the data is extracted from a plurality of business systems and contains historical data, thus avoiding that some data are error data and some data conflict with each other, and cleaning the error data and the conflict data. For example, in the loan application process, if character information appears in the identity card information in the personal basic information, the character information is regarded as error data, and the identity card information data in the personal basic information is cleaned. The data supplement is to improve the missing data, for example, in the process of bank transaction, the data of each transaction is important data to be referred to for prediction, when the missing data occurs, the data has an influence on the prediction of future transaction, and the complete transaction data needs to be supplemented. Data pre-processing normalizes the data, which can slow down learning with large inputs, so the normalized data is scaled using StandardScaler, which normalizes the data for each dimension.
S12: and performing trend prediction on the second original data through the prediction model to obtain a predicted value of the second original data.
In step S12, the second original data is another part of the history data in the application different from the first original data, and the second original data is substituted into the prediction model to obtain a predicted value of the second original data. For example, two weeks of bank savings card transaction amount data, when the original data of the first week is used for training to obtain a prediction model, the original data of the first week is the first original data, the original data of the second week is predicted to obtain a predicted value, and the original data of the second week is the second original data.
In this embodiment, the second original data is transaction data of a bank savings card, and a data prediction model is obtained through an LSTM network to predict the second original data of the savings card transaction amount and obtain a predicted value. In this embodiment, each minute of the deposit card transaction amount is taken as a node, and the prediction value of each node is obtained through a prediction model.
S13: and if the difference value set between the predicted value and the second original data is larger than a first dynamic threshold value set, outputting alarm information.
In step S13, the predicted value and the second original data both include multiple data, when the predicted value and the second original data are compared, the obtained difference is a difference set, the data in the difference set is compared with the data in the dynamic first threshold set, so as to obtain the data in the difference set that is greater than the dynamic first threshold set, and if the difference set is greater than the data in the dynamic first threshold set, it is determined that an abnormal condition exists in the second original data, and alarm information is output. The alarm information is an alarm behavior, and can be output according to different rules, for example, to eliminate misjudgment of sudden abnormal conditions, a piece of alarm information can be generated when 3 abnormal conditions occur continuously.
In this embodiment, the second original data of the transaction amount of the savings card is predicted to obtain a predicted value, the predicted value is compared with the second original data value to obtain a difference set of the predicted value and the second original data value, and data in the difference set is selected to be larger than threshold data in the dynamic first threshold set. In this embodiment, when the dynamic first threshold is set, different thresholds are set according to different times, and the thresholds are changed over time, for example, in the data transaction process, it is considered that the transaction data in the daytime is larger, the transaction data in the nighttime is smaller, and the maximum limit of the difference between the prediction result and the original data value is also different, when the transaction data is larger, the fluctuation range limit between the prediction value and the original data may be smaller, and when the transaction data is smaller, the fluctuation range limit between the prediction value and the original data may be larger. Therefore, different thresholds are set according to different times, in the embodiment, at the time point from 8:00 to 20:00, the threshold is set to be 5% of the original data, and when the difference value between the predicted value and the original data or the difference value between the original data and the predicted value is greater than 5% of the original data, the point is considered as an abnormal point. At the time of 20:00-8:00, the threshold value is set to be 10% of the original data, and when the difference value between the predicted value and the original data or the difference value between the original data and the predicted value is larger than 10% of the original data, the point is considered as an abnormal point. And outputting alarm information according to the abnormal point.
It should be noted that the dynamic threshold setting is determined according to different service scenarios or different types of data. The threshold is set according to time, and different thresholds can be set according to different time periods, for example, when a periodic prediction model is used, if data is increased or decreased firstly in one period, when the threshold is set, the threshold is set according to the period, and the threshold is set on the basis of original data, the threshold is increased after the percentage of the original data is decreased. The setting of the dynamic threshold improves the real-time accuracy of the quasi-prediction and increases the application range.
As an embodiment of the present application, step S13 specifically includes:
in a preset period, averaging the predicted value and the second original data to obtain an average predicted value and an average second original data in the period; obtaining abnormal cycle points when a difference value set between the average predicted value and the average second original data in the cycle is larger than a first dynamic threshold value set;
and outputting alarm information according to the abnormal period point and a preset period threshold value.
In this embodiment, when the predicted value is compared with the original data, in order to reduce the complexity of calculation, an average value over a period of time is used as a comparison object, for example, an average value every 5 minutes is used as a comparison object, and the average predicted value is compared with the average second original data with a period of 5 minutes.
In this embodiment, after the abnormal period point is obtained, the position of the abnormal period point is recorded, the number of the abnormal period points is continuously obtained, and according to the convergence policy, when a plurality of abnormal points continuously appear, it is considered that an alarm appears. And outputting alarm information when the number of the continuous abnormal periodic points is larger than the preset periodic threshold. For example, when 3 abnormal cycle points occur continuously, the data is considered to be abnormal, and an alarm is generated. The continuous abnormal period points are selected to converge, so that the salient abnormal point caused by burst reasons can be eliminated, for example, if the data is abnormal at a certain time because of network reasons and then the network is recovered to be normal, the abnormal point is considered as the salient abnormal point, and a warning message cannot be generated. Alarm information is output through a convergence strategy, so that the accuracy of outputting the alarm information is improved, and the burr phenomenon in data is weakened.
S14: and constructing an alarm strategy chain based on the incidence relation among the alarm items in the alarm information and the abnormal reason of the alarm information.
In step S14, the reason of the alarm information is the root cause of the abnormal data, there are various reasons for the abnormal data, when the CPU is loaded, the usage rate of the CPU in the database exceeds 100%, and there are many logs to be queried slowly, which will cause the abnormal data, when there are various changes in the interface or there are no fields, the abnormal data will occur in the interface, when the data is redundant, the integrity and consistency of the image data will be maintained, when querying the database, the abnormal data will occur, when the connection between the middleware and the database is not released, the connection will continue to increase, the internal consumption will be exhausted, the abnormal data will occur, and so on. The reasons for generating the data abnormity are various, the reasons for generating the data abnormity are classified by utilizing the decision tree, the determined abnormal reasons are used for constructing an alarm strategy chain, the alarm strategy chain reflects the association between alarm information, and when one piece of alarm information appears, the alarm information which is about to appear can be predicted according to the alarm strategy chain.
In this embodiment, the alarm information is obtained according to the second original data, and is used as a training set to analyze the reason of generating the alarm information, where the characteristics of the data abnormality generated by different reasons are different, for example, when the CPU load generates the data abnormality, the data abnormality is generated in a peak period, which is the alarm information a, so that the data abnormality is characterized by being generally in a fixed time period and having a short duration. When the middleware connection database is not released, the middleware connection database is continuously increased to exhaust the internal consumption, and the characteristic of generating data abnormity is continuous abnormity, namely alarm information B, and the time is long. And obtaining abnormal reasons corresponding to different alarm information by using a decision tree algorithm. Then the association between the different alert information is determined. For example, if the alarm information A and the alarm information B are associated, an alarm strategy chain related to the alarm information A and the alarm information B is constructed.
As an embodiment of the present application, step S14 specifically includes:
classifying and analyzing each alarm item in the alarm information, and obtaining the abnormal reason of the alarm item by using a decision tree algorithm; obtaining association relations among all alarm items in the alarm information by using an Apriori algorithm, an FP-Growth algorithm and abnormal reasons of the alarm items through a data mining technology; and taking the incidence relation among all the alarm items in the alarm information as an alarm strategy chain.
In this embodiment, the decision tree is a decision analysis method for obtaining the probability that the expected value of the net present value is greater than or equal to zero by constructing the decision tree on the basis of the known occurrence probability of each condition, evaluating the risk of the project, and judging the feasibility of the project, and is a graphical method for intuitively applying probability analysis. In machine learning, a decision tree is a prediction model, which represents a mapping relationship between object attributes and object values, where data anomalies are object attributes and causes of anomalies are object values, and a decision tree algorithm is used to obtain the mapping relationship between data anomalies and causes of anomalies. Then the association between the different alert information is determined. For example, if there is alarm information a, alarm information B and alarm information C, association between the alarm information a, the alarm information B and the alarm information C is obtained through Apriori and FP-Growth algorithms, and if there is association between the alarm information a and the alarm information B and the alarm information C, when one of the alarm information is detected, the alarm information B and the alarm information C can be predicted. For example, if data abnormality occurs when data redundancy of a database, an alarm information a is generated, data abnormality occurs due to CPU load, an alarm information B is generated, association between the alarm information a and the alarm information B is obtained through Apriori and FP-Growth algorithms, and the alarm information a and the alarm information B generate an alarm policy chain, the alarm information B can be predicted according to the alarm policy chain when the alarm information a is detected to be generated.
S15: and if the real-time alarm information in the real-time service data is matched with the target node information in the alarm strategy chain, sending early warning information to an early warning terminal according to the target node information.
In step S15, the real-time alarm information is generated by comparing the prediction model with the real-time service data, and is matched with the obtained alarm policy chain to obtain the other alarm information associated with the alarm policy chain, and then the configuration system sends the early warning information. The content management system is used by creators, editors and publishers of the configured content to submit, modify, approve and publish the content. And when the early warning information is generated, the early warning information is sent to the early warning terminal through the configuration system.
In this embodiment, when the alarm policy chain is obtained as alarm information a, alarm information B, and alarm information C are all associated with each other, when the real-time alarm information bit in the real-time service data is alarm information a, the alarm information a is matched with the alarm information a of the target node in the alarm policy chain, so that the obtained early warning information is alarm information B and alarm information C, and the configuration system sends the alarm information B and the alarm information C to the early warning terminal. And after receiving the early warning information, the configuration system sends the early warning information to the early warning terminal.
As an embodiment of the present application, step S15 specifically includes:
comparing the difference set of the predicted value of the real-time service data and the real-time service data with the first dynamic threshold set to obtain real-time alarm information; searching node information associated with the target node information in the alarm strategy chain through matching of the real-time alarm information and the target node information in the alarm strategy chain to obtain the associated node alarm information of the target node in the alarm strategy chain;
in this embodiment, the predicted value of the real-time service data is compared with the real-time service data to obtain the real-time alarm information, for example, when the data monitoring process is performed, the real-time service data is 100, the predicted value of the real-time service data is 50, and the difference between the predicted values of the real-time service data and the real-time service data is greater than the threshold in the dynamic first threshold set, a real-time alarm information a is generated. Searching for alarm information A from an alarm strategy chain, wherein the alarm information A in the alarm strategy chain is target node information, obtaining the associated node alarm information of a target node in the alarm strategy chain according to the target node information, designating a target early warning terminal in a configuration system, and sending early warning information to the target early warning terminal by using the configuration system.
The application provides an early warning method based on big data analysis, which trains the data characteristics of first original data by a preset algorithm, constructs a prediction model, realizes accurate prediction of service data, is used for predicting the trend of the data, utilizes the prediction model to predict the trend of second original data to obtain the predicted value of the second original data, compares the predicted value with the second original data, outputs warning information if the difference set between the predicted value and the second original data is larger than a first dynamic threshold value set, checks the abnormal condition of the data according to the warning information, constructs a warning strategy chain based on the incidence relation among warning items in the warning information and the abnormal reason of the warning information, describes the incidence relation among the warning information, is favorable for monitoring the warning information, and matches the warning real-time information in the real-time service data with target node information in the warning strategy chain, early warning information is sent to the early warning terminal according to the target node information, the fault position can be rapidly located according to the early warning information, and the usability of the system is improved.
Referring to fig. 2, fig. 2 is a block diagram of an apparatus of an early warning method based on big data analysis according to an embodiment of the present disclosure. In this embodiment, the server includes 5 units for executing the steps in the embodiment corresponding to fig. 1, and refer to fig. 1 and the related description in the embodiment corresponding to fig. 1 specifically. For convenience of explanation, only the portions related to the present embodiment are shown. Referring to fig. 2, the big data analysis-based early warning method apparatus 20 includes: training unit 21, prediction unit 22, alarm unit 23, construction unit 24, and sending unit 25, wherein:
the training unit 21 is configured to train data characteristics of the first original data by using a preset algorithm to construct a prediction model;
the prediction unit 22 is configured to perform trend prediction on second original data through the prediction model to obtain a prediction value of the second original data;
the alarm unit 23 is configured to output alarm information if a difference set between the predicted value and the second original data is greater than a first dynamic threshold set;
the construction unit 24 is configured to construct an alarm strategy chain based on an association relationship between alarm items in the alarm information and an abnormal reason of the alarm information; (ii) a
And a sending unit 25, configured to send, if the real-time alarm information in the real-time service data matches with the target node information in the alarm policy chain, the early warning information to the early warning terminal according to the target node information.
As an embodiment of the present application, the early warning method apparatus 20 based on big data analysis further includes:
a first executing unit 26, configured to process the first raw data by using a principal component analysis technique to obtain a data feature of the first raw data; and training the data characteristics of the first original data through a preset algorithm to construct a prediction model.
The second execution unit 27 is configured to average the predicted value and the second original data in a preset period to obtain an average predicted value and an average second original data in the period; obtaining abnormal cycle points when a difference value set between the average predicted value and the average second original data in the cycle is larger than a first dynamic threshold value set; and outputting alarm information according to the abnormal period point and a preset period threshold value.
As an embodiment of the present application, the alarm unit is specifically configured to obtain abnormal data in the second original data according to the predicted value, the second original data, and the first dynamic threshold value set, obtain an abnormal period point according to the abnormal data and a preset period, and output alarm information when the number of the abnormal period points that continuously appear is greater than the preset period threshold value through the abnormal period point and the preset period threshold value.
A third executing unit 28, configured to perform classification analysis on each alarm item in the alarm information, and obtain an abnormal reason of the alarm item by using a decision tree algorithm; obtaining an association relation between alarm items in the alarm information by using an Apriori algorithm and an FP-Growth algorithm through a data mining technology and the abnormal reasons of the alarm items; and taking the incidence relation among all the alarm items in the alarm information as an alarm strategy chain.
The fourth execution unit 29 is configured to compare the difference set between the predicted value of the real-time service data and the real-time service data with the first dynamic threshold set to obtain real-time alarm information; searching node information associated with the target node information in the alarm strategy chain through matching of the real-time alarm information and the target node information in the alarm strategy chain to obtain the associated node alarm information of the target node in the alarm strategy chain; and sending early warning information to an early warning terminal by using a configuration system based on the associated node warning information.
It should be understood that, in the structural block diagram of the apparatus of the early warning method based on big data analysis shown in fig. 2, each unit is used to execute each step in the embodiment corresponding to fig. 1, and for each step in the embodiment corresponding to fig. 1, the detailed explanation has been given in the above embodiment, specifically please refer to fig. 1 and the related description in the embodiment corresponding to fig. 1, and details are not repeated here.
In one embodiment, a computer device is provided, the computer device is a server, and the internal structure diagram of the computer device can be as shown in fig. 3. The computer device 30 includes a processor 31, an internal memory 33, and a network interface 34 connected by a system bus 32. Wherein the processor 31 of the computer device is used to provide computing and control capabilities. The memory of the computer device 30 comprises a readable storage medium 35, an internal memory 33. The readable storage medium 35 stores an operating system 36, computer readable instructions 37, and a database 38. The internal memory 33 provides an environment for the operation of an operating system 36 and computer readable instructions 37 in a readable storage medium 35. The database 38 of the computer device 30 is used to store data relating to pre-warning methods based on big data analysis. The network interface 33 of the computer device 30 is used for communication with an external terminal through a network connection. The computer readable instructions 37, when executed by the processor 31, implement a big data analysis based early warning method. The readable storage medium 35 provided by the present embodiment includes a nonvolatile readable storage medium and a volatile readable storage medium.
It will be understood by those of ordinary skill in the art that all or part of the processes of the methods of the above embodiments may be implemented by hardware related to computer readable instructions, which may be stored in a non-volatile readable storage medium or a volatile readable storage medium, and when executed, the computer readable instructions may include processes of the above embodiments of the methods. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A big data analysis-based early warning method is characterized by comprising the following steps:
training the data characteristics of the first original data by using a preset algorithm to construct a prediction model; the preset algorithm is selected from a preset database according to the data characteristics;
performing trend prediction on second original data through the prediction model to obtain a predicted value of the second original data;
if the difference set between the predicted value and the second original data is larger than a first dynamic threshold value set, outputting alarm information; the alarm information is used for representing the abnormal condition in the second original data;
constructing an alarm strategy chain based on the incidence relation among alarm items in the alarm information and the abnormal reason of the alarm information; the incidence relation between alarm items in the alarm information is obtained by analyzing the alarm information by using an Apriori algorithm and an FP-Growth algorithm; the abnormal reason of the alarm information is obtained through a decision tree algorithm;
and if the real-time alarm information in the real-time service data is matched with the target node information in the alarm strategy chain, sending early warning information to an early warning terminal according to the target node information.
2. The big data analysis-based early warning method according to claim 1, wherein the training of the data features of the first raw data by using the preset algorithm to construct the prediction model comprises:
processing the first original data by utilizing a principal component analysis technology to obtain data characteristics of the first original data;
and training the data characteristics of the first original data through a preset algorithm to construct a prediction model.
3. The big data analysis-based early warning method according to claim 1, wherein if the difference set between the predicted value and the second original data is greater than a first dynamic threshold set, outputting warning information, including:
in a preset period, averaging the predicted value and the second original data to obtain an average predicted value and an average second original data in the period;
obtaining abnormal cycle points when a difference value set between the average predicted value and the average second original data in the cycle is larger than a first dynamic threshold value set;
and outputting alarm information according to the abnormal period point and a preset period threshold value.
4. The big data analysis-based early warning method according to claim 3, wherein the outputting of the warning information according to the abnormal period point and a preset period threshold comprises:
and outputting alarm information when the number of the continuous abnormal periodic points is larger than the preset periodic threshold.
5. The big data analysis-based early warning method according to claim 1, wherein the constructing a warning strategy chain based on the association relationship between the warning items in the warning information and the abnormal reason of the warning information comprises:
classifying and analyzing each alarm item in the alarm information, and obtaining the abnormal reason of the alarm item by using a decision tree algorithm;
obtaining the association relation between the alarm items in the alarm information by using an Apriori algorithm, an FP-Growth algorithm and the abnormal reasons of the alarm items through a data mining technology;
and taking the incidence relation among all the alarm items in the alarm information as an alarm strategy chain.
6. The big data analysis-based early warning method according to claim 1, wherein if real-time warning information in the real-time service data matches target node information in the warning policy chain, sending early warning information to an early warning terminal according to the target node information, comprises:
comparing the difference set of the predicted value of the real-time service data and the real-time service data with the first dynamic threshold set to obtain real-time alarm information;
searching node information associated with the target node information in the alarm strategy chain through matching of the real-time alarm information and the target node information in the alarm strategy chain to obtain the associated node alarm information of the target node in the alarm strategy chain;
and sending early warning information to an early warning terminal by using a configuration system based on the associated node warning information.
7. The big data analysis-based early warning method according to claim 6, wherein the sending early warning information to an early warning terminal by using a configuration system based on the associated node warning information comprises:
and based on the associated node warning information, a target early warning terminal is designated in a configuration system, and early warning information is sent to the target early warning terminal.
8. An early warning method device based on big data analysis, which is characterized in that,
a training unit: training the data characteristics of the first original data by using a preset algorithm to construct a prediction model; the preset algorithm is selected from a preset database according to the data characteristics;
a prediction unit: performing trend prediction on second original data through the prediction model to obtain a predicted value of the second original data;
an alarm unit: if the difference set between the predicted value and the second original data is larger than a first dynamic threshold value set, outputting alarm information; the alarm information is used for representing the abnormal condition in the second original data;
a construction unit: constructing an alarm strategy chain based on the incidence relation among alarm items in the alarm information and the abnormal reason of the alarm information; the incidence relation between alarm items in the alarm information is obtained by analyzing the alarm information by using an Apriori algorithm and an FP-Growth algorithm; the abnormal reason of the alarm information is obtained through a decision tree algorithm;
a transmission unit: and if the real-time alarm information in the real-time service data is matched with the target node information in the alarm strategy chain, sending early warning information to an early warning terminal according to the target node information.
9. A computer device comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein the computer readable instructions are readable instructions generated by the engine of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions for causing the computer to perform the steps of the method of any of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210581629.4A CN114881349A (en) | 2022-05-26 | 2022-05-26 | Big data analysis-based early warning method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210581629.4A CN114881349A (en) | 2022-05-26 | 2022-05-26 | Big data analysis-based early warning method and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114881349A true CN114881349A (en) | 2022-08-09 |
Family
ID=82677222
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210581629.4A Pending CN114881349A (en) | 2022-05-26 | 2022-05-26 | Big data analysis-based early warning method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114881349A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115514613A (en) * | 2022-11-15 | 2022-12-23 | 阿里云计算有限公司 | Alarm strategy obtaining method and device |
| CN116089224A (en) * | 2023-04-11 | 2023-05-09 | 宇动源(北京)信息技术有限公司 | Alarm analysis method, alarm analysis device, calculation node and computer readable storage medium |
| CN116128312A (en) * | 2023-04-17 | 2023-05-16 | 南昌工程学院 | Dam safety early warning method and system based on monitoring data analysis |
| CN116453310A (en) * | 2023-06-14 | 2023-07-18 | 昆山工业大数据创新中心有限公司 | Method and device for alarming temperature abnormality of copper bar |
-
2022
- 2022-05-26 CN CN202210581629.4A patent/CN114881349A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115514613A (en) * | 2022-11-15 | 2022-12-23 | 阿里云计算有限公司 | Alarm strategy obtaining method and device |
| CN115514613B (en) * | 2022-11-15 | 2023-04-11 | 阿里云计算有限公司 | Alarm strategy obtaining method and device |
| CN116089224A (en) * | 2023-04-11 | 2023-05-09 | 宇动源(北京)信息技术有限公司 | Alarm analysis method, alarm analysis device, calculation node and computer readable storage medium |
| CN116089224B (en) * | 2023-04-11 | 2023-06-30 | 宇动源(北京)信息技术有限公司 | Alarm analysis method, alarm analysis device, calculation node and computer readable storage medium |
| CN116128312A (en) * | 2023-04-17 | 2023-05-16 | 南昌工程学院 | Dam safety early warning method and system based on monitoring data analysis |
| CN116128312B (en) * | 2023-04-17 | 2023-07-14 | 南昌工程学院 | Dam safety early warning method and system based on monitoring data analysis |
| CN116453310A (en) * | 2023-06-14 | 2023-07-18 | 昆山工业大数据创新中心有限公司 | Method and device for alarming temperature abnormality of copper bar |
| CN116453310B (en) * | 2023-06-14 | 2023-09-05 | 昆山工业大数据创新中心有限公司 | Method and device for alarming temperature abnormality of copper bar |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114881349A (en) | Big data analysis-based early warning method and storage medium | |
| US11410187B2 (en) | Feature drift hardened online application origination (OAO) service for fraud prevention systems | |
| CN108153603B (en) | Database server fault handling method, device and storage medium | |
| CN112800116B (en) | Method and device for detecting abnormity of service data | |
| US20200166921A1 (en) | System and method for proactive repair of suboptimal operation of a machine | |
| KR20230030542A (en) | AI-based facility data anomaly detection system and method using random cut forest algorithm | |
| CN114091930A (en) | Service index early warning method and device, electronic equipment and storage medium | |
| Gęca | Performance comparison of machine learning algotihms for predictive maintenance | |
| CN116909712A (en) | Intelligent task scheduling system and method based on machine learning | |
| CN116680196A (en) | Test management method and device based on artificial intelligence and electronic equipment | |
| CN113535449B (en) | Abnormal event restoration processing method and device, computer equipment and storage medium | |
| CN115174129A (en) | Abnormal node detection method and device, computer equipment and storage medium | |
| CN114676021A (en) | Job log monitoring method and device, computer equipment and storage medium | |
| Andersson et al. | Anomaly detection for the centralised Elasticsearch service at CERN | |
| CN113822421B (en) | Neural network-based anomaly locating method, system, equipment and storage medium | |
| Rajasekaran et al. | Anomaly detection of smart grid equipment using machine learning applications | |
| CN114741673B (en) | Behavior risk detection method, clustering model construction method and device | |
| CN117807406B (en) | Enterprise account management method, system, equipment and storage medium of payment platform | |
| US20240126910A1 (en) | Providing a graphical representation of anomalous events | |
| KR102617150B1 (en) | Device, method and program for preventing false positives based on artificial intelligence using rule filtering | |
| Ranaweera et al. | Anomaly detection in complex trading systems | |
| CN113822421A (en) | Neural network based anomaly positioning method, system, equipment and storage medium | |
| CN114186031A (en) | System fault prediction method, device, computer equipment and storage medium | |
| CN114386619A (en) | Machine learning model management method, device, computer equipment and storage medium | |
| CN117579332A (en) | Network threat detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |