CN117421684B

CN117421684B - Abnormal data monitoring and analyzing method based on data mining and neural network

Info

Publication number: CN117421684B
Application number: CN202311718358.3A
Authority: CN
Inventors: 林明; 胡琴; 卢山
Original assignee: Yizhigu Technology Group Co ltd
Current assignee: Yizhigu Technology Group Co ltd
Priority date: 2023-12-14
Filing date: 2023-12-14
Publication date: 2024-03-12
Anticipated expiration: 2043-12-14
Also published as: CN117421684A

Abstract

The invention relates to the technical field of abnormal data monitoring, in particular to a method for monitoring and analyzing abnormal data based on data mining and a neural network, which comprises the following steps: the method comprises the steps of preprocessing real-time communication data, encoding the data, detecting and identifying the abnormal communication data by using an abnormal data detection method, carrying out standardized processing on the abnormal communication data, accurately identifying and controlling abnormal behaviors of equipment by using the abnormal detection of an enhanced weighted random forest algorithm, identifying the classified abnormal data by using an enhanced stream type abnormal detection algorithm and a trained neural network model, detecting and early warning the abnormal data in real time by using an adaptive method, and adjusting by using an adaptive learning rate according to an adaptive gradient adjustment factor. The method has the advantages of good effectiveness and accuracy, high efficiency, low energy consumption, high intelligence and the like.

Description

Abnormal data monitoring and analyzing method based on data mining and neural network

Technical Field

The invention relates to the technical field of abnormal data monitoring, in particular to a method for monitoring and analyzing abnormal data based on data mining and a neural network.

Background

Anomaly detection is a hot topic in various fields at present, and is widely applied to the fields of health care, intelligent transportation, large-scale production systems, network security and the like. Abnormality detection targets in different fields are different, for example, in the field of health care, abnormality detection is used to monitor human diseases; in intelligent traffic, it is used to find traffic accidents; in large production systems, for equipment failure diagnosis; in network security, it is used to detect network intrusion and the like. In the field of mobile communication systems, common abnormal data include signal quality anomalies, dropped call rate anomalies, call completion rate anomalies, data transmission rate anomalies, base station failure or anomalies, and traffic anomalies.

The existing anomaly detection method comprises clustering, random forest, single-class support vector machine and the like, training is carried out on data by using a machine learning algorithm, and then the anomaly data is detected through a model. And judging whether the abnormality exists or not by analyzing the statistical characteristics of the data. The existing abnormality detection method also comprises ARIMA model, exponential smoothing and the like, wherein data are regarded as time series, and whether abnormality exists or not is judged by analyzing the trend and periodicity of the series. The current communication system is mainly focused on providing high-speed, stable and safe communication connection, meeting the requirements of different users and diversified communication modes of application scenes, and has good expandability and interoperability. Meanwhile, requirements on mobility, low delay, high reliability, privacy protection and the like are also increasing. Since existing anomaly detection methods fail to meet the needs of current communication systems, these anomaly data are monitored and analyzed in conjunction with data mining and neural networks.

In the prior art, little research is done on abnormal data monitoring by using a neural network and data mining, and chinese patent application No. 201811522835.8 discloses a mobile communication data traffic abnormality monitoring system, which includes a traffic monitoring unit, an information following unit, a traffic analyzing unit, a personal database, a processor, a display unit, an event recording unit, a reminding unit and a data confirmation unit. The flow monitoring unit is used for monitoring communication flow of the communication equipment, wherein the communication flow comprises data flow and real-time rate information. The data flow represents the total consumption of the data flow until the current month, and the real-time rate information represents the real-time network speed when the network access is performed. According to the scheme, the data traffic service condition of the mobile equipment is monitored in real time through the traffic monitoring unit, and the communication traffic is transmitted to the traffic analysis unit. The flow analysis unit is combined with the abnormality analysis step to calculate and obtain an instant stable value. And then, calculating a stable difference value according to the instant stable value, and judging whether the data flow access of the user is abnormal or not by utilizing the difference value. But normal traffic typically occupies the vast majority and abnormal traffic is relatively small, which causes tag imbalance problems.

Most of the existing anomaly detection methods adopt neural network training anomaly detection models, and Chinese patent application number 202110397166.1 discloses an anomaly data detection method based on improved EMD and neural network models, which comprises the following steps: drawing an envelope curve on the original signal by using an envelope function; inputting the drawn envelope signals into a modified EMD algorithm to extract characteristic variables (IMF components); modifying the characteristic variable extraction flow, namely modifying a cubic spline interpolation function into an fminbnd function, wherein an envelope function adopts an inventcope function; inputting the extracted characteristic variables into a neural network model; after three layers of screening of the neural network model and matching with the frequency spectrum of the fault cause, finding out the fault point and the fault cause; however, the method has the phenomenon of modal aliasing, noise interferes with the sampled signal, and the characteristic variable of the signal cannot be accurately extracted.

The invention aims to solve the technical problems of low analysis speed and inaccurate analysis result in the existing communication abnormal data detection.

For this purpose, abnormal data monitoring and analysis methods based on data mining and neural networks are proposed.

Disclosure of Invention

The invention aims to provide an abnormal data monitoring and analyzing method based on data mining and a neural network, which is characterized in that real-time communication data is preprocessed, then data encoding is carried out, abnormal communication data is detected and identified by using an abnormal data detection method, and real-time communication data trend is predicted and early-warned by using a data mining and neural network technology. And identifying the abnormality of the standardized data by using a neural network related algorithm, and realizing intelligent early warning of communication abnormal data.

In order to achieve the above purpose, the present invention provides the following technical solutions:

the abnormal data monitoring and analyzing method based on the data mining and the neural network comprises the following steps:

acquiring a high-dimensional real-time communication data set of a user side, and classifying and marking the high-dimensional real-time communication data set;

preprocessing the classified high-dimensional real-time communication data set;

extracting features of the preprocessed high-dimensional real-time communication data set by using a bi-directional cyclic neural network BiRNN model;

performing dimension reduction on the extracted features of the high-dimension real-time communication data set by using a Principal Component Analysis (PCA) method, and encoding the dimension-reduced features by using a discrete encoding method to form encoded data;

Classifying the encoded data into a normal data set and an abnormal data set by an enhanced weighted random forest algorithm;

inputting the abnormal data set into a convolutional neural network detection model for training;

calculating a local anomaly factor value by comparing the average density of each data point with the adjacent neighbor points, the data points with the local anomaly factor value less than a certain threshold being anomaly points;

periodically updating the average density of the neighborhood set, and updating the local anomaly factor value of the data points in the neighborhood according to the updated average density;

processing the abnormal data set in real time by using an enhanced flow type abnormal detection algorithm and a trained convolutional neural network model, dynamically adjusting according to the environment and the data distribution of the data set change, and automatically identifying the abnormal type of the abnormal data;

outputting results and early warning, and detecting the abnormal data set in real time by a self-adaptive method; judging whether the data sample is abnormal or not according to the threshold value, performing corresponding early warning processing, and feeding early warning information back to an early warning system;

the self-adaptive method tracks the change of the data in real time, dynamically adjusts the model and the parameters, can be more suitable for different types and distribution of data, provides more accurate abnormality detection and early warning results, and ensures that the early warning accuracy reaches more than 90 percent.

Preferably, the data preprocessing specifically includes:

unique attribute processing: the unique attribute refers to the characteristic capable of uniquely identifying the sample to be identified, but has no influence on the identification of abnormal data, and is directly deleted;

missing data processing: finding the position of the missing value; filling the missing value; quickly acquiring index information of a missing value in the characteristic data, and processing the missing value by using a position logic index;

correlation attribute merging: for characteristic data with obvious correlation, combining the correlated data into one data by using a data operation, and deleting non-correlated data; the data is combined by adding two attribute columns feature 1 and feature 2 into the data, and the deleting operation is performed after the attribute combination.

Automatic cleaning: automatically identifying and processing the problems of missing values, abnormal values and repeated values in the data by using an automatic algorithm and a tool through a semi-supervised learning algorithm and an abnormal detection algorithm; the method comprises the following specific steps:

loading data to be cleaned into an appropriate computing environment;

missing value processing: and (3) missing value detection: detecting missing values in the data using a statistical index, a visualization method, or a clustering algorithm; missing value filling: filling the missing value by using an interpolation method according to the characteristics and meaning of the data;

Outlier processing: abnormal value detection: identifying outliers in the data using an outlier detection algorithm; outlier processing: selecting an abnormal value deleting method for processing according to the property of the abnormal value;

repeating the value processing: duplicate value detection: detecting duplicate records in the data using a unique identifier of the data or a combination of fields; repeating the value processing: selecting to reserve the first record to process the repeated value according to the service requirement;

verification and evaluation: verifying and evaluating the cleaned data, checking the cleaning effect, and comparing the cleaning effect with the original data; the missing value ratio, the outlier ratio, are used to evaluate the accuracy and integrity of the cleaning results.

Preferably, the encoding the dimension reduction feature by using a discrete encoding method to form encoded data specifically includes:

among the relevant attributes, one or more discrete attributes are categorized into One category, and each category is individually encoded using the one_hot encoding method.

Preferably, the training process of the convolutional neural network detection model comprises the following steps:

initializing parameters with random values;

inputting the coded data in the abnormal data set, and obtaining an output value through forward propagation of a convolution layer, a pooling layer and a full connection layer;

Calculating training errors between the output value and the target value of each layer of the convolution layer, the pooling layer and the full-connection layer; the target value refers to the classified abnormal data set;

performing back propagation updating weight according to the training error;

when the training error does not change significantly within 100 iterations, the training process is terminated; if the termination condition is not met, re-executing the input data;

and obtaining a trained convolutional neural network model.

Preferably, the number of neurons in the input layer of the convolutional neural network is set according to the number of the input abnormal communication data features and the coding bit number of each abnormal communication data feature, and the calculation method is as follows:

；

where M is the number of features of the input abnormal communication data,is the number of coding bits of the ith feature, +.>Is the number of neurons of the input layer; the abnormal communication data characteristic types comprise frequency abnormality, time delay abnormality, abnormal data packet frequency, abnormal data packet size, signal strength abnormality, abnormal protocol behavior and data integrity abnormality.

Preferably, the encoded data in the abnormal data set is input into a convolutional neural network detection model M1 for training;

the convolutional neural network model M1 comprises 12 convolutional layers and 8 pooling layers, the convolutional layers adopt 3 small convolutional kernels of 3*3, and the 8 pooling layers adopt maximum pooling;

Respectively adding residual error connection modules into a 3 rd convolution layer, a 4 th convolution layer and a 5 th convolution layer in the model M1, wherein one residual error connection module consists of two convolution layers, and batch normalization and activation functions are contained between the convolution layers; adding the output and the input of the 3 rd convolution layer of the model M1, transmitting the added result to the 4 th convolution layer as input, applying the residual error connection module again, adding the output and the input of the 4 th convolution layer, transmitting the added result to the 5 th convolution layer, continuously applying the residual error connection module, and obtaining a convolution neural network model M2;

inputting the encoded data in the abnormal data set into the convolutional neural network model M2, and performing convolutional, residual error connection and pooling operation;

the middle layer is 1 layer, and the neuron number of the middle layer is。

Preferably, the number of neurons of the output layer is equal to the number of demand categories; the one_hot encoding method is used to encode the demand state and the function of the output neuron is selected as the log function.

Preferably, classification is performed by an enhanced weighted random forest algorithm, comprising the steps of:

inputting the encoded data into an enhanced weighted random forest model for training;

Performing anomaly detection and classification using a trained, enhanced weighted random forest model, separating the encoded data into a normal data set and an anomaly data set;

assigning a weight to each abnormal data sample;

for unlabeled new samples, the trained model uses the learned parameters and weights thereof to conduct classification inference according to the learned rules, and the new samples are distributed to normal categories or abnormal categories; the parameters and weights of the model are learned from the labeled samples by an optimization algorithm in the training process.

Preferably, the tag classification of the data refers to dividing all data sets into normal data sets and abnormal data sets, the encoded data is classified by an enhanced weighted random forest algorithm, and the weight function of the enhanced weighted random forest algorithm is as follows:

；

wherein,is the unbalance of the decision tree, N is the number of decision trees, < >>Is the voting weight of the decision tree;

given N balanced sub-training sets, training is carried out on the N balanced sub-training sets to obtain N decision tree classifiers，/>Is a natural number from 1 to N;

the final classifier is obtained by weighted voting and is expressed as follows:

；

Wherein Y is an abnormal data set;

the finally obtained classifierThe method is used for testing the classification effect by the test set.

Preferably, the abnormal data set is identified by using an enhanced flow type abnormal detection algorithm and a trained convolutional neural network model, and the calculation process of the local abnormal factor in the enhanced flow type abnormal detection algorithm is as follows:

for each outlier data point in the real-time communication dataset：

Calculation ofK nearest neighbor of (2) and obtain the neighborhood set +.>；

Calculating data pointsTo data point->K reachable distance +.>Which is data point>K adjacent distance>And (4) point->And->European distance between->Is the maximum value of (2);

；

according to the set distance thresholdDefining the data points with the distance smaller than or equal to the threshold value as the data points in the neighborhood of the target point; target point->Is expressed as:

；

wherein,representing target point->A set of data points within a neighborhood of (a); d is the dataset; />Representing data points +.>Is->A Euclidean distance between them; />Indicating that the distance is less than or equal to a set threshold +.>Data point->Belonging to the target point->Is within a neighborhood of (2); different k reachable distances correspond to different distance thresholds, and the threshold is dynamically adjusted according to the k reachable distances >The method comprises the steps of carrying out a first treatment on the surface of the Data point +.>To data point->K reachable distance +.>And threshold->Comparing; if the k reachable distance of a certain data point is smaller than the threshold +.>Marking it as an anomaly of the same type; if successive outliers occur, the threshold value +.>To improve accuracy; if no outlier occurs, the threshold value +.>To increase sensitivity;

calculation and calculationNeighborhood set of->Is expressed as> ；

；

Calculation ofIs->；

Wherein M is the number of data points; by dynamically adjusting threshold valuesSo that abnormal data points in more adjacent areas are detected.

Preferably, the self-adaptive method is utilized to enable the abnormality detection model to be automatically adjusted according to the change of the data, so as to adapt to the new data distribution and mode change; in the adaptive methodThe updated formula of (c) is as follows:

；

wherein,is a parameter->Is (are) updated value->Is self-adaptive learning rate->Is an adaptive gradient adjustment factor,/->Is the first moment of the gradient, i.e. the mean,/->Is the second moment of the gradient, i.e. the variance, +.>Is a smooth item->Is the current gradient;

the learning rate is adjusted according to the self-adaptive gradient adjustment factor, and the parameters of the model are updated by using the adjusted learning rate; when the adaptive gradient adjustment factor becomes large, the learning rate becomes small; when the self-adaptive gradient adjustment factor becomes smaller, the learning rate becomes larger; the adaptive gradient adjustment factor affects not only the absolute magnitude of the learning rate, but also the rate of change of the learning rate; the change amplitude of the learning rate in the training process is adjusted by controlling the change rate of the self-adaptive gradient adjustment factor; an adaptive gradient adjustment factor greater than a certain threshold may result in a slower change in learning rate, while an adaptive gradient adjustment factor less than a certain threshold may result in a faster change in learning rate.

Compared with the prior art, the invention has the beneficial effects that:

1. the abnormal data points are effectively identified by classifying through an abnormal detection algorithm of the reinforced weighted random forest algorithm, the classification accuracy is improved, the characteristics of the abnormal data are better captured, a large-scale high-dimensional data set is effectively processed, and the real-time abnormal detection requirement is met. The enhanced weighted random forest algorithm allows the model to better handle sample size differences between different classes. By giving different weights to the data samples, the model is more concerned about the samples of few categories, thereby improving the classification accuracy. The weighted random forest algorithm considers the importance of the features and identifies the most discriminative features for classification or anomaly detection tasks by evaluating the weights of the features. This helps to improve the accuracy and efficiency of the model and reduces the impact of irrelevant or redundant features. The random forest algorithm has strong robustness, and has certain robustness on noise and abnormal values in data. The stability of the model is further improved through the improvement of the weighted random forest algorithm, and the problem of over-fitting of abnormal samples is reduced, so that the overall model performance is improved.

2. And predicting and early warning the trend of the real-time communication data by utilizing an enhanced stream type anomaly detection algorithm, processing the high-dimensional data stream in real time, immediately identifying the anomaly data, continuously updating a model according to new data, and adaptively adapting to the change of data distribution. The reinforced flow type anomaly detection algorithm has self-adaptability, and automatically learns and adjusts the model to adapt to the change of data distribution. Different k reachable distances correspond to different distance thresholds, and the threshold is dynamically adjusted according to the k reachable distancesThis improves the robustness and adaptability of the algorithm, ensuring that the performance of the model remains efficient during long-term operation. The streaming anomaly detection algorithm uses incremental learning to model update new data samples without the need to reprocess the entire data set. Thus, the calculation resources are saved, and the algorithm efficiency is improved. By dynamically adjusting threshold->So that abnormal data points in more adjacent areas are detected.

3. The abnormal data is detected and early-warned through a self-adaptive method, and a detection model or rule can be automatically adjusted according to the actual condition of the data so as to adapt to different data distribution and abnormal modes. The method ensures that the learning rate is automatically adapted to different gradient conditions in the training process, thereby achieving better optimization effect; the method can adapt to dynamic change of data and new abnormal modes, and improves robustness and adaptability of the system. And an abnormality detection algorithm is dynamically adjusted according to the distribution and the change of the actual data by using an adaptive method, so that the detection accuracy is improved. The self-adaptive method can capture the change modes and characteristics of the data and correspondingly adjust the abnormality detection model so as to better identify the abnormal data. The self-adaptive method can analyze and process the data in real time, timely detect abnormal data and make early warning.

Drawings

FIG. 1 is a flow chart of a method for monitoring and analyzing abnormal data based on data mining and neural network according to the present invention;

FIG. 2 is a graph of classification effects of various forms of models of the present invention;

FIG. 3 is a graph comparing weight functions of the present invention;

FIG. 4 is a graph comparing local anomaly factors of the present invention;

FIG. 5 is a graph comparing adaptive learning rates of the adaptive method of the present invention;

fig. 6 is a graph comparing early warning accuracy of the adaptive method of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The embodiments described herein are merely some, but not all embodiments of the invention. All other embodiments, which come within the scope of the invention without inventive work, are within the scope of the invention.

Referring to fig. 1 to 6, the present invention provides an abnormal data monitoring and analyzing method based on data mining and neural network, and the technical scheme is as follows:

Acquiring a high-dimensional real-time communication data set of a user side, and carrying out marking classification on the high-dimensional real-time communication data set; the high-dimensional dataset includes the following 10 dimensional data: timestamp, sender and receiver identification, communication mode, communication duration, communication quality indicator, bandwidth usage, communication location, data traffic, network topology, and user behavior; removing users without abnormal communication behaviors, and retaining abnormal data samples; the abnormal data comprise abnormal signal quality, abnormal call drop rate, abnormal call completing rate, abnormal data transmission rate, base station fault or abnormal state and abnormal flow;

preprocessing the classified high-dimensional real-time communication data set; the data preprocessing comprises automatic cleaning, multi-source data integration, abnormality detection and data restoration, unique attribute deletion, relevant attribute integration and missing value processing, and the acquired high-dimensional data is converted into a data set which does not contain missing data and only contains effective characteristics;

the data preprocessing specifically comprises the following steps:

loading data to be cleaned into an appropriate computing environment;

Extracting features of the preprocessed high-dimensional real-time communication data set by using a bi-directional cyclic neural network BiRNN model, extracting features suitable for anomaly detection, removing data which cannot possibly have anomalies from the data, and extracting the anomaly data;

normalizing the abnormal data, performing dimension reduction on the extracted characteristics of the high-dimension real-time communication data set by using a Principal Component Analysis (PCA) method, and encoding the dimension reduction characteristics by using a discrete encoding method to form encoded data;

the step of encoding the dimension reduction feature by using a discrete encoding method to form encoded data specifically comprises the following steps:

assigning a weight to each abnormal data sample;

The marking classification of the data refers to that all data sets are divided into normal data sets and abnormal data sets, the coded data are classified through an enhanced weighted random forest algorithm, and the weight function of the enhanced weighted random forest algorithm is as follows:

（1）；

（2）；

wherein Y is an abnormal data set;

the finally obtained classifier The method is used for testing the classification effect by the test set.

Inputting the abnormal data into a convolutional neural network detection model for training;

the training process of the convolutional neural network detection model comprises the following steps:

initializing parameters with random values;

performing back propagation updating weight according to the training error;

and obtaining a trained convolutional neural network model.

The method comprises the steps of setting the number of neurons in an input layer of a convolutional neural network according to the number of input abnormal communication data features and the coding bit number of each abnormal communication data feature, wherein the calculation method is shown in the following formula:

；（3）

And inputting the encoded data in the abnormal data set into a convolutional neural network detection model M1 for training.

the middle layer is 1 layer, and the neuron number of the middle layer is。

The number of neurons of the output layer is equal to the number of demand classifications; the one_hot encoding method is used to encode the demand state and the function of the output neuron is selected as the log function.

identifying the abnormal data set by using an enhanced flow type abnormal detection algorithm and a trained convolutional neural network model, wherein the calculation process of local abnormal factors in the enhanced flow type abnormal detection algorithm is as follows:

for each outlier data point in the real-time communication dataset：

Calculation ofK nearest neighbor of (2) and obtain the neighborhood set +.>；

；

according to the set distance thresholdDefining the data points with the distance smaller than or equal to the threshold value as the data points in the neighborhood of the target point; target point- >Is expressed as:

；

wherein,representing target point->A set of data points within a neighborhood of (a); d is the dataset; />Representing data points +.>Is->A Euclidean distance between them; />Indicating that the distance is less than or equal to a set threshold +.>Data point->Belonging to the target point->Is within a neighborhood of (2); different k reachable distances correspond to different distance thresholds, and the threshold is dynamically adjusted according to the k reachable distances>The method comprises the steps of carrying out a first treatment on the surface of the Data point +.>To data point->K reachable distance +.>And threshold->Comparing; if the k reachable distance of a certain data point is smaller than the threshold +.>Marking it as an anomaly of the same type; if a continuous number of outliers occurs, the threshold value +.>To improve accuracy; if no or few outliers occur, the threshold value +.>To increase sensitivity;

calculation and calculationNeighborhood set of->Is expressed as> ；

；

；（4）

Calculation ofIs->；（5）

The self-adaptive method is utilized to enable the anomaly detection model to be automatically adjusted according to the change of the data, and adapt to new data distribution and mode change; in the adaptive methodThe updated formula of (c) is as follows:

；（6）

the learning rate is adjusted according to the self-adaptive gradient adjustment factor, and the parameters of the model are updated by using the adjusted learning rate; when the adaptive gradient adjustment factor becomes large, the learning rate becomes small; when the self-adaptive gradient adjustment factor becomes smaller, the learning rate becomes larger; the method ensures that the learning rate is automatically adapted to different gradient conditions in the training process, thereby achieving better optimization effect; the adaptive gradient adjustment factor affects not only the absolute magnitude of the learning rate, but also the rate of change of the learning rate; the change amplitude of the learning rate in the training process is adjusted by controlling the change rate of the self-adaptive gradient adjustment factor; an adaptive gradient adjustment factor greater than a certain threshold may result in a slower change in learning rate, while an adaptive gradient adjustment factor less than a certain threshold may result in a faster change in learning rate.

As an embodiment of the present invention, a mobile communication network operator in a certain area desires to find out problems and potential failure causes by analyzing abnormal data such as signal quality abnormality, call drop rate abnormality, call completion rate abnormality, data transmission rate abnormality, base station failure or abnormal state, and traffic abnormality, so as to take corresponding measures to improve network quality. Various anomaly data including signal quality, call drop rate, call completion rate, data transmission rate, base station status, and traffic data are first collected. The data is preprocessed, including data cleaning, outlier removal, normalization, etc. Exploratory analysis, including statistical description, visual analysis, etc., is then performed on each anomaly data. For example, a line graph or a bar graph is drawn to observe the time-series change condition of abnormal data and the correlation with other indexes. An anomaly detection algorithm (random forest algorithm) is used to identify outlier data points. For each outlier data point, the cause and influencing factors behind it are further analyzed, e.g. looking at the network equipment, base station location, weather conditions etc. related to the outlier. And establishing an abnormality detection model according to the existing data and the characteristics of the known abnormality. At the same time, features such as signal strength, network load, antenna direction, etc. are extracted based on existing knowledge. And determining specific reasons for the abnormality according to the result of the model and the feature importance analysis, and providing corresponding problem solutions. For example, if a base station failure rate in a certain area is found to be high, engineering maintenance or increased investment may be required to improve the stability of the base station apparatus. According to the proposed problem solution, corresponding improvement measures are implemented and the improved data changes are of interest. The improvement effect is monitored and evaluated, and if the results are still unsatisfactory, further optimization of the solution or re-identification of other potential anomalies is required.

As an embodiment of the invention, reference is made to fig. 1, which is a flow chart of the method according to the invention.

Acquiring a high-dimensional real-time communication data set of a user side, and carrying out marking classification on the high-dimensional real-time communication data set; removing users without abnormal communication behaviors, and retaining abnormal data samples;

normalizing the abnormal data, performing dimension reduction on the extracted high-dimension real-time communication data set features by using a Principal Component Analysis (PCA) method, and encoding the dimension reduction features by using a discrete encoding method to form encoded data;

Classifying by an anomaly detection algorithm of the enhanced weighted random forest algorithm; dividing the encoded data into a normal data set and an abnormal data set;

processing the data flow in real time by using an enhanced flow type anomaly detection algorithm and a trained convolutional neural network model, dynamically adjusting according to the continuously changing data distribution, and automatically identifying the anomaly type of the anomaly data;

and outputting and early warning results, detecting and early warning the abnormal data set in real time through a self-adaptive method, and feeding early warning information back to an early warning system.

As an embodiment of the present invention, referring to fig. 2, a classification effect diagram of various form models is shown.

The abnormal communication data is processed using a time sequence construction method and then input into a classifier. After operation, the classification effect of the traditional model and various models is obtained. The results of the operation of each model are shown in fig. 2.

As can be seen from fig. 2, the recall of the time series exponential model in the form of the ratio and the first relative value is highest. The time series exponential model in differential form is best in terms of accuracy. It is also seen from fig. 2 that the recall and accuracy exhibit a one-time fluctuation law. The comparison of classification accuracy under different hidden layer structures is shown in table 1.

As seen from table 1, concealment levels 1, 2, 3 and 4 show good classification accuracy, all reaching 91% or more. After 1000 iterations, the classification accuracy of the second stage reaches 99.22%, which is the maximum of classification accuracy. Therefore, it is concluded that the convolutional neural network model with the 4-layer hidden layer structure has good classification accuracy.

Table 1 comparison of classification accuracy under different hidden layer structures

As can be seen from Table 2, the classification and identification error rate of the classifier designed by the invention is 27.51%, and the accuracy is 72.49%. The values and actual values in table 2 represent the number of times there is no physical unit. Experiments show that the main cause of the error rate is the delay of the communication anomaly detection result. According to the example analysis, the abnormal data monitoring and analyzing method based on the data mining and the neural network has good effectiveness and accuracy and has a certain practical value.

Table 2 error rate of neural network classification

As an embodiment of the present invention, refer to fig. 3, a comparison graph of weight functions.

Assuming 5 decision trees, different weights are given to different samples according to the frequency of the samples in the data set or the importance degree of specific attributes. When (when) Taking 10%>Take 20->Taking 30%>40 parts of (I) in the middle of (II)>Taking 50, N is equal to 5, then +.>，/>，，/>，/>The method comprises the steps of carrying out a first treatment on the surface of the When->50 parts of (I) in the middle of (II)>40 parts of (I) in the middle of (II)>Taking 30%>Take 20->Taking 10, N is equal to 5, then +.>，，/>，/>，/>。

As shown in fig. 3, as the number of decision trees increases,the weight value decreases accordingly. With decreasing unbalance of the decision tree, +.>The weight value increases. The reinforced weighted random forest algorithm performs weighting treatment on the samples by introducing a weight function so as to improve the accuracy and the robustness of the model. By introducing a weight function, the weighted random forest is more concerned with important or scarce samples and effectively handles unbalanced data sets. This improves the accuracy of the classification model over a few classes or important samples and reduces the risk of misclassification. The weighted random forest enables the model to more robustly cope with noise, outliers and outliers in the data by weighting the samples. This helps to reduce the impact of these interference factors on the model and improves the stability of the model. By carrying out weighting treatment on the samples, the weighted random forest better captures important characteristics and modes of sample distribution, so that the generalization capability of the model is improved. This helps reduce the over-fitting phenomenon and improves the predictive power of the model for unknown data.

As an embodiment of the present invention, reference is made to fig. 4, which is a comparison graph of local anomaly factors.

Assuming that for each data point, its 10 nearest neighbors neighborhood is determined, the reachable distance of each data point to each point in its neighborhood is calculated, and then the average of the inverse of the reachable distance is calculated. For each data point, the LOF value for each point in its neighborhood is calculated and then the average of these LOF values is taken as the LOF value for that data point. A threshold is set based on the calculated LOF value, and if the LOF value of the data point exceeds the threshold, it is determined as an outlier.

As shown in fig. 4, as the number of nearest neighbors increases, the LOF value decreases. As the achievable distance increases, the LOF value decreases. The LOF algorithm can effectively find and identify outliers in the flow anomaly detection. The LOF algorithm is able to capture local anomaly patterns, not just global anomaly patterns, by taking into account the density and outlier degree of the data points relative to their neighborhood. Because local neighborhood calculation is adopted, the LOF algorithm is sensitive to the change of data distribution, and a new abnormal mode in the data stream can be captured in time. When the LOF algorithm calculates the average density and the local anomaly factors, some optimization techniques such as approximate calculation and index structure are adopted, so that the calculation complexity is reduced, and the algorithm efficiency is improved.

As an embodiment of the present invention, referring to fig. 5, a comparison chart of adaptive learning rates of an adaptive method is shown.

Assume that1->Taking 0.3-0.7%>Taking 300%>Taking 100.

As shown in fig. 5, with the smoothing termAn increase in the update value; as the learning rate increases, the update value decreases. By adaptive gradient adjustment factor->Offset correction is performed to adaptively adjust the learning rate, and the learning rate is divided by the square root of these to perform parameter updating. Not only can the learning rate be adaptively adjusted, but also the updating amplitude and the dynamic range of different parameters can be adapted, so that the training effect and the training stability are improved. The self-adaptive learning rate method can dynamically adjust the learning rate according to the current situation, thereby improving the efficiency and performance of the algorithm.

As an embodiment of the present invention, referring to fig. 6, a comparison chart of early warning accuracy of the adaptive method is shown.

As shown in fig. 6, as the learning rate increases, the higher the early warning accuracy is, the more than 90%; along with the increase of the self-adaptive gradient adjustment factor, the early warning accuracy is improved. The adaptive learning rate may dynamically adjust the learning rate based on the model's performance during training so that the model can converge to an optimal solution more quickly. The adaptive gradient adjustment factor can dynamically adjust the update amplitude according to the information of the gradient, so that the model parameters can be updated towards the optimal direction more quickly. The proper self-adaptive learning rate and gradient adjustment factor can help the model to better converge to the optimal solution, and the accuracy and stability of the model parameters are improved. When the model parameters are more accurate and stable, the early warning accuracy of the abnormal detection and prediction model can be correspondingly improved. The self-adaptive method can dynamically adjust the model and parameters according to the change and heterogeneity of the data, and can be better adapted to different abnormal conditions. The dynamic adaptability can improve the accuracy of anomaly detection and prediction, thereby improving the early warning accuracy.

In summary, the abnormal data monitoring and analyzing method based on data mining and neural network comprises the following steps:

the data preprocessing specifically comprises the following steps:

loading data to be cleaned into an appropriate computing environment;

Carrying out unified mark classification on the coded data;

assigning a weight to each abnormal data sample;

；

wherein Y is an abnormal data set;

The invention classifies through the abnormal detection algorithm of the reinforced weighted random forest algorithm, improves the accuracy of a classification model, better processes the unbalanced data problem and improves the stability and the reliability of the overall classification. The method helps to reduce dimensionality, redundancy characteristics and time consumption of model training and prediction, and is suitable for abnormal detection and classification tasks of large data volume and high-frequency data.

initializing parameters with random values;

performing back propagation updating weight according to the training error;

and obtaining a trained convolutional neural network model.

；（3）

Inputting the abnormal data set into a convolutional neural network detection model M1 for training;

the middle layer is 1 layer, and the neuron number of the middle layer is。

identifying the classified abnormal data by using an enhanced flow type abnormal detection algorithm and a trained convolutional neural network model, wherein the calculation process of local abnormal factors in the enhanced flow type abnormal detection algorithm is as follows:

for each outlier data point in the real-time communication dataset：

Calculation ofK nearest neighbor of (2) and obtain the neighborhood set +.>；

；

calculation and calculationNeighborhood set of->Is expressed as> ；

；

；（4）

Calculation ofIs->；（5）

The method and the device identify the classified abnormal data by utilizing the reinforced streaming abnormal detection algorithm and the trained model, monitor and analyze the abnormal data in real time, detect the newly-appearing abnormal data in time, have the capability of efficiently processing the real-time data stream, and timely send out early warning and response to the abnormal data.

；（6）

According to the invention, the abnormal data is detected and early-warned by a self-adaptive method, the parameters and the threshold value of an abnormal detection algorithm are flexibly adjusted according to the characteristics and the change modes of the abnormal data, the accuracy of abnormal detection is improved, the abnormal data is dynamically adjusted according to the change of the abnormal data, and the abnormal data detection system adapts to abnormal conditions under different time periods and different data distribution, so that the abnormal detection system can respond to a new abnormal mode in real time and send early warning timely.

The present disclosure is a system, and/or computer program product. The computer program product includes a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.

Finally, it should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that: modifications and equivalents may be made to the specific embodiments of the invention without departing from the spirit and scope of the invention, which is intended to be covered by the claims.

Claims

1. The abnormal data monitoring and analyzing method based on the data mining and the neural network is characterized by comprising the following steps:

acquiring a high-dimensional real-time communication data set of a user side, and performing classification marking and preprocessing;

extracting features by using a bi-directional cyclic neural network BiRNN model, and reducing the dimension by adopting a principal component analysis method;

encoding by a discrete encoding method to form encoded data;

the self-adaptive method tracks the change of data in real time, dynamically adjusts the model and parameters, can be more suitable for different types and distribution of data, provides more accurate anomaly detection and early warning results, and ensures that the early warning accuracy reaches more than 90 percent;

classification is performed by an enhanced weighted random forest algorithm, comprising the steps of:

assigning a weight to each abnormal data sample;

for unlabeled new samples, the trained model uses the learned parameters and weights thereof to conduct classification inference according to the learned rules, and the new samples are distributed to normal categories or abnormal categories; the parameters and weights of the model are learned from the marked samples through an optimization algorithm in the training process;

The tag classification of data refers to the classification of all data sets into normal data sets and abnormal data sets, the encoded data is classified by an enhanced weighted random forest algorithm, and the weight function of the enhanced weighted random forest algorithm is as follows:

given N balanced sub-training sets, training is carried out on the N balanced sub-training sets to obtain N decision tree classifiers，Is a natural number from 1 to N;

wherein Y is an abnormal data set;

the finally obtained classifierThe testing set is used for testing classification effects;

for each outlier data point in the real-time communication dataset：

Calculation ofK nearest neighbor of (2) and obtain the neighborhood set +.>；

Calculating data pointsTo data point->K reachable distance +.>Which is data point>K adjacent distance>And (4) point->And (3) withEuropean distance between- >Is the maximum value of (2);

according to the distance thresholdDefining the data points with the distance smaller than or equal to the threshold value as the data points in the neighborhood of the target point; target point->Is expressed as:

wherein,representing target point->A set of data points within a neighborhood of (a); d is a real-time communication dataset; />Representing data points +.>Is->A Euclidean distance between them; />Indicating that the distance is less than or equal to the threshold value%>Data point->Belonging to the target point->Is within a neighborhood of (2); different k reachable distances correspond to different distance thresholds, and the threshold is dynamically adjusted according to the k reachable distances>The method comprises the steps of carrying out a first treatment on the surface of the Data point +.>To data point->K reachable distance +.>And threshold->Comparing; if the k reachable distance of a certain outlier data point is smaller than the threshold +.>Marking it as an anomaly of the same type; if successive outliers occur, the threshold value +.>To improve accuracy; if no outlier occurs, the threshold value +.>To increase sensitivity;

calculation and calculationNeighborhood set of->Is expressed as>；

Calculation ofIs->；

Wherein M is the number of data points;

the self-adaptive method is utilized to enable the anomaly detection model to be automatically adjusted according to the change of the data, and adapt to new data distribution and mode change; in the adaptive method The updated formula of (c) is as follows:

the learning rate is adjusted according to the self-adaptive gradient adjustment factor, and the parameters of the model are updated by using the adjusted learning rate; when the adaptive gradient adjustment factor becomes large, the learning rate becomes small; when the self-adaptive gradient adjustment factor becomes smaller, the learning rate becomes larger;

acquiring a high-dimensional real-time communication data set of a user side, and carrying out marking classification on the high-dimensional real-time communication data set;

the high-dimensional dataset includes the following 10 dimensional data: timestamp, sender and receiver identification, communication mode, communication duration, communication quality indicator, bandwidth usage, communication location, data traffic, network topology, and user behavior;

removing users without abnormal communication behaviors, and retaining abnormal data samples;

the abnormal data comprise abnormal signal quality, abnormal call drop rate, abnormal call completing rate, abnormal data transmission rate, base station fault or abnormal state and abnormal flow.

2. The method for monitoring and analyzing abnormal data based on data mining and neural network according to claim 1, wherein the step of encoding the dimension-reduction feature using a discrete encoding method to form encoded data specifically comprises:

3. The method for monitoring and analyzing abnormal data based on data mining and neural network according to claim 1, wherein the training process of the convolutional neural network detection model comprises:

initializing parameters with random values;

performing back propagation updating weight according to the training error;

And obtaining a trained convolutional neural network model.

4. The abnormal data monitoring and analyzing method based on data mining and neural network according to claim 3, wherein the number of neurons in the input layer of the convolutional neural network is set according to the number of input abnormal communication data features and the number of coding bits of each abnormal communication data feature, and the calculating method is as follows:

where M is the number of features of the input abnormal communication data,is the firstCoding bit number of i features, < >>Is the number of neurons of the input layer; the types of the abnormal communication data characteristics comprise frequency abnormality, time delay abnormality, abnormal data packet frequency, abnormal data packet size, signal strength abnormality, abnormal protocol behavior and data integrity abnormality.

5. The method for monitoring and analyzing abnormal data based on data mining and neural network according to claim 4, wherein the encoded data in the abnormal data set is inputted into a convolutional neural network detection model M1 for training,

the middle layer is 1 layer, and the neuron number of the middle layer is。

6. The abnormal data monitoring and analyzing method based on data mining and neural network according to claim 3, wherein the number of neurons of the output layer is equal to the number of demand classifications; the one_hot encoding method is used to encode the demand state and the function of the output neuron is selected as the log function.