CN115438035A - A data anomaly processing method based on KPCA and hybrid similarity - Google Patents

Abstract

The invention discloses a data exception handling method based on KPCA and mixed similarity, which comprises the following steps: s1: the terminal generates a task and uploads the task to the edge terminal; s2: the edge terminal receives the task and divides the data related to the task into high-dimensional data and low-dimensional data; s3: processing the high-dimensional data and the low-dimensional data; s4: and the edge terminal uploads the processed data to the cloud terminal. Through the mode, the data exception handling method provided by the invention has higher integrity on data feature mining, and the data exception handling method based on KPCA and mixed similarity has higher accuracy, so that the quality management level of a data set is improved, and the safe, stable and high-quality operation of a cloud end and an edge end to a task is promoted.

Description

A data anomaly processing method based on KPCA and hybrid similarity

technical field

The invention relates to the field of big data processing, in particular to a data anomaly processing method based on KPCA and mixed similarity.

Background technique

In recent years, traditional industrial control systems have been gradually connected with the Internet and cloud platforms to form an industrial Internet platform. At the same time, with the vigorous development of the Internet of Things and 5G technology, mobile terminal devices generate massive amounts of data. All data collected by terminal devices will be transmitted to the cloud through the network, where cleaning, mining and other work will be performed. This will not only cause huge pressure on the network bandwidth and long delay, but also waste the computing resources of the cloud computing center. Therefore, it is very important to upload the clean data to the cloud for storage and utilization after reasonable data cleaning processing at the edge. It is necessary. The existing technology often includes the detection and cleaning of industrial data outliers, but rarely includes the deduplication of redundant data. However, since many redundant data can be deduced from other data, the redundant Information will not play a role in data processing, and deduplication of redundant data can effectively reduce data transmission pressure and task load.

Chinese invention patent (Application No.: 201811519395.0, Publication No.: CN 109635958 A) discloses a smart power data anomaly detection method, which reduces the dimensionality of effective offline data samples and calculates time-series sample sequences, including: using PCA principal component analysis The method performs dimensionality reduction processing on effective offline data samples, removes the correlation of each dimension feature above three dimensions, and obtains offline data samples after dimensionality reduction; serializes offline data samples after dimensionality reduction to obtain a sequence of time series samples. The disadvantages of this scheme are: traditional industrial data are mostly high-dimensional data with strong nonlinearity, the PCA algorithm has a general effect on nonlinear data processing, the data information after dimensionality reduction is poorly preserved, and nonlinear features are difficult to obtain, resulting in Data accuracy after anomaly detection is lower.

Chinese invention patent (application number: 201911423436.0, publication number: CN 111275288 A) discloses a multi-dimensional data anomaly detection method and device based on XGBoost, including: data collection and cleaning, standardized processing of cleaned data, and unification of different dimensions of data Between dimensions; feature extraction and dimensionality reduction, building anomaly detection model training, using the XGBoost method to train dimensionality reduction data, and establishing a prediction model for equipment anomalies; online anomaly detection, if it exceeds a given threshold, then it is determined that it has occurred abnormal. The disadvantage of this scheme is that it only considers the Pearson correlation coefficient, which is only good for data sets with strong correlation, but poor for industrial data with strong nonlinear relationship, and the detection accuracy of redundant data is insufficient. Lead to poor deduplication effect.

Contents of the invention

In order to solve the above technical problems or at least partly solve the above technical problems, the present disclosure provides a data anomaly processing method based on KPCA and hybrid similarity, which is characterized in that it includes the following steps:

S1: The terminal generates a task and uploads the task to the edge;

S2: The edge end receives the task, and divides the data involved in the task into high-dimensional data and low-dimensional data according to dimensions;

S3: Processing the high-dimensional data and low-dimensional data;

S4: The edge uploads the processed data to the cloud.

Further, the high-dimensional data is data with dimension >= 3;

The low-dimensional data is data with a dimension <3;

Further, the processing of the high-dimensional data and low-dimensional data includes:

S31. Perform anomaly detection on the high-dimensional data and low-dimensional data, and obtain a detection result;

S32. Cleaning the detection results to obtain a cleaned data set;

S33. Perform redundant data judgment and processing on the cleaned data set.

Further, the anomaly detection is performed on the high-dimensional data and low-dimensional data, and detection results are obtained, including:

S311. Using iForest to perform anomaly detection on the low-dimensional data, and obtain the path length and abnormal score corresponding to each low-dimensional data;

S312. Convert the high-dimensional data into characteristic data using the KPCA algorithm, and then use iForest to perform anomaly detection on the characteristic data, and obtain the path length and abnormal score corresponding to each high-dimensional data;

Further, the high-dimensional data is converted into feature data using the KPCA algorithm, including:

A high-dimensional data mapping database is established, and all original high-dimensional data and corresponding feature data are recorded in the high-dimensional data mapping database.

Further, the cleaning of the detection results includes:

S321. Obtain the path length and abnormal score of the high-dimensional data and the low-dimensional data, and calculate the average path length;

S322. Use the data whose average path length is in the range of 0 to 0.15 and the abnormal score in the range of 0.85 to 1 as an abnormal value, and perform data cleaning.

Further, the cleaning of the detection results, the high-dimensional data and the low-dimensional data are performed separately using the methods involved in the above S31, S32, and S33.

Further, the judging and processing the redundant data of the cleaned data set includes:

，则将

视为冗余数据；其中，所述S331步骤中，低维数据与高维数据均采用上述方法，并分开同步进行；S331. Obtain data with similar average path lengths and abnormal scores, and assume the obtained data as

, then the

It is regarded as redundant data; wherein, in the step S331, both the low-dimensional data and the high-dimensional data adopt the above method, and are performed separately and synchronously;

的数据类型，若

为低维冗余数据，则转S333，若

为高维冗余数据，转S334；S332. Analysis

data type, if

It is low-dimensional redundant data, then turn to S333, if

For high-dimensional redundant data, turn to S334;

；公式如下：S333. Using the Pearson correlation coefficient to obtain the similarity of the low-dimensional redundant data

;The formula is as follows:

对应的原始高维数据

，采用混合相似度算法获取所述高维冗余数据的相似度

；公式如下：S334. Obtain the above from the high-dimensional data mapping database

Corresponding original high-dimensional data

, using a hybrid similarity algorithm to obtain the similarity of the high-dimensional redundant data

;The formula is as follows:

为斯皮尔曼相关系数所占权重，

为

数据的斯皮尔曼相关系数，

为

的互信息值；in

is the weight of the Spearman correlation coefficient,

for

The Spearman correlation coefficient of the data,

for

mutual information value;

或

与预设阈值

比较，若H₁>δ或H₂>δ，则表示

中存在冗余数据，进行数据清除。S335. The said

or

with preset threshold

Comparison, if H ₁ >δ or H ₂ >δ, it means

If there is redundant data in , clear the data.

、预设阈值

的由人工取值，

范围为0~1，优选取值为0.5，

范围不超过计算出的相似度最大值，优选的，

取值设为最大相似度值的90%。Further, the

, preset threshold

The value is taken manually,

The range is 0~1, the preferred value is 0.5,

The range does not exceed the calculated maximum value of the similarity, preferably,

The value is set to 90% of the maximum similarity value.

中随机选择一个数据进行删除。Further, the data cleaning includes: in the

Randomly select a piece of data to delete.

Compared with the prior art, the technical solution provided by the invention has the following advantages:

The data anomaly processing method based on KPCA and hybrid similarity provided by the present invention can analyze the tasks generated by the terminal and uploaded to the edge end, and divide the data involved in the tasks into high-dimensional data and low-dimensional data. The above high-dimensional data and low-dimensional data are processed, and the edge end uploads the processed data to the cloud. At the same time, in view of the characteristics of large changes in the dimensions of industrial data, the present invention divides the data types into high-dimensional data and low-dimensional data, uses the KPCA algorithm for data processing on the high-dimensional data, and reduces the dimension of the data set through feature extraction , to realize the abnormal detection of high-dimensional data and low-dimensional data; for the characteristics that the nonlinear characteristics of industrial data are difficult to mine, the present invention uses the Pearson correlation coefficient combined with the mixed similarity algorithm to realize the detection of redundant data, wherein, for the high-dimensional data There is a certain dependence between the nonlinear features and the similarity between high-dimensional data, and the Spearman correlation coefficient combined with the mutual information value method is used to calculate the similarity of high-dimensional data. In this way, the data anomaly processing method provided by the present invention has high integrity for data feature mining, and the data anomaly detection and deduplication scheme provided has high accuracy, thereby improving the quality management level of data sets and promoting cloud computing. And the edge end is safe, stable and high-quality for tasks.

Description of drawings

FIG. 1 is a flow chart of a data anomaly processing method based on KPCA and hybrid similarity provided by the present invention.

FIG. 2 is a flowchart of a high-dimensional data and low-dimensional data processing method based on a data anomaly processing method based on KPCA and mixed similarity provided by the present invention.

Fig. 3 is a flow chart of cleaning abnormal data of a data abnormal processing method based on KPCA and hybrid similarity provided by the present invention.

detailed description

The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, so as to define the protection scope of the present invention more clearly.

In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure can also be implemented in other ways than described here; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, and Not all examples.

Fig. 1 is a flow chart of a data anomaly processing method based on KPCA and hybrid similarity provided by the present invention, the method comprising:

S1: The terminal generates a task and uploads the task to the edge;

S2: The edge end receives the task, and divides the data involved in the task into high-dimensional data and low-dimensional data according to dimensions;

S3: Processing the high-dimensional data and low-dimensional data;

S4: The edge uploads the processed data to the cloud.

Further, the high-dimensional data is data with dimension >= 3;

The low-dimensional data is data with a dimension <3;

Further, referring to FIG. 2, the processing of the high-dimensional data and low-dimensional data includes:

S31. Perform anomaly detection on the high-dimensional data and low-dimensional data, and obtain a detection result;

S32. Cleaning the detection results to obtain a cleaned data set;

S33. Perform redundant data judgment and processing on the cleaned data set.

Further, the anomaly detection is performed on the high-dimensional data and low-dimensional data, and detection results are obtained, including:

S311. Using iForest to perform anomaly detection on the low-dimensional data, and obtain the path length and abnormal score corresponding to each low-dimensional data;

S312. Convert the high-dimensional data into characteristic data using the KPCA algorithm, and then use iForest to perform anomaly detection on the characteristic data, and obtain the path length and abnormal score corresponding to each high-dimensional data;

Further, the calculation formula of the path length is:

为路径长度，

为样本数，

为欧拉常数；Among them, the

is the path length,

is the number of samples,

is Euler's constant;

The formula for calculating the abnormal score is:

表示异常分数，

表示路径长度期望，所述

为调和函数，

。Among them, the

Indicates the outlier score,

represents the path length desired, the

is the harmonic function,

.

为数据在所有iTree上的路径长度期望，经过iForest算法输出结果为0~1的值。said

It is the expected path length of data on all iTrees, and the output result of the iForest algorithm is a value of 0~1.

Further, the high-dimensional data is converted into feature data using the KPCA algorithm, including:

Establishing a high-dimensional data mapping database, recording all original high-dimensional data and corresponding feature data in the high-dimensional data mapping database;

It can be understood that the feature data is obtained by reducing the dimensionality of the original high-dimensional data. In the anomaly detection of the high-dimensional data and low-dimensional data, the high-dimensional data has nonlinear characteristics, so the effective The KPCA algorithm obtains the feature data of high-dimensional data, and processes the feature data; while judging and processing the redundant data of the cleaned data set, in order to ensure the integrity of high-dimensional data information, it is selected The original high-dimensional data is processed; the purpose of constructing the high-dimensional data mapping database is to ensure the preservation of the original high-dimensional data and feature data, so that the scheme has higher flexibility and reliability.

Further, the cleaning of the detection results includes:

S321. Obtain the path length and abnormal score of the high-dimensional data and the low-dimensional data, and calculate the average path length;

S322. Using the average path length in the range of 0 to 0.15 and the data with an abnormal score in the range of 0.85 to 1 as abnormal values, perform data cleaning;

Specifically, those skilled in the art can determine the range according to data characteristics and actual needs, and the values provided here can be used as a reference and not as a limitation.

Further, in the cleaning of the detection results, the high-dimensional data and low-dimensional data adopt the methods involved in the above-mentioned S31, S32, and S33 respectively, and are carried out separately and synchronously, wherein the high-dimensional data each select the same dimension For example, if the dimension of high-dimensional data is N _i , ( _i =0, 1, .

Further, referring to FIG. 3 , the redundant data judgment and processing of the cleaned data set includes:

，则将

视为冗余数据；其中，所述S331步骤中，低维数据与高维数据均采用上述方法，并分开同步进行；S331. Acquire data with similar average path lengths and abnormal scores, and assume the acquired data as

, then the

It is regarded as redundant data; wherein, in the step S331, both the low-dimensional data and the high-dimensional data adopt the above method, and are performed separately and synchronously;

的数据类型，若

为低维冗余数据，则转S333，若

为高维冗余数据，转S334；S332. Analysis

data type, if

It is low-dimensional redundant data, then turn to S333, if

For high-dimensional redundant data, turn to S334;

；公式如下：S333. Using the Pearson correlation coefficient to obtain the similarity of the low-dimensional redundant data

;The formula is as follows:

对应的原始高维数据

，采用混合相似度算法获取所述高维冗余数据的相似度

；公式如下：S334. Obtain the above from the high-dimensional data mapping database

Corresponding original high-dimensional data

, using a hybrid similarity algorithm to obtain the similarity of the high-dimensional redundant data

;The formula is as follows:

为斯皮尔曼相关系数所占权重，

为

数据的斯皮尔曼相关系数，

为

的互信息值,其中：in

is the weight of the Spearman correlation coefficient,

for

The Spearman correlation coefficient of the data,

for

The mutual information value of , where:

，

表示数据

的联合概率，

表示

、

出现的概率，log的底数通常取为e。

,

represent data

the joint probability of

express

,

The probability of occurrence, the base of log is usually taken as e.

=[0,0,1] ,

=[1,1,0]，可得

，

，E.g:

=[0,0,1] ,

=[1,1,0], available

,

,

,

，而本例的

=

+

=0.6365。

,

, and in this case the

=

+

=0.6365.

In this scheme, mutual information is a measure of the degree of interdependence between two data, and the greater the value of mutual information, the greater the degree of dependence between the two data;

或

与预设阈值

比较，若H₁>δ或H₂>δ，则表示

中存在冗余数据，进行数据清除。S335. The said

or

with preset threshold

Comparison, if H ₁ >δ or H ₂ >δ, it means

If there is redundant data in , clear the data.

、预设阈值

的取值可视情形而定，

优选为0.5。Further, the

, preset threshold

The value of can depend on the situation,

Preferably it is 0.5.

的确定本领域技术人员可根据数据特征与实际需求设置，优选的，为人为设定的固定阈值为当前相似度上限值的90％，此处提供的值可作参考，并不作为限定。specific,

Those skilled in the art can set it according to data characteristics and actual needs. Preferably, the artificially set fixed threshold is 90% of the current upper limit of similarity. The values provided here can be used as a reference and not as a limitation.

中随机选择一个数据进行删除。Further, the data cleaning includes: in the

Randomly select a piece of data to delete.

The flow charts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations that may be implemented by methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

It should be noted that, in this article, relational terms are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations . Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Claims (9)

1. A data anomaly processing method based on KPCA and hybrid similarity, is characterized in that, comprises the following steps: S1: The terminal generates a task and uploads the task to the edge; S2: The edge end receives the task, and divides the data involved in the task into high-dimensional data and low-dimensional data; S3: Processing the high-dimensional data and low-dimensional data; S4: The edge uploads the processed data to the cloud. 2. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 1, is characterized in that, The high-dimensional data is data with dimensions >= 3; The low-dimensional data is data with dimension <3. 3. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 1, is characterized in that, The processing of the high-dimensional data and low-dimensional data includes: S31. Perform anomaly detection on the high-dimensional data and low-dimensional data, and obtain a detection result; S32. Cleaning the detection results to obtain a cleaned data set; S33. Perform redundant data judgment and processing on the cleaned data set. 4. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 3, is characterized in that, The anomaly detection is performed on the high-dimensional data and low-dimensional data to obtain detection results, including: S311. Using iForest to perform anomaly detection on the low-dimensional data, and obtain the path length and abnormal score corresponding to each low-dimensional data; S312. Convert the high-dimensional data into feature data using the KPCA algorithm, and then use iForest to perform anomaly detection on the feature data, and obtain path lengths and abnormal scores corresponding to each high-dimensional data. 5. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 4, is characterized in that, The described high-dimensional data adopts KPCA algorithm to be converted into feature data, including: A high-dimensional data mapping database is established, and all original high-dimensional data and corresponding feature data are recorded in the high-dimensional data mapping database. 6. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 5, is characterized in that, The cleaning of the detection results includes: S321. Obtain the path length and abnormal score of the high-dimensional data and the low-dimensional data, and calculate the average path length; S322. Use the data whose average path length is in the range of 0 to 0.15 and whose abnormal score is in the range of 0.85 to 1 as an abnormal value, and perform data cleaning. 7. A kind of data anomaly processing method based on KPCA and hybrid similarity as described in any one of claim 4-6, it is characterized in that, In the cleaning of the detection results, the high-dimensional data and the low-dimensional data are respectively carried out using the methods involved in the above S31, S32, and S33, and the high-dimensional data are each selected from data with the same dimension for processing. . 8. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 6, is characterized in that, The redundant data judgment and processing of the cleaned data set includes: S331. Acquire data with similar average path lengths and abnormal scores, and assume the acquired data as

, then the

It is regarded as redundant data; wherein, in the step S331, both the low-dimensional data and the high-dimensional data adopt the above method, and are performed separately and synchronously; S332. Analysis

data type, if

It is low-dimensional redundant data, then turn to S333, if

For high-dimensional redundant data, turn to S334; S333. Using the Pearson correlation coefficient to obtain the similarity H ₁ of the low-dimensional redundant data; the formula is as follows: H ₁ =corr

S334. Obtain the above from the high-dimensional data mapping database

Corresponding original high-dimensional data

, using a hybrid similarity algorithm to obtain the similarity H ₂ of the high-dimensional redundant data; the formula is as follows:

Where μ is the weight of the Spearman correlation coefficient,

for

The Spearman correlation coefficient of the data,

for

mutual information value; S335. Comparing the H ₁ or H ₂ with the preset threshold δ, if H ₁ >δ or H ₂ >δ, it means

If there is redundant data in , clear the data. 9. a kind of data exception processing method based on KPCA and hybrid similarity as claimed in claim 8, is characterized in that, The μ and the preset threshold δ are manually selected, the range of μ is 0-1, and the range of δ does not exceed the calculated maximum value of the similarity.

Images