CN110727669A - A power system sensor data cleaning device and cleaning method - Google Patents

Abstract

The invention belongs to the technical field of wireless sensor networks and neural networks, and in particular relates to a power system sensor data cleaning device and a cleaning method, which are mainly used to solve how to distinguish abnormal data caused by events from abnormal data caused by other factors such as the environment. The invention includes: an abnormal data detection module, an abnormal data classification module and a noise data restoration module. The invention constructs a sensor data cleaning model for cleaning sensor data, which classifies the sensor data into normal data, noise data and fault data, completes the restoration of the noise data, removes the interference of the noise data, and uses Its fault diagnosis classification can greatly enhance the accuracy. The method is real-time and can quickly clean up the data transmitted in real-time; it is universal and can be applied to most industrial sensor networks.

Description

A power system sensor data cleaning device and cleaning method

technical field

The invention belongs to the technical field of wireless sensor networks and neural networks, and in particular relates to a power system sensor data cleaning device and a cleaning method, which are mainly used to solve how to distinguish abnormal data caused by events from abnormal data caused by other factors such as the environment.

Background technique

The real-time condition monitoring of various facilities in the power system is very important to the safety production management. The monitoring data of the equipment can be used to determine whether the operating equipment is functioning properly or what kind of failure has occurred. The rapid development of the Internet has made many devices automatically monitored by various types of sensors. Wireless sensor network technology has also been successfully applied in major factories. Sensor data can be uniformly transmitted through the network, which greatly saves labor costs and improves. real time monitoring. However, various factors, such as environmental factors, sensor hardware performance, wireless transmission interference, etc., cause monitoring data to be prone to anomalies. Secondly, due to changes in the status of the monitoring equipment, such as failures, abnormal sensor data can also be caused. Since these two kinds of anomalies are somewhat similar, the accuracy of state analysis and fault detection will be greatly reduced during data cleaning, and these fault data and noise data are abnormal data. Obviously, these data are very important to judge whether the equipment is faulty or not.

Since noisy and faulty data tend to be quite similar, outliers should not be smoothed or discarded directly during data cleaning.

SUMMARY OF THE INVENTION

In order to solve the problem that the accuracy of state analysis and fault detection using sensor data is relatively low due to the very similar noise data and fault data in abnormal sensor data, the present invention proposes a power system sensor data cleaning device and method, which can Classify fault data from noisy data, and repair noisy data to improve the accuracy of state analysis and fault detection.

In order to realize the above-mentioned purpose of the invention, the present invention adopts the following technical solutions:

A power system sensor abnormal data cleaning device, comprising: an abnormal data detection module, an abnormal data classification module and a noise data repair module;

Abnormal data detection module detects abnormal data of power system sensors;

The abnormal data classification module is used to divide the abnormal data of the sensor into noise data and fault data;

The noise data repair module fits the noise data to normal data, and then replaces the noise data with the corresponding SDAE-fitted data to complete the repair.

The abnormal data detection module uses the normal data of the sensor to train SDAE, learns the characteristics of the normal data of the sensor, fits the normal data of the sensor, and uses the maximum value of the difference between the SDAE fitting data and the original data as the normal value of the data. Threshold; when new data is input, the new data is input into SDAE for fitting, and the difference between the fitted data and the original data is obtained, and the data with the difference greater than the threshold is judged as abnormal data.

The abnormal data classification module is used to divide the sensor data set X into m windows X={L ₁ , . . . , L _m } according to time, and calculate the correlation between the sensor data in each window. , find out all abnormal data at the same time and record the correlation between the sensor where these abnormal data are located and sensor i within the window; there are at least w-1 abnormal data at time t, and the time series of the sensor where these abnormal data are located and the time of the sensor When the correlations of the sequences are all greater than the minimum correlation threshold, the data is identified as fault data; if it does not meet this condition, it is noise data; a data category matrix including normal data, fault data and noise data is established.

The calculation formula of the correlation degree RT _ij between the sensor i and the sensor j is:

Among them, x _ij represents the data of the ith sensor at time j, _{X_it} is the t-th data of the sensor in the window, _{X_jt} is the t-th data of the sensor j in the window, and the length of each window is s.

数据矩阵Y为修复完成的数据集。The noise data repair module is used to establish a data matrix Y={y _ij }, let Y=X; find out the data x _ij of k _ij =1 in the data category matrix, let

The data matrix Y is the repaired dataset.

The data cleaning method of a power system sensor abnormal data cleaning device includes:

Input the new data to be cleaned into the trained SDAE, and output the fitted data; make the difference between the fitted data and the original data, and the data whose difference exceeds the specified threshold is abnormal data;

For each abnormal data, find out whether there is abnormal data in other sensors at the same time, and record other sensors with abnormality at the same time; calculate the correlation between the sensor and other recorded sensors, if the correlation is greater than the specified threshold, then determine There is a strong correlation between them; if the number of sensors with strong correlation with the sensor is greater than the specified number threshold, the abnormal data of the sensor at this moment is fault data; otherwise, it is noise data;

For each noise data, find the corresponding SDAE fitting data; replace the noise data with the fitting data to complete the restoration.

The specific steps of the abnormal data detection include:

S1: Collect the normal data of the sensor, learn the characteristics of the normal data of the sensor, and fit the normal data of the sensor;

S2: Calculate the difference between the SDAE fitting data and the original data when fitting normal data, make a histogram of the difference sequence, draw its normal distribution curve, take a confidence interval of 0.99, and obtain the upper threshold Th _L and the lower threshold Th _U ;

其中

表示第i个传感器的j时刻的拟合数据，将拟合数据与原数据做差，求得差值

其中d_ij表示第i个传感器的j时刻的拟合数据与原数据的差值；S3: Suppose the new data is X={x _ij }, where x _ij represents the data of the ith sensor at time j, input the data into SDAE, fit the data, and obtain the fitted data

in

Indicates the fitted data at time j of the i-th sensor, and the fitted data is compared with the original data to obtain the difference

where d _ij represents the difference between the fitted data at time j of the ith sensor and the original data;

S4: Compare the difference with the threshold, if Th _L <d _ij <Th _U , then x _ij is normal data, and k _ij =0; otherwise, x _ij is abnormal data, and k _ij =1.

The specific steps of the abnormal data classification include:

S1: Divide the sensor data set X into m windows X={L ₁ ,...,L _m } according to time, where each window

L _j ={X _1j ,...,X _nj } ^T ,X _ij ={xi,(j-1)×s+1,...,xi,j×s},

The length of each window is s;

S2: Calculate the correlation between sensor data in each window;

S3: For the abnormal data x _it with k _it = 1, find out all abnormal data at time t and record the correlation between the sensor where these abnormal data are located and sensor i within the window; at least w-1 abnormal data exists at time t , and the correlation between the time series of the sensor where these abnormal data are located and the time series of the sensor are all greater than the minimum correlation threshold, that is, Num((RT _ij >RT _min )&(k _jt =1))>w-1, it is determined that this If the data is fault data, let k _it =2; if this condition is not met, it is noise data, and k _it =1 remains unchanged;

S4: Establish a data category matrix K={k _ij }, k _ij ∈ {0,1,2}, k _ij =0 indicates that the data x _ij is normal data, k _ij =1 indicates that the data x _ij is noise data, k _ij =2 indicates that the data x _ij is fault data.

The calculation formula of the correlation degree RT _ij between the sensor i and the sensor j is:

Among them, x _ij represents the data of the ith sensor at time j, _{X_it} is the t-th data of the sensor in the window, _{X_jt} is the t-th data of the sensor j in the window, and the length of each window is s.

The specific steps of the noise data restoration are as follows:

S1: establish a data matrix Y={y _ij }, let Y=X;

S2: Find the data x _ij with k _ij =1 in the data category matrix, let

S3: Use the data matrix Y as the inpainted dataset.

The present invention has the following advantages and beneficial effects:

The invention constructs a sensor data cleaning model for cleaning sensor data, which classifies the sensor data into normal data, noise data and fault data, completes the restoration of the noise data, removes the interference of the noise data, and uses Its fault diagnosis classification can greatly enhance the accuracy. The method is real-time and can quickly clean up the data transmitted in real time. It is universal and can be applied to most industrial sensor networks.

Description of drawings

In order to facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

1 is a flow chart of a method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of dividing five sensor time series data into several windows according to the present invention.

Detailed ways

The present invention is a power system sensor data cleaning device and cleaning method. As shown in FIG. 1 , FIG. 1 is a flowchart of a method according to an embodiment of the present invention.

The power system sensor data cleaning device of the present invention is divided into three stages: an abnormal data detection module, an abnormal data classification module and a noise data restoration module. These three stages run sequentially from front to back to complete the task of data cleaning.

The abnormal data detection module adopts three-layer superimposed denoising self-encoder SDAE to detect abnormal data of power system sensors;

The abnormal data classification module is used to divide the abnormal data of the sensor into noise data and fault data, and use the correlation between the sensor data to classify the abnormal data. If multiple sensor data at the same time are judged as abnormal data at the same time, it will be regarded as a fault. data, otherwise noise data;

The noise data repair module uses SDAE to fit the noise data to normal data, and then replaces the noise data with the corresponding SDAE-fitted data to complete the repair.

The abnormal data detection stage is responsible for detecting abnormal data in the sensor data; by applying the normal data of the sensor to train the three-layer superimposed denoising auto-encoder SDAE, the three-layer superimposed denoising auto-encoder SDAE can learn the sensor data. features and can fit the sensor data. The maximum value of the difference between the three-layer superposition denoising auto-encoder SDAE fitting normal data is recorded as the threshold for judging normal and abnormal data. When new data arrives, the data is input into SDAE for fitting, and the difference between the fitted data and the original data is obtained, and the data with the difference greater than the threshold value is judged as abnormal data.

The abnormal data classification stage is responsible for classifying abnormal data into noise data and fault data. By using the correlation between sensor data, the abnormal data detected before is classified, and multiple relevant sensor data at the same time are simultaneously determined as Abnormal data is regarded as fault data, otherwise it is noise data.

The noise data repairing stage is responsible for repairing the noise data discriminated in the previous stage into normal data, so that it does not affect subsequent data analysis and processing work. By using the three-layer superposition denoising auto-encoder SDAE, the noise can be fitted to the characteristics of normal data, and the noise data can be replaced with the data fitted by the corresponding three-layer superposition denoising auto-encoder SDAE to complete the restoration.

The present invention is a data cleaning method of a power system sensor abnormal data cleaning device. The data is classified, and then the noise data is repaired, while the normal data and the fault data are retained to analyze the equipment failure. Specifically include:

For abnormal data detection, use normal sensor data to train three-layer superimposed denoising auto-encoder SDAE; input the new data to be cleaned into the trained SDAE, and output the fitted data; make the difference between the fitted data and the original data, the difference is Data whose value exceeds the specified threshold is abnormal data;

Abnormal data classification: for each abnormal data, find out whether there is abnormal data in other sensors at the same time, and record other sensors with abnormality at the same time; calculate the correlation between the sensor and other recorded sensors, if the correlation is greater than the specified If the number of sensors with strong correlation with the sensor is greater than the specified number threshold, the abnormal data of the sensor at this moment is fault data; otherwise, it is noise data;

Abnormal data repair: For each noise data, find the corresponding SDAE fitting data; replace the noise data with the fitted data to complete the repair.

The specific steps of the abnormal data detection module include:

The normal sensor data is collected in advance for training the three-layer denoising auto-encoder SDAE, which can make the three-layer denoising auto-encoder SDAE learn the characteristics of the sensor's normal data, and can fit the sensor's normal data.

The maximum value of the difference between the three-layer superposition denoising auto-encoder SDAE fitting normal data is recorded as the threshold for judging normal and abnormal data. Due to the existence of errors, we cannot directly take the maximum value. First, calculate the difference between the SDAE fitting data of the three-layer denoising auto-encoder and the original data when fitting normal data, make a histogram of the difference sequence, and draw its The normal distribution curve, taking the confidence interval of 0.99, obtains the upper threshold Th _L and the lower threshold Th _U .

其中表示第i个传感器的j时刻的拟合数据。将拟合数据与原数据做差，求得差值

其中d_ij表示第i个传感器的j时刻的拟合数据与原数据的差值。Assuming that the new data is X={x _ij }, where x _ij represents the data of the ith sensor at time j, input the data into the three-layer denoising auto-encoder SDAE, and the three-layer denoising auto-encoder SDAE can be based on The previously learned features are fitted to the data to obtain the fitted data

in Represents the fitted data at time j of the ith sensor. Calculate the difference between the fitted data and the original data

where d _ij represents the difference between the fitted data of the ith sensor at time j and the original data.

Compare the difference with the threshold, if Th _L <d _ij <Th _U , then x _ij is normal data, and k _ij =0; otherwise, x _ij is abnormal data, and k _ij =1.

The specific steps of the abnormal data classification module include:

Divide the sensor dataset X into m windows X={L ₁ ,...,L _m } according to time, where each window L _j ={X _1j ,...,X _nj } ^T ,X _ij ={ xi,(j-1)×s+1,...,xi,j×s}, the length of each window is s, Figure 2 shows the window division for 5 sensor data, the window length in the figure is 200. It can be seen that the correlation between the sensors in the L3 window is obviously different from that _of other windows. Therefore, it is necessary to calculate the correlation after dividing the window.

To calculate the correlation between sensor data in each window, the calculation formula of the correlation RT _ij between sensor i and sensor j is:

For the abnormal data x _it with k _it =1, find out all the abnormal data at time t and record the correlation between the sensor where these abnormal data are located and the sensor i within the window. Only at time t, there are at least w-1 abnormal data, and the correlation between the time series of the sensor where these abnormal data are located and the time series of the sensor are all greater than the minimum correlation threshold, that is

Num((RT _ij >RT _min )&(k _jt =1))>w-1, consider the data as fault data, let k _it =2; if this condition is not met, it is noise data, k _it =1 constant.

Since then, the data category matrix K={k _ij }, k _ij ∈{0,1,2} has been established, and the classification of abnormal data has been completed: k _ij =0 indicates that the data x _ij is normal data, and k _ij =1 indicates that the data x _ij is noise data, and k _ij =2 indicates that the data x _ij is noise data.

The specific steps of the noise data repair module include:

矩阵Y便为修复完成的数据集。Create a data matrix Y={y _ij }, let Y=X. Find the data x _ij with k _ij = 1 in the data category matrix, let

The matrix Y is the inpainted dataset.

It should be understood by those skilled in the art that the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Modifications or equivalent replacements are made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. The utility model provides an abnormal data cleaning device of electric power system sensor which characterized in that: the method comprises the following steps: the abnormal data detection module, the abnormal data classification module and the noise data restoration module;

the abnormal data detection module detects abnormal data of a sensor of the power system;

the abnormal data classification module is used for dividing the sensor abnormal data into noise data and fault data;

and the noise data restoration module fits the noise data into normal data, and then replaces the noise data with the data fitted by the corresponding SDAE to finish restoration.

2. The abnormal data cleaning device for the sensor of the power system according to claim 1, wherein: the abnormal data detection module trains the SDAE by using the normal data of the sensor, learns the characteristics of the normal data of the sensor, fits the normal data of the sensor, and takes the maximum value of the difference value between the SDAE fitted data and the original data as a threshold value for judging whether the data is normal or not; and when new data is input, inputting the new data into SDAE for fitting, solving the difference value between the fitting data and the original data, and judging the data with the difference value larger than the threshold value as abnormal data.

3. The abnormal data cleaning device for the sensor of the power system according to claim 1, wherein: the abnormal data classification module is used for dividing the sensor data set X into m windows X ═ L according to time₁,...,L_mCalculating the correlation degree between the sensor data in each window, finding out all abnormal data at the same time for the abnormal data, and recording the correlation degree between the sensor where the abnormal data is located and the sensor i in the window; when at least w-1 abnormal data exist at the time t and the correlation degree of the sensor time sequence where the abnormal data exist and the sensor time sequence is larger than the minimum correlation degree threshold value, the data are determined to be fault data; if the condition is not met, the data is noise data; a data category matrix is established that includes normal data, fault data, and noise data.

4. The device for cleaning abnormal data of sensor in power system as claimed in claim 3, wherein correlation RT between sensor i and sensor j is_ijThe calculation formula of (2) is as follows:

wherein x is_ijData representing the i-th sensor at time j, X \ u_itIs the t-th data, X _, of the sensor within the window_jtIs the t-th data of sensor j in a window, each window being s in length.

5. The abnormal data cleaning device for the sensor of the power system as claimed in claim 3, wherein the noise data repairing module is used for establishing a data matrix Y ═ { Y ═ Y }_ijLet Y be X; finding k in data category matrix_ijData x of 1_ijLet us order

The data matrix Y is a repaired data set.

6. A data cleaning method using the abnormal data cleaning apparatus for a power system sensor according to any one of claims 1 to 5, comprising:

inputting new data to be cleaned into the trained SDAE, and outputting the fitted data; making a difference between the fitted data and the original data, wherein the data with the difference value exceeding a specified threshold value is abnormal data;

for each abnormal data, searching whether other sensors have abnormal data at the same moment, and recording other sensors with abnormal data at the same moment; calculating the correlation degree between the sensor and other recorded sensors, and if the correlation degree is greater than a specified threshold value, judging that the sensor and other recorded sensors have strong correlation; if the number of the sensors with strong correlation with the sensors is larger than a specified number threshold, the abnormal data of the sensors at the moment is fault data; otherwise, it is noise data;

for each noise data, finding out the corresponding SDAE fitting data; and replacing the noise data with the fitting data to complete the restoration.

7. The data cleansing method according to claim 6, characterized in that: the specific steps of abnormal data detection comprise:

s1, collecting normal data of the sensor, learning the characteristics of the normal data of the sensor, and fitting the normal data of the sensor;

s2, calculating the difference between the SDAE fitting data and the original data when fitting the normal data, making a histogram of the difference sequence, drawing a normal distribution curve, taking a confidence interval of 0.99, and obtaining an upper threshold Th_LAnd a lower threshold Th_U；

S3: suppose the new data is X ═ X_ijIn which x_ijData indicating the j time of the i-th sensor is input to SDAE, andfitting the data to obtain fitting data

Wherein

Fitting data representing the j moment of the ith sensor is obtained, the fitting data is subtracted from the original data, and a difference value is obtained

Wherein d is_ijRepresenting the difference between the fitted data of the ith sensor at the time j and the original data;

s4: comparing the difference with a threshold value, if Th_L＜d_ij＜Th_UThen x_ijIs normal data, k_ij0, otherwise, x_ijAs abnormal data, k_ij＝1。

8. The data cleansing method according to claim 7, characterized in that:

the specific steps of the abnormal data classification include:

s1: time-wise partitioning of a sensor data set X into m windows X ═ L₁,...,L_mIn which each window L_j＝{X_1j,...,X_nj}^T,X_ij＝{x_i,(j-1)×s+1,...,x_i,j×sH, the length of each window is s;

s2, calculating the correlation degree between the sensor data in each window;

s3 for k_itAbnormal data x of 1_itFinding out all abnormal data at the time t and recording the correlation degree of the sensor where the abnormal data are located and the sensor i in the window; when at least w-1 abnormal data exist at the time t and the correlation degree of the sensor time sequence where the abnormal data exist and the sensor time sequence is larger than the minimum correlation degree threshold value, namely Num ((RT)_ij＞RT_min)&(k_jt1) is greater than w-1, the data is considered as fault data, and k is made_it2; if this condition is not satisfied, the data is noise data, k_it1 is unchanged;

s4: establishing a data category matrix K ═ K_ij},k_ij∈{0,1,2}，k_ij0 indicates data x_ijIs normal data, k_ij1 denotes the data x_ijFor noisy data, k_ij2 denotes the data x_ijIs failure data.

9. The data cleansing method according to claim 8, characterized in that:

correlation RT between sensor i and sensor j_ijThe calculation formula of (2) is as follows:

wherein x is_ijData representing the i-th sensor at time j, X \ u_itIs the t-th data, X _, of the sensor within the window_jtIs the t-th data of sensor j in a window, each window being s in length.

10. The data cleansing method according to claim 7, characterized in that:

the specific steps of the noise data restoration are as follows:

s1: establishing a data matrix Y ═ Y_ijLet Y be X;

s2: finding k in data category matrix_ijData x of 1_ijLet us order

S3: the data matrix Y is used as the repair-completed data set.

Images