CN112418324B - Cross-modal data fusion method for electrical equipment state perception - Google Patents

Abstract

The invention discloses a cross-modal data fusion method for electrical equipment state perception. Based on two types of cross-modal data, multi-sensor data and image data, the electrical equipment state is fused and perceived. The invention first converts multi-sensor time series data into recursive graphs; then uses different convolutional neural networks to extract features from recursive graphs and electrical equipment image data; The fused features undergo further feature extraction and state-level perception. The invention makes full use of two types of cross-modal data, such as multi-sensors and images, in the monitoring data of electrical equipment, and solves the problems of low accuracy and poor fault tolerance in perception based on single-modal data to a certain extent.

Description

A Cross-modal Data Fusion Method for State Awareness of Electrical Equipment

technical field

The invention belongs to the technical field of electrical equipment status, and in particular relates to a cross-modal data fusion method for electrical equipment status perception.

Background technique

Effective state awareness of electrical equipment, timely detection of hidden dangers and corresponding measures are the keys to the safe and stable operation of the power grid.

The state of electrical equipment is the result of the joint action of a variety of parameters. The effective integration of various influencing factors makes them complement and enhance each other, so that the state perception can be carried out from multiple angles and all directions, which is beneficial to improve the accuracy of perception. At the same time, when the data quality of a certain type of parameter is degraded due to communication and measurement errors, there are still other types of data to supplement, which can effectively improve the fault tolerance of perception.

With the gradual development of digital infrastructure and power Internet of Things, the number and types of power sensing terminals are increasing, providing an effective way to obtain data for all-round, multi-angle state sensing of electrical equipment. The multi-modal parameters generated by the multi-source sensing terminal under the power Internet of Things include structured parameters such as time series represented by electrical measurements, as well as unstructured parameters such as images and maintenance reports. There is a big difference, namely multimodal parameters. For the data of the same mode, due to the same data form, it can be regarded as a parameter under the same coordinate system, and the difficulty of fusion is relatively small; for the data of cross-modality, due to its different structural forms and physical meanings, it is difficult to unify Description, integration is difficult. It can be seen that the rapid development of the power grid provides sufficient data sources for electrical equipment state perception based on cross-modal data fusion, but the cross-type and multi-dimensional data analysis technology is weak, and the correlation analysis and mining capabilities between state quantities are insufficient. lack of state awareness.

At present, image parameters and various structural parameters are widely used in the state assessment of electrical equipment, such as using infrared image data to monitor the temperature of the equipment, and using various sensors to monitor the amount of gas in the operating environment of the equipment. However, the existing methods are still mainly based on state evaluation based on unimodal data, which has a single description angle, low accuracy and poor fault tolerance.

SUMMARY OF THE INVENTION

The technical problem solved by the present invention is to provide a cross-modal data fusion method for electrical equipment state perception, which solves the problems that the existing electrical equipment state perception method only relies on single-modal data, and its perception accuracy is low and fault tolerance is poor. .

The present invention provides a cross-modal data fusion method for electrical equipment state perception, comprising the following steps:

S1, normalize the multi-sensor time series data, and generate a multi-parameter recursive graph;

S2, using the convolutional neural network to perform feature extraction on the multi-parameter recursive graph to obtain multi-sensor parameter features;

S3. Use the Faster R-CNN network to perform feature extraction on the image data to obtain image parameter features;

S4. Build a cross-modal data fusion model, which fuses multi-sensor parameter features and image parameter features according to weights, extracts features from the fused features, and finally classifies electrical equipment status through a multi-category neural network;

S5, train the cross-modal data fusion model with the corresponding multi-sensor time series data, image data and electrical equipment state;

S6. Use the trained cross-modal data fusion model to perceive the state of the electrical equipment.

Further, in step S1, the specific method for normalizing the multi-sensor time series data is as follows:

表示传感器某一参量i在t₁到t_n时刻的采样值，n表示时间序列的长度。In the formula,

It represents the sampling value of a certain parameter i of the sensor from time t ₁ to t _n , and n represents the length of the time series.

Further, in step S1, generating a multi-parameter recursive graph specifically includes:

S11. Set the length of the sensor time series as n, then generate a recursive matrix R _n ,

表示第t_i时刻j类传感器的采样值，x_j(t_i)为第j类传感器在第t_i时刻的采样值。In the formula, R _ij represents the elements in the recursive matrix R _n ,

represents the sampling value of the j-th sensor at time t _i , and x _j (t _i ) is the sampling value of the j-th sensor at time t _i .

S12. Multiply all elements in the recursive matrix by a preset matching coefficient, so that the recursive matrix matches the element values of the image, and draws a recursive graph with the matched element values as pixel values.

Further, the convolutional neural network structure in step S2 is specifically:

The first layer: input layer, the input size is n×n multi-sensor recursive graph;

The second layer: convolutional layer, 32 convolutional layers of size 5×5, stride s=1, padding=2, and non-linear transformation with relu function;

The third layer: pooling layer, the maximum pooling function of size 2×2, step size s=2, padding=0;

The fourth layer: convolutional layer, 32 convolutional layers of size 5×5, stride s=1, padding=2, and use the relu function for nonlinear transformation;

The fifth layer: pooling layer, the maximum pooling function of size 2×2, step size s=2, padding=0;

The sixth layer: a fully connected layer of size 1×125.

Further, in step S3, using the Faster R-CNN network to perform feature extraction on the image data is specifically: using the first fully connected layer after ROI Pooling in the Faster R-CNN and its previous network as the feature extraction network of the image data. to extract features from the image data.

Further, in step S4, the weighting factors of the multi-sensor parameter features and the image parameter features are obtained through training.

Further, in step S4, the multi-sensor parameter features and the image parameter features are fused according to weights, specifically:

The weight of the multi-sensor parameter feature is set as W ₁ , and the weight of the image data feature as W ₂ , and the two are multiplied by the corresponding weights respectively and then spliced.

Further, in step S4, feature extraction is performed on the fused features, and finally the electrical equipment state classification is performed through a multi-category neural network. Specifically, the fused features are respectively passed through a 4096 fully connected layer and an m fully connected layer. , and finally connected to a 1×m multi-category neural network; where m is the number of layers of the multi-category neural network, that is, the number of categories of electrical equipment states.

Further, in step S5, the cross-modal data fusion model is trained with the corresponding multi-sensor time series data, image data and electrical equipment state, specifically: using the corresponding multi-sensor time series data and image data pairs as the model input, to The electrical equipment state category is the output, and the model is trained.

Further, the objective function of model training is L _cls :

表示第i个样本的真实分类结果。In the formula, m represents the number of training samples, pi is the predicted classification result of the _ith sample, which is calculated by the multi-class neural network classification formula,

represents the true classification result of the ith sample.

The beneficial effects of the present invention are as follows: the cross-modal data fusion method for electrical equipment state perception of the present invention is based on two types of cross-modal data, multi-sensor time series data and image data, to fuse and perceive the electrical equipment state, making full use of the The multi-sensor and image cross-modal data in electrical equipment monitoring data solves the problems of low accuracy and poor fault tolerance in perception based on single-modal data.

Description of drawings

FIG. 1 is a flow chart of the cross-modal data fusion method for electrical equipment state perception according to the present invention.

FIG. 2 is a network diagram of multi-sensor parameter feature extraction of electrical equipment based on recursive graph of the present invention.

FIG. 3 is a network diagram of image feature extraction of electrical equipment according to the present invention.

FIG. 4 is a feature fusion network diagram based on weighting factors of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings:

The cross-modal data fusion method for electrical equipment state perception of the present invention is based on two types of cross-modal data, multi-sensor data and image data, to perform fusion perception on the electrical equipment state. The invention first converts multi-sensor time series data into recursive graphs; then uses different convolutional neural networks to extract features from recursive graphs and electrical equipment image data; The fused features undergo further feature extraction and state-level perception. The present invention makes full use of two types of cross-modal data, such as multi-sensors and images, in the monitoring data of electrical equipment, and solves the problems of low accuracy and poor fault tolerance in perception based on single-modal data to a certain extent.

The cross-modal data fusion method for electrical equipment state perception according to the embodiment of the present invention, as shown in FIG. 1 , includes the following steps:

S1, normalize the multi-sensor time series data, and generate a multi-parameter recursive graph;

S2, using the convolutional neural network to perform feature extraction on the multi-parameter recursive graph to obtain multi-sensor parameter features;

S3. Use the Faster R-CNN network to perform feature extraction on the image data to obtain image parameter features;

S4. Build a cross-modal data fusion model, which fuses multi-sensor parameter features and image parameter features according to weights, extracts features from the fused features, and finally classifies electrical equipment status through a multi-category neural network;

S5, train the cross-modal data fusion model with the corresponding multi-sensor time series data, image data and electrical equipment state;

S6. Use the trained cross-modal data fusion model to perceive the state of the electrical equipment.

Further, in step S1, the specific method for normalizing the multi-sensor time series data is as follows:

表示传感器某一参量i在t₁到t_n时刻的采样值，n表示时间序列的长度，由图像数据和传感器数据的采样频率决定。为了使时序数据与图像数据相对应，即在时间上相匹配，多传感器时序数据中最后一个数据的采样时刻与图像采样时刻相同或相近。In the formula,

It represents the sampling value of a certain parameter i of the sensor from time t ₁ to t _n , and n represents the length of the time series, which is determined by the sampling frequency of image data and sensor data. In order to make the time series data correspond to the image data, that is, match in time, the sampling time of the last data in the multi-sensor time series data is the same as or close to the image sampling time.

Further, in step S1, generating a multi-parameter recursive graph specifically includes:

S11. Set the length of the sensor time series as n, then generate a recursive matrix R _n ,

表示第t_i时刻j类传感器的采样值，x_j(t_i)为第j类传感器在第t_i时刻的采样值。In the formula, R _ij represents the elements in the recursive matrix R _n ,

represents the sampling value of the j-th sensor at time t _i , and x _j (t _i ) is the sampling value of the j-th sensor at time t _i .

S12. In order to match the element values in the recursive graph with the element values in the image data, multiply all the elements in the recursive matrix by a preset matching coefficient, which is 100 in this embodiment, so that the recursive matrix matches the element values of the image Match, draw the recursive graph with the matched element value as the pixel value.

Further, the convolutional neural network structure in step S2 is shown in Fig. 2, specifically:

The first layer: input layer, the input size is n×n multi-sensor recursive graph;

The second layer: convolutional layer, 32 convolutional layers of size 5×5, stride s=1, padding=2, and non-linear transformation with relu function;

The third layer: pooling layer, the maximum pooling function of size 2×2, step size s=2, padding=0;

The fourth layer: convolutional layer, 32 convolutional layers of size 5×5, stride s=1, padding=2, and use the relu function for nonlinear transformation;

The fifth layer: pooling layer, the maximum pooling function of size 2×2, step size s=2, padding=0;

The sixth layer: a fully connected layer of size 1×125.

Among them, k is the number of sensor categories, and n is the length of the time series. conv(32*5*5) represents 32 convolution kernels of size 5*5; maxpooling1(2*2) represents the maximum pooling of size 2*2; s is the step in each dimension of the image during convolution long; p is padding, that is, the part filled around the image, if p=2, it means that two circles of 0 are filled around the image.

Further, in step S3, using the Faster R-CNN network to perform feature extraction on the image data is specifically: using the first fully connected layer after ROI Pooling (region of interest pooling) in Faster R-CNN and the network before it. As a feature extraction network for image data, as shown in Figure 3, feature extraction is performed on image data.

Further, in step S4, the multi-sensor parameter feature and the image parameter feature are fused according to the weight, specifically: set the weight of the multi-sensor parameter feature to be W ₁ and the weight of the image data feature to be W ₂ , and combine the two with the corresponding The weights are multiplied and stitched together. In order to avoid the influence of human factors, the weighting factors of the multi-sensor parameter features and the image parameter features in step S4 can be obtained through training.

Further, in step S4, feature extraction is performed on the fused features, and finally the electrical equipment state classification is performed through a multi-category neural network. Specifically, the fused features are respectively passed through a 4096 fully connected layer and an m fully connected layer. , and finally connected with a 1×m multi-category neural network; where m is the number of layers of the multi-category neural network, that is, the number of categories of electrical equipment states. In this embodiment, the feature extraction network of the multi-parameter sensor parameters outputs a feature matrix with a size of 1×125, and the feature extraction network of the image outputs a feature matrix with a size of 1×4096. Let the weight corresponding to the multi-sensor parameter feature be W ₁ , the weight corresponding to the image data feature is W ₂ , as shown in Figure 4, the two are spliced according to the weight to form a feature with a size of 1×4221, and then the feature extraction is performed on the fused feature: this feature is used as input , to perform feature extraction and parameter training with the goal of electrical equipment state perception, that is, through a fully connected layer of 4096 and m respectively, and finally connected to a 1×m softmax (multi-category neural network) for electrical equipment state classification. The states of electrical devices are divided into m classes, so softmax outputs a 1×m matrix, where the ith element represents the probability that the input is the ith class.

For softmax, let its input be x, and the corresponding probability of the j-th target is p(y=j|x),

where W is the model parameter from x to y.

Further, in step S5, the cross-modal data fusion model is trained with the corresponding multi-sensor time series data, image data and electrical equipment state, specifically: using the corresponding multi-sensor time series data and image data pairs as the model input, to The electrical equipment state category is the output, and the model is trained. Generate a joint label for multi-sensor data and image data, that is, take the multi-sensor data and image data at the corresponding time as the data pair, generate a common label, and then use it as the model input with the multi-sensor number and the image data pair as the electrical data. The device state category is the output, and the model is trained.

Further, the objective function of model training is L _cls :

表示第i个样本的真实分类结果。In the formula, m represents the number of training samples, pi is the predicted classification result of the _ith sample, which is calculated by the multi-class neural network classification formula,

represents the true classification result of the ith sample.

Until L _cls ≤σ is satisfied, the training is stopped, and σ is the error threshold, which can be determined according to the actual situation.

After the training is completed, the trained cross-modal data fusion model can be used to perceive the state of electrical equipment with the corresponding multi-sensor data and image data pairs as input.

The present invention also provides another embodiment, which uses multi-sensor data and infrared data to sense the state of the transformer. Among them, the sensor data is structured data, the data volume is small, so the transmission cost is small, and it is collected every 10 minutes. Table 1 shows the sensor data categories; the camera collects unstructured data, the data volume is large, and the transmission cost is high, usually every 2 collected every hour.

Table 1 Multi-sensor parameter type and sampling frequency table

The data set used in this embodiment contains 6432 infrared image data. Referring to the multi-parameter recursive graph conversion method in step 1, one infrared image corresponds to a 10×20 sensor parameter matrix, so the total sensor parameters are 10×20×6432. As shown in Table 2, 5432 data pairs are used as training set and 1000 data pairs are used as test set. In order to facilitate the verification of the method proposed by the present invention, the state of the transformer is divided into two categories in this embodiment, namely, abnormal and non-abnormal.

Table 2 Data distribution table

The specific steps are as follows:

Step 1, select the length n=20 of the multi-sensor time series, normalize ten types of parameters including winding temperature, full current, capacitance value, etc., and generate a multi-parameter recursion graph with a size of 20×20.

In step 2, feature extraction is performed on the generated multi-parameter recursion graph, and the output is a feature vector with a size of 1×125.

Step 3: Use Faster R-CNN to extract features from the infrared image data corresponding to the multi-sensor parameters, and output a feature vector with a size of 1×4096.

Step 4, splicing the two types of features according to the weights to generate a fusion feature vector with a size of 1×4221.

Step 5, with the transformer state level as the target, the fused features are trained through a fully connected layer of size 1×4096 and 1×2 respectively.

Step 6, repeat steps 1-5 to train the model.

Step 7, use the trained model for testing.

In this implementation, precision (P), recall (Recall, R), average precision (AveragePrecision, AR) and average recall (Average Recall, AR) are used to evaluate the model testing metrics. The test results are shown in Table 3. It can be seen that the cross-modal data fusion method for electrical equipment state perception proposed in the present invention can more accurately perceive the electrical equipment state, and the average precision rate and average recall rate of perception are 76.3% and 72.9%, respectively.

Table 3 Comparison of the results of each model

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the described specific embodiments or substitute in similar manners, but will not deviate from the spirit of the present invention or go beyond the definitions of the appended claims range.

Claims (5)

1. A cross-mode data fusion method for sensing the state of electrical equipment is characterized by comprising the following steps:

s1, carrying out normalization processing on the multi-sensor time sequence data and generating a multi-parameter recursion graph;

wherein: the specific method for carrying out normalization processing on the multi-sensor time sequence data comprises the following steps:

in the formula (I), the compound is shown in the specification,

indicating that a certain parameter i of the sensor is at t₁To t_nSampling values at time, n representing the length of the time series;

the generating of the multi-parameter recursive graph specifically comprises:

s11, if the length of the sensor time sequence is n, generating a recursive matrix R_n：

In the formula, R_ijRepresenting a recursive matrix R_nThe elements (A) and (B) in (B),

denotes the t-th_iSampling value, x, of a sensor of type j at a time instant_j(t_i) At t for class j sensors_iSampling values of the moments;

s12, multiplying all elements in the recursion matrix by a preset matching coefficient, matching the recursion matrix with the element values of the image, and drawing a recursion graph by taking the matched element values as pixel values;

s2, performing feature extraction on the multi-parameter recursive graph by using a convolutional neural network to obtain multi-sensor parameter features;

s3, extracting the characteristics of the image data by using a Faster R-CNN network to obtain image parameter characteristics;

s4, constructing a cross-modal data fusion model, fusing the multi-sensor parameter characteristics and the image parameter characteristics according to weights, extracting the characteristics of the fused characteristics, and finally classifying the states of the electrical equipment through a multi-class neural network;

wherein, the weighting factors of the multi-sensor parameter characteristic and the image parameter characteristic are obtained through training;

fusing the multi-sensor parameter characteristics and the image parameter characteristics according to weights specifically comprises the following steps: let the weight of the multi-sensor parameter feature be W₁The weight of the image data feature is W₂Respectively multiplying the two weights by corresponding weights and splicing;

s5, training a cross-mode data fusion model according to the corresponding multi-sensor time sequence data, image data and electric equipment state; the method comprises the following specific steps: taking corresponding multi-sensor time sequence data and image data pairs as model input, and taking the state type of the electrical equipment as output, and performing model training;

and S6, sensing the state of the electrical equipment by using the trained cross-modal data fusion model.

2. The cross-modal data fusion method for electrical device state awareness according to claim 1, wherein the convolutional neural network structure in step S2 is specifically:

a first layer: an input layer that inputs a multi-sensor recurrence map of size n × n;

a second layer: convolutional layers, 32 convolutional layers of size 5 × 5, with step s equal to 1, padding equal to 2, and non-linearly transformed with relu function;

and a third layer: the pooling layer is a maximal pooling function with the size of 2 multiplied by 2, the step length s is 2, and the padding is 0;

a fourth layer: convolutional layers, 32 convolutional layers of size 5 × 5, with step s equal to 1, padding equal to 2, and non-linearly transformed with relu function;

and a fifth layer: pooling layers with a maximum pooling function of 2 × 2, step length s is 2, padding is 0;

a sixth layer: fully connected layers of size 1 x 125.

3. The cross-modal data fusion method for sensing the state of the electrical device according to claim 1, wherein in step S3, the performing the feature extraction on the image data by using the Faster R-CNN network specifically comprises: and (3) utilizing the first full-connection layer after ROI Pooling in the Faster R-CNN and the network before the first full-connection layer as a feature extraction network of the image data to extract the features of the image data.

4. The method for fusing cross-modal data for sensing the state of the electrical device according to claim 1, wherein in step S4, the feature extraction is performed on the fused features, and finally the classification of the state of the electrical device through the multi-class neural network is specifically as follows: the fused features respectively pass through a 4096 full-link layer and a m full-link layer, and are finally connected with a 1 x m multi-class neural network; wherein m is the number of layers of the multi-class neural network, namely the class number of the state of the electrical equipment.

5. The method of claim 1, wherein an objective function of model training is L_cls：

Wherein m represents the number of samples for training, p_iIs the prediction classification result of the ith sample and is obtained by calculation of a multi-class neural network classification formula,

representing the true classification result of the ith sample.

Images