CN116610935A - A Mechanical Fault Detection Method Based on Multimodal Analysis of Engine Vibration Signals - Google Patents

Abstract

The invention relates to a mechanical fault detection method based on engine vibration signal multi-mode analysis, firstly adopting a multi-mode feature extraction network to extract image features related to abnormal signals from one-dimensional amplitude data and two-dimensional image data for the extracted diesel engine vibration signal, then adopting a mixed channel feature fusion detection network to split a feature image into two groups, respectively adopting a spatial attention mechanism and a channel attention mechanism to carry out weighting calculation on the feature image in a spatial domain and a channel domain, grouping the calculated feature image again, obtaining a multi-dimensional weighting feature image through merging operation, and finally adopting a multi-scale detector to detect the three feature images simultaneously to judge whether the signal in the period has an abnormal state. Compared with the prior art, the invention has the advantages of high accuracy, good noise resistance and the like.

Description

A mechanical fault detection method based on multimodal analysis of engine vibration signals

Technical Field

The present invention relates to the technical field of machine learning, and in particular to a mechanical fault detection method based on multimodal analysis of engine vibration signals.

Background Art

Fault detection of diesel engines can extend their service life and enhance their safety, which has important economic value and social benefits.

However, existing fault detection methods cannot cope with strong noise environments in practical applications and are not universal under multiple working conditions. At the same time, existing fault diagnosis models use a single data detection scheme, such as using only one-dimensional data as input for abnormal analysis and detection. Such detection schemes often lack consideration of the inherent correlation and distribution gap of the same data source in different forms, and have limitations in the exploration of multi-source data.

Summary of the invention

The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a mechanical fault detection method based on multimodal analysis of engine vibration signals.

The purpose of the present invention can be achieved by the following technical solutions:

A mechanical fault detection method based on multimodal analysis of engine vibration signals comprises the following steps:

Collect vibration signals of generators;

Inputting the vibration signal into a multimodal feature extraction network to obtain multiple feature information;

Select p feature information and input it into the mixed channel feature fusion detection network, perform secondary feature processing and detection, and output the detection results

The multimodal feature extraction network includes a shaping network, a convolution module, a fully connected module and a multimodal Transformer module;

The shaping network is used to convert the input engine vibration signal into a two-dimensional image, and the convolution module performs feature extraction based on the two-dimensional image to obtain a two-dimensional feature image;

The fully connected module is used to extract a one-dimensional feature vector of the input engine vibration signal; the multimodal Transformer module is used to integrate the one-dimensional feature vector and the two-dimensional feature image;

The mixed-channel feature fusion detection network includes a feature aggregation module, a feature mixing module and a multi-scale detection module;

The feature aggregation module is used to aggregate the p feature information to obtain an aggregated feature map FM;

The feature mixing module is used to perform feature mixing on the aggregated feature map FM to obtain a plurality of feature maps of different sizes;

The multi-scale detection module is used to detect the mixed feature map, and to determine whether there is an abnormal vibration signal in the diesel engine cylinder during the current period by detecting whether there is an abnormal or irregular texture area in the feature map.

Furthermore, the multimodal feature extraction network includes a q-layer structure, and each layer structure includes a convolution module, a fully connected module and a multimodal Transformer module.

Furthermore, the multimodal feature extraction network extracts features from the vibration signal to obtain a plurality of feature information, including the following steps:

S1, the vibration signal is input into the shaping network and the fully connected module respectively, the shaping network outputs a two-dimensional image, and the two-dimensional image is input into the convolution module for feature extraction;

S2, the fully connected module outputs a one-dimensional feature vector;

S3, the convolution module outputs a two-dimensional feature image;

S4, inputting the one-dimensional feature vector and the two-dimensional feature image into a multimodal Transformer module to obtain integrated feature information;

S5, inputting the integrated feature information and the two-dimensional feature image into the convolution module of the next layer, and inputting the one-dimensional feature vector into the fully connected module of the next layer;

S6. Repeat steps S2-S5 until the qth layer structure of the multimodal feature extraction network is reached, and perform feature extraction layer by layer to obtain multiple feature information.

Furthermore, the multimodal Transformer module includes two multi-head attention networks, which respectively correspond to the long-distance relationship interaction between the one-dimensional feature vector and the two-dimensional feature image.

Furthermore, the multimodal Transformer module integrates the one-dimensional feature vector and the two-dimensional feature image, including the following steps:

Divide the one-dimensional feature vector into n tokens sub-vectors;

Divide the two-dimensional feature image into n feature blocks, and extend the feature blocks into n tokens sub-vectors;

In the first multi-head attention network, the tokens corresponding to the two-dimensional feature image are input into the matrix In the example, the tokens corresponding to the one-dimensional feature vector are input into the matrix and In the process, the query matrix Q with the image feature information is calculated and the key matrix K with the one-dimensional amplitude feature n information is matched, and the obtained matching degree is assigned to the corresponding eigenvalue matrix V to complete the operation of mapping the image feature to the amplitude feature;

In the second multi-head attention network, the tokens corresponding to the two-dimensional feature image are input into the matrix In the example, the tokens corresponding to the one-dimensional feature vector are input into the matrix and In the process, the query matrix Q with the image feature information is calculated and the key matrix K with the one-dimensional amplitude feature n information is matched, and the obtained matching degree is assigned to the corresponding eigenvalue matrix V to complete the operation of mapping the image feature to the amplitude feature;

Among them, in the two multi-head attention networks, the corresponding two sets of Q, K, and V vectors are respectively and The matrix is calculated;

The output one-dimensional feature vectors of the two multi-head attention networks are merged, and the fully connected layer and ReLU activation function are used to activate the merged features. Finally, the obtained 1×1×C vector is integer-calculated to obtain a feature map of H×W×C dimensions.

Furthermore, the feature aggregation module aggregates the feature maps FM ₁ , FM ₂ and FM ₃ output by the three convolution modules at the bottom layer of the multimodal feature extraction network, including the following steps:

Improve the feature information of each feature map;

FM ₃ doubles the size of the feature map by deconvolution and compresses the number of channels to half of the original. The features of FM ₃ are assigned to FM ₂ using the add fusion method to obtain FM′ ₂ ;

Upsample and channel compress the FM ₂ feature map, and fuse it with the features of the FM ₁ feature map to obtain FM′ ₁ ;

The concat layer is used to merge the three feature maps FM′ ₁ , FM′ ₂ and FM _3. During the merging operation, FM′ ₁ is downsampled and FM ₃ is upsampled to obtain the feature map FM.

Furthermore, the feature mixing module performs feature mixing on the aggregated feature map FM, including the following steps:

A 1x1 convolutional layer is used to compress and merge the feature channels of FM to the same number as FM′ ₂ , and a grouped convolution is used to divide FM into two feature maps, FM_1 and FM_2, which have the same size as FM and half the number of channels as FM.

Perform feature extraction operations on the grouped feature maps FM_1 and FM_2 to obtain FM_1′ and FM_2′;

The same parallel convolution module is used to extract feature maps of different sizes and number of channels from FM_1′ and FM_2′ to form two paired groups;

A fusion module composed of a concat layer and a 1x1 convolution layer is used to merge and mix the feature maps in the pairing groups to obtain three feature maps of different sizes, namely FM1, FM2 and FM3.

Furthermore, spatial attention calculation is used on the feature map FM_1 to enhance the morphological feature information FM_1′ corresponding to the abnormal signal; channel attention calculation is used on FM_2 to enhance the semantic feature weight FM_2′ of the abnormal signal;

The pairing group extracted from FM_1′ contains the spatial attention weighted feature value, and the pairing group extracted from FM_2′ contains the channel attention weighted feature value.

Furthermore, the multi-scale detection module is used to detect the feature map output by the feature mixing module, and the cross entropy classification loss is used to adjust the classification module of the detector, and the position loss is calculated based on the CIoU position evaluation relationship.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention takes into account the multi-modality of the generator vibration signal, extracts the one-dimensional amplitude vector and the two-dimensional feature vector in the engine vibration signal through the multi-modal feature extraction network, and performs secondary feature processing and detection on the extracted features through the mixed channel feature fusion detection network. The mixed channel feature fusion module includes a space and channel dual-dimensional attention calculation mechanism, and performs weighted calculations in the space domain and the channel domain on the feature map to achieve the suppression of vibration noise, so that the network can resist the influence of environmental noise and changes in operating conditions on the final detection results, and is suitable for strong noise environments in practical applications, and has universality under multiple working conditions;

The accuracy and anti-noise performance of the present invention are significantly better than those of the prior art. On four data sets constructed under different working conditions, even when the signal-to-noise ratio is -4dB, the accuracy of the present invention reaches at least 99.008%, which is much higher than other methods.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the structure of a multimodal feature extraction network of the present invention.

FIG. 2 is a two-dimensional diagram of waveform signals of a cylinder under normal and abnormal working conditions in an embodiment of the present invention, wherein (2a) represents a normal working condition and (2b) represents an abnormal working condition.

FIG3 is a schematic diagram of the structure of a multimodal Transformer module in an embodiment of the present invention.

FIG4 is a schematic diagram of the structure of a mixed channel feature fusion detection network in an embodiment of the present invention.

FIG5 is a schematic diagram of the MITDCNN network structure in an embodiment of the present invention.

FIG6 is a typical time domain signal of a single cylinder misfire of a diesel engine at an operating speed of 1800 rpm in an embodiment of the present invention.

FIG7 is a schematic diagram of a time domain signal of a single cylinder misfire in a diesel engine at an operating speed of 1800 rpm converted into a two-dimensional image in an embodiment of the present invention.

FIG8 is a loss iteration curve of network 3 (MITDCNN) in an embodiment of the present invention.

FIG9 is an AP iteration curve of network 3 (MITDCNN) in an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

In order to solve the problems existing in the prior art, the present invention proposes a convolutional neural network (MITDCNN) based on multimodal Transformer feature extraction for diesel engine misfire diagnosis under strong environmental noise and different working conditions.

This embodiment collects vibration signals of the engine cylinder head at different speeds through experiments, and extracts one-dimensional amplitude vector features and two-dimensional image features from them, and inputs them into the multimodal feature extraction network, and the extracted features are subjected to secondary feature processing and detection by the mixed channel feature fusion detection network. The mixed channel feature fusion module contains a spatial and channel dual-dimensional attention calculation mechanism to achieve the suppression of vibration noise, so that the network can resist the influence of environmental noise and changes in operating conditions on the final detection results. This embodiment verifies the effectiveness of the proposed method through the data set collected by the experiment and the comparison with the existing representative algorithms. The results show that the accuracy and anti-noise performance of the proposed MITDCNN are significantly better than the existing algorithms. On the four data sets constructed under different working conditions, even if the signal-to-noise ratio is -4dB, the accuracy of the proposed method is at least 99.008%, which is much higher than other methods.

In order to enhance the sensitivity of the network to abnormal signals, the present invention designs a multimodal feature extraction network for detecting abnormal signals of diesel engines. Compared with a single-modal feature extraction network, the network designed by the present invention simultaneously extracts features from one-dimensional and two-dimensional data, thereby improving the richness of the extracted feature information. The overall network structure is shown in FIG1 .

As shown in Figure 1, the data information extracted from the diesel generator cylinder is a waveform signal. First, the waveform signal is converted from one-dimensional data to a two-dimensional image according to time and amplitude through a reshaping network (ReshapeNet). Secondly, the waveform signal is extracted according to the time unit. The corresponding amplitude forms a one-dimensional vector, thereby obtaining two different types of data in the same time period.

The waveform signal output under normal working condition of the cylinder has certain regularity, and the converted two-dimensional image is an image with certain texture regularity. When the working state of the cylinder is abnormal, irregular areas appear in the converted image. The two-dimensional images converted under normal working condition and abnormal working condition are shown in Figure 2.

Similarly, for one-dimensional vector data, when the cylinder is abnormal, its value will change irregularly. As shown in Figure 1, the network is divided into two directions, the x-axis and the y-axis, to extract and aggregate features. In the x-axis, the waveform signal is shaped to obtain a two-dimensional image, and then the convolution module (ConvNet) extracts features to obtain a two-dimensional feature image, and the extracted two-dimensional feature image is input into the multi-modal Transformer module (Multi-Modal Transformer); the output features of the multi-modal Transformer module are aggregated with the output of the convolution module, and the multi-modal features are mapped into the feature map. In the backbone network, the multi-modal feature extraction network is composed of the superposition of the six above structures. On the other hand, in the y-axis, the extracted one-dimensional vector is vectorized and then extracted by the fully connected module (FCNet) to obtain a one-dimensional feature vector to obtain the characteristic relationship between the timing and the amplitude. The obtained feature vector is input into the multi-modal Transformer module to complement the feature information of the two-dimensional image and learn the characteristic pattern of the abnormal waveform.

In summary, the multimodal feature extraction network mainly extracts the feature information of two-dimensional images, while adding the extraction and fusion of one-dimensional temporal features as a supplement to expand the richness of network features. The core unit of the network is the multimodal Transformer module, which is responsible for building the association between one-dimensional and two-dimensional input features.

For one-dimensional time series features, the multimodal Transformer module proposed in the present invention has better long-dependency solving ability than recurrent neural networks, and can obtain feature information in longer time series. And the multimodal Transformer module is used to extract image features, so the correlation between features of each region of the image, that is, the global features, can be extracted. In the present invention, in order to fuse one-dimensional and two-dimensional input information, a multimodal Transformer module is designed, and multi-head attention calculations are performed on the two types of inputs at the same time, and the extracted feature information is fused. The structure of the multimodal Transformer module is shown in Figure 3.

As shown in Figure 3, the multimodal Transformer module structure contains two multi-head attention networks (Multi Head Attention), which correspond to the long-distance relationship interaction between the one-dimensional feature vector and the two-dimensional feature image.

The two types of data input into the multimodal Transformer module are preprocessed as follows: the one-dimensional amplitude vector is extracted into a one-dimensional feature vector through the fully connected layer, and then the one-dimensional feature vector is divided into n tokens sub-vectors; the two-dimensional feature image is extracted into a feature map by the convolution module, and then the feature map is equally divided into n feature blocks, and then the feature blocks are extended into n one-dimensional feature vector tokens.

In the two Transformer multi-head attention networks, the corresponding two sets of Q, K, and V vectors are respectively and Six matrices are calculated.

First, in the first Transformer module (Transformer-1Multi Head Attention), the tokens corresponding to the two-dimensional feature image of the transformed image are input into In the matrix, the tokens corresponding to the one-dimensional magnitude vector are input into and In the matrix, the query matrix Q with the image feature information is calculated and the key matrix K with the one-dimensional amplitude feature n information is matched, and the obtained matching degree is assigned to the corresponding eigenvalue matrix V, so as to complete the operation of mapping the image feature to the amplitude feature;

Similarly, the tokens corresponding to the one-dimensional amplitude features are input into the first Transformer module (Transformer-2Multi Head Attention) In the example, the tokens corresponding to the image features are input into and The matrix is used to map the amplitude features into the feature map.

Then, the one-dimensional feature vectors output by the multi-head attention network are merged, and the fully connected layer (FCLayer) and ReLU activation function are used to activate the merged features to enhance their nonlinearity. Finally, the obtained 1×1×C vector is integer-calculated to obtain a feature map of H×W×C dimensions.

After completing the extraction of multimodal features, the present invention designs a mixed-channel feature fusion network to merge network features at different levels to improve the comprehensiveness of the features. Since the appearance duration of abnormal signals is different, the abnormal picture blocks displayed in the converted image are also different. Therefore, for the detection of abnormal signal images, the present invention adopts a multi-scale detection scheme to detect feature maps of different sizes. The network structure designed in summary is shown in Figure 4.

The mixed-channel feature fusion network can be divided into three parts: feature aggregation module, feature mixing module and multi-scale detection module. The workflow and function of each part are as follows:

First, in the feature aggregation module, the feature maps output by the three convolution modules at the bottom of the multimodal feature extraction network are aggregated. As shown in Figure 4, three feature maps of different sizes are input to the feature aggregation module, named FM ₁ , FM ₂ and FM ₃ (Feature Map, FM). From the perspective of feature dimension, FM ₁ to FM ₃ show a state of step-by-step enhancement of semantic features. Therefore, during the aggregation operation, the feature information of each feature map is first improved (UpSample). FM ₃ doubles the size of the feature map by deconvolution and compresses the number of channels to half of the original. The add fusion method is used to assign its features to FM ₂ to obtain FM′ _2. Similarly, the fused FM ₂ feature map is upsampled and the channel is compressed, and fused with the features of the FM ₁ feature map to obtain FM′ ₁ . Then, the concat layer is used to merge the three feature maps FM′ ₁ , FM′ ₂ and FM _3. During the merging operation, FM′ ₁ is downsampled (DownSample) and FM ₃ is upsampled. The aggregated feature map FM has the same size as FM′ _2. This feature map contains three levels of feature information, so it is superior to FM ₁ and other three feature maps in terms of feature information richness.

Secondly, the feature mixing module is used to mix the features of the aggregated feature map FM. A 1x1 convolution layer is used to compress and merge the number of feature channels of FM to the same as FM′ ₂ , and then a grouped convolution is used to divide FM into two feature maps, FM_1 and FM_2, which have the same size as FM and half the number of channels as FM. Feature extraction operations are performed on the grouped feature maps to enhance the weights of the effective feature features in the feature maps. In the weighting operation, spatial attention calculation (Spatial Attention model) is used on the feature map FM_1 to enhance the morphological feature information FM_1′ corresponding to the abnormal signal; channel attention calculation (Channel Attention model) is used for FM_2 to enhance the semantic feature weight FM_2′ of the abnormal signal. Then, the same parallel convolution module is used to extract feature maps of different sizes and numbers of channels from FM_1′ and FM_2′. Taking FM_1′ as an example, a 3x3 convolution with a sliding step size of 2 is used to obtain a feature map FM_1′ ₁ with the same size as FM ₁ and half the number of channels; a 3x3 convolution with a step size of 1 is used to obtain FM_1′ ₂ with the same size as FM 2 and half the number of channels _; and finally, a 3x3 deconvolution is used to obtain FM_1′ ₃ with the same size as FM ₃ and half the number of channels. Similarly, the FM_2′ feature map is calculated using a parallel convolution module to obtain FM_2′ ₁ , FM_2′ ₂ , and FM_2′ ₃ . In terms of feature map parameters, FM_1′ ₁ is the same as FM_2′ ₁ , FM_1′ ₂ is the same as FM_2′ ₂ , and FM_1′ ₃ is the same as FM_2′ _3. The above three groups of feature maps are called paired groups. Each paired group contains spatial attention weighted feature values and channel attention weighted feature values. The two feature maps in the paired group are merged and mixed using a fusion module composed of a concat layer and a 1x1 convolutional layer to obtain three feature maps of different sizes, namely FM1, FM2 and FM3.

Finally, the three mixed feature maps are detected using a multi-scale detection module. By detecting whether there are abnormal and irregular texture areas in the feature maps, it is determined whether there are abnormal vibration signals in the diesel engine cylinder during the current period. For the detector, the classification loss and regression loss used are as follows: In the detection task of the present invention, only the transformed graphic area is judged whether there are abnormal signals, so it is a two-classification problem. The cross entropy classification loss is used to adjust the classification module of the detector, and its loss function formula is:

Loss＝-y*log(p)-(1-y)*log(1-p)

In the calculation of position loss, the image block area generated by the abnormal information number is a regular rectangle without complex boundary information. Therefore, the position loss is based on the CIoU position evaluation relationship, and its loss formula can be expressed as:

Loss = 1-CIoU

In summary, in order to more accurately detect the misfire signal of the diesel engine, the present invention constructs a MITDCNN network based on a multimodal feature extraction network and a mixed channel feature fusion detection network, and the overall network structure is shown in Figure 5. The overall working principle of the network is: for the extracted diesel engine vibration signal, a multimodal feature extraction network is first used to extract image features about abnormal signals from one-dimensional amplitude data and two-dimensional image data, and then a mixed channel feature fusion detection network is used to split the feature map into two groups, and the spatial attention mechanism and the channel attention mechanism are used to perform weighted calculations on the feature map in the spatial domain and the channel domain respectively, and the calculated feature maps are grouped again, and a multi-dimensional weighted feature map is obtained through a merging operation, and finally a multi-scale detector is used to simultaneously detect the three feature maps to determine whether the signal in this period is in an abnormal state. The details of the constructed network are shown in Table 1 below:

Table 1. MITDCNN network layer structure composition table

Based on the above, this embodiment studies the misfire of a diesel engine at three different operating speeds. The three operating speeds are 1300rpm, 1800rpm, and 2200rpm, corresponding to the simulated low-speed, medium-speed, and high-speed operating conditions, respectively. Since the misfire of three or more cylinders will cause severe vibration of the engine, it can be observed by the operator with the naked eye without fault detection, so this embodiment focuses on single-cylinder misfire and double-cylinder (mixed cylinder) misfire. As shown in Table 4, single-cylinder misfire detection is performed at three speeds of 1300rpm, 1800rpm, and 2200rpm, and double-cylinder misfire detection is performed at 1800rpm. In addition to the misfire fault, each group uses the normal operation situation as a comparative reference. Common engine misfire faults are basically shown in the misfire types listed in Table 2. Since the relevant data involved in the above faults have been collected more comprehensively during the experiment, the data set used in this embodiment is universal.

Table 2 Diesel engine failure at different operating speeds

In this embodiment, the vibration signal is collected at a sampling frequency of 25.6kHz, the sampling time for each misfire type is 41s, the sampling time includes at least 900 working cycles, and a total of 1,049,600 vibration sequence points are obtained. The typical time domain signal of a single cylinder misfire at different operating speeds is shown in Figure 6. Since the difference between the time domain signals of different misfire types under the same operating conditions is extremely small, it is difficult to directly diagnose the specific misfire type based on the time domain signal. Therefore, it is necessary to use computer vision to analyze the "texture structure" of the signal and then determine whether it is abnormal. At the same time, the sample after converting the vibration signal into a two-dimensional image is shown in Figure 7.

The data set division results under four different working conditions are shown in Table 3. Datasets A, B and C correspond to single cylinder misfires at three operating speeds of 1300rpm, 1800rpm and 2200rpm, respectively, and data set D corresponds to mixed cylinder misfires at 1800rpm. Since 1,049,600 vibration sequence points were collected at different speeds for each misfire fault, 511 samples can be obtained for each label, and each data set has five labels, Ⅰ, Ⅱ, Ⅲ, Ⅳ, and Ⅴ. It can be seen that each data set has a total of 2555 samples. Finally, the training set, test set and validation set account for 80%, 10% and 10% of these data sets respectively.

Table 3 Dataset division under different working conditions

Secondly, the environment used in this embodiment is shown in Table 4.

Table 4 Experimental environment

In this embodiment, in the hardware environment, the CPU uses Inter i7 12100, and the graphics processor uses Nvidia RTX 3080, which has 8704 CUDA cores and 184 Tensor cores, and has 10GB of video memory. In terms of software, PyTorch 1.11.0 is used as the deep learning framework API, the CUDA computing platform version is 11.3, and the cuDNN computing acceleration library version is 8.2.1.

This embodiment uses the following evaluation indicators to judge the performance of the network model:

(1) Precision: The accuracy of the model in judging whether a signal is abnormal, i.e., the precision rate. The calculation formula is:

(2) Recall: evaluates the model's sensitivity to abnormal signals, i.e., recall rate. Its calculation formula is:

(3) F1-Score: The harmonic mean of Precision and Recall indicators. Its calculation formula is:

(4) Average Precision (AP): Evaluates the average detection accuracy of the model. It reflects the overall performance of the model by calculating the lower bounding area of the Precision-Recall index curve. The calculation formula is:

In the above formula, TP, FP, and FN are elements in the confusion matrix, and the meaning of each element is:

TP: the number of correctly detected targets, that is, the number of image areas with abnormal signals correctly detected;

FP: the number of false detection targets, that is, the number of image areas with normal signals judged as abnormal signals;

FN: The number of missed targets, that is, the number of image areas where abnormal signals are not detected.

In the ablation experiment, the network MITDCNN designed by the present invention is first disassembled into the following three groups of networks:

Network 1: In the feature extraction network, only the convolution module is used to extract the two-dimensional image features, and the mixed channel feature fusion network is not used, and only the detection network is used;

Network 2: In the feature extraction network, a multimodal feature extraction network is used to extract one-dimensional and two-dimensional features. The mixed-channel feature fusion network is not used, and only the detection network is used.

Network 3: A multimodal feature extraction network and a mixed channel feature fusion detection network, namely the MITDCNN network.

The above three networks were tested on the vibration signal data set of the engine at different working speeds using the above evaluation indicators. The comparative test results are shown in Tables 5-8.

Table 5 Comparison of 1300rpm single cylinder low speed condition data set

The comparative test results of different networks on dataset A are shown in Table 5. The Precision, Recall, F1-Score and AP indicators of network 1 are 83.128%, 84.035%, 83.579% and 88.268% respectively, all within the range of [80%, 90%]. The Precision, Recall, F1-Score and AP indicators of network 2 are all over 90%, which are 9.229%, 11.017%, 10.106% and 6.589% higher than those of network 1. In network 3, its Precision, Recall, F1-Score and AP indicators are 99.738%, 99.888%, 99.812% and 99.926% respectively, which are extremely close to 100%, and its standard deviation is controlled within 1%.

Table 6 Comparison of 1800rpm single cylinder medium speed condition data set

The comparative test results of different networks on dataset B are shown in Table 6. The Precision of Network 1 is 83.571% and the Recall is 84.268%. The F1-Score and AP indicators of Network 2 and Network 3 are 93.724%, 94.968% and 99.735%, 99.853% respectively, all exceeding 90%.

Table 7 Comparison of 2200rpm single cylinder high-speed working condition data set

The comparative test results of different networks on dataset C are shown in Table 7. The Precision and Recall of network 1 are 84.287% and 85.937% respectively, which are higher than the Precision and Recall values of network 1 in datasets A and B. The Recall, F1-Score and AP indicators of network 2 are all higher than 95%, which are 96.872%, 95.201% and 95.587% respectively. In network 3, its Precision, Recall, F1-Score and AP indicators are 99.184%, 99.179%, 99.181% and 99.217% respectively, which are 14.897%, 13.242%, 14.077% and 11.292% higher than those of network 1.

Table 8 Comparison of 1800rpm mixing cylinder medium speed condition data set

The comparative test results of different networks on dataset D are shown in Table 8. The Precision, Recall, F1-Score and AP indicators of network 1 are 81.918%, 84.268%, 83.076% and 86.687% respectively, and the standard deviation is within 4%. The Precision, Recall, F1-Score and AP indicators of network 2 are improved by 8.619%, 11.225%, 9.872% and 4.571% compared with network 1. In network 3, its Precision, Recall, F1-Score and AP indicators are all higher than 99%, which are improved by 8.473%, 4.089%, 6.347% and 8.481% compared with network 2.

According to the comparison in Tables 5-8, network 1 only uses the convolution module to extract relevant features from the converted image, and the gray point texture map obtained by signal conversion has a difference that cannot clearly indicate whether the signal is normal or not. Therefore, the precision and recall indicators obtained on the four data sets are low, which makes its F1-Score and AP comprehensive performance indicators also low. In network 2, compared with network 1, it uses multimodal feature extraction for feature extraction. The richness of features has been greatly improved after the backbone network fuses the -dimensional amplitude features, and the use of the Transformer module can extract global features to complement the local features of the convolution module, and the feature level is more complete. Therefore, in the comparison results in the above table, the various indicators of network 1 and network 2 have been greatly improved. Finally, in network 3, the main difference compared with network 2 is the addition of a mixed channel feature fusion detection network, which improves the significance of abnormal signal features by mixing and aggregating the extracted features at different levels, and weighting features from the two dimensions of spatial domain and channel domain. From the test and comparison results, all the data of network 3 are the best, and the improvement is also larger than that of network 2. Therefore, it also shows that the mixed channel feature fusion detection network is very helpful in improving network performance.

Secondly, in order to simulate the misfire fault signal detection performance under different noise environments, noise with different signal-to-noise ratios is added to the original data set. Let P _signal and P _noise be the signal and noise energy respectively, then the signal-to-noise ratio SNR is defined as follows:

In the ablation experiment in a noisy environment, Gaussian white noise with different signal-to-noise ratios (-4, -2, 0, 2, 4, 6, 8, 10dB) was added to the original signals of the engine at different operating speeds and cylinder numbers. The detection performance of each network for abnormal signals under different working conditions and noise environments was compared and tested. The test results of the three networks are shown in Tables 9-11 respectively.

Table 9 Detection performance test of network 1 under various working environments with different signal-to-noise ratios

Table 9 shows the detection performance indicators of network 1 under different data sets and noise interference of different intensities. In data set A, the average indicators of its precision, recall, F1-Score (measurement of the model's ability to find positive examples) and average precision (AP) are 76.636%, 78.058%, 77.340%, and 82.038%. In data sets B and C, its Precision values are very close, 75.616% and 75.278% respectively. In data set D, its Precision value is 73.350%, the lowest value among the four data sets. Under -4dB noise interference, the average values of Precision, Recall, F1-Score and AP of network 1 in the four data sets fluctuate in the range of [70%, 80%], which are 70.760%, 72.757%, 71.743%, and 76.995% respectively.

Table 10 Detection performance test of network 2 under various working environments with different signal-to-noise ratios

The detection performance indicators of network 2 under different data sets and noise interference of different intensities are shown in Table 10. In data set A, its Recall value is higher than 90%, which is 91.786%, which is 13.728% higher than the Recall value of network 1 in data set A. The average indicators of its AP value in data sets B, C, and D are 92.482%, 92.965%, and 89.345%, respectively, and the standard deviation in data set C is 1.414%. Under -2dB noise interference, the average values of Recall, F1-Score, and AP of network 2 in the four data sets are all higher than 90%, which are 91.940%, 90.845%, and 92.225%, respectively, which are 1.934%, 1.587%, and 1.455% higher than the average values of Recall, F1-Score, and AP under 8dB noise interference, respectively.

Table 11 Detection performance test of network 3 under various working environments with different signal-to-noise ratios

Table 11 shows the detection performance indicators of network 3 under different data sets and noise interference of different strengths. In data set A, the average indicators of its Precision, Recall, F1-Score and AP are all higher than 99.7%, which are 99.711%, 99.754%, 99.732% and 99.856% respectively. In data sets B and C, the average indicators of its Precision, Recall, F1-Score and AP fluctuate in the range of [99.1%-99.9%], which is extremely close to 100%. In data set D, the average indicators of its Precision, Recall, F1-Score and AP are 98.995%, 99.304%, 99.149% and 99.699%, which are much higher than the average indicators of Precision, Recall, F1-Score and AP of networks 1 and 2 in data set D.

By comparing the test results of the three networks under different working conditions and noise environments, the following conclusions can be drawn: for network 1, only the convolution module is used to extract features, and the image texture features converted after adding noise are greatly disturbed. Therefore, in the test results, the performance indicators of network 1 fluctuate greatly; while network 2 adopts a multimodal feature extraction scheme. The noise resistance of the network is improved to a certain extent by adding one-dimensional features. Compared with the other two networks, network 3 adds a mixed channel feature fusion module, which includes spatial attention and channel attention calculations, and can suppress noise in the spatial domain and channel domain. Therefore, in the test results, the fluctuations caused by noise in various indicators of network 3 are small, reflecting strong robustness. Secondly, the iterative curves of the loss function value and accuracy value during the training of the network 3 (MITDCNN) model are shown in Figures 8 and 9.

In this embodiment, after multiple experimental tests, the following optimal hyperparameter settings are obtained: the number of epoch iterations is 200, the initial learning rate is set to 0.001, and gradually converges to 0.00001 after training, the momentum parameter momentum is set to 0.954, and the weight decay rate weight decay is set to 0.0013. As can be seen from Figures 8-9, after 100 epochs, the loss and model accuracy remain stable. The loss function value quickly decreases and converges in a short period of time, and the AP value also increases rapidly, indicating that the network has a strong learning ability for the target features.

The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims (10)

1. A mechanical fault detection method based on multimodal analysis of engine vibration signals, characterized in that it comprises the following steps: Collect vibration signals of generators; Inputting the vibration signal into a multimodal feature extraction network to obtain multiple feature information; Select p feature information and input it into the mixed channel feature fusion detection network, perform secondary feature processing and detection, and output the detection results The multimodal feature extraction network includes a shaping network, a convolution module, a fully connected module and a multimodal Transformer module; The shaping network is used to convert the input engine vibration signal into a two-dimensional image, and the convolution module performs feature extraction based on the two-dimensional image to obtain a two-dimensional feature image; The fully connected module is used to extract a one-dimensional feature vector of the input engine vibration signal; the multimodal Transformer module is used to integrate the one-dimensional feature vector and the two-dimensional feature image; The mixed-channel feature fusion detection network includes a feature aggregation module, a feature mixing module and a multi-scale detection module; The feature aggregation module is used to aggregate the p feature information to obtain an aggregated feature map FM; The feature mixing module is used to perform feature mixing on the aggregated feature map FM to obtain a plurality of feature maps of different sizes; The multi-scale detection module is used to detect the mixed feature map, and to determine whether there is an abnormal vibration signal in the diesel engine cylinder during the current period by detecting whether there is an abnormal or irregular texture area in the feature map. 2. According to claim 1, a mechanical fault detection method based on multimodal analysis of engine vibration signals is characterized in that the multimodal feature extraction network includes a q-layer structure, and each layer of the structure includes a convolution module, a fully connected module and a multimodal Transformer module. 3. A mechanical fault detection method based on multimodal analysis of engine vibration signals according to claim 2, characterized in that the multimodal feature extraction network extracts features from the vibration signal to obtain multiple feature information, comprising the following steps: S1, the vibration signal is input into the shaping network and the fully connected module respectively, the shaping network outputs a two-dimensional image, and the two-dimensional image is input into the convolution module for feature extraction; S2, the fully connected module outputs a one-dimensional feature vector; S3, the convolution module outputs a two-dimensional feature image; S4, inputting the one-dimensional feature vector and the two-dimensional feature image into a multimodal Transformer module to obtain integrated feature information; S5, inputting the integrated feature information and the two-dimensional feature image into the convolution module of the next layer, and inputting the one-dimensional feature vector into the fully connected module of the next layer; S6. Repeat steps S2-S5 until the qth layer structure of the multimodal feature extraction network is reached, and perform feature extraction layer by layer to obtain multiple feature information. 4. According to claim 1, a mechanical fault detection method based on multimodal analysis of engine vibration signals is characterized in that the multimodal Transformer module includes two multi-head attention networks, which respectively correspond to the long-distance relationship interaction between the one-dimensional feature vector and the two-dimensional feature image. 5. A mechanical fault detection method based on multimodal analysis of engine vibration signals according to claim 4, characterized in that the multimodal Transformer module integrates the one-dimensional feature vector and the two-dimensional feature image, comprising the following steps: Divide the one-dimensional feature vector into n tokens sub-vectors; Divide the two-dimensional feature image into n feature blocks, and extend the feature blocks into n tokens sub-vectors; In the first multi-head attention network, the tokens corresponding to the two-dimensional feature image are input into the matrix In the example, the tokens corresponding to the one-dimensional feature vector are input into the matrix and In the process, the query matrix Q with the image feature information is calculated and the key matrix K with the one-dimensional amplitude feature n information is matched, and the obtained matching degree is assigned to the corresponding eigenvalue matrix V to complete the operation of mapping the image feature to the amplitude feature; In the second multi-head attention network, the tokens corresponding to the two-dimensional feature image are input into the matrix In the example, the tokens corresponding to the one-dimensional feature vector are input into the matrix and In the process, the query matrix Q with the image feature information is calculated and the key matrix K with the one-dimensional amplitude feature n information is matched, and the obtained matching degree is assigned to the corresponding eigenvalue matrix V to complete the operation of mapping the image feature to the amplitude feature; Among them, in the two multi-head attention networks, the corresponding two sets of Q, K, and V vectors are respectively and The matrix is calculated; The output one-dimensional feature vectors of the two multi-head attention networks are merged, and the fully connected layer and ReLU activation function are used to activate the merged features. Finally, the obtained 1×1×C vector is integer-calculated to obtain a feature map of H×W×C dimensions. 6. A mechanical fault detection method based on multimodal analysis of engine vibration signals according to claim 1, characterized in that the feature aggregation module aggregates the feature maps FM ₁ , FM ₂ and FM ₃ output by the three convolution modules at the bottom layer of the multimodal feature extraction network, comprising the following steps: Improve the feature information of each feature map; FM ₃ doubles the size of the feature map by deconvolution and compresses the number of channels to half of the original. The features of FM ₃ are assigned to FM ₂ using the add fusion method to obtain FM ₂ ′; Upsample and channel compress the FM ₂ feature map, and fuse it with the features of the FM ₁ feature map to obtain FM ₁ ′; The concat layer is used to merge the three feature maps FM ₁ ′, FM ₂ ′ and FM _3. During the merging operation, FM ₁ ′ is downsampled and FM ₃ is upsampled to obtain the feature map FM. 7. A mechanical fault detection method based on multimodal analysis of engine vibration signals according to claim 6, characterized in that the feature mixing module performs feature mixing on the aggregated feature map FM, comprising the following steps: A 1x1 convolutional layer is used to compress and merge the number of feature channels of FM to the same as that of FM ₂ ′, and a grouped convolution is used to divide FM into two feature maps, FM_1 and FM_2, which have the same size as FM and half the number of channels as FM. Perform feature extraction operations on the grouped feature maps FM_1 and FM_2 to obtain FM_1′ and FM_2′; The same parallel convolution module is used to extract feature maps of different sizes and number of channels from FM_1′ and FM_2′ to form two paired groups; A fusion module composed of a concat layer and a 1x1 convolution layer is used to merge and mix the feature maps in the pairing groups to obtain three feature maps of different sizes, namely FM1, FM2 and FM3. 8. A mechanical fault detection method based on multimodal analysis of engine vibration signals according to claim 7, characterized in that spatial attention calculation is used for feature map FM_1 to enhance the morphological feature information FM_1′ corresponding to the abnormal signal; channel attention calculation is used for FM_2 to enhance the semantic feature weight FM_2′ of the abnormal signal; The pairing group extracted from FM_1′ contains the spatial attention weighted feature value, and the pairing group extracted from FM_2′ contains the channel attention weighted feature value. 9. According to claim 1, a mechanical fault detection method based on multimodal analysis of engine vibration signals is characterized in that the multi-scale detection module is used to detect the feature map output by the feature mixing module, and the cross entropy classification loss is used to adjust the classification module of the detector. 10. According to claim 1, a mechanical fault detection method based on multimodal analysis of engine vibration signals is characterized in that the multi-scale detection module is used to detect the feature map output by the feature mixing module, and the position loss is calculated based on the CIoU position evaluation relationship.