CN116842423A - Aeroengine fault diagnosis method and system based on multi-mode deep learning - Google Patents
Aeroengine fault diagnosis method and system based on multi-mode deep learning Download PDFInfo
- Publication number
- CN116842423A CN116842423A CN202310717006.XA CN202310717006A CN116842423A CN 116842423 A CN116842423 A CN 116842423A CN 202310717006 A CN202310717006 A CN 202310717006A CN 116842423 A CN116842423 A CN 116842423A
- Authority
- CN
- China
- Prior art keywords
- data
- deep learning
- engine
- modal
- mode
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003745 diagnosis Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013135 deep learning Methods 0.000 title claims abstract description 25
- 238000012423 maintenance Methods 0.000 claims abstract description 53
- 238000013136 deep learning model Methods 0.000 claims abstract description 44
- 238000007781 pre-processing Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 15
- 238000013461 design Methods 0.000 claims abstract description 7
- 239000013598 vector Substances 0.000 claims description 55
- 238000004458 analytical method Methods 0.000 claims description 15
- 239000000446 fuel Substances 0.000 claims description 13
- 238000012544 monitoring process Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 11
- 230000006872 improvement Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 9
- 230000001133 acceleration Effects 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 230000015654 memory Effects 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000013024 troubleshooting Methods 0.000 claims description 6
- 238000005311 autocorrelation function Methods 0.000 claims description 4
- 238000002485 combustion reaction Methods 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000012800 visualization Methods 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 208000024891 symptom Diseases 0.000 description 7
- 230000004927 fusion Effects 0.000 description 6
- 230000008439 repair process Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 206010000369 Accident Diseases 0.000 description 1
- 238000000342 Monte Carlo simulation Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 230000005389 magnetism Effects 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000010183 spectrum analysis Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2123/00—Data types
- G06F2123/02—Data types in the time domain, e.g. time-series data
Abstract
The invention discloses an aeroengine fault diagnosis method and system based on multi-mode deep learning, which belong to the field of aeroengine fault diagnosis and prediction, and comprise the following steps: collecting multi-modal data, wherein the multi-modal data comprises image data, sound data, text data and time sequence data; preprocessing the acquired multi-mode data; designing a multi-mode deep learning model for an aeroengine maintenance scene; after the design is completed, training the multi-mode deep learning model by using training data and optimizing the multi-mode deep learning model so as to achieve expected performance indexes; fusing the data of different modes; and inputting the fused multi-modal data into the multi-modal deep learning model, and performing fault diagnosis and prediction on the engine by using the trained multi-modal deep learning model. The invention improves the accuracy and efficiency of the fault diagnosis of the engine and reduces the maintenance cost and risk.
Description
Technical Field
The invention relates to the field of aeroengine fault diagnosis and prediction, in particular to an aeroengine fault diagnosis method and system based on multi-mode deep learning.
Background
The aeroengine is used as a heart of the aircraft to provide a flight power source for the aircraft, the reliability of the aeroengine is critical to the safe flight of the aircraft, if the aeroengine is abnormal, serious consequences are caused, the aircraft returns to the air abnormally when the aircraft is light, and the aircraft is destroyed when the aircraft is heavy. There are studies that have been counted: among 785 flight accidents, the accident 361 caused by the engine accounts for 45.99% of the total number of aircraft accidents. Therefore, how to ensure the normal operation of the engine becomes a primary problem to be considered.
The fault diagnosis technology is the power for the innovation of the maintenance mode of the engine, and the maintenance mode can be divided into 4 stages. The first stage: after maintenance, the engine is repaired after the engine fails, and the engine can be used for any sudden failure. And a second stage: the maintenance is planned, the parts with multiple faults are maintained in a targeted mode, the planning is high, and the fault repair task can be efficiently completed. And a third stage: and (5) according to the conditions, monitoring the state of the engine, collecting relevant fault judging data, processing and analyzing the data, and judging whether the engine needs to be repaired or not. Fourth stage: predicting maintenance, predicting possible fault modes in the future by means of state monitoring and fault analysis information, and making a predictive maintenance plan in advance. Predictive maintenance must be a marker for future advanced engines.
The prior art has the following problems: because of the complexity of the engine and the diversity of fault forms, a simple one-to-one correspondence does not exist between the fault mechanism and the symptoms, one fault can cause multiple symptoms, and the same symptom can also be caused by different faults, so that the difficulty is increased for fault identification.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides an aeroengine fault diagnosis method and system based on multi-mode deep learning, which improves the accuracy and efficiency of engine fault diagnosis, reduces maintenance cost and risk and the like.
The invention aims at realizing the following scheme:
an aeroengine fault diagnosis method based on multi-mode deep learning comprises the following steps:
s01, acquiring multi-modal data, wherein the multi-modal data comprises image data, sound data, text data and time sequence data;
s02, preprocessing the acquired multi-mode data;
s03, designing a multi-mode deep learning model for an aeroengine maintenance scene; after the design is completed, training the multi-mode deep learning model by using training data and optimizing the multi-mode deep learning model so as to achieve expected performance indexes;
s04, fusing the data of different modes;
s05, inputting the fused multi-modal data into the multi-modal deep learning model, and performing fault diagnosis and prediction on the engine by using the trained multi-modal deep learning model.
Further, after step S05, the method further includes the steps of: the output results of the multi-designed modal deep learning model are visualized, wherein the visualization mode comprises the mode of displaying the output of the model through an interactive interface of images or animations.
Further, after step S05, the method further comprises the steps of: embedding the designed multi-mode deep learning model into a real-time monitoring system, and continuously analyzing and monitoring engine data in real time; and when the multi-mode deep learning model detects an abnormal condition, an alarm is timely generated.
Further, after step S05, the method further includes the steps of: recommending maintenance steps and tools according to the analysis result and the suggestion of the multi-mode deep learning model, and providing operation guidance to optimize the maintenance process; or setting an automatic maintenance process according to the analysis result and the suggestion of the multi-mode deep learning model.
Further, in step S01, the image data is acquired by photographing the image data inside the engine through the industrial endoscope; the sound data are stored in a designated folder after being collected, and the sound data comprise fan sound data, high-pressure compressor sound data, combustion chamber sound data, high-pressure turbine sound data and tail nozzle sound data; the time sequence data is stored in a time sequence database after being collected, and the time sequence data comprises vibration acceleration data, engine rotating speed data, engine exhaust temperature data, fuel flow data, engine thrust data, turbine gas temperature data and fuel pressure data; the text data is obtained from an engine maintenance manual and an troubleshooting manual provided by an engine manufacturer.
Further, in step S02, the text data is preprocessed by removing stop words from the obtained text data, and obtaining an embedded vector of the text data through a dictionary; the voice data preprocessing mode is to acquire engine audio data stored in a designated folder, and noise in the audio data is reduced through Fourier transformation; the image data preprocessing is used for removing noise points in the picture and enhancing the image; time series data preprocessing is used to remove outliers.
Further, in step S03, the designing a multi-modal deep learning model for an aeroengine repair scenario includes the sub-steps of:
modeling time sequence data and extracting feature vectors: modeling the preprocessed time sequence data, capturing the time sequence characteristics of the data by using a cyclic neural network RNN or a long-short-term memory network LSTM, inputting a sensor data sequence designed to be in sequence of time, and outputting a sensor data sequence designed to be in sequence of time as a time sequence characteristic vector; performing feature engineering processing on the time sequence data to extract feature data, wherein the feature data comprises sliding window statistical features or frequency domain features or time lag autocorrelation functions of the time sequence data;
modeling text data and extracting feature vectors: modeling the preprocessed text data, and obtaining feature vectors of the text data from embedded vectors of the text data through a feature extractor of a transducer;
modeling image data and extracting feature vectors: carrying out feature extraction on the preprocessed image data through convolution and pooling to obtain feature vectors of the image data;
extracting the characteristic vector of the audio data: and carrying out time domain and frequency domain feature extraction on the waveform of the preprocessed sound data, and extracting MFCC parameters and GFCC parameters representing sound features.
Further, in step S04, the fusing the data of different modalities includes the following sub-steps:
firstly converting feature vectors of different modes into high-dimensional feature expression, and then fusing common features of different mode data in a high-dimensional space at selected positions by adopting a method based on attention; the selected position is specifically a position capable of realizing information complementation among different modal data and is positioned in the middle layer of the model; the feature vectors of the different modalities include an image data feature vector, a text data feature vector, a time series data feature vector, and an audio data feature vector.
Further, in step S05, the fault diagnosis and prediction of the engine using the trained multi-modal deep learning model includes the following sub-steps:
fault diagnosis and prediction: inputting real-time aeroengine sensor data into a model, analyzing and predicting the model according to time sequence characteristics of the data, and outputting a fault diagnosis result or a fault prediction probability; and/or the number of the groups of groups,
model evaluation and improvement: evaluating the established model, and evaluating the accuracy, recall rate and accuracy index of the model; according to the evaluation result, carrying out model improvement and optimization; the improvement and optimization includes adjusting model structure, adding training data, and adjusting hyper-parameters.
An aeroengine fault diagnosis system based on multi-modal deep learning, comprising a computer device having a program stored in a memory of the computer device, which when loaded by a processor performs the aeroengine fault diagnosis method based on multi-modal deep learning as defined in any one of the above.
The beneficial effects of the invention include:
the invention adopts data of different modes to fuse and integrate, so that the system can comprehensively understand the state and the problem of the engine from multiple angles. Through the designed multi-mode deep learning model, the multi-mode data is subjected to joint analysis, the model can automatically learn the relevance between different fault modes and characteristics, and more accurate fault identification and prediction are realized, so that maintenance measures are taken in advance, and the downtime and maintenance cost are reduced.
The invention can help to improve the efficiency, accuracy and safety of the maintenance of the aeroengine, provide more powerful tools and decision support for maintenance personnel, and reduce the maintenance cost and risk.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a flow chart of an embodiment of the present invention;
fig. 2 is a diagram of a model network structure according to an embodiment of the present invention.
Detailed Description
All of the features disclosed in all of the embodiments of this specification, or all of the steps in any method or process disclosed implicitly, except for the mutually exclusive features and/or steps, may be combined and/or expanded and substituted in any way.
In view of the problems in the background, after further thought and analysis, the inventors of the present invention have found that the existing aeroengine fault prediction and diagnosis schemes mainly include the following:
scheme one: direct fault diagnosis based on physicochemical principles. The method generally observes the change rule and characteristics of the aeroengine directly according to various means such as vibration, abnormal sound, light, heat, electricity, magnetism and the like, and takes the change rule and characteristics as the basis of whether the engine has faults.
Scheme II: fault diagnosis based on signal processing. And establishing a threshold model through signal processing and analysis methods such as time domain, frequency domain, amplitude, wavelet analysis, autoregressive spectrum analysis and the like to carry out fault diagnosis.
Scheme III: rule-based expert system diagnostics. Rules describing the relationship between faults and symptoms are summarized from the experience of the expert, and then fault diagnosis is performed through the rules.
Scheme IV: fault tree based diagnostic methods. The fault tree is a directed graph showing the relation of fault propagation, and takes the most unexpected event of the diagnosis object as a top event, and spreads layer by layer according to the structure and function relation of the object until an inseparable event (bottom event). The method comprises the steps of establishing a corresponding fault tree by receiving an information instruction of establishing the fault tree by a user, receiving an instruction of binding preset basic data by the user aiming at a designated node in the fault tree, binding the preset basic data with the designated node, and acquiring and storing the fault tree; and obtaining engine parameter data, determining a fault tree corresponding to the engine parameter data, performing fault diagnosis according to the fault tree corresponding to the engine parameter data, and obtaining a diagnosis result.
Scheme five: a neural network-based fault diagnosis method.
Scheme six: fault diagnosis based on the monte carlo method.
Scheme seven: mainly adopts a fuzzy theory diagnosis method, a gray theory diagnosis method, a fault diagnosis method of multi-sensor information fusion and a fuzzy neural network method.
The inventors of the present invention further conducted inventive analysis and found that the above-described scheme mainly has the following problems: the first scheme, the sixth scheme and the seventh scheme can detect partial faults only; the second disadvantage of the scheme is that the diagnosis precision is not high; the third disadvantage of the scheme is that the fault mode covered by the knowledge base is limited, the fault diagnosis effect on the fault which does not occur and has insufficient experience is poor, and misdiagnosis is easy to cause; the fourth disadvantage of the scheme is that the fault tree is established on the basis of element connection and fault mode analysis, unpredictable faults cannot be diagnosed, and the diagnosis result depends on the completeness of fault tree information; the fifth, sixth and seventh disadvantages are that the neural network diagnosis method is a 'black box method', the potential relationship inside the system cannot be revealed, the diagnosis process cannot be clearly explained, and in addition, the problems of slow fault diagnosis speed and the like caused by huge model exist.
Aiming at the problems of the scheme, due to the complexity of the engine and the diversity of fault forms, a simple one-to-one correspondence relationship does not exist between a fault mechanism and symptoms, one fault can cause multiple symptoms, and the same symptom can be caused by different faults, so that difficulty is increased for fault identification. In the invention, the invention aims to combine engine sound signals, vibration acceleration signals, engine rotating speed signals, engine exhaust temperature signals, fuel flow signals, engine thrust signals, turbine gas temperature signals, engine maintenance manuals, engine troubleshooting manual data and image data shot by an industrial endoscope, fuse multi-mode data such as audio, numbers, texts, images and the like, perform joint learning, extract characteristic values representing faults, perform characteristic fusion, establish a fault identification model with unified representation of multi-mode fault characteristics, and finally form fault diagnosis decisions through a decision mechanism. Compared with a fault identification model of single-mode data, the accuracy of engine fault diagnosis can be greatly improved.
Specifically, the embodiment of the invention provides an application scheme of multi-mode deep learning in an aircraft engine maintenance scene, which comprises an aircraft engine fault diagnosis method based on multi-mode deep learning, as shown in fig. 1, and specifically comprises the following steps:
s01: multimodal data collection: multimodal data is collected including sensor data, image data, sound data, text data, and the like. The data come from monitoring systems of the engine, sensor networks and other related devices such as high-definition images taken of the interior of the engine through the hole detection device, engine maintenance manuals and troubleshooting manuals.
S02: multimodal data preprocessing: preprocessing the collected multi-modal data, including data cleaning, noise removal, data alignment, and the like. The aim of preprocessing is to ensure the quality and consistency of the data, and provide accurate and reliable input for the subsequent deep learning model.
S03: multimodal model design and training: and designing a multi-mode deep learning model suitable for an aeroengine maintenance scene. In this step, a combination of a plurality of models is involved, including an image processing model, a sound processing model, a time series model, a text processing model, and the like. After the design is completed, training the model by using training data, and tuning to reach the expected performance index, as shown in fig. 2, is a model structure diagram designed in this step.
S04: multimodal data fusion: the data of different modes are fused, and a model such as a multi-mode attention mechanism (optionally, a graph convolution neural network can also be used) is used for better fusing and extracting the information of the multi-mode data.
S05: fault diagnosis and prediction: and performing fault diagnosis and prediction on the engine by using the trained model. The multi-mode data is input into the model, the health condition of the engine is judged through the output of the model, the possible fault modes are identified, and the occurrence probability or time window of the fault is predicted.
On the basis of the steps, the method further comprises the following steps:
s06: visualization assistance: the output of the model is visualized for a service person to better understand and interpret the engine data. The output of the model is displayed through images, animations or other interactive interfaces, which helps maintenance personnel to quickly locate problems and formulate corresponding maintenance strategies.
S07: real-time monitoring and alerting: the model is embedded into a real-time monitoring system, and engine data is continuously analyzed and monitored in real time. When the model detects an abnormal condition, an alarm is generated in time, and relevant personnel are informed to respond and maintain.
S08: and (3) optimizing a maintenance process: and optimizing the maintenance process according to the analysis result and the suggestion of the model. Optimal repair steps and tools can be recommended, providing operational guidance, and even supporting automated repair procedures to improve efficiency and accuracy.
On the basis of the implementation steps, the invention also comprises the following implementation steps of step S01:
s001: audio and time-ordered data collection. Relevant state performance data during test bed and run in an engine room is collected by installing sound sensors, temperature sensors, acceleration and rotation speed sensors, vibration sensors, thrust sensors, turbine flow sensors, pressure transmitters and the like on key parts (such as fans, high-pressure compressors, combustion chambers, high-pressure turbines, tail pipes and the like) of an aircraft engine. The collected data are mainly: engine audio data, vibration acceleration data, engine speed data, engine exhaust temperature data, fuel flow data, engine thrust data, turbine gas temperature data, and fuel pressure data.
S002: image data collection: image data of the interior of the engine is acquired by photographing through an industrial endoscope such as a hole detection device. The data collected by various sensors are stored in a designated position of the ground computer, wherein the engine audio data is stored in a designated folder, and the vibration acceleration data, the engine rotation speed data, the engine exhaust temperature data, the fuel flow data, the engine thrust data, the turbine gas temperature data and the fuel pressure data are stored in a time sequence database TDengine.
S003: text data acquisition. And collecting an engine maintenance manual and an troubleshooting manual provided by an engine manufacturer.
On the basis of the above implementation steps, the present invention further includes the following implementation steps, where step S02 includes:
s004: text data preprocessing: and removing stop words from the acquired text data, and obtaining embedded vectors of the text data through a dictionary.
S005: preprocessing image data: and removing noise points in the picture and performing image enhancement through preprocessing the picture such as hog, PCA, K-means and the like.
S006: audio data preprocessing: engine audio data stored in a specified folder is acquired, and noise in the audio data is subjected to noise reduction processing by fourier transformation.
S007: preprocessing time sequence data: and removing outliers from time sequence data such as vibration acceleration data, engine rotation speed data, engine exhaust temperature data, fuel flow data, engine thrust data, turbine gas temperature data, fuel pressure data and the like in the TDengine.
On the basis of the above implementation steps, the present invention further includes the following implementation steps, where step S03 includes:
s008: modeling time sequence data and extracting features: modeling the preprocessed time series data. A Recurrent Neural Network (RNN) or long term memory network (LSTM) model is used to capture the timing characteristics of the data. The input may be a time-sequential sequence of sensor data and the output a time-sequential feature vector. In this step, the time series data is preferably feature engineered to extract more useful features. For example, sliding window statistics (e.g., mean, maximum, minimum, etc.) may be calculated, frequency domain features (e.g., fourier transforms, wavelet transforms, etc.) extracted, or time-lapse autocorrelation functions using time-series data, etc.
S009: modeling text data and extracting features: the embedded vector of the text data is used for obtaining the feature vector of the text data through a feature extractor of a transducer.
S010: modeling image data and extracting features: and carrying out feature extraction on the preprocessed image data through convolution, pooling and other operations to obtain feature vectors of the image data.
S011: extracting audio data characteristics: and extracting time domain and frequency domain characteristics of the waveform of the audio. The MFCC parameters and GFCC parameters characterizing the sound feature are extracted. The extraction process of the MFCC comprises the steps of preprocessing, fast Fourier transformation, mel filter bank, logarithmic operation, discrete cosine transformation, dynamic feature extraction and the like. In this step, the characteristic quantity is a state parameter reflecting the signal, and in the state monitoring, the obtained original data is often huge, so that in order to effectively classify the running state of the engine, the original data needs to be transformed to obtain a parameter which can most reflect the essential characteristic of the engine, that is, the characteristic extraction. The good characteristic quantity should have the following characteristics: (1) distinguishability; (2) reliability; (3) independence; (4) the number is small.
On the basis of the implementation steps, the invention further comprises the following implementation steps of step S04:
s012: and (5) multi-mode feature fusion. Feature vectors (image data feature vectors, text data feature vectors, time sequence data feature vectors and audio feature vectors) of different modes are firstly converted into high-dimensional feature expression, and then proper positions are selected for fusion by using common features of different mode data in a high-dimensional space by adopting an attention-based method. In the step, preferentially, the most suitable feature extraction model is selected for processing the data of each mode, the original information of the data is kept as much as possible, and then a suitable position is selected in the middle layer of the model for feature fusion, so that the information complementation among the data of different modes can be fully realized, and the operation level has stronger flexibility,
on the basis of the implementation steps, the invention further comprises the following implementation steps of step S05:
s013: fault diagnosis and prediction: and performing fault diagnosis and prediction by using the trained time sequence model. Inputting real-time aeroengine sensor data into a model, analyzing and predicting the model according to time sequence characteristics of the data, and outputting fault diagnosis results or fault prediction probability.
S014: model evaluation and improvement: and evaluating the established time sequence model, and evaluating indexes such as accuracy, recall rate, accuracy and the like of the model. And (3) carrying out model improvement and optimization according to the evaluation result, such as model structure adjustment, training data addition, super parameter adjustment and the like. In this step, preferably, a proper loss function and optimization algorithm are selected, and tuning and verification of the model are performed, so that the performance and generalization capability of the model are ensured.
Compared with the prior art, the invention has the following advantages:
(1) Aeroengine repair involves various types of data, such as sensor data, image data, sound data, etc. The invention fuses and integrates the data of different modes, so that the machine can comprehensively understand the state and the problem of the engine from multiple angles.
(2) Through carrying out joint analysis on the multi-mode data, the algorithm can automatically learn the relevance between different fault modes and characteristics, and realize more accurate fault identification and prediction, thereby taking maintenance measures in advance and reducing downtime and maintenance cost.
(3) Visual assistance, by converting multimodal data into visual images, provides intuitive and easy to understand information, helps maintenance personnel to better understand and interpret engine data, and makes accurate decisions during engine maintenance.
(4) The real-time monitoring and alarming functions of the engine are realized. By constantly analyzing the multi-modal data stream, the algorithm can quickly detect anomalies and generate alarms that alert maintenance personnel to take action in time to prevent further damage or failure.
(5) The maintenance process is optimized, and the multi-mode deep learning provides intelligent support and optimization in the engine maintenance process. The model based on the multi-mode data can recommend the best maintenance steps and tools, provide operation guidance and support an automatic maintenance process, thereby improving efficiency and accuracy.
The invention can help to establish a multi-mode fault diagnosis model of the aeroengine, realize fault diagnosis and prediction, improve the efficiency, accuracy and safety of aeroengine maintenance, provide stronger tools and decision support for maintenance personnel, and reduce maintenance cost and risk.
It should be noted that, within the scope of protection defined in the claims of the present invention, the following embodiments may be combined and/or expanded, and replaced in any manner that is logical from the above specific embodiments, such as the disclosed technical principles, the disclosed technical features or the implicitly disclosed technical features, etc.
Example 1
An aeroengine fault diagnosis method based on multi-mode deep learning comprises the following steps:
s01, acquiring multi-modal data, wherein the multi-modal data comprises image data, sound data, text data and time sequence data;
s02, preprocessing the acquired multi-mode data;
s03, designing a multi-mode deep learning model for an aeroengine maintenance scene; after the design is completed, training the multi-mode deep learning model by using training data and optimizing the multi-mode deep learning model so as to achieve expected performance indexes;
s04, fusing the data of different modes;
s05, inputting the fused multi-modal data into the multi-modal deep learning model, and performing fault diagnosis and prediction on the engine by using the trained multi-modal deep learning model.
Example 2
On the basis of embodiment 1, after step S05, the following steps are further included: the output results of the multi-designed modal deep learning model are visualized, wherein the visualization mode comprises the mode of displaying the output of the model through an interactive interface of images or animations.
Example 3
On the basis of embodiment 1, after step S05, the following steps are further included: embedding the designed multi-mode deep learning model into a real-time monitoring system, and continuously analyzing and monitoring engine data in real time; and when the multi-mode deep learning model detects an abnormal condition, an alarm is timely generated.
Example 4
On the basis of embodiment 1, after step S05, the following steps are further included: recommending maintenance steps and tools according to the analysis result and the suggestion of the multi-mode deep learning model, and providing operation guidance to optimize the maintenance process; or setting an automatic maintenance process according to the analysis result and the suggestion of the multi-mode deep learning model.
Example 5
On the basis of embodiment 1, in step S01, the image data is acquired by photographing the image data inside the engine through the industrial endoscope; the sound data are stored in a designated folder after being collected, and the sound data comprise fan sound data, high-pressure compressor sound data, combustion chamber sound data, high-pressure turbine sound data and tail nozzle sound data; the time sequence data is stored in a time sequence database after being collected, and the time sequence data comprises vibration acceleration data, engine rotating speed data, engine exhaust temperature data, fuel flow data, engine thrust data, turbine gas temperature data and fuel pressure data; the text data is obtained from an engine maintenance manual and an troubleshooting manual provided by an engine manufacturer.
Example 6
Based on embodiment 1, in step S02, the text data is preprocessed by removing stop words from the obtained text data, and obtaining an embedded vector of the text data through a dictionary; the voice data preprocessing mode is to acquire engine audio data stored in a designated folder, and noise in the audio data is reduced through Fourier transformation; the image data preprocessing is used for removing noise points in the picture and enhancing the image; time series data preprocessing is used to remove outliers.
Example 7
On the basis of embodiment 1, in step S03, the design of the multi-modal deep learning model for an aeroengine maintenance scenario comprises the sub-steps of:
modeling time sequence data and extracting feature vectors: modeling the preprocessed time sequence data, capturing the time sequence characteristics of the data by using a cyclic neural network RNN or a long-short-term memory network LSTM, inputting a sensor data sequence designed to be in sequence of time, and outputting a sensor data sequence designed to be in sequence of time as a time sequence characteristic vector; performing feature engineering processing on the time sequence data to extract feature data, wherein the feature data comprises sliding window statistical features or frequency domain features or time lag autocorrelation functions of the time sequence data;
modeling text data and extracting feature vectors: modeling the preprocessed text data, and obtaining feature vectors of the text data from embedded vectors of the text data through a feature extractor of a transducer;
modeling image data and extracting feature vectors: carrying out feature extraction on the preprocessed image data through convolution and pooling to obtain feature vectors of the image data;
extracting the characteristic vector of the audio data: and carrying out time domain and frequency domain feature extraction on the waveform of the preprocessed sound data, and extracting MFCC parameters and GFCC parameters representing sound features.
Example 8
Based on embodiment 1, in step S04, the fusing the data of different modalities includes the following sub-steps:
firstly converting feature vectors of different modes into high-dimensional feature expression, and then fusing common features of different mode data in a high-dimensional space at selected positions by adopting a method based on attention; the selected position is specifically a position capable of realizing information complementation among different modal data and is positioned in the middle layer of the model; the feature vectors of the different modalities include an image data feature vector, a text data feature vector, a time series data feature vector, and an audio data feature vector.
Example 9
On the basis of embodiment 1, in step S05, the fault diagnosis and prediction of the engine by using the trained multi-mode deep learning model includes the following sub-steps:
fault diagnosis and prediction: inputting real-time aeroengine sensor data into a model, analyzing and predicting the model according to time sequence characteristics of the data, and outputting a fault diagnosis result or a fault prediction probability; and/or the number of the groups of groups,
model evaluation and improvement: evaluating the established model, and evaluating the accuracy, recall rate and accuracy index of the model; according to the evaluation result, carrying out model improvement and optimization; the improvement and optimization includes adjusting model structure, adding training data, and adjusting hyper-parameters.
Example 10
An aeroengine fault diagnosis system based on multi-modal deep learning, comprising a computer device having a program stored in a memory of the computer device, which when loaded by a processor performs the aeroengine fault diagnosis method based on multi-modal deep learning of any one of embodiments 1 to 9.
The units involved in the embodiments of the present invention may be implemented by software, or may be implemented by hardware, and the described units may also be provided in a processor. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
According to an aspect of embodiments of the present invention, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read from the computer-readable storage medium by a processor of a computer device, and executed by the processor, cause the computer device to perform the methods provided in the various alternative implementations described above.
As another aspect, the embodiment of the present invention also provides a computer-readable medium that may be contained in the electronic device described in the above embodiment; or may exist alone without being incorporated into the electronic device. The computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the methods described in the above embodiments.
Claims (10)
1. The aeroengine fault diagnosis method based on multi-mode deep learning is characterized by comprising the following steps of:
s01, acquiring multi-modal data, wherein the multi-modal data comprises image data, sound data, text data and time sequence data;
s02, preprocessing the acquired multi-mode data;
s03, designing a multi-mode deep learning model for an aeroengine maintenance scene; after the design is completed, training the multi-mode deep learning model by using training data and optimizing the multi-mode deep learning model so as to achieve expected performance indexes;
s04, fusing the data of different modes;
s05, inputting the fused multi-modal data into the multi-modal deep learning model, and performing fault diagnosis and prediction on the engine by using the trained multi-modal deep learning model.
2. The method for diagnosing an aircraft engine fault based on multi-modal deep learning as recited in claim 1, further comprising, after step S05, the steps of: the output results of the multi-designed modal deep learning model are visualized, wherein the visualization mode comprises the mode of displaying the output of the model through an interactive interface of images or animations.
3. The method for diagnosing an aircraft engine fault based on multi-modal deep learning as claimed in claim 1, further comprising the steps of, after step S05: embedding the designed multi-mode deep learning model into a real-time monitoring system, and continuously analyzing and monitoring engine data in real time; and when the multi-mode deep learning model detects an abnormal condition, an alarm is timely generated.
4. The method for diagnosing an aircraft engine fault based on multi-modal deep learning as recited in claim 1, further comprising, after step S05, the steps of: recommending maintenance steps and tools according to the analysis result and the suggestion of the multi-mode deep learning model, and providing operation guidance to optimize the maintenance process; or setting an automatic maintenance process according to the analysis result and the suggestion of the multi-mode deep learning model.
5. The method for diagnosing an aeroengine fault based on multi-modal deep learning as claimed in claim 1, wherein in step S01, the image data is acquired by photographing the image data inside the engine through an industrial endoscope; the sound data are stored in a designated folder after being collected, and the sound data comprise fan sound data, high-pressure compressor sound data, combustion chamber sound data, high-pressure turbine sound data and tail nozzle sound data; the time sequence data is stored in a time sequence database after being collected, and the time sequence data comprises vibration acceleration data, engine rotating speed data, engine exhaust temperature data, fuel flow data, engine thrust data, turbine gas temperature data and fuel pressure data; the text data is obtained from an engine maintenance manual and an troubleshooting manual provided by an engine manufacturer.
6. The method for diagnosing an aircraft engine fault based on multi-modal deep learning according to claim 1, wherein in step S02, the preprocessing mode of the text data is to remove stop words from the acquired text data, and obtain an embedded vector of the text data through a dictionary; the voice data preprocessing mode is to acquire engine audio data stored in a designated folder, and noise in the audio data is reduced through Fourier transformation; the image data preprocessing is used for removing noise points in the picture and enhancing the image; time series data preprocessing is used to remove outliers.
7. The method for diagnosing an aircraft engine fault based on multi-modal deep learning according to claim 1, wherein in step S03, the designing of the multi-modal deep learning model for an aircraft engine maintenance scenario includes the sub-steps of:
modeling time sequence data and extracting feature vectors: modeling the preprocessed time sequence data, capturing the time sequence characteristics of the data by using a cyclic neural network RNN or a long-short-term memory network LSTM, inputting a sensor data sequence designed to be in sequence of time, and outputting a sensor data sequence designed to be in sequence of time as a time sequence characteristic vector; performing feature engineering processing on the time sequence data to extract feature data, wherein the feature data comprises sliding window statistical features or frequency domain features or time lag autocorrelation functions of the time sequence data;
modeling text data and extracting feature vectors: modeling the preprocessed text data, and obtaining feature vectors of the text data from embedded vectors of the text data through a feature extractor of a transducer;
modeling image data and extracting feature vectors: carrying out feature extraction on the preprocessed image data through convolution and pooling to obtain feature vectors of the image data;
extracting the characteristic vector of the audio data: and carrying out time domain and frequency domain feature extraction on the waveform of the preprocessed sound data, and extracting MFCC parameters and GFCC parameters representing sound features.
8. The method for diagnosing an aircraft engine fault based on multi-modal deep learning according to claim 1, wherein in step S04, the fusing of data of different modalities includes the following sub-steps:
firstly converting feature vectors of different modes into high-dimensional feature expression, and then fusing common features of different mode data in a high-dimensional space at selected positions by adopting a method based on attention; the selected position is specifically a position capable of realizing information complementation among different modal data and is positioned in the middle layer of the model; the feature vectors of the different modalities include an image data feature vector, a text data feature vector, a time series data feature vector, and an audio data feature vector.
9. The method for diagnosing an aircraft engine fault based on multi-modal deep learning according to claim 1, wherein in step S05, the fault diagnosis and prediction of the engine using the trained multi-modal deep learning model comprises the following sub-steps:
fault diagnosis and prediction: inputting real-time aeroengine sensor data into a model, analyzing and predicting the model according to time sequence characteristics of the data, and outputting a fault diagnosis result or a fault prediction probability; and/or the number of the groups of groups,
model evaluation and improvement: evaluating the established model, and evaluating the accuracy, recall rate and accuracy index of the model; according to the evaluation result, carrying out model improvement and optimization; the improvement and optimization includes adjusting model structure, adding training data, and adjusting hyper-parameters.
10. An aeroengine fault diagnosis system based on multi-modal deep learning, comprising computer means, in the memory of which a program is stored, which when loaded by a processor performs the aeroengine fault diagnosis method based on multi-modal deep learning according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310717006.XA CN116842423A (en) | 2023-06-16 | 2023-06-16 | Aeroengine fault diagnosis method and system based on multi-mode deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310717006.XA CN116842423A (en) | 2023-06-16 | 2023-06-16 | Aeroengine fault diagnosis method and system based on multi-mode deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116842423A true CN116842423A (en) | 2023-10-03 |
Family
ID=88166230
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310717006.XA Pending CN116842423A (en) | 2023-06-16 | 2023-06-16 | Aeroengine fault diagnosis method and system based on multi-mode deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116842423A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117011690A (en) * | 2023-10-07 | 2023-11-07 | 广东电网有限责任公司阳江供电局 | Submarine cable hidden danger identification method, submarine cable hidden danger identification device, submarine cable hidden danger identification equipment and submarine cable hidden danger identification medium |
| CN117095188A (en) * | 2023-10-19 | 2023-11-21 | 中国南方电网有限责任公司 | Electric power safety strengthening method and system based on image processing |
| CN117725529A (en) * | 2024-02-18 | 2024-03-19 | 南京邮电大学 | Transformer fault diagnosis method based on multi-mode self-attention mechanism |
-
2023
- 2023-06-16 CN CN202310717006.XA patent/CN116842423A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117011690A (en) * | 2023-10-07 | 2023-11-07 | 广东电网有限责任公司阳江供电局 | Submarine cable hidden danger identification method, submarine cable hidden danger identification device, submarine cable hidden danger identification equipment and submarine cable hidden danger identification medium |
| CN117011690B (en) * | 2023-10-07 | 2024-02-09 | 广东电网有限责任公司阳江供电局 | Submarine cable hidden danger identification method, submarine cable hidden danger identification device, submarine cable hidden danger identification equipment and submarine cable hidden danger identification medium |
| CN117095188A (en) * | 2023-10-19 | 2023-11-21 | 中国南方电网有限责任公司 | Electric power safety strengthening method and system based on image processing |
| CN117095188B (en) * | 2023-10-19 | 2023-12-29 | 中国南方电网有限责任公司 | Electric power safety strengthening method and system based on image processing |
| CN117725529A (en) * | 2024-02-18 | 2024-03-19 | 南京邮电大学 | Transformer fault diagnosis method based on multi-mode self-attention mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116842423A (en) | Aeroengine fault diagnosis method and system based on multi-mode deep learning | |
| EP1609033B1 (en) | Data analysis system and method | |
| US8332337B2 (en) | Condition-based monitoring system for machinery and associated methods | |
| JP5016519B2 (en) | Diagnostic system and method for predictive condition monitoring | |
| EP3105644B1 (en) | Method of identifying anomalies | |
| KR102141391B1 (en) | Failure data management method based on cluster estimation | |
| JP5342708B1 (en) | Anomaly detection method and apparatus | |
| US20100030521A1 (en) | Method for analyzing and classifying process data | |
| WO2011125130A1 (en) | Plant diagnosis device, diagnosis method, and diagnosis program | |
| CN104102773A (en) | Equipment fault warning and state monitoring method | |
| CN108027611B (en) | Decision assistance system and method for machine maintenance using expert opinion supervised decision mode learning | |
| KR102301201B1 (en) | Apparatus and method for precise state diagnosis of rotating machinery based on IoT sensor | |
| CN110307990B (en) | Intelligent diagnosis method and system for faults of diesel engine oil system | |
| KR102102346B1 (en) | System and method for condition based maintenance support of naval ship equipment | |
| CN117131110B (en) | Method and system for monitoring dielectric loss of capacitive equipment based on correlation analysis | |
| EP3469434B1 (en) | Automatic visual and acoustic analytics for event detection | |
| KR102306244B1 (en) | Method and apparatus for generating fault detecting model for device | |
| Allen et al. | Gas turbine machinery diagnostics: A brief review and a sample application | |
| KR102108975B1 (en) | Apparatus and method for condition based maintenance support of naval ship equipment | |
| Mishra et al. | Hybrid models for rotating machinery diagnosis and prognosis: estimation of remaining useful life | |
| JP6685124B2 (en) | Diagnostic device and diagnostic method | |
| CN117435908A (en) | Multi-fault feature extraction method for rotary machine | |
| US11339763B2 (en) | Method for windmill farm monitoring | |
| KR102350636B1 (en) | Apparatus and method of plant failure prediction | |
| KR102321602B1 (en) | Rotating machine fault detecting apparatus and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |