KR20210077389A - Deep learning based abnormal symptom detection system using mechanical sound of mechanical equipment - Google Patents

Abstract

The present invention relates to a deep learning-based abnormal symptom detection system through noise or vibration of mechanical equipment, which includes: a sensor for data collection; a feature extraction unit; a deep learning unit configured to perform deep learning through any one or a combination of two or more among artificial neural networks (ANN), deep neural networks (DNN), convolution neural networks (CNN), long short term memory recurrent neural networks (LSTM RNN), connectionist temporal classification (CTC), and attention mechanism; and an abnormality detection unit for detecting abnormal signs of mechanical equipment.

Description

Deep learning based abnormal symptom detection system using mechanical sound of mechanical equipment

The present invention relates to a system for detecting anomalies based on deep learning through noise or vibration of mechanical equipment, and more particularly, by collecting machine sounds generated from one or more facilities with a data collection sensor and applying a feature extraction algorithm to the characteristics of the machine sound It relates to a deep learning-based abnormal symptom detection system using machine sounds of machinery that reduces installation and maintenance costs by extracting and judging whether the feature data extracted from the machine sound is normal or abnormal through the learned deep learning model.

The conventional facility diagnosis technology collects data generated inside the facility and uses it to monitor the deterioration and/or failure of the driving unit. CBM (Condition Based Management) has been performed to minimize it.

For example, torque and speed data that can be collected from a motor driver, internal/external vibration data of equipment, temperature, slope, current, barometric pressure, pressure, alarm data, etc. Acquisition), etc., are used to predict equipment failure diagnosis.

However, there are several difficulties in the existing facility diagnosis technology, and the representative difficulties are as follows.

The first is that time series data is generally obtained from equipment, and it is difficult to diagnose equipment by directly quantifying the time series data obtained in the time domain.

This is because, according to the structural characteristics of each driving unit, the vibration corresponding to the natural frequency of each component is included in the data. Therefore, in order to perform quantitative diagnosis of equipment, the domain is changed from time to frequency and the diagnosis is performed. In order to accurately measure frequency characteristics in a wider range, the sampling rate is increased. Accordingly, there is a problem in that the data size increases.

Second, since the characteristics of data for each facility, process, model, and drive axis are all different, an individual diagnosis algorithm reflecting the characteristics of each data is required for effective diagnosis. However, a lot of time, money, and effort are required to construct such an individual diagnostic algorithm, and also, the built diagnostic algorithm is applied only to the corresponding data, so there is a limit in terms of utility.

Third, when acquiring data, most of it is normal data, and the amount of intermittent failure data is very small. However, in the diagnosis of equipment, only failure data is usually used, and most of the normal data is discarded immediately after collection.

Therefore, there is a problem in terms of utilization of normal data, and there may also be problems in terms of reliability of equipment diagnosis.

The present invention has been devised to solve the above problems, and collects machine sounds or vibrations generated in one or more facilities with a data collection sensor, applies a feature extraction algorithm to extract features for machine sounds, and uses the learned deep learning model. The purpose of this is to provide a deep learning-based abnormal symptom detection system using the machine sound of mechanical equipment that reduces installation and maintenance costs by determining whether the feature data extracted from the machine sound is normal or abnormal.

The present invention has the following features to achieve the above object.

The present invention provides a system for detecting anomalies based on deep learning using machine sounds of mechanical equipment, the system comprising: a data collection sensor for collecting machine sounds generated from one or more mechanical equipment; a feature extraction unit for receiving the machine sound collected from the data collection sensor and extracting features of the machine sound; a deep learning unit configured to perform deep learning using a combination of a convolutional neural network (CNN), a LSTM long short term memory recurrent neural network (RNN), and an attention mechanism using machine sound feature data; and an abnormality detection unit that receives the characteristic data extracted from the characteristic extraction unit and detects abnormal signs of the mechanical equipment through the deep learning unit.

Here, the sensor for data collection is a micro-microphone that collects mechanical sounds generated from mechanical equipment, and the deep learning-based abnormal symptom detection system provides maintenance history information when maintenance is performed according to abnormalities or settings of mechanical equipment. A maintenance input unit is included, and the deep learning unit performs deep learning through machine sound feature data before and after performing maintenance.

In addition, the feature extraction unit receives the machine sound collected from the data collection sensor, converts the machine sound into a frequency domain, and extracts features through noise removal and Mel Filter Bank.

In addition, the parameter used in extracting the feature by the feature extraction unit is any one or more of a frame window size, a window shift rate, a low level descriptor (LLD), or a high-level statistical function (HSF).

In addition, the deep learning-based anomaly detection system performs repeated cross-validation in the learning process of the deep learning unit using a 4-fold cross validation technique with more than 1000 machine sound data.

According to the present invention, the deep learning unit can detect abnormal signs of mechanical equipment by collecting data from the machine sound generated by the machine, and it is possible to collect the machine sound of several facilities with one micromicro, and predict the failure of important or expensive parts and general facilities. It has the effect of reducing the cost of sensor attachment and maintenance by judging by only the mechanical sound.

1 is a diagram schematically illustrating a deep learning-based anomaly detection system according to an embodiment of the present invention.
2 is a block diagram showing the internal configuration of a deep learning-based anomaly detection system according to an embodiment of the present invention.
3 is a diagram schematically illustrating a feature extraction process through a feature extraction unit according to an embodiment of the present invention.
4 is a diagram illustrating a process of obtaining an MFCC in a feature extraction unit according to an embodiment of the present invention.
5 is a diagram showing the structure of a CLA model according to an embodiment of the present invention.
6 is a diagram illustrating a state in which repeated cross-validation is performed in the learning process of the deep learning unit by a 4-fold cross validation technique according to an embodiment of the present invention.

In order to explain the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, preferred embodiments of the present invention are exemplified below and will be described with reference to them.

First, the terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention, and the singular expression may include a plural expression unless the context clearly indicates otherwise. Also in the present application, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a diagram schematically showing a deep learning-based anomaly detection system according to an embodiment of the present invention, and FIG. 2 is a deep learning-based abnormal symptom detection system according to an embodiment of the present invention according to an embodiment of the present invention. It is a block diagram showing the internal configuration of the sensing system.

3 is a diagram schematically illustrating a feature extraction process through a feature extraction unit according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating a process for obtaining an MFCC by the feature extraction unit according to an embodiment of the present invention, FIG. 5 is a diagram showing the structure of a CLA model according to an embodiment of the present invention, and FIG. 6 is a 4-fold cross validation technique according to an embodiment of the present invention. Repeated cross-validation is performed in the learning process of the deep learning unit. It is a drawing showing the appearance.

Referring to the drawings, a deep learning-based abnormal symptom detection system 100 according to an embodiment of the present invention includes a data collection sensor 10 for collecting machine sounds generated from one or more mechanical equipment, and the data collection sensor 10 ), a feature extraction unit 20 that extracts the characteristics of the machine sound by receiving the collected machine sound, and a Convolution Neural Network (CNN), Long Short Term Memory Recurrent Neural Network (LSTM RNN), Attention using the feature data of the machine sound. The deep learning unit 30 configured to perform deep learning through a combination of mechanisms, and receiving the feature data extracted from the feature extraction unit 20, abnormal symptoms for the machine equipment through the deep learning unit 30 It consists of an abnormality detection unit 40 that detects , and a maintenance input unit 50 that provides maintenance history information when maintenance is performed according to an abnormality or setting of mechanical equipment.

Here, the sensor 10 for data collection is provided to collect mechanical sounds generated from mechanical equipment, and preferably, a micro-microphone is applied.

The sensor 10 for data collection according to the present invention collects mechanical sounds generated from one or several mechanical equipment, and if necessary, a sensor that collects vibration sound or vibration itself rather than mechanical sound may be applied.

In addition, the feature extracting unit 20 is provided to receive the machine sound collected from the data collection sensor 10 and extract the features of the corresponding machine sound, and the feature data extracted by the feature extracting unit 20 is in various forms. may be formed, but according to an embodiment of the present invention, as shown in FIG. 3 , the mechanical sound collected from the data collection sensor 10 is received and the corresponding mechanical sound is converted into a frequency domain, and the noise removal and Mel filter bank ( Mel Filter Bank), etc., can be used to extract features.

The feature extracted by the feature extraction unit 20 is a process of performing deep learning in the deep learning unit 30 or extracting a component useful for detecting an abnormality in the abnormality detecting unit 40 from the machine sound.

The parameter used for extracting the feature by the feature extractor 20 may include any one or more of a frame window size, a window shift rate, a low level descriptor (LLD), and a high-level statistical function (HSF).

Here, as LLD, mfcc, log energy, filter bank energy, etc. may correspond, and as HSF, mean, variance, max, min, mode, quant. etc. may apply.

The extraction of such feature data is generally related to information compression and dimensionality reduction processes. The main research area of feature data extraction is the extraction of features that are robust to various noise environments/speakers/channel variations and that express temporal changes well.

As a reflection of auditory characteristics in the commonly used feature extraction process, filter bank analysis applying the frequency response of the cochlea, center frequency arrangement in mel or Bark scale units, increase in bandwidth according to frequency, and pre-emphasis filter are used.

The most widely used method for improving robustness is cepstral mean subtraction (CMS) to reduce the influence of convolutive channels. In order to reflect the dynamic characteristics of the speech signal, the first (delta) and second derivative values of the cepstrum are used.

CMS and differentiation can be thought of as filtering in the time axis direction, and are processes of obtaining temporally uncorrelated feature vectors in the time axis direction. The process of obtaining the cepstrum from the filterbank coefficients can be thought of as an orthogonal transform to transform the filterbank coefficients into uncorrelated.

There is a Mel frequency cepstral coefficient (MFCC) as a feature extracted by the feature extraction unit 20 according to the present invention.

Here, the method of obtaining the MFCC is briefly described. The audio signal is converted into a digital signal x(n) through an anti-aliasing filter and then A/D conversion. A digital audio signal passes through a digital pre-emphasis filter having a high-pass characteristic.

The reason for using this filter is first, high-band filtering is performed to model the frequency characteristics of the human outer/middle ear. This compensates for the 20 dB/decade attenuation by radiation from the lips, so that only the vocal tract characteristics are obtained from the voice.

Second, it compensates somewhat for the fact that the auditory system is sensitive to the spectral region above 1 kHz. In PLP feature extraction, the equal-loudness curve, which is the frequency characteristic of the human auditory organ, is used for direct modeling. The characteristic H(z) of the pre-emphasis filter is as follows, and a value in the range of 0.95-0.98 is used.

The pre-emphasized signal is divided into frames in block units by covering the Hamming window. All subsequent processing is performed on a frame-by-frame basis. The size of a frame is usually 20-30 ms, and 10 ms is commonly used for frame movement. A voice signal of one frame is converted into a frequency domain using FFT. Divide the frequency band into several filter banks and find the energy in each bank. After taking the logarithm of the band energy, the final MFCC is obtained by performing discrete cosine transform (DCT).

The shape of the filter bank and the method of setting the center frequency are determined in consideration of the auditory characteristics of the ear (frequency characteristics in the cochlea). In Fig. 4, a triangular-shaped filter is used, and the center frequency is linearly located up to 1 kHz and consists of 20 banks distributed on a mel scale above that.

Meanwhile, the deep learning unit 30 is configured to perform deep learning through a combination of a Convolution Neural Network (CNN), a Long Short Term Memory Recurrent Neural Network (RNN), and an Attention mechanism using the feature data of the machine sound.

Recently, as sensing technology and computing power have improved, high-quality data can be acquired. Based on this, deep learning-based fault diagnosis research is being actively conducted. For example, convolutional neural networks based on vibration signals Network (CNN) training to try to diagnose electronic device failures, and research on performing fault diagnosis by learning Deep Belief Network (DBN) after imaging the vibration signal of a rotating body.

CNN, which is frequently used in troubleshooting technology, is the most used network for image analysis. However, in general, CNN is hierarchical, RNN is expressed as sequential, and RNN is considered to be superior in models that use sequence data such as natural language processing. Therefore, it can be seen that RNN is more suitable for the task of analyzing the signal obtained from the bearing, which is the sequence data, to determine whether there is a machine failure.

Also, in a factory environment where ambient noise may be mixed, the processing of the obtained signal requires longer-term judgment. Attention mechanism is an architecture that is being studied to perform this function.

In one study, the attention mechanism consists of an encoder and a decoder. In the encoder, the hidden states of the LSTM are made into a matrix, and in the decoder, important states among the hidden states are emphasized.

This method made it possible to focus on important output vectors over a certain distance, showing excellent performance in neural machine translation (NMT).

Therefore, the present invention proposes a model (CLA) that combines CNN, LSTM RNN, and attention mechanism. The CLA model utilizes the advantages of CNN, which is advantageous for local feature identification, LSTM RNN, which is advantageous for sequential data processing, and the attention mechanism, which allows you to focus on important information, to induce better performance than a model using simple CNN. do.

The CLA model used in the present invention is converted into an 80-dimensional mel spectrogram during the feature extraction process of the machine sound, and 300 framesets in units of 3 seconds are input to the model. The model extracts the spatial characteristics of the spectrum with the CNN layer, the temporal information with the LSTM layer, and then collects the information by concentrating the local characteristics with the attention layer.

In the output layer, it is designed to output as 0 or 1 to determine whether there is an abnormality. An embodiment of the CLA model is as shown in FIG. 5, and the specification of each layer is shown in [Table 1].

This CLA model calculates the final output ((1), (2), (3), (4) of FIG. 5, respectively) after going through the CNN layer, LSTM layer, and attention layer in order when an input is received. The final output is a value between 0 and 1, which is a probability value of whether the machine is defective.

The feature input to this model is an 80 mel-band spectrogram, which was created by moving a 20ms-sized window by 10ms. The audio from the machine is 3 seconds long and has a sampling rate of 16000 Hz. Therefore, the finally generated spectrogram has a time step of 300 frames, and the dimension of each frame is 80.

CNN layer is a layer that repeats convolution operation and max pooling twice. The CNN layer used in this model is a layer that analyzes the input spectrogram and works to extract features more suitable for fault detection.

This is done through convolution operation and max pooling, and the convolution operation is a process of extracting necessary features through operation from a local part of the input data. The Max pooling operation reduces the dimension of data while leaving only necessary features, and serves to lower the computational complexity of the LSTM and attention layer that will be performed later.

In the LSTM layer, LSTM is a type of Recurrent Neural Network (RNN) architecture, and in general, when referring to RNN, it is a structure that is used so much that it consists of an LSTM structure.

In this CLA model, it is used to obtain temporal information from the feature vectors obtained through the CNN layer. This is to discriminate the noise caused by the failure or the noise that can only be known when long-distance information is processed using the sequence information processing capability of the RNN.

In this CLA model, an LSTM having 64 cells was used as two layers, and data was input in the forward direction.

The attention layer is a layer created by utilizing the attention mechanism and is used to create a context vector for making a final judgment from the information vector obtained through CNN and LSTM.

The input of the attention mechanism is the output of the LSTM layer H=h ₁ , h ₂ , h ₃ , … , hT, hi=(p ₁ , p ₂ , …, p _C ) and calculate the output using the weight parameter q=(q ₁ , q ₂ , q ₃ , …, q _{C ).} In the above expression, T and C are the timestep length of the LSTM input and the number of cells in the LSTM, respectively, and the output o of the attention layer is as follows.

In the above equation, · represents the dot production operation between two vectors, and * represents the multiplication between a scalar value and a vector. The o obtained in the above equation is a vector of size C and is used as the input of the output layer.

The output layer is a fully connected layer (FCL) composed of one unit. It is a layer that converts the input vector into the final probability value. It is output as a value between 0 and 1 using the sigmoid function as the activation function. In the final judgment, if the value is over 0.5, it is judged that there is a malfunction in the machine where the input audio was recorded. If the value is less than 0.5, the input audio It was judged that there was no malfunction in the recorded machine.

[Example 1]

The data used in this experiment are CNC, CWRU, and WP datasets, respectively, and a noise dataset was additionally used.

-CNC

A CNC data set is a collection of recorded data from a CNC machine. The total number of data is 177, the number of normal sounds is 91, and the number of abnormal sounds is 86.

-CWRU

The CWRU dataset is

48000 samples per of Normal Baseline Data, 12k Drive End Bearing Fault Data, 48k Drive End Bearing Fault Data, and Fan-End Bearing Fault Data provided at https://csegroups.case.edu/bearingdatacenter/pages/download-data-file It is a data set created by excluding the 48k drive end bearing fault data recorded as the second.

The format of data provided by this site is in Matlab format for use in Matlab, and includes data obtained from the position of the motor fan and drive end.

Using the function provided by the scipy module of Python, a data file can be read and composed into a dictionary structure. The structure of the read dictionary is as follows.

dict_keys(['__header__', '__version__', '__globals__', 'X097_DE_time', 'X097_FE_time', 'X097RPM'])

The header, version, and globals keys of the data dictionary store the basic structure and creation date of the matlab file, and the DE_time and FE_time keys contain samples recorded at the drive end and fan positions of the motor, respectively. In this experiment, the data of the DE_time key was used.

The total number of data is 333, the number of normal sounds is 45, and the number of abnormal sounds is 288.

-WP

The WP dataset is a dataset created by recording the sound of a water pump. The normal sound is the sound of water being sprayed, and the abnormal sound is the sound of the spraying machine running without water being sprayed. The total number of data is 1124, the number of normal sounds is 907, and the number of abnormal sounds is 217.

-Noise

The noise dataset was added to the above three datasets and used to increase the diagnostic difficulty of the data, and the dataset consists of background sound downloaded from the freesound.org site.

The background sound is a total of 7 files, of which 4 files are the speech sounds of people obtained from restaurants, cafes, and phone calls. The other three files are files that record the sound of a vehicle on the street, and it consists of the sound of a car. Each file is about 2 to 5 minutes long.

In this experiment, we compare the performance between the model using only CNN and the CLA model. The CNN model consists of 4 layers of CNN, and 2 layers of FCL are used behind the CNN. Each data was randomly mixed with noise and then used for training and evaluation of CNN and CLA models. At this time, the ratio of training and evaluation data was set to 8:2, and it was repeated 100 epochs.

As performance indicators, precision, recall, and F1-score were used as shown below, and are calculated as follows.

True Positive (TP): Predicting the correct answer that is actually True as True (correct answer)

False Positive (FP): Predicting a true false answer as true (wrong answer)

False Negative (FN): Predict the correct answer that is true as False (wrong answer)

True Negative (TN): Predicting a correct answer that is actually False as False (correct answer)

[Table 2] shows the experimental results of the CLA model and the CNN model.

As can be seen in [Table 2] above, the CLA model shows better performance than the CNN model. This is considered to be because the existing CNN model, which does not use the attention mechanism and LSTM, is weak in analyzing data mixed with noise.

The ability to understand the long-term context of the attention mechanism and LSTM is because it has the ability to distinguish noise mixed in audio for a long time and to extract only important features to determine the presence or absence of a malfunction.

For this reason, the CLA model was able to get 2.81%, 0.73%, and 3.64% better performance than CNN for CNC, CWRU, and WP data sets mixed with noise, respectively.

In the present invention, the CLA model for determining the presence or absence of failure of various machines such as bearing failure and machine abnormality is applied to the deep learning unit 30 . The CLA model is a model that adds LSTM and Attention Mechanism that can identify long-term contexts based on CNN. It uses audio recordings obtained from an environment that determines whether a machine is malfunctioning in real time as an input.

At this time, noise such as human speech or other machine sounds may be mixed depending on the surrounding conditions of the machine. In the case of the existing CNN model, it may be difficult to distinguish the noise because the long-term ability to grasp the context is weak due to the nature of the CNN. The CLA model is a model that can expect good performance even in an environment in which noise is mixed by utilizing the long-term contextual ability of LSTM and attention mechanism.

After creating a dataset by mixing human speech and traffic sounds with CNC, CWRU, and WP data as noise, training and evaluation were performed, and it showed good performance compared to the CNN model. From this, it can be seen that LSTM and attention mechanism can help to distinguish unnecessary noise for machine diagnosis.

In addition, the deep learning-based abnormal symptom detection system 100 according to the present invention includes a maintenance input unit 50 that provides maintenance history information when maintenance is performed according to an abnormality or setting of mechanical equipment, such maintenance The maintenance input unit 50 may be configured so that an administrator who manages the mechanical equipment inputs the corresponding maintenance history information through a terminal connected to the feature extraction unit 20 or the deep learning unit 30 to enable data communication.

This is because, when the deep learning unit 30 according to the present invention performs deep learning, it is most efficient to perform deep learning based on machine sound characteristic data before and after maintenance history information.

Accordingly, it is preferable that deep learning be performed mainly on machine sound data before and after the maintenance through maintenance history information, rather than learning the entire machine sound data collected by the sensor 10 for data collection.

On the other hand, the deep learning unit 30 of the deep learning-based anomaly detection system 100 according to the present invention repeatedly cross-validates 1000 or more machine sound data in a 4-fold cross validation technique in the learning process as shown in FIG. carry out

With such repeated cross-validation, the deep learning model structure and parameters are continuously supplemented, so that the reliability of the system according to the present invention can be improved.

As described above, the present invention has been described with reference to one embodiment shown in the drawings, which is merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and equivalent other embodiments are possible therefrom. will understand

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

10: sensor for data collection 20: feature extraction unit
30: deep learning unit 40: abnormality detection unit
50: maintenance input unit
100: Deep learning-based anomaly detection system

Claims (6)

In the deep learning-based abnormal symptom detection system using the machine sound of mechanical equipment,
The deep learning-based anomaly detection system 100 is
a sensor 10 for collecting data for collecting mechanical sounds generated from one or more mechanical equipment;
a feature extracting unit 20 for receiving the machine sound collected from the data collection sensor 10 and extracting features of the corresponding machine sound;
A deep learning unit 30 configured to perform deep learning through a combination of a Convolution Neural Network (CNN), a Long Short Term Memory Recurrent Neural Network (LSTM RNN), and an Attention mechanism using machine sound feature data;
Machine equipment comprising an abnormality detection unit 40 that receives the feature data extracted from the feature extraction unit 20 and detects abnormal symptoms of the mechanical equipment through the deep learning unit 30 Deep learning-based anomaly detection system using machine sound.
According to claim 1,
The data collection sensor 10 is
Deep learning-based anomaly detection system using machine sound of mechanical equipment, characterized in that it is a micro-microphone that collects mechanical sounds generated from mechanical equipment.
According to claim 1,
The deep learning-based anomaly detection system 100 is
A maintenance input unit 50 that provides maintenance history information when maintenance is performed according to an abnormality or setting of mechanical equipment is further included,
The deep learning unit 30 is a deep learning-based abnormal symptom detection system using machine sound of machine equipment, characterized in that it performs deep learning through machine sound feature data before and after maintenance is performed.
According to claim 1,
The feature extraction unit 20
Using the mechanical sound of mechanical equipment, characterized in that receiving the machine sound collected from the data collection sensor 10, converting the machine sound into a frequency domain, and extracting features through noise removal and Mel Filter Bank Deep learning-based anomaly detection system.
5. The method of claim 4,
The parameters used for feature extraction in the feature extraction unit 20 are
A deep learning-based anomaly detection system using a machine sound of a mechanical facility, characterized in that it is at least one of frame window size, window shift rate, LLD (Low level descriptor) or HSF (High-level Statisticla Functions).
According to claim 1,
The deep learning-based anomaly detection system 100 is
Deep learning-based abnormal symptom detection system using machine sound of mechanical equipment, characterized in that it repeatedly cross-validates in the learning process of the deep learning unit 30 with a 4-fold cross validation technique with more than 1000 machine sound data.

Images