KR20210125366A - Method for detecting recording device failure using neural network classifier, server and smart device implementing the same - Google Patents

Abstract

The present invention relates to a method for a server to detect failure of a recording device, which detects failure of a recording device of a smart device by using a neural network classifier. According to the present invention, the method comprises: a step of receiving training audio signals for each failure type to extract at least one spectrogram and audio information for each training audio signal; a step of generating training data in which the at least one spectrogram and audio information for each training audio signal are mapped to a corresponding failure type to train a neural network classifier; and a step of when receiving an audio signal from the smart device, using the trained neural network classifier to detect and transfer the failure type of the audio signal to a smart device, wherein the trained neural network classifier includes: a first neural network model trained to output spectrogram feature values of the training audio signals by failure type; a second neural network model trained to output audio information feature values of the training audio signals for each failure type; and a classification model trained to output the failure type of the corresponding training audio signal by connecting the feature values output from the first neural network model and the second neural network model.

Description

Method for detecting recording device failure using neural network classifier, server and smart device implementing the same}

The present invention relates to a method for detecting a recording apparatus failure in a smart device equipped with a recording apparatus using a neural network classifier that determines the failure of the recording apparatus and notifies a user or a service provider, a server implementing the same, and a smart device .

Due to the development of technology, various products are equipped with a voice recognition function. The number of artificial intelligence speakers around the world alone has already exceeded 100 million. If you consider other IoT products, a huge number of voice recognition devices are already in use around us.

In such smart devices, the performance of the microphone leads to the accuracy of speech recognition. When the voice comes in clear, the smart device produces more accurate results. Therefore, we invest a lot of effort to improve voice quality, such as increasing the number of microphones in smart devices and using the latest voice processing technology.

Despite these efforts, if an abnormality such as a malfunction of the microphone or foreign material occurs, the smart device collects distorted voice. In this case, the smart device understands the sentence incorrectly.

However, it is difficult for the user to know whether the recording device is malfunctioning. This is because the user cannot directly hear the voice being recorded. In addition, the user has no way of knowing whether the cause of inaccurate voice recognition is a microphone, noise, or lack of performance of voice recognition software.

In addition, in the prior art, a specially produced reference signal is reproduced from a speaker of a smart device, and a malfunction is determined when a signal coming into the microphone of the smart device differs from the reference signal by a predetermined or more.

To this end, after the reference signal is reproduced from the speaker of the normal device, the signal coming into the microphone of the same device is recorded as a recording sample. After reproducing the reference signal in the device to be tested, compare the signal coming into the microphone with the recorded sample and if it differs by more than a certain amount, it is determined as a failure.

Such a prior art is difficult to use in a noisy environment because comparison must be made with reference signals. Therefore, in order to minimize the effect of noise or signal distortion, artificial sound such as a sine wave must be reproduced. And, since it only judges whether the signals are similar or not, there is a problem that it is impossible to determine what type of failure it is, and since the signals between microphones must be compared, it is impossible to apply to a smart device to which a single recording device has no comparison target. do.

Therefore, the present invention detects a recording device failure using a neural network classifier that detects a failure type of an audio signal collected by a smart device using a neural network classifier learned to output a failure type using audio signals for each failure type. A method, a server implementing the same, and a smart device are provided.

As a method of detecting a failure of a recording apparatus of a smart device by a server, which is a feature of the present invention for achieving the technical problem of the present invention,

receiving learning audio signals for each failure type, extracting at least one spectrogram and audio information for each learning audio signal, respectively, mapping at least one spectrogram and audio information for each learning audio signal to a corresponding failure type Generating one training data, training a neural network classifier using the training data, and receiving an audio signal from the smart device, using the learned neural network classifier to detect a failure type of the audio signal, and transmitting to a device, wherein the learned neural network classifier outputs a first neural network model trained to output spectrogram feature values of the learning audio signals for each failure type, and audio information feature values of the learning audio signals for each failure type. a second neural network model trained to do so, and a classification model trained to output the failure type of the corresponding learning audio signal by connecting the feature values output from the first neural network model and the second neural network model.

The first neural network model may be a convolutional recurrent neural network (CRNN)-based model, and the second neural network model may be a recurrent neural network (RNN)-based model.

The failure type may include at least one of a normal type, an instantaneous failure type, or a persistent failure type.

The step of extracting each of the spectrogram and audio information includes extracting a plurality of time section signals from each training audio signal, converting each time section signal into a frequency domain, and converting the spectrogram and audio information of the corresponding time section signal into a frequency domain. and extracting, wherein each of the plurality of time interval signals may be extracted such that at least a portion of the time interval overlaps with a previous time interval or a subsequent time interval.

As another feature of the present invention for achieving the technical problem of the present invention, a smart device including a recording apparatus is a method of detecting a failure of the recording apparatus,

receiving an audio signal collected from a recording device, extracting at least one spectrogram and audio information for the audio signal, and learning the spectrogram and audio information of the audio signal as learning audio signals for each failure type and inputting it into an established neural network model, and detecting a failure type of the audio signal through the neural network model.

Before receiving the audio signal, receiving audio signals for learning by failure type, extracting at least one spectrogram and audio information for each audio signal for learning, respectively, at least one spectrogram for each audio signal for learning Generating training data by mapping gram and audio information to a corresponding failure type, and training a neural network classifier using the training data, wherein the training of the neural network classifier includes: when the spectrogram is input, the spectrogram is input training a first neural network model included in the neural network classifier to output a gram feature value, training a second neural network model included in the neural network classifier to output an audio information feature value when the audio information is input, and Learning the classification model to output the failure type of the input audio signal based on the probability value calculated by connecting the spectrogram feature value output from the first neural network model and the audio information feature value output from the second neural network model may include more.

Prior to receiving the audio signal, the method may further include receiving, from a server interworking with the smart device, a neural network model learned from the server.

The step of extracting the spectrogram and audio information may include extracting a plurality of time section signals from the received audio signal, converting each time section signal into a frequency domain, and providing a spectrogram and audio information of the corresponding time section signal and extracting the signals, wherein each of the plurality of time interval signals may be extracted such that at least a portion of the time interval overlaps with a previous time interval or a subsequent time interval.

In the training of the classification model, the spectrogram feature value and the audio information feature value may be completely concatenated to generate a connectivity map, and an activation function may be applied to the generated connectivity map to calculate the probability value.

If there are a plurality of recording devices, the acquiring the failure type may include: a spectrogram and audio information of a first audio signal collected by a first recording device among the plurality of recording devices, and a second recording device collected by a second recording device. 2 extracting a spectrogram of an audio signal and audio information, and a difference value between the first audio signal and the second audio signal as input information, and inputting the input information into the learned neural network model, and the neural network model It may include obtaining the failure types of the first audio signal and the second audio signal from

The step of extracting the input information may include: generating a first frequency domain value and a second frequency domain value from the first audio signal and the second audio signal; calculating a first difference value, and n-squaring the first audio signal and the second audio signal, and calculating a second difference value from respective frequency domain values of the n-squared first audio signal and the second audio signal .

As a server for detecting a failure of a recording apparatus included in a smart device, which is another feature of the present invention for achieving the technical problem of the present invention,

A memory including at least one instruction and storing audio signals for learning for each failure type, an interface for receiving an audio signal from the smart device, and a processor, wherein the processor includes a spectrogram and audio signal from the audio signal for learning for each failure type. Information is extracted, a neural network classifier is trained using the spectrogram and audio information, and a failure type of the audio signal is detected by inputting the received audio signal to the learned neural network classifier.

The processor may extract a plurality of section signals to overlap a predetermined section from the received audio signal, and convert the extracted section signals into frequency domain values to extract the plurality of audio information.

If there are a plurality of recording apparatuses of the smart device, the processor may calculate a difference value between audio signals each collected by the plurality of recording apparatuses.

The processor may generate a connectivity map by performing full concatenation operation processing on the spectrogram feature value and the audio information feature value, and may calculate the probability value by applying an activation function to the generated connectivity map.

According to the present invention, since it is possible to determine the failure of the recording device without the need to reproduce a specially prepared reference signal, it is possible to continuously monitor the failure of the recording apparatus without playing music or special manipulation through the user's voice. can

In addition, it is possible to detect not only the failure of the recording device, but also what type of failure has occurred, so that the service provider can receive the corresponding sound data to analyze the problem in more detail.

In addition, since it is applicable regardless of the number of recording devices, it is possible to cope with the failure of the recording devices even if all recording devices fail equally and receive the same signal.

In addition, more accurate detection is possible by using additional spectral characteristics and differential values in addition to Mel spectrogram or voice feature data commonly used in existing voice classifiers.

Considering Mel spectrogram and other characteristics, using a proprietary neural network structure that divides into CRNN (Convolutional Recurrent Neural Network) and RNN, more accurate results can be obtained with less computation.

1 is an exemplary diagram of an environment to which a neural network classifier for detecting a failure of a recording apparatus according to an embodiment of the present invention is applied.
2 is a flowchart illustrating a method for a server to detect a failure of a recording device according to the first embodiment of the present invention.
3 is a flowchart of a method for a smart device equipped with a single microphone to detect a failure of a recording apparatus according to a second embodiment of the present invention.
4 is an exemplary diagram of an audio frame according to an embodiment of the present invention.
5 is an exemplary diagram of a Mel spectrogram according to an embodiment of the present invention.
6 is an exemplary diagram of a neural network classifier according to an embodiment of the present invention.
7 is a flowchart of a method for a smart device equipped with a plurality of microphones according to an embodiment of the present invention to self-detect whether a recording apparatus fails.
8 is an exemplary diagram illustrating signals of two recording devices for differential value extraction according to an embodiment of the present invention.
9 is another exemplary diagram of a neural network classifier according to an embodiment of the present invention.
10 is an exemplary diagram of a graph in which a probability value for each failure type outputted according to an embodiment of the present invention is visualized.
11 is a structural diagram of a smart device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Hereinafter, with reference to the drawings, a server, a smart device, a method for learning a neural network classifier using the same, and a method for self-detection of a recording device failure according to an embodiment of the present invention will be described.

In the first embodiment of the present invention, the smart device 100 trains the neural network classifier with an audio signal for learning for each failure type in order to detect whether the recording apparatus 200 fails. Then, whether the audio signal received by the recording apparatus 200 has a failure and a failure type is detected with the learned neural network classifier.

In addition, in the second embodiment of the present invention, the server 300 interworking with the smart device 100 trains the neural network classifier with the learning audio signal for each failure type. And after detecting the failure and failure type of the audio signal received from the smart device 100 , the result is provided to the smart device 100 .

An environment in which a neural network classifier detects a failure of a recording apparatus according to such an embodiment will be described with reference to FIG. 1 .

1 is an exemplary diagram of an environment to which a neural network classifier for detecting a failure of a recording apparatus according to an embodiment of the present invention is applied.

As shown in FIG. 1 , the smart device 100 includes one recording device 200 or a plurality of recording devices 200 ′. In the embodiment of the present invention, a microphone is used as the recording device 200 as an example, but any means capable of collecting sound may be substituted.

The recording apparatus 200 collects audio signals such as a sound or sound source generated through a speaker (not shown) provided in the smart device 100 or an external sound source, voice, and noise. In the embodiment of the present invention, it is illustrated that only the recording apparatus 200 is included in the smart device 100 for convenience of explanation, but various configurations for providing a service to the user through the smart device 100 may be included.

The smart device 100 transmits the audio signal collected by the recording apparatus 200 to the server 300 according to the first embodiment, and requests to detect whether or not the recording apparatus 200 malfunctions and the failure type. After the server 300 detects the failure and the failure type of the recording apparatus 200 that has collected the audio signal using the learned neural network classifier, the result is provided to the smart device 100 .

In addition, the smart device 100 extracts input information including a Mel spectrogram and audio information from the audio signal collected by the recording apparatus 200 according to the second embodiment, and uses the extracted input information to perform a learned neural network classifier. Use to check if there is an error. And when it is determined that there is an abnormality in the input information, the smart device 100 monitors the occurrence frequency of the abnormality.

When the smart device 100 determines that there is an abnormality in the sound with a preset threshold frequency or more, it is determined that there is an abnormality in the recording apparatus 200 that has collected the sound. Then, the smart device 100 notifies the service provider or the user who uses the smart device 100 that there is a malfunction in the recording apparatus 200 .

At this time, the server 300 according to the first embodiment and the smart device 100 according to the second embodiment separate standards within the smart device 100 or the server 300 to detect whether an audio signal is abnormal. It does not generate a signal. In addition, the smart device 100 or the server 300 may classify the audio information of the received audio signal through a neural network classifier to determine whether there is an abnormality and check the frequency of occurrence of the abnormality to determine whether there is a failure.

Therefore, since it is possible to determine the failure of the recording device without the need to reproduce a specially prepared reference signal, it is possible to continuously monitor the failure of the recording apparatus without special manipulation through sounds such as music or voices uttered by the user. can do. In addition, it is possible to detect not only whether the recording device 200 has failed, but also what type of failure has occurred, so that the service provider can receive the corresponding sound data to analyze the problem in more detail.

Here, one recording apparatus 200 may be provided in the smart device 100 as shown in (a) of FIG. 1 . In addition, as shown in (b) of Figure 1 may be provided in plurality.

If a plurality of recording apparatuses 200' are provided in the smart device 100' as shown in FIG. Two recording devices 200' are selected in a preset order or randomly. In addition, it is detected whether at least one of the plurality of recording apparatuses 200' included in the smart device 100' has a failure in the recording apparatus. This will be described in detail later.

In an embodiment of the present invention, the neural network classifier is trained in the smart device 100 equipped with one or a plurality of recording devices, and the collected audio signal is input to the learned neural network classifier to determine the failure type of the recording device in which the audio signal is collected. can be discerned. In addition, in the embodiment of the present invention, the server 300 trains the neural network classifier, and receives the audio signal from the smart device 100 to determine the failure type of the recording device that collects the audio signal through the learned neural network classifier. have. In addition, in the embodiment of the present invention, the server 300 trains the neural network classifier and transmits it to the smart device 100 , collects audio signals from the smart device 100 and records them through the neural network classifier learned in the server 300 . It is possible to determine the type of failure of the device.

In the above environment, a method in which the server 300 trains the neural network classifier and detects a failure of the recording apparatus according to the first embodiment of the present invention will be described with reference to FIG. 2 .

2 is a flowchart illustrating a method for a server to detect a failure of a recording device according to the first embodiment of the present invention.

As shown in FIG. 2 , the server 300 extracts at least one spectrogram and audio information from the training audio signal when receiving the training audio signals collected for each failure type as input ( S100 ). Here, the form in which the spectrogram and audio information are mapped to the corresponding failure type is referred to as learning data, and the method of extracting the learning data from the audio signal for learning will be described later when a method of detecting a failure of the recording device is described. .

The server 300 inputs the spectrogram extracted in step S100 to the first neural network model constituting the neural network classifier, and trains the first neural network model to output spectrogram feature values (S101). Then, the server 300 inputs the audio information extracted in step S100 to the second neural network model constituting the neural network classifier, and trains the second neural network model to output audio information feature values (S102). The server 300 connects the spectrogram feature value and the audio information feature value to learn the classification model constituting the neural network classifier so that the failure type of the corresponding learning audio signal is output ( S103 ).

In this way, when the server 300 learns the neural network classifier, the smart device 100 receives an audio signal using the recording apparatus 200 (S104), and transmits the received audio signal to the server 300 for recording. A request is made to detect whether the device 200 has a failure and a failure type (S105).

When the server 300 receives the audio signal from the smart device 100 in step S105, it extracts the spectrogram and audio information as input information from the audio signal (S106). In order to extract the input information, when the audio signal for learning is input, the server 300 extracts a time period signal of a certain period, that is, an audio frame.

Although the embodiment of the present invention describes extracting an audio frame for convenience of description, it may also be expressed as extracting an overlapping time period signal so that the audio signal overlaps a predetermined time period.

In the exemplary embodiment of the present invention, an audio signal having a length of 30 ms is extracted as an audio frame every 10 ms. That is, one audio signal having a length of 30 ms may be extracted as three audio frames. In the embodiment of the present invention, the length of the audio signal is defined as 30 ms and the length of one audio frame is defined as 10 ms, but the present invention is not limited thereto.

Then, the server 300 applies a windowing function to the extracted audio frames. The windowing function is a technique of smoothing equally spaced data, ie, audio frames, using a preset weight (window function). If a windowing function is applied to reduce the sudden blocking effect of the window, the server 300 may obtain the window size and some overlapping audio frames.

In the embodiment of the present invention, the application of window hanning among windowing functions in the process of acquiring audio frames will be described as an example. However, various window functions such as Hamming, Blackman, and Gaussian may be used, and the window function is not limited to any one.

After obtaining audio frames by applying a windowing function, the server 300 performs Fast Fourier Transform (FFT) on the audio frames. Based on the fast Fourier transform, the server 300 converts an audio frame in a time domain represented by time and volume into an audio frame in a frequency domain represented by frequency and volume. A method for the server 300 to extract an audio frame at a predetermined interval from an audio signal or to perform a fast Fourier transform is already known, and a detailed description thereof will be omitted in the embodiment of the present invention.

The server 300 measures the signal strength of the audio signal by obtaining the sum of the areas of the audio frames converted into the frequency domain. A method of measuring the signal strength of an audio signal from the sum of the areas of the audio frames is already known, and detailed description thereof will be omitted in the exemplary embodiment of the present invention.

And only when the measured signal strength is greater than the preset signal strength, the corresponding audio frame is used to detect a failure. A method for the server 300 to obtain the area sum of audio frames in the frequency domain is already known, and a detailed description thereof will be omitted in the embodiment of the present invention.

The server 300 extracts a Mel-scaled spectrogram and audio information by using the audio frames. In an embodiment of the present invention, as audio information extracted by the server 300, the spectral centroid, spectral spread, spectral skewness, and spectral kurtosis confirming the overall sound pitch (spectral centroid) Spectral kurtosis, spectral decrease, spectral flux indicating whether sound changes are severe or constant, spectral roll-off, pitch, etc. It is not limited to such audio information.

In order for the server 300 to extract each piece of audio information, the following equations are used.

Here, f(n) is the nth frequency value, and x(n) is the nth FFT value. And x(T-1)(n) means the FFT value of the previous frame.

In addition, the server 300 uses a pitch detection algorithm such as a YIN algorithm or an MPM algorithm to extract a pitch value. The pitch detection algorithm is already known, and detailed description thereof will be omitted in the embodiment of the present invention.

When the Mel spectrogram and audio information are extracted through the above procedure, the server 300 inputs the Mel spectrogram and audio information to the neural network classifier, and whether there is an abnormality in the audio signal received from the smart device 100 in step S105. to confirm (S107).

If it is confirmed that there is an abnormality in the audio signal, it is confirmed that there is an abnormality in the recording device 200 in which the sound is recorded. Accordingly, the server 300 classifies the failure type of the audio signal according to the failure type definition of the recording device through the learned neural network classifier.

Here, the server 300 inputs a Mel spectrogram having a correlation between adjacent filter banks to the learned first neural network model and outputs a Mel spectrogram feature to check whether there is an abnormality. In addition, audio information having a low correlation between adjacent filter banks is input to the learned second neural network model.

The first neural network according to an embodiment of the present invention uses a convolution recurrent neural network (CRNN), and the second neural network uses a recurrent neural network (RNN)-based model of a Gated Recurrent Unit (GRU). It will be described using an example.

The server 300 uses the Mel spectrogram and audio information extracted as input information in step S106 to determine the failure type of the recording device 200 in advance to determine whether there is an abnormality in the recording device 200 as shown in Table 1 below. define it

failure type Justice 0 normal One momentary noise 2 momentary signal missing 3 persistent additive noise 4 Missing persistence signal 5 Missing certain frequency bands 6 Poor microphone sealing 7 bad microphone circuit

After checking and classifying whether the audio signal is abnormal, the server 300 checks the frequency of abnormality classified as having an abnormality in the audio signal (S108). And if the number of times of checking that there is an abnormality in the audio signal is greater than or equal to the preset number, the server 300 notifies the smart device 100 of the failure of the recording apparatus and the type of failure (S109). The smart device 100 notifies a user using the smart device 100 or a service provider who provides a service through the smart device 100 of the failure and the failure type received from the server 300 of the failure occurrence (S110). ).

In the above, the first embodiment has been described in which the server 300 trains the neural network classifier, and notifies whether the recording apparatus 200 has a failure and the failure type using the learned neural network classifier. However, it is also possible to train the neural network classifier and to determine whether or not the recording apparatus 200 fails and the type of failure in the smart device 100 . This will be described with reference to FIG. 3 .

3 is a flowchart of a method for a smart device equipped with a single microphone to detect a failure of a recording apparatus according to a second embodiment of the present invention.

Here, since the detailed method of each procedure is the same as the method described with reference to FIG. 2 , detailed description thereof will be omitted in FIG. 3 .

As shown in FIG. 3 , a spectrogram and audio information are extracted from an audio signal for learning input from the outside ( S200 ), and a neural network classifier is trained using the extracted spectrogram and audio information ( S210 ).

The smart device 100 collects audio signals through one or a plurality of recording devices 200 (S220). Here, the audio signal corresponds to the voice or music reproduced by the smart device 100 , or external noise or the user's voice.

The smart device 100 extracts the input information of the audio signal from the audio signal collected in step S220 (S230). Then, the extracted input information is input to the neural network classifier learned in step S210, and the failure type and failure type of the recording device 200 that has collected the audio signal are classified (S240).

The smart device 100 checks and classifies whether the audio signal is abnormal through step S240, and then checks the frequency of abnormalities classified as abnormal in the audio signal (S250). And, if the number of times that the audio signal is confirmed to be abnormal is greater than or equal to the preset number, the smart device 100 determines that a failure has occurred in the recording apparatus 200 . Then, the user using the smart device 100 or a service provider providing a service through the smart device 100 is notified of the occurrence of a failure ( S260 ).

Through the above procedure, when the server 300 or the smart device 100 extracts the input information of the audio signal during the process of self-diagnosing whether the recording apparatus 200 is malfunctioning, an example of an audio frame extracted from the audio signal is also shown. 4 will be described with reference. In the embodiment of the present invention, it is expressed that an 'audio frame' is extracted for convenience of description, but it may also be expressed as extracting sections from an audio signal. And for convenience of explanation, it is shown that the audio frame is extracted from the server 300 , but may also be extracted from the smart device 100 .

4 is an exemplary diagram of an audio frame according to an embodiment of the present invention.

As shown in FIG. 4 , in order to analyze the audio signal, when the server 300 receives the audio signal, it extracts a plurality of audio frames at predetermined intervals. At this time, in the embodiment of the present invention, an audio signal having a length of 30 ms is extracted as an audio frame every 10 ms as an example.

In the first audio signal collected by the server 300 at the first time point (t=1) and the second audio signal collected at the second time point (t=2) immediately after the first time point, a certain portion of the audio frames overlap. do. This is to reduce the sudden blocking effect of the window by acquiring audio frames so that the window size partially overlaps.

Next, a Mel spectrogram among input information of an audio signal will be described with reference to FIG. 5 .

5 is an exemplary diagram of a Mel spectrogram according to an embodiment of the present invention.

As shown in FIG. 5 , the server 300 extracts a 40th-order Mel spectrogram using the result of fast Fourier transforming the audio signal. The x-axis of the Mel spectrogram shown in FIG. 5 is the time axis, and the y-axis is the frequency axis.

At this time, taking the nth audio frame composed of three frames as an example, 15 frames (n-15th frame to n-1th frame) preceding the nth time, which is the start position of the nth audio frame, 1 in the nth audio frame From the frame (n+1th frame), the 15th frames (n+2th frame to n+16th frame) are connected. This is an exemplary diagram shown in FIG. 5 to obtain and visualize a 31*40 two-dimensional matrix, which is a Mel spectrogram of a total of 31 frames (n-15th frame to n+16th frame).

Next, in an embodiment of the present invention, when the audio signal is collected from the smart device 100 provided with one recording apparatus 200 and the server 300 checks whether the audio signal is abnormal, the An implemented neural network classifier will be described with reference to FIG. 6 .

6 is an exemplary diagram of a neural network classifier according to an embodiment of the present invention.

As shown in FIG. 6 , the neural network classifier includes an input layer 110 including a first input layer 111 receiving a Mel spectrogram as an input and a second input layer 112 receiving audio information as an input.

The Mel spectrogram input to the first input layer 111 is input to the first neural network model 120 trained by the artificial intelligence algorithm, and the audio information input to the second input layer 112 is learned by the artificial intelligence algorithm. It is input to the second neural network model 130 . Here, the first neural network model 120 is a convolutional recurrent neural network (CRNN)-based model, and the second neural network model 130 is a recurrent neural network (RNN)-based model.

According to the input information input to the input layer 110 of the neural network classifier in the embodiment of the present invention, the characteristic values are output by passing through the appropriate neural network models 120 and 130 . A connection map is generated by connecting the output feature values in the classification model 140 which is a connection layer.

Then, the probability value is calculated using the activation function, softmax, in the connection map. In this case, the probability values are calculated for each preset failure type. Here, the classification model 140, which is a connected layer, will be described as an example that is implemented as a fully connected layer.

In FIG. 6 , numbers in parentheses indicate the number of nodes of the corresponding layer. For example, CNN 5*5(8) of the first neural network model 120 means that one Mel spectrogram is processed using 8 CNN layers using 5*5 filters, respectively.

Each of the layers of the neural network classifier will be described. When the input layer 110 receives the Mel spectrogram and audio information, it normalizes the Mel spectrogram and the audio information using a batch normalization algorithm. The batch normalization is to adjust the activation values in each layer to be appropriately distributed, and the batch normalization method is already known, and a detailed description thereof will be omitted in the embodiment of the present invention.

The convolutional recursive neural network layer corresponding to the first neural network model 120 includes a convolutional product layer, an activation layer, and a pooling layer. The convolutional product layer applies the convolutional product operation to the batch normalized Mel spectrogram. The convolution operation corresponds to the filter operation referred to in image processing. It moves the filter window at regular intervals and multiplies the input data, the batch normalized Mel spectrogram, with the elements corresponding to the filter to obtain the total sum.

The activation layer applies an activation function to the Mel spectrogram to which the composite product operation is applied. Here, the use of the hyperbolic tangent function (tanh(x)) as the activation function is described as an example, but is not necessarily limited thereto.

The pooling layer reduces the vertical and horizontal space in the frequency domain of the Mel spectrogram to which the activation sum is applied. To this end, in the embodiment of the present invention, the size of the Mel spectrogram is reduced by using max pooling.

In the embodiment of the present invention, the first neural network model 120 is described using a convolutional recursive neural network structure as an example, and a recurrent neural network structure such as RNN and LSTM is used instead of GRU included in the first neural network model 120. can In addition, the detailed number of nodes or the number of layers of the first neural network model 120 is not limited to the number shown in FIG. 6 and may be changed.

The recurrent neural network layer corresponding to the second neural network model 130 uses a GRU cell that does not have a vanishing gradient problem and has a small amount of computation. The details of the GRU cell are already known, and detailed description will be omitted in the embodiment of the present invention.

In the embodiment of the present invention, the second neural network model 130 will be described using an RNN structure as an example, and a recurrent neural network structure such as RNN and LSTM may be used instead of GRU. In addition, the detailed number of nodes or the number of layers of the second neural network model 130 is not limited to the number shown in FIG. 6 and may be changed.

In addition, the above-mentioned neural network classifier is trained by using the sound collected in advance by the failed recording device 200 as data input, and the failure type as a one-hot-vector output. The learned neural network classifier outputs probability values of predefined failure types for every frame.

In the above, an example of detecting whether a malfunction occurs when a single microphone is provided in the smart device 100 has been described. On the other hand, when the smart device 100' is provided with a plurality of microphones, another embodiment in which the smart device 100' detects a failure will be described with reference to FIG. 7 . 7, the neural network classifier of the smart device 100' provided with a plurality of recording apparatuses 200' is learned by failure type in the server 300 as described in FIG. 2, or the smart device ( 100') can be learned. In the embodiment of the present invention, for convenience of explanation, a neural network classifier trained in the smart device 100' will be described as an example.

7 is a flowchart of a method for a smart device equipped with a plurality of microphones according to an embodiment of the present invention to self-detect whether a recording apparatus fails.

7, the smart device 100' extracts the spectrogram and audio information from the learning audio signal input from the outside (S300), and uses the extracted spectrogram and audio information to learn the neural network classifier ( S310).

The smart device 100 collects audio signals through a plurality of recording devices 200 . At this time, audio signals are respectively collected through two recording apparatuses selected from among the plurality of recording apparatuses 200' (S320).

Here, the smart device 100' may select two recording apparatuses from among the plurality of recording apparatuses 200' in a preset order, or may randomly select two recording apparatuses. And, since there are a plurality of recording apparatuses 200', the smart device 100' includes identification information of each recording apparatus 200' in the audio signal extracted from the collected sound, and collects it.

The smart device 100' extracts input information from the two audio signals collected in step S320, respectively (S330). Here, the input information includes spectrograms of two audio signals and audio information. In order for the smart device 100' to extract input information from the two audio signals, two audio signals each obtained from two sounds are cut into frame units at regular intervals and extracted as audio frames.

In the exemplary embodiment of the present invention, two audio signals having a length of 30 ms are extracted as an audio frame every 10 ms as an example. Accordingly, although one frame is defined as 10 ms in the embodiment of the present invention, it is not necessarily limited to this.

Then, the smart device 100' applies windowing to the extracted audio frames, respectively. After obtaining audio frames from two audio signals by applying a windowing function to each, the smart device 100 ′ performs Fast Fourier Transform (FFT) on the audio frames in a time domain expressed in time and volume. ) is converted into an audio frame in the frequency domain expressed in terms of frequency and volume.

The smart device 100' obtains the sum of the areas of the two audio frames converted into the frequency domain, and measures the signal strength of the two audio signals, respectively. And only when the measured signal strength is greater than the preset signal strength, the corresponding audio frame is used to detect a failure.

The smart device 100 ′ extracts a Mel-scaled spectrogram and audio information by using the audio frames. The audio information extracted by the smart device 100 ′ is the same as the audio information described in FIG. 2 . Then, two difference measures between the two recording apparatuses 200' are calculated (S340).

When the Mel spectrogram and audio information are extracted through the above procedure and the differential value is calculated, the smart device 100 ′ checks whether there is an abnormality in the audio signals collected in step S320 through the neural network classifier. And if it is confirmed that there is an abnormality in the audio signals, it is confirmed that there is an abnormality in any one of the two recording devices 200 ′ that have collected the audio signals.

In the embodiment of the present invention, only detecting whether there is an abnormality in at least one of the plurality of recording apparatuses 200' included in the smart device 100' is described as an example. However, after separately detecting whether each of the two recording apparatuses 200 ′ having compared the two audio frames has failed, it is also possible to check which error occurred in which recording apparatus.

Accordingly, the smart device 100' determines that a failure has occurred in any one of the two recording apparatuses 200' based on the differential value between the two recording apparatuses 200'. Then, according to the predefined failure type definition of the recording device, the failure type of the audio signal collected in step S320 is classified (S350).

Here, the smart device 100 ′ learns a Mel spectrogram having a correlation between adjacent filter banks through the first neural network model and checks whether there is an abnormality. In addition, audio information having a low correlation between adjacent filter banks is learned through the second neural network model.

At this time, the smart device 100' uses the Mel spectrogram and audio information extracted as input information in step S330, and the differential value of the recording apparatuses 200' calculated in step S340, to cause abnormalities in the recording apparatus 200. The failure types for determining whether there are are as defined in Table 1 above.

The smart device 100' checks whether the audio signal is abnormal through step S350 and classifies it, and then checks the frequency of the abnormality (S360). And if the number of times that the audio signal is confirmed to be abnormal is greater than or equal to a preset frequency, it is determined that a failure has occurred in the recording apparatus 200 ', and the service is provided through the user or the smart device 100 using the smart device 100'. A failure occurrence is notified to the providing service provider (S370).

When the smart device 100' self-diagnoses whether or not the recording apparatus 200' is malfunctioning through the above procedure, the smart device 100' determines two differential values between the two selected recording apparatuses 200'. must be extracted This will be described with reference to FIG. 8 .

8 is an exemplary diagram illustrating signals of two recording devices for differential value extraction according to an embodiment of the present invention.

FIG. 8(a) shows audio signals of sound collected by two recording devices, and FIG. 8(b) is an exemplary view showing audio signals checked by n squared on the audio signals.

The smart device 100 ′ may obtain a difference value between the first audio signal of the sound collected by the first recording apparatus and the second audio signal of the sound collected by the second recording apparatus by calculating the area between the two signals. Even when the recording device is in a normal state, the two audio signals show some difference according to the location where the recording device is installed, as shown in (a) of FIG. 8 .

The difference between the audio signals tends to be proportional to the amplitude of the audio signal. Therefore, as shown in Equations 8 and 9, the smart device 100 divides the difference in magnitude (energy) between the two signals by the sum of the total signal magnitudes and normalizes it to obtain a difference value.

In an embodiment of the present invention, the smart device 100 obtains two differential values. When at least one of the two recording apparatuses 200' has an abnormality, a difference occurs between signals entering each recording apparatus 200', and the difference value increases. The normal graph shown in FIG. 8(a) has almost the same bottom area, but as shown in FIG. 8(b), when one of the two recording devices 200' is broken, the area difference is large. You can increase the accuracy of the neural network by using these changes as additional information input to the neural network.

That is, the first difference value of the first audio signal and the second audio signal is obtained by using Equation 8, and the second difference value is obtained by using Equation 9 by squaring the first audio signal and the second audio signal. . However, it is not necessarily limited in this way.

Next, a neural network classifier for determining whether a smart device 100 ′ having a plurality of recording devices 200 ′ is abnormal in an audio signal will be described with reference to FIG. 9 .

9 is another exemplary diagram of a neural network classifier according to an embodiment of the present invention.

In the embodiment of the present invention, even if a plurality of recording apparatuses are provided, an example in which the smart device 100 selects two recording apparatuses and checks whether an audio signal is abnormal will be described as an example. Accordingly, the neural network structure for checking whether two audio signals are abnormal is shown in FIG. 8 .

As shown in FIG. 9 , the neural network classifier receives a Mel spectrogram and audio information through an input layer 110 ′. At this time, since there are a plurality of recording apparatuses 200', the input layer 110' receives processing information obtained by processing the audio signal of the sound acquired by the plurality of recording apparatuses 200', respectively. In addition, two differential values calculated through the two audio signals are also received as inputs.

The function of each layer is the same as described with reference to FIG. 5 . That is, the Mel spectrogram input to the input unit 110' is learned by an artificial intelligence algorithm and is input to the first neural network model 120', which is a model based on a convolutional recursive neural network. Then, the audio information and the differential values are learned by an artificial intelligence algorithm and are input to the second neural network model 130 ′, which is a recursive neural network-based model.

According to the input information input to the input layer 110' of the neural network classifier, the appropriate neural network models 120' and 130' are passed through the embodiment of the present invention, and feature values according to each input information are output. Then, one connection map is generated by connecting the feature values output from the neural network models 120' and 130' in the classification model 140', which is a connection layer.

The probability value is calculated using the activation function softmax in the connection map. In this case, the probability values are calculated for each preset failure type. Here, the classification model 140 ′ will be described as an example implemented as a fully connected layer.

At this time, a graph visualizing the probability values output by the neural network classifier will be described with reference to FIG. 10 .

10 is an exemplary diagram of a graph visualizing a probability value for each type of failure outputted according to an embodiment of the present invention.

In FIG. 10, among the eight failure types described in Table 1, the probability values for normal and the probability values for instantaneous failure and persistent failure are visualized and the probability values are calculated for each of the eight failure types.

When an input audio signal is classified using a neural network classifier, error detection results are generated at intervals of a predetermined period (10 ms). In the neural network classifier, probability values are output for each failure type, and when visualized so as to see changes over time, as shown in FIG. 9 .

10 shows probability values for each failure type for one recording device 200 . However, when a plurality of recording apparatuses 200' are used, as many graphs as the number of recording apparatuses 200' are generated.

Here, the sum of the normal probability value, the persistent failure probability value, and the instantaneous failure probability value at an arbitrary point in time becomes 1. And, as indicated by the dotted line, the probability value in the normal case is approximately 0.1, but when the probability value of the instantaneous failure is 0.9, the smart device 100 indicates that the instantaneous failure occurred in the recording apparatus 200 at that time. can be judged as

That is, the probability values of instantaneous failure and persistent failure are significantly different. Accordingly, different failure discrimination criteria are required according to the types of instantaneous and persistent failures.

In the case of a momentary failure, an error occurs intermittently only for a very short period of time. To detect this, the smart device 100 searches for a failure type having a maximum value among the probability values calculated by the neural network classifier. If the failure type having the maximum value is an instantaneous failure, it is determined that a failure of the failure type has occurred.

In order to prevent erroneous detection, the smart device 100 notifies as a failure only when the same abnormality occurs over a certain number of times for a long time. For example, when an abnormality occurs in excess of 10 frames for 30 minutes, the recording apparatus 200 is notified that there is a failure.

On the other hand, in the case of continuous failure, abnormal symptoms occur continuously. In general, even when the recording apparatus 200 is normal, the probability value of the failure type may be calculated instantaneously. When all of these cases are detected as failures, erroneous detection may be performed even though the recording apparatus 200 is normal.

In order to prevent this, when an abnormality occurs continuously for a certain period of time, it is detected as a failure. For example, if it is recognized that an error has occurred in the audio signal continuously for 5 minutes or more, the recording apparatus 200 is notified that the continuous failure has occurred.

In this way, if the smart device 100 self-detects a failure of the recording apparatus 200 of the smart device 100, if the recording apparatus 200 is defective, it can be induced to immediately notify the customer and take action. And, it is possible to know the defect of the recording apparatus 200 without collecting the smart device 100 .

In addition, it is possible to detect a defect in the recording apparatus 200 through a general sound, such as a sound output from the smart device 100 or a user's voice, without reproducing a special signal. And, by clearly identifying the type of failure of the detected recording apparatus 200 , it can be used to improve the smart device 100 .

11 is a structural diagram of a computer system according to an embodiment of the present invention.

Referring to FIG. 11 , in the computer system 400 operated by at least one processor, a program including instructions described for executing the operation of the present invention is executed. The program may be stored in a computer-readable storage medium and may be distributed. Here, the structure of the computer system 400 may be the structure of the smart device 100 according to the embodiment of the present invention or the structure of the server 300 .

Hardware of the computer system 400 may include at least one processor 410 , a memory 420 , a storage 430 , and a communication interface 440 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computer system 400 may be loaded with various software including an operating system capable of driving a program.

The processor 410 is a device for controlling the operation of the computer system 400 , and may be various types of processors that process instructions included in a program, for example, a central processing unit (CPU), a micro processor unit (MPU) ), microcontroller unit (MCU), graphic processing unit (GPU), and the like.

The memory 420 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 410 . The memory 420 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 430 stores various data, programs, etc. required for executing the operation of the present invention. The communication interface 440 may be a wired/wireless communication module.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (16)

As a method for the server to detect the failure of the recording device of the smart device,
receiving the learning audio signals for each failure type, and extracting at least one spectrogram and audio information for each learning audio signal, respectively;
generating training data in which at least one spectrogram and audio information for each training audio signal are mapped to a corresponding failure type;
training a neural network classifier using the training data, and
Receiving an audio signal from the smart device, detecting a failure type of the audio signal using a learned neural network classifier, and transmitting it to the smart device
including,
The learned neural network classifier comprises: a first neural network model trained to output spectrogram feature values of the learning audio signals for each failure type;
A second neural network model trained to output audio information feature values of the audio signals for learning for each failure type, and
A classification model trained to output the failure type of the corresponding learning audio signal by connecting the feature values output from the first neural network model and the second neural network model.
A fault detection method comprising: According to claim 1,
The first neural network model is a convolutional recurrent neural network (CRNN)-based model,
The second neural network model is a recurrent neural network (RNN)-based model, a failure detection method. 3. The method of claim 2
The failure detection method, wherein the failure type includes at least one of a normal type, a transient failure type, or a persistent failure type. The method of claim 1
The step of extracting the spectrogram and the audio information, respectively,
extracting a plurality of time interval signals from each learning audio signal, and
Transforming each time interval signal into a frequency domain and extracting spectrogram and audio information of the corresponding time interval signal
including,
Each of the plurality of time interval signals is extracted such that at least some time intervals overlap with a previous time interval or a subsequent time interval. A method for a smart device including a recording device to detect a failure of the recording device, the method comprising:
receiving the audio signal collected from the recording device;
extracting at least one spectrogram and audio information for the audio signal; and
inputting the spectrogram and audio information of the audio signal into a neural network model trained as audio signals for learning for each failure type, and detecting a failure type of the audio signal through the neural network model
Including, a fault detection method. 6. The method of claim 5,
Before the step of receiving the audio signal,
receiving the learning audio signals for each failure type, and extracting at least one spectrogram and audio information for each learning audio signal, respectively;
generating training data in which at least one spectrogram and audio information for each training audio signal are mapped to a corresponding failure type; and
training a neural network classifier using the training data
including,
The step of training the neural network classifier comprises:
training a first neural network model included in the neural network classifier to output a spectrogram feature value when the spectrogram is input;
training a second neural network model included in the neural network classifier to output an audio information feature value when the audio information is input; and
Training the classification model to output the failure type of the input audio signal based on the probability value calculated by connecting the spectrogram feature value output from the first neural network model and the audio information feature value output from the second neural network model
Further comprising, a failure detection method. 6. The method of claim 5,
Before the step of receiving the audio signal,
Receiving the neural network model learned from the server from the server interworking with the smart device
Further comprising, a failure detection method. 6. The method of claim 5
The step of extracting the spectrogram and audio information,
extracting a plurality of time interval signals from the received audio signal; and
Transforming each time interval signal into a frequency domain and extracting spectrogram and audio information of the corresponding time interval signal
including,
Each of the plurality of time interval signals is extracted such that at least some time intervals overlap with a previous time interval or a subsequent time interval. 7. The method of claim 6,
The first neural network model is a convolutional recurrent neural network (CRNN)-based model,
The second neural network model is a recurrent neural network (RNN)-based model, a failure detection method. 7. The method of claim 6,
The step of training the classification model comprises:
A method for detecting a failure, wherein the spectrogram feature value and the audio information feature value are completely concatenated to generate a connectivity map, and an activation function is applied to the generated connectivity map to calculate the probability value. 6. The method of claim 5,
If there are a plurality of recording devices, obtaining the failure type includes:
The spectrogram and audio information of the first audio signal collected by the first recording device among the plurality of recording devices, the spectrogram and audio information of the second audio signal collected by the second recording device, and the first audio signal extracting the difference value of the second audio signal as input information, and
inputting the input information into the learned neural network model, and obtaining failure types of the first audio signal and the second audio signal from the neural network model
Including, a fault detection method. 12. The method of claim 11,
The step of extracting the input information is,
generating first and second frequency domain values from the first and second audio signals;
calculating a first difference value between the first frequency domain value and the second frequency domain value; and
n-squaring the first audio signal and the second audio signal, and calculating a second difference value from respective frequency domain values of the n-squared first audio signal and the second audio signal;
Including, a fault detection method. As a server for detecting the failure of the recording device included in the smart device,
A memory including at least one instruction and storing audio signals for learning for each failure type;
an interface for receiving an audio signal from the smart device, and
processor
including,
The processor is
Spectrogram and audio information are extracted from the learning audio signal for each failure type, a neural network classifier is trained using the spectrogram and audio information, and the received audio signal is input to the learned neural network classifier to determine the failure type of the audio signal. to detect, the server. 14. The method of claim 13,
The processor is
A server for extracting a plurality of section signals to overlap a predetermined section from the received audio signal, and converting the extracted section signals into frequency domain values to extract the plurality of audio information. 15. The method of claim 14,
The processor is
If there are a plurality of recording apparatuses of the smart device, the server calculates a difference value between the audio signals collected by the plurality of recording apparatuses, respectively. 16. The method of claim 15,
The processor is
A server for generating a connection map by performing full connection operation processing on the spectrogram feature value and the audio information feature value, and calculating the probability value by applying an activation function to the created connection map.

Images