KR102281676B1 - Audio classification method based on neural network for waveform input and analyzing apparatus - Google Patents

Abstract

A sound source classification method based on a neural network model for analyzing a waveform sound source signal includes the steps of: an analyzing device receiving a sound source signal in a waveform form, the analyzing device inputting the sound source signal to a neural network model, and the analyzing device using the neural network model and classifying the sound source signal based on the output information. The neural network model includes a plurality of convolution blocks, the convolution blocks include a one-dimensional convolutional layer and a pooling layer, and the size of a filter in the convolutional layer is 2 samples or 3 samples.

Description

A sound source classification method and analysis device based on a neural network model that analyzes a waveform sound source signal {AUDIO CLASSIFICATION METHOD BASED ON NEURAL NETWORK FOR WAVEFORM INPUT AND ANALYZING APPARATUS}

The technology to be described below relates to a technology for directly classifying a waveform sound source signal.

Music information retrieval (MIR) is a technical field for extracting and analyzing certain information from a sound source (music) signal. The MIR system converts a frequency domain signal into a sound source signal, which is typically a time domain signal, and analyzes the frequency domain signal. For example, the MIR system can analyze the sound source signal by changing it into a signal such as mel-spectrograms, which are logarithmic values of the frequency domain. Meanwhile, a sound source signal analysis technique using an artificial neural network model is being actively studied.

S. Dieleman and B. Schrauwen, "End-to-end learning for music audio," in Proc. Int. Conf. Acoust., Speech, Signal Process., 2014, pp. 6964-6968.

Most artificial neural network models classify sound sources by analyzing sound source signals in the frequency domain. Therefore, in the prior art, it is essential to change the initial sound source to the frequency domain. The technology to be described below is intended to provide a technique for classifying a sound source by directly analyzing a raw waveform of the sound source signal.

A sound source classification method based on a neural network model for analyzing a waveform sound source signal includes the steps of: an analyzing device receiving a sound source signal in a waveform form, the analyzing device inputting the sound source signal to a neural network model, and the analyzing device using the neural network model and classifying the sound source signal based on the output information. The neural network model includes a plurality of convolution blocks, the convolution blocks include a one-dimensional convolutional layer and a pooling layer, and the size of a filter in the convolutional layer is 2 samples or 3 samples.

A sound source analysis device using a neural network model for analyzing a waveform sound source signal is an input device for receiving a sound source signal in a waveform form, a storage device for storing a neural network model for analyzing a sound source signal, and inputs the sound source signal to the neural network model, and an arithmetic unit for classifying the sound source signal based on information output by the neural network model. The neural network model includes a plurality of convolution blocks, the convolution blocks include a one-dimensional convolutional layer and a pooling layer, and the size of a filter in the convolutional layer is 2 samples or 3 samples.

The technique to be described below effectively classifies a waveform sound source signal by using a neural network model that processes short samples. Furthermore, the technique to be described below classifies the sound source signal using an extended neural network model with an effective structure reinforced.

1 is an example of a sound source classification system.
2 is an example of a general CNN.
3 is an example of a neural network model for classifying a sound source.
4 is an example of the structure of an input region of a sample CNN.
5 is an example of a block of a neural network model for classifying a sound source.
6 is a result of comparing the performance of a spectrogram-based CNN and a sample CNN.
7 is a performance evaluation result for a convolution block.
8 is an example of a sound source analysis device.

The technology to be described below may have various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the technology described below.

Terms such as first, second, A, and B may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. used only as For example, a first component may be named as a second component, and similarly, a second component may also be referred to as a first component without departing from the scope of the technology to be described below. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of the terms used herein, the singular expression should be understood to include a plural expression unless the context clearly dictates otherwise, and terms such as "comprises" refer to the described feature, number, step, operation, and element. , parts or combinations thereof are to be understood, but not to exclude the possibility of the presence or addition of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is intended to clarify that the classification of the constituent parts in the present specification is merely a division according to the main function that each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of the other constituent units in addition to the main function it is responsible for, and may additionally perform some or all of the functions of the other constituent units. Of course, it may be carried out by being dedicated to it.

In addition, in performing the method or the method of operation, each process constituting the method may occur differently from the specified order unless a specific order is clearly described in context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

A technique to be described below is a technique for classifying a sound source.

The sound source or sound source signal basically means a waveform signal. The technology to be described below analyzes the sound source signal itself in the form of a waveform. A sound source is sound data in various forms. For example, the sound source includes music, speech, and acoustic scene sound.

Classification refers to the process of recognizing the content of a sound source or determining the type of sound source based on the characteristics of the sound source. For example, the sound source classification may refer to any one of tasks such as determining a genre of music, searching for a specific keyword in a human voice, classifying an environment in which a specific sound is generated, and recognizing that a specific event occurs in a sound source.

It is a technology for classifying sound sources with a machine learning model to be described below.

Machine learning is a field of artificial intelligence and refers to the field of developing algorithms so that computers can learn. A machine learning model or learning model refers to a model developed so that a computer can learn. There are various types of learning models, such as artificial neural networks and decision trees, depending on the approach.

The technology to be described below classifies a sound source using a neural network model. As the neural network model, various models such as Recurrent Neural Networks (RNN), feedforward neural network (FFNN), and convolutional neural network (CNN) may be used. The neural network that classifies the sound source may be one of various types. However, for the convenience of the description below, the CNN model will be mainly described.

The analyzer or sound source analyzer is a device for analyzing and classifying sound sources. The analysis device refers to a computing device capable of certain data processing. The analysis device may be implemented in various forms, such as a PC, a smart device, a server in a network, and a chipset dedicated to sound processing.

1 is an example of a sound source classification system. 1 shows three types of systems or devices. The analysis device is the subject that classifies the sound source. The analysis device classifies the corresponding sound source based on the waveform sound source signal. In FIG. 1 , the analysis device is illustrated in the form of analysis servers 110 and 210 and a smart device 300 .

FIG. 1A is an example of a system 100 including an analysis server 110 and a sound source database 120 . The sound source database 120 stores the sound source source file in the form of a waveform. The analysis server 110 receives a specific sound source from the sound source base 120 and classifies the received sound source. The analysis server 110 classifies the sound source using the neural network model. The user terminal 10 may request the analysis server 110 to classify the sound source. The user terminal 10 may receive the sound source classification result from the analysis server 110 .

2(B) is an example of a system 200 including an analysis server 210 . The user terminal 20 transmits a waveform sound source signal to the analysis server 210 . The analysis server 110 classifies the received sound source. The analysis server 110 classifies the sound source using the neural network model. The user terminal 10 may receive the sound source classification result from the analysis server 110 .

2 (C) is an example of an analysis device in the form of a smart device (310). The smart device 310 may receive a sound source directly into the microphone. The smart device 310 classifies the input sound source. The smart device 310 classifies the sound source using the neural network model. The smart device 310 may output the classification result and deliver it to the user.

Unlike FIG. 2C , the smart device 310 or the computer terminal can access the sound source stored in a storage medium (USB, SD card, hard disk, etc.). The smart device 310 or the computer terminal may classify the sound source stored in the storage medium using a neural network model.

2 is an example of a general CNN. 2 is for explaining the general structure and operation of the CNN model.

A CNN includes a convolution layer (Conv), a pooling layer (pool), and a fully connected layer. A plurality of convolutional layers and pooling layers may be repeatedly disposed. The CNN of FIG. 2 may have a structure of five convolutional layers, two pooling layers, and two fully connected layers.

The convolution layer outputs a feature map through a convolution operation on input data. In this case, a filter that performs a convolution operation is also called a kernel. The size of the filter is called the filter size or kernel size. Operation parameters constituting the kernel are called kernel parameters, filter parameters, or weights.

The convolutional layer performs convolutional and nonlinear operations. The convolution layer may include a batch normalization layer that normalizes output data.

The convolution operation is performed in a window of a constant size. The window can be moved one space from the upper left to the lower right of the 2D input data, and the moving size can be adjusted at a time. The amount of movement is called the stride. The convolution layer performs a convolution operation on all regions of the input data while moving the window in the input data. The convolution layer may maintain the dimension of the input data after the convolution operation by padding the edges of the input data.

In this case, a filter that performs a convolutional operation is also called a kernel. The size of the filter is called the filter size or kernel size. Operation parameters constituting the kernel are called kernel parameters, filter parameters, or weights. In the convolutional layer, different types of filters can be used for one input.

The nonlinear operation layer is a layer that determines output values from neurons (nodes). The nonlinear computation layer uses a transfer function. Transfer functions include Relu and sigmoid functions.

A pooling layer sub-sampling a feature map obtained as a result of an operation in the convolutional layer. Pooling operations include max pooling and average pooling. Maximum pooling selects the largest sample value within the window. Average pooling samples the average value of the values included in the window.

The pre-connected layer finally classifies the input data. The all-connected layer receives all the values output from the previous convolutional layer and performs final classification. In FIG. 2, the all-connection layer outputs a classification result using a softmax function.

3 is an example of a neural network model 400 for classifying sound sources. 3 is an example of a CNN-based neural network model. 3 shows that the CNN model does not process frame-level samples (eg, 256 or 512 samples), but processes a very small number of samples in the first convolutional layer. A CNN model with such a structure is called a sample CNN.

The input data is a waveform sound source signal.

The sample CNN 400 includes a plurality of convolutional layers and a pooling layer. Convolutional layers and pooling layers are one-dimensional. The size of the filter in all layers is very small, 2 or 3 samples. A small size filter reduces the likelihood that the input signal will be learned with the same filter shape, though it is out of phase in the time domain.

The sample CNN 400 includes a convolution-stride layer 410 , a plurality of convolution blocks 420 , and a fully connected layer (FC, 430 ).

3 is an example illustrating one convolution-stride layer 410 and nine convolution blocks 420 .

The convolution-stride layer (conv-stride) performs the stride convolution operation. A convolution-stride layer may also be viewed as a convolution block. A convolution-stride layer can perform a convolution operation with a filter having a size of three samples. In addition, the convolution-stride layer can perform strides of 3 sizes in one dimension.

A convolution block includes a convolution layer and a pooling layer. Pooling may use max pooling. The pooling layer can perform maximum pooling in units of three sample sizes.

The all-connection layer 430 may include two consecutive all-connection layers. The all-connection layer 430 classifies the sound source based on the information output by the convolution block.

Meanwhile, the sample CNN may have a structure different from that of FIG. 3 . (1) The number of a plurality of convolution blocks may vary. Convolution blocks may be less than 9 or greater than 9. (2) There may be no convolution-stride layer. Furthermore, the convolution-stride layer (conv-stride) may be composed of a plurality of layers performing stride convolution. For example, a convolution-stride layer may include two stride convolution layers.

4 is an example of the structure of an input region of a sample CNN. 4 is an example of the structure of the front end to which a waveform sound source signal is input and processed in the sample CNN. For convenience of description, the front end is referred to as an input end region. The input end region may include the aforementioned (i) convolution-stride layer, (ii) convolution-stride layer and convolution block, or (iii) a plurality of convolution blocks. The layer belonging to the input stage is composed of filters of small size (size of 2-3 samples). The layer belonging to the input region has a structure in which small-sized filters are stacked. In FIG. 4 , conv3 denotes a convolutional layer with a filter size of 3, max3 denotes a pooling layer with a filter size of 3, and strided conv3 denotes a convolution-stride layer with a filter size of 3. That is, FIG. 4 is an example of an input region composed of a layer having a filter size of 3 .

4(A) is an example of an input region without a convolution-stride layer. The input region of FIG. 4A includes a first convolutional layer (conv3_1), a first maximum pooling layer (max3_1), a second convolutional layer (conv3_2), and a second maximum pooling layer (max3_2). Of course, a layer for data normalization may be disposed between the convolution layer and the pooling layer. The second convolutional layer conv3_2 receives information sampled from the first maximum pooling layer max3_1.

4(B) is an example of an input region composed of one convolution-stride layer and a convolution block. The input region of FIG. 4B includes a first convolution-stride layer (strded conv3_1), a first convolution layer (conv3_1), and a first maximum pooling layer. Of course, a layer for data normalization may be disposed between the convolution layer and the pooling layer. The first convolution-stride layer (strded conv3_1) performs stride convolution with three filters, and the first convolution layer (conv3_1) receives the value output by the first convolution-stride layer (strded conv3_1). .

4(C) is an example of an input region composed of two convolution-stride layers. The input region of FIG. 4B includes a first convolution-stride layer (strded conv3_1) and a second convolution-stride layer (strded conv3_2).

The first convolution-stride layer (strided conv3_1) performs stride convolution with three filters, and the second convolution-stride layer (strided conv3_2) outputs the output of the first convolution-stride layer (strided conv3_1). get input value. The second convolution-stride layer (strded conv3_2) performs stride convolution with three filters, and passes the output to the next placed convolution block.

The sample CNN may use a small size filter in the entire layer. Alternatively, the sample CNN may process the waveform sound source signal by using a filter of a small size only in the input region shown in FIG. 4 . Furthermore, the sample CNN may process signals using filters of different sizes in a plurality of layers.

5 is an example of a block of a neural network model for classifying a sound source. FIG. 5 is an example of the structure of the convolution block described in FIG. 3 . Accordingly, in the sample CNN 400, at least one convolution block may be any one of the blocks shown in FIG. 5 . The sample CNN 400 may be composed of the same type of convolution block. In some cases, the sample CNN 400 may be configured in a form in which different types of convolution blocks are combined.

5(A) is an example of a structure 500 of a basic block. The basic block 500 includes a one-dimensional convolutional layer (Conv1D, 511), a batch normalization layer (BatchNorm, 512), and a maximum pooling layer (MaxPool, 513). Each of the convolutional layer 511 and the pooling layer 513_ performs the aforementioned functions. The batch normalization layer 512 normalizes the output in mini-batch units. The batch normalization layer 512 has an activation value in each layer. Let it be properly distributed The batch normalization layer can be an optional configuration.

5(B) is an example of a structure of a residual (Res-n) block 600 . The residual-n block 600 is defined as a block in which one additional path (skip connection) is added to the basic block 500 of FIG. 5A . The residual block 600 uses an additional path so that the gradient can propagate well.

A residual-n block means a residual block having n convolutional layers. A residual block used for sound source classification is a block with n = 1 or 2.

5(B) is an example of a residual block with n=2. The residual block of FIG. 5B is largely composed of two layers. The first layer includes a first convolutional layer (Conv1D, 611), a first batch normalization layer (BatchNorm, 612), and a dropout (dropout, 613) layer. The first layer is indicated by a dashed box.

The drop-out layer 613 traditionally turns off neurons randomly during learning in the all-connected layer in order to prevent over-fitting, thereby preventing a phenomenon in which learning is biased toward training data. The residual block causes the output of the first layer to be randomly lost. The drop ratio is configurable. For example, the loss rate may be set to 0.2. The second layer includes a second convolutional layer (Conv1D, 621), a second batch normalization layer (BatchNorm, 622), and a maximum pooling layer (MaxPool, 623). Before the maximum pooling layer (623) there is an additional path through which the input data is input. As such, in the residual block, the value input to the previous layer is also input to the subsequent layer, so that the gradient is transmitted well.

For reference, a residual block with n = 1 has a structure without a dotted line box portion (first layer) in FIG. 5B .

5(C) is an example of the structure of a squeeze and excitation (SE) block 700 . The SE block 700 includes a one-dimensional convolution layer (Conv1D, 711), a batch normalization layer (BatchNorm, 712) and a maximum pooling layer (MaxPool, 713), and an extraction/rebalance layer 720 . The SE block has a structure in which an extraction and re-adjustment layer indicated by a solid line box is added to the basic block of FIG. 5(A).

Since each of the filters operates in a local area to which the filter is applied, information on other areas cannot be used. The importance may be different for each channel, and it is difficult to consider this in a general CNN structure.

The extraction/re-adjustment layer 720 is a configuration that scales the feature map according to importance for each channel. Eventually, the extraction/recalibration layer 720 recalibrates the features. The extraction/re-adjustment layer 720 consists of an operation of extracting features for each channel (squeeze) and an operation of re-adjusting the extracted features in consideration of a relationship between channels (excitation).

The extraction (squeeze) layer 721 performs an operation of extracting statistics information (statistics) for each channel. The extraction layer 721 obtains channel-wise statistics by performing global average pooling for each channel for a predetermined time. The extraction layer 721 extracts a scalar value representing the channel for each channel. As shown on the right side of FIG. 5(D), the C × T feature map is reduced to statistical information for each C × 1 channel through global average pooling. C stands for filter channel and T stands for dimensionality in time.

The excitation layer receives the output of the extraction layer as an input value. The rebalancing layer computes a weight for each channel. Weights are learned through two all-connected layers (FC, 722, 723). The first pre-connected layer 722 receives the output of the extraction layer as an input value and performs ReLu function operation, and the second pre-connected layer 723 receives the output of the first pre-connected layer 721 as an input, sigmoid operation. The dimensional number between the two heat transfer layers can also be controlled by the hyperparameter α. As a result of experiments on CNN for sound source classification, it was found that α preferably has a value less than 1.

Finally, each channel is multiplied by the readjusted channel statistical information of C × 1 size, and the feature map is rescaled (rescaling, 724). In the right side of FIG. 5(D), readjusted features (statistical information) for each channel are expressed in color.

5(D) is an example of the structure of a residual and extraction/rebalancing (ReSE-n) block 800 . The ReSE-n block 800 is a form in which the structures of the residual block of FIG. 5B and the extraction/rearrangement block of FIG. 5C are merged.

The ReSE-n block 800 may be divided into two layers (a first layer and a second layer) when classified based on a layer performing a convolution operation.

The first layer includes a first convolutional layer (Conv1D, 811), a first batch normalization layer (BatchNorm, 812), and a dropout layer (dropout, 813). The second layer 820 includes a second convolution layer (Conv1D, 821), a second batch normalization layer (BatchNorm, 822), an extraction/rebalance layer, and a maximum pooling layer (MaxPool, 827). Before the max pooling layer, there is an additional path where the input data enters. The extraction/rearrangement layer is the same as the configuration described in FIG. 5(C).

The extraction/remediation layer includes an extraction layer 823 and a remediation layer 824 , 825 , 826 . The extraction layer 823 obtains channel-wise statistics by performing global average pooling for each channel for a predetermined time. The extraction layer 823 extracts a scalar value representing the channel for each channel. As shown on the right side of FIG. 5(D), the C × T feature map is reduced to statistical information for each C × 1 channel through global average pooling.

The excitation layer receives the output of the extraction layer as an input value. The rebalancing layer computes a weight for each channel. Weights are learned through two all-connected layers (FC, 824, 825). The first pre-connected layer 824 receives the output of the extraction layer as an input value and performs ReLu function operation, and the second pre-connected layer 825 receives the output of the first pre-connected layer 824 as an input, sigmoid operation. The dimensional number between the two heat transfer layers can also be controlled by the hyperparameter α. As a result of experiments on CNN for sound source classification, it was found that α preferably has a value less than 1.

Finally, each channel is multiplied by the readjusted channel statistical information of C × 1 size, and the feature map is rescaled (rescaling, 826). In the right side of FIG. 5(C), readjusted features (statistical information) for each channel are expressed in color.

In the maximum pooling layer 827 , a value obtained by adding a value output from the extraction/re-adjustment layer and a value input to the first convolution layer 811 is inputted.

Hereinafter, an experiment to verify the effect of a sample CNN for sound source classification will be described. Effects on sound sources of three different domains were tested. The three domains are music, voice and sound scene sound. The data set for the experiment and the trained model structure are shown in Table 1 below.

Music auto-tagging is a multi-classification operation for sound sources. For example, the music classification may be classified according to criteria such as genre, mood, instrument, vocal level, and the like. In the experiment, the MTT (MagnaTagATun) data set, which is frequently used for MIR, was used. The evaluation criterion was based on the ranking accuracy of the classified music. The classification accuracy was evaluated by ROC-AUC (area under receiver operating characteristic). The ROC-AUC score was calculated and averaged for all tags as the score. A sound source clip having at least one positive label and a length of 29.1 seconds or longer was used. Meanwhile, for comparison with other techniques, a MSD (Million Song Dataset) data set having a Last.FM tag was used. MTT and MDS were evaluated by pretreatment in the same manner.

Keyword Spotting corresponds to a multi-classification task for speech signals. Key word detection is mainly used to recognize short sentences that are often used in AI speakers. The TensorFlow community recently published a speech dataset for speech command recognition. In this experiment, a data set containing 35 general instructions was used. That is, it was evaluated whether it correctly specified one of 35 possible commands.

Acoustic scene tagging is a multi-classification operation for sound sources. The data set published in 2017 by DCASE (Detection and Classification of Acoustic Scenes and Events) was used. In this experiment, we used the data set of the non-timestamped version (audio tagging). Assessment was performed using an instance-based F-score. The average F-score of the audio clips of the data set for testing was used.

All CNN models for testing have a batch size of 23 and are trained by gradient descent with a Nesterov momentum of 0.9. The learning rate was initially set to 0.01, and if the effectiveness loss did not decrease in two epochs, the learning rate was divided by 5. A droop-out layer with a loss rate of 0.5 is inserted before the last full-connection layer of the CNN model. In the training process, predictions were made for each segment, and the predictions of the segments were averaged to make a final prediction for each audio clip.

6 is a result of comparing the performance of a spectrogram-based CNN and a sample CNN. Fig. 6(A) is a result of automatic music tagging, Fig. 6(B) is a result of key word detection, and Fig. 6(C) is a result of sound scene tagging.

Spectrogram-based CNN is a traditional model that classifies sound sources by receiving Mel-spectrograms as input. For accurate comparison with the sample CNN, the spectrogram-based CNN has a structure similar to the sample CNN used in the experiment as much as possible. The sample CNN used a model in which the filter and maximum unwrapping/striding have all three sizes.

At the bottom of FIG. 6, the window and hop size of the Mel-spectrogram are indicated, and the filter and hop size used in the first convolutional layer of the sample CNN are indicated correspondingly. For example, in FIG. 6 , if the window/filter size is 729 ( 3 ⁶ ), the hop/stride size is also 829 . By reducing the window and filter size, the accuracy of the model was evaluated. However, for the spectrogram-based CNN, if the size is too small, it is difficult to express in the frequency domain, so when the window/hop size reaches 81, the size is not reduced any more.

Referring to FIG. 6 , the sample CNN basically shows better performance than the spectrogram-based CNN. In addition, the spectrogram-based CNN converges to a certain level as the window and hop sizes are reduced, but the sample CNN shows better performance as the filter and stride sizes are reduced. The sample CNN performed best with the smallest filter and stride sizes.

In addition, two convolution blocks were evaluated. Models with basic blocks and SE blocks were evaluated, respectively. The sample CNN consistently outperformed the model with the SE block than the model with the base block.

In the sample CNN, the model parameters increased as the depth of the model increased. Therefore, it can be seen that the performance improvement is due to an increase in the model size. To verify this, additional experiments were performed on the sample CNN. For a sample CNN with basic blocks, the number of parameters was fixed by adjusting the number of filters. Table 2 below summarizes the experimental results.

Looking at Table 2, it can be seen that the performance improvement of the sample CNN is due to the structure (depth).

A convolution block constituting a sample CNN has been described in FIG. 5 . We evaluated the performance of different types of convolution blocks that can compose a sample CNN. 7 is a performance evaluation result for a convolution block. 7 is a performance evaluation result for six sample CNNs each consisting of a basic block, an SE block, a Res-1 block, a Res-2 block, a ReSE-1 block, and a RESE-2 block. 7 shows the mean and standard deviation for the performance of each model. 7(A) is an evaluation result for automatic music tagging, and FIG. 7(B) is an evaluation result for PR (Precision Recall)-AUC for automatic music tagging. 7(C) is an evaluation result for key word detection. 7(D) is an evaluation result for acoustic scene tagging.

Referring to FIG. 7 , overall, the SE block-based model (SE), the Res-2 block-based model (Res-2), and the ReSE-2 block-based model (RsSE-2) show superior effects compared to other models. However, the specific model did not have superior performance in all three domains.

8 is an example of a sound source analysis device. The analysis device 900 is a device corresponding to the analysis device 110 , 210 or 310 of FIG. 1 .

The analysis device 900 classifies the sound source using the above-described neural network model (sample CNN). The analysis device 900 may be physically implemented in various forms. For example, the analysis device 900 may have the form of a computer device such as a PC, a server of a network, a chipset dedicated to sound processing, and the like. The computer device may include a mobile device such as a smart device or the like.

The analysis device 900 includes a storage device 910 , a memory 920 , an arithmetic device 930 , an interface device 940 , a communication device 950 , and an output device 960 .

The storage device 910 stores a neural network model (sample CNN) that analyzes a sound source signal. Neural network models must be trained in advance. Furthermore, the storage device 910 may store other programs or source codes required for data processing. The storage device 910 may store an input sound source file or sound source data.

The memory 920 may store data and information generated in the process of analyzing the data received by the analysis apparatus 900 .

The interface device 940 is a device that receives predetermined commands and data from the outside. The interface device 940 may receive sound source data from a physically connected input device or an external storage device. The interface device 940 may receive a learning model for sound analysis. The interface device 940 may receive learning data, information, and parameter values for training a learning model.

Furthermore, the interface device 940 may be a microphone device that directly receives a sound from the field.

The communication device 950 refers to a configuration for receiving and transmitting certain information through a wired or wireless network. The communication device 950 may receive sound source data from an external object. The communication device 950 may also receive data for model learning. The communication device 950 may transmit the analysis result of the sound source to an external object.

The communication device 950 and the interface device 940 are devices that receive predetermined data or commands from the outside. The communication device 950 or the interface device 940 may be referred to as input devices.

The input device may input or receive sound source data to be analyzed. For example, the input device may receive sound source data from an external server or DB. The input device may collect sound sources generated in the field. The input device may receive sound source data directly from the storage medium.

The output device 960 is a device that outputs certain information. The output device 960 may output an interface necessary for a data processing process, an analysis result, and the like.

The computing device 930 may classify the sound source by using the neural network model or program stored in the storage device 910 . The computing device 930 may classify the sound source based on a value output by the neural network model. The value output by the neural network model may be any one of multiple classifications. The computing device 930 may classify the sound source by directly using the output value of the neural network model. Furthermore, the computing device 930 may classify the sound source by processing or additionally analyzing the output value of the neural network model.

Meanwhile, the computing device 930 may train a learning model for classifying a sound source by using the given training data.

The computing device 930 may be a device such as a processor, an AP, or a program embedded chip that processes data and processes a predetermined operation.

In addition, the sound source classification method and sample CNN as described above may be implemented as a program (or application) including an executable algorithm that can be executed in a computer. The program may be provided by being stored in a non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the various applications or programs described above may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

This embodiment and the drawings attached to this specification merely clearly show a part of the technical idea included in the above-described technology, and within the scope of the technical idea included in the specification and drawings of the above-described technology, those skilled in the art can easily It will be apparent that all inferred modified examples and specific embodiments are included in the scope of the above-described technology.

Claims (13)

receiving, by the analysis device, a sound source signal in the form of a waveform;
inputting, by the analysis device, the sound source signal to a neural network model; and
Comprising the step of the analysis device classifying the sound source signal based on the information output by the neural network model,
The neural network model includes a plurality of convolution blocks, the convolution block includes a one-dimensional convolution layer and a pooling layer, and the size of the filter in the convolution layer is 2 samples or 3 samples,
The neural network model further comprises a stride convolution layer for performing strided convolution in one dimension in front of the plurality of convolution blocks,
The stride convolution layer is a sound source classification method based on a neural network model that analyzes a waveform sound source signal in which one or two consecutive stride convolution layers are performed. delete According to claim 1,
At least one block among the plurality of convolution blocks is a residual block,
The residual block is a one-dimensional first convolutional layer, a one-dimensional second convolutional layer disposed after the first convolutional layer, and is disposed between the first convolutional layer and the second convolutional layer, A waveform sound source signal comprising a drop out layer having a specific loss rate and a pooling layer for maximally pooling information output from the second convolution layer and information input to the first convolution layer A sound source classification method based on the neural network model to be analyzed. According to claim 1,
At least one block among the plurality of convolution blocks is a squeeze and excitation block,
The extraction and reconditioning block includes a one-dimensional first convolutional layer, a pooling layer maximally pooling the output of the first convolutional layer, and an extraction/rebalancing layer,
The extraction/reconditioning layer analyzes a waveform sound source signal including an extraction layer that extracts statistical information for each channel by performing global average pooling and a readjustment layer that includes two all-connected layers that determine a weight for readjusting the statistical information A sound source classification method based on a neural network model 5. The method of claim 4,
A sound source classification method based on a neural network model that analyzes a waveform sound source signal controlled by a hyperparameter having a dimension number less than one between the two all-connected layers. According to claim 1,
At least one block among the plurality of convolution blocks is a residual and an extraction / re-regulation (squeeze and excitation) block,
The residual and extraction/rebalancing blocks include a one-dimensional first convolutional layer, a one-dimensional second convolutional layer disposed after the first convolutional layer, the first convolutional layer and the second convolutional layer. A drop out layer disposed between and having a specific loss rate, an extraction/re-adjustment layer that receives the output of the second convolutional layer and reflects the characteristics of each channel, and information output from the extraction/re-adjustment layer and the first 1 Including a pooling layer that maximally pools the summed information of the information input to the convolution layer,
The extraction/reconditioning layer analyzes a waveform sound source signal including an extraction layer that extracts statistical information for each channel by performing global average pooling and a readjustment layer that includes two all-connected layers that determine a weight for readjusting the statistical information A sound source classification method based on a neural network model. A computer-readable recording medium recording a program for executing a sound source classification method based on a neural network model for analyzing a waveform sound source signal according to any one of claims 1 to 6 in a computer. an input device receiving a sound source signal in the form of a waveform;
A storage device that stores a neural network model that analyzes a sound source signal, and
A computing device for inputting the sound source signal to the neural network model and classifying the sound source signal based on information output by the neural network model,
The neural network model includes a plurality of convolution blocks, the convolution block includes a one-dimensional convolution layer and a pooling layer, and the size of the filter in the convolution layer is 2 samples or 3 samples,
At least one block among the plurality of convolution blocks is a residual block,
The residual block is a one-dimensional first convolutional layer, a one-dimensional second convolutional layer disposed after the first convolutional layer, and is disposed between the first convolutional layer and the second convolutional layer, A waveform sound source signal comprising a drop out layer having a specific loss rate and a pooling layer for maximally pooling information output from the second convolution layer and information input to the first convolution layer A sound source analysis device using a neural network model to analyze. 9. The method of claim 8,
The neural network model further comprises a stride convolution layer for performing strided convolution in one dimension in front of the plurality of convolution blocks,
The stride convolution layer is a sound source analyzer using a neural network model that analyzes a waveform sound source signal in which one or two consecutive stride convolution layers are used. delete 9. The method of claim 8,
At least one block among the plurality of convolution blocks is a squeeze and excitation block,
The extraction and reconditioning block includes a one-dimensional first convolutional layer, a pooling layer maximally pooling the output of the first convolutional layer, and an extraction/rebalancing layer,
The extraction/reconditioning layer analyzes a waveform sound source signal including an extraction layer that extracts statistical information for each channel by performing global average pooling and a readjustment layer that includes two all-connected layers that determine a weight for readjusting the statistical information A sound source analysis device using a neural network model. 12. The method of claim 11,
A sound source analysis apparatus using a neural network model for analyzing a waveform sound source signal controlled by a hyperparameter having a dimension number less than one between the two all-connected layers. 9. The method of claim 8,
At least one block of the plurality of convolution blocks is an extraction / reconditioning (squeeze and excitation) block,
The extraction/rearrangement block is a one-dimensional first convolutional layer, a one-dimensional second convolutional layer disposed after the first convolutional layer, and disposed between the first convolutional layer and the second convolutional layer. and a drop out layer having a specific loss rate, an extraction/re-adjustment layer that receives the output of the second convolution layer and reflects the characteristics of each channel, and information output from the extraction/re-adjustment layer and the first convolution Including a pooling layer for maximally pooling the summed information of information input to the layer,
The extraction/reconditioning layer analyzes a waveform sound source signal including an extraction layer that extracts statistical information for each channel by performing global average pooling and a readjustment layer that includes two all-connected layers that determine a weight for readjusting the statistical information A sound source analysis device using a neural network model.

Images