KR102018346B1 - Method for classifying sounds and system therefor - Google Patents

Abstract

The present invention relates to technique for classifying sound signals using a multi-stream convolution neural network. The method may comprise the steps of: generating a full spectrogram of sound signals; generating a plurality of partial spectrograms from the full spectrogram, wherein each partial spectrogram includes a predetermined frequency interval of the full spectrogram; and determining a first parameter used for a convolution operation performed in a first convolution neural network for the full spectrogram and a second parameter used for a convolution operation performed in a second convolution neural network for the partial spectrograms.

Description

Method and system for classifying acoustic signals {METHOD FOR CLASSIFYING SOUNDS AND SYSTEM THEREFOR}

The present invention relates to a technique for classifying acoustic signals, and more particularly, to a technique for classifying acoustic signals using a multi-stream convolution neural network.

Using a spectrogram-based feature to classify the acoustic signals of the surrounding environment as input to the convolutional deep neural network, it is possible to learn various frequency-time modulation patterns of the environmental sound to obtain high quality features.

However, when learning the entire spectrogram pattern of each environmental acoustic signal using a single stream convolutional deepening neural network, there is a limit in learning all the details of the spectrogram characteristics, thus classifying the environmental sounds having similar patterns. Difficulties arise.

Korea Patent Registration 10-1749254 (registered June 14, 2017)

In an embodiment of the present invention, a technique for classifying a sound signal by performing multi-stream convolution deep learning on a spectrogram of the sound signal and classifying the sound signal may be proposed.

In addition, the embodiment of the present invention, by applying the global pooling (global pooling) to the convolution deepening learning results, to propose a technique that can reduce the amount of computation and over-fitting in the deep neural network.

The problem to be solved by the present invention is not limited to the above-mentioned, another problem to be solved is not mentioned can be clearly understood by those skilled in the art from the following description. will be.

According to an embodiment of the present invention, there is provided a method for classifying an acoustic signal using a multi-stream convolutional neural network, the method comprising: generating an entire spectrogram of an acoustic signal, and generating a plurality of spectrograms from the entire spectrogram; Generating four partial spectrograms, each of the partial spectrograms comprising a predetermined frequency interval of the entire spectrogram, and used for a convolution operation performed in a first convolutional neural network for the full spectrogram And a second parameter used in a convolution operation performed in a second convolutional neural network for the plurality of partial spectrograms.

The determining may include predicting a first result using the first convolutional neural network for the entire spectrogram of the acoustic signal, and for the plurality of partial spectrograms of the acoustic signal, Predicting a second result using a convolutional neural network, learning the first parameter using the first result, and learning the second parameter using the second result. .

In addition, global pooling may be applied to feature information generated by a convolution operation in the second convolutional neural network for the plurality of partial spectrograms.

In addition, the applied global pooling may be Global Average Pooling (GAP) and Global Max Pooling (GMP).

The method may further include classifying the sound signal by applying the first parameter and the second parameter to the multi-stream convolutional neural network for the entire spectrogram and the plurality of partial spectrograms of an arbitrary sound signal. have.

The classifying the sound signal may include performing convolution operation on the entire spectrogram, generating feature information on the entire spectrogram, and performing convolution operation on the plurality of partial spectrograms. Generating feature information for the plurality of partial spectrograms, applying a first activation function to feature information for the entire spectrogram and feature information for the plurality of partial spectrograms; Merging the feature information of the entire spectrogram to which the first activation function is applied and the feature information of the plurality of partial spectrograms, and then applying a second activation function and a result of applying the second activation function. Classifying the acoustic signal by applying a third activation function The.

In addition, global pooling may be applied to the feature information on the entire spectrogram and the feature information on the plurality of partial spectrograms.

In addition, the first activation function and the second activation function may be a rectified linear unit (ReLU) function, and the third activation function may be a sigmoid function.

The method may further include generating the plurality of partial spectrograms after expanding the size of frequency bins of the entire spectrograms.

According to an embodiment of the present invention, a system for classifying an acoustic signal using a multi-stream convolutional neural network, comprising: a preprocessor for generating an entire spectrogram of an acoustic signal, and a plurality of partial spectrograms from the entire spectrogram; A spectrogram classifier, each of the partial spectrograms comprising a predetermined frequency interval of the entire spectrogram, a first parameter used for a convolution operation performed on the entire spectrogram, and the plurality of portions A system for classifying acoustic signals including a learning model unit for determining a second parameter used in a convolution operation performed on a spectrogram may be provided.

Here, the learning model unit includes a first convolutional neural network for predicting a first result with respect to the entire spectrogram of the sound signal, and learning a first parameter using the first result, and a plurality of parts of the sound signal. A second convolutional neural network that predicts a second result for a spectrogram and learns a second parameter using the second result, and for the entire spectrogram and a plurality of partial spectrograms of any acoustic signal, And a third convolutional neural network classifying the sound signal by applying one parameter and the second parameter to the multi-stream convolutional neural network.

The first convolutional neural network may further include: a convolution layer configured to apply a first activation function to the entire spectrogram to generate convolutional feature information for the entire spectrogram, and a second to the convolutional feature information. And a fully connected layer for generating linear concealment feature information by applying an activation function, and an output layer for outputting a result of learning the first parameter by applying a third activation function to the linear concealment feature information.

The second convolutional neural network may further include: a convolution layer configured to apply a first activation function to the plurality of partial spectrograms to generate convolution feature information for the plurality of partial spectrograms, and the convolution characteristic information. A global pooling layer applying global pooling to the fully-connected layer generating linear hidden feature information by applying a second activation function to the convolution feature information to which the global pooling is applied, and And an output layer that outputs a result of learning the second parameter by applying an activation function.

In addition, the first activation function may be a leaky rectified linear unit (LReLU) function, the second activation function may be a rectified linear unit (ReLU) function, and the third activation function may be a sigmoid function.

In addition, the third convolutional neural network may apply the first parameter and the second parameter to generate convolution characteristic information for all spectrograms of an arbitrary sound signal and convolution characteristic information for a plurality of partial spectrograms. A convolution layer, a global pooling layer that applies global pooling to the convolutional feature information for the entire spectrogram and the convolutional feature information for the partial spectrogram, and a convolution of the entire spectrogram to which the global pooling is applied A fully-connected layer generating linear concealment feature information by applying a first activation function to the convolution feature information and the convolution feature information of the partial spectrogram, and applying the second activation function after merging the linear concealment feature information. A third activation function for the merged layer and the result of applying the second activation function Applied may comprise an output layer for outputting a classification result of the sound signal.

In addition, the first activation function and the second activation function may be a ReLU function, and the third activation function may be a sigmoid function.

According to an embodiment of the present invention, by performing multi-stream convolution deep learning on the spectrogram of the sound signal, it is possible to increase the accuracy of sound signal classification accuracy, and by applying global pooling to the convolution deep learning result, The amount of computation and overfitting can be reduced.

1 is a block diagram of a system for acoustic signal classification according to an embodiment of the present invention.
FIG. 2 is a diagram exemplarily illustrating a convolution operation on an entire spectrogram performed by the learning model unit 300 of FIG. 1.
FIG. 3 is a diagram for explaining a convolution operation on a plurality of partial spectrograms performed by the learning model unit 300 of FIG. 1.
4 is a diagram illustrating an example of a convolution operation on the entire spectrogram and the plurality of partial spectrograms performed by the learning model unit 300 of FIG. 1.
FIG. 5 is a numerical comparison of sound signal classification results by a typical single stream convolution operation and sound signal classification results by a multi-stream convolution operation according to an embodiment of the present invention.
6 is a diagram illustrating a confusion matrix by a multi-stream convolution operation according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms, only the embodiments are to make the disclosure of the present invention complete, and those skilled in the art to which the present invention pertains. It is provided to fully inform the scope of the invention, and the scope of the invention is defined only by the claims.

In describing the embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted unless they are actually necessary in describing the embodiments of the present invention. The terms to be described below are terms defined in consideration of functions in the embodiments of the present invention, which may vary according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

In an embodiment of the present invention, a full spectrogram of an acoustic signal is generated, a plurality of partial spectrograms including a predetermined frequency section of the entire spectrogram are generated, and a multi-stream is generated for the full spectrogram and the plurality of partial spectrograms. By applying a convolutional neural network to classify the acoustic signal, a technique for improving the accuracy of acoustic signal classification is proposed. In addition, in an embodiment of the present invention, by applying global pooling to the convolution deep learning results of the entire spectrogram and partial spectrogram, it is proposed a technique that can reduce the amount of computation and overfit in the deep neural network.

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the present invention.

1 is a block diagram of a system for acoustic signal classification according to an embodiment of the present invention.

As shown in FIG. 1, the acoustic signal classification system 10 may include a preprocessor 100, a spectrogram classifier 200, a learning model unit 300, and an acoustic classifier 400.

First, the preprocessor 100 may perform a preprocessing process for generating the entire spectrogram of the input sound signal.

In an embodiment of the present invention, for example, a log-scale Mel spectrogram may be used to classify an acoustic signal, and to generate a log-scaled mel spectrogram, the following process may be performed. have.

First, a Hamming window equal to the sample length N is applied to the discrete signal y (n) as shown in [Equation 1].

Thereafter, a Discrete Fourier Transform (DFT) may be performed while moving by n _s samples to obtain a result value Y _{l, k} as shown in [Equation 2].

If you apply absolute value to the result of Equation 2, you can generate spectrogram. If you filter Mel filter ( F _{k, m} ) on the generated spectrogram, and scale it with log value, A log scaled mel spectrogram such as 3] can be generated.

The spectrogram classifier 200 may generate a plurality of partial spectrograms from all spectrograms generated by the preprocessor 100, for example, a log scaled mel spectrogram.

Here, each of the plurality of partial spectrograms may include a predetermined frequency section of the entire spectrogram. For example, a frequency bin of the entire spectrogram is divided into a low frequency interval, a low-medium frequency interval, and a medium-high frequency interval, the low frequency interval being a first partial spectrogram, and the low-medium frequency interval being a first. The two partial spectrograms may be divided into a third partial spectrogram to generate a plurality of partial spectrograms.

At this time, since the size of the frequency bin of each partial spectrogram is very small (e.g., 32 bins), when the depth of the convolutional neural network becomes deep, the size of the receptive field is determined by the input characteristic. It can be almost equal in size. In this case, since there is a limit in classifying the local frequency patterns of the environmental sound in detail, in the embodiment of the present invention, the size of the entire spectrogram input is reset to extend the size of the frequency bin and then the respective partial spectrograms may be modified. Can be implemented to generate For example, after doubling the frequency axis section of the entire spectrogram (32, 128-> 64, 128), a plurality of partial spectrograms are generated, and a plurality of partial spectrograms generated from the extended full spectrogram are convolutioned. Can be set as the input value of the neural network.

The training model unit 300 classifies the sound signal by performing deep learning on each of the entire spectrogram and the plurality of partial spectrograms input through the spectrogram classifier 200, for example, by performing operations on a convolutional neural network. Acoustic signal classification model can be generated. The process of generating the acoustic signal classification model may include a process of determining (training) a parameter used in the convolution operation of the learning model unit 300. Here, the parameter may include, for example, weight and bias used in the convolution operation.

In addition, the training model unit 300 is based on the parameters used in the convolution operation of the entire spectrogram, and the parameters used in the convolution operation of the plurality of partial spectrogram, the entire spectrogram and a plurality of Multistream convolution operations on partial spectrograms can be performed.

The sound classifier 400 may classify an arbitrary sound signal based on a result of the multi-stream convolution operation performed by the training model unit 300.

Hereinafter, the sound signal classification method according to the embodiment of the present invention together with the above-described configuration will be described in detail.

FIG. 2 is a diagram exemplarily illustrating a convolution operation on an entire spectrogram performed by the learning model unit 300 of FIG. 1. In the embodiment of FIG. 2, the entire spectrogram may be named a global feature .

As shown in FIG. 2, the convolution operation on the entire spectrogram predicts a first result for the entire spectrogram of an acoustic signal having a specific magnitude, for example 128 × 128, and uses this first result. By performing a first convolutional neural network learning the first parameter.

Here, the first convolutional neural network may include a convolution layer 302a for generating convolution characteristic information for the entire spectrogram by applying a first activation function to the entire spectrogram, and a second activation for the convolution characteristic information. Fully connected layer 306a for applying linear function to generate linear hidden feature information, and output layer 310a for outputting a result of learning a first parameter by applying a third activation function to linear hidden feature information. ) May be included.

The convolutional layer 302a may include, for example, three layers C1, C2, and C3, each time a convolution operation of each of the layers C1, C2, and C3 is completed, It is possible to reduce the dimension of the entire spectrogram input by max pooling while moving by a predetermined stride (for example, window range: 2X2, stride: 2). Convolution characteristic information f _Global (X _E ) generated by the convolution operation of the convolution layer 302a may be expressed as Equation 4 below.

In Equation 4, f () on the right side represents a first activation function, for example, a Leaky Rectified Linear Unit (LReLU) function, and * on the right side represents a convolution operation. In addition, W _{Li and} b _Li may represent weights and biases of the i th layer of the convolutional neural network, respectively.

The generated convolution characteristic information f _Global (X _E ) is converted into one high-dimensional vector and then transferred to the fully connected layer 306a.

The fully connected layer 306a may generate linear concealment feature information by applying a second activation function, for example, a rectified linear unit (ReLU) function, to the convolution feature information generated in the convolution layer 302a. Such linear concealment feature information X _fc may be expressed as Equation 5 below.

In Equation 5, f () on the right side represents a ReLU function, which is a second activation function, and W _{L4 and} b _L4 represent weights and biases required for generating linear hidden feature information, respectively.

The generated linear concealment feature information X _fc is passed to the output layer 310a.

The output layer 310a may output a result of learning the first parameter by applying a third activation function, for example, a sigmoid function, to the linear concealment feature information generated in the fully connected layer 306a. have. The resulting value is output by this output layer (310a) (output) can be expressed as: [Equation 6].

In Equation 6, f () on the right side represents a sigmoid function, which is a third activation function, and W _L5 represents a weight required for calculating a result value. The bias is not applied to the operation of the output layer 310a.

FIG. 3 is a diagram for explaining a convolution operation on a plurality of partial spectrograms performed by the learning model unit 300 of FIG. 1. In the embodiment of FIG. 3, the plurality of partial spectrograms may be named local features .

The input features of a plurality of partial spectrograms, called local features, are three spectrograms separated in the frequency direction, each of which is divided into a low frequency, a low-medium frequency, and a mid-high frequency interval. It may include.

Here, in the case of the low frequency, low-medium frequency, middle-high frequency in order can mean the (1 ~ 32), (33 ~ 64), (65 ~ 96) frequency empty interval of the spectrogram, In the examples, spectrograms having a high frequency range are not considered.

Since the frequency bin size of each spectrogram is very small (e.g., 32 bins), the embodiment of the present invention resizes the input spectrogram to double the frequency axis section (32, 128-> 64, 128) It was set as the input value of the convolutional neural network.

As shown in FIG. 3, the convolution operation on the plurality of partial spectrograms predicts a second result for the partial spectrogram of the acoustic signal of each particular magnitude, for example 64 × 128 each, The second result may be performed in a second convolutional neural network that learns the second parameter.

Here, the second convolutional neural network may include a convolution layer 302b for generating convolution characteristic information for the plurality of partial spectrograms by applying a first activation function to the plurality of partial spectrograms, and the convolution characteristic information. A global pooling layer 304b for applying global pooling to the global pooling layer, a fully connected layer 306b for applying a second activation function to the convolution feature information to which the global pooling is applied to generate linear hidden feature information, and a linear hidden feature information. It may include an output layer 310b for outputting a result of learning the second parameter by applying a third activation function for.

Convolution layer 302b may include three convolution layers 302b-1, 302b-2, and 302b-3, each of which is connected to a total of three convolution streams.

The first convolutional layer 302b-1 of the three convolutional layers 302b-1, 302b-2, 302b-3, for example, performs a convolution operation on the first partial spectrogram of the low frequency interval. can do. This first convolutional layer 302b-1 may include, for example, three layers C4, C5, and C6, each time a convolutional operation of each of the layers C4, C5, and C6 is completed. It is possible to reduce the dimensions of a plurality of partial spectrograms input by pulling the maximum value while moving by a predetermined stride to a window of a predetermined range (for example, window range: 2X2 and stride: 2). The first partial convolution feature information f _Local1 (X _L ) generated by the convolution operation of the first convolution layer 302b-1 may be expressed as Equation 7 below.

In addition, the second convolutional layer 302b-2 of the three convolutional layers 302b-1, 302b-2, 302b-3 is, for example, a convolution for the second partial spectrogram of the low-medium frequency interval. You can perform a calculation operation. This second convolutional layer 302b-2 may comprise, for example, three layers C7, C8, C9, and as in the first convolutional layer 302b-1, each layer C7, C8 , Maximum pooling may be performed whenever the convolutional operation of C9) is completed. The second partial convolution feature information f _Local2 (X _LM ) generated by the convolution operation of the second convolution layer 302b-2 may be expressed as Equation 8 below.

In addition, the third convolutional layer 302b-3 of the three convolutional layers 302b-1, 302b-2, 302b-3 is, for example, a convolution for the third partial spectrogram of the mid-high frequency interval. You can perform a calculation operation. Such third convolutional layer 302b-3 may comprise, for example, three layers C10, C11, C12, in the first and second convolutional layers 302b-1, 302b-2. Similarly, the maximum pooling may be performed at the end of the convolution operation of each layer C10, C11, and C12. The third partial convolution feature information f _Local3 (X _MH ) generated by the convolution operation of the third convolution layer 302b-3 may be expressed as Equation 9 below.

F () on the right side of [Equation 7], [Equation 8] and [Equation 9] means a first activation function, for example, an LReLU function, and * on the right side means a convolution operation. In addition, W _{LiSj and} b _LiSj may represent weights and biases of the i th layer and the j th stream of the convolutional neural network, respectively.

Each generated partial convolution feature information (( f _Local1 (X _L ) , ( f _Local2 (X _LM ) , ( f _Local3 (X _MH ) )) is converted into a single high-dimensional vector and then the global pooling layer 304b Is passed to.

The global pooling layer 304b may apply global pooling to respective feature maps output from the output of the last convolutional layer when classifying the acoustic signal of the surrounding environment and then give it as an input value of the fully connected layer 306b. In an embodiment of the present invention among various global pooling methods, for example, global pooling combining a global average pooling (GAP) and a global max pooling (GMP) may be applied. GAP has strong normalization capabilities and can consider the overall nature of each feature map. GMP is robust to noise because it selects the element with the largest energy value in the feature map, and can obtain strong local properties of the feature.

Feature map set Suppose, Denotes the i th feature map. Each feature map is a two-dimensional matrix of size h by w. Is It can be represented as a set consisting of. A set of values generated by applying global pooling to each feature map Set to, Since is the result of applying the global pooling to the i-th feature map, the result of applying GAP and GMP to the i-th feature map is as shown in [Equation 10], and the result of applying GMP to the i-th feature map is [Equation 10] 11].

Therefore, [Equation 7], [Expression 8] and the first part of the convolution characteristics obtained respectively from the equation (9)] information _{(f Local1 (X L))} , the second portion convolution feature information (f _Local2 (X _LM ) ) and the third partial convolution characteristic information ( f _Local3 (X _MH ) ), respectively, are the result of applying both GAP and GMP as shown in [Equation 12], [Equation 13] and [Equation 14]. Can be expressed.

G () on the right side of [Equation 12], [Equation 13] and [Equation 14] is Means.

Convolution feature information to which global pooling has been applied by the global pooling layer 304b is passed through L2-normalization and then to the fully connected layer 306b.

The fully connected layer 306b may generate a plurality of linear concealment feature information by applying a second activation function, eg, a ReLU function, to the plurality of partial convolution feature information generated in the convolution layer 302b. For example, the plurality of linear concealment feature information may include first linear concealment feature information ( X _LAS1 ), second linear concealment feature information ( X _LAS2 ), and third linear concealment feature information ( X _LAS3 ). Equation 15, Equation 16 and Equation 17 may be represented.

In Equation 15, Equation 16 and Equation 17, the right side f () represents the ReLU function, which is the second activation function, and W _LiSj, b _LiSj represents the i th layer and j of the convolutional neural network. The weight and the bias of the first stream may be represented, respectively.

Each linear concealment feature information generated is passed to output layer 310b.

The output layer 310b may output a result of learning a second parameter by applying a third activation function, for example, a sigmoid function, to the linear concealment feature information generated in the fully connected layer 306b. The output layer (310b) the result (output) output by the can be expressed as: [Equation 18].

In Equation 18, f () on the right side represents a sigmoid function which is a third activation function, and W _LiSj, denotes the i th layer and the j th weight. The bias is not applied to the operation of the output layer 310b.

As such, the respective result values output through the output layer 310b may be summed.

4 is a diagram exemplarily illustrating a convolution operation on an entire spectrogram and a plurality of partial spectrograms of an arbitrary sound signal performed by the learning model unit 300 of FIG. 1.

In FIG. 4, the convolution operation on the entire spectrogram and the plurality of partial spectrograms for an arbitrary sound signal may include a third convolution that classifies the sound signal by applying the first parameter determined in FIG. 2 and the second parameter determined in FIG. 3. May be performed in a neural network.

Here, the third convolutional neural network applies the first parameter and the second parameter to generate a convolution characteristic information for all spectrograms of an arbitrary sound signal and convolution characteristic information for a plurality of partial spectrograms. Layer 302c, global pooling layer 304c that applies global pooling to convolutional feature information for the entire spectrogram and convolutional feature information for the partial spectrogram, and convolution of the entire spectrogram with global pooling The fully connected layer 306c generating linear concealment feature information by applying the first activation function to the convolution feature information of the feature information and the partial spectrogram, and applying the second activation function after merging the linear concealment feature information. Classification of acoustic signals by applying a third activation function to the merged layer 308c and the result of applying the second activation function And an output layer 310c for outputting an output.

The convolution layer 302c may generate feature information by applying the first parameter and the second parameter trained in FIG. 2 and FIG. 3 to the convolution operation for the entire spectrogram and the plurality of partial spectrograms. have. Each feature information may be represented by f _global , f _local1 , f _local2 , and f _local3 .

These feature information may be connected to each other to be generated as one deep feature information, and the generated deep feature information may be represented by Equation 19 in consideration of both the overall characteristics and the partial characteristics of the acoustic signal.

The generated deep feature information may be delivered to the global pooling layer 304c, which may be delivered to the fully connected layer 306c after applying global pooling to the feature maps, undergoing L2-normalization, and the like. have.

The fully connected layer 306c may apply a first activation function, eg, a ReLU function, to the generated deep feature information to generate linear concealment feature information having a full feature and a plurality of partial features, as Equation 20] can be expressed as

In Equation 20, f () on the right side represents a ReLU function, which is a first activation function, and W _{total and} b _total may represent weights and biases of the fully connected layer 306c, respectively.

Each of the generated linear hidden feature information may be transferred to the merging layer 308c. In the merging layer 306c, the respective characteristic values from the fully connected layer 306c are concatenated to combine the overall characteristic information and the partial characteristic information. It is possible to generate integrated feature information having all. A second activation function, for example a ReLU function, may be applied when merging the linear concealment feature information in the merging layer 308c.

The resulting value to which the second activation function is applied in the merge layer 308c may be transferred to the output layer 310c.

The output layer 310c may output a classification result of the acoustic signal by applying a third activation function, for example, a sigmoid function, to the result value transferred to the merging layer 308c. The result (output) to be output by this output layer (310c) may be expressed as: [Equation 21].

In Equation 21, f () on the right side represents a sigmoid function, which is a third activation function, and W _final represents a weight. The bias is not applied to the operation of the output layer 310c.

On the other hand, the cost function used to optimize each deep neural network can be expressed as shown in Equation 22 below.

In the case of the second convolutional neural network generating a plurality of partial feature information, the sum of the output values of each stream is set as a loss function. In Equation 22, f () means the cross-entropy of the predicted result and the actual label.

FIG. 5 is a numerical comparison of sound signal classification results by a typical single stream convolution operation and sound signal classification results by a multi-stream convolution operation according to an embodiment of the present invention.

The data set used in the experiment for the evaluation of each sound clip is the “Urban Sound8K dataset”, which contains a total of 8,732 sound clips and consists of a total of 10 types of environmental sounds: “Air conditional”, “Car horn”, “Children playing”, “Dog bark”, “Drilling”, “Engine idling”, “Gun shot”, “Jackhammer”, “Siren”, “Street music”.

Each sound clip has a time length of less than 4 seconds and was created by recording environmental sounds that occurred in a real urban environment. As a method for evaluation, the “probability voting” method was applied. This method splits a sound clip into a certain size and averages the results of each of the pieces to evaluate the estimated sound clip. In the embodiment of the present invention, the classification of the environmental acoustics with the limited technology shows that the average of the 10 environmental acoustic recognition results is 83%, which is 73% (79% with the data amplification technology), which shows better classification performance than the prior art. have.

6 is a diagram illustrating a confusion matrix by a multi-stream convolution operation according to an embodiment of the present invention. In the confusion matrix of FIG. 6, the horizontal axis represents an output value and the vertical axis represents an answer.

According to an embodiment of the present invention as described above, a full spectrogram of an acoustic signal is generated, a plurality of partial spectrograms including a predetermined frequency section of the entire spectrogram are generated, and a full spectrogram and a plurality of partial spectrograms are generated. By classifying acoustic signals by applying multi-stream convolutional neural networks to grams, acoustic signal classification accuracy can be improved, and global pooling can be applied to deeper convolution deep learning results of whole spectrograms and partial spectrograms. Implemented to reduce the amount of computation and overfit of.

On the other hand, the combination of each block in the accompanying block diagram and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment are described in each block of the block diagram. It creates a means to perform the functions.

These computer program instructions may be stored on a computer usable or computer readable recording medium (or memory) or the like that may be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, thereby making the computer available. Alternatively, instructions stored on a computer readable recording medium (or memory) may produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram.

In addition, computer program instructions may be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to generate a computer or other program. Instructions that perform possible data processing equipment may also provide steps for performing the functions described in each block of the block diagram.

In addition, each block may represent a portion of a module, segment, or code that includes at least one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative embodiments, the functions noted in the blocks may occur out of order. For example, the two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the corresponding function.

10: acoustic signal classification device
100: pretreatment unit
200: spectrogram classification
300: learning model
302a, 302b, 302c: convolutional layer
304b, 304c: global pooling layer
306a, 306b, 306c: Fully Connected Layer
308c: merge layer
310a, 310b, 310c: output layer
400: sound classification unit

Claims (17)

A method of classifying acoustic signals using a multi-stream convolution neural network,
Generating a full spectrogram of the acoustic signal,
Generating a plurality of partial spectrograms from the full spectrogram, each of the partial spectrograms comprising a predetermined frequency interval of the full spectrogram;
A first parameter used for a convolution operation performed on the first convolution neural network for the entire spectrogram, and a second parameter used for a convolution operation performed on a second convolution neural network for the plurality of partial spectrograms Determining,
The determining step,
Predicting a first result using the first convolutional neural network for the entire spectrogram of the acoustic signal;
Predicting, for the plurality of partial spectrograms of the acoustic signal, a second result using the second convolutional neural network;
Learning the first parameter using the first result, and learning the second parameter using the second result.
Sound signal classification method. delete The method of claim 1,
Global pooling is applied to feature information generated by a convolution operation in the second convolutional neural network for the plurality of partial spectrograms.
Sound signal classification method. The method of claim 3, wherein
The applied global pooling is Global Average Pooling (GAP) and Global Max Pooling (GMP).
Sound signal classification method. The method of claim 1,
Classifying the sound signal by applying the first parameter and the second parameter to the multi-stream convolutional neural network for the full spectrogram and the plurality of partial spectrograms of an arbitrary sound signal.
Sound signal classification method. The method of claim 5,
Classifying the sound signal,
Generating feature information on the entire spectrogram by performing a convolution operation on the entire spectrogram;
Generating feature information on the plurality of partial spectrograms by performing a convolution operation on the plurality of partial spectrograms;
Applying a first activation function to the feature information for the entire spectrogram and the feature information for the plurality of partial spectrograms;
Applying a second activation function after merging feature information on the entire spectrogram to which the first activation function is applied and feature information on the plurality of partial spectrograms;
Classifying the sound signal by applying a third activation function to a result of applying the second activation function;
Sound signal classification method. The method of claim 6,
Global pooling is applied to the feature information on the entire spectrogram and the feature information on the plurality of partial spectrograms.
Sound signal classification method. The method of claim 6,
The first activation function and the second activation function are ReLU (Reectified Linear Unit) functions, and the third activation function is a sigmoid function.
Sound signal classification method. The method of claim 1,
Generating the plurality of partial spectrograms after expanding the size of frequency bins of the entire spectrograms;
Sound signal classification method. A system for classifying acoustic signals using a multi-stream convolutional neural network,
A preprocessor for generating a full spectrogram of the acoustic signal,
A spectrogram classifier for generating a plurality of partial spectrograms from the full spectrogram, each of the partial spectrograms comprising a predetermined frequency interval of the full spectrogram;
A learning model unit for determining a first parameter used in a convolution operation performed on the entire spectrogram and a second parameter used in a convolution operation performed on the plurality of partial spectrograms,
The learning model unit,
A first convolutional neural network predicting a first result with respect to the entire spectrogram of the acoustic signal, and learning a first parameter using the first result;
A second convolutional neural network for predicting a second result with respect to the plurality of partial spectrograms of the acoustic signal and learning a second parameter using the second result;
A third convolutional neural network for classifying the acoustic signal by applying the first parameter and the second parameter to the multi-stream convolutional neural network for all spectrograms and a plurality of partial spectrograms of an arbitrary acoustic signal;
System for classifying acoustic signals. delete The method of claim 10,
The first convolutional neural network,
A convolution layer for generating convolution characteristic information for the entire spectrogram by applying a first activation function to the entire spectrogram;
A fully connected layer generating linear concealment feature information by applying a second activation function to the convolution feature information;
An output layer for outputting a result of learning the first parameter by applying a third activation function to the linear concealment feature information;
System for classifying acoustic signals. The method of claim 10,
The second convolutional neural network,
A convolution layer for generating convolution characteristic information for the plurality of partial spectrograms by applying a first activation function to the plurality of partial spectrograms;
A global pooling layer applying global pooling to the convolutional feature information;
A fully connected layer generating linear concealment feature information by applying a second activation function to the convolution feature information to which the global pooling is applied;
And an output layer for outputting a result of learning the second parameter by applying a third activation function to the linear concealment feature information.
System for classifying acoustic signals. The method according to claim 12 or 13,
The first activation function is a leaky linear linear unit (LReLU) function, the second activation function is a rectified linear unit (ReLU) function, and the third activation function is a sigmoid function.
System for classifying acoustic signals. The method of claim 10,
The third convolutional neural network,
A convolution layer applying the first parameter and the second parameter to generate convolution feature information for the entire spectrogram of an arbitrary sound signal and convolution feature information for the plurality of partial spectrograms;
A global pooling layer applying global pooling to the convolution feature information for the entire spectrogram and the convolution feature information for the partial spectrogram;
A fully connected layer generating linear concealment feature information by applying a first activation function to the convolution feature information of the entire spectrogram to which the global pooling is applied and the convolution feature information of the partial spectrogram;
A merging layer for applying a second activation function after merging the linear hidden feature information;
And an output layer configured to apply a third activation function to a result of applying the second activation function to output a classification result of the acoustic signal.
System for classifying acoustic signals. The method of claim 15,
The first activation function and the second activation function are ReLU (Reectified Linear Unit) functions, and the third activation function is a sigmoid function.
System for classifying acoustic signals. Generating a full spectrogram of the acoustic signal,
Generating a plurality of partial spectrograms from the full spectrogram, each of the partial spectrograms comprising a predetermined frequency interval of the full spectrogram;
Determining a first parameter used in a convolution operation performed on the entire spectrogram and a second parameter used in a convolution operation performed on the plurality of partial spectrograms, wherein the determining comprises: Predicting a first result using a first convolutional neural network for the entire spectrogram of the acoustic signal, and predicting a second result using a second convolutional neural network for the plurality of partial spectrograms of the acoustic signal And learning the first parameter using the first result, and learning the second parameter using the second result.
Computer-readable recording media.