KR20180101057A - Method and apparatus for voice activity detection robust to noise - Google Patents

Abstract

The present invention relates to a method for voice activity detection robust to noise and an apparatus thereof. According to an embodiment of the present invention, the method for voice activity detection includes: a step of generating a first matrix representing the waveform and spectral characteristics of an input audio signal; a step of generating a second matrix including one or more feature vectors based on the first matrix; and a step of determining whether a speech candidate activity corresponds to a speech activity based on a harmonic-to-percussive ratio (HPR) value determined based on the second matrix, with regard to the speech candidate activity determined based on the first matrix.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for robust voice-

The present disclosure is directed to voice detection, and more particularly, to a noise-robust voice interval detection method and apparatus.

Recently, various technologies for processing multimedia contents have been developed in accordance with the development of communication technology. In particular, a voice activity detection (VAD) technique for detecting the presence or absence of voice in an audio signal may reduce power consumption by preventing a voice processing apparatus from generating or transmitting unnecessary data for a non-voice section, It can also be used for various purposes such as sound editing and preprocessing of voice data retrieval methods.

The most problematic in the speech segment detection technique is various noise included in the audio signal. Particularly, multimedia contents include various audio signals such as voice, music, sound effect, and noise, so it is required to improve the performance of voice section detection.

Recently, it has been studied to apply a machine learning technique including deep learning for voice section detection. Although the deep learning technique generally shows higher performance than the existing speech section detection technology, there is a problem that the performance depends on the amount and quality of samples of multimedia contents including voice. Especially, in case of multimedia contents including various audio signals such as voice, music, sound effect, noise, etc., it may include sporadic noise that occurs temporarily at a certain time. This sporadic noise has a relatively low number of audio signals compared with audio signals such as music and music. Therefore, in the case of sporadic noise, it is difficult to construct a probability model because of lack of learning samples. Therefore, it is difficult to expect a high performance for an audio signal including sporadic noise.

As described above, there is a demand for a technology capable of high-performance speech segment detection for audio signals including sporadic noises, but no specific measures have been made.

SUMMARY OF THE INVENTION The present invention is directed to a method and apparatus for detecting noise-robust speech intervals in an audio signal including sporadic noise.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

According to an aspect of the present disclosure, there is provided a method of detecting a speech interval, the method comprising: generating a first matrix representing a waveform and spectral characteristics of an input audio signal; Generating a second matrix including one or more feature vectors based on the first matrix; And determining whether or not the speech candidate section corresponds to a speech section based on a Harmonic-to-Percussive Ratio (HPR) value determined based on the second matrix, for a speech candidate section determined based on the first matrix Based on the result of the determination.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, a method and apparatus for detecting noise-robust speech intervals in an audio signal including sporadic noise can be provided.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

1 is a block diagram showing an exemplary configuration of a speech interval detection apparatus according to the present disclosure;
2 is a diagram showing an example of a spectrogram of an audio signal having a harmonic component.
3 is a diagram showing an example of a spectrogram of an audio signal having a perch only component.
4 is a diagram showing an example of a spectrogram for sporadic noise and speech signals.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising "or" having "another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

The definitions of the terms used in the present disclosure are as follows.

- sporadic noise: transient noise. Sporadic noise includes impulsive noise and sudden noise.

Impulse noise: Noise present for a short time. Impulsive noise has very large amplitude and simple waveform characteristics.

- Sudden Noise: Noise that exists for a relatively long time compared to impulsive noise. Sudden noises have a complexity in signal components with similar amplitude to speech.

- Harmonics-to-Percussive Ratio (HPR): The ratio of the harmonic to percussive components of the audio signal.

- Harmonic component: A component that has a pitch (or pitch) in the audio signal. The harmonics component is a component that is maintained for a predetermined time period and has a property that is continuous in the time domain by periodicity. For example, the sound of stringed instruments.

- Percussive component: A component in the audio signal that has no pitch (or pitch). The percussive component is a component having continuous properties in the frequency domain due to the instantaneous and non-periodic properties. For example, the sound of a percussion instrument.

The definitions of the above terms are merely illustrative, and the scope of the present invention is not limited by these definitions. For example, the definition of such terms does not exclude the basic meaning that is easily understood by those skilled in the art.

Various examples of the present disclosure for a noise-robust voice interval detection method and apparatus are described below.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The examples of this disclosure relate to a method for detecting a speech interval using the characteristics of a speech signal. For example, according to the present disclosure, noise can be detected in an audio signal in a multimedia content, and a noise-detection method robust against noise can be provided.

The most problematic in the detection of the speech interval is the various noise included in the audio signal. Especially, in case of multimedia contents, various signals such as music, sound effect and noise are included. Recently, voice segment detection technology using deep learning technique has been actively studied all over the world.

Deep learning is a technique of machine learning, which performs a high level of abstractions through a combination of several nonlinear transformation techniques. Deep learning, for example, can be defined as a set of algorithms that perform the task of extracting core content or functionality in a large amount of data or complex data. As such, deep learning is a field of machine learning that teaches computers how people think.

Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), and Deep Belief Networks (DBN) are examples of deep-learning algorithms that can be applied to computer vision, speech recognition, natural language processing, speech processing, . The DNN algorithm is composed of several layers and has excellent performance for modeling nonlinear relations among speech features. Therefore, it is possible to use DNN Many attempts have been made to apply algorithms. For example, by learning a probability model for voice segment detection using a conventional voice / music database (DB), YouTube contents, movie contents, Defense Advanced Research Projects Agency (US) DARPA) project in Korea.

As shown in the result report of the deep learning based speech segment detection test, the performance of the deep learning based speech segment detection may be degraded when there are music, sound effects, various noises, and the like as multimedia contents. This degradation is noticeable in the case of sporadic noise.

Here, the noise can be divided into a noise distributed uniformly over the entire section of the signal and a sporadic noise temporally present in a specific section. Among these, sporadic noise can be divided into impulsive noise and sudden noise. Impulsive noise exists for a short period of time, and its amplitude is very large and has a simple waveform. However, sudden noise exists for a relatively long time compared with impulsive noise, and the amplitude and the signal components are similar in complexity to the voice.

Such sporadic noise is relatively ineffective compared to voice signals and music signals in multimedia contents, and training for a deep running probability model such as DNN and CNN is not properly performed. In other words, in the deep learning technique, a large number of audio samples having similar characteristics are acquired and collected to learn a probability model. However, sudden noise has amplitude and signal components similar to those of a voice signal, Therefore, there is a problem that the learning required in the deep learning technique is difficult to be properly performed.

In order to solve such a problem, in the examples of the present disclosure, when judging whether or not the audio signal is speech or non-speech, the speech signal includes a sudden noise similar in distribution type of frequency spectrum to the audio signal A method for determining whether sporadic noise is present will be described. Accordingly, it is possible to distinguish whether speech is strong or not and to detect a speech section or a non-speech section.

1 is a block diagram showing an exemplary configuration of a speech interval detection apparatus according to the present disclosure;

The audio analysis unit 110 may output the first matrix based on an analysis result of the input audio signal.

For example, the input audio signal may be PCM (Pulse Code Modulation) data, and the first matrix may be a matrix representing characteristics of a waveform and a spectrum of the input audio signal.

As a concrete example, the audio analysis unit 110 may perform a STFT (Short Time Fourier Transform) on the PCM data to output a complex spectrogram matrix.

The scope of the present disclosure is not limited to the example described above, and the first matrix may include data representing time-frequency characteristics of the audio signal.

The feature extraction unit 120 may output a second matrix for an audio signal based on a first matrix input from the audio analysis unit 110. [

For example, the second matrix may be a matrix of audio feature vector (s). In particular, the second matrix may comprise one or more feature vectors constructed based on the spectrogram matrix. As a more specific example, the feature vector may be constructed using spectrogram values, or may be configured using various subband processing based audio features of the TF (Time-Frequency) domain.

In the case of constructing the feature vector matrix using the spectrogram value, the number of rows of the feature vector matrix corresponds to the order of FFT (Fast Fourier Transform), and the number of columns of the feature vector matrix may correspond to the number of audio frames .

Mel-Frequency Cepstral Coefficients (MFCC) can be used as an example of subband processing of the TF domain. MFCC value is obtained by calculating the coefficient by taking the TF component of the first matrix output from the audio analyzer 110 by performing a cosine transform on the log power spectrum in the frequency domain of Mel-scale have. In this case, the number of rows of the feature vector matrix corresponds to the number of Mel filter banks, and the number of columns of the feature vector matrix may correspond to the number of audio frames.

The scope of the present disclosure is not limited to the example described above, and the second matrix may include one or more feature vectors configured based on the time-frequency characteristics of the audio signal.

The HPR calculation unit 130 may calculate the ratio of the harmonic component to the percussive component in a predetermined time interval and a predetermined frequency interval of the audio signal based on the input first matrix.

The characteristics of the harmonic component and the percussive component will be described with reference to Figs. 2 to 4. Fig.

2 is a diagram showing an example of a spectrogram of an audio signal having a harmonic component.

3 is a diagram showing an example of a spectrogram of an audio signal having a perch only component.

4 is a diagram showing an example of a spectrogram for sporadic noise and speech signals.

The sporadic noise is present for a relatively long time compared to the impulsive noise such as impulsive noise and sudden noise (for example, applause, laughter, cheering, whistling, explosion, etc.) Complex noise), and this sporadic noise has characteristics similar to the percussive signal.

For example, the example of FIG. 2 corresponds to the spectrogram for the violin sound, and the example of FIG. 3 corresponds to the spectrogram for the castanet sound. As shown in the example of FIG. 2, it can be seen that the string sound is mainly composed of a harmonic component in which energy corresponding to various kinds of frequency harmonics appears horizontally and continuously along the time axis. As in the example of FIG. 3, it can be seen that the percussion sound is mainly composed of a percussive component in which energy is vertically concentrated along the time axis at a specific time and energy is continuously distributed evenly over the entire frequency axis.

In the example of FIG. 4, it can be seen that, in a time period corresponding to sporadic noise (for example, laughter), the punsive component is relatively more included than a time period corresponding to voice. In addition, it can be seen that, in the time interval corresponding to the speech, the harmonic components are relatively more included than the time interval corresponding to the sporadic noise.

Therefore, it is possible to determine, using the ratio of the harmonic component and the percussive component (i.e., HPR), whether a particular time interval (e.g., frame) is likely to correspond to a speech interval or is likely to correspond to sporadic noise have.

For example, to compute the HPR, a harmonic component and a percussive component can be separated using a median filter. Specifically, the HPR at a predetermined time interval and a predetermined frequency interval of the audio signal can be calculated.

Here, the predetermined time period may be any one of a time period in which the audio signal exists and a time period in which the audio signal exists, divided by a plurality of time units of the same size. For example, the time unit may be a frame. The scope of the present disclosure is not limited thereto, and the HPR can be calculated for a time unit of arbitrary size.

The predetermined frequency interval may be any one of a whole frequency range in which the audio signal exists, divided by a plurality of frequency units of the same size. For example, the frequency unit may be expressed as a frequency bin. The scope of the present disclosure is not limited thereto, and the HPR can be calculated for a frequency unit of any size.

The median filter is a nonlinear digital filter technique that is often used to remove signal noise in image processing and is typically used to perform high-performance noise cancellation on images prior to high-level processing such as in-image contour detection do. In this disclosure, an intermediate value filter can be used to compute HPR.

For example, it is possible to output a result of applying the filtering by the first median filter to the first matrix as a first filtered first matrix. Specifically, the first intermediate value filter may be a horizontal median filter, and the first filtered first matrix may be expressed as Spect_h to which horizontal median filtering of the spectrogram is applied.

Also, it is possible to output a result of applying the filtering by the second intermediate value filter to the first matrix as a second filtered first matrix. Specifically, the first intermediate value filter may be a horizontal median filter, and the second filtered first matrix may be expressed as Spect_v to which vertical median filtering for the spectrogram is applied.

The HPR for each frequency unit can be calculated using the power values of the filtered spectrograms (i.e., the first filtered spectrogram and the second filtered spectrogram). For example, the HPR for each frequency unit (e.g., frequency bin) may be calculated according to Equation 1 below.

In Equation (1), i is a frame index and j is a frequency bin index. That is, HPR [i, j] represents the HPR in the i-th frame and the j-th frequency bin. Spect_h [i, j] represents the power value in the i-th frame and the j-th frequency bin of the spectrogram to which the first filtering (for example, horizontal median filtering) is applied. Spect_v [i, j] represents the power value in the i-th frame and the j-th frequency bin of the spectrogram to which the second filtering (e.g., vertical median filtering) is applied.

For example, the larger the Spect_h at the i-th frame and the j-th frequency than the Specht_v, the greater the energy of the horizontally filtered component in the TF two-dimensional region as shown in FIG. The fact that the energy of the horizontally filtered component is large can be interpreted that the harmonic component is included more. Therefore, the larger the value of HPR [i, j], the higher the probability that the region corresponding to the i-th frame and the j-th frequency corresponds to the voice section.

Or, as the Spect_h at the i-th frame and the j-th frequency is relatively smaller than Specht_v, the energy of the vertically filtered component in the TF two-dimensional region as shown in FIG. 4 is large. The fact that the energy of the vertically filtered component is large can be interpreted to include more percussive components. Therefore, the smaller the value of HPR [i, j], the higher the probability that the region corresponding to the i-th frame and the j-th frequency corresponds to the non-speech interval (or sporadic noise).

In the speech / non-speech discrimination unit 140, the second matrix output from the feature extraction unit 120 and the HPR value output from the HPR calculation unit 130 may be input.

The speech / non-speech discrimination unit 140 may determine a speech candidate section based on the second matrix and finally determine the speech interval based on the HPR for the speech candidate section.

First, statistical modeling based on information contained in the second matrix (e.g., feature vector) may be used to determine the speech candidate interval.

For example, it is possible to determine whether a speech interval or a non-speech interval for each frame is determined using DNN, and a speech candidate interval for a frame determined as a speech interval. Specifically, the DNN model for determining whether a speech or a non-speech segment is a probabilistic model that has been previously learned (or trained) using an audio database in which a speech segment or a non-speech segment is marked can be used.

When the speech candidate section is determined, the HPR value in the frame corresponding to the speech candidate section can be confirmed from the HPR values of the frame-frequency unit input from the HPR calculation section 130. [ If the determined HPR value is equal to or greater than a predetermined threshold value, the voice candidate section can be finally determined as the voice section. If the determined HPR value is less than the predetermined threshold value, the speech candidate section may be finally determined as a non-speech section (or a sporadic noise section).

Here, the predetermined threshold value may be adjusted to an optimal value based on a history of checking whether a frame determined as a voice interval actually includes voice. Alternatively, the predetermined threshold value may be adjusted to a value desired by the user, corresponding to the sensitivity of the voice interval detection.

An index (e.g., Speech_index) indicating a voice interval is given to the time interval (or frame) finally determined as the voice interval, and the voice interval can be easily identified and retrieved based on the index.

5 is a flowchart for explaining a method of voice interval detection according to the present disclosure.

In step S510, the speech segment detection apparatus may analyze the input audio signal to generate a first matrix (e.g., a spectrogram matrix) expressing characteristics of the waveform and spectrum of the input audio signal.

In step S520, a second matrix (e.g., a feature vector matrix) may be generated based on the first matrix.

In step S530, the HPR may be calculated based on the first matrix. The HPR can be generated in units of time intervals (e.g., frames) and frequency intervals (e.g., frequency bins), respectively.

In step S540, the speech candidate section may be determined based on the second matrix. For example, based on the statistical model, it may be determined whether the feature represented by the feature vector matrix corresponds to a speech interval or a non-speech interval, and thus a speech candidate interval may be determined.

In step S550, it may be determined whether the HPR value for the speech candidate section (for example, a unit corresponding to a specific frame and a specific frequency bin) is less than a predetermined threshold value.

If the HPR value is equal to or greater than the predetermined threshold value, the process proceeds to step S560, where the voice candidate section can be finally determined as the voice section.

If the HPR value is less than the predetermined threshold value, the process proceeds to step S570, where the voice candidate section can be finally determined as the non-voice section.

According to the method and apparatus for detecting a voice section according to the present disclosure, it is possible to detect a voice section and a non-voice section which are robust against noise through comparison of a harmonic component and a pucker's component of an input audio signal, It is possible to improve service quality and efficiency in preprocessing for data retrieval.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to illustrate representative aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

100 ... Voice section detection device
110 ... audio analysis unit
120 ... feature extraction unit
130 ... HPR calculation section
140 ... voice / non-voice discrimination unit

Claims (1)

A method for detecting a speech interval,
Generating a first matrix representing a waveform and spectral characteristics of an input audio signal;
Generating a second matrix including one or more feature vectors based on the first matrix; And
For a speech candidate section determined based on the first matrix, whether or not the speech candidate section corresponds to a speech section, based on a Harmonic-to-Percussive Ratio (HPR) value determined based on the second matrix And determining the voice interval.

Images