KR20210105688A - Method and apparatus for reconstructing speech signal without noise from input speech signal including noise using machine learning model - Google Patents

Abstract

Provided is a speech signal processing method that inputs an amplitude signal of an input speech signal including noise to acquire a mask to be applied to the amplitude signal as output for a pre-trained machine learning model, inputs a first phase signal of a first frequency band among phase signals of the input speech signal to acquire a mask to be applied to a fast Fourier transform (FFT) coefficient as output for the machine learning model, and restores a noise-cancelled speech signal based on the input speech signal and the masks acquired from the machine learning model.

Description

METHOD AND APPARATUS FOR RECONSTRUCTING SPEECH SIGNAL WITHOUT NOISE FROM INPUT SPEECH SIGNAL INCLUDING NOISE USING MACHINE LEARNING MODEL

The embodiments relate to a method and apparatus for reconstructing (reconstructing) a speech signal from which noise has been removed from an input speech signal including noise, and in particular, applying a machine learning model to an amplitude signal and a phase signal constituting the input speech signal. By doing so, the present invention relates to a method and apparatus for reconstructing a denoised voice signal.

Recently, as interest in Internet calls such as VoIP and development and provision of content utilizing other voice/sound signals increases, interest in technology for removing noise from voice signals is also increasing.

A voice signal generally has a bandwidth of 8 kHz or 16 kHz, and a technology for removing noise from the voice signal has also been developed targeting the voice signal having a bandwidth of 8 kHz or 16 kHz. However, it is difficult to apply this noise removal technique to remove noise included in a high-quality voice signal having a bandwidth of 24 kHz or higher (ie, full band). In other words, it is difficult to apply the existing noise removal technology to a service that provides a full-band voice signal, and when the sampling rate is increased to remove noise from such a full-band voice signal, the amount of computation increases rapidly. there is a problem that

In addition, in the case of restoring only the amplitude signal and using the signal containing noise as the phase signal as it is, when the noise is severe (eg, when the SNR is 0 dB or less), the voice signal from which the noise is properly removed. cannot be restored, and in particular, it is difficult to effectively remove noise included in the amplitude signal and the phase signal for a high sampling rate audio signal.

Korean Patent Laid-Open No. 10-2018-0067608 (published on June 20, 2018) describes a method and apparatus for determining a noise signal, and a method and apparatus for removing voice noise.

The information described above is for understanding only, and may include content that does not form a part of the prior art, and may not include what the prior art can present to a person skilled in the art.

One embodiment is to obtain a mask to be applied to the amplitude signal as an output by inputting the amplitude signal of the input voice signal including noise to the pre-trained machine learning (ML) model, and to the corresponding machine learning model , to obtain a mask to be applied to the FFT coefficients of the input speech signal as an output by inputting a first phase signal of a first frequency band among the phase signals of the input speech signal, and based on the mask obtained from the input speech signal and the machine learning model Thus, it is possible to provide a method of processing a voice signal to restore a voice signal from which noise has been removed.

An embodiment, using a machine learning model, uses the FFT coefficients of the input speech signal as a mask for the FFT coefficients of the first phase signal of the first frequency band, which is a low-band (ie, low-frequency band) among the phase signals of the input speech signal. obtain a mask to be applied to , a mask to be applied to a first amplitude signal in a second frequency band among amplitude signals of the input voice signal, and a third frequency band in a third frequency band that is a higher frequency band than the second frequency band among the amplitude signals 2 Acquire a mask to be applied to the amplitude signal, and apply the mask to the phase signal of a high band (that is, high frequency band) other than the first frequency band, which is a low band among the phase signals, and the input speech signal to which the mask obtained by the machine learning model is applied. A method of reconstructing a voice signal from which noise has been removed may be provided.

In one aspect, in a method for processing a noise-containing voice signal using a computer system, an amplitude signal and a phase ) obtaining a signal, inputting the amplitude signal to a pre-trained machine learning model to estimate a mask to be applied to the input speech signal to remove the noise, and inputting the amplitude signal as an output of the machine learning model obtaining a mask to be applied to a signal, inputting a first phase signal of a first frequency band among the phase signals to the machine learning model, and for FFT coefficients of the input speech signal as an output of the machine learning model A method for processing a speech signal is provided, comprising: obtaining a mask to be applied; and reconstructing the speech signal from which the noise has been removed based on the mask obtained from the input speech signal and the machine learning model.

The first phase signal is a signal of a relatively low frequency band among the phase signals, and a phase signal of a frequency band other than the first frequency band among the phase signals may be used as it is in reconstructing the voice signal from which the noise has been removed. .

The step of obtaining a mask to be applied to the amplitude signal includes inputting a first amplitude signal of a second frequency band among the amplitude signals to the machine learning model, and applying the first amplitude signal to the first amplitude signal as an output of the machine learning model. obtaining a first mask to be applied to the signal, dividing a second amplitude signal of a third frequency band that is a larger frequency band than the second frequency band among the amplitude signals into amplitude signals of a plurality of bandwidth sections, the division calculating an average energy for each of the amplitude signals and, for the machine learning model, inputting the calculated average energy, and obtaining a second mask to be applied to the second amplitude signal as an output of the machine learning model may include the step of

The second amplitude signal may be divided into amplitude signals of the plurality of bandwidth sections by dividing the third frequency band in units of a bark scale.

wherein the first mask is an ideal ratio mask (IRM) for the first amplitude signal, the second mask is an IRM for the calculated average energy, and restoring the noise-removed speech signal. In , the first mask may be multiplied by the first amplitude signal, and the second mask may be multiplied by the second amplitude signal.

The obtaining of the first mask may include calculating a predetermined number of Mel-Frequency Cepstral Coefficients (MFCCs) based on the first amplitude signal and using the calculated MFCCs to obtain the first mask. input to the machine learning model.

Acquiring the first mask may include calculating a Zero Crossing Rate (ZCR) based on the first amplitude signal and inputting the calculated ZCR to the machine learning model to obtain the first mask. It may include the step of

The input voice signal is a signal having a bandwidth of 24 kHz, the first frequency band is a band of 0 or more and less than 4 kHz, the second frequency band is a band of 0 or more and less than 8 kHz, and the third frequency band is 8 kHz or more. The band may be less than 24 kHz.

The step of restoring the noise-removed speech signal may include estimating the noise-removed amplitude signal by multiplying the amplitude signal and a mask to be applied to the amplitude signal, and adding an FFT coefficient of the input speech signal and the first phase signal. estimating a noise removal phase signal of the first frequency band by multiplying a mask to be applied to Restoring the FFT coefficient of the noise-removed voice signal based on the phase signal of the frequency band and performing Inverse Fast Fourier Transform (IFFT) based on the restored FFT coefficient to restore the noise-removed voice signal may include the step of

The input voice signal may be a signal having a bandwidth of 24 kHz, the first frequency band may be a band of 0 or more and less than 4 kHz, and a frequency band other than the first frequency band may be a band of 4 kHz or more and less than 24 kHz.

The machine learning model may be configured as a fully connected layer.

The machine learning model is a model further trained for estimating Voice Activity Detection (VAD) information for distinguishing between speech and silence, and the method for processing the speech signal includes the input as an output of the machine learning model. The method may further include obtaining VAD information for the voice signal, wherein the VAD information for the input voice signal may be used in frequency restoration of the voice signal from which the noise has been removed.

The mask to be applied to the first phase signal is a complex ideal ratio mask (CIRM) with respect to the FFT coefficient of the first phase signal of the first frequency bandwidth, and is used to restore the voice signal from which the noise has been removed. In the step, the CIRM may be multiplied by an FFT coefficient of the input speech signal.

The amplitude signal and the phase signal may be signals in units of predetermined frequency bins, and the voice signal from which the noise has been removed may be restored in units of the frequency bins.

The input voice signal input to the machine learning model may consist of a plurality of frames.

In another aspect, a computer system for processing a voice signal including noise, comprising at least one processor implemented to execute computer-readable instructions, wherein the at least one processor comprises: Fast Fourier Transform (FFT) ) obtained by performing a magnitude signal and a phase signal from an input speech signal containing noise, and pre-trained machine learning to estimate a mask to be applied to the input speech signal to remove the noise. For a model, input the amplitude signal, obtain a mask to be applied on the amplitude signal as an output of the machine learning model, and for the machine learning model, obtain a first phase signal of a first frequency band among the phase signals input, obtain a mask to be applied to the FFT coefficients of the input speech signal as an output of the machine learning model, and generate the noise-removed speech signal based on the input speech signal and the mask obtained from the machine learning model A computer system for restoring is provided.

In another aspect, in a method of processing a noise-containing voice signal using a computer system, an amplitude signal and a phase from an input voice signal containing noise subjected to FFT (Fast Fourier Transform) obtaining a phase signal, inputting the phase signal to a pre-trained machine learning model to estimate a mask to be applied to the input speech signal to remove the noise, and as an output of the machine learning model obtaining a mask to be applied to the FFT coefficients of the input speech signal as a mask for the FFT coefficients of a first phase signal of a first frequency band among the phase signals; inputting the amplitude signal to the machine learning model; A first mask to be applied to a first amplitude signal in a second frequency band among the amplitude signals as an output of the machine learning model, and a second amplitude in a third frequency band that is a frequency band larger than the second frequency band among the amplitude signals obtaining a second mask to be applied to a signal; and a product of a phase signal of a frequency band other than the first frequency band among the phase signals by an FFT coefficient of the mask and the input voice signal, the first mask and the A method for processing a voice signal is provided, comprising the step of reconstructing the noise-removed voice signal based on the product of the first amplitude signal and the product of the second mask and the second amplitude signal.

Among the input voice signals, the amplitude signal and phase signal belonging to the low-band frequency band are restored using a machine learning model, but the amplitude signal belonging to the high-band frequency band is restored using the machine learning model in units of the bark scale. By restoring and using the phase signal belonging to the high frequency band as it is, it is possible to reduce the amount of computation required to restore the noise-removed audio signal.

As a machine learning model for reconstructing a denoised speech signal, the amount of computation required for reconstructing a denoised speech signal by using a Deep Neural Network (DNN) composed only of fully connected layers. can reduce

By reducing the amount of computation required to remove noise from a high-quality voice signal such as a full-band voice signal, it is possible to provide a noise removal method that can be implemented in a device such as a mobile terminal.

1 illustrates a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.
FIG. 2 illustrates a structure of a computer system for reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.
3 is a flowchart illustrating a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.
4 is a flowchart illustrating a method of obtaining a mask to be applied to an amplitude signal using a machine learning model, according to an example.
5 is a flowchart illustrating a method of determining a parameter input to a machine learning model in generating a mask to be applied to a first amplitude signal of a second frequency band among amplitude signals using a machine learning model, according to an example; am.
6 is a flowchart illustrating a method of reconstructing a noise-removed speech signal using an input speech signal and a mask from a machine learning model, according to an example.
7 illustrates a mask estimated by a machine learning model, according to an example.
8 illustrates a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an example.
9 shows parameters input to the machine learning model and the structure of the machine learning model, according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

1 illustrates a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.

Referring to FIG. 1 , a method of obtaining a voice signal 120 restored by removing noise from an input voice signal 105 including noise and a voice signal will be described.

The input audio signal 105 may be, for example, a full-band audio signal having a bandwidth of 24 kHz (or more). The input voice signal 105 may be a signal having a sampling rate of 48 kHz.

The method of processing the input voice signal 105 of the embodiment may be performed by the computer system 100 to be described later.

The input speech signal 105 may be subjected to Fast Fourier Transform (FFT), and an amplitude signal representing the amplitude of the input speech signal 105 from the input speech signal 105 on which the FFT has been performed. and a phase signal indicating a phase may be obtained.

As shown, in an embodiment, an amplitude signal belonging to a low band (eg, a bandwidth of less than 8 kHz) and a phase signal belonging to a low band (eg, a bandwidth of less than 4 kHz) may be reconstructed using the machine learning model 110 . . On the other hand, the amplitude signal belonging to the high band (eg, bandwidth of 8 kHz or more) may be divided in units of Bark scale, and the average energy of the divided amplitude signal may be input to the machine learning model 110 and restored. In addition, a phase signal belonging to a high band (eg, a bandwidth of 4 kHz or more) may be directly used to reconstruct the voice signal 120 without special processing. In addition, along with the amplitude signal belonging to the low band, MFCC (Mel frequency cepstral coefficient)(s) generated based on the amplitude signal belonging to the corresponding low band may be input to the machine learning model 110 as a parameter.

The machine learning model 110 may be implemented, for example, through an artificial neural network (eg, CNN and/or DNN).

The output from the machine learning model 110 may be a mask to apply to the input speech signal 105 (or at least a portion of the input speech signal 105 ). The voice signal 120 may be restored using the input voice signal 105 to which the mask is applied and the phase signal belonging to the high band. For example, the FFT coefficients of the speech signal 120 may be reconstructed using the input speech signal 105 to which the mask is applied and the phase signal belonging to the high band, and inverse fast Fourier transform (IFFT) is performed to the speech signal 120 . can be restored.

In the embodiment, a speech signal from which noise is removed using an amplitude signal belonging to a high band (for example, compared to a case where an amplitude signal belonging to the high band and a phase signal belonging to the high band are directly input to the machine learning model 110) The amount of computation for reconstructing (120) and the amount of computation for reconstructing the noise-removed voice signal 120 using a phase signal belonging to a high band may be remarkably reduced.

Therefore, although the computer system 100 is an electronic device such as a mobile terminal, and does not correspond to a computing device such as a high-performance PC or server, in the embodiment, a full-band voice signal ( 105) can be effectively removed.

In an embodiment, as shown, only a phase signal of, for example, a 0-4 kHz bandwidth corresponding to a low band may be set as an input of the machine learning model 110 . According to the results of the listening evaluation experiment, even when the amplitude signal is clean, if the phase signal contains noise, the corresponding voice signal may be perceived by the user as containing noise (noisy). However, even if the phase signal includes noise, if the phase signal of the 0 to 4 kHz bandwidth does not include noise, the corresponding voice signal may be perceived as clean by the user. Accordingly, in the embodiment, only a phase signal of a bandwidth corresponding to a low band having a dominant influence on noise felt by a user may be used as an input of the machine learning model 110 . Accordingly, the amplitude of the network of the machine learning model 110 can be reduced.

In addition, in the embodiment, according to the psychoacoustics theory, the amplitude signal of the high band (eg, 8 to 24 kHz bandwidth) corresponding to the voice signal perceived by the user with low resolution is divided into Bark scale units, and the separated By using the average energy of the amplitude signal as an input parameter to the machine learning model 110 , it is possible to reduce the amount of computation in the machine learning model 110 for the high-band amplitude signal.

As such, according to the psychoacoustic theory, the user feels the sound in a specific bandwidth section with respect to the high-bandwidth amplitude signal, so even if the voice signal is restored in the specific bandwidth section, the user does not experience deterioration in sound quality or noise. may not feel it. In addition, with respect to the high-band phase signal, the user may not feel the signal change well. Therefore, in the case of the embodiment, a similar result can be obtained even with a smaller amount of computation compared to the method of reconstructing the full-band 24 kHz voice signal.

A more specific method of reconstructing the noise-removed voice signal 120 by processing the noise-containing input voice signal 105 will be described in more detail with reference to FIGS. 2 to 9 to be described later.

FIG. 2 illustrates the structure of a computer system for reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.

The illustrated computer system 100 may correspond to the computer system 100 described above with reference to FIG. 1 . The computer system 100 may be an electronic device in which a lightweight inference model (eg, the machine learning model 110) for removing noise from the input speech signal 105 is mounted. Alternatively, unlike shown, the computer system 100 uses a machine learning model 110 existing in an electronic device or server external to the computer system 100 to remove noise from the input voice signal 105 voice signal ( 120) may be a device for obtaining. In this case, the computer system 100 may acquire the voice signal 120 through communication with an external electronic device or server.

The computer system 100 is, for example, a personal computer (PC), a laptop computer, a smart phone, a tablet, a wearable computer, an Internet of Things (Internet Of Things) device, etc. may include. For example, the computer system 100 is a device such as a mobile terminal, and may not correspond to a computing device such as a high-performance PC or server.

The computer system 100 may include a communication unit 210 and a processor 220 . The computer system 100 may include a microphone 230 for receiving a voice signal 105 from a user, and a speaker 240 for outputting a noise-removed voice signal 120 . The microphone 230 may generate a voice signal having a full-band bandwidth from a voice input from the user or from the outside. The speaker 240 may be configured to output a voice signal having a bandwidth of a full band.

Also, although not illustrated, the computer system 100 may further include a display for displaying information input from a user and/or information/content provided according to a user's request.

The communication unit 210 may be a device for the computer system 100 to communicate with other servers or other devices. In other words, the communication unit 210 transmits/receives data and/or information to/from other servers or other devices, such as a network interface card, a network interface chip, and a networking interface port of the computer system 100, or a hardware module or network device driver. (driver) or a software module such as a networking program.

The processor 220 may manage components of the computer system 100 and execute programs or applications used by the computer system 100 . For example, the processor 220 may obtain the voice signal 105 input through the microphone 230 or pre-input, and process the acquired voice signal 105 using the machine learning model 110 . and may generate the voice signal 120 from which noise is removed from the voice signal 105 . The processor 220 may process an operation required to execute a program or application required to perform the above operation, process data, and the like. The processor 220 may be at least one processor of the computer system 100 or at least one core within a processor.

Although not shown, computer system 100 may include memory. The memory is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the non-volatile mass storage device may be separated from the memory and included as separate permanent storage devices. Also, an operating system and at least one program code may be stored in the memory. These software components may be loaded from a computer-readable recording medium separate from the memory. The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory through the communication unit 210 rather than a computer-readable recording medium.

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The command may be provided to the processor 220 by the memory or communication unit 210 . For example, the processor 220 may be configured to execute a received instruction according to a program code loaded into the memory. By the operation of the processor 220 , the computer system 100 may generate the voice signal 120 from which noise is removed from the voice signal 105 .

As an example, the processor 220 may obtain an amplitude signal and a phase signal from the input voice signal 105 including noise on which FFT has been performed, and select a mask to be applied to the input voice signal 105 to remove the noise. For a pretrained machine learning model 110 for estimating, an amplitude signal may be input to obtain a mask to be applied to the amplitude signal as an output of the machine learning model 110 . In addition, the processor 220 inputs the first phase signal of the first frequency band among the phase signals to the machine learning model 110 , and as an output of the machine learning model 110 , the FFT coefficient of the input speech signal 105 . It is possible to obtain a mask to be applied to . The processor 220 may reconstruct the noise-removed speech signal 120 based on the input speech signal 105 and the mask obtained from the machine learning model 110 .

The above operation may be performed by at least one software and/or hardware module configured by the processor 220 .

The machine learning model 110 may be a pre-trained model for estimating a mask to be applied to the input speech signal 105 to remove noise included in the input speech signal 105 . The machine learning model 110 may be trained by a set of a plurality of training input voice signals that know the correct answer. The machine learning model 110 may be implemented through DNN. The machine learning model 110 may include a plurality of layers constituting the DNN, and may consist of only a fully connected layer in order to reduce the amount of computation. The machine learning model 110 will be described in more detail with reference to FIG. 9 to be described later.

For a more specific method of processing the input voice signal 105 including noise and reconstructing the noise-removed voice signal 120 using the computer system 100, refer to FIGS. 3 to 9 to be described later. described in more detail.

As described above, the description of the technical features described above with reference to FIG. 1 may be applied to FIG. 2 as it is, and thus a redundant description will be omitted.

In the detailed description to be described later, the operations performed by the components of the computer system 100 or the processor 220 or the operations performed by the applications/programs executed by the computer system 100 or the processor 220 are for convenience of description. It may be described as an operation performed by the computer system 100 .

3 is a flowchart illustrating a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an exemplary embodiment.

In step 310 , the computer system 100 may obtain an amplitude signal and a phase signal from the input voice signal 105 including noise on which FFT has been performed. The input voice signal 105 may be a voice signal input to the computer system 100 through the microphone 230 or a voice signal pre-stored in the computer system 100 . The noise included in the input voice signal 105 may be noise included in the amplitude signal and/or the phase signal, and may represent noise other than the desired voice signal that the user desires to hear.

The input voice signal 105 may be a signal having a bandwidth (full band) of 24 kHz. The sampling rate of the input voice signal 105 may be 48 kHz.

In step 320 , the computer system 100 applies the pretrained machine learning model 110 to the pretrained machine learning model 110 to estimate a mask to be applied to the input speech signal 105 to remove noise contained in the input speech signal 105 . , the amplitude signal of the input speech signal 105 may be input, and a mask to be applied to the amplitude signal may be obtained as an output of the machine learning model 110 .

A specific method of obtaining a mask to be applied to the amplitude signal will be described in more detail with reference to FIGS. 4 and 5, which will be described later.

The amplitude signal (or a parameter obtained from the amplitude signal) of the input speech signal 105 may be an input parameter for performing inference in the machine learning model 110 . By applying (eg, multiplying) the amplitude signal of the input speech signal 105 with a mask output by the machine learning model 110 , an amplitude signal from which noise has been removed can be obtained.

In step 330 , the computer system 100 may input a first phase signal of a first frequency band among the phase signals of the input speech signal 105 to the machine learning model 110 , and the machine learning model ( As an output of 110 , it is possible to obtain a mask to be applied to the FFT coefficients of the input speech signal 105 . The first phase signal of the first frequency band may represent a phase signal corresponding to a relatively low frequency band (low band) among the phase signals. For example, the first frequency band may indicate a band of 0 or more and less than 4 kHz, and the first phase signal may indicate a phase signal of such a band of 0 or more and less than 4 kHz. Meanwhile, among the phase signals, a phase signal of a frequency band other than the first frequency band (ie, a phase signal of a relatively high frequency band (high band)) may be used as it is in reconstructing the noise-free voice signal 120 .

The first phase signal (or a parameter obtained from the first phase signal) may be an input parameter for performing inference in the machine learning model 110 . By applying (eg, multiplying) the mask output by the machine learning model 110 to the FFT coefficients of the input speech signal 105 , a denoised phase signal can be obtained.

Meanwhile, a phase signal of a frequency band other than the first frequency band among the phase signals of the input voice signal 105 may not be input to the machine learning model 110 . That is, for example, a phase signal of a band of 4 kHz or more and less than 24 kHz corresponding to a high band may not be input to the machine learning model 110 . As described above, as a result of the listening evaluation experiment, the phase signal corresponding to the high band does not significantly affect the noise felt by the user, so to reduce the amount of computation in the machine learning model 110 , the phase signal corresponding to the high band is machine learning It may not be input to the model 110 . That is, the phase signal corresponding to the high band may be used to restore the voice signal 120 from which the noise has been removed as it is.

In step 340 , the computer system 100 may reconstruct the noise-removed speech signal 120 based on the mask obtained from the input speech signal 105 and the machine learning model 110 . For example, computer system 100 may apply the mask obtained in step 320 to the amplitude signal, and may apply the mask obtained in step 330 to FFT coefficients of the input speech signal 105 . . The computer system 100 may restore the noise-removed voice signal 120 based on the mask-applied signals and the phase signal corresponding to the high band that is not input to the machine learning model 110 .

A detailed method of reconstructing the noise-removed voice signal 120 will be described in more detail with reference to FIG. 6 to be described later.

According to the embodiment, noise removal processing for the input voice signal 105 input to the computer system 100 may be performed in real time. For example, when a user makes a call using the computer system 100 , noise removal processing for a voice signal corresponding to a voice uttered by the user or a voice signal received by the computer system 100 may be performed in real time. Alternatively, according to an embodiment, noise removal processing may be performed by acquiring (ie, loading) the input voice signal 105 previously stored in the computer system 100 .

Meanwhile, the machine learning model 110 may be a more trained model for estimating Voice Activity Detection (VAD) information for distinguishing between speech and silence from the input speech signal 105 . Although not shown as a step in FIG. 3 , the computer system 100 may obtain VAD information for the input speech signal 105 as an output of the machine learning model 110 . The obtained VAD information on the input voice signal 105 may be used for frequency restoration in restoring the noise-removed voice signal 120 . The VAD information may be used in the machine learning model 110 in order to improve the quality of the reconstructed speech signal 120 (ie, to increase the performance of noise removal).

The machine learning model 110 may include a plurality of layers for estimating VAD information, and an output of each layer for estimating VAD information may be used as an input of a frequency reconstruction network. For example, the output of each layer for estimating the VAD information may be used as an input of the layer for determining the mask obtained in step 320 and the input of the layer for determining the mask obtained in step 330 .

The description of the technical features described above with reference to FIGS. 1 and 2 can be applied to FIG. 3 as it is, and thus the overlapping description will be omitted.

4 is a flowchart illustrating a method of obtaining a mask to be applied to an amplitude signal using a machine learning model, according to an example.

A specific method of obtaining a mask to be applied to an amplitude signal will be described with reference to steps 410 to 440 of FIG. 4 .

In step 410 , the computer system 100 may input a first amplitude signal of a second frequency band among the amplitude signals of the input speech signal 105 to the machine learning model 110 , and the machine learning model ( 100), a first mask to be applied to the first amplitude signal may be obtained. The first amplitude signal of the second frequency band may represent an amplitude signal corresponding to a low band among amplitude signals of the input voice signal 105 . For example, the second frequency band may represent a band of 0 or more and less than 8 kHz, and the first amplitude signal may represent an amplitude signal of such a band of 0 or more and less than 8 kHz.

The first amplitude signal (and/or a parameter obtained from the first amplitude signal) may be an input parameter for performing inference in the machine learning model 110 . By applying (eg, multiplying) the first mask output by the machine learning model 110 to the first amplitude signal, a denoised amplitude signal (ie, the denoised first amplitude signal) will be obtained. can

The first mask may be an ideal ratio mask (IRM) for the first amplitude signal. As described above, this first mask can be applied to the first amplitude signal by multiplying it with the first amplitude signal.

In step 420 , the computer system 100 converts a second amplitude signal of a third frequency band, which is a higher frequency band than the second frequency band, among the amplitude signals of the input voice signal 105 , as an amplitude signal of a plurality of bandwidth sections. can be divided into The second amplitude signal of the third frequency band may represent an amplitude signal corresponding to a high band among the amplitude signals of the input voice signal 105 . For example, the second frequency band may represent a band of 8 kHz or more and less than 24 kHz, and the first amplitude signal may represent an amplitude signal of such a band of 0 or more and less than 8 kHz.

For example, the computer system 100 may divide the second amplitude signal into amplitude signals of a plurality of bandwidth sections by dividing the third frequency band of the second amplitude signal by at least one bark scale unit.

The Bark scale may be a scale based on psychoacoustics. This may be a measure for distinguishing different sounds in order to specifically express the characteristics of the sound in relation to sound characteristics such as amplitude, height, length, tone, and the like of the sound that humans can distinguish using the auditory organ.

According to a psychoacoustics theory, a user can recognize a high-bandwidth (eg, 8-24kHz bandwidth) amplitude signal with low resolution, and the amplitude signal corresponding to the high-bandwidth voice signal is a Bark scale unit. By dividing by , it can be divided into amplitude signals of a plurality of bandwidth sections. The plurality of bandwidth sections may be, for example, 8000-9600Hz, 9600-12000Hz, 12000-15600Hz, 15600-20000Hz, and 20000-24000Hz according to the Bark scale, and the amplitude signal corresponding to the high-bandwidth voice signal is It can be divided into an amplitude signal of each bandwidth section.

In step 430 , the computer system 100 may calculate an average energy for each of the amplitude signals identified in step 420 . The computer system 100 may calculate an average energy (ie, an average of frequency energy) in a bandwidth section corresponding to each amplitude signal for each of the divided amplitude signals.

In step 440 , the computer system 100 may input the average energy calculated in step 430 to the machine learning model 110 , and as an output of the machine learning model 110 , the second amplitude signal It is possible to obtain a second mask to be applied to .

As such, the average energy calculated in step 430 may be an input parameter for performing inference in the machine learning model 110 . By applying (eg, multiplying) the second mask output by the machine learning model 110 to the second amplitude signal, a denoised amplitude signal (ie, denoised second amplitude signal) will be obtained. can

The second mask may be the IRM for the average energy calculated in step 430 . As described above, this second mask can be applied to the second amplitude signal by multiplying it with the second amplitude signal.

In an embodiment, as in step 410 , the low-band amplitude signal may be input to the machine learning model 110 (ie, the low-band amplitude signal will be an input parameter to the machine learning model 110 ). ), and an operation for noise removal may be performed by inference by the machine learning model 110 .

However, as in steps 420 to 440 , with respect to the high-band amplitude signal, the average energy of each of the amplitude signals divided into a plurality of bandwidth sections is calculated, and the calculated average energy is calculated for the machine learning model 110 . By being used as an input parameter, the amount of computation in the machine learning model 110 for a high-band amplitude signal may be reduced.

As described above, the description of the technical features described above with reference to FIGS. 1 to 3 can be applied to FIG. 4 as it is, and thus the overlapping description will be omitted.

5 is a flowchart illustrating a method of determining a parameter input to a machine learning model in generating a mask to be applied to a first amplitude signal of a second frequency band among amplitude signals using a machine learning model, according to an example; am.

Referring to steps 510-1 to 520-2 of FIG. 5, parameters input to the machine learning model 110 based on the first amplitude signal of the second frequency band corresponding to the low band among the amplitude signals are calculated. Explain how to decide.

In operation 510-1, the computer system 100 may calculate a predetermined number of Mel-Frequency Cepstral Coefficients (MFCCs) based on the first amplitude signal of the second frequency band.

In step 520-1, the computer system 100 may input the calculated MFCCs to the machine learning model 110 to obtain a first mask to be applied to the first amplitude signal.

That is, together with the first amplitude signal, MFCCs for the first amplitude signal may be input parameters for performing inference in the machine learning model 110 . A predetermined number (eg, 20) of coefficients for the first amplitude signal may be calculated and input to the machine learning model 110 . The MFCCs may provide information about the shape of the entire frequency of the first amplitude signal.

The MFCC may be a coefficient required for feature vectorizing a speech signal. For example, the MFCC may be a feature of the first amplitude signal.

The MFCC can be calculated (extracted) from the first amplitude signal based on the Mel-scale that recognizes the voice signal of the relatively low frequency band well and considers the characteristic of the cochlea that does not recognize the voice signal of the high frequency band well. can The MFCC may be calculated for each section by dividing the first amplitude signal into a plurality of sections according to the Mel scale.

In operation 510 - 2 , the computer system 100 may calculate a Zero Crossing Rate (ZCR) based on the first amplitude signal of the second frequency band.

In step 520 - 2 , the computer system 100 may input the calculated ZCR to the machine learning model 110 to obtain a first mask to be applied to the first amplitude signal.

That is, together with the first amplitude signal, the ZCR for the first amplitude signal may be an input parameter for performing inference in the machine learning model 110 . ZCR can be calculated by analyzing the first amplitude signal on the time axis. The ZCR may provide information about noise included in the time axis component of the first amplitude signal. ZCR may represent a rate of change of a sign according to a (voice) signal, that is, a rate at which the signal changes. That is, the ZCR may represent a rate at which the signal passes through 0 and the sign of the signal changes.

The computer system 100 may input a first amplitude signal, the MFCCs for the first amplitude signal, and the ZCR for the first amplitude signal to the machine learning model 110 , and as an output of the machine learning model 100 . A first mask to be applied to a 1-amplitude signal may be obtained.

As described above, the description of the technical features described above with reference to FIGS. 1 to 4 can be applied to FIG. 5 as it is, and thus a redundant description will be omitted.

6 is a flowchart illustrating a method of reconstructing a noise-removed speech signal by using an input speech signal and a mask from a machine learning model, according to an example.

A method of reconstructing a noise-removed voice signal will be described in detail with reference to steps 610 to 640 of FIG. 6 .

In step 610, the computer system 100 may estimate the denoised amplitude signal (the denoised amplitude signal) by multiplying the amplitude signal of the input speech signal 105 by a mask to be applied to the amplitude signal. . For example, as described above, the computer system 100 may be configured to multiply the first mask obtained in step 410 with a first amplitude signal, and multiply the second mask obtained in step 440 with a second amplitude signal. The amplitude signal can be estimated as an amplitude signal from which noise has been removed. By multiplying the first mask and the first amplitude signal, the noise-removed first amplitude signal may be estimated, and by multiplying the second mask and the second amplitude signal, the noise-removed second amplitude signal may be estimated. have.

In step 620, the computer system 100 multiplies the FFT coefficients of the input speech signal 105 by a mask to be applied to the first phase signal obtained in step 330 to thereby generate a denoising phase signal ( A phase signal from which noise has been removed) can be estimated. The computer system determines that the noise of the input voice signal 105 is removed based on the phase signal of a frequency band other than the first frequency band among the phase signal and the noise-removed phase signal of the first frequency band estimated by step 620 . A phase signal can be obtained.

The mask to be applied to the first phase signal may be a complex ideal ratio mask (CIRM) with respect to the FFT coefficient of the first phase signal of the first frequency bandwidth. This CIRM may be multiplied by the FFT coefficient of the input speech signal 105 .

In step 630, the computer system 100 generates a noise-removed voice based on a noise-removed amplitude signal, a noise-removed phase signal of a first frequency band, and a phase signal of a frequency band other than the first frequency band among the phase signals. The FFT coefficients of the signal 120 may be reconstructed. In other words, the computer system 100 determines the denoised amplitude signal of the input speech signal 105 obtained by step 610 and the denoised phase signal of the input speech signal 105 obtained by step 620 . Based on the signal, the FFT coefficient of the voice signal 120 from which the noise has been removed may be restored.

In operation 640 , the computer system 100 may reconstruct the noise-removed speech signal 120 by performing Inverse Fast Fourier Transform (IFFT) based on the reconstructed FFT coefficients.

The restored voice signal 120 may be output from the computer system 100 , for example, through the speaker 240 .

In one example, the input voice signal 105 may be a signal having a bandwidth of 24 kHz, and a first frequency band (which is a low band) of the phase signal may be a band of 0 or more and less than 4 kHz, and a first frequency band (which is a high band) of the phase signal. The frequency band other than the frequency band may be a band of 4 kHz or more and less than 24 kHz.

As described above, the description of the technical features described above with reference to FIGS. 1 to 5 can be applied to FIG. 6 as it is, and thus a redundant description will be omitted.

As will be described with reference to FIG. 8 to be described later, the amplitude signal and the phase signal of the input voice signal 105 described in the embodiment may be signals in units of predetermined frequency bins. Also, the voice signal 120 from which noise is removed may be restored in units of such frequency bins. In other words, in an embodiment, a speech signal in the low band (phase signal is 4 kHz or less (less than) / amplitude signal is 8 kHz or less (less than)) can be reconstructed using the machine learning model 110 in units of frequency bins of frequency and , the high-bandwidth (phase signal > 4 kHz (greater than)/amplitude signal > 8 kHz (greater than)) is restored in units of Bark scale (amplitude signal), or the voice signal containing noise is used as it is (phase signal ) can be restored.

On the other hand, as another embodiment, the input voice signal 105 (amplitude signal and/or phase signal) in which the first frequency band to the third frequency band is divided is not directly input to the machine learning model 110 . , when the input speech signal 105 is input to the machine learning model 110, the machine learning model 110 is the input speech signal 105 (amplitude signal and / or phase signal).

As an example, the computer system 100 may run the input speech signal 105 against a pre-trained machine learning model 110 to estimate a mask to be applied on the input speech signal 100 to denoise the input speech signal 105 . A phase signal of 105 may be input, and an FFT coefficient of the input speech signal 105 as a mask for the FFT coefficient of a first phase signal of a first frequency band among the phase signals as an output of the machine learning model 110 A mask (eg, CIRM) to be applied may be obtained. That is, the machine learning model 110 may be targeted to output a mask for the FFT coefficients of the first phase signal of the first frequency band when the phase signal of the input speech signal 105 is input.

In addition, the computer system 100 may input an amplitude signal of the input voice signal 105 to the machine learning model 110 , and as an output of the machine learning model 110 , the amplitude signal of the second frequency band among the amplitude signals. A first mask to be applied to the first amplitude signal and a second mask to be applied to a second amplitude signal in a third frequency band that is a higher frequency band than a second frequency band among the amplitude signals may be obtained. That is, the machine learning model 110 may be targeted to output the first mask and the second mask when the amplitude signal of the input speech signal 105 is input.

The computer system 100 is configured to multiply a phase signal of a frequency band other than the first frequency band among the phase signals of the input voice signal 105 by a mask CIRM and an FFT coefficient of the input voice signal 105, and a first mask. Based on the product of the first amplitude signal and the second mask and the second amplitude signal, the noise-removed voice signal 120 may be restored.

For the input parameters for the machine learning model 110, the acquisition of the mask by the machine learning model 110, and the restoration of the voice signal 120, the description of the technical features described above with reference to FIGS. 1 to 6 can be applied as it is. Therefore, redundant descriptions are omitted.

7 illustrates a mask estimated by a machine learning model, according to an example.

Each of <a> to <d> shown in FIG. 7 may represent an example of a mask (IRM or CIRM) estimated through inference by the machine learning model 110 . In other words, each of <a> to <d> shown may represent an optimal value estimated by the machine learning model 110 .

By multiplying the mask with respect to the input voice signal 105 , noise included in the input voice signal 105 may be suppressed.

In the illustrated <a> to <d>, for example, the x-axis may represent a frequency (or time), and the y-axis may represent a value multiplied with respect to the input voice signal 105 .

The amplitude of the shape and value of the mask estimated by the machine learning model 110 will be different from that shown depending on the results of the estimation by the machine learning model 110 and the input parameters to the machine learning model 110 described above. can

As described above, the description of the technical features described above with reference to FIGS. 1 to 6 can be applied to FIG. 7 as it is, and thus the overlapping description will be omitted.

8 illustrates a method of reconstructing a noise-removed voice signal by processing a voice signal including noise, according to an example.

With reference to FIG. 8 , a method of obtaining a reconstructed voice signal 120 from which noise has been removed from the input voice signal 105 including noise and voice signals according to the embodiment will be described in more detail.

Hereinafter, a method of obtaining the reconstructed voice signal 120 from which noise has been removed from the input voice signal 105 of the embodiment according to steps 1 to 18 shown in FIG. 8 will be described. Processes 1 to 18 may correspond to the steps described with reference to FIGS. 3 to 6 . In FIG. 8 , the machine learning model 110 is illustrated as a DNN.

1. The computer system 100 may perform FFT on the input noise-containing input voice signal 105 . The input voice signal 105 may include a noise-free voice signal and noise (eg, background noise) as an audio signal. As FFT, 1024-point-FFT may be performed. The sampling rate of the input speech signal 105 may be 48 kHz, the FFT size may be 20 ms (960 samples), and the hop size may be 2.5 ms (120 samples, 12.5% overlap).

2. For the input speech signal 105 on which the FFT has been performed, the computer system 100 calculates the amplitude (root) by taking the square root of the squared sum of the real and imaginary values of the FFT coefficients. magnitude) signal can be obtained.

3. The computer system 100 may obtain the phase signal by calculating the arc tangent of the real and imaginary values of the FFT coefficients. The phase signal may be calculated to have a value in the range (-pi to pi) to enable restoration to the original voice signal.

4. The computer system 100 may set an amplitude signal of a bandwidth of 0-8 kHz (a first amplitude signal of the above-described second frequency bandwidth) as an input of the DNN 110 . The computer system 100 may further calculate the MFCC and ZCR for the input amplitude signal in order to increase the accuracy of learning and inference of the DNN 110 , and additionally input them to the DNN 110 . can MFCC can be calculated in the range of 0-8 kHz and calculated as 20 coefficients. ZCR may be calculated using the input amplitude signal of the time axis. The MFCC may provide information about the shape of the entire frequency of the amplitude signal, and the ZCR may provide information about the noise included in the time-axis component of the amplitude signal (eg, information about the extent to which the noise is included). .

5. The computer system 100 may calculate an average energy for each bandwidth section of the Bark scale unit with respect to the amplitude signal of the 8-24 kHz bandwidth (the second amplitude signal of the third frequency bandwidth described above). According to a psychoacoustic theory, this may be in consideration of human perception of high-bandwidth speech signals with low resolution. The divided bandwidth section may include, for example, bandwidth sections of 8000 to 9600, 9600 to 12000 Hz, 12000 to 15600 Hz, 15600 to 20000 Hz and 20000 to 24000 Hz, according to a predefined Bark scale, and the frequency energy for each section. An average can be calculated.

6. The computer system 100 may set the average value of energy for each of the five bandwidth sections calculated in step 5 as an input value of the DNN 110 .

7. The computer system 100 may set the low-bandwidth 0-4 kHz phase signal (the first phase signal of the first frequency bandwidth described above) as an input of the DNN 110 . According to the results of the listening evaluation experiment, even when the amplitude signal is clean, if the phase signal contains noise, the corresponding voice signal may be perceived by the user as containing noise (noisy). However, even if the phase signal includes noise, if the phase signal of the 0 to 4 kHz bandwidth does not include noise, the corresponding voice signal may be perceived as clean by the user. Therefore, in the embodiment, in order to reduce the amplitude of the network constituting the DNN 110 , only a phase signal of a bandwidth of 0 to 4 kHz may be used as an input (input parameter) of the DNN 110 .

8. The input of the DNN 110 may be configured with the above-described five parameters (amplitude signal, phase signal, Bark scale energy, MFCC and ZCR). In order to utilize the temporal continuity of the speech signal, parameters for the current frame and 7 frames in the past may be used, and accordingly, parameters for a total of 8 frames may be used as an input of the DNN 110 . In addition, in order to reduce the amount of computation, the DNN 110 may be configured only with a fully connected layer.

The mask corresponding to the output of the DNN 110 for restoring the frequency of the voice signal 120 is i) IRM for an amplitude signal of 0-8 kHz bandwidth (which is the second frequency band), ii) (the third frequency band) i) IRM for Bark scale energy according to amplitude signal of 8-24 kHz bandwidth, iii) FFT coefficient of 0-4 kHz bandwidth (which is the first frequency band) (eg, FFT coefficient of input voice signal 105 of that bandwidth, or CIRM for the FFT coefficient of the first phase signal). In addition, as additional information for increasing the performance of inference by the DNN 110 , the DNN 110 may further include a network for outputting VAD information for additionally learning information about a voice signal. The output of each layer constituting the network outputting the VAD information may be used as an input of the frequency recovery network in reconstructing the voice signal 120 .

9. As one of the outputs of the DNN 110 , an IRM for an amplitude signal of a bandwidth of 0 to 8 kHz (which is a second frequency band) may be output.

10. The computer system 100 may estimate the denoised amplitude signal by taking the product of the IRM at 9 and the amplitude signal of the input speech signal 105 (ie, the first amplitude signal).

11. As one of the outputs of the DNN 110 , an IRM for Bark scale energy according to an amplitude signal of a bandwidth of 8 to 24 kHz (which is a third frequency band) may be output.

12. The computer system 100 may estimate the denoised amplitude signal by taking the product of the IRM at 11 and the amplitude signal (ie, the second amplitude signal) of the input speech signal 105 . At this time, the computer system 100 may configure a group of the second amplitude signal according to the range of the calculated section of the IRM of the Bark scale, and may multiply the IRM in units of the group. That is, the IRM may be multiplied by the second amplitude signal to adjust the average energy for each divided Bark scale bandwidth section.

13. As one of the outputs of the DNN 110 , a CIRM for a phase signal (ie, a first phase signal) having a bandwidth of 0 to 4 kHz (which is a first frequency band) may be output.

14. The computer system 100 may obtain the product of the FFT coefficient of the input speech signal 105 (or the first phase signal) and the CIRM. As a result of the product, a complex value including both amplitude and phase information can be calculated. Among them, amplitude information is not used, and only phase information is extracted using arctan to restore the voice signal 120 . Can be used.

15. The computer system 100 may use the inputted phase signal as it is to restore the voice signal 120 with respect to a phase signal of a bandwidth of 4 to 24 kHz (which is a frequency band other than the first frequency band).

16. As one of the outputs of the DNN 110 , VAD information may be output. As described above, the VAD information may be an additional parameter for increasing the performance of inference by the DNN 110 .

17. The computer system 100 may reconstruct the FFT coefficients of the speech signal 120 using the amplitude signal and the phase signal estimated by 10, 12 and 14 described above (noise removed). The computer system 100 may reconstruct the speech signal 120 by performing IFFT using the reconstructed FFT coefficients.

18. The computer system 100 may output the restored voice signal 120 according to the result of the IFFT.

As described above, the description of the technical features described above with reference to FIGS. 1 to 7 can be applied to FIG. 8 as it is, and thus the overlapping description will be omitted.

9 shows parameters input to the machine learning model and the structure of the machine learning model, according to an example.

In FIG. 9 , the machine learning model 110 is illustrated as a DNN.

As shown, the input voice signal 105 input to the machine learning model 110 may be composed of a plurality of frames. For example, the machine learning model 110 sets parameters for the current frame and the past 7 frames (ie, a total of 8 frames) in order to utilize the temporal continuity of the speech signal (audio signal). ) can be used as an input to perform inference.

With respect to each frame of input speech signal 105 , amplitude signal, phase signal, Bark scale energy, MFCC and ZCR may be input parameters of machine learning model 110 , as described above.

The machine learning model 110 may estimate the IRM from the amplitude signal for each of the eight frames, and the CIRM from the phase signal. In addition, the machine learning model 110 may estimate VAD information based on the input.

A network for estimating IRM, a network for estimating CIRM, and a network for estimating VAD information may be configured as a plurality of all fully connected layers.

As shown, the output of each layer constituting the network for estimating VAD information may be used as an input of the frequency recovery network in reconstructing the voice signal 120 . In other words, the output of each layer constituting the network for estimating VAD information may be input to a layer constituting a network for estimating CIRM and a layer constituting a network for estimating IRM. Accordingly, the accuracy of estimation of CIRM and IRM may be improved by the input VAD information.

Hereinafter, the DNN of FIG. 9 will be described in more detail.

In FIG. 9 , a number in ( ) in relation to an input parameter for the DNN may indicate the number of FFT bins. Regarding the amplitude, “170” may indicate the number of FFT bins of the amplitude signal. For example, when 1024-FFT is performed on a signal sampled at a sampling rate of 48000 Hz, the resolution of one frequency bin may be expressed as 48000/1024 = 46.875. If the low-band (ie, second frequency band) 8 kHz for the above-described amplitude signal is calculated with the above resolution, 8000/46.875 = about 170.66, therefore, the amplitude signal input to the DNN is expressed by 170 frequency bins. can be Meanwhile, since the low band (ie, the first frequency band) of the above-described phase signal is 4 kHz, 85 frequency bins, which are half of the 170, may be used for the phase signal. Accordingly, the input parameter of the phase signal to the DNN in FIG. 9 may be represented by the phase 85 . Since the Bark scale energy is 5 as the number of bandwidth sections of the high band (ie, the third frequency band), it can be expressed as 5 as shown. 20 and 1 corresponding to MFCC and ZCR may indicate the number of calculated coefficients of MFCC and ZCR, respectively.

The number in parentheses ( ) of each layer of the illustrated DNN may indicate the number of nodes of each layer. In an embodiment, the DNN may be experimentally designed such that the number of nodes gradually decreases from input to output. Alternatively, the number of nodes in each of the layers constituting the DNN may be configured differently from the illustrated one.

In the learning process for VAD information, with respect to an input voice signal including noise, 1 may be labeled for a section corresponding to voice and 0 may be labeled for a section corresponding to non-voice, respectively. Accordingly, outputs corresponding to VAD information may be trained to be 0 and 1, and thus, the last layer for learning/inferring VAD information may have only one node. The 64 nodes that receive the input for learning/inferring VAD information for the first time may be learned to distinguish characteristics of sections corresponding to speech/non-voice. Accordingly, the DNN can learn features specific to the input speech signal, and the output from the layer for learning/inferring VAD information is used as an input to the layer for learning/inferring IRM and CIRM, so that the IRM of the DNN and CIRM learning/inference performance can be improved.

For the layer for learning/inferring IRM and CIRM, features representing the characteristics of the input voice signal are input, so that IRM and CIRM corresponding to the voice signal from which the noise has been removed can be inferred. In other words, a layer for learning/inferring IRM and CIRM can be learned to infer these IRM and CIRM. For example, a layer for learning/inferring IRM and CIRM may be trained to infer IRM and CIRM based on the characteristics of the voice signal of the previous frame and the current frame of the input voice signal, and IRM according to the result of such learning and CIRM can be inferred.

As described above, the description of the technical features described above with reference to FIGS. 1 to 8 can be applied to FIG. 9 as it is, and thus a redundant description will be omitted.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store the program executable by the computer, or to temporarily store the program for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store for distributing applications, a site for supplying or distributing other various software, and a server.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (18)

A method for processing, using a computer system, a voice signal including noise, the method comprising:
obtaining an amplitude signal and a phase signal from an input voice signal including noise on which Fast Fourier Transform (FFT) has been performed;
input the amplitude signal to a pre-trained machine learning model to estimate a mask to be applied to the input speech signal to remove the noise, and obtain a mask to be applied to the amplitude signal as an output of the machine learning model to do;
inputting a first phase signal of a first frequency band among the phase signals to the machine learning model, and obtaining a mask to be applied to the FFT coefficients of the input speech signal as an output of the machine learning model; and
Restoring the speech signal from which the noise has been removed based on the mask obtained from the input speech signal and the machine learning model
A method of processing a voice signal comprising: According to claim 1,
The first phase signal is a signal of a relatively low frequency band among the phase signals,
and a phase signal of a frequency band other than the first frequency band among the phase signals is used as it is in reconstructing the noise-free voice signal. According to claim 1,
Obtaining a mask to be applied to the amplitude signal comprises:
inputting a first amplitude signal of a second frequency band among the amplitude signals to the machine learning model, and obtaining a first mask to be applied to the first amplitude signal as an output of the machine learning model;
dividing a second amplitude signal of a third frequency band, which is a higher frequency band than the second frequency band, among the amplitude signals into amplitude signals of a plurality of bandwidth sections;
calculating an average energy for each of the divided amplitude signals; and
inputting the calculated average energy to the machine learning model, and obtaining a second mask to be applied to the second amplitude signal as an output of the machine learning model;
A method of processing a voice signal comprising: 4. The method of claim 3,
The second amplitude signal is divided into amplitude signals of the plurality of bandwidth sections by dividing the third frequency band by a bark scale unit. 4. The method of claim 3,
The first mask is an ideal ratio mask (IRM) for the first amplitude signal,
the second mask is the IRM for the calculated average energy,
In the step of restoring the voice signal from which the noise has been removed,
wherein the first mask is multiplied by the first amplitude signal and the second mask is multiplied by the second amplitude signal. 4. The method of claim 3,
Obtaining the first mask comprises:
calculating a predetermined number of Mel-Frequency Cepstral Coefficients (MFCCs) based on the first amplitude signal; and
inputting the calculated MFCCs to the machine learning model to obtain the first mask;
A method of processing a voice signal comprising: 4. The method of claim 3,
Obtaining the first mask comprises:
calculating a Zero Crossing Rate (ZCR) based on the first amplitude signal; and
inputting the calculated ZCR to the machine learning model to obtain the first mask;
A method of processing a voice signal comprising: 4. The method of claim 3,
The input voice signal is a signal having a bandwidth of 24 kHz,
The first frequency band is a band of 0 or more and less than 4 kHz,
The second frequency band is a band of 0 or more and less than 8 kHz,
The third frequency band is a band of 8 kHz or more and less than 24 kHz, the method of processing a voice signal. According to claim 1,
The step of restoring the voice signal from which the noise has been removed includes:
estimating a denoising amplitude signal by multiplying the amplitude signal by a mask to be applied to the amplitude signal;
estimating a denoising phase signal of the first frequency band by multiplying an FFT coefficient of the input speech signal by a mask to be applied to the first phase signal;
Restoring an FFT coefficient of the noise-removed speech signal based on the noise-removed amplitude signal, the noise-removed phase signal of the first frequency band, and a phase signal of a frequency band other than the first frequency band among the phase signals step; and
reconstructing the noise-removed speech signal by performing Inverse Fast Fourier Transform (IFFT) based on the reconstructed FFT coefficients
A method of processing a voice signal comprising: 10. The method of claim 9,
The input voice signal is a signal having a bandwidth of 24 kHz,
The first frequency band is a band of 0 or more and less than 4 kHz,
A frequency band other than the first frequency band is a band of 4 kHz or more and less than 24 kHz. According to claim 1,
The machine learning model is composed of a fully connected layer (Fully Connected Layer), a method of processing a voice signal. According to claim 1,
The machine learning model is a more trained model to estimate Voice Activity Detection (VAD) information for distinguishing between speech and silence,
obtaining VAD information for the input speech signal as an output of the machine learning model;
further comprising,
The VAD information of the input voice signal is used in frequency restoration of the noise-removed voice signal. According to claim 1,
The mask to be applied to the first phase signal is a complex ideal ratio mask (CIRM) for the FFT coefficient of the first phase signal of the first frequency bandwidth,
In the step of restoring the voice signal from which the noise has been removed,
wherein the CIRM is multiplied by an FFT coefficient of the input speech signal. According to claim 1,
The amplitude signal and the phase signal are signals of a predetermined frequency bin unit,
The voice signal from which the noise has been removed is restored in units of the frequency bin. According to claim 1,
The method of claim 1, wherein the input speech signal input to the machine learning model is composed of a plurality of frames. A program recorded on a computer-readable recording medium for performing the method of any one of claims 1 to 15. A computer system for processing a voice signal including noise, the computer system comprising:
at least one processor implemented to execute computer-readable instructions
including,
the at least one processor,
To obtain a magnitude signal and a phase signal from an input speech signal including noise on which FFT (Fast Fourier Transform) has been performed, and to estimate a mask to be applied to the input speech signal to remove the noise For a pretrained machine learning model, input the amplitude signal, obtain a mask to be applied on the amplitude signal as an output of the machine learning model, and for the machine learning model, input a first phase signal, obtain a mask to be applied for FFT coefficients of the input speech signal as an output of the machine learning model, and based on the input speech signal and a mask obtained from the machine learning model, the noise is A computer system that restores a removed speech signal. A method for processing, using a computer system, a voice signal including noise, the method comprising:
obtaining an amplitude signal and a phase signal from an input voice signal including noise on which Fast Fourier Transform (FFT) has been performed;
input the phase signal to a pre-trained machine learning model for estimating a mask to be applied to the input speech signal to remove the noise, and output the first frequency band of the phase signal as an output of the machine learning model obtaining a mask to be applied to the FFT coefficients of the input speech signal as a mask for the FFT coefficients of the first phase signal;
For the machine learning model, input the amplitude signal, and as an output of the machine learning model, a first mask to be applied to a first amplitude signal of a second frequency band of the amplitude signal and the second frequency band of the amplitude signal obtaining a second mask to be applied for a second amplitude signal of a third frequency band, which is a larger frequency band; and
a product of a phase signal in a frequency band other than the first frequency band among the phase signals, a product of the mask and an FFT coefficient of the input audio signal, a product of the first mask and the first amplitude signal, and a second mask; Restoring the speech signal from which the noise has been removed based on the product of the second amplitude signal
A method of processing a voice signal comprising:

Images