KR20230120734A - Voice enhancing method - Google Patents

Abstract

The present invention relates to a voice enhancement method for performing noise elimination and voice enhancement by applying a noise elimination function using a machine learning model and a voice enhancement function.
The voice enhancement method of the present invention includes a first step of generating a voice input signal in the frequency domain by performing an FFT function on the voice input signal, and using the voice input signal in the frequency domain as an input value to generate noise by a first machine learning module. A second step of generating a masking spectrum as an output value, and a voice input signal in the frequency domain and an attention value of the noise masking spectrum as input values, generating a first voice enhancement signal in which voice components are enhanced by a second machine learning module Including the 3rd step.

Description

Voice enhancement method {VOICE ENHANCEING METHOD}

The present invention relates to a voice enhancement method, and more particularly, to a voice enhancement method for removing noise and enhancing voice by applying a noise removal function using a machine learning model and a voice enhancement function.

Recently, as interest in Internet calls such as VoIP and development and provision of content utilizing other voice/sound signals has increased, interest in technology for removing noise from voice signals has also increased.

A voice signal generally has a bandwidth of 8 kHz or 16 kHz, and a technology for removing noise from the voice signal has also been developed targeting the voice signal having a bandwidth of 8 kHz or 16 kHz. However, it is difficult to apply this noise removal technique to remove noise included in a high-quality voice signal having a bandwidth of 24 kHz or more (ie, full band). In other words, it is difficult to apply the existing noise removal technology to a service that provides a full-band voice signal, and when the sampling rate is increased to remove noise from the full-band voice signal, the amount of computation increases rapidly. there is a problem

In addition, in restoring a voice signal, when only an amplitude signal is restored and a signal containing noise is used as it is as a phase signal, when the noise is severe (eg, when the SNR is 0 dB or less), the voice signal from which the noise is properly removed cannot be restored, and in particular, it is difficult to effectively remove noise included in an amplitude signal and a phase signal for a high sampling rate audio signal.

An object of the present invention is to provide a voice enhancement method for performing noise removal and voice enhancement by applying a noise removal function using a machine learning model and a voice enhancement function.

The voice enhancement method of the present invention includes a first step of generating a voice input signal in the frequency domain by performing an FFT function on the voice input signal, and using the voice input signal in the frequency domain as an input value to generate noise by a first machine learning module. A second step of generating a masking spectrum as an output value, and a voice input signal in the frequency domain and an attention value of the noise masking spectrum as input values, generating a first voice enhancement signal in which voice components are enhanced by a second machine learning module Including the 3rd step.

In addition, the voice enhancement method includes a fourth step of generating a second voice enhancement signal by removing noise by multiplying the first voice enhancement signal by a noise mask of the noise masking spectrum, and performing an IFFT function on the second voice enhancement signal It is preferable to include a fifth step of generating an audio output signal by doing so.

Also, the attention value is preferably between 0 and 1.0.

The present invention has an effect of performing noise removal and voice enhancement by applying a noise removal function using a machine learning model and a voice enhancement function.

1 is a control block diagram of a voice enhancement device that performs a voice enhancement method according to the present invention.
2 is a control flow chart of a voice enhancement method according to the present invention.

Hereinafter, the present invention will be described in detail through examples and drawings. However, it should be understood that this is not intended to limit the present invention to the specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the present invention. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has", "may have", "includes", or "may include" refer to the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this document, expressions such as "A or B", "at least one of A and/and B", or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B", "at least one of A and B", or "at least one of A or B" includes (1) at least one A, (2) at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as "first", "second", "first", or "second" as used herein may modify various elements, in any order and/or importance, and may refer to one element as another. It is used to distinguish from components, but does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this document, a first element may be called a second element, and similarly, the second element may also be renamed to the first element.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, third components) do not exist between the other components.

As used in this document, the expression "configured to" means "suitable for", "having the capacity to", depending on the situation. ", "designed to", "adapted to", "made to", or "capable of" can be used interchangeably. The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or set) to perform A, B, and C" may include a dedicated processor (eg, an embedded processor) to perform those operations, or one or more software programs stored in a memory device that executes By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, in an ideal or excessively formal meaning. not interpreted In some cases, even terms defined in this document cannot be interpreted to exclude the embodiments of this document.

1 is a control block diagram of a voice enhancement device that performs a voice enhancement method according to the present invention.

The voice enhancement device 10 includes an input unit 1 that acquires or receives a voice input signal and applies it to a processor 9, and receives an audio output signal in which voice components are enhanced and noise is removed or suppressed from the processor 9. The output unit 2 that emits or transmits sound and the audio input signal in the frequency domain applied from the processor 9 is input as an input value, and the noise masking spectrum (noise mask, attention value or weight is calculated) including) as an output value, a voice input signal in the frequency domain applied from the processor 9, and a first voice enhancement signal in which the voice component is enhanced by receiving the attention value as an input value A voice enhancement module 5 that generates an output value and applies it to the processor 9, and a voice enhancement function that removes noise from the voice input signal and enhances the voice to output the voice output signal by controlling the above-described components. It is configured to include a processor 9 and the like. However, since the power supply unit (not shown) corresponds to a technology naturally recognized by those familiar with the technical field to which the present invention belongs, a detailed description thereof will be omitted.

In this embodiment, the voice input signal is a signal containing voice and/or noise.

The input unit 1 may be implemented as a microphone that acquires noise and/or voice, or a communication unit that receives or receives a voice input signal from an external device.

The output unit 2 may be implemented as a speaker that emits a sound output signal or a communication unit that transmits or applies an audio output signal to an external device.

The noise removal module 4 may be an algorithm that performs machine learning or deep learning, or may be implemented as an executor that performs and outputs an operation according to such an algorithm, and an RNN-based LSTM may be used or embedded. The noise removal module 4 takes a voice input signal in the frequency domain (a mixed signal of noise and a clean voice signal) as an input value, transfers the clean voice signal as an answer, and converts the feature value into a sigmoid function. Through this, a noise masking spectrum is generated and output to the processor 9. That is, the noise removal module 4 learns to reduce the difference between the product of the noise mask (or noise masking spectrum) and the input spectrum of the voice input signal in the frequency domain and the correct answer, the clean voice signal. Alternatively, the noise removal module 4 may be an executor generated by such learning.

The noise removal module 4 applies a noise masking spectrum to the processor 9 to perform a function of reducing the noise component while maintaining the voice component in the voice input signal. In this embodiment, noise masking is a result of mapping a value between 0.0 and 1.0 to each frequency bin through a sigmoid function in the input spectrum of a voice input signal in the frequency domain, and is an attention value ) has the same meaning. For example, the noise masking spectrum has a gain value between 0.0 and 1.0 and an attention value through a sigmoid function for each frequency bin. ) is given, and a low gain (low weight) is given to the frequency bin determined to be a noise component to remove the noise component.

The voice enhancement module 5 may be implemented as an algorithm that performs machine learning or deep learning, or an executor that performs and outputs an operation according to such an algorithm, and a CNN-based model or an RNN-based model may be used or embedded. there is.

The voice enhancement module 5 takes the voice input signal and the attention value in the frequency domain as input values, and applies the attention value, which is a weight, to the value of each frequency bin of the voice input signal in the frequency domain using the attention value, so that the voice component is The emphasized or enhanced first audio enhancement signal is output to the processor 9 as an output value. The voice enhancement module 5 corrects each frequency bin value of the input spectrum of the voice input signal in the frequency domain. The higher the attention value used for correction, the higher the probability that there is a voice component in the corresponding frequency bin. It may be an executor created by performing learning or performing learning in a direction (performing reinforcement or emphasis correction) to prevent loss of the corresponding frequency bin value (loss of speech component).

The processor 9 is a processor that performs a voice enhancement function, and includes a fast Fourier transform (FFT) function and an inverse fast Fourier transform (IFFT) function, and an arithmetic function (eg, a multiplication function) and a storage function (eg, a multiplication function). For example, memory, etc.) is an electronic and / or electrical circuit device having. The voice enhancement function performed by the processor 9 is described in detail in FIG. 2 .

2 is a control flow chart of a voice enhancement method according to the present invention.

In step S1, the processor 9 receives or receives a voice input signal from the input unit 1. The processor 9 proceeds to step S3 while receiving the voice input signal, but may proceed to step S3 for each data frame unit having a predetermined size.

In step S3, the processor 9 performs an FFT function on the voice input signal to generate a voice input signal in the frequency domain, and converts the generated voice input signal into the noise removal module 4 and the voice enhancement module. (5) Apply to each, and proceed to step S5.

In step S5, the noise removal module 4 takes the applied frequency domain audio input signal as an input value and, as described above, generates a noise masking spectrum as an output value and applies it to the processor 9, and the processor ( 9) receives the noise masking spectrum, and applies the attention value of the applied noise masking spectrum to the voice enhancement module 5. Processor 9 performs step S5 and proceeds to step S7.

In step S7, the voice enhancement module 5 uses the applied frequency domain voice input signal and the attention value as input values to generate a first voice enhancement signal in which voice components are enhanced, as described above, and use the processor as an output value. (9) is authorized. The processor 9 receives the first voice enhancement signal and proceeds to step S9.

In step S9, the processor 9 multiplies the first voice enhancement signal and the noise mask to remove or suppress noise to generate a second voice enhancement signal. Processor 9 performs step S9 and proceeds to step S11.

In step S11, the processor 9 generates and stores a time-domain audio output signal by performing an IFFT function on the second audio enhancement signal, which is a frequency domain signal, or applies it to the output unit 2.

The processor 9 generates an audio output signal by performing the above-described audio enhancement function on the audio input signal in units of data frames, and applies the generated audio output signal to the output unit 2 .

At least a part of a device (eg, a processor or functions thereof) or a method (eg, operations) according to various embodiments may be stored in a computer-readable storage medium in the form of a program module. Can be implemented as stored instructions. When the command is executed by a processor, the one or more processors may perform a function corresponding to the command. A computer-readable storage medium may be, for example, a memory.

Computer-readable recording media include hard disks, floppy disks, magnetic media (eg magnetic tape), optical media (eg CD-ROM, DVD (Digital Versatile Disc), magnetic- It may include optical media (e.g., a floptical disk), hardware devices (e.g., ROM, RAM, or flash memory, etc.), etc. In addition, the program instructions may include the same as generated by a compiler. It may include not only machine code but also high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform operations of various embodiments, The reverse is the same.

Processors or functions by processors according to various embodiments may include at least one or more of the aforementioned elements, some may be omitted, or additional elements may be further included. Operations performed by modules, program modules, or other components according to various embodiments may be executed in a sequential, parallel, repetitive, or heuristic manner. Also, some actions may be performed in a different order, omitted, or other actions may be added.

As described above, the present invention is not limited to the specific preferred embodiments described above, and anyone having ordinary knowledge in the technical field to which the present invention pertains can do various things without departing from the gist of the present invention claimed in the claims. Of course, variations are possible, and such variations are within the scope of the claims.

1: input 2: output
4: noise cancellation module 5: voice enhancement module

Claims (5)

a first step of generating a voice input signal in a frequency domain by performing an FFT function on the voice input signal;
a second step of generating a noise masking spectrum as an output value by a first machine learning module by using the voice input signal in the frequency domain as an input value;
A third step of generating a first speech enhanced signal in which speech components are enhanced by a second machine learning module using the speech input signal in the frequency domain and the attention value of the noise masking spectrum as input values. method. According to claim 1,
The voice enhancement method includes a fourth step of generating a second voice enhancement signal by multiplying a first voice enhancement signal and a noise mask of a noise masking spectrum to remove noise;
and a fifth step of generating an audio output signal by performing an IFFT function on the second audio enhancement signal. According to claim 1,
The voice reinforcement method, characterized in that the attention value is a value between 0 and 1.0. an input unit for applying a voice input signal to the processor;
a noise removal module that is a machine learning module that receives the voice input signal in the frequency domain applied from the processor as an input value and outputs a noise masking spectrum as an output value;
a voice enhancement module that is a machine learning module that receives a voice input signal of a frequency domain applied from the processor and an attention value of a noise masking spectrum as input values, generates a first voice enhancement signal as an output value, and applies the output value to the processor;
The FFT function is performed on the voice input signal from the input unit, and the voice input signal in the frequency domain is applied to the noise control module and the voice enhancement module, and the noise masking spectrum is received from the noise control module and applied to the voice enhancement module, and the voice enhancement A speech enhancement device comprising a processor to receive a first speech enhancement signal from the module. According to claim 4,
The processor multiplies the first voice enhancement signal with a noise mask of the noise masking spectrum to generate a second voice enhancement signal, and performs an IFFT on the generated second voice enhancement signal to generate a voice output signal. Characterized in that voice enhancer.