KR20240030483A - Electronic device performing noise cancellation using a plurality of artificial intelligence modules - Google Patents

Abstract

An electronic device that performs noise removal using a plurality of artificial intelligence modules, which is an embodiment, an input unit for applying an acoustic input signal to a processor, and an input unit for receiving the acoustic input signal on a frame-by-frame basis from the processor and removing noise from the acoustic input signal frame. A first artificial intelligence module that infers a first mask and applies the inferred first mask to the processor, and a second mask that receives an audio input signal in units of frames from the processor and removes noise from the audio input signal frame. a second artificial intelligence module for inferring and applying the inferred second mask to the processor; applying an acoustic input signal from the input unit to each of the first and second artificial intelligence modules in units of frames; and Receive first and second masks from each of the intelligence modules, multiply the sound input signal frame and each of the applied first and second masks to obtain an sound input signal frame to which the first mask is applied and an sound input to which the second mask is applied. Generate a signal frame, and depending on the size of the first signal-to-noise ratio of the acoustic input signal frame to which the generated first mask is applied and the second signal-to-noise ratio of the acoustic input signal frame to which the generated second mask is applied, the first mask or the second mask and a processor for applying the 2 mask or the arithmetic mean of the first and second masks as a final mask for the acoustic input signal frame.

Description

Electronic device that performs noise removal using a plurality of artificial intelligence modules {ELECTRONIC DEVICE PERFORMING NOISE CANCELLATION USING A PLURALITY OF ARTIFICIAL INTELLIGENCE MODULES}

The embodiment relates to an electronic device, and in particular, uses a plurality of different artificial intelligence modules to remove noise according to the characteristics of the sound through machine learning (machine learning) to more effectively remove noise from sounds containing voice and noise. It relates to an electronic device that performs noise removal using a plurality of artificial intelligence modules.

Recently, noise reduction/suppression technology using machine learning/deep learning has become widely used as a technology to cancel out noise that interferes with conversation while wearing devices in the form of wired or wireless earphones or headphones. It is becoming. This method can usually remove noise and maintain voice signals by inferring values such as masks for input signals such as the time axis and frequency axis.

In the case of the conventional noise removal mode, noise cannot be effectively removed in an acoustic environment where voice and noise coexist.

The embodiment uses a plurality of different artificial intelligence modules to remove noise according to the characteristics of the sound through machine learning, and uses a plurality of artificial intelligence modules to more effectively remove noise from sound containing voice and noise. The object is to provide an electronic device that performs removal.

An electronic device that performs noise removal using a plurality of artificial intelligence modules, which is an embodiment, has an input unit that applies an acoustic input signal to a processor, and performs machine learning to remove noise from a composite signal of voice and noise with a signal-to-noise ratio greater than 0. , a first artificial intelligence module that receives an acoustic input signal in frame units from a processor, infers a first mask for removing noise from the acoustic input signal frame, and applies the inferred first mask to the processor, and a signal-to-noise ratio Perform machine learning to remove noise from a composite signal of voice and noise smaller than 0, receive the audio input signal in frame units from the processor, infer a second mask to remove noise from the audio input signal frame, and perform inference. a second artificial intelligence module for applying the second mask to the processor, and applying an acoustic input signal from the input unit to each of the first and second artificial intelligence modules in units of frames, and each of the first and second artificial intelligence modules Receive first and second masks from and multiply the audio input signal frame and each of the applied first and second masks to generate an audio input signal frame to which the first mask is applied and an audio input signal frame to which the second mask is applied. And, depending on the size of the first signal-to-noise ratio of the acoustic input signal frame to which the generated first mask is applied and the second signal-to-noise ratio of the acoustic input signal frame to which the generated second mask is applied, the first mask, the second mask, or the second mask. and a processor that applies the arithmetic mean of the first and second masks as a final mask for the acoustic input signal frame.

Additionally, when the first signal-to-noise ratio and the second signal-to-noise ratio are greater than 0, the processor preferably applies the first mask as the final mask for the audio input signal frame.

Additionally, when the first signal-to-noise ratio and the second signal-to-noise ratio are less than 0, the processor preferably applies the second mask as the final mask for the audio input signal frame.

Additionally, if the first signal-to-noise ratio is greater than 0 and the second signal-to-noise ratio is less than 0, or the first signal-to-noise ratio is less than 0 and the second signal-to-noise ratio is greater than 0, the processor calculates the arithmetic mean of the first and second masks. It is desirable to apply it as a final mask for the acoustic input signal frame.

In addition, the electronic device performs machine learning to determine the type of acoustic environment from the acoustic signal, receives the acoustic input signal frame from the processor, infers the type of acoustic environment, and applies the inferred type of acoustic environment to the processor. It is desirable to have a third artificial intelligence module.

In addition, the processor applies an acoustic input signal frame to the third artificial intelligence module, receives approval of the type of acoustic environment inferred from the third artificial intelligence module, and when the type of the inferred acoustic environment is conversation, the acoustic input signal frame It is desirable to apply to each of the first and second artificial intelligence modules.

The embodiment uses a plurality of different artificial intelligence modules that perform noise removal according to the characteristics of the sound through machine learning, and determines the mask to be applied using the SNRs of the sound to which the plurality of masks are respectively applied, so that the voice and noise are reduced. It has the effect of more effectively removing noise from the included sound.

1 is a control configuration diagram of an electronic device that performs noise removal using a plurality of artificial intelligence modules according to an embodiment.
Figure 2 is a schematic diagram showing a processing process in an electronic device.

Hereinafter, embodiments are described in detail through the drawings. However, this is not intended to be limiting to specific embodiments, and the described embodiments should be understood to include various modifications, equivalents, and/or alternatives of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B”, “at least one of A and B”, or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first", "second", "first", or "second" may describe various elements in any order and/or importance, and may refer to one element as another. It is only used to distinguish from components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be renamed a second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

The expression “configured to” used in this document may mean, for example, “suitable for,” “having the capacity to,” or “having the capacity to.” It can be used interchangeably with ", "designed to," "adapted to," "made to," or "capable of." The term “configured (or set) to” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or executing one or more software programs stored on a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing the corresponding operations.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, they may be interpreted in an ideal or excessively formal sense. It is not interpreted. In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

FIG. 1 is a control configuration diagram of an electronic device that performs noise removal using a plurality of artificial intelligence modules according to an embodiment, and FIG. 2 is a schematic diagram showing a processing process in the electronic device.

The electronic device 10 includes an input unit 1 that acquires sound, generates an acoustic input signal, and applies it to the processor 9, and an output unit 2 that receives the sound output signal from the processor 9 and emits or transmits the sound. , a communication unit 3 that communicates with an electronic communication device (e.g., smartphone, tablet, etc.), and a frequency domain acoustic input signal from the processor 9 by performing machine learning using artificial intelligence. A first artificial intelligence module (4) that receives authorization and infers the type of acoustic environment of the acoustic input signal and applies it to the processor (9), and performs machine learning using artificial intelligence to provide frequency domain acoustic input from the processor (9). A second and second processor for receiving a signal, inferring first and second masks or mask values for removing noise included in the acoustic input signal, respectively, and applying the inferred first and second masks or mask values to the processor 9. The third artificial intelligence module (5), (6) and the above-described components are controlled to communicate with the electronic communication device through the communication unit (3) to provide a sound reproduction function, a phone call function, and a hearing assistance function (e.g., hearing aid function). function), applies an acoustic input signal in the frequency domain on a frame basis to the first artificial intelligence module (4), and depending on the type of acoustic environment from the first artificial intelligence module (4), the second and third artificial intelligence Perform noise removal mode using the intelligence modules (5) and (6), and use one of the first and second masks from each of the second and third artificial intelligence modules (5) and (6) as the final mask. Alternatively, the arithmetic mean value of the first and second masks is used as the final mask, and the acoustic input signal in the frequency domain is multiplied by the final mask to generate an acoustic input signal in the frequency domain from which noise has been removed, and the generated acoustic input signal in the frequency domain is It is configured to include a processor 9 that converts an audio input signal in the time domain and applies it to the output unit 2 as an audio output signal. However, the power supply unit (not shown) and the communication unit 3 correspond to technologies naturally recognized by those familiar with the technical field to which the present invention pertains, and their detailed descriptions are therefore omitted.

The input unit 1 may be implemented as a microphone that acquires noise and/or sound or voice, or as a communication unit that receives or receives a voice input signal in the time domain from an external device.

The output unit 2 may be implemented as a speaker that emits an audio output signal, or a communication unit 3 that transmits or applies the audio output signal to an external device.

The first to third artificial intelligence modules 4 to 6 are, for example, algorithms that perform deep learning or machine learning using RNN, LSTM, CNN, DNN models, etc., or perform operations according to such algorithms. It can be implemented as an executor that outputs. The first to third artificial intelligence modules 4 to 6 receive frequency domain acoustic input signals including magnitude values and phases for each frequency for each frame from the processor 9, that is, the first to third artificial intelligence modules 4 to 6 The input value of modules 4 to 6 is an acoustic input signal in the frequency domain. In an embodiment, the input value is mainly the magnitude of each frequency of the acoustic input signal.

First, the first artificial intelligence module 4 uses the frequency domain acoustic input signal (acoustic input signal frame) applied in frame units from the processor 9 as an input value, and determines the type of acoustic environment of the acoustic input signal (e.g. For example, it is an artificial intelligence model that judges speech (SPEECH, WARNING, etc.) and outputs the type of acoustic environment to the processor 9 as an output value. In the first artificial intelligence module 4, train data are sound source files corresponding to each acoustic environment, and target data includes a label indicating which acoustic environment file the train data is. The first artificial intelligence module 4 previously performs a learning process on the train data and target data.

The second and third artificial intelligence modules 5 and 6 each use an audio input signal (acoustic input signal frame) in the frequency domain applied in frames from the processor 9 as an input signal, and an audio signal from which noise has been removed. is delivered as the correct answer, and the feature value of the acoustic input signal in the frequency domain is used to differentiate between noise and speech through the sigmoid function and infer a mask for each frequency (or frequency band) to remove noise. The operation of outputting the output value in (9) is performed.

In this embodiment, the noisy signal is a signal containing clean sound (voice) and noise, the noise signal is a signal containing only noise, and the clean signal is a signal containing only noise. As a signal containing only clean sound (voice) without noise, a noisy signal is created by combining a noise signal and a clean signal.

The second artificial intelligence module 5 learns a noisy signal synthesized from a clean signal and a noise signal, and performs machine learning using the noisy signal with a signal-to-noise ratio (SNR) greater than 0 as train data to remove noise and generate a clean signal. This is a module that infers a mask for human speech. On the other hand, the third artificial intelligence module 6 synthesizes a noisy signal synthesized from a clean signal and a noise signal, and performs machine learning using the noisy signal with a signal-to-noise ratio (SNR) less than 0 as train data to remove noise. This is a module that infers a mask for voice, which is a clean signal. This mask inference process corresponds to a technique naturally recognized by a person skilled in the art who is familiar with the technical field to which the embodiment belongs, so detailed description thereof is omitted. If the mask is '0', the size of the relevant frequency must be removed (sound including noise or noisy speech), and if the mask is '1', the size of the relevant frequency must be maintained (preserved). means (voice or sound including voice with low noise). The mask inferred by the second and third artificial intelligence modules 5 and 6 respectively falls within the range of 0 or more and within 1.

In the learning process of a single artificial intelligence module, when learning a noisy signal with an SNR greater than 0 and a noisy signal with an SNR less than 0, significant confusion occurs in the process of refining a clean signal from a sound source that is too diverse in train data. Since this problem is occurring, in this embodiment, the second and third artificial intelligence modules (5) and (6) are utilized, respectively.

The processor 9 is equipped with a fast Fourier transform (FFT) function and an inverse fast Fourier transform (IFFT) function, and has arithmetic functions (e.g., multiplication function, Amadard product (element-wise multiplication function, etc.)) and storage function ( It is an electronic and/or electrical circuit device including (for example, memory, etc.).

The processor 9 performs FFT on the acoustic input signal applied from the input unit 1 in frame units of a preset size, generates an acoustic input signal frame in the frequency domain, and continuously applies it to the first artificial intelligence module 4. do. In this embodiment, the acoustic input signal frames in the frequency domain generated by the processor 9 are divided into a first frame (Frame 1), a second frame (Frame 2), and a third frame (Frame 3) according to the order in which they are generated. ), ..., is referred to as the nth frame (Frame n).

First, the processor 9 applies the first frame (frame 1) to the first artificial intelligence module 4. The first artificial intelligence module 4 infers the type of acoustic environment from the approved first frame (frame 1) and applies the inferred type of acoustic environment to the processor 9.

The processor 9 determines whether to apply the first frame (frame 1) to the second and third artificial intelligence modules 5 and 6 according to the type of the inferred acoustic environment. If the type of the inferred acoustic environment is conversation, the processor 9 applies the first frame (frame 1) to the second and third artificial intelligence modules 5 and 6, respectively, to perform the noise removal mode. do. If the inferred type of acoustic environment is not a conversation, the processor 9 does not transmit the first frame to the second and third artificial intelligence modules 5 and 6 to perform a mode other than the noise removal mode. No.

The second and third artificial intelligence modules 5 and 6 each infer the first and second masks (mask1) and (mask2) from the first frame (frame 1), and the inferred first and second masks (mask1) and (mask2) are respectively 2 Masks (mask1) and (mask2) are applied to the processor 9, respectively.

As shown in FIG. 2, the processor 9 receives the first and second masks (mask1) and (mask2), respectively, performs multiplication with the first frame (Frame 1), and generates the first mask (mask1) to which the first mask (mask1) is applied. One frame (Mask1_1) and a first frame (Mask2_1) to which the second mask (mask2) is applied are generated. Additionally, the processor 9 calculates the first signal-to-noise ratio (SNR1) and the second signal-to-noise ratio (SNR2) of the first frame (Mask1_1) and the first frame (Mask2_1), respectively. The processor 9 compares the first signal-to-noise ratio (SNR1) and the second signal-to-noise ratio (SNR2) with a reference value of 0. If both the first signal-to-noise ratio (SNR1) and the second signal-to-noise ratio (SNR2) are greater than 0, the processor 9 determines that the first frame (Frame 1) is a frame with a signal-to-noise ratio (SNR) greater than 0. , the first mask (mask1) is used to remove noise from the first frame (Frame 1), that is, the first mask (mask1) from the second artificial intelligence module 5 is used for the first frame (Frame 1). Apply as a final mask. In the case of the first frame (Frame 1) in FIG. 2, IFFF is performed on the first frame (Mask1_1) to generate and store an audio input signal frame in the time domain or apply it to the output unit 2 as an audio output signal. .

If both the first signal-to-noise ratio (SNR1) and the second signal-to-noise ratio (SNR2) are less than 0, the processor 9 determines that the first frame (Frame 1) is a frame with a signal-to-noise ratio (SNR) less than 0. Therefore, the second mask (mask2) is used to remove noise from the first frame (Frame 1), that is, the second mask (mask2) from the third artificial intelligence module 5 is applied to the first frame (Frame 1). Apply as a final mask for

In addition to the two cases above, the first signal-to-noise ratio (SNR1) is greater than 0 and the second signal-to-noise ratio (SNR2) is less than 0, or the first signal-to-noise ratio (SNR1) is less than 0 and the second signal-to-noise ratio ( If SNR2) is greater than 0, the processor 9 applies the arithmetic mean of the first and second masks (mask1) and (mask2) as the final mask for the first frame (Frame 1), Frame 1) and the final mask are multiplied to generate the first frame (Frame 1) from which noise has been removed.

In FIG. 2, in the case of the second frame (Frame 2), the processor 9 applies the second mask (mask2) of the third artificial intelligence module 6 as the final mask and sets IFFF for the second frame (Mask2_2). By performing this, an audio input signal frame in the time domain is generated and stored or applied to the output unit 2. The case of the third frame (Frame 3) is the same.

In Figure 2, the case of the nth frame (Frame n) is the same as the case of the first frame (Frame 1).

In Figure 2, frames determined as final frames are marked with red squares for ease of understanding.

The above-described frame is an acoustic input signal in the frequency domain, but the noise removal mode may be performed by applying the acoustic input signal in the time domain on a frame-by-frame basis.

In this embodiment, the electronic device includes earphones, a headset, etc.

At least a portion of the device (e.g., processor or functions thereof) or method (e.g., operations) according to various embodiments is stored in a computer-readable storage media, for example, in the form of a program module. Can be implemented as stored instructions. When the instruction is executed by a processor, the one or more processors may perform the function corresponding to the instruction. A computer-readable storage medium may be, for example, memory.

Computer-readable recording media include hard disks, floppy disks, magnetic media (e.g. magnetic tape), optical media (e.g. CD-ROM, DVD (Digital Versatile Disc), magnetic media) It may include magnetic media (e.g., a floptical disk), hardware devices (e.g., ROM, RAM, or flash memory, etc.), etc. Additionally, program instructions may include information such as those generated by the compiler. It may include not only machine language code but also high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform the operations of various embodiments, The same goes for the station.

The processor or functions provided by the processor according to various embodiments may include at least one of the above-described components, some of them may be omitted, or other additional components may be included. Operations performed by modules, program modules, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or in a heuristic manner. Additionally, some operations may be executed in a different order, omitted, or other operations may be added.

As explained above, it is not limited to the specific preferred embodiments described above, and various modifications can be made by anyone skilled in the art without departing from the gist of the claims. Of course, such changes are within the scope of the claims.

1: Input section 2: Output section
3: Communication Department 4: First Artificial Intelligence Module
5: Second artificial intelligence module 6: Third artificial intelligence module
9: processor

Claims (6)

an input unit that applies an audio input signal to the processor;
Perform machine learning to remove noise from a composite signal of voice and noise with a signal-to-noise ratio greater than 0, receive the audio input signal on a frame-by-frame basis from the processor, and infer a first mask to remove noise from the audio input signal frame. and a first artificial intelligence module that applies the inferred first mask to the processor;
Perform machine learning to remove noise from a composite signal of voice and noise with a signal-to-noise ratio of less than 0, receive the audio input signal on a frame-by-frame basis from the processor, and infer a second mask to remove noise from the audio input signal frame. and a second artificial intelligence module for applying the inferred second mask to the processor;
An audio input signal from the input unit is applied to each of the first and second artificial intelligence modules in units of frames, first and second masks are received from each of the first and second artificial intelligence modules, and an audio input signal frame and Each of the applied first and second masks is multiplied to generate an acoustic input signal frame to which the first mask is applied and an acoustic input signal frame to which the second mask is applied, and the first of the generated acoustic input signal frames to which the first mask is applied is Depending on the signal-to-noise ratio and the magnitude of the second signal-to-noise ratio of the acoustic input signal frame to which the generated second mask is applied, the first mask or the second mask or the arithmetic mean of the first and second masks is used as the final signal for the acoustic input signal frame. An electronic device that performs noise removal using a plurality of artificial intelligence modules, characterized in that it includes a processor applied as a mask. According to claim 1,
When the first signal-to-noise ratio and the second signal-to-noise ratio are greater than 0, the processor applies the first mask as the final mask for the acoustic input signal frame. An electronic device that performs noise removal using a plurality of artificial intelligence modules. device. According to claim 1,
When the first signal-to-noise ratio and the second signal-to-noise ratio are less than 0, the processor applies the second mask as the final mask for the acoustic input signal frame. An electronic device that performs noise removal using a plurality of artificial intelligence modules. device. According to claim 1,
When the first signal-to-noise ratio is greater than 0 and the second signal-to-noise ratio is less than 0, or the first signal-to-noise ratio is less than 0 and the second signal-to-noise ratio is greater than 0, the processor uses the arithmetic mean of the first and second masks as the acoustic input. An electronic device that performs noise removal using a plurality of artificial intelligence modules, characterized in that they are applied as a final mask to the signal frame. According to claim 1,
The electronic device performs machine learning to determine the type of acoustic environment from the acoustic signal, receives the acoustic input signal frame from the processor, infers the type of acoustic environment, and applies the inferred type of acoustic environment to the processor. An electronic device that performs noise removal using a plurality of artificial intelligence modules, including an artificial intelligence module. According to claim 5,
The processor applies an acoustic input signal frame to the third artificial intelligence module, receives approval of the type of acoustic environment inferred from the third artificial intelligence module, and, if the inferred type of acoustic environment is conversation, provides an acoustic input signal frame. An electronic device that performs noise removal using a plurality of artificial intelligence modules, characterized in that the application is applied to each of the first and second artificial intelligence modules.