WO2019223650A1

WO2019223650A1 - Beamforming method, multi-beam forming method and apparatus, and electronic device

Info

Publication number: WO2019223650A1
Application number: PCT/CN2019/087621
Authority: WO
Inventors: 周舒然; 李志飞
Original assignee: 出门问问信息科技有限公司
Priority date: 2018-05-22
Filing date: 2019-05-20
Publication date: 2019-11-28

Abstract

Disclosed are a beamforming method, multi-beam forming method and apparatus, and an electronic device. The product of a spatial filtering parameter and an original frequency domain signal corresponding to a target sound source orientation is calculated to obtain beamforming output of the target sound source orientation, and noise reduction processing is performed on a non-target sound source orientation to improve the signal-to-noise ratio of the beamforming output of the target sound source orientation. Therefore, it can be ensured that a target spatial orientation is undistorted; moreover, sounds of the other target spatial orientations are effectively suppressed, thereby improving the signal-to-noise ratio of a sound of the target spatial orientation.

Description

Beam forming method, multi-beam forming method, device and electronic equipment

This application requires the application number 2018104970698 filed on May 22, 2018 and the invention name as "Method, Device and Electronic Equipment for Multi-Beam Beamforming", and the application number 2018104964502, invention filed on May 22, 2018 A method, device and electronic device for multi-beam beamforming, and a Chinese patent application filed on May 22, 2018 with an application number of 2018104964485 and an invention name of "method, device and electronic device for beamforming" Priority, the entire contents of which are incorporated herein by reference.

Technical field

The embodiments of the present application relate to the field of speech processing technologies, and in particular, to a beam forming method, a multi-beam forming method, a device, and an electronic device.

Background technique

With the rapid popularization of smart terminal technology, users have increasingly higher requirements for the functions and intelligence of smart terminals. How to make smart terminals more intelligent and professional has become one of the current research directions.

For example: Basically all smart terminals are equipped with a recording function, and most of the recording functions use beamforming. Beamforming is a signal processing technology (such as a microphone array) used for sensor arrays and used for directional signals. Receive and perform appropriate signal processing on the received sound signals. Beamforming allows the microphone component to receive sound signals in order to achieve the effect of selectively processing electrical signals. For example, the processing of sound information from one sound source is different from the processing of sound information from different sound sources.

Currently, speech processing is usually performed by fusing the calculation of the beamforming driving weights in the time domain filter and the frequency domain, but this does not reduce unwanted environmental noise.

Summary of the Invention

In view of this, embodiments of the present application provide a beam forming method, a multi-beam forming method, a device, and an electronic device, so as to ensure that the sound directed by the target space is not distorted, and effectively suppress the sound directed by other target spaces, thereby improving The signal-to-noise ratio of the sound pointed at the target space.

In a first aspect, an embodiment of the present invention provides a beamforming method, including:

Acquiring a spatial filtering parameter, which is different with different angles and subband frequencies; determining the sound source direction corresponding to the spatial filtering parameter, and obtaining the original frequency domain signal corresponding to the sound source direction;

Calculate a product of the spatial filtering parameter and the original frequency domain signal, where the product is used to perform beamforming in a manner that suppresses other frequency domain signals other than the original frequency domain signal pointed by the sound source.

Further, before acquiring the spatial filtering parameters, the method further includes:

Calculate the spatial filtering parameters.

Further, the calculating spatial filtering parameters includes:

Calculate the delay time for the sound source to reach the microphone array;

Constructing a signal vector function according to the delay time, and calculating a sound source direction according to the signal vector function and the delay time;

Calculating a spatial filtering parameter when the loss function approaches a minimum value according to a preset first limiting condition and a second limiting condition, and the loss function is constructed according to the spatial filtering parameter and the signal vector function;

The first limitation condition is specifically a white noise gain limitation; the second limitation condition is that a product of the spatial filtering parameter and the signal vector function is a first preset value.

Further, calculating the delay time for the sound source to reach the microphone array includes:

Determine the spacing between the microphones in the microphone array, and the speed at which the sound source propagates the sound;

Determining an angle at which the sound source is pointing;

The delay time is calculated according to a distance between the microphones, a speed at which the sound source propagates sound, and an angle at which the sound source points.

Further, calculating the sound source direction according to the signal vector function and the delay time includes:

Determine a matrix corresponding to all subband frequencies;

Calculate a sound source direction according to the matrix corresponding to all subband frequencies, the signal vector function, and the delay time.

Further, the spatial filtering parameter is a matrix.

Further, the sound source is directed to an arbitrary angle of 0 ° -180 ° of a plane wave.

In a second aspect, an embodiment of the present invention provides a beamforming apparatus, including:

A first obtaining unit, configured to obtain spatial filtering parameters, which are different with different angles and subband frequencies;

A determining unit, configured to determine a sound source corresponding to the spatial filtering parameter obtained by the first obtaining unit;

A second obtaining unit, configured to obtain that the sound source determined by the determining unit points to a corresponding original frequency domain signal;

A first calculation unit is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal, and the product is used to perform suppression in a frequency domain signal other than the original frequency domain signal pointed by the sound source. Beamforming.

According to a third aspect, an embodiment of the present invention provides a multi-beam beamforming method, including:

Calculate the target sound source pointing to the corresponding beamforming output;

Calculate noise parameters based on the blocking matrix;

Performing noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output directed by the target sound source according to the noise parameter.

Further, calculating the target beamforming output of the target sound source includes:

Acquiring spatial filtering parameters, and determining the target sound source direction corresponding to the spatial filtering parameters;

Acquiring that the target sound source points to a corresponding original frequency domain signal;

Calculate a product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source pointing to obtain the beamforming output pointed by the target sound source.

Further, calculating the noise parameters according to the blocking matrix includes:

Calculate the frequency response of the sound signal to the microphone in turn;

Constructing the blocking matrix according to the frequency response;

Calculate the noise parameter according to the blocking matrix and the non-target sound source pointing to a corresponding original frequency domain signal.

Further, performing noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output to the target sound source according to the noise parameter includes:

Calculate multi-channel optimal filtering parameters through multi-channel filtering algorithm and iterative algorithm;

Performing noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output according to the beamforming output of the target sound source, the multi-channel optimal filtering parameter, and the noise parameter .

According to a fourth aspect, an embodiment of the present invention provides a multi-beam beamforming apparatus, including:

A first calculation unit, configured to calculate that a target sound source points to a corresponding beamforming output;

A second calculation unit, configured to calculate a noise parameter by using a blocking matrix;

A noise reduction unit, configured to perform, according to the noise parameter calculated by the second calculation unit, a signal pointed by the target sound source calculated by the first calculation unit to a non-target sound source other than a corresponding beamforming output; Noise reduction.

Further, the first calculation unit includes:

A first acquisition module, configured to acquire spatial filtering parameters;

A determining module, configured to determine a target sound source direction corresponding to the spatial filtering parameter obtained by the first obtaining module;

A second acquisition module, configured to acquire a target sound source acquired by the first acquisition module to point to a corresponding original frequency domain signal;

A calculation module is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source pointing to obtain a beamforming output pointed by the target sound source.

Further, the second calculation unit includes:

A first calculation module, configured to calculate a frequency response in which a sound signal reaches the microphone in order;

A construction module, configured to construct the blocking matrix according to the frequency response calculated by the first calculation module;

A second calculation module is configured to calculate the noise parameter according to the blocking matrix constructed by the construction module and the non-target sound source pointing to a corresponding original frequency domain signal.

Further, the noise reduction unit includes:

A calculation module for calculating multi-channel optimal filtering parameters through a multi-channel filtering algorithm and an iterative algorithm;

A noise reduction module, configured to point the target sound source to a non-target sound source other than the corresponding beamforming output according to the beamforming output of the target sound source, the multi-channel optimal filtering parameter, and the noise parameter The pointed signal is denoised.

In a fifth aspect, the present invention provides a multi-beam beamforming method, including:

Calculate the product of the spatial filtering parameters and at least two sound sources pointing to the corresponding original frequency domain signals to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the subband frequency. The at least two The sound source pointing includes a target sound source and at least one non-target sound source pointing;

Calculating the enhanced speech pointed to by the target sound source;

Calculate the energy ratio according to the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source;

A product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the energy ratio is calculated, and the speech corresponding to the product is output.

Further, before calculating the product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the product of the energy ratio, the method further includes:

Perform smoothing frame-by-frame on the current frame and the previous frame through the smoothing parameters.

Further, calculating the product of the spatial filtering parameters and the original frequency domain signals corresponding to the at least two sound source directions respectively to obtain multi-beam beamforming includes:

Acquiring spatial filtering parameters, and determining at least two sound source directions respectively corresponding to the spatial filtering parameters;

Acquiring at least two sound sources pointing to respective original frequency domain signals;

Calculate products of the spatial filtering parameters and the original frequency domain signals corresponding to the at least two sound source directions, respectively.

Further, the calculation of the enhanced speech pointed by the target sound source includes:

Calculate a ratio gain between the energy pointed by the target sound source and the energy sum pointed by all sound sources with each subband as a unit;

Calculate a product of a first product and the ratio gain to obtain the enhanced speech, wherein the first product is a product between the target sound source pointing to a corresponding original frequency domain signal and the spatial filtering parameter.

Further, calculating the energy ratio according to the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source includes:

Combine the energy corresponding to all subbands in the current frame to calculate the energy sum of all subbands in the current frame;

Calculate the ratio between the energy of the subband corresponding to the target sound source and the energy sum of all the subbands pointed to by at least one non-target sound source to obtain the energy ratio.

Further, performing smoothing processing on a frame-by-frame basis for the current frame and the previous frame by using a smoothing parameter includes:

Set the smoothing parameters of the current frame so that the sum of the smoothing parameters of the current frame and the smoothing parameters of the previous frame is the second preset value;

Calculate the ratio gain of the previous frame and the smoothing parameters of the previous frame to obtain the second product;

Calculate the product of the ratio gain of the current frame and the smoothing parameter of the current frame to obtain a third product;

Performing frame-by-frame smoothing processing on the current frame according to the sum of the second product and the third product.

Further, calculating the product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the energy ratio, and outputting the speech corresponding to the product includes:

A product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the energy ratio is calculated, and the speech corresponding to the product is output according to the smoothing processing result.

According to a sixth aspect, an embodiment of the present invention provides a multi-beam beamforming apparatus, including:

A first calculation unit is configured to calculate a product of spatial filtering parameters and original frequency domain signals corresponding to at least two sound source directions, respectively, to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the subband frequency. Differently, the at least two sound source points include a target sound source point and at least one non-target sound source point sound source point;

A second calculation unit, configured to separately calculate the enhanced speech pointed by the target sound source;

A third calculation unit, configured to calculate an energy ratio based on the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source;

A fourth calculation unit is configured to calculate a product of the original frequency domain signal pointed by the target sound source, a corresponding enhanced speech pointed by the target sound source, and the energy ratio, and output a speech corresponding to the product.

In a seventh aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored, and the program is executed by a processor to implement the method according to the first aspect of the embodiment of the present invention and / or the method according to the first embodiment of the present invention. The method described in the third aspect and / or the method described in the fifth aspect of the embodiments of the present invention.

In an eighth aspect, an embodiment of the present invention provides an electronic device, where the electronic device includes a processor, a memory, and a bus; the processor and the memory communicate with each other through the bus; And storing program instructions that are executed by the processor to implement the method according to the first aspect of the embodiment of the present invention and / or the method according to the third aspect of the embodiment of the present invention and / or the method according to the present invention The method described in the fifth aspect of the embodiment.

In the embodiment of the present invention, the beamforming output of the target sound source is obtained by calculating the product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source direction, and the target is improved by performing noise reduction processing on the non-target sound source direction The signal-to-noise ratio of the beamforming output pointed by the sound source. Therefore, it is possible to ensure that the sound pointed by the target space is not distorted, and effectively suppress the sound pointed by other target spaces, thereby improving the signal-to-noise ratio of the sound pointed by the target space.

BRIEF DESCRIPTION OF THE DRAWINGS

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the detailed description of the preferred embodiments below. The drawings are only for the purpose of illustrating preferred embodiments, and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference numerals are used throughout the drawings to refer to the same parts. In the drawings:

FIG. 1 is a flowchart of a beamforming method according to an embodiment of the present invention;

2 is a schematic diagram of a microphone array according to an embodiment of the present invention;

3 is a schematic diagram of another microphone array according to an embodiment of the present invention;

4 is a flowchart of a method for calculating spatial filtering parameters according to an embodiment of the present invention;

5 is a flowchart of a multi-beam beamforming method according to an embodiment of the present invention;

6 is a schematic diagram of a final voice output pointed by a target sound source according to an embodiment of the present invention;

7 is a flowchart of another multi-beam beamforming method according to an embodiment of the present invention;

8 is a flowchart of still another multi-beam beamforming method according to an embodiment of the present invention;

9 is a schematic diagram of a beamforming apparatus according to an embodiment of the present invention;

10 is a schematic diagram of another beamforming apparatus according to an embodiment of the present invention;

11 is a schematic diagram of a multi-beam beamforming apparatus according to an embodiment of the present invention;

12 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention;

13 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention;

14 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention;

FIG. 15 is a structural block diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a thorough understanding of the present disclosure, and to fully convey the scope of the present disclosure to those skilled in the art.

FIG. 1 is a flowchart of a beamforming method according to an embodiment of the present invention. The beamforming method of the sound source in this embodiment is shown in FIG. 1 and includes the following steps:

Step S110, acquiring spatial filtering parameters. Among them, the spatial filtering parameters are different with different angles and subband frequencies.

In this embodiment, the beamforming in a fixed spatial direction (sound source pointing) can be enhanced by using spatial filtering parameters to ensure that the sound in the pointing direction is substantially unchanged, and the sound in other directions will be suppressed to a certain extent.

The spatial filtering parameter in the embodiment of the present invention is a filter parameter in the frequency domain, and its purpose is to perform corresponding gain or suppression on the subband frequency of the signal of each frame. In an optional implementation manner, the spatial filtering parameter in this embodiment is a matrix, and the spatial filtering parameter is calculated by a computer device, and the obtained spatial filtering parameter is stored in the method for performing the method described in the embodiment of the present invention. The electronic devices are directly used by the power supply sub-devices, thereby reducing the time consumption of beamforming.

In order to facilitate the description, the following embodiments will take the direction of the beam pointing directly in front of 90 ° as an example, that is, the sound source is pointing directly in front of 90 °, but it should be noted that this method is not easy to perform in a limited beam. It is 90 °. In practical applications, the sound source is directed at any angle of plane wave 0 ° -180 °, such as 30 °, 60 °, 120 ° and so on.

In step S120, the sound source corresponding to the spatial filtering parameter is determined.

Step S130: Acquire a sound source pointing to a corresponding original frequency domain signal.

The sound source reaches the microphone array from different directions, causing different microphones to receive signals with different degrees of delay time. The delay time can be used to locate the direction of the beam focus and determine the direction of the sound source that is consistent with the spatial filtering parameters (such as positive 90 ° ahead).

The microphone array is composed of a certain number of acoustic sensors (usually microphones), which are used to sample the spatial characteristics of the sound field. In practical applications, the number of microphones can be 4 in a linear pattern (as shown in Figure 2), with even spacing. Distribution, 6 evenly spaced evenly spaced lines, 8 evenly spaced evenly spaced circles (as shown in Figure 3), 12 or 14 evenly spaced evenly spaced circles, rectangles, crescents, etc. The number and arrangement of the microphone array are not limited in the embodiment of the present invention. However, for the convenience of description, the embodiment of the present invention will be described by taking the four linear microphone arrays 2 shown in FIG. 2 as an example, but it should be clear that this description method is not a specific limitation on the microphone array. .

In the practical application process, considering the characteristics of sound waves, when laying out microphones, the distance between each microphone cannot be easily set too large, nor can it be set too small. If the set distance is not suitable for the sound source, There is an error in the focus positioning. In general, the equidistance between microphones can be set to less than 80 mm and greater than 30 mm.

In this embodiment, when positioning the direction of the beam focus by the delay time, the following method may be adopted, but the delay time for the sound source to reach each microphone may be calculated by the physical structure of the microphone arrangement. Assumption: Determine the microphone distance d, the sound propagation speed c, and the angle Ω at which the sound source points (that is, the angle of the direction in which you want to receive and focus, such as 90 ° directly in front). In the microphone array, select a reference object that reaches the microphone first (such as Mic1 in Figure 2), and calculate the delay time of the first microphone Mic1 as: tau_0 = d * sin (Ω) / c; the second microphone Mic2 The delay time of tau_1 = 2 * d * sin (Ω) / c, the delay time of the third microphone Mic4 is: tau_2 = 3 * d * sin (Ω) / c, and the delay time of the fourth microphone Mic4 is: tau_3 = 4 * d * sin (Ω) / c. Taking the angle Ω of the sound source as 90 ° as an example, usually the first microphone Mic1 is the reference microphone, and the delay time is 0. tau_1 refers to the delay time from the sound field to the second microphone Mic2. The above calculation method of delay time is suitable for linearly spaced microphone arrays. Other calculation methods for microphone distribution and non-equally spaced may be different from the above methods.

The signal vector function is constructed according to the delay time of each microphone array, and the sound source direction is calculated according to the signal vector function and the delay time. When constructing a signal vector function, a matrix corresponding to all subband frequencies needs to be determined. The signal vector function is:

Among them, Ω is the direction angle of sound receiving and focusing, j is the phase at a certain time, ω = 2 * π * f, where f is a matrix corresponding to all subband frequencies, and τ ₀ is the sound source to the first The delay time of the microphone, N is the number of microphones, and τ _(N-1) is the delay time from the sound source to the Nth microphone. Therefore, the sound source direction can be calculated according to the signal vector function and the delay time corresponding to each microphone. Optionally, the matrix corresponding to the subband frequencies corresponding to the sound source is first determined, and the target sound source direction is calculated according to the matrices corresponding to all the subband frequencies corresponding to the sound source, the above-mentioned signal vector function, and the delay time.

In the practical application process, in order to facilitate the subsequent use of sound, it is necessary to first convert the sound signal through the Fourier transform to the time-domain signal (sound signal) that was originally difficult to process into a frequency-domain signal that is easy to analyze. For any continuously measured time sequence or signal, it can be expressed as an infinite superposition of sine wave signals of different frequencies, and the Fourier transform algorithm created according to this principle uses the directly measured original signal to calculate the different sine in the signal in an additive manner. The frequency, amplitude, and phase of the wave signal. A specific implementation manner of the Fourier transform is not described in this embodiment of the present invention.

It should be noted that there is no restriction on the execution between step 110 and step 120. In practical applications, step 110 may be performed first, and then step 120 may be performed, or step 110 and step 120 may be performed simultaneously. This embodiment of the present invention This is not limited.

Step S140: Calculate a product of the obtained spatial filtering parameter and the original frequency domain signal of the sound source to obtain a beamforming output pointed by the sound source. Wherein, the product performs beamforming in a manner of suppressing the original frequency domain signal corresponding to a non-target sound source other than the original frequency domain signal pointed by the sound source.

Wherein, the spatial filtering parameters and the original frequency domain signal are both matrices, and the two matrices are multiplied together, and the product is generated from the original frequency domain signal corresponding to a non-target sound source other than the original frequency domain signal pointed by the sound source. The beamforming is performed in a suppression manner, so that sound signals in a fixed direction are not distorted, and sound signals in other directions are suppressed.

In the beamforming method provided by the embodiment of the present invention, an electronic device obtains a spatial filtering parameter, and the spatial filtering parameter is different with different angles and subband frequencies; determining a sound source direction corresponding to the spatial filtering parameter, and acquiring the sound The source points to the corresponding original frequency domain signal; the product of the spatial filtering parameter and the original frequency domain signal is calculated, and the product is used to suppress the frequency domain signals other than the original frequency domain signal pointed by the sound source. Compared with the prior art, the present invention can not only save the time of beamforming by presetting the spatial filtering parameters in advance, but also can achieve no distortion of sound signals in a fixed direction.

In this embodiment, a computer equipment is used to pre-calculate the spatial filtering parameters corresponding to any angle of plane wave 0 ° -180 °, so as to obtain the corresponding spatial filtering parameters when beamforming the sound source.

FIG. 4 is a flowchart of a method for calculating spatial filtering parameters according to an embodiment of the present invention. In an optional implementation manner, as shown in FIG. 4, calculating the spatial filtering parameters specifically includes the following steps:

Step S1: Calculate the delay time for the sound source to reach the microphone array. The sound source reaches the microphone array from different directions, resulting in different degrees of delay time for signals received by different microphones. The delay time can be used to locate the direction of the beam focus and determine the direction of the sound source that is consistent with the spatial filtering parameters (such as positive 90 ° ahead).

In this embodiment, the calculation of the delay time from the arrival of the sound source to the microphone array may specifically include, but is not limited to, the following steps: determine the microphone distance d, the sound propagation speed c, and the angle Ω that the sound source is pointing at (i.e., you want to receive and focus) Direction angle, such as 90 ° directly in front). The above-mentioned delay time is calculated according to the determined microphone distance d, the sound propagation speed c, and the angle Ω at which the sound source is pointed. For specific methods, refer to step S120, and details are not described herein again.

In step S2, a signal vector function is constructed according to the delay time of each microphone array, and the sound source direction is calculated according to the signal vector function and the delay time. When constructing a signal vector function, a matrix corresponding to all subband frequencies needs to be determined. The signal vector function is:

Among them, Ω is the direction angle of sound receiving and focusing, j is the phase at a certain time, ω = 2 * π * f, where f is a matrix corresponding to all subband frequencies, and τ ₀ is the sound source to the first The delay time of the microphone, N is the number of microphones, and τ _(N-1) is the delay time from the sound source to the Nth microphone. Therefore, the sound source direction can be calculated according to the signal vector function and the delay time corresponding to each microphone. Optionally, the matrix corresponding to the subband frequencies corresponding to the sound source is first determined, and the target sound source direction is calculated according to the matrices corresponding to all the subband frequencies corresponding to the sound source, the above-mentioned signal vector function, and the delay time. For specific explanation, please refer to step S120, which is not repeated here.

Step S3: Calculate a spatial filtering parameter when the loss function approaches a minimum value according to a preset first limitation condition and a second limitation condition. Among them, the loss function is constructed according to the spatial filtering parameters and the signal vector function.

In an optional implementation manner, the preset first limiting condition is a white noise gain limitation.

W _f (ω) is the spatial filtering parameter, T is the transpose operation, H is the conjugate transpose, ω = 2 * π * f, f is the matrix corresponding to all subband frequencies, and Ω is the direction angle of the sound and focus . g (ω, Ω) is a signal vector function. γ is a gain limit of white noise. Optionally, the gain limit of white noise is gamma_db = -20db, and γ is specifically exp (gamma_db / 10). Specifically, the embodiment of the present invention does not limit the specific value of γ.

In an optional implementation manner, the preset second limitation condition is that the product of the spatial filtering parameter and the signal vector function is a first preset value. Preferably, the first preset value is 1. That is, the second restriction condition is: W _f (ω) * g (ω, Ω) = 1. Among them, the spatial filtering parameters and the signal vector function are both matrices, and in general, the matrix of the signal vector function hardly changes.

In the embodiment of the present invention, the spatial conditions of beamforming are limited. In a specific implementation process, the first restriction condition and the second restriction condition must be satisfied at the same time. Optionally, in addition to satisfying the above two limiting conditions, it may also include satisfying a third limiting condition. The third limiting condition is: determining the convexity of the loss function.

Among them, R _nn is the covariance matrix of noise, g (ω, Ω) is a signal vector function, and H is a conjugate transpose.

The loss function constructed according to the spatial filtering parameters and the signal vector function is:

Among them, the loss function b_hat makes the final response at each angle Ω:

According to the first limitation condition and the second limitation condition, calculating the spatial filtering parameter when the loss function approaches a minimum value is as follows:

When calculating the spatial filtering parameters when the loss function approaches the minimum value, it is also necessary to establish equations with the first, second, and third constraints, and use mathematical solutions to solve the spatial filtering parameters. Algorithms for mathematically solving the equations The embodiments of the present invention will not be repeated here.

FIG. 5 is a flowchart of a multi-beam beamforming method according to an embodiment of the present invention. As shown in FIG. 5, the multi-beam beamforming method in this embodiment includes the following steps:

Step S210: Calculate that the target sound source points to the corresponding beamforming output.

In the implementation of the present invention, the source angle of the beam forming sound is directed by at least two sound sources, forming a multi-beam beam forming. In practical applications, the sound source is directed at any angle of plane wave 0 ° -180 °, which needs to be explained. Yes, the at least two sound source directions described in the embodiment of the present invention include a target sound source and at least one other sound source direction. For ease of description, the following embodiments will use beam directions: 0 °, 30 °, 60 °, 90 ° , 120 °, 150 °, and 180 ° directions (a total of 7 directions) are used as an example for explanation. Among them, the target sound source is directed at 90 °. However, it should be noted that this method is not easy to perform in a limited beam. The angle can be 53 °, 80 °, and the target sound source can also be 60 °. It is not limited.

Calculate the product of the original frequency-domain signal and the spatial filtering parameters corresponding to each sound source respectively, and get each single beamforming. The result is also a matrix, and its expression is spectrum. When calculating the product of the corresponding original frequency-domain signal and the spatial filtering parameter of each sound source pointing, each sound source pointing needs to be determined through a microphone array, which specifically includes: the microphone array is composed of a certain number of acoustic sensors (generally microphones) , Used to sample the spatial characteristics of the sound field. In practical applications, the number of microphones can be uniformly distributed at 4 equal intervals (as shown in Figure 2), uniformly distributed at 6 equal intervals, and 8 in circles. Shapes are uniformly distributed at equal intervals (as shown in FIG. 3), 12 or 14 are uniformly distributed at equal intervals such as circles, rectangles, and crescents, etc. The specific embodiment of the present invention does not limit the number and arrangement of microphone arrays. However, for convenience of description, the embodiment of the present invention will be described by taking the four linear microphone arrays 3 shown in FIG. 3 as an example, but it should be clear that this description method is not a specific limitation on the microphone array. .

As another implementation method of the embodiment of the present invention, when calculating the target sound source pointing to the corresponding beamforming output, a GSC (Generalized Sidelobe Cancellation) algorithm may also be used to calculate the beamforming algorithm of a single sound source pointing. There is no limitation on the beamforming algorithm for calculating the direction of a single sound source.

S220. Calculate a noise parameter according to the blocking matrix. Among them, the blocking matrix is used to characterize the frequency response of the sound signal. The purpose of calculating the noise parameter is to reduce the noise of the sound pointed by the non-target sound source. For example, the beam pointing is: 0 °, 30 °, 60 °, 90 °, 120 °, 150 °, 180 ° directions (a total of 7 directions), and the target sound source is 90 °, so the noise parameter is used for the sound Sources are: 0 °, 30 °, 60 °, 120 °, 150 °, 180 ° for noise reduction.

Step S230: Perform noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output to the target sound source according to the noise parameter.

In the specific implementation process, from the target sound source calculated in step S210 to the corresponding beamforming output signal, the signal pointed by the non-target sound source in step S220 is filtered, that is, the noise parameter is used to reduce the signal pointed by the non-target sound source. Noise, in this way, can not only ensure that the target sound source is not distorted by the sound, but also reduce the interference of other sound sources to the sound.

The multi-beam beamforming method provided in the embodiment of the present invention calculates a target sound source to point to a corresponding beamforming output; calculates a noise parameter through a blocking matrix; and points the target sound source outside the corresponding beamforming output according to the noise parameter. Compared with the prior art, the embodiments of the present invention can ensure that the sound pointed by the target sound source is not distorted, and perform noise reduction on the sound pointed by other sound sources, which can effectively suppress other sounds. Directional interference.

Further, as a further extension and refinement of the foregoing embodiment, a specific implementation method of each step is described below in order.

When step S210 is performed to calculate the target sound source pointing to the corresponding beamforming output, the following methods may be adopted, for example: acquiring spatial filtering parameters, and determining the target sound source corresponding to the spatial filtering parameters, and obtaining the target sound source Pointing to the corresponding original frequency domain signal; calculating the product of the spatial filtering parameter and the target sound source pointing to the corresponding original frequency domain signal to obtain the beamforming pointed by the target sound source.

The spatial filtering parameter according to the embodiment of the present invention is a filter parameter in the frequency domain, and its purpose is to make a corresponding gain on the subband frequency of the signal of each frame. In practical applications, the spatial filtering parameters described in the embodiments of the present invention are a matrix. The spatial filtering parameters are calculated by computer equipment, and the obtained spatial filtering parameters are stored in an electronic device that executes the method according to the embodiments of the present invention. In the device, the power supply sub-device is used directly, thereby reducing the time consumption of beamforming.

Obtaining a spatial filtering parameter W _f (ω), and determining a target sound source direction corresponding to the spatial filtering parameter W _f (ω), and respectively obtaining a target sound source pointing to a corresponding original frequency domain signal; calculating the spatial filtering parameter W _{The product of f} (ω) and the original frequency domain signal corresponding to different sound source directions respectively.

In the present embodiment, determining the spatial filtering parameters W _f (ω) corresponding to the target sound source directed at the direction of the beam focused by the delay time positioning, i.e., determining the spatial filter parameters W _f (ω) corresponding to the target sound source point, The following method can be adopted, but not limited to, the delay time of the sound source reaching each microphone can be calculated through the physical structure of the microphone arrangement. Assumption: Determine the microphone distance d, the sound propagation speed c, and the angle Ω at which the sound source points (that is, the angle of the direction in which you want to receive and focus, such as 90 ° directly in front). In the microphone array, select a reference object that reaches the microphone first (such as Mic1 in Figure 2), and calculate the delay time of the first microphone Mic1 as: tau_0 = d * sin (Ω) / c; the second microphone Mic2 The delay time of tau_1 = 2 * d * sin (Ω) / c, the delay time of the third microphone Mic4 is: tau_2 = 3 * d * sin (Ω) / c, and the delay time of the fourth microphone Mic4 is: tau_3 = 4 * d * sin (Ω) / c. Usually the first microphone Mic1 is the reference microphone, so the delay time is 0, tau_1 refers to the delay time from the sound field to the second microphone Mic2. The above calculation method of delay time is suitable for linearly spaced microphone arrays. Other calculation methods for microphone distribution and non-equally spaced may be different from the above methods.

In the practical application process, in order to facilitate the subsequent use of sound, it is necessary to first convert the sound signal through the Fourier transform to the time-domain signal (sound signal) that was originally difficult to process into a frequency-domain signal that is easy to analyze. The principle of the Fourier transform For any continuously measured time sequence or signal, it can be expressed as an infinite superposition of sine wave signals of different frequencies, and the Fourier transform algorithm created according to this principle uses the directly measured original signal to calculate the different sine in the signal in an additive manner. The frequency, amplitude, and phase of the wave signal. A specific implementation manner of the Fourier transform is not described in this embodiment of the present invention.

Further, the spatial filtering parameter W _f (ω) and the original frequency domain signal Z (t, e ^jω ) are both matrices, and the two matrices are multiplied: Y (ω, Ω) = W _f (ω) Z (t , e ^jω ), the product Y (ω, Ω) performs beamforming in a manner that suppresses other frequency-domain signals except the original frequency-domain signal pointed by the target sound source, so that the sound signal in a fixed direction is not distorted.

In step S220, the noise parameter is calculated through the blocking matrix, which can be adopted, but not limited to, for example, by calculating the frequency response of the sound signal reaching the microphone in sequence, and constructing the blocking matrix based on the frequency response, according to the blocking matrix and the non-target sound The source points to the corresponding original frequency domain signal, and the noise parameters are calculated. The purpose of calculating the noise parameter is to reduce the noise of the sound pointed by the non-target sound source.

In an optional implementation manner, first calculate the frequency response of the sound signal reaching the first microphone: A-1 (e ^jω ), and the frequency response of reaching the second microphone: A-2 (e ^jω ), ..., Frequency response of the sound signal reaching the Mth microphone: AM (e ^jω ), A is used to characterize the frequency response function of the microphone.

Construct a group blocking matrix based on the above frequency response:

After the blocking matrix H (e ^jω ) is constructed, the noise parameter is calculated according to the blocking matrix H (e ^jω ) and the non-target sound source pointing to the corresponding original frequency domain signal Z (t, e ^jω ):

U (t, e ^jω ) = H (t, e ^jω ) Z (t, e ^jω )

Among them, t represents the input time of each frame signal.

In the specific implementation process, from the target sound source calculated in step S210 to the corresponding beamforming output signal, the signal pointed by the non-target sound source in step S220 is filtered, that is, the noise parameter U (t, e ^jω ) The signal pointed by the target sound source is denoised, so that it can not only ensure that the target sound source is not distorted, but also reduce the interference of non-target sound sources.

In the actual application process, during the propagation of sound signals, some stable and weak noises such as fans and air conditioners will be included. In order to reduce these noises, in step S230, noise reduction is performed according to the noise parameter U (t, e ^jω ) on the signals pointed by the sound source other than the corresponding beamforming output, which can be adopted, but not limited to The following methods include: calculating multi-channel optimal filtering parameters through a multi-channel filtering algorithm and an iterative algorithm; and pointing the target sound source to the corresponding beam forming output according to the beam forming output, optimal filtering parameters, and noise parameters of the target sound source. Signals pointed to by other sound sources are denoised.

The embodiment of the present invention is described by taking a multi-channel filtering algorithm as a multi-channel Wiener filtering as an example. In order to minimize the impact of the energy of the target sound source pointing to the output, the optimal filtering parameter G is calculated by using a multi-channel Wiener filter and an NLMS iterative method (Normalized Least Mean Square). (t, e ^jω ), further filtering out the stable background noise, and calculating the optimal filtering parameter G (t, e ^jω ), must make E {|| Y (t, e ^jω ) -G (t, e ^jω ) U (t, e ^jω ) || ² } is the smallest, and then an optimal filtering parameter G (t, e ^jω ) is obtained.

After calculating the optimal filtering parameter G (t, e ^jω ) and noise parameter U (t, e ^jω ), the speech output to which the final target sound source is directed is output:

Y = Y (ω, Ω) -G (t, e ^jω ) * U (t, e ^jω )

In order to facilitate the understanding of the final voice output, as shown in FIG. 6, FIG. 6 shows a schematic diagram of the final voice output pointed to by a target sound source according to an embodiment of the present invention, where Y (ω, Ω) in FIG. 7 It is expressed as Y _FBF (t, e ^jω ), and G (t, e ^jω ) * U (t, e ^jω ) is expressed as Y _NC (t, e ^jω ).

In this embodiment, by calculating the target beamforming output corresponding to the target sound source, and reducing the noise of the signal pointed by the non-target sound source according to the noise parameter, this can further ensure that the sound pointed by the target sound source is not distorted and further suppress the non- The interference pointed by the target sound source.

FIG. 7 is a flowchart of another multi-beam beamforming method according to an embodiment of the present invention. As shown in FIG. 7, the multi-beam beamforming method in this embodiment includes the following steps:

Step S340: Calculate a product of the spatial filtering parameters and the original frequency domain signals corresponding to the at least two sound source directions, respectively, to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the frequency of the subband. At least two sound source directions include a target sound source and at least one non-target sound source direction.

The spatial filtering parameter described in this embodiment is a filter parameter in the frequency domain, and its purpose is to make a corresponding gain on the subband frequency of the signal of each frame. In practical applications, the spatial filtering parameters described in the embodiments of the present invention are a matrix. The spatial filtering parameters are obtained through calculation by a computer device. After the calculation results are obtained, the spatial filtering parameters are stored in the electronic device according to the embodiments of the present invention. In the use of power supply equipment, the time consumption of beamforming is shortened. In an optional implementation manner, in this embodiment, the method described in steps S1 to S3 in FIG. 4 may be used to calculate the spatial filtering parameters, and details are not described herein again.

The beam angle of the sound source in this embodiment is directed by at least two sound sources, which constitutes multi-beam beamforming. In practical applications, the sound source is directed at any angle of plane wave 0 ° -180 °. It should be noted that The at least two sound source directions described in the embodiment of the present invention include a target sound source and at least one other sound source direction. For ease of description, the following embodiments will use beam directions: 0 °, 30 °, 60 °, 90 °, 120 The directions of °, 150 °, and 180 ° (a total of 7 directions) are used as an example for description. The target sound source is pointed at 90 °. However, it should be noted that this method is not easy to perform in a limited beam. The above angle can also point to 53 °, 80 °, and the target sound source can also be 60 °, etc., which is not specifically limited.

Calculate the product of the original frequency-domain signal and the spatial filtering parameter corresponding to each sound source pointing separately to obtain each single beamforming. The result is also a matrix whose representation is the frequency spectrum. When calculating the product of the corresponding original frequency-domain signal and the spatial filtering parameter of each sound source pointing, each sound source pointing needs to be determined through a microphone array, which specifically includes: the microphone array is composed of a certain number of acoustic sensors (generally microphones) , Used to sample the spatial characteristics of the sound field. In practical applications, the number of microphones can be uniformly distributed at 4 linear shapes and evenly spaced (as shown in Figure 2), uniformly distributed at 6 linear shapes and evenly spaced, and 8 circled. Shapes are uniformly distributed at equal intervals (as shown in FIG. 3), 12 or 14 are uniformly distributed at equal intervals such as circles, rectangles, and crescents, etc. The specific embodiment of the present invention does not limit the number and arrangement of microphone arrays. However, for the convenience of description, the embodiments of the present invention will be described later using the microphone array style and quantity in FIG. 2 as an example, but it should be clear that this description manner does not specifically limit the microphone array.

Step S320: Calculate the enhanced speech pointed by the target sound source.

In this embodiment, the microphone array 2 in FIG. 2 is used as an example. After obtaining sounds in 7 directions, the 7 segments of sound are subjected to Fourier transform to obtain 7 4 * 512 matrices, where 4 represents the number of microphones. , 512 represents that the spectrum corresponding to different directions is decomposed into 512 subbands respectively. The purpose of this step is to perform filtering processing from the perspective of the subbands, and determine the proportion of all subbands corresponding to the target sound source on each subband.

Assuming that the target sound source is pointed at 90 °, the frequency spectrum corresponding to the target sound source corresponds to α1: 4 * 512 subbands, the 0 ° sound source points to the corresponding spectrum corresponding to α2: 4 * 512 subbands, and the 30 ° sound source points to the corresponding frequency spectrum. α3: 4 * 512 subbands, 60 ° sound source points to the corresponding spectrum corresponds to α4: 4 * 512 subbands, 120 ° sound source points to the corresponding spectrum corresponds to α5: 4 * 512 subbands, and 150 ° sound source points to the corresponding spectrum Corresponding to α6: 4 * 512 subbands, and a 180 ° sound source pointing to the corresponding spectrum corresponds to α7: 4 * 512 subbands. In one implementation, calculating the ratio gain of the target sound source pointing is: α1 / (α1 + α2 + α3 + α4 + α5 + α6 + α7); in another implementation, calculating the target sound source pointing corresponding The ratio gain is: α1 / (α2 + α3 + α4 + α5 + α6 + α7). After obtaining the ratio gain corresponding to the target sound source, obtain the target sound source direction according to the ratio gain and the multi-beam beamforming output calculated in step S310 (that is, the product of the spatial filtering parameter and the original frequency domain signals corresponding to at least two sound source directions). Enhanced speech. Optionally, the product of the first product and the ratio gain corresponding to the target sound source is calculated. Wherein, the first product is a product between the target sound source pointing to the corresponding original frequency domain signal and the spatial filtering parameter.

Step S330: Calculate an energy ratio based on the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source.

In an optional implementation manner, multiple subbands of the current frame spectrum decomposition are combined, and the energy of the combined subbands is obtained. The current frame includes a target sound source and a non-target sound source. In the specific implementation process, the 512 subbands corresponding to the target sound source are combined first, and the combined subband energy is determined. Secondly, merge the 512 sub-bands pointed by the other 6 sound sources (or 7 sound sources, including the target sound source) in order, and determine the sub-band energy pointed by each combined sound source. Finally, calculate 6 The sum of the energy of all subbands pointed by the sound source (or 7 sound sources, including the target sound source). The energy sum is a matrix.

The energy ratio is calculated based on the sum of the energy of the subbands corresponding to the target sound source and the energy of all the subbands pointed by the 6 sound sources (or 7 sound sources, including the target sound source).

Step S340: Calculate a product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech and the energy ratio pointed by the target sound source to reduce noise to the non-target sound source, and output the speech corresponding to the product.

Obtain the target sound source pointing to the corresponding original frequency domain signal, and calculate the product between the original frequency domain signal and the target sound source obtained at step S320 pointing to the corresponding enhanced speech, the energy ratio calculated at step S330, and the beam obtained according to the product The shaping can ensure that the sound pointed by the target sound source is not distorted, and at the same time, can suppress the noise generated in the direction of the non-target sound source.

In the method for multi-beam beamforming provided by the embodiment of the present invention, multi-beam beamforming is obtained by calculating a product of a spatial filtering parameter and at least two sound source directions corresponding to original frequency domain signals, respectively. The product of the voice, energy ratio, and the original frequency domain signal pointed by the target sound source to output the speech corresponding to the product, thereby achieving noise reduction processing for non-target sound sources and ensuring that the sound pointed by the target sound source is not distorted.

FIG. 8 is a flowchart of another multi-beam beamforming method according to an embodiment of the present invention. As a refinement and extension of the foregoing embodiment, an embodiment of the present invention also provides another method for multi-beam beamforming. As shown in FIG. 8, the method for multi-beam beamforming in this embodiment includes the following steps:

Step S410: Calculate a product of the spatial filtering parameter and the original frequency domain signals corresponding to the at least two sound source directions, respectively, to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the frequency of the subband. At least two sound source directions include a target sound source and at least one non-target sound source direction.

When calculating the product of the spatial filtering parameter W _f (ω) and the original frequency-domain signals corresponding to at least two sound source directions respectively, to obtain multi-beam beamforming, the following methods can be adopted, but not limited to:

Acquire a spatial filtering parameter W _f (ω), and determine the respective sound source directions corresponding to the spatial filtering parameter W _f (ω), and separately obtain each sound source pointing to a corresponding original frequency domain signal; calculate the spatial filtering parameter The products of W _f (ω) and original frequency domain signals corresponding to different sound source directions respectively.

In the present embodiment, the spatial filter is determined parameter W _f (ω) corresponding to at least two points when the sound source direction of the beam focused by positioning the delay time, i.e., determining the spatial filter parameters W _f (ω) corresponding to the target sound source For pointing, you can use but not limited to the following methods to calculate the delay time for the sound source to reach each microphone through the physical structure of the microphone arrangement. Assumption: Determine the microphone distance d, the sound propagation speed c, and the angle Ω at which the sound source points (that is, the angle of the direction in which you want to receive and focus, such as 90 ° directly in front). In the microphone array, select a reference object that reaches the microphone first (such as Mic1 in Figure 2), and calculate the delay time of the first microphone Mic1 as: tau_0 = d * sin (Ω) / c; the second microphone Mic2 The delay time of tau_1 = 2 * d * sin (Ω) / c, the delay time of the third microphone Mic4 is: tau_2 = 3 * d * sin (Ω) / c, and the delay time of the fourth microphone Mic4 is: tau_3 = 4 * d * sin (Ω) / c. Usually the first microphone Mic1 is the reference microphone, so the delay time is 0, tau_1 refers to the delay time from the sound field to the second microphone Mic2. The above calculation method of delay time is suitable for linearly spaced microphone arrays. Other calculation methods for microphone distribution and non-equally spaced may be different from the above methods.

Further, the spatial filtering parameter W _f (ω) and the original frequency domain signal Z (t, e ^jω ) are both matrices, and the two matrices are multiplied: Y (ω, Ω) = W _f (ω) Z (t , e ^jω ), the product Y (ω, Ω) performs beamforming in a manner that suppresses frequency-domain signals other than the original frequency-domain signal pointed by the target sound source, so that the sound signal in a fixed direction is not distorted, and, Suppress sound signals in other directions.

In this embodiment, it is assumed that there are 7 sound source points (including a 90 ° target sound source point) and 4 microphones (such as the microphone array 3 shown in FIG. 2) to collect sounds. The beam directions calculated by the above method are respectively : 0 °, 30 °, 60 °, 90 °, 120 °, 150 °, 180 ° directions (7 directions in total) for single beam forming. 7 7 * 512 matrices are obtained, 4 represents the number of microphones, and 512 represents the frequency spectrum corresponding to different directions is decomposed into 512 subbands respectively.

Step S420: Calculate the enhanced speech pointed by the target sound source.

In practical applications, the following methods are used to calculate the enhanced speech pointed by the target sound source, including:

Using each subband as a unit, calculate the ratio gain between the energy pointed by the target sound source and the energy sum directed by all sound sources; calculate the product of the first product B (ω, Ω) and the ratio gain to obtain enhanced speech, where: The first product is a product between the target sound source pointing to a corresponding original frequency domain signal and the spatial filtering.

When calculating the energy sums pointed by all sound sources, the essence is to merge the 4 microphones, that is, to obtain 7 1 * 512 matrices, and obtain the energy sums pointed by all sound sources as Spectrum power of other directions, continue Obtain the energy pointed by the target sound source and record it as: Spectrum power of target directions. Calculate the ratio of the energy pointed by the target sound source to the energy of the target direction and the energy pointed by all sound sources and the Spectrum power of other directions to get the ratio gain Gain- mask.

Continue to calculate the product of the first product B (ω, Ω) and the ratio gain Gain-mask to obtain the enhanced speech Gain-mask-frame = B (ω, Ω) * Gain-mask.

Step S430: Calculate an energy ratio based on the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source.

Specifically, the energies of all subbands in the current frame are combined, and the energy sum of all subbands in the current frame is calculated; the energy of the subband corresponding to the target sound source and the energy of all subbands pointed to by the non-target sound source are calculated. The ratio between and to get the energy ratio. Alternatively, the ratio between the energy of the subband corresponding to the target sound source and the energy sum of all subbands in the current frame is calculated to obtain the energy ratio.

The current frame contains all subbands in the direction of the 7 sound sources. The energy corresponding to all subbands in the current frame is combined. First, all the subbands pointed by each sound source are combined to obtain the spectrum corresponding to the different directions. 7 * 1 matrix, where 7 is the direction of 7 sound sources and 1 is the combined subband (spectrum). Secondly, all subbands corresponding to different directions are combined to obtain a 1 * 1 matrix, that is, the energy sum of all subbands is obtained according to the matrix, and it is denoted as Energy of each bin direction. Third, obtain the subband energy corresponding to the target sound source, and record it as: Energy of each target in the target directions. Finally, calculate the subband energy corresponding to the target sound source and the energy of all subbands pointed by the non-target sound source. (The energy sum of all the sound sources pointing to the corresponding subbands in the current frame) to get the energy ratio, which is recorded as: Gain-mask-frame-bin.

Step S240: Perform frame-by-frame smoothing processing on the current frame and the previous frame through the smoothing parameters.

In the embodiment of the present invention, the purpose of performing the smoothing process is to enable a smooth transition of speech before two consecutive frames. Therefore, when smoothing the current frame and the previous frame frame by frame using the smoothing parameter, the following manners can be adopted but not limited to:

Set the smoothing parameters of the current frame so that the sum of the smoothing parameters of the current frame and the smoothing parameters of the previous frame is the second preset value. Preferably, the second preset value is 1. Calculate the product of the ratio gain of the previous frame and the smoothing parameter corresponding to the previous frame to obtain a second product, and calculate the product of the ratio gain and the smoothing parameter corresponding to the current frame to obtain a third product. The frame-by-frame smoothing process is performed on the sound source in the current frame according to the sum of the second product and the third product.

In an optional implementation manner, the smoothing parameter γ is an empirical value, and the smoothing parameter γ of the current frame can be set to 0.8, then the smoothing parameter of the previous frame is (1-γ) = 0.2. Specifically, the present invention The embodiment does not limit this. Thus, the ratio gain of the current frame can be obtained to perform frame-by-frame smoothing processing on the sound source in the current frame. Assume that the ratio gain of the previous frame is the ratio gain of the previous frame is Previous Gain. Then the current frame's ratio gain Current Gain = Previous Gain * (1-γ) + γ * Gain-mask = Previous Gain * (1-γ) + γ * Spectrum power of target directions / Spectrum power of other directions.

Step S450: Calculate a product of the corresponding enhanced speech and energy ratio pointed by the target sound source and the original frequency domain signal pointed by the target sound source, and output the speech corresponding to the product according to the smoothing result.

In this embodiment, multi-beam beamforming is obtained by calculating a product of spatial filtering parameters and original frequency domain signals corresponding to at least two sound source directions respectively, and by calculating an enhanced voice, an energy ratio, and a target sound source direction of the target sound source, The product of the original frequency domain signal, while smoothing the current frame and the previous frame by smoothing parameters, and outputting the speech corresponding to the product according to the smoothing processing result, further reducing the noise of non-target sound sources, and further ensuring The sound pointed by the target sound source is not distorted.

Further, as an implementation of the method shown in FIG. 1 described above, another embodiment of the present invention further provides a voice processing apparatus. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment does not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can correspondingly implement the foregoing method implementation. Everything in the example.

Further, as an implementation of the method shown in FIG. 1, another embodiment of the present invention further provides a beamforming apparatus. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment does not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can correspondingly implement the foregoing method implementation. Everything in the example.

FIG. 9 is a schematic diagram of a beamforming apparatus according to an embodiment of the present invention. FIG. 10 is a schematic diagram of another beamforming apparatus according to an embodiment of the present invention. As shown in FIG. 9, the beamforming apparatus 9 of this embodiment includes a first obtaining unit 91, a determining unit 92, a second obtaining unit 93, and a first calculating unit 94.

The first obtaining unit 91 is configured to obtain spatial filtering parameters, and the spatial filtering parameters are different according to different angles and subband frequencies. The determining unit 92 is configured to determine a sound source corresponding to the spatial filtering parameter obtained by the first obtaining unit 91. The second obtaining unit 93 is configured to obtain that the sound source determined by the determining unit 92 points to a corresponding original frequency domain signal. The first calculation unit 94 is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal, and the product is used to suppress other frequency domain signals except the original frequency domain signal pointed by the sound source.

Further, as shown in FIG. 10, the beamforming device 9 further includes:

The second calculation unit 95 is configured to calculate the spatial filtering parameters before the first obtaining unit 93 obtains the spatial filtering parameters.

Further, as shown in FIG. 10, the second calculation unit 95 includes:

The first calculation module 951 is configured to calculate a delay time when the sound source reaches the microphone array. A building module 952 is used to build a signal vector function. A second calculation module 953 is configured to calculate a sound source direction according to the signal vector function constructed by the construction module 952 and the delay time calculated by the first calculation module 951. The first setting module 954 is configured to set a first limiting condition, where the first limiting condition is a white noise gain limitation. The second setting module 955 is configured to set a second limitation condition, where the second limitation condition is that a product of the spatial filtering parameter and the signal vector function is 1. A construction module 956 is configured to construct a loss function according to the spatial filtering parameter and the signal vector function. A third calculation module 957 is configured to calculate the loss function according to the first restriction condition set by the first setting module 954 and the second restriction condition set by the second setting module 955. Spatial filtering parameters towards the minimum.

Further, as shown in FIG. 10, the first calculation module 951 includes:

The first determining sub-module 951a is configured to determine a distance between microphones in the microphone array, and a speed at which a sound source propagates sound. The second determining sub-module 951b is configured to determine an angle pointed by the sound source. A calculation sub-module 951c is configured to calculate a delay time according to a distance, a speed, and an angle between the microphones.

Further, as shown in FIG. 12, the second calculation module 953 includes:

A determining sub-module 953a is configured to determine a matrix corresponding to all sub-band frequencies. A calculation sub-module 953b is configured to calculate a sound source direction according to the matrices corresponding to all the sub-band frequencies, the signal vector function, and the delay time determined by the determination sub-module.

Further, the spatial filtering parameter is a matrix.

Since the beamforming device described in this embodiment is a device that can execute the beamforming method in the embodiment of the present invention, based on the beamforming method described in the embodiment of the present invention, those skilled in the art can understand the The specific implementations of the beamforming device and its various variations, so how to implement the beamforming method in the embodiment of the present invention with the beamforming device will not be described in detail here. As long as a device used by a person skilled in the art to implement the beamforming method in the embodiment of the present invention falls within the protection scope of the present application.

Further, as an implementation of the method shown in FIG. 5, another embodiment of the present invention further provides a multi-beam beamforming apparatus. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment does not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can correspondingly implement the foregoing method implementation. Everything in the example.

FIG. 11 is a schematic diagram of a multi-beam beamforming apparatus according to an embodiment of the present invention. FIG. 12 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention. As shown in FIG. 11, the multi-beam beamforming apparatus 11 of this embodiment includes a first calculation unit 111, a second calculation unit 112, and a noise reduction unit 113.

The first calculation unit 111 is configured to calculate that the target sound source points to a corresponding beamforming output. The second calculation unit 112 is configured to calculate a noise parameter by using a blocking matrix. The noise reduction unit 113 is configured to perform, according to the noise parameter calculated by the second calculation unit 112, a signal pointed by the target sound source calculated by the first calculation unit 111 to a non-target sound source other than the corresponding beamforming output. Noise reduction.

Further, as shown in FIG. 12, the first calculation unit 111 includes:

The first obtaining module 1111 is configured to obtain spatial filtering parameters.

A determining module 1112 is configured to determine a target sound source corresponding to the spatial filtering parameter obtained by the first obtaining module 1111.

The second acquisition module 1113 is configured to acquire the target sound source acquired by the first acquisition module 1111 to point to the corresponding original frequency domain signal.

A calculation module 1114 is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source pointing to obtain the beamforming pointed by the target sound source.

Further, as shown in FIG. 12, the second calculation unit 112 includes:

The first calculation module 1121 is configured to calculate a frequency response of the sound signal reaching the microphone in order.

A construction module 1122 is configured to construct the blocking matrix according to the frequency response calculated by the first calculation module.

A second calculation module 1123 is configured to calculate the noise parameter according to the blocking matrix constructed by the construction module and the other sound sources pointing to corresponding original frequency domain signals.

Further, as shown in FIG. 12, the noise reduction unit 113 includes:

A calculation module 1131 is configured to calculate a multi-channel optimal filtering parameter by using a multi-channel filtering algorithm and an iterative algorithm.

The noise reduction module 1132 is configured to perform, according to the beamforming output of the target sound source, an optimal filtering parameter, and the noise parameter, a signal directed by the sound source directed by a sound source other than the corresponding beamforming output. Noise reduction.

Since the multi-beam beamforming apparatus described in this embodiment is an apparatus that can execute the multi-beam beamforming method in the embodiment of the present invention, based on the multi-beam beamforming method described in the embodiment of the present invention, those skilled in the art The specific implementations of the multi-beam beamforming apparatus of this embodiment and its various variations can be understood, so how to implement the multi-beam beamforming apparatus in the embodiment of the present invention will not be described in detail here. As long as a device used by a person skilled in the art to implement the multi-beam beamforming method in the embodiment of the present invention falls within the scope of the present application.

Further, as an implementation of the method shown in FIG. 7, another embodiment of the present invention further provides a multi-beam beamforming apparatus. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment does not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can correspondingly implement the foregoing method implementation. Everything in the example.

FIG. 13 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention. FIG. 14 is a schematic diagram of another multi-beam beamforming apparatus according to an embodiment of the present invention. As shown in FIG. 13, the multi-beam beamforming apparatus 13 in this embodiment includes a first calculation unit 131, a second calculation unit 132, a third calculation unit 133, and a fourth calculation unit 134.

The first calculation unit 131 is configured to calculate a product of the spatial filtering parameter and the original frequency domain signals corresponding to the at least two sound source directions respectively to obtain multi-beam beamforming. The spatial filtering parameter varies with the angle of the sound source and the subband frequency. Each sound source point includes a target sound source point and at least one non-target sound source point.

The second calculation unit 132 is configured to separately calculate the enhanced speech pointed by the target sound source.

The third calculation unit 133 is configured to calculate an energy ratio according to the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source.

A fourth calculation unit 134 is configured to calculate a product of the original frequency domain signal pointed by the target sound source and the enhanced speech and energy ratio corresponding to the target sound source direction, and output the speech corresponding to the product.

Further, as shown in FIG. 14, the multi-beam beamforming device 13 further includes:

A processing unit 135, configured to: before the fourth calculation unit 134 calculates a product of the original frequency domain signal pointed by the target sound source and the target speech source pointed by the corresponding enhanced speech and energy ratio, smoothing the current frame One frame is smoothed frame by frame.

Further, as shown in FIG. 14, the first calculation unit 131 includes:

The first obtaining module 1311 is configured to obtain spatial filtering parameters.

A determining module 1312 is configured to determine at least two sound source directions respectively corresponding to the spatial filtering parameters obtained by the first obtaining module 1311.

A second acquisition module 1313 is configured to acquire at least two sound sources determined by the determination module to point to corresponding original frequency domain signals.

A calculation module 1314 is configured to calculate products of the spatial filtering parameters and original frequency domain signals corresponding to different sound source directions, respectively.

Further, as shown in FIG. 14, the second calculation unit 132 includes:

The first calculation module 1321 is configured to calculate a ratio gain between the energy pointed by the target sound source and the energy sum pointed by all the sound sources by using each subband as a unit.

A second calculation module 1322 is configured to calculate a product of a first product and a ratio gain to obtain enhanced speech, where the first product is a signal between the target sound source pointing to a corresponding original frequency domain signal and the spatial filtering. product.

Further, as shown in FIG. 14, the third calculation unit 133 includes:

A combining module 1331 is configured to combine the energy corresponding to all subbands in the current frame.

The first calculation module 1332 is configured to calculate energy sums of all subbands in the current frame.

A second calculation module 1333 is configured to calculate a ratio between the energy of the sub-band corresponding to the target sound source and the energy of all the sub-bands pointed to by at least one non-target sound source to obtain an energy ratio.

Further, as shown in FIG. 14, the processing unit 135 includes:

A setting module 1351 is used to set the smoothing parameters of the current frame so that the sum of the smoothing parameters of the current frame and the smoothing parameters of the previous frame is 1.

A calculation module 1352 is configured to calculate a product of a ratio gain of a previous frame and a corresponding smoothing parameter to obtain a second product, and calculate a product of a smoothing parameter of the current frame and the ratio gain to obtain a third product.

The processing module 1353 is configured to perform frame-by-frame smoothing processing on the current frame according to the sum of the first product and the second product.

Further, the fourth calculation unit 134 is further configured to calculate a product of the target sound source pointing to a corresponding enhanced voice, an energy ratio, and the original frequency domain signal pointed to by the target sound source, and output a smoothing result according to a smoothing result. The speech corresponding to the product is described.

The apparatus for multi-beam beamforming provided by the embodiment of the present invention calculates a product of a spatial filtering parameter and at least two sound sources pointing to corresponding original frequency-domain signals to obtain multi-beam beamforming. The spatial filtering parameter varies with the angle of the sound source. Different from the sub-band frequency, the at least two sound source directions include a target sound source and at least one other sound source direction; calculating an enhanced speech pointed by the target sound source; and according to the sub-band energy corresponding to the target sound source and at least one Sum of the energy of all subbands pointed by other sound sources, calculate the energy ratio; calculate the product of the original frequency-domain signal pointed by the target sound source and the target sound source pointed to the corresponding enhanced speech, energy ratio, and output the product corresponding to the product Compared with the prior art, the embodiment of the present invention can ensure that the sound pointed by the target sound source is not distorted, and can effectively suppress interference from other sound directions.

Each of the foregoing devices includes a processor and a memory. Each unit in the device is stored in the memory as a program unit, and the processor executes the program unit stored in the memory to implement a corresponding function. The processor contains a kernel, and the kernel retrieves the corresponding program unit from the memory. The kernel can set one or more, and when adjusting the kernel parameters to implement the above method, ensure that the sound pointed by the target space is not distorted, and the sound pointed by other spaces is effectively suppressed. Memory may include non-persistent memory, random access memory (RAM), and / or non-volatile memory in computer-readable media, such as read-only memory (ROM) or flash memory (flashRAM). Memory includes at least one storage chip.

An embodiment of the present invention provides a storage medium on which a program is stored, and when the program is executed by a processor, the foregoing voice processing method is implemented.

An embodiment of the present invention provides a processor, where the processor is configured to run a program, and when the program runs, the foregoing voice processing method is performed.

FIG. 15 is a structural block diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 15, the electronic device 17 includes:

At least one processor 151;

And at least one memory 152 and bus 153 connected to the processor 151;

The processor 151 and the memory 152 complete communication with each other through the bus 153;

The processor 151 is configured to call program instructions in the memory 152 to execute any one of the foregoing methods.

The electronic devices in this article can be servers, PCs, PADs, mobile phones, smart TVs, and other smart devices that include microphones.

The electronic device provided by the embodiment of the present invention obtains a beamforming output pointed by the target sound source by calculating a product of a spatial filtering parameter and a target original sound source signal corresponding to the target sound source pointing, and performs noise reduction by pointing to a non-target sound source The processing improves the signal-to-noise ratio of the beamforming output pointed by the target sound source. Therefore, it is possible to ensure that the sound pointed by the target space is not distorted, and effectively suppress the sound pointed by other target spaces, thereby improving the signal-to-noise ratio of the sound pointed by the target space.

An embodiment of the present invention further provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute any one of the foregoing voice processing methods.

This application also provides a computer program product that, when executed on a data processing device, implements the functions of any of the above-mentioned speech processing methods.

Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, this application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

This application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, so that instructions generated by the processor of the computer or other programmable data processing device may be used to Means for implementing the functions specified in one or more flowcharts and / or one or more blocks of the block diagrams.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to work in a specific manner such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, the instructions The device implements the functions specified in one or more flowcharts and / or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of steps can be performed on the computer or other programmable device to produce a computer-implemented process, which can be executed on the computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

Memory may include non-persistent memory, random access memory (RAM), and / or non-volatile memory in computer-readable media, such as read-only memory (ROM) or flash memory (flashRAM). Memory is an example of a computer-readable medium.

Computer-readable media includes permanent and non-persistent, removable and non-removable media. Information storage can be accomplished by any method or technology. Information may be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transmitting medium may be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "including", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, product, or device that includes a series of elements includes not only those elements, but also Other elements not explicitly listed, or those that are inherent to such a process, method, product, or device. Without more restrictions, the elements defined by the sentence "including a ..." do not exclude that there are other identical elements in the process, method, product or equipment including the elements.

The above are only examples of the present application and are not intended to limit the present application. For those skilled in the art, this application may have various modifications and changes. Any modification, equivalent replacement, and improvement made within the spirit and principle of this application shall be included in the scope of claims of this application.

Claims

A beamforming method includes:

Acquiring a spatial filtering parameter, which is different with different angles and subband frequencies; determining the sound source direction corresponding to the spatial filtering parameter, and obtaining the original frequency domain signal corresponding to the sound source direction;

Calculate a product of the spatial filtering parameter and the original frequency domain signal, where the product is used to perform beamforming in a manner that suppresses other frequency domain signals other than the original frequency domain signal pointed by the sound source.
The method according to claim 1, wherein before the obtaining the spatial filtering parameters, the method further comprises:

Calculate the spatial filtering parameters.
The method according to claim 2, wherein the calculating spatial filtering parameters comprises:

Calculate the delay time for the sound source to reach the microphone array;

Constructing a signal vector function according to the delay time, and calculating a sound source direction according to the signal vector function and the delay time;

Calculating a spatial filtering parameter when the loss function approaches a minimum value according to a preset first limiting condition and a second limiting condition, and the loss function is constructed according to the spatial filtering parameter and the signal vector function;

The first limitation condition is specifically a white noise gain limitation; the second limitation condition is that a product of the spatial filtering parameter and the signal vector function is a first preset value.
The method according to claim 3, wherein calculating the delay time for the sound source to reach the microphone array comprises:

Determine the spacing between the microphones in the microphone array, and the speed at which the sound source propagates the sound;

Determining an angle at which the sound source is pointing;

The delay time is calculated according to a distance between the microphones, a speed at which the sound source propagates sound, and an angle at which the sound source points.
The method according to claim 3, wherein calculating a sound source direction according to the signal vector function and the delay time comprises:

Determine a matrix corresponding to all subband frequencies;

Calculate a sound source direction according to the matrix corresponding to all subband frequencies, the signal vector function, and the delay time.
The method according to any one of claims 1-5, wherein the spatial filtering parameter is a matrix.
The method according to any one of claims 1-5, wherein the sound source is directed at any angle of 0 ° -180 ° of a plane wave.
A beamforming device includes:

A first obtaining unit, configured to obtain spatial filtering parameters, which are different with different angles and subband frequencies;

A determining unit, configured to determine a sound source corresponding to the spatial filtering parameter obtained by the first obtaining unit;

A second obtaining unit, configured to obtain that the sound source determined by the determining unit points to a corresponding original frequency domain signal;

A first calculation unit is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal, and the product is used to perform suppression in a frequency domain signal other than the original frequency domain signal pointed by the sound source. Beamforming.
A method for multi-beam beamforming includes:

Calculate the target sound source pointing to the corresponding beamforming output;

Calculate noise parameters based on the blocking matrix;

Performing noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output directed by the target sound source according to the noise parameter.
The method according to claim 9, wherein calculating the target sound source pointing to the corresponding beamforming output comprises:

Acquiring spatial filtering parameters, and determining the target sound source direction corresponding to the spatial filtering parameters;

Acquiring that the target sound source points to a corresponding original frequency domain signal;

Calculate a product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source pointing to obtain the beamforming output pointed by the target sound source.
The method according to claim 10, wherein calculating the noise parameter according to the blocking matrix comprises:

Calculate the frequency response of the sound signal to the microphone in turn;

Constructing the blocking matrix according to the frequency response;

Calculate the noise parameter according to the blocking matrix and the non-target sound source pointing to a corresponding original frequency domain signal.
The method according to claim 11, wherein performing noise reduction on a signal pointed by a non-target sound source other than the corresponding beamforming output of the target sound source according to the noise parameter comprises:

Calculate multi-channel optimal filtering parameters through multi-channel filtering algorithm and iterative algorithm;

Performing noise reduction on a signal directed by a non-target sound source other than the corresponding beamforming output according to the beamforming output of the target sound source, the multi-channel optimal filtering parameter, and the noise parameter .
A multi-beam beamforming device, comprising:

A first calculation unit, configured to calculate that a target sound source points to a corresponding beamforming output;

A second calculation unit, configured to calculate a noise parameter by using a blocking matrix;

A noise reduction unit, configured to perform, according to the noise parameter calculated by the second calculation unit, a signal pointed by the target sound source calculated by the first calculation unit to a non-target sound source other than a corresponding beamforming output; Noise reduction.
The apparatus according to claim 13, wherein the first calculation unit comprises:

A first acquisition module, configured to acquire spatial filtering parameters;

A determining module, configured to determine a target sound source direction corresponding to the spatial filtering parameter obtained by the first obtaining module;

A second acquisition module, configured to acquire a target sound source acquired by the first acquisition module to point to a corresponding original frequency domain signal;

A calculation module is configured to calculate a product of the spatial filtering parameter and the original frequency domain signal corresponding to the target sound source pointing to obtain a beamforming output pointed by the target sound source.
The apparatus according to claim 14, wherein the second calculation unit comprises:

A first calculation module, configured to calculate a frequency response in which a sound signal reaches the microphone in order;

A construction module, configured to construct the blocking matrix according to the frequency response calculated by the first calculation module;

A second calculation module is configured to calculate the noise parameter according to the blocking matrix constructed by the construction module and the non-target sound source pointing to a corresponding original frequency domain signal.
The apparatus according to claim 14, wherein the noise reduction unit comprises:

A calculation module for calculating multi-channel optimal filtering parameters through a multi-channel filtering algorithm and an iterative algorithm;

A noise reduction module, configured to point the target sound source to a non-target sound source other than the corresponding beamforming output according to the beamforming output of the target sound source, the multi-channel optimal filtering parameter, and the noise parameter The pointed signal is denoised.
A method for multi-beam beamforming includes:

Calculate the product of the spatial filtering parameters and at least two sound sources pointing to the corresponding original frequency domain signals to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the subband frequency. The at least two The sound source pointing includes a target sound source and at least one non-target sound source pointing;

Calculating the enhanced speech pointed to by the target sound source;

Calculate the energy ratio according to the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source;

A product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the energy ratio is calculated, and the speech corresponding to the product is output.
The method according to claim 17, wherein before calculating a product of the original frequency domain signal pointed by the target sound source, the target sound source pointed at the corresponding enhanced speech, and the product of the energy ratio, the method further include:

Perform smoothing frame-by-frame on the current frame and the previous frame through the smoothing parameters.
The method according to claim 18, wherein the calculating the product of the spatial filtering parameters and the original frequency domain signals corresponding to the at least two sound source directions respectively to obtain multi-beam beamforming comprises:

Acquiring spatial filtering parameters, and determining at least two sound source directions respectively corresponding to the spatial filtering parameters;

Acquiring at least two sound sources pointing to respective original frequency domain signals;

Calculate products of the spatial filtering parameters and the original frequency domain signals corresponding to the at least two sound source directions, respectively.
The method according to claim 19, wherein the calculating the enhanced speech pointed by the target sound source comprises:

Calculate a ratio gain between the energy pointed by the target sound source and the energy sum pointed by all sound sources with each subband as a unit;

Calculate a product of a first product and the ratio gain to obtain the enhanced speech, wherein the first product is a product between the target sound source pointing to a corresponding original frequency domain signal and the spatial filtering parameter.
The method according to claim 20, wherein calculating an energy ratio based on a sum of subband energy corresponding to the target sound source and energy of all subbands pointed to by at least one non-target sound source comprises:

Combine the energy corresponding to all subbands in the current frame to calculate the energy sum of all subbands in the current frame;

Calculate the ratio between the energy of the subband corresponding to the target sound source and the energy sum of all the subbands pointed to by at least one non-target sound source to obtain the energy ratio.
The method according to claim 21, wherein performing frame-by-frame smoothing processing on the current frame and the previous frame by using a smoothing parameter comprises:

Set the smoothing parameters of the current frame so that the sum of the smoothing parameters of the current frame and the smoothing parameters of the previous frame is the second preset value;

Calculate the ratio gain of the previous frame and the smoothing parameters of the previous frame to obtain the second product;

Calculate the product of the ratio gain of the current frame and the smoothing parameter of the current frame to obtain a third product;

Performing frame-by-frame smoothing processing on the current frame according to the sum of the second product and the third product.
The method according to any one of claims 18 to 22, wherein a product of an original frequency domain signal pointed by the target sound source, a corresponding enhanced speech pointed by the target sound source, and the energy ratio is calculated, And outputting the speech corresponding to the product includes:

A product of the original frequency domain signal pointed by the target sound source, the corresponding enhanced speech pointed by the target sound source, and the energy ratio is calculated, and the speech corresponding to the product is output according to the smoothing processing result.
A multi-beam beamforming device, comprising:

A first calculation unit is configured to calculate a product of spatial filtering parameters and original frequency domain signals corresponding to at least two sound source directions, respectively, to obtain multi-beam beamforming. The spatial filtering parameters vary with the angle of the sound source and the subband frequency. Differently, the at least two sound source points include a target sound source point and at least one non-target sound source point sound source point;

A second calculation unit, configured to separately calculate the enhanced speech pointed by the target sound source;

A third calculation unit, configured to calculate an energy ratio based on the sum of the energy of the subband corresponding to the target sound source and the energy of all the subbands pointed to by at least one non-target sound source;

A fourth calculation unit is configured to calculate a product of the original frequency domain signal pointed by the target sound source, a corresponding enhanced speech pointed by the target sound source, and the energy ratio, and output a speech corresponding to the product.
A storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the method according to any one of claims 1-7 and / or any one of claims 9-12 And / or the method according to any of claims 17-23.
An electronic device, characterized in that the electronic device includes a processor, a memory, and a bus; the processor and the memory complete communication with each other through the bus; and the memory is used to store program instructions, The program instructions are executed by the processor to implement the method according to any of claims 1-7 and / or the method according to any of claims 9-12 and / or claim 17 The method according to any of -23.