WO2022253003A1

WO2022253003A1 - Speech enhancement method and related device

Info

Publication number: WO2022253003A1
Application number: PCT/CN2022/093969
Authority: WO
Inventors: 魏善义; 吴超; 邱炎; 廖猛; 范泛; 彭世强; 李斌; 赵文斌; 李江; 李海婷; 黄雪妍
Original assignee: 华为技术有限公司
Priority date: 2021-05-31
Filing date: 2022-05-19
Publication date: 2022-12-08
Also published as: CN117480554A; US20240096343A1

Abstract

A speech enhancement method and a related device. The method comprises: after a terminal device enters a PNR mode, obtaining a first noisy speech signal and target speech related data, the first noisy speech signal comprising an interference noise signal and a speech signal of a target user, and the target speech related data being used for indicating a speech feature of the target user (S301); and according to the target speech related data, performing noise reduction processing on the first noisy speech signal by means of a speech noise reduction model to obtain a noise reduction speech signal of the target user, wherein the speech noise reduction model is implemented on the basis of a neural network (S302). Speech enhancement of the target user and suppression of interference are realized.

Description

Speech enhancement method and related equipment

This application is required to be submitted to the State Intellectual Property Office of China on May 31, 2021. The application number is 202110611024.0. The title of the invention is "Voice Enhancement Method and Related Equipment". It is submitted to the State Intellectual Property Office of China on June 22, 2021. The application number is 202110694849.3, the title of the invention is "Voice Enhancement Method and Related Equipment" and the Chinese patent application with the application number 202111323211.5 and the title of invention "Voice Enhancement Method and Related Equipment" submitted to the State Intellectual Property Office of China on November 09, 2021 priority, the entire contents of which are incorporated by reference into this application.

technical field

The present application relates to the field of speech processing, in particular to a speech enhancement method and related equipment.

Background technique

In recent years, smart devices have greatly enriched people's lives. When the device works in a quiet scene, the voice call quality and voice interaction (wake-up and recognition rate) functions can better meet the needs, but when the device works in the environment noise 1. Under the scene conditions of voice interference, the experience effect of voice call quality, wake-up rate and recognition rate will decrease, and it is necessary to rely on the voice enhancement algorithm to achieve the purpose of enhancing the target voice and filtering out interference.

Environmental noise suppression and speech interference suppression have always been hot issues. A general noise reduction method, one way is to estimate the background noise by using the signal collected within a period of time according to the difference in spectral characteristics between the background noise signal and the voice music signal, and then perform environmental noise suppression according to the estimated background noise characteristics , this method works well for stationary noise, but completely fails for speech interference. In addition to using the difference in spectral characteristics between the background noise signal and the speech music signal, another way also uses the difference in the correlation between different channels, such as multi-channel noise suppression or microphone array beamforming technology. The direction of voice interference can be suppressed to a certain extent, but the tracking effect of the direction change of the interference source often cannot meet the demand, and the voice enhancement of the specific target person cannot be realized.

At present, the realization of voice enhancement and interference suppression functions is mainly realized through traditional or artificial intelligence (AI)-based general noise reduction, separation and other algorithms. Under certain conditions, it is difficult to achieve the effect of highlighting the target voice and suppressing the interference voice, and the experience is poor.

Contents of the invention

The embodiment of the present application provides a speech enhancement method and related equipment. By adopting the embodiment of the present application, all interference noises except the target user's voice can be suppressed and the target user's voice can be highlighted in various environmental noise and speech interference scenarios. , which improves the user experience of voice calls and voice interactions.

In the first aspect, the embodiment of the present application provides a method for speech enhancement, including: after the terminal device enters into a person-specific noise reduction (personalized noise reduction, PNR) mode, acquiring a noisy speech signal and target speech related data, wherein the noisy The speech signal contains the interference noise signal and the speech signal of the target user; the target speech related data is used to indicate the speech characteristics of the target user; according to the target speech related data, the first noisy speech signal is denoised through the trained speech noise reduction model processing to obtain the noise-reduced speech signal of the target user; wherein, the speech noise-reduction model is realized based on a neural network.

Wherein, the interfering noise signal includes a voice signal of a non-target user, an environmental noise signal (such as a car whistle, a sound emitted by a machine during operation) and the like.

Optionally, the target voice-related data can be the registered voice signal of the target user, can be a voice pickup (voice pick up, VPU) signal of the target user, can also be the voiceprint feature of the target user or the video lip movement information of the target user Wait.

The voice noise reduction model is guided by the target voice-related data to extract the target user's voice signal from the noisy voice signal, suppressing all interference noises except the target user's voice, highlighting the target user's voice, and improving the user's voice communication and The experience of voice interaction and so on.

In a feasible embodiment, the method of the present application also includes:

Acquiring the speech enhancement coefficient of the target user; based on the speech enhancement coefficient of the target user, the noise reduction speech signal of the target user is enhanced to obtain the enhanced speech signal of the target user, wherein the amplitude of the enhanced speech signal of the target user is the same as that of the target user The ratio of the amplitude of the noise-reduced speech signal is the speech enhancement coefficient of the target user.

By introducing the target user's voice enhancement coefficient, the target user's voice signal can be further enhanced, so as to further highlight the target user's voice and suppress the non-target user's voice, and improve the user's experience in voice calls and voice interactions.

Further, the interference noise signal is also obtained through the noise reduction processing, and the method of the present application also includes:

Obtain the interference noise suppression coefficient; suppress the interference noise signal based on the interference noise suppression coefficient to obtain the interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient; The noise-suppressed signal is fused with the target user's enhanced speech signal to obtain an output signal.

Optionally, the value range of the interference noise suppression coefficient is (0,1).

By introducing the interference noise suppression coefficient, the voice of the non-target user is further suppressed, and the voice of the target user is highlighted indirectly.

In a feasible embodiment, the interference noise signal is also obtained through noise reduction processing, and the method of the present application also includes:

Obtain the interference noise suppression coefficient; suppress the interference noise signal based on the interference noise suppression coefficient to obtain the interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient; The noise suppression signal is fused with the target user's noise reduction speech signal to obtain an output signal.

In practical applications, only the voice of the target user appears in the ear without noise, which will make the user very uncomfortable. Therefore, by introducing the interference noise suppression coefficient and the interference noise signal, the interference noise signal can be suppressed by introducing the interference noise suppression coefficient. At the same time, noise signals can be heard during a call, which improves user experience.

In a feasible embodiment, the target users include M, the target voice-related data includes the voice-related data of the M target users, the noise-reduced voice signals of the target users include the noise-reduced voice signals of the M target users, and the target user's voice The enhancement coefficient includes speech enhancement coefficients of M target users, and M is an integer greater than 1,

Carry out denoising processing on the first noisy voice signal through the voice denoising model according to the relevant data of the target voice, and obtain the denoising voice signal of the target user, including:

For any target user A in the M target users, according to the voice-related data of the target user A, the first noisy speech signal is denoised by the speech noise reduction model, so as to obtain the noise-reduced speech signal of the target user A; for M Each target user in the target users is all processed according to this method, and the noise-reduced voice signals of M target users can be obtained;

Based on the speech enhancement coefficient of the target user, the noise reduction speech signal of the target user is enhanced to obtain the enhanced speech signal of the target user, including:

Process the noise-reduced speech signal of target user A based on the speech enhancement coefficient of target user A to obtain the enhanced speech signal of target user A; The ratio of is the speech enhancement coefficient of the target user A; according to this method, the noise-reduced speech signal of each of the M target users is processed, and the enhanced speech signals of the M target users can be obtained.

The method of the present application further includes: obtaining an output signal based on the enhanced voice signals of the M target users.

The voice signals of multiple target users can be enhanced by using the above parallel method, and for multiple target users, the enhanced voice signals of the target users can be further adjusted by setting the voice enhancement coefficient, thus solving the problem of voice degradation in the case of multiple people. noise problem.

In a feasible embodiment, the target users include M, and the target speech-related data includes the speech-related data of M target users, and the noise-reduced speech signals of the target users include the noise-reduced speech signals of M target users, and M is greater than 1 an integer of

Carry out denoising processing on the first noisy voice signal through the voice denoising model according to the relevant data of the target voice, so as to obtain the denoising voice signal and the interference noise signal of the target user, including:

According to the voice-related data of the first target user among the M target users, the first noisy voice signal is denoised through the voice noise reduction model, and the noise-reduced voice signal of the first target user and the first target user are obtained. The first noisy speech signal of the user's speech signal; the first noisy speech signal not containing the speech signal of the first target user through the speech noise reduction model according to the speech related data of the second target user in the M target users Carry out noise reduction processing, to obtain the noise-reduced speech signal of the 2nd target user and the first band noise speech signal that does not contain the speech signal of the 1st target user and the speech signal of the 2nd target user; Repeat above-mentioned process, until According to the voice-related data of the Mth target user, the noise reduction process is performed on the first noisy voice signal that does not contain the voice signals of the 1st to M-1 target users through the voice noise reduction model, and the noise reduction of the Mth target user is obtained. Noise-reduced speech signals and interference noise signals; so far, the noise-reduced speech signals and interference noise signals of M target users are obtained.

The voice signals of multiple target users can be enhanced by adopting the above serial method, thereby solving the problem of voice noise reduction in the case of multiple people.

In a feasible embodiment, the target users include M, and the target speech-related data includes the speech-related data of M target users, and the noise-reduced speech signals of the target users include the noise-reduced speech signals of M target users, and M is greater than 1 is an integer, according to the target voice-related data, the first noisy voice signal is denoised by the voice denoising model, so as to obtain the denoised voice signal and the interference noise signal of the target user, including:

According to the voice-related data of the M target users, the first noisy voice signal is denoised through the voice denoising model, so as to obtain the denoised voice signals and interference noise signals of the M target users.

In a feasible embodiment, for the voice-related data of M target users, the relevant data of each target user includes the registered voice signal of the target user, and the registered voice signal of target user A is when the noise decibel value is lower than the preset The voice signal of the target user A collected under the environment of the value, the voice noise reduction model includes M first encoding network, second encoding network, time convolution network (time convolution network, TCN), first decoding network and M first The three-decoding network performs noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the target speech-related data, so as to obtain the target user's noise-reduced speech signal and interference noise signal, including:

Use M first coding networks to extract features of the registered speech signals of M target users respectively, and obtain the feature vectors of the registered speech signals of M target users; use the second coding network to perform feature extraction on noisy speech signals, and obtain The eigenvector of noise speech signal; Obtain the first eigenvector according to the eigenvector of the registration speech signal of M target users and the eigenvector of the first band noisy speech signal; Obtain the second eigenvector according to TCN and the first eigenvector; According to M Each of the third decoding networks in the third decoding networks obtains the noise-reduced speech signals of M target users for the second feature vector and the feature vectors output by the first encoding network corresponding to the third decoding network; according to the first The decoding network, the second eigenvector and the eigenvector of the first noisy speech signal obtain an interference noise signal.

By adopting the above method, the voice signals of multiple target users can be denoised, thereby solving the problem of voice denoising in the case of multiple people.

In a feasible embodiment, the target users include M, and the relevant data of the target users include the registration voice signals of the target users, and the registration voice signals of the target users are target users collected in an environment where the noise decibel value is lower than a preset value The speech signal, the speech noise reduction model includes the first coding network, the second coding network, TCN and the first decoding network,

Carry out denoising processing to the first noisy voice signal through the voice denoising model according to the target voice related data, and obtain the denoising voice signal of the target user, including:

Utilize the first encoding network and the second encoding network to carry out feature extraction respectively to the registered voice signal of the target user and the first noisy voice signal, obtain the feature vector of the registered voice signal of the target user and the feature vector of the first noisy voice signal; Obtain the first eigenvector according to the eigenvector of the registered speech signal of the target user and the eigenvector of the noisy speech signal; obtain the second eigenvector according to the TCN and the first eigenvector; obtain the target user according to the first decoding network and the second eigenvector noise-reduced speech signal.

Further, the method of the present application also includes:

An interfering noise signal is also obtained from the first decoding network and the second eigenvector.

In a feasible embodiment, the relevant data of the target user A includes the registered voice signal of the target user A, and the registered voice signal of the target user A is the voice of the target user A collected in an environment where the noise decibel value is lower than a preset value signal, the speech noise reduction model includes a first coding network, a second coding network, a TCN and a first decoding network, and performs noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech related data of the target user A, so as to Obtain the noise-reduced voice signal of the target user A, including:

Use the first encoding network and the second encoding network to extract the features of the registration speech signal of the target user A and the first noisy speech signal, so as to obtain the feature vector of the registration speech signal of the target user A and the first noise speech signal. Feature vector; According to the feature vector of the registered voice signal of target user A and the feature vector of the first noisy voice signal, the first feature vector is obtained; according to the TCN and the first feature vector, the second feature vector is obtained; according to the first decoding network and the first feature vector The second eigenvector obtains the noise-reduced speech signal of the target user A.

In a feasible embodiment, the relevant data of the i-th target user among the M target users includes the registration voice signal of the i-th target user, i is an integer greater than 0 and less than or equal to M, and the voice noise reduction model includes an encoding network, a second encoding network, a TCN and a first decoding network,

Utilize the first coding network and the second coding network to carry out feature extraction respectively to the registration speech signal of target user and the first noise signal, obtain the feature vector of the registration speech signal of the ith target user and the feature vector of this first noise signal; Wherein, the first noise signal is the first noisy speech signal that does not contain the speech signals of the 1st to i-1 target users; according to the feature vector of the registration speech signal of the i target user and the feature vector of the first noise signal Obtain the first eigenvector; Obtain the second eigenvector according to the TCN and the first eigenvector; Obtain the noise-reduced voice signal and the second noise signal of the i-th target user according to the first decoding network and the second eigenvector, wherein, the second The noise signal is a first noisy speech signal that does not include the speech signals of the 1st to i-th target users.

By registering the voice signal of the target user in advance, in the subsequent voice interaction, the voice signal of the target user can be enhanced, interference voice and noise can be suppressed, and only the voice signal of the target user can be input during voice wake-up and voice interaction to improve voice The effect and accuracy of wake-up and speech recognition; and construct a speech noise reduction model based on the TCN causal hole convolution network, and realize the low-latency output speech signal of the speech noise reduction model.

In a feasible embodiment, the relevant data of the target user includes the VPU signal of the target user, and the speech noise reduction model includes a preprocessing module, a third coding network, a gated recurrent unit (gated recurrent unit, GRU), a second decoding network And the post-processing module, according to the relevant data of the target voice, the noise reduction processing is carried out to the first noisy voice signal by the voice noise reduction model, so as to obtain the noise-reduced voice signal of the target user, including:

Carry out time-frequency transformation to the VPU signal of the first band noise speech signal and the target user respectively by the preprocessing module, to obtain the first frequency domain signal of the first band noise speech signal and the second frequency domain signal of the VPU signal; The frequency domain signal is fused with the second frequency domain signal to obtain a first fused frequency domain signal; the first fused frequency domain signal is successively processed by a third encoding network, a GRU, and a second decoding network to obtain a voice signal of a target user The mask of the third frequency domain signal; the first frequency domain signal is post-processed by the post-processing module according to the mask of the third frequency domain signal to obtain the third frequency domain signal; the third frequency domain signal is frequency-time Transform to obtain the noise-reduced voice signal of the target user; wherein, the third encoding module and the second decoding module are all implemented based on the convolutional layer and the frequency transformation block (frequency transformation block, FTB).

Among them, the post-processing includes mathematical operations, such as dot multiplication and so on.

Further, the mask of the first frequency domain signal is obtained by processing the first fused frequency domain signal successively through the third coding network, the GRU and the second decoding network; the mask of the first frequency domain signal is processed by the post-processing module A frequency-domain signal is post-processed to obtain a fourth frequency-domain signal of the interference noise signal; frequency-time transformation is performed on the fourth frequency-domain signal to obtain the interference noise signal.

Optionally, since the first noisy speech signal includes the speech signal of the target user and the interference noise signal, after obtaining the noise-reduced speech signal of the target user, the first noisy speech signal is processed according to the noise-reduced speech signal of the target user. processing to obtain an interference noise signal, that is, subtract the noise-reduced speech signal of the target user from the first noisy speech signal to obtain an interference noise signal.

In a feasible embodiment, the relevant data of the target user A includes the VPU signal of the target user A, and the speech noise reduction model includes a preprocessing module, a third encoding network, a GRU, a second decoding network and a postprocessing module, according to the target user A The voice-related data of A uses the voice noise reduction model to perform noise reduction processing on the first noisy voice signal to obtain the noise-reduced voice signal of the target user A, including:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user A through the preprocessing module to obtain the first frequency domain signal of the first noisy speech signal and the ninth frequency domain of the VPU signal of the target user A signal; the first frequency domain signal and the ninth frequency domain signal are fused to obtain the second fused frequency domain signal; the second fused frequency domain signal is successively processed by the third encoding network, the GRU and the second decoding network to obtain the target The mask of the tenth frequency domain signal of the voice signal of user A; The first frequency domain signal is post-processed by the mask of the tenth frequency domain signal by the post-processing module to obtain the tenth frequency domain signal; for the tenth frequency domain The signal is subjected to frequency-time transformation to obtain the noise-reduced speech signal of the target user A; wherein, the third encoding module and the second decoding module are both implemented based on the convolutional layer and FTB.

In a feasible embodiment, the relevant data of the i-th target user among the M target users includes the VPU signal of the i-th target user, where i is an integer greater than 0 and less than or equal to M,

Both the first noise signal and the VPU signal of the i-th target user are time-frequency transformed by the preprocessing module to obtain the eleventh frequency domain signal of the first noise signal and the tenth frequency domain signal of the i-th target user's VPU signal. Two frequency domain signals; the eleventh frequency domain signal and the twelfth frequency domain signal are fused to obtain the third fusion frequency domain signal; wherein the first noise signal is the voice that does not contain the 1st to i-1 target users The first noisy speech signal of the signal; the mask sum of the thirteenth frequency domain signal of the speech signal of the i-th target user is obtained by processing the third fusion frequency domain signal successively through the third encoding network, the GRU and the second decoding network The mask of the eleventh frequency domain signal; the mask of the eleventh frequency domain signal is post-processed by the post-processing module according to the mask of the thirteenth frequency domain signal and the mask of the eleventh frequency domain signal, to obtain the thirteenth frequency domain signal Domain signal and the 14th frequency domain signal of the second noise signal; Carry out frequency-time transformation to the 13th frequency domain signal and the 14th frequency domain signal, obtain the noise reduction voice signal and the second noise signal of the i-th target user , the second noise signal is the first noisy speech signal that does not contain the speech signals of the 1st to i-th target users; wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.

By using the VPU signal of the target user as auxiliary information, it is used to extract the voice features of the target user in real time, and this feature is fused with the noisy voice signal collected by the microphone to guide the voice enhancement of the target user and the suppression of interference such as the voice of the non-target user, and This embodiment also proposes a new speech noise reduction model based on FTB and GRU for the suppression of interference such as speech enhancement of target users and speech of non-target users; The user registers the voice feature information in advance, and can use the real-time VPU signal as auxiliary information to obtain the enhanced voice of the target user and suppress the interference of non-target voice.

In a feasible embodiment, based on the speech enhancement coefficient of the target user, the noise reduction speech signal of the target user is enhanced to obtain the enhanced speech signal of the target user, including:

For any target user in the M target users, the noise-reduced speech signal of target user A is enhanced based on the speech enhancement coefficient of target user A to obtain the enhanced speech signal of target user A; the enhanced speech signal of target user A The ratio of the amplitude of the amplitude and the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

The interference noise suppression signal is fused with the target user's enhanced speech signal to obtain an output signal, including:

The enhanced speech signals of the M target users are fused with the interference noise suppression signal to obtain an output signal.

For the noise-reduced speech signals of multiple target users, by introducing the speech enhancement coefficients of the multiple target users, the magnitudes of the enhanced speech signals of the multiple target users can be adjusted as required.

In a feasible embodiment, the relevant data of the target user includes the target user's VPU signal, and the method of the present application further includes: acquiring the target user's in-ear sound signal;

Carry out denoising processing on the first noisy voice signal through the voice denoising model according to the relevant data of the target voice, so as to obtain the denoising voice signal of the target user, including:

Carrying out time-frequency transformation to the first noisy speech signal and the in-ear sound signal respectively to obtain the first frequency-domain signal of the first noisy speech signal and the fifth frequency-domain signal of the in-ear sound signal; according to the VPU signal of the target user , the first frequency domain signal and the fifth frequency domain signal obtain the covariance matrix of the first noisy speech signal and the in-ear sound signal; obtain the first minimum variance distortionless response (minimum variance distortionless response, MVDR) weight based on the covariance matrix ; Obtain the sixth frequency domain signal of the first noisy speech signal and the seventh frequency domain signal of the ear sound signal of the target user based on the first MVDR weight, the first frequency domain signal and the fifth frequency domain signal; according to the sixth frequency domain domain signal and the seventh frequency domain signal to obtain the eighth frequency domain signal of the noise-reduced speech signal of the target user; and perform frequency-time transformation on the eighth frequency-domain signal to obtain the noise-reduced speech signal of the target user.

Further, an interference noise signal is obtained according to the noise-reduced speech signal of the target user and the first noisy speech signal.

In a feasible embodiment, the relevant data of the target user A includes the VPU signal of the target user A, and the method of the present application further includes: acquiring the in-ear sound signal of the target user A;

According to the voice-related data of the target user A, the first noisy voice signal is denoised through the voice noise reduction model to obtain the denoised voice signal of the target user A, including:

Perform time-frequency transformation on the first noisy speech signal and the target user A's in-ear sound signal respectively to obtain the first frequency domain signal of the first noisy speech signal and the fifteenth frequency domain of the target user A's in-ear sound signal signal; according to the VPU signal of target user A, the first frequency domain signal and the 15th frequency domain signal, obtain the covariance matrix of the first noisy speech signal and the in-ear sound signal of target user A; obtain the 2nd based on this covariance matrix Two MVDR weights; based on the second MVDR weight, the first frequency domain signal and the fifteenth frequency domain signal, the sixteenth frequency domain signal of the first noisy speech signal and the seventeenth frequency of the ear sound signal of the target user A are obtained domain signal; obtain the eighteenth frequency domain signal of the noise reduction speech signal of target user A according to the sixteenth frequency domain signal and the seventeenth frequency domain signal; carry out frequency-time transformation to the eighteenth frequency domain signal, to obtain target user A noise-reduced speech signal.

With this method, it is not necessary for the target user to register its voice feature information in advance, and the enhanced voice signal of the target user or target user A can be obtained according to the real-time VPU signal as auxiliary information, and interference such as voice of non-target users can be suppressed.

In a feasible embodiment, the method of the present application also includes:

Obtain the first noise segment and the second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are temporally continuous noise segments; obtain the signal-to-noise ratio (SNR) of the first noise segment ) and sound pressure level (sound pressure level, SPL); if the SNR of the first noise segment is greater than the first threshold and the SPL of the first noise segment is greater than the second threshold, then extract the first temporary feature vector of the first noise segment; based on The first temporary speech feature vector performs noise reduction processing on the second noise segment to obtain the second noise reduction noise segment; based on the second noise reduction noise segment and the second noise segment, the damage assessment is performed to obtain the first damage score; if the second - If the damage score is not greater than the third threshold, enter the PNR mode;

Obtaining the first noisy speech signal includes:

A first noisy speech signal is determined from the noise signal generated after the first noise segment; the feature vector of the registered speech signal includes a first temporary feature vector.

Further, if the first damage score is not greater than the third threshold, the method of the present application also includes:

The first prompt message is sent by the terminal device, and the first prompt message is used to prompt whether the terminal device enters the PNR mode; the PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.

Through this method, it can be judged whether it is necessary to adopt the solution of the present application for voice noise reduction, which avoids the situation that noise reduction is not performed when noise reduction is required, realizes flexible automatic noise reduction, and improves user experience.

In a feasible embodiment, the relevant data of the target user includes the microphone array signal of the auxiliary device, and the method of the present application further includes:

Acquiring the first noise segment and the second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time; acquiring the microphone array of the auxiliary device of the terminal device The signal collected by the environment is calculated by using the collected signal to obtain the signal angle of arrival (direction of arrival, DOA) and sound pressure level (sound pressure level, SPL) of the first noise segment; if the DOA of the first noise segment is greater than the ninth threshold and less than The tenth threshold, and the SPL of the first noise segment is greater than the eleventh threshold, then extract the second temporary feature vector of the first noise segment; based on the second temporary speech feature vector, the second noise segment is denoised to obtain the first Four denoising noise segments; performing damage assessment based on the fourth denoising noise segment and the second noise segment to obtain a fourth damage score; if the fourth damage score is not greater than the twelfth threshold, enter the PNR mode.

Obtaining the first noisy speech signal includes:

A first noisy speech signal is determined from a noise signal generated after the first noise segment; the feature vector of the registered speech signal includes a second temporary feature vector.

Wherein, the DOA and SPL of the first noise segment are calculated by using the collected signal, which may specifically include:

Time-frequency transform is performed on the signal collected by the microphone array to obtain a nineteenth frequency domain signal, and based on the nineteenth frequency domain signal, DOA and SPL of the first noise segment are calculated.

Further, if the fourth damage score is not greater than the twelfth threshold, the method of the present application also includes:

The fourth prompt message is sent by the terminal device, and the fourth prompt message is used to prompt whether the terminal device enters the PNR mode; the PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.

Optionally, the auxiliary device may be a device with a microphone array, such as a computer, a tablet computer, and the like.

In a feasible embodiment, the method of the present application also includes:

When it is detected that the terminal device is used again, obtain the second noisy voice signal; and use the traditional noise reduction algorithm, that is, use the non-PNR mode to perform noise reduction processing on the second noisy voice signal, and obtain the noise reduction of the current caller voice signal

When the SNR of the second noisy speech signal is lower than the fourth threshold, the second noisy speech signal is subjected to noise reduction processing according to the first temporary feature vector to obtain the current user's noise-reduced speech signal; based on the current user's The noise-reduced speech signal and the second noisy speech signal are subjected to damage assessment to obtain a second damage score; when the second damage score is not greater than the fifth threshold, a second prompt message is sent through the terminal device, and the second prompt message is used to Prompting the current user that the terminal device can enter the PNR mode; after detecting the operation command agreed to enter the PNR mode, the terminal device is made to enter the PNR mode to perform noise reduction processing on the third noisy voice signal. The third noisy voice signal is Acquired after the second noisy speech signal; after detecting the current user's operation instruction of not agreeing to enter the PNR mode, use the non-PNR mode to perform noise reduction processing on the third noisy speech signal.

What needs to be explained here is that after performing temporary speech feature extraction on the first noise segment to obtain the temporary feature vector of the first noise segment, the terminal device stores the temporary feature vector, and obtains the temporary feature vector directly when it needs to be used later, avoiding This prevents the subsequent inability to obtain the voice features of the current user in a noisy scene, thus making it impossible to perform damage assessment. Here, the temporary feature vector of the first noise segment may be the first temporary feature vector or the second temporary feature vector.

Optionally, the fourth threshold may or may not be the same as the first threshold; the fifth threshold may or may not be the same as the third threshold.

In a feasible embodiment, the method of the present application also includes:

If the SNR of the first noise segment is not greater than the first threshold or the SPL of the first noise segment is not greater than the second threshold, and the terminal device has stored the reference temporary voiceprint feature vector, obtain the third noise segment; according to the reference temporary voiceprint feature vector Perform denoising processing on the third noise segment to obtain a third denoising noise segment; perform damage assessment based on the third noise segment and the third denoising noise segment to obtain a third damage score; if the third damage score is greater than the sixth threshold And the SNR of the third noise segment is less than the seventh threshold, or the third damage score is greater than the eighth threshold and the SNR of the third noise segment is not less than the seventh threshold, then the third prompt information is sent through the terminal device, and the third prompt information is used to Prompt the current user that the terminal device can enter the PNR mode; after detecting the current user's operation instruction agreeing to enter the PNR mode, the terminal device enters the PNR mode to perform noise reduction processing on the fourth noisy voice signal; After the operator does not agree to enter the operation instruction of PNR mode, adopt the non-PNR mode to carry out noise reduction processing on the fourth noisy speech signal; wherein, the fourth noisy speech signal is determined from the noise signal generated after the third noise segment of.

Wherein, the reference temporary voiceprint feature vector is the voiceprint feature vector of the historical user.

Optionally, the seventh threshold may be 10dB or other values, the sixth threshold may be 8dB or other values, and the eighth threshold may be 12dB or other values.

In a feasible embodiment, the method of the present application also includes:

When it is detected that the terminal device is in the handheld talking state, it will not enter the PNR mode;

When it is detected that the terminal device is in the hands-free call state, enter the PNR mode, wherein the target user is the owner of the terminal device or the user who is using the terminal device;

When it is detected that the terminal device is in a video call state, enter the PNR mode, wherein the target user is the owner of the terminal device or the user closest to the terminal device;

When it is detected that the terminal device is connected to the headset for a call, enter the PNR mode, wherein the target user is a user wearing the headset; the first noisy voice signal and the target voice-related data are collected through the headset; or,

When it is detected that the terminal device is connected to a smart large-screen device, a smart watch or a vehicle-mounted device, it enters the PNR mode, where the target user is the owner of the terminal device or the user who is using the terminal device, and the first noisy voice signal is related to the target voice The data is collected by the audio collection hardware of smart large-screen devices, smart watches or vehicle-mounted devices.

Based on different application scenarios, it is judged whether to enable the PNR noise reduction function, which realizes flexible automatic noise reduction and improves user experience.

In a feasible embodiment, the method of the present application also includes:

Obtain the decibel value of the audio signal in the current environment. If the decibel value of the audio signal in the current environment exceeds the preset decibel value, it is determined whether the PNR function corresponding to the application started by the terminal device is enabled; if it is not enabled, the PNR function started by the terminal device is enabled. The application corresponds to the PNR function and enters the PNR mode.

Wherein, the application program is an application program installed on the terminal device, such as call, video call, video recording application program, WeChat, QQ and so on.

Based on the size of the audio signal in the current environment, it is judged whether to enable the PNR function, which realizes flexible automatic noise reduction and improves user experience.

In a feasible embodiment, the terminal device includes a display screen, and the display screen includes a plurality of display areas, wherein each display area in the plurality of display areas displays a label and a corresponding function key, and the function key is used to control its corresponding label Indicates the activation and deactivation of the function or application's PNR function.

The interface displayed on the display screen of the terminal device is set to control the opening and closing of the PNR function of a certain application program (such as calling, recording, etc.) of the terminal device, so that the user can turn on and off the PNR function as required.

In a feasible embodiment, when voice data transmission is performed between the terminal device and another terminal device, the method of the present application further includes:

Receive a voice enhancement request sent by another terminal device, the voice enhancement request is used to instruct the terminal device to enable the PNR function of the call function; in response to the voice enhancement request, send a third prompt message through the terminal device, the third prompt message is used to prompt Whether to enable the terminal device to enable the PNR function of the call function; after detecting the operation instruction confirming the PNR function of the call function, enable the PNR function of the call function and enter the PNR mode; send a voice enhancement response message to another terminal device, the The voice enhancement response message is used to indicate that the terminal device has enabled the PNR function of the call function.

During the call, when the other party is found to be in a noisy environment, a request for enabling the PNR function of the call function of the other party's terminal equipment is sent to the other party, which improves the quality of the call between the two parties. Of course, this embodiment can also be applied to video calls and the like.

In a feasible embodiment, when the terminal device starts the video call or video recording function, the display interface of the terminal device includes a first area and a second area, the first area is used to display the content of the video call or video recording, and the second area The area is used to display M controls and corresponding M labels. The M controls correspond to the M target users one by one. Each of the M controls includes a sliding button and a sliding bar. By controlling the sliding button to slide on the sliding bar, to adjust the speech enhancement coefficient of the target user indicated by the label corresponding to the control.

By adjusting the size of the speech enhancement coefficient according to the user's need, the user can adjust the intensity of noise reduction according to his need. Of course, the interference noise suppression coefficient can also be adjusted in this way.

In a feasible embodiment, when the terminal device starts the video call or video recording function, the display interface of the terminal device includes a first area, and the first area is used to display the video call content or the video recording content;

When an operation on any object in the video call content or video recording content is detected, the control corresponding to the object is displayed in the first area, the control includes a sliding button and a sliding bar, and the sliding button is controlled to slide on the sliding bar to Adjusts the speech enhancement factor for this object.

In a feasible embodiment, when the terminal device is an intelligent interactive device, the target voice-related data includes a voice signal including a wake-up word, and the first noisy voice signal includes an audio signal including a command word.

Optionally, the smart interactive devices include devices such as smart speakers, sweeping robots, smart refrigerators, and smart air conditioners.

This method is used to perform noise reduction processing on the instruction voice for controlling the intelligent interactive device, so that the intelligent interactive device can quickly obtain accurate instructions, and then complete the actions corresponding to the instructions.

In a second aspect, an embodiment of the present application provides a terminal device, where the terminal device includes a unit or a module configured to execute the method in the first aspect.

In a third aspect, an embodiment of the present application provides a terminal device, including a processor and a memory, wherein the processor is connected to the memory, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method of the first aspect part or all of.

In the fourth aspect, the embodiment of the present application provides a chip system, which is applied to electronic equipment; the chip system includes one or more interface circuits, and one or more processors; the interface circuits and processors are interconnected through lines; the interface The circuit is used to receive a signal from the memory of the electronic device and send a signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in the first aspect.

In a fifth aspect, an embodiment of the present application provides a computer storage medium, where the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the first aspect.

In the sixth aspect, the embodiment of the present application further provides a computer program product, including computer instructions, which, when the computer instructions are run on the terminal device, enable the terminal device to implement part of the method described in the first aspect or all.

These or other aspects of the present application will be more concise and understandable in the description of the following embodiments.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 2a is a schematic diagram of a speech noise reduction processing principle provided by an embodiment of the present application;

FIG. 2b is a schematic diagram of another speech noise reduction processing principle provided by the embodiment of the present application;

FIG. 3 is a schematic flowchart of a speech enhancement method provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of a speech noise reduction model provided in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a speech noise reduction model provided in an embodiment of the present application;

Figure 6a shows the framework structure of the TCN model;

Figure 6b illustrates the structure of the causal dilated convolutional layer unit;

FIG. 7 is a schematic structural diagram of another speech noise reduction model provided by the embodiment of the present application;

Fig. 8 is the specific structure diagram of neural network in Fig. 7;

FIG. 9 is a schematic diagram of a speech noise reduction process provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of another speech noise reduction process provided by the embodiment of the present application;

FIG. 11 is a schematic diagram of a multi-person speech noise reduction process provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of a multi-person speech noise reduction process provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of a multi-person speech noise reduction process provided by an embodiment of the present application;

FIG. 14 is a schematic structural diagram of another speech noise reduction model provided by the embodiment of the present application;

FIG. 15 is a schematic diagram of a UI interface provided by the embodiment of the present application;

FIG. 16 is a schematic diagram of another UI interface provided by the embodiment of the present application;

FIG. 17 is a schematic diagram of another UI interface provided by the embodiment of the present application;

FIG. 18 is a schematic diagram of another UI interface provided by the embodiment of the present application;

FIG. 19 is a schematic diagram of a UI interface in a call scenario provided by an embodiment of the present application;

FIG. 20 is a schematic diagram of a UI interface in another call scenario provided by the embodiment of the present application;

FIG. 21 is a schematic diagram of a video recording UI interface provided by an embodiment of the present application;

FIG. 22 is a schematic diagram of a video call UI interface provided by an embodiment of the present application;

FIG. 23 is a schematic diagram of another video call UI interface provided by the embodiment of the present application;

FIG. 24 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

FIG. 25 is a schematic structural diagram of another terminal device provided by an embodiment of the present application;

FIG. 26 is a schematic structural diagram of another terminal device provided by an embodiment of the present application.

Detailed ways

Each will be described in detail below.

The terms "first", "second", "third" and "fourth" in the description and claims of this application and the drawings are used to distinguish different target users, rather than to describe specific order. Furthermore, the terms "include" and "have", as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally further includes For other steps or units inherent in these processes, methods, products or apparatuses.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

"Multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.

Embodiments of the present application are described below in conjunction with the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. The application scenario includes an audio collection device 102 and a terminal device 101. The terminal device can be a smart phone, a smart watch, a TV, a smart vehicle/vehicle terminal, a headset, a PC, a tablet, a notebook computer, a smart speaker, a robot, a recording collection device, etc. On the terminal equipment that needs to collect sound signals, such as for mobile phone voice enhancement, process the noisy voice signal collected by the microphone, output the noise-reduced voice signal of the target user, as the uplink signal of the voice call, or voice wake-up and voice Input signal to the recognition engine.

Of course, the collected sound signal can also be collected by an audio collection device 102 connected to the terminal device in a wired or wireless manner. Computer or recording collection equipment, etc.

Optionally, the audio collection device 102 and the terminal device 101 are integrated together.

Fig. 2a and Fig. 2b schematically illustrate the principle of speech noise reduction processing. As shown in Figure 2a, after collecting the noisy speech signal obtained by mixing the speech of the target user, the speech of the interfering person and other noises, the noisy speech signal and the registered speech of the target user are input into the speech noise reduction model for processing to obtain the noise-reduced speech signal of the target user, or as shown in Figure 2b, input the noisy speech signal and the VPU signal of the target user into the speech noise reduction model for processing to obtain the noise-reduced speech signal of the target user.

The enhanced voice signal can be used for voice calls or voice wake-up and voice recognition functions. For private devices (such as mobile phones, PCs and various personal wearable products, etc.), the target user is fixed, and only the voice information of the target user is kept as the registered voice or VPU signal during the call and voice interaction, and then the voice enhancement is performed in the above-mentioned manner , which can greatly improve the user experience. In limited public equipment (such as smart home, car, conference room, etc.), users are relatively fixed, and voice enhancement can be performed through multi-user voice registration (as shown in Figure 2a), which can improve the experience of multi-user scenarios.

Referring to FIG. 3 , FIG. 3 is a schematic flowchart of a speech enhancement method provided by an embodiment of the present application. As shown in Figure 3, the method includes:

S301. After the terminal device enters the PNR mode, acquire the first noisy voice signal and target voice-related data, wherein the first noisy voice signal includes an interference noise signal and the voice signal of the target user, and the target voice-related data is used to indicate the target The user's speech characteristics.

Optionally, the target voice-related data may be the target user's registered voice signal, or the target user's VPU signal, or the target user's voiceprint features, or the target user's video lip movement information.

In an example, the voice signal of the target user with a preset duration collected by the microphone in a quiet scene is the registered voice signal of the target user; wherein, the sampling frequency of the microphone can be 16000 Hz, assuming that the preset duration is 6s , the registered voice signal of the target user includes 96000 sampling points. Wherein, the quiet scene specifically means that the sound level of the scene is not higher than a preset decibel; optionally, the preset decibel may be 1dB, 2dB, 5dB, 10dB or other values.

In another example, the target user's VPU signal is acquired through a device with a bone voiceprint sensor, and the VPU sensor in the bone voiceprint sensor can pick up the target user's voice signal through bone conduction. Compared with the signal collected by the microphone, the difference of the VPU signal is that it only picks up the voice of the target user and can only pick up low-frequency signals (generally below 4kHz).

Wherein, the first noisy speech signal includes the target user's speech signal and other noise signals, and the other noise signals include other user's speech signals and/or noise signals generated by non-human beings, such as noise signals generated by automobiles and construction site machines.

S302. Perform noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech related data to obtain a noise reduction speech signal of the target user, wherein the speech noise reduction model is implemented based on a neural network.

For different target speech related data, the speech noise reduction model has different network structures, that is to say, the speech noise reduction model adopts different processing methods for different target speech related data. For the target voice related data is the registered voice of the target user or the video lip movement information of the target user, the voice noise reduction model corresponding to the method can be used to carry out noise reduction processing on the target voice related data and the first noisy voice signal; for the target voice The relevant data includes the VPU signal of the target user, and the voice noise reduction model corresponding to mode 2 or mode 3 may be used to perform noise reduction processing on the target voice related data and the first noisy voice signal. The following specifically describes the processing procedures of the manner 1, the manner 2 and the manner 3.

The first method is specifically described by taking the target voice-related data as the registered voice signal of the target user as an example.

Mode 1: As shown in Figure 4, according to the relevant data of the target voice, the noise reduction processing is performed on the first noisy voice signal through the voice noise reduction model to obtain the noise-reduced voice signal of the target user, which specifically includes the following steps:

Utilize the first encoding network to extract the feature vector of the registered voice signal from the registered voice signal of the target user; utilize the second encoding network to extract the feature vector of the noisy voice signal from the noisy voice signal; according to the feature vector of the registered voice signal and the eigenvector of the noisy speech signal to obtain the first eigenvector, specifically, perform a mathematical operation on the eigenvector of the speech signal and the eigenvector of the noisy speech signal, such as dot multiplication, to obtain the first eigenvector; utilize TCN to The first eigenvector is processed to obtain the second eigenvector, and then the first decoding network is used to process the second eigenvector to obtain the noise-reduced speech signal of the target user. It can be known from the above description that in the first manner, the speech noise reduction model includes a first encoding network, a second encoding network, a TCN, and a first decoding network.

Specifically, as shown in a in Figure 5, the first encoding network includes a convolutional layer, layer normalization (256), an activation function PReLU (256) and an averaging layer, and the size of the convolution kernel of the convolutional layer can be is 1*1; the registered voice with 96000 sampling points is input with 40 sampling points as a frame, and the feature matrix with a size of 4800*256 is obtained through the convolutional layer, layer normalization and activation function PReLU, in which, and two adjacent The frontal overlap rate of the sampling points of the frame can be 50%, and the overlap rate can of course be other values; then the feature matrix is averaged in the time dimension by the averaging layer to obtain the feature vector of the registered speech signal with a size of 1*256. When the first noisy speech signal is collected by the microphone, 20 sampling points are taken as a frame, and input to the second encoding network frame by frame for feature extraction, and the speech feature vector of each frame is obtained. Among them, as shown in b in Figure 5, the second encoding network includes a convolutional layer, layer normalization and activation function; specifically, the noisy speech is passed through the convolutional layer, layer Normalize and activate the function to obtain the speech feature vector of each frame; perform mathematical operations, such as dot multiplication, on the target speech feature vector and the speech feature vector of each frame in the first noisy speech, so as to obtain the first feature vector. Optionally, the above mathematical operations may be dot multiplication or other operations. The TCN model uses a causal atrous convolution model. Figure 6a shows the framework structure of the TCN model. As shown in Figure 6a, the TCN model includes M blocks, and each block consists of N causal atrous convolutional layer units. Figure 6b shows the structure of the causal dilated convolutional layer unit, and the corresponding convolution expansion rate of the nth layer is 2 ^n-1 . In this embodiment, the TCN model includes 5 blocks, and each block includes 4 layers of causal atrous convolutional layer units, so the expansion rates corresponding to

layers

1, 2, 3, and 4 in each block are 1, 2, and 4 respectively. ,8, the convolution kernel is 3x1. The first eigenvector is passed through the TCN model to obtain the second eigenvector, and the dimension of the second eigenvector is 1x256. As shown in c in Figure 5, the first decoding network includes an activation function PReLU (256) and a deconvolution layer (256x20x2); the second feature vector passes through the activation function and the deconvolution layer to obtain the voice signal of the target user. For the structure of the second encoding network, refer to the structure of the first encoding network. Compared with the first encoding network, the second encoding network lacks the function of averaging in the time dimension.

It should be noted here that 256 in the above-mentioned layer normalization (256) and the above-mentioned activation function PReLU (256) indicates the number of feature dimensions output by layer normalization and activation function, and 256x20x2 in the deconvolution layer (256x20x2) Indicates the size of the convolution kernel used by the deconvolution layer. The above description is only an illustration, not a limitation of the present application.

It should be pointed out that the target user's video lip movement information includes multiple frames of images containing the target user's lip movement information. If the target voice-related data is the target user's video lip movement information, the target user's registered voice The signal is replaced with the video lip movement information of the target user, and the feature vector of the video lip movement information of the target user is extracted through the first coding network, and then the subsequent processing is performed according to the first described method.

By registering the voice signal of the target user in advance, in the subsequent voice interaction, the voice signal of the target user can be enhanced, interference voice and noise can be suppressed, and only the voice signal of the target user can be input during voice wake-up and voice interaction to improve voice The effect and accuracy of wake-up and speech recognition; and the TCN causal hole convolution network is used to build a speech noise reduction model, which can realize the low-latency output speech signal of the speech noise reduction model.

Mode 2 and Mode 3 are specifically described by taking the target voice-related data as the VPU signal of the target user as an example.

Mode 2: As shown in Figure 7, the VPU signal of the target user and the first noisy voice signal are subjected to noise reduction processing using the voice noise reduction model to obtain the noise-reduced voice signal of the target user, which specifically includes the following steps:

Carry out time-frequency conversion to the VPU signal of the target user and the first band noise voice signal respectively by the preprocessing module, obtain the frequency domain signal of the VPU signal of the target user and the frequency domain signal of the first band noise voice signal; The frequency domain signal of the VPU signal is fused with the frequency domain signal of the first noisy speech signal to obtain the first fused frequency domain signal; the first fused frequency domain signal is processed through the third encoding network, the GRU and the second decoding network respectively , to obtain the mask of the frequency domain signal of the voice signal of the target user; the frequency domain signal of the first noisy voice signal is post-processed through the post-processing module according to the mask of the frequency domain signal of the voice signal of the target user, such as mathematical operations The dot product in is to obtain the frequency domain signal of the voice signal of the target user, and perform frequency-time transformation on the frequency domain signal of the voice signal of the target user to obtain the noise-reduced voice signal of the target user. It can be seen from the above that the speech noise reduction model of the second mode includes a preprocessing module, a third encoding network, a GRU, a second decoding network and a postprocessing module.

Specifically, fast Fourier transform (fast Fourier transform, FFT) is performed on the VPU signal of the target user and the first noisy voice signal through the preprocessing module to obtain the frequency domain signal of the VPU signal of the target user and the first noisy voice signal. The frequency domain signal of the voice signal; and through the preprocessing module, the frequency domain signal of the VPU frequency domain signal of the target user and the frequency domain signal of the noisy voice are spliced and combined in the frequency domain, or the frequency spectrum of the frequency domain signal of the VPU signal of the target user is Superimpose with the spectrum of the frequency domain signal of the first noisy speech signal, or perform a dot product operation on the frequency domain signal of the VPU signal of the target user and the frequency domain signal of the first noisy speech signal, so as to obtain the first fusion frequency domain signal . For example, the frequency domain signal of 0-1.5kHz is extracted from the frequency domain signal of the VPU signal of the target user, and the frequency domain signal of 1.5kHz-8kHz is extracted from the frequency domain signal of the first noisy speech signal. The extracted two groups of frequency domain signals are directly spliced and combined in the frequency domain to obtain the first fused frequency domain signal. At this time, the frequency range of the first fused frequency domain signal is 0-8 kHz. As shown in Figure 8; the first fused frequency domain signal is input into the third encoding network for feature extraction to obtain the feature vector of the first fused frequency domain signal; then the eigenvector of the first fused frequency domain signal is input into the GRU performing processing to obtain a third eigenvector; inputting the third eigenvector into a second decoding network for processing to obtain a mask of a frequency domain signal of the voice signal of the target user. As shown in Figure 8, both the third encoding network and the second decoding network include 2 convolutional layers and 1 FTB. Among them, the size of the convolution kernel of the convolution layer is 3x3. The mask of the frequency domain signal of the speech signal of the target user is carried out point multiplication with the frequency domain signal of the first band noise speech signal by the post-processing module, obtains the frequency domain signal of the speech signal of the target user; Then the speech signal of the target user Inverse fast Fourier transform (IFFT) is performed on the frequency domain signal to obtain the noise-reduced speech signal of the target user. The above description is only an illustration, not a limitation of the present application.

By using the VPU signal of the target user as auxiliary information, it is used to extract the voice features of the target user in real time, and this feature is fused with the first noisy voice signal collected by the microphone to guide the voice enhancement of the target user and the suppression of interference such as the voice of non-target users , and this embodiment also proposes a new speech noise reduction model based on FTB and GRU for the suppression of interference such as speech enhancement of target users and speech of non-target users; it can be seen that by adopting the scheme of this embodiment, It does not require the user to register voice feature information in advance, and the real-time VPU signal can be used as auxiliary information to obtain the enhanced voice of the target user and suppress the interference of non-target voice.

Mode 3: Time-frequency transform is performed on the first noisy speech signal and the target user's in-ear sound signal respectively to obtain the frequency domain signal of the first noisy speech signal and the frequency domain signal of the target user's sound signal; The covariance matrix of the first band noise speech signal and the ear sound signal of the target user is obtained based on the VPU signal and the frequency domain signal of the first noisy speech signal and the frequency domain signal of the target user's ear sound signal respectively; The covariance matrix of the in-ear sound signal of the noisy speech signal and the target user obtains the first MVDR weight; based on the first MVDR weight, the frequency domain signal of the first noisy speech signal and the frequency domain signal of the target user's in-ear sound signal Obtain the frequency domain signal of the first voice signal and the frequency domain signal of the second voice signal; wherein, the frequency domain signal of the first voice signal is related to the first noisy voice signal, and the frequency domain signal of the second voice signal is related to the target user's Intra-ear sound signal correlation, according to the frequency domain signal of the first voice signal and the frequency domain signal of the second voice signal, the frequency domain signal of the noise reduction voice signal of the target user is obtained; the frequency domain signal of the noise reduction voice signal of the target user is performed Frequency-time transformation to obtain the noise-reduced speech signal of the target user.

Specifically, an earphone device with a bone voiceprint sensor, the device includes a bone voiceprint sensor, an in-ear microphone and an out-of-ear microphone, and the VPU sensor in the bone voiceprint sensor can pick up the sound signal of the speaker through bone conduction; The microphone is used to pick up the sound signal in the ear; the microphone outside the ear is used to pick up the sound signal outside the ear, which is the first noisy voice signal in this application;

As shown in Figure 9, the VPU signal of the target user is processed by the voice activity detection (VAD) algorithm, and the processing result is obtained; according to the processing result, it is judged whether the target user is speaking; if it is judged that the target user is speaking, then the The first identification is set to the first value (such as 1 or true); if it is judged that the target user does not speak, the first identification is set to the second value (such as 0 or false);

When the value of the first identifier is the second value, update the covariance matrix, which specifically includes: respectively performing time-frequency transformation, such as FFT, on the first noisy speech signal and the target user's in-ear sound signal to obtain the first noisy speech signal The frequency domain signal of the signal and the frequency domain signal of the target user's in-ear sound signal; then calculate the target user's ear based on the frequency domain signal of the first noisy speech signal and the frequency domain signal of the target user's in-ear sound signal. The covariance matrix of the inner sound signal and the first noisy speech signal; wherein, the covariance matrix can be expressed as: R _n (f)=X(f)X ^H (f); wherein, X(f) is the target user The two-channel frequency-domain signal of the in-ear sound signal and the first noisy speech signal, X ^H (f) is the Hermitian transformation of X (f), or the conjugate transpose of X (f); f is a frequency point; then The MVDR weight is obtained based on the covariance matrix; among them, the MVDR weight can be expressed as:

Among them, a(f,θ _s )=[a ₁ (f,θ _s )a ₂ (f,θ _s )…a _M (f,θ _s )] ^T represents the corresponding signal azimuth θ _s at frequency f Guidance vector, f is the frequency point, θ _s is the target orientation, and the θ _s is the preset value, such as 90 degrees in the vertical direction (the earphone wearing posture is relatively fixed to the position of the mouth), M is the number of microphones, a ^H (f, θ _s ) is the Hermitian transformation of a(f,θ _s ),

is the inverse matrix of R _n (f);

Based on the first MVDR weight, the frequency domain signal of the first noisy speech signal and the frequency domain signal of the target user's in-ear sound signal, the frequency domain signal of the first speech signal and the frequency domain signal of the second speech signal are obtained; wherein, The frequency domain signal of the first voice signal is related to the first noisy voice signal, the frequency domain signal of the second voice signal is related to the ear sound signal of the target user, and the frequency domain signal of the first voice signal and the second voice signal The frequency domain signal can be expressed as: Y _n (f)=w _n (f,θ _s )X _n (f); it needs to be pointed out that w _n (f,θ _s ) contains two vectors, corresponding to the first speech The frequency domain signal of the signal and the frequency domain signal of the second voice signal; the frequency domain signal of the first noisy voice signal and the frequency domain signal of the ear sound signal of the target user are respectively multiplied by two vectors to obtain the first The frequency domain signal of the voice signal and the frequency domain signal of the second voice signal; according to the frequency domain signal of the first voice signal and the frequency domain signal of the second voice signal, the frequency domain signal of the noise reduction voice signal of the target user is obtained, specifically, Adding the frequency-domain signal of the first speech signal and the frequency-domain signal of the second speech signal frequency-by-frequency, specifically combining the first frequency point of the frequency-domain signal of the first speech signal with the frequency-domain signal of the second speech signal Add the first frequency point of the signal, add the second frequency point of the frequency domain signal of the first speech signal to the second frequency point of the frequency domain signal of the second speech signal, until the first speech signal All corresponding frequency points of the frequency domain signal of the frequency domain signal and the frequency domain signal of the second voice signal are all added to obtain the frequency domain signal of the noise reduction voice signal of the target user; IFFT is carried out to the frequency domain signal of the noise reduction voice signal of the target user to obtain noise-reduced voice signal of the target user;

When the value of the first identifier is the first value, the locked covariance matrix is not updated, that is to say, the historical covariance matrix is used for calculating the first MVDR weight.

Adopting the third method, the user does not need to register the voice feature information in advance, and the real-time VPU signal can be used as the auxiliary information to obtain the enhanced voice signal while suppressing the interference noise.

In a feasible embodiment, in order to further enhance the noise-reduced speech signal of the target user, the speech enhancement coefficient of the target user is obtained, and the noise-reduced speech signal of the target user is enhanced based on the speech enhancement coefficient of the target user to obtain the target user The enhanced speech signal of the target user, wherein the ratio of the amplitude of the enhanced speech signal of the target user to the amplitude of the noise-reduced speech signal of the target user is the speech enhancement coefficient of the target user.

Since outputting the voice signal of the user alone will reduce the user experience, an interference noise signal is added on the basis of the voice signal of the target user, thereby improving the user experience. In a feasible embodiment, for the speech noise reduction models in

ways

1 and 2, during training, the decoding network (including the first decoding network and the second decoding network) in the speech noise reduction model can not only transmit the target user The enhanced speech signal can also output the interference noise signal. For the third way, after the noise-reduced speech signal of the target user is obtained, the interference noise signal can be obtained by subtracting the noise-reduced speech signal of the target user from the first noisy speech signal.

For the second method, the second decoding network of the speech noise reduction model also outputs the mask of the frequency domain signal of the first noisy speech signal, and the post-processing module also performs the first processing according to the mask of the frequency domain signal of the first noisy speech signal. The frequency domain signal of the noisy speech signal is post-processed, such as dot multiplication, to obtain the frequency domain signal of the interference noise, and then frequency-time transform is performed on the frequency domain signal of the interference noise, such as IFFT, to obtain the interference noise signal.

Optionally, after the noise-reduced speech signal of the target user is obtained, the first noisy speech signal is processed according to the noise-reduced speech signal of the target user to obtain an interference noise signal. Specifically, the interference noise signal can be obtained by subtracting the noise-reduced speech signal of the target user from the first noisy speech signal.

Optionally, for mode one or mode two or mode three, after the interference noise signal is obtained, the interference noise signal is fused with the enhanced voice signal of the target user to obtain an output signal; the output signal is the enhanced voice signal of the target user and It is obtained by mixing the interference noise signal.

Alternatively, as shown in Figure 10, the interference noise suppression coefficient is obtained, and the interference noise signal is suppressed based on the interference noise suppression coefficient to obtain the interference noise suppression signal, wherein the amplitude of the interference noise suppression signal and the amplitude of the interference noise The ratio of is the interference noise suppression coefficient; then the interference noise suppression signal is fused with the enhanced speech signal of the target user to obtain an output signal; the output signal is obtained by mixing the enhanced speech signal of the target user and the interference noise suppression signal.

Alternatively, the interference noise suppression coefficient is acquired, and the interference noise signal is suppressed based on the interference noise suppression coefficient to obtain an interference noise suppression signal; then the interference noise suppression signal is fused with the target user's noise-reduced voice signal to obtain an output signal. The output signal is obtained by mixing the target user's noise-reduced voice signal and the interference noise-suppressed signal.

Wherein, the interference noise suppression coefficient α and the target speech enhancement coefficient β may be preset by the system, for example α=0, β=1. It can also be set by the user. For example, the user can set the interference noise suppression coefficient α and the target speech enhancement coefficient β through the UI interface of the terminal device.

In the scenarios of conferences and video calls, there are many people participating, and there may be more than one target user who needs voice enhancement; therefore, methods 4, 5 and 6 can be used for voice enhancement of multiple people.

The target users include M, the target voice-related data include the relevant data of M target users, the noise-reduced voice signals of the target users include the noise-reduced voice signals of M target users, and the voice enhancement coefficients of the target users include the voices of M target users Enhancement coefficient; the first noisy speech signal includes speech signals of M target users and interference noise signals.

Method 4: As shown in Figure 11, input the speech-related data and the first noisy speech signal of the first target user among the M target users into the speech noise reduction model for noise reduction processing, and obtain the first target user The noise-reduced speech signal of the first target user and the first noisy speech signal not containing the speech signal of the first target user; The noisy speech signal is input into the speech noise reduction model for noise reduction processing, and the noise-reduced speech signal of the second target user and the first speech signal not including the speech signal of the first target user and the speech signal of the second target user are obtained. Noisy speech signal; repeat the above steps until the speech-related data of the M target user and the first noisy speech signal that does not contain the speech of the 1st to M-1 target users are input into the speech noise reduction model for reduction Noise processing, obtain the noise-reduced speech signal of the M target user and the interference noise signal, this interference noise signal is the first band noise speech signal that does not contain the speech signal of the 1st to M target users; Based on M target users The speech enhancement coefficients are respectively enhanced to the noise-reduced speech signals of M target users to obtain the enhanced speech signals of M target users; for any target user O in the M target users, the enhanced speech signal of target user O is The ratio of the amplitude and the amplitude of the noise-reduced speech signal of the target user is the speech enhancement coefficient of the target user O; based on the interference noise suppression coefficient, the interference noise signal is suppressed to obtain the interference noise suppression signal, wherein, the interference noise suppression The ratio of the amplitude of the signal to the amplitude of the interference noise signal is the interference noise suppression coefficient; the enhanced speech signals of M target users are fused with the interference noise suppression signal to obtain an output signal. The output signal is obtained by mixing the enhanced speech signals of the M target users and the interference noise suppression signals.

For the voice noise reduction model in mode four, when the voice-related data of the M target users are registered voice signals or video lip movement information, the structure of the voice noise reduction model in mode four can be the structure described in mode one; When the voice-related data of M target users is a VPU signal, the structure of the speech noise reduction model in the fourth method may be the structure described in the second method, or the speech noise reduction model in the fourth method realizes the function described in the third method.

In an example, after the noise-reduced speech signals and interference noise signals of M target users are obtained according to the fourth manner, the noise-reduced speech signals and interference noise signals of the M target users are directly fused to obtain an output signal. The output signal is obtained by mixing noise-reduced speech signals and interference noise signals of M target users.

Mode 5: the target users include M, as shown in Figure 12, input the voice-related data and the first noisy voice signal of the first target user among the M target users into the voice noise reduction model for noise reduction processing, Obtain the noise-reduced speech signal of the first target user; input the speech-related data and the first noisy speech signal of the second target user into the speech noise reduction model for noise reduction processing, and obtain the denoised speech signal of the second target user. Noisy voice signal; repeat the steps above, until the voice-related data of the M target user and the first noisy voice signal are input into the voice noise reduction model for noise reduction processing, to obtain the noise reduction voice signal of the M target user; Based on the speech enhancement coefficients of the M target users, the noise-reduced speech signals of the M target users are respectively enhanced to obtain the enhanced speech signals of the M target users; for any target user O in the M target users, the target user The ratio of the amplitude of the enhanced speech signal of O to the amplitude of the noise-reduced speech signal of the target user O is the speech enhancement coefficient of the target user O; the enhanced speech signals of M target users are fused to obtain an output signal. The output signal is obtained by mixing the enhanced speech signals of M target users.

It should be understood that the voice-related data of the above M target users and the first noisy voice signal are input into the voice noise reduction model in parallel, so the above actions may be processed in parallel.

For the voice noise reduction model in mode five, when the voice-related data of M target users are registered voice signals or video lip movement information, the structure of the voice noise reduction model in mode five can be the structure described in mode one; When the voice-related data of M target users is a VPU signal, the structure of the voice noise reduction model in the fifth way can be the structure described in the second way, or the voice noise reduction model in the fifth way can realize the function described in the third way.

In an example, after the enhanced voice signals of the M target users are obtained through the voice noise reduction model, the enhanced voice signals of the M target users may be directly fused to obtain the above output signal. The output signal is obtained by mixing the enhanced speech signals of M target users.

Method 6: As shown in Figure 13, input the voice-related data of M target users and the first noisy voice signal into the voice noise reduction model for noise reduction processing, and obtain the noise-reduced voice signals of M target users; based on The speech enhancement coefficients of the M target users respectively enhance the noise-reduced speech signals of the M target users to obtain the enhanced speech signals of the M target users; for any target user O in the M target users, the target user O's The ratio of the amplitude of the enhanced speech signal to the amplitude of the noise-reduced speech signal of the target user O is the speech enhancement coefficient of the target user O; based on the interference noise suppression coefficient, the interference noise signal is suppressed to obtain the interference noise suppression signal, where , the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient; the enhanced speech signals of M target users are fused with the interference noise suppression signal to obtain an output signal. The output signal is obtained by mixing the enhanced speech signals of the M target users and the interference noise suppression signals.

Further, the speech noise reduction model in way six is shown in Figure 14, the speech noise reduction model includes M first encoding networks, second encoding networks, TCN and first decoding networks; use M first encoding networks to respectively Feature extraction is performed on the registered voice signals of M target users to obtain the feature vectors of the registered voice signals of M target users; the second coding network is used to extract the features of the first noisy voice signal to obtain the features of the first noisy voice signal vector; perform mathematical operations on the eigenvectors of the registered voice signals of M target users and the eigenvectors of the first noisy voice signal, such as dot multiplication, to obtain the first eigenvector; use TCN to process the first eigenvector to obtain the first eigenvector Two feature vectors; and use the first decoding network for processing to obtain the noise-reduced voice signal and interference noise signal of the target user.

It should be noted that during a multi-person teleconference or call, there are multiple people at one end, and everyone is wearing headphones. Through these headphones, the VPU signal of each person can be collected, and then the noise reduction scheme based on the VPU signal can be carried out according to the above-mentioned VPU signal. Perform noise reduction processing.

In a feasible embodiment, for the interference noise suppression coefficient, it can be the default value, or it can be set by the target user based on their own needs. For example, as shown in the left figure in Figure 15, after the PNR function is enabled on the terminal device , the terminal device enters the PNR mode, and the display interface of the terminal device displays the stepless sliding control shown in the right figure in Figure 15. The target user can adjust the interference noise suppression coefficient by controlling the gray knob on the stepless sliding control, where the interference noise suppression The value range of the coefficient is [0,1]; when the control gray knob is slid to the far left, the interference noise suppression coefficient is 0, indicating that the PNR mode is not entered, and the interference noise is not suppressed; when the control gray knob is slid to the right When it is on the right side, the interference noise suppression coefficient is 1, which means that the interference noise is completely suppressed; when the control gray knob slides to the middle, it means that the interference noise is not completely suppressed.

Adjust the strength of noise reduction by adjusting the value of interference noise suppression coefficient.

Optionally, the infinite sliding control may be in the shape of a disk as shown in FIG. 15 , or in a bar shape, or in other shapes, which are not limited here.

It should be noted here that the speech enhancement coefficient may also be adjusted in the above manner.

In a feasible embodiment, the following method can be used to determine whether to use the traditional noise reduction algorithm or the noise reduction method disclosed in the present application for noise reduction. The method of the present application also includes:

Obtain the first noise segment and the second noise segment of the environment where the terminal device is located, wherein the first noise segment and the second noise segment are continuous in time; obtain the SNR and SPL of the first noise segment, if the first noise segment The SNR of the first noise segment is greater than the first threshold and the SPL of the first noise segment is greater than the second threshold, then the first temporary feature vector of the first noise segment is extracted; the second noise segment is degraded based on the first temporary speech feature vector of the first noise segment Noise processing to obtain the second noise reduction noise segment; damage assessment based on the second noise reduction noise segment and the first noise segment to obtain the first damage score; if the first damage score is not greater than the third threshold, enter the PNR mode, and determine the first noisy speech signal from the noise signal generated after the first noise segment, and use the first temporary feature vector as the feature vector of the registered speech signal.

Further, if the first damage score is not greater than the third threshold, the terminal device sends a first prompt message to the target user, the first prompt message is used to prompt the target user whether to make the terminal device enter the PNR mode; when the target user is detected Enter the PNR mode only after agreeing with the operation instruction that the terminal device enters the PNR mode.

Specifically, when the user uses the terminal device for the first time, the default microphone of the terminal device collects a voice signal, and processes the collected voice signal through a traditional noise reduction algorithm to obtain the user's noise-reduced voice signal; (For example, every 10 minutes) Acquire the first noise segment (such as the 6s voice signal currently collected by the microphone) and the second noise segment (such as the subsequent 10s voice signal of the 6s voice signal currently collected by the microphone) in the environment where the terminal device is located ), and obtain the SNR and SPL of the first noise segment; judge whether the SNR of the first noise segment is greater than 20dB and whether the SPL is greater than 40dB; if the SNR of the first noise segment is greater than the first threshold (such as 20dB) and the SPL is greater than the second threshold (such as 40dB), then extract the first temporary feature vector of the first noise segment; Utilize the first temporary feature vector to carry out noise reduction processing to the second noise segment to obtain the second noise reduction noise segment; based on the second noise reduction noise segment Perform damage assessment with the second noise segment to obtain a first damage score, wherein the first damage score is used to characterize the damage degree of the signal collected by the microphone of the terminal device, if the first damage score is greater, the damage degree is higher; if the first If the damage score is not greater than the third threshold, it means that the voice signal collected by the microphone is not damaged, and the terminal device sends a first prompt message to the user, and the first prompt message is used to prompt the user whether to make the terminal device enter the PNR mode; the prompt message can be It is voice information, it can be text information displayed on the display screen of the terminal device, and of course it can also be other forms of information, which are not limited here; the user's instruction for the prompt information is detected, and the instruction can be voice instruction, touch instruction , gesture commands, etc.; if the command is used to indicate that the user does not agree to enter the PNR mode, the traditional noise reduction algorithm will be used for noise reduction; if the command is used to indicate the user to agree to enter the PNR mode, after waiting for the user to finish speaking this sentence , enter the PNR mode, and determine the first noisy speech signal from the noise signal generated after the first noise segment, that is, obtain the first noisy speech signal from the second noise segment or the noise signal collected after the second noise segment Speech signal, and store the first temporary feature vector as the feature vector of the registered speech signal; if the first impairment score is greater than the third threshold, reacquire the first noise segment and the second noise segment at preset intervals, and repeat the execution above steps.

Wherein, determining the first noisy speech signal from the noise signal generated after the first noise segment may be understood as the first noisy speech signal is part or all of the noise signal generated after the first noise segment.

Optionally, the impairment score may be a signal-to-distortion ratio (SDR) value or a (perceptual evaluation of speech quality, PESQ) value.

In a feasible embodiment, the method of the present application also includes:

Specifically, if the SNR of the first noise segment is not greater than the first threshold or the SPL of the first noise segment is not greater than the second threshold, that is, the scene where the target voice feature cannot be extracted during this call, at this time, if the terminal device has Stored voiceprint information of historical users (such as voiceprint feature vectors), the terminal device detects that there is continuous voice (ie vad=1) in the input signal for more than 2 seconds, and the terminal device collects the voice signal to obtain the third noise segment , and perform denoising processing on the third noise segment based on the stored historical user's voiceprint feature vector to obtain a third denoising noise segment; perform damage assessment based on the third noise segment and the third denoising noise segment, To obtain the third impairment score; when the third impairment score is greater than the sixth threshold (such as 8dB) and the SNR of the third noise segment is less than the seventh threshold (such as 10dB), or when the third impairment score is greater than the eighth threshold (such as 12dB ) and the SNR of the third noise segment is not less than the seventh threshold value, it means that the voiceprint feature of the current user matches the stored voice feature, and a third prompt message is sent to the user through the terminal device, and the third prompt message is used to prompt Whether the current user makes the terminal device enter the PNR mode; the third prompt information can be voice information, can be text information displayed by the display screen of the terminal device, and can certainly be other forms of information, which are not limited here; detection To the user's instruction for the prompt information, the instruction may be a voice instruction, a touch instruction, a gesture instruction, and the like. If it is detected that the current user agrees to the operation instruction of turning on the PNR function of the terminal device, the terminal device enters the PNR mode, and performs noise reduction processing on the fourth noisy voice signal, which is after the third noise segment Acquired; if it is detected that the current user does not agree with the operation instruction to enable the PNR function of the terminal device, then maintain the traditional noise reduction algorithm to perform noise reduction processing on the fourth noisy speech signal.

In a feasible embodiment, the method of the present application also includes:

When it is detected that the terminal device is used again, the second noisy voice signal is obtained, and the traditional noise reduction algorithm is used, that is, the non-PNR mode is used to perform noise reduction processing on the second noisy voice signal to obtain the current user's noise-reduced voice signal; simultaneously judge whether the SNR of the second noisy speech signal is lower than the fourth threshold; when the SNR of the second noisy speech signal is lower than the fourth threshold, perform speech on the second noisy speech signal according to the first temporary feature vector Noise reduction processing to obtain the noise-reduced voice signal of the current user; performing damage assessment based on the noise-reduced voice signal of the current user and the second noisy voice signal to obtain a second damage score; when the second damage score is not greater than the fifth When the threshold is reached, a second prompt message is sent to the current user through the terminal device, and the second prompt message is used to prompt the current user that the terminal device can enter the PNR mode; after detecting the current user's operation instruction that the terminal device enters the PNR mode , enter the PNR mode to perform noise reduction processing on the third noisy voice signal, which is obtained after the second noisy voice signal; after detecting the current user's disagreement to enter the PNR mode operation instruction After that, continue to use the traditional noise reduction algorithm to perform noise reduction processing on the third noisy speech signal.

Specifically, when it is detected that the terminal device is used for a call again, the default microphone of the terminal device collects the second noisy voice signal, and uses a traditional noise reduction algorithm to process the second noisy voice signal, and outputs the current user's reduced voice signal. noisy speech signal. Determine whether the current environment is noisy at the same time, specifically judge whether the SNR of the second noisy speech signal is less than the fourth threshold; when the SNR of the second noisy speech signal is less than the fourth threshold (for example, the SNR is less than 10dB), it means that the current environment is noisy; according to this The noise reduction algorithm of the application uses the previously stored speech features (i.e. the first temporary feature vector) to perform noise reduction processing on the second noisy speech signal to obtain the current user's noise reduction speech signal; based on the current user's noise reduction The noisy speech signal and the second noisy speech signal are subjected to damage assessment to obtain the second damage score. The specific process can be referred to the above method, which will not be described here; if the second score is lower than the fifth threshold, it means that the current user and the storage match the voice features represented by the first temporary feature vector; send a second prompt message to the current user through the terminal device, and the second prompt message is used to prompt the current user to enable the PNR call function of the terminal device. If it is detected that the current user agrees to an operation command to enable the PNR function of the terminal device, the terminal device enters the PNR mode, and performs noise reduction processing on the third noisy voice signal, which is generated in the second noisy voice signal. After the signal is acquired; if it is detected that the current user does not agree to the operation instruction of turning on the PNR function of the terminal device, the traditional noise reduction algorithm is maintained to perform noise reduction processing on the third noisy voice signal.

Acquiring the first noise segment and the second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time; acquiring the microphone array of the auxiliary device of the terminal device The signal collected by the environment is used to calculate the DOA and SPL of the first noise segment; if the DOA of the first noise segment is greater than the ninth threshold and less than the tenth threshold, and the SPL of the first noise segment is greater than the eleventh threshold, Then extract the second temporary feature vector of the first noise segment; based on the second temporary speech feature vector, the second noise segment is denoised to obtain the fourth noise reduction noise segment; based on the fourth noise reduction noise segment and the second noise Perform damage assessment on the segment to obtain a fourth damage score; if the fourth damage score is not greater than the twelfth threshold, enter the PNR mode.

Obtaining the first noisy speech signal includes:

In a specific scenario, the terminal device is connected to a computer (a case of an auxiliary device) in a wired or wireless manner, and the microphone array of the computer collects the signal of the environment where the terminal device is located; then the terminal device obtains the The signals collected by the microphone array are then processed in the manner described above, which will not be described here.

It should be noted here that after extracting the first temporary feature vector or the second temporary feature vector, the terminal device stores the first temporary feature vector or the second temporary feature vector, and directly obtains the first temporary feature vector or the second temporary feature vector when it needs to be used later. The second temporary eigenvector avoids the failure to obtain the speech characteristics of the current user in a scene with high noise, thereby making it impossible to perform damage assessment.

Various noise reduction methods are disclosed in this application. For different scenarios, it is possible to judge whether to enter the PNR mode based on the scene information, and automatically identify the target user or object, and select the corresponding noise reduction method:

When it is detected that the terminal device is in the hands-free call state, it enters the PNR mode, and the owner who has registered the voiceprint feature is the target user; obtain the voice signal of the current user during the call for t seconds for voiceprint recognition, and compare the recognition result with the registration Compare the features of the voiceprint. If it is determined that the current user is not the owner of the phone, use the acquired voice signal of the current user for t seconds during the call as the registered voice signal of the user, and use the current user as the target user. Noise reduction is performed in the manner described above; wherein, the above t can be 3 or other values.

When it is detected that the terminal device is in the video call state, enter the PNR mode, and when the terminal device is in the video call, face recognition is performed on the image collected by the camera to determine the identity of the current user in the image; if the image contains multiple people, then use The person closest to the camera is the current user; the distance between the person in the image and the camera can be determined through sensors such as depth sensors on the terminal device; after the current user is determined, the terminal device detects whether the registered voice of the current user has been stored or The voice characteristics of the current user; if the registered voice of the current user or the voice characteristics of the current user have been stored, the current user is determined as the target user, and the registered voice or voice characteristics of the current user are used as the voice-related data of the current user; if the terminal If the device does not store the registered voice or voice features of the current user, the terminal device detects whether the current user is speaking through the lip shape detection method, and when it detects that the current user is speaking, intercepts the voice signal of the current user from the voice signal collected by the microphone , as the registered voice of the current user, the registered voice of the current user can be obtained by concatenating multiple signals, and the total duration is not less than 6s; obtain the first noisy voice signal through the microphone of the terminal device, and use method 1 or Noise reduction processing is performed in the manner described in the fourth manner.

When it is detected that the terminal device is connected to the headset and the terminal device is in a call state, it enters the PNR mode; and the terminal device detects whether the headset has a bone voiceprint sensor, and if so, collects the VPU of the target user through the bone voiceprint sensor of the headset signal, and use the methods described in methods 2, 3 and 4 to perform noise reduction processing; if the earphone does not have a bone voiceprint sensor, the user with the voice signal that has been registered in the earphone will be used as the target user by default, and the The user's registered voice and the first noisy voice signal collected by the headset are sent to the terminal device, and the terminal device uses the methods described in methods 1 and 4 to perform noise reduction; The microphone acquires the call voice of the user who is currently wearing the earphone, uses a part of the voice as the user's registered voice, and sends the registered voice and the first noisy voice signal collected by the earphone to the terminal device. The terminal device adopts method one Perform noise reduction in the same way as described in Method 4.

When it is detected that the terminal device is connected to a smart device (such as a smart large-screen device or a smart watch or a car Bluetooth device) and is in a video call state, enter the PNR mode to determine whether the current user's registered voice signal has been stored in the terminal device, if If the registered voice signal of the current user has been stored in the terminal device, the first noisy voice signal is collected through the smart device, and the first noisy voice signal is sent to the terminal device, and the terminal device adopts the methods described in method 1 and method 4 Perform noise reduction.

In a feasible embodiment, since the PNR is mainly used in an environment with relatively strong noise, and the user may not always be in an environment with relatively strong noise, it can provide An interface for users to set a specific function or the PNR function of an application. Apps can be various applications that require specific voice enhancement functions, such as calls, voice assistants, Changlian, recorders, etc.; specific functions can be various functions that require local voice recording, such as answering calls, video recording, using voice Assistant etc. As shown in the left figure in Figure 16, the display interface of the terminal device displays 3 function labels and 3 PNR control buttons corresponding to the 3 function labels; the user can control 3 functions respectively through the 3 PNR control buttons Turn off and turn on the PNR function; as shown in the left figure of Figure 16, the PNR function corresponding to the call and voice assistant is turned on, and the PNR function of video recording is turned off; as shown in the right figure of Figure 16, the display interface of the terminal device There are 5 application labels and 5 PNR control buttons corresponding to the 5 application labels. The user can control the PNR function of the 5 applications to be turned off and on through the 5 PNR control buttons; as shown in the right figure in Figure 16 , the PNR functions of Changba, Recorder and Changlian are turned on, and the PNR functions of calls and WeChat are turned off. It should be pointed out that, for example, when the PNR function of a call is enabled, when the user uses the terminal device to make a call, the terminal device directly enters the PNR mode. By adopting the above method, for different voice functions of the terminal device, the user can flexibly set whether to enable the PNR function.

As shown in Figure 17, the display interface of the terminal device taking the "Call" application program/"Answer a call" function as an example, provides a switch for enabling the PNR function on this interface, such as the "Enable PNR" function button in Figure 17; The left picture in Figure 17 is a schematic diagram of the display interface of the terminal device when a call comes in, the display interface displays the information of the caller, the "open PNR" function button, the "hang up" function button and the "answer" function button; in Figure 17 The picture on the right is a schematic diagram of the display interface of the terminal device when answering a call; the display interface displays the information of the caller, the "Enable PNR" function button, and the "Hang up" function button.

It should be pointed out here that some specific functions of the terminal device in this application are essentially functions of application programs installed on the terminal device. For example, the call function of the terminal device is realized through the application program "Phone".

Optionally, after it is detected that the target user has targeted the "Enable PNR" function button on the call interface (the interface shown in Figure 17), the display interface of the terminal device jumps to display the interface shown in the left figure in Figure 15, Target users can adjust the intensity of noise reduction by controlling the gray knob in Figure 15 to adjust the size of the interference noise suppression coefficient.

Through the UI interface shown in FIG. 16 , target users can flexibly enable or disable specific functions or PNR functions of applications according to their needs.

In a feasible embodiment, in order to reduce user operations, the present application further includes: judging whether the decibel value of the current ambient sound exceeds a preset decibel value (such as 50dB), or detecting whether the current ambient sound contains the voice of a non-target user ; If it is determined that the decibel value of the current ambient sound exceeds the preset decibel value, or a non-target user's voice is detected in the current ambient sound, the PNR function is enabled. When the target user uses the terminal device and needs to perform noise reduction, it directly enters the PNR mode; in other words, for a specific function or application program of the terminal device, the corresponding PNR function can be enabled in the above manner.

Further, when the target user clicks on the PNR as shown in a in Figure 18, and enters the PNR setting interface, the target user can turn on the "Smart Open" of the PNR through the "Smart Open" switch function key shown in Figure 18 b Function, after the PNR smart activation function is enabled, the PNR function can be enabled in the above-mentioned way for specific functions or applications of the terminal device. When the "smart open" function of the PNR is turned off, the display interface of the terminal device displays the content shown in c in Figure 18; the target user can turn on or off the specific function or application according to the demand through the PNR function key corresponding to the specific function or application Program the PNR function.

Enabling the smart PNR function as described above makes the terminal device more intelligent, reduces user operations, and makes user experience better.

In a feasible embodiment, in a call scenario, after the terminal device (also the local device) enables the PNR function, only the opposite end user knows the effect of the call after the PNR function is enabled, and it is difficult for the target user to judge whether to enable the PNR function. The function or the noise reduction strength set can make the peer user hear clearly. Whether the PNR function of the terminal device is enabled or the noise reduction strength is set by the peer device.

After the peer device (that is, another terminal device) detects that the user of the peer device activates the PNR function of the terminal device, the peer device sends a voice enhancement request to the terminal device, and the voice enhancement request is used to request to turn on the terminal device. The PNR function of the call function of the device; after the terminal device receives the enhanced voice request, it responds to the voice enhanced request and displays a reminder label on the display interface of the terminal device, which is the third prompt message. The reminder label is used to remind the target user The peer device requests to enable the PNR function of the call function of the local device, whether to enable the terminal device to enable the PNR function of the call function; the reminder label also includes a confirmation function button; when the terminal device detects the operation of the target user on the confirmation function button Afterwards, the terminal device turns on the PNR function of the call function, enters the PNR mode, and sends a response message to the peer device. The response message is used to respond to the above-mentioned enhanced voice request, and the response message is used to inform the peer device that the PNR function; after receiving the response message, the peer device displays a prompt label on the display interface of the peer device, and the prompt label is used to remind the user of the peer device that the voice of the target user has been enhanced.

Optionally, after the terminal device (also the local device) turns on the PNR function of the call, the peer device sends the interference noise suppression coefficient to the terminal device to adjust the noise reduction strength of the terminal device; or the peer device sends a signal to the terminal device The voice enhancement request carries the interference noise suppression coefficient. Optionally, when the peer device sends the interference noise suppression coefficient to the terminal device, the peer device also sends the speech enhancement coefficient of the target user to the terminal device.

Take the conversation between user A and user B as an example. As shown in Figure 19, the terminal device of user A (peer device) and the terminal device of user B (the above-mentioned terminal device is also the local device) conduct voice calls through the base station. The transmission of data realizes the call between user A and user B. The environment where user A is located is very noisy, and user B cannot hear what user A is saying; user B clicks the "enhance the other party's voice" function button displayed on the display interface of user B's terminal device to enhance user A's voice; User B's terminal device detects that user B presses the "enhance the other party's voice" function button, as shown in a in Figure 20, and sends an enhanced voice request to user A's terminal device, and the enhanced voice request is used to request user A's terminal device Turn on the PNR function of the call function; after user A's terminal device receives the voice enhancement request, a reminder label is displayed on the display interface of user A's terminal device, as shown in b in Figure 20, and the reminder label displays "the other party's request Enhance your voice, do you accept" to remind user A that user B requests to enhance his voice; if user B agrees to enhance his voice, user B clicks the "accept" function button displayed on the display interface of his terminal device; user B After detecting user B's operation on the "Accept" function button, user B's terminal device turns on the PNR function of the call function, and sends a response message to user A's terminal device through the base station. The response message is used to inform user A The PNR function of the call function of user B's terminal device has been turned on; after user B's terminal device receives the above-mentioned response message fed back by the base station, it will display a prompt label "the other party's voice is being enhanced" on its display interface to inform user B that it has been enhanced User A's voice, as shown in c in Figure 20.

It should be understood that the terminal device (local device) may also control the peer device to enable the PNR function of the call function in the above manner.

It should be pointed out here that the data transmitted between the terminal device and the peer device (including voice enhancement requests, response messages, etc.) is realized through the communication link established based on the phone number of the terminal device and the phone number of the peer device. Transmission.

During a call, the user of the peer device can decide whether to control the local device to enable the PNR function of the call function according to the voice quality of the target user it hears; of course, the target user can The voice quality of the user of the end device determines whether to control the PNR function of the end device to enable the call function, thereby improving the efficiency of the call between the two parties.

In a feasible embodiment, in a video recording scene, for example, when a parent records a video for a child, the child is far away from the terminal device (such as the shooting terminal), and the parent is relatively close to the terminal device, resulting in the effect of the child's video recording. The voice is small, while the voice of the parents is loud, but it is actually a video in which the voice of the child is loud, and the voice of the parents is weakened or even absent. For this problem, this application has the following solutions:

When recording a video or a video call, the display interface of the terminal device includes a first area and a second area, wherein the first area is used to display the video recording result or the content of the video call in real time, and the second area is used to display and adjust multiple Controls and corresponding labels of voice enhancement coefficients of objects (or target users); The operation instruction of the control of the voice enhancement coefficient obtains the voice enhancement coefficients of multiple objects, and then respectively performs enhancement processing on the noise-reduced voice signals of the multiple objects according to the voice enhancement coefficients of the multiple objects, so as to obtain the enhanced voice signals of the multiple objects ; Then an output signal is obtained based on the enhanced speech signals of multiple objects. The output signal is obtained by mixing the enhanced speech signals of multiple objects.

Optionally, after the noise-reduced speech signals and interference noise signals of multiple objects are obtained according to the fourth or sixth method, the speech enhancement coefficients of the multiple objects are obtained according to the above-mentioned method, and then the speech enhancement coefficients of the multiple objects are respectively assigned to The noise-reduced speech signal of one object is enhanced to obtain the enhanced speech signal of multiple objects; then the output signal is obtained based on the enhanced speech signal of multiple objects and the interference noise signal. The output signal is a mixture of enhanced speech signals and interference noise signals of multiple objects.

Optionally, after the noise-reduced speech signals and interference noise signals of multiple objects are obtained according to method four or method six, the above-mentioned second area is also used to display controls for adjusting the interference noise suppression coefficient, based on the user's target of the terminal device Operation instructions for the control for adjusting the speech enhancement coefficients of multiple objects and the control for adjusting the interference noise suppression coefficients Acquire the speech enhancement coefficients and interference noise suppression coefficients of multiple objects, and then perform multi- The noise-reduced speech signal of an object is enhanced to obtain the enhanced speech signal of multiple objects; the interference noise signal is suppressed according to the interference noise suppression coefficient to obtain the interference noise suppression signal; then based on the enhanced speech signal of multiple objects and the The interference noise suppresses the signal to obtain the output signal. The output signal is obtained by mixing the enhanced speech signals of multiple objects and the interference noise suppression signals.

It should be noted here that sound samples of multiple objects have been registered.

Taking object 2 as an example to record video for object 1, as shown in Figure 21, the display interface of the terminal device includes an area for displaying the video recording result for image 1, displaying the voice enhancement coefficient and object The control of the speech enhancement coefficient of 2, the control includes a bar-shaped sliding bar and a sliding button; object 2 can adjust the speech enhancement coefficient of object 1 by dragging the sliding button of object 1 on the sliding bar, and can drag the object The sliding button of 2 slides on the sliding bar to adjust the size of the voice enhancement coefficient of object 2, so as to realize the adjustment of the sound volume of object 1 and object 2 during video recording.

It should be pointed out that the object 2 becomes the photographer by dragging the object 2, which is not shown in FIG. 21 .

In a video call scenario, such as a video call between family members, as shown in Figure 22, the terminal device is in the hands of the daughter (subject 1), the mother (subject 2) cooks at a certain distance behind the daughter, the father is at the far end, and the father I want to hear my mother speak but can't hear clearly. Object 1 can increase the voice enhancement factor of object 2 by dragging the sliding button of object 2 to slide on the slide bar, thereby increasing the voice of object 2, that is, the voice of mother.

Optionally, as shown in the left figure in Figure 23, the controls for adjusting the speech enhancement coefficients of Object 1 and Object 2 are not displayed when the speech enhancement coefficients do not need to be adjusted. When the terminal device detects that Object 1 When it is necessary to adjust the speech enhancement coefficient of

object

1 or 2, the control for adjusting the speech enhancement coefficient of object 1 or object 2 is displayed on the display interface of the terminal device; as shown in the right figure in Figure 23, object 1 needs Adjust the voice enhancement coefficient of object 2. Object 1 long presses or clicks the display area of object 2 on the display interface of the terminal device. Of course, it can also be other operations. To adjust the control of the voice enhancement coefficient of object 2, within a period of time when object 1 slides the control of the voice enhancement coefficient of object 2, the terminal device does not detect any operation on the control for adjusting the voice enhancement coefficient of object 2 , hides the controls for adjusting the speech enhancement factor for object 2.

It should be pointed out that after the terminal device detects the operation on the area where the object 2 is displayed, the terminal device determines the voice signal feature of the object 2 from the database storing the voice signal features corresponding to the object, and then proceeds according to the noise reduction method of this application. noise reduction.

It should be understood that the operations on the area where the object 2 is displayed include but are not limited to long press and click, and of course other forms of operations are also possible.

When the terminal device detects a click, long press or other operations on the display interface, the terminal device first needs to recognize the object displayed in the operated area, and then determine the object that needs to be enhanced based on the pre-recorded relationship between the object and the voice signal. The speech signal, and then set the corresponding speech enhancement coefficient.

In a feasible embodiment, when the terminal device is an intelligent interactive device, the target voice-related data includes a voice signal including a wake-up word, and the noisy voice signal includes an audio signal including a command word.

Wherein, the above-mentioned smart interactive device is a device capable of voice interaction with the user, such as a sweeping robot, a smart speaker, a smart refrigerator, and the like.

For smart speakers and smart robots, it is often not possible to strictly limit user identities. For example, the smart speakers used in the family not only need to be able to be controlled by voice by family members, but also need to be able to use voice to interact with visiting guests. Family members can collect voice registration in advance, but for temporary visiting guests, voice registration cannot be collected in advance. For intelligent robots engaged in public services, it is necessary to respond to every possible user, and it is also impossible to require all possible users to collect voice registration in advance. However, when these devices are used, they often encounter complex situations with noisy backgrounds and many speakers. There is a stronger demand for voice enhancement of target users and suppression of other interference. In response to this requirement, this application provides the following solutions:

Taking the voice command of a smart speaker as an example, the microphone collects audio signals, and the voice wake-up module analyzes the collected audio signals to determine whether to wake up the device; the voice wake-up module first detects the collected signals and divides the voice segment come out. Then perform wake-up word recognition on the voice segment to determine whether the set wake-up word is included. For example, when using voice commands to control smart speakers by voice, the user generally needs to speak the wake-up word first, such as "little A little A".

The audio signal containing the wake-up word obtained by the voice wake-up module is used as the registered voice signal of the target user; the microphone collects the audio signal containing the user's voice command. Under normal circumstances, the user will speak specific commands after waking up the device, such as "what's the weather like tomorrow?", "play where is spring please" and other specific commands.

Take the user who speaks the wake-up word as the target user, and use the audio signal containing the voice command as the noisy voice signal, and use the method 1 to perform noise reduction processing to obtain the enhanced voice signal or output signal of the target user. The enhanced voice signal or output signal of the target user The voice signal or the output signal enhances the voice signal of the target user speaking the wake-up word, and effectively suppresses other interfering speakers and background noise.

It is judged whether there is a new wake-up word voice, and if so, the new voice signal containing the wake-up word is used as the registration voice signal of the new target user, and the user who speaks the new voice signal containing the wake-up word is the target user.

For example, user C speaks the wake-up word, "Little A, little A", and then user C can continue to use voice to control the smart speaker, but user B cannot use voice to control the smart speaker, only when user B outputs the wake-up word After "Little A, Little A", user B takes over the control of the speaker. At this time, user C's voice command will no longer be responded to by the speaker. Only after user C says "Little A, Little A" again can he take over the control of the speaker Control.

It can be seen that this embodiment provides a solution that does not need to register the voice in advance, and does not need to rely on images or other sensor information to enhance the voice of the target person and suppress other background noises and interfering voices. It is suitable for Smart speakers, smart robots, etc. are multi-user-oriented, and users have temporary devices.

It can be seen that in the solution of this application, the target user's noise-reduced voice signal is obtained by using the target voice-related data and the voice noise reduction model to perform noise reduction processing on the noisy voice signal, and the enhancement and suppression of the target user's voice is realized. Interference noise; through the introduction of voice enhancement coefficient and interference noise suppression coefficient, it meets the needs of users to adjust the noise reduction intensity; adopts the voice noise reduction model based on TCN or FTB+GRU structure for noise reduction, and delays in voice calls or video calls small, and the user has a good subjective sense of hearing; the noise reduction method of this application can also be used for noise reduction in multi-person scenarios, which meets the needs of multi-user noise reduction in multi-user scenarios; Targeted noise reduction in video scenes can automatically identify the target user, and retrieve the voiceprint information corresponding to the target user from the database to perform noise reduction, thereby improving the user experience; in the call scene or video call scene, based on the Enabling the PNR function according to the noise reduction requirement of the end user can improve the call quality of both parties in the call; adopting the method of this application to automatically enable the PNR function can improve the usability.

Referring to FIG. 24, FIG. 24 is a schematic structural diagram of a terminal device provided in an embodiment of the present application. As shown in Figure 24, the terminal device 2400 includes:

The acquiring unit 2401 is configured to acquire a noisy voice signal and target voice-related data after the terminal device enters the PNR mode, wherein the noisy voice signal includes an interference noise signal and a target user's voice signal; the target voice-related data is used to indicate the target user the user's voice characteristics;

The noise reduction unit 2402 is used to perform noise reduction processing on the first noisy speech signal through the trained speech noise reduction model according to the target speech related data, to obtain the noise reduction speech signal of the target user, wherein the speech noise reduction model is based on implemented by neural networks.

In a feasible embodiment, the acquiring unit 2401 is also configured to acquire the speech enhancement coefficient of the target user;

The noise reduction unit 2402 is further configured to perform enhancement processing on the target user's noise-reduced speech signal based on the target user's speech enhancement coefficient to obtain the target user's enhanced speech signal, wherein the amplitude of the target user's enhanced speech signal is the same as the target user's The ratio of the amplitude of the noise-reduced speech signal is the speech enhancement coefficient of the target user.

Further, the obtaining unit 2401 is also configured to obtain the interference noise suppression coefficient after obtaining the interference noise signal through the noise reduction processing;

The noise reduction unit 2402 is further configured to perform noise reduction processing on the interference noise signal based on the interference noise suppression coefficient to obtain the interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient ; The interference noise suppression signal is fused with the enhanced speech signal of the target user to obtain an output signal.

In one possible embodiment,

The obtaining unit 2401 is further configured to obtain the interference noise suppression coefficient after obtaining the interference noise signal through the noise reduction processing;

The noise reduction unit 2402 is further configured to suppress the interference noise signal based on the interference noise suppression coefficient to obtain the interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient; The interference noise suppressed signal is fused with the noise-reduced speech signal of the target user to obtain an output signal.

In a feasible embodiment, the target users include M, the target voice-related data includes the voice-related data of the M target users, the noise-reduced voice signals of the target users include the noise-reduced voice signals of the M target users, and the target user's voice The enhancement coefficient includes speech enhancement coefficients of M target users, M is an integer greater than 1, and the first noisy speech signal is subjected to noise reduction processing through a speech noise reduction model according to the target speech related data to obtain a noise-reduced speech signal of the target user aspect, the noise reduction unit 2402 is specifically used for:

For any target user A in the M target users, according to the voice-related data of the target user A, the first noisy voice signal is denoised by the voice noise reduction model, so as to obtain the denoising voice signal of the target user A;

In terms of enhancing the noise-reduced speech signal of the target user based on the speech enhancement coefficient of the target user to obtain the enhanced speech signal of the target user, the noise reduction unit 2402 is specifically used for:

Based on the speech enhancement coefficient of target user A, the noise-reduced speech signal of target user A is enhanced to obtain the enhanced speech signal of target user A; The ratio of amplitude is the speech enhancement coefficient of target user A; According to this mode, the noise reduction speech signal of each target user in M target users is processed, and the enhanced speech signals of M target users can be obtained;

The noise reduction unit 2402 is further configured to obtain an output signal based on the enhanced voice signals of the M target users.

In a feasible embodiment, the target users include M, and the target speech-related data includes the speech-related data of M target users, and the noise-reduced speech signals of the target users include the noise-reduced speech signals of M target users, and M is greater than 1 is an integer, in the aspect of performing noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the target speech-related data, so as to obtain the target user's noise-reduced speech signal and interference noise signal, the noise reduction unit 2402 is specifically used for:

In terms of performing noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the target speech related data to obtain the noise-reduced speech signal of the target user, the noise reduction unit 2402 is specifically used for:

Further, the noise reduction unit 2402 is also used for:

In a feasible embodiment, the relevant data of the target user A includes the registered voice signal of the target user A, and the registered voice signal of the target user A is the voice of the target user A collected in an environment where the noise decibel value is lower than a preset value signal, the speech noise reduction model includes a first coding network, a second coding network, a TCN and a first decoding network, and performs noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech related data of the target user A, In order to obtain the noise-reduced speech signal of the target user A, the noise reduction unit 2402 is specifically used for:

In a feasible embodiment, the relevant data of the i-th target user among the M target users includes the registration voice signal of the i-th target user, i is an integer greater than 0 and less than or equal to M, and the voice noise reduction model includes A coding network, a second coding network, a TCN and a first decoding network, the noise reduction unit 2402 is specifically used for:

In a feasible embodiment, for the voice-related data of M target users, the relevant data of each target user includes the registered voice signal of the target user, and the registered voice signal of target user A is when the noise decibel value is lower than the preset The voice signal of the target user A collected under the environment of the value, the voice noise reduction model includes M first encoding network, second encoding network, TCN, first decoding network and M third decoding network, according to the target voice related data Perform noise reduction processing on the first noisy speech signal through the speech noise reduction model, so as to obtain the noise reduction speech signal and the interference noise signal of the target user. The noise reduction unit 2402 is specifically used for:

In a feasible embodiment, the relevant data of the target user includes the VPU signal of the target user, and the speech noise reduction model includes a preprocessing module, a third encoding network, a GRU, a second decoding network and a postprocessing module,

In terms of performing noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the target speech related data, so as to obtain the noise-reduced speech signal of the target user, the noise reduction unit 2402 is specifically used for:

Carry out time-frequency transformation to the VPU signal of the first band noise speech signal and the target user respectively by the preprocessing module, to obtain the first frequency domain signal of the first band noise speech signal and the second frequency domain signal of the VPU signal; The frequency domain signal is fused with the second frequency domain signal to obtain a first fused frequency domain signal; the first fused frequency domain signal is successively processed through a third encoding network, a GRU, and a second decoding network to obtain a voice signal of a target user The mask of the third frequency domain signal; the first frequency domain signal is post-processed by the post processing module according to the mask of the third frequency domain signal to obtain the third frequency domain signal; the third frequency domain signal is frequency-time transformed, A noise-reduced speech signal of the target user is obtained; wherein, the third encoding module and the second decoding module are both implemented based on convolutional layers and frequency FTB.

In a feasible embodiment, the noise reduction unit 2402 is specifically used for:

Process the first fused frequency domain signal successively through the third encoding network, GRU and second decoding network to obtain the mask of the first frequency domain signal; through the post-processing module, the first frequency domain signal is processed according to the mask of the first frequency domain signal The signal is post-processed to obtain a fourth frequency domain signal of the interference noise signal; and frequency-time transformation is performed on the fourth frequency domain signal to obtain the interference noise signal.

In a feasible embodiment, the relevant data of the target user A includes the VPU signal of the target user A, and the voice noise reduction model includes a preprocessing module, a third encoding network, a GRU, a second decoding network and a postprocessing module. The voice-related data of user A is used for the first noisy voice signal through the voice noise reduction model to obtain the noise-reduced voice signal of target user A. The noise reduction unit 2402 is specifically used for:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user A through the preprocessing module to obtain the first frequency domain signal of the first noisy speech signal and the ninth frequency domain of the VPU signal of the target user A signal; the first frequency domain signal and the ninth frequency domain signal are fused to obtain the second fused frequency domain signal; the second fused frequency domain signal is successively processed by the third encoding network, the GRU and the second decoding network to obtain the target The mask of the tenth frequency domain signal of the voice signal of user A; The first frequency domain signal is post-processed by the mask of the tenth frequency domain signal by the post-processing module to obtain the tenth frequency domain signal; for the tenth frequency domain The signal is subjected to frequency-time transformation to obtain the noise-reduced speech signal of the target user A;

Wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.

In a feasible embodiment, the relevant data of the i-th target user among the M target users includes the VPU signal of the i-th target user, i is an integer greater than 0 and less than or equal to M, and the noise reduction unit 2402 is specifically used to :

Both the first noise signal and the VPU signal of the i-th target user are time-frequency transformed by the preprocessing module to obtain the eleventh frequency domain signal of the first noise signal and the tenth frequency domain signal of the i-th target user's VPU signal. Two frequency domain signals; the eleventh frequency domain signal and the twelfth frequency domain signal are fused to obtain the third fusion frequency domain signal; wherein the first noise signal is the voice that does not contain the 1st to i-1 target users The noisy voice signal of the signal; the mask of the thirteenth frequency domain signal and the tenth frequency domain signal of the voice signal of the i-th target user are obtained by processing the third fusion frequency domain signal successively through the third encoding network, GRU and the second decoding network A mask of the frequency-domain signal; post-processing the eleventh frequency-domain signal through the post-processing module according to the mask of the thirteenth frequency-domain signal and the mask of the eleventh frequency-domain signal, to obtain the thirteenth frequency-domain signal and the fourteenth frequency-domain signal of the second noise signal; the frequency-time transformation is carried out to the thirteenth frequency-domain signal and the fourteenth frequency-domain signal to obtain the noise-reduced voice signal and the second noise signal of the i-th target user, the first The second noise signal is the first noisy speech signal that does not contain the speech signals of the 1st to i-th target users; wherein, the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.

In a feasible embodiment, the noise reduction unit 2402 is specifically configured to:

For any target user A among the M target users, the noise-reduced voice signal of target user A is enhanced based on the voice enhancement coefficient of target user A to obtain the enhanced voice signal of target user A; the enhanced voice signal of target user A The ratio of the amplitude of the signal to the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

In terms of fusing the interference noise suppression signal with the enhanced speech signal of the target user to obtain an output signal, the noise reduction unit 2402 is specifically used for:

In a feasible embodiment, the relevant data of the target user includes the VPU signal of the target user, and the acquiring unit 2401 is further configured to: acquire the in-ear sound signal of the target user;

Carrying out time-frequency transformation to the first noisy speech signal and the in-ear sound signal respectively to obtain the first frequency-domain signal of the first noisy speech signal and the fifth frequency-domain signal of the in-ear sound signal; according to the VPU signal of the target user , the first frequency domain signal and the fifth frequency domain signal obtain the covariance matrix of the first noisy speech signal and the in-ear sound signal; obtain the first minimum variance distortion-free response MVDR weight based on the covariance matrix; based on the first MVDR weight, The first frequency domain signal and the fifth frequency domain signal obtain the sixth frequency domain signal of the first noisy speech signal and the seventh frequency domain signal of the in-ear sound signal; The eighth frequency domain signal of the noise speech signal; performing frequency-time transformation on the eighth frequency domain signal to obtain the noise-reduced speech signal of the target user.

Further, the noise reduction unit 2402 is also used for:

An interference noise signal is obtained for the first noisy speech signal according to the noise-reduced speech signal of the target user.

In a feasible embodiment, the relevant data of the target user A includes the VPU signal of the target user A, and the acquiring unit 2401 is also used to acquire the in-ear sound signal of the target user A;

In terms of performing noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech-related data of the target user A to obtain the noise-reduced speech signal of the target user A, the noise reduction unit 2402 is specifically used for:

Perform time-frequency transformation on the first noisy speech signal and the target user A's in-ear sound signal respectively to obtain the first frequency domain signal of the first noisy speech signal and the fifteenth frequency domain of the target user A's in-ear sound signal signal; according to the VPU signal of target user A, the first frequency domain signal and the 15th frequency domain signal, obtain the covariance matrix of the first noisy speech signal and the in-ear sound signal of target user A; obtain the second based on the covariance matrix MVDR weight; Based on the second MVDR weight, the first frequency domain signal and the fifteenth frequency domain signal, the sixteenth frequency domain signal of the first noisy speech signal and the seventeenth frequency domain of the ear sound signal of the target user A are obtained Signal; Obtain the eighteenth frequency domain signal of the noise reduction voice signal of target user A according to the sixteenth frequency domain signal and the seventeenth frequency domain signal; Carry out frequency-time transformation to the eighteenth frequency domain signal, to obtain target user A noise-reduced speech signal.

In a feasible embodiment, the obtaining unit 2401 is also used to:

Obtain the first noise segment and the second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are consecutive noise segments in time; obtain the signal-to-noise ratio SNR and sound pressure level SPL of the first noise segment

The terminal device 2400 also includes:

A determining unit 2403, configured to extract a first temporary feature vector of the first noise segment if the SNR of the first noise segment is greater than a first threshold and the SPL of the first noise segment is greater than a second threshold; based on the first temporary speech feature vector pair Perform noise reduction processing on the second noise segment to obtain the second noise reduction segment; perform damage assessment based on the second noise reduction segment and the second noise segment to obtain the first damage score; if the first damage score is not greater than the third Threshold, enter PNR mode;

In terms of obtaining the first noisy speech signal, the obtaining unit 2401 is specifically used to:

In a feasible embodiment, if the first damage score is not greater than the third threshold, the determining unit 2403 is further configured to:

The first prompt message is sent by the terminal device, and the first prompt message is used to prompt whether to enable the terminal device to enter the PNR mode; the PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.

In a feasible embodiment, the obtaining unit 2401 is further configured to obtain the second noisy speech signal when it is detected that the terminal device is used again;

The noise reduction unit 2402 is further configured to: when the SNR of the second noisy speech signal is lower than the fourth threshold, perform noise reduction processing on the second noisy speech signal according to the first temporary feature vector, so as to obtain the noise reduction of the current user. noisy speech signal;

The determination unit 2403 is further configured to perform damage assessment based on the current user's noise-reduced voice signal and the second noise-containing voice signal to obtain a second damage score; when the second damage score is not greater than the fifth threshold, send a message through the terminal device The second prompt information, the second prompt information is used to remind the current user that the terminal equipment can enter the PNR mode; after detecting the current user's consent to enter the PNR mode operation instruction, the terminal equipment is allowed to enter the PNR mode to respond to the third noisy voice The signal is subjected to noise reduction processing, and the third noisy speech signal is obtained after the second noisy speech signal; after detecting the current user's disagreement with the operation instruction to enter the PNR mode, use the non-PNR mode to process the third noisy speech signal. Noise reduction processing for noisy speech signals.

In a feasible embodiment, the acquiring unit 2401 is further configured to: if the SNR of the first noise segment is not greater than the first threshold or the SPL of the first noise segment is not greater than the second threshold, and the terminal device has stored the reference temporary sound pattern feature vector, to obtain the third noise segment;

The noise reduction unit 2402 is further configured to perform noise reduction processing on the third noise segment according to the reference temporary voiceprint feature vector to obtain a third noise reduction noise segment;

The determining unit 2403 is further configured to perform damage assessment according to the third noise segment and the third denoising noise segment to obtain a third damage score; if the third damage score is greater than the sixth threshold and the SNR of the third noise segment is less than the seventh threshold , or the third damage score is greater than the eighth threshold and the SNR of the third noise segment is not less than the seventh threshold, then a third prompt message is sent through the terminal device, and the third prompt message is used to prompt the current user that the terminal device can enter the PNR mode; After detecting the operation instruction of the current user agreeing to enter the PNR mode, the terminal device enters the PNR mode to perform noise reduction processing on the fourth noisy voice signal; after detecting the operation instruction of the current user not agreeing to enter the PNR mode , using a non-PNR mode to perform noise reduction processing on the fourth noisy speech signal; wherein, the fourth noisy speech signal is determined from the noise signal generated after the third noise segment.

In a feasible embodiment, the acquisition unit 2401 is also configured to acquire the first noise segment and the second noise segment of the environment where the terminal device 2400 is located; the first noise segment and the second noise segment are temporally continuous noise segments ; Obtain the signal collected by the microphone array of the auxiliary device of the terminal device 2400 for the environment where the terminal device 2400 is located;

The terminal device 2400 also includes:

The determining unit 2403 is configured to use the collected signal to calculate the signal arrival angle DOA and SPL of the first noise segment; if the DOA of the first noise segment is greater than the ninth threshold and less than the tenth threshold, and the SPL of the first noise segment is greater than the first threshold Eleven thresholds, then extract the second temporary feature vector of the first noise segment, and carry out noise reduction processing to the second noise segment based on the second temporary feature vector, to obtain the third noise reduction noise segment; based on the third noise reduction noise segment and Perform damage assessment on the second noise segment to obtain a fourth damage score; if the fourth damage score is greater than the twelfth threshold, enter the PNR mode;

In a feasible embodiment, if the fourth damage score is not greater than the twelfth threshold, the determining unit 2403 is further configured to:

The fourth prompt message is sent by the terminal device 2400, and the fourth prompt message is used to prompt whether to make the terminal device 2400 enter the PNR mode; the terminal device 2400 enters the PNR mode only after detecting an operation instruction of the target user agreeing to enter the PNR mode.

In a feasible embodiment, the terminal device 2400 also includes:

The detection unit 2404 is configured to not enter the PNR mode when it is detected that the terminal device is in the handset talking state;

When it is detected that the terminal device is in a video call, enter the PNR mode, wherein the target user is the owner of the terminal device or the user closest to the terminal device;

When it is detected that the terminal device is connected to the headset for talking, enter the PNR mode, wherein the target user is a user wearing the headset; the first noisy voice signal and the target voice-related data are collected through the headset; or,

When it is detected that the terminal device is connected to a smart large screen device, a smart watch or a vehicle-mounted device, it enters the PNR mode, where the target user is the owner of the terminal device or the user who is using the terminal device, the first noisy voice signal and the target voice-related data It is collected by the audio collection hardware of smart large-screen devices, smart watches, or vehicle-mounted devices.

In a feasible embodiment, the acquiring unit 2401 is also configured to: acquire the decibel value of the audio signal in the current environment,

The terminal device 2400 also includes:

The control unit 2405 is configured to determine whether the function activated by the terminal device or the PNR function corresponding to the application is enabled if the decibel value of the audio signal in the current environment exceeds the preset decibel value; if not enabled, then enable the application program activated by the terminal device Corresponding PNR function, and enter PNR mode.

In a feasible embodiment, the terminal device 2400 includes a display screen 2408, and the display screen 2408 includes multiple display areas,

Wherein, each display area in the plurality of display areas displays a label and a corresponding function key, and the function key is used to control the opening and closing of the PNR function of the application program indicated by the corresponding label.

In a feasible embodiment, when voice data transmission is performed between the terminal device and another terminal device, the terminal device 2400 further includes:

The receiving unit 2406 is configured to receive a voice enhancement request sent by another terminal device, where the voice enhancement request is used to instruct the terminal device to enable the PNR function of the call function;

The control unit 2405 is configured to send third prompt information through the terminal device in response to the voice enhancement request, and the third prompt information is used to prompt whether to enable the terminal device to enable the PNR function of the call function; After the PNR function of the call function, turn on the PNR function of the call function and enter the PNR mode;

The sending unit 2407 is configured to send a voice enhancement response message to another terminal device, where the voice enhancement response message is used to indicate that the terminal device has enabled the PNR function of the call function.

In a feasible embodiment, when the terminal device starts the video call or video recording function, the display interface of the terminal device includes a first area and a second area, the first area is used to display the content of the video call or video recording, and the second area The area is used to display M controls and corresponding M labels. The M controls correspond to the M target users one by one. Each control in the M controls includes a sliding button and a sliding bar. By controlling the sliding button to slide on the sliding bar, to adjust the speech enhancement coefficient of the target user indicated by the label corresponding to the control.

In a feasible embodiment, when the terminal device starts the video call or video recording function, the display interface of the terminal device includes a first area, and the first area is used to display the content of the video call or video recording; the terminal device 2400 also includes:

The control unit 2405 is configured to display a control corresponding to the object in the first area when an operation on any object in the video call content or video recording content is detected, the control includes a sliding button and a sliding bar, and the sliding button is controlled to move between Slide the slider to adjust the voice enhancement factor of the object.

In a feasible embodiment, when the terminal device is an intelligent interactive device, the target voice related data is the target user's voice signal including the wake-up word, and the noisy voice signal is the target user's audio signal including the command word.

It should be noted that the above-mentioned units (acquisition unit 2401, noise reduction unit 2402, determination unit 2403, detection unit 2404, control unit 2405, receiving unit 2406, sending unit 2407, and display screen 2408) are used to execute the relevant steps of the above-mentioned method .

In this embodiment, the terminal device 2400 is presented in the form of a unit. The "unit" here may refer to an application-specific integrated circuit (ASIC), a processor and memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above functions . In addition, the acquisition unit 2401 , noise reduction unit 2402 , determination unit 2403 , detection unit 2404 and control unit 2405 above may be implemented by the processor 2601 of the terminal device shown in FIG. 26 .

Referring to FIG. 25 , FIG. 25 is a schematic structural diagram of another terminal device provided by the implementation of the present application. As shown in Figure 25, the terminal device 2500 includes:

The sensor collection unit 2501 is configured to collect noisy speech signals, registered speech signals of the target user, VPU signals, video images, depth images and other information that can be used to determine the target user.

The storage unit 2502 is configured to store noise reduction parameters (including target user's speech enhancement coefficient and interference noise suppression coefficient), registered target users and their speech feature information.

The UI interaction unit 2504 is configured to receive user interaction information and send it to the noise reduction control unit 2506, and feed back the information fed back by the noise reduction control unit 2506 to the local user.

The communication unit 2505 is configured to send and receive interaction information with the peer user, and optionally, transmit a noisy voice signal of the peer and voice registration information of the peer user.

The processing unit 2503 includes a noise reduction control unit 2506 and a PNR processing unit 2507, wherein,

The noise reduction control unit 2506 is configured to configure the PNR noise reduction parameters according to the interaction information received by the local end and the peer end and the information stored in the storage unit, including but not limited to determining the user or target user for voice enhancement, voice enhancement coefficient and interference noise suppression coefficient, whether to enable the noise reduction function and the noise reduction method.

The PNR processing unit 2507 is configured to process the noisy speech signal collected by the sensor collection unit according to the configured noise reduction parameters to obtain an enhanced audio signal, that is, an enhanced speech signal of the target user.

It should be noted here that, for the specific functions of the PNR processing unit 2507, reference may be made to the relevant description of the functions of the noise reduction unit 2402.

As shown in FIG. 26 , the terminal device 2600 can be implemented with the structure in FIG. 26 , and the terminal device 2600 includes at least one processor 2601 , at least one memory 2602 , at least one display screen 2604 and at least one communication interface 2603 . The processor 2601 , the memory 2602 , the display screen 2604 and the communication interface 2603 are connected through the communication bus and complete mutual communication.

The processor 2601 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the programs of the above solutions.

The communication interface 2603 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (Wireless Local Area Networks, WLAN), etc.

The memory 2602 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (random access memory, RAM) or other types that can store information and instructions It can also be an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a read-only disc (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be programmed by a computer Any other medium accessed, but not limited to. The memory can exist independently and be connected to the processor through the bus. Memory can also be integrated with the processor.

The display screen 2604 may be an LCD display screen, an LED display screen, an OLED display screen, a 3D display screen or other display screens.

Wherein, the memory 2602 is used to store the application program codes for executing the above solutions, and the execution is controlled by the processor 2601, and the function buttons, labels, etc. described in the above method embodiments are displayed on the display screen. The processor 2601 is configured to execute application program codes stored in the memory 2602 .

The code stored in the memory 2602 can execute any of the speech enhancement methods provided above, for example: after the terminal device enters the PNR mode, obtain the noisy speech signal and the target speech related data, wherein the noisy speech signal contains the interference noise signal and the target speech signal. The user's voice signal; the target voice-related data is used to indicate the voice characteristics of the target user; according to the target voice-related data, the first noisy voice signal is denoised through the trained voice noise reduction model to obtain the target user's denoising Speech signal; wherein, the speech noise reduction model is implemented based on a neural network.

The embodiment of the present application also provides a computer storage medium, wherein the computer storage medium can store a program, and the program includes some or all steps of any speech enhancement method described in the above method embodiments when executed.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Depending on the application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or can be Integrate into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. Several instructions are included to make a computer device (which may be a personal computer, server or network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes such as U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash disk , Read-only memory (English: Read-Only Memory, referred to as: ROM), random access device (English: Random Access Memory, referred to as: RAM), magnetic disk or optical disc, etc.

The embodiments of the present application have been introduced in detail above, and specific examples have been used in this paper to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; meanwhile, for Those skilled in the art will have changes in specific implementation methods and application scopes based on the ideas of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

A voice enhancement method, the method is applied to a terminal device, characterized in that it comprises:

After the terminal device enters the specific person noise reduction PNR mode, the first noisy speech signal and target speech related data are acquired, wherein the first noisy speech signal includes an interference noise signal and a target user's speech signal; The target voice-related data is used to indicate the voice characteristics of the target user;

Perform noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech related data to obtain a noise reduction speech signal of the target user; wherein the speech noise reduction model is based on a neural network Achieved.
The method according to claim 1, further comprising:

Acquiring the speech enhancement coefficient of the target user;

The noise-reduced speech signal of the target user is enhanced based on the speech enhancement coefficient of the target user to obtain the enhanced speech signal of the target user, wherein the amplitude of the enhanced speech signal of the target user is the same as that of the target user The ratio of the amplitude of the user's noise-reduced speech signal is the speech enhancement coefficient.
The method according to claim 2, wherein the interference noise signal is also obtained through the noise reduction processing; the method further comprises:

Obtain the interference noise suppression coefficient;

The interference noise signal is suppressed based on the interference noise suppression coefficient to obtain an interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient;

The interference noise suppression signal is fused with the enhanced speech signal of the target user to obtain an output signal.
The method according to claim 1, wherein the interference noise signal is also obtained through the noise reduction processing; the method further comprises:

Obtain the interference noise suppression coefficient;

The interference noise signal is suppressed based on the interference noise suppression coefficient to obtain an interference noise suppression signal, wherein the ratio of the amplitude of the interference noise suppression signal to the amplitude of the interference noise signal is the interference noise suppression coefficient;

The interference noise suppressed signal is fused with the noise-reduced speech signal of the target user to obtain an output signal.
The method according to claim 2, wherein the target users include M, the target voice-related data include the voice-related data of the M target users, and the noise-reduced voice signals of the target users include the The noise reduction voice signals of the M target users, the voice enhancement coefficients of the target users include the voice enhancement coefficients of the M target users, and the M is an integer greater than 1;

The step of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain a noise-reduced speech signal of the target user includes:

For any target user A among the M target users, according to the speech-related data of the target user A, the noise reduction process is performed on the first noisy speech signal through the speech noise reduction model, so as to obtain the The noise-reduced voice signal of the target user A;

The step of enhancing the noise-reduced speech signal of the target user based on the speech enhancement coefficient of the target user to obtain the enhanced speech signal of the target user includes:

Based on the speech enhancement coefficient of the target user A, the noise-reduced speech signal of the target user A is enhanced to obtain the enhanced speech signal of the target user A; the amplitude of the enhanced speech signal of the target user A is related to the The ratio of the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

The method also includes:

An output signal is obtained based on the enhanced speech signals of the M target users.
The method according to claim 3, wherein the target users include M, the target voice-related data include the voice-related data of the M target users, and the noise-reduced voice signals of the target users include the The denoising voice signals of the M target users, the M being an integer greater than 1;

Perform noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech related data, to obtain the target user's noise-reduced speech signal and the interference noise signal, including:

Perform noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech related data of the first target user among the M target users, so as to obtain the noise reduction of the first target user a voice signal and a first noisy voice signal that does not contain the voice signal of the first target user;

Perform noise reduction processing on the first noisy speech signal that does not contain the speech signal of the first target user through the speech noise reduction model according to the speech-related data of the second target user among the M target users. , obtaining the noise-reduced speech signal of the second target user and the first noisy speech signal not including the speech signal of the first target user and the speech signal of the second target user;

Repeat the above process until the first noisy speech signal that does not contain the speech signals of the 1st to M-1 target users is subjected to noise reduction processing through the speech noise reduction model according to the speech related data of the Mth target user , to obtain the noise-reduced speech signal of the Mth target user and the interference noise signal.
The method according to claim 3, wherein the target users include M, the target voice-related data include the voice-related data of the M target users, and the noise-reduced voice signals of the target users include the The denoising voice signals of the M target users, the M being an integer greater than 1;

Performing noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the target speech related data, so as to obtain the noise-reduced speech signal of the target user and the interference noise signal, including:

Perform noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech-related data of the M target users, so as to obtain the noise-reduced speech signals of the M target users and the interference noise Signal.
The method according to any one of claims 1-4, wherein the target user comprises M, the relevant data of the target user comprises the registered voice signal of the target user, and the voice noise reduction model comprises A first encoding network, a second encoding network, a temporal convolutional network TCN and a first decoding network;

The step of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain a noise-reduced speech signal of the target user includes:

Using the first coding network and the second coding network to perform feature extraction on the registration speech signal of the target user and the first noisy speech signal respectively, so as to obtain a feature vector of the registration speech signal of the target user and the feature vector of the first noisy speech signal;

Obtaining a first eigenvector according to the eigenvector of the registration voice signal of the target user and the eigenvector of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

Obtaining the noise-reduced speech signal of the target user according to the first decoding network and the second feature vector.
The method according to claim 8, characterized in that the method further comprises:

The interference noise signal is also obtained according to the first decoding network and the second eigenvector.
The method according to claim 5, wherein the relevant data of the target user A includes the registered voice signal of the target user A, and the voice noise reduction model includes a first coding network, a second coding network, a TCN and the first decoding network;

According to the voice-related data of the target user A, the noise reduction processing is performed on the first noisy voice signal through a voice noise reduction model, so as to obtain the noise-reduced voice signal of the target user A, including:

Using the first encoding network and the second encoding network to perform feature extraction on the registration voice signal of the target user A and the first noisy voice signal, so as to obtain the registration voice signal of the target user A A feature vector and a feature vector of the first noisy speech signal;

Obtaining a first feature vector according to the feature vector of the registered voice signal of the target user A and the feature vector of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

The noise-reduced speech signal of the target user A is obtained according to the first decoding network and the second feature vector.
The method according to claim 6, wherein the relevant data of the i-th target user among the M target users includes the registration voice signal of the i-th target user, and the i is greater than 0 and less than or An integer equal to M, the speech noise reduction model includes a first coding network, a second coding network, a TCN and a first decoding network,

Using the first coding network and the second coding network to perform feature extraction on the registration speech signal of the target user and the first noise signal respectively, to obtain the feature vector and the registration speech signal of the i-th target user. The eigenvector of the first noise signal; wherein, the first noise signal is the first noisy speech signal that does not contain the speech signals of the 1st to i-1 target users;

Obtaining a first feature vector according to the feature vector of the i-th target user's registered voice signal and the feature vector of the first noise signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

According to the first decoding network and the second eigenvector, the noise-reduced speech signal and the second noise signal of the i-th target user are obtained, wherein the second noise signal does not contain the first to i-th targets A first noisy speech signal of the user's speech signal.
The method according to claim 7, wherein, for the voice-related data of the M target users, the relevant data of each target user includes the registered voice signal of the target user, and the voice noise reduction model includes M a first encoding network, a second encoding network, a TCN, a first decoding network and M third decoding networks;

performing noise reduction processing on the noisy speech through the speech noise reduction model according to the speech-related data of the M target users, so as to obtain the noise-reduced speech signals of the M target users and the interference noise signal ,include:

Utilize the M first encoding networks to perform feature extraction on the registered voice signals of the M target users respectively, to obtain feature vectors of the registered voice signals of the M target users; use the second encoding network to extract the features of the first Performing feature extraction on the noisy speech signal to obtain a feature vector of the first noisy speech signal;

Obtaining a first eigenvector according to the eigenvectors of the registration voice signals of the M target users and the eigenvectors of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

According to each of the M third decoding networks, the second feature vector and the feature vector output by the first encoding network corresponding to the third decoding network, the noise-reduced voices of M target users are obtained. Signal;

The interference noise signal is obtained according to the first decoding network, the second feature vector and the feature vector of the first noisy speech signal.
The method according to any one of claims 1-4, wherein the relevant data of the target user includes the voice pickup VPU signal of the target user, and the voice noise reduction model includes a preprocessing module, a third coding A network, a gated recurrent unit GRU, a second decoding network and a post-processing module;

The step of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain a noise-reduced speech signal of the target user includes:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user through the preprocessing module, so as to obtain the first frequency domain signal and the VPU signal of the first noisy speech signal The second frequency domain signal;

merging the first frequency domain signal and the second frequency domain signal to obtain a first fused frequency domain signal;

sequentially processing the first fused frequency domain signal through the third encoding network, the GRU, and the second decoding network to obtain a mask of a third frequency domain signal of the speech signal of the target user;

performing post-processing on the first frequency-domain signal by the post-processing module according to the mask of the third frequency-domain signal, to obtain the third frequency-domain signal;

performing frequency-time transformation on the third frequency domain signal to obtain a noise-reduced speech signal of the target user;

Wherein, both the third encoding module and the second decoding module are implemented based on a convolutional layer and a frequency domain transformation module FTB.
The method according to claim 13, characterized in that,

Processing the first fused frequency domain signal successively through the third encoding network, the GRU, and the second decoding network to obtain a mask of the first frequency domain signal;

performing post-processing on the first frequency-domain signal by the post-processing module according to the mask of the first frequency-domain signal, to obtain a fourth frequency-domain signal of the interference noise signal;

Perform frequency-time transformation on the fourth frequency domain signal to obtain the interference noise signal.
The method according to claim 5, wherein the relevant data of the target user A includes the VPU signal of the target user A, and the speech noise reduction model includes a preprocessing module, a third encoding network, a GRU, a first The second decoding network and post-processing module, the first noisy speech signal is subjected to noise reduction processing through a speech noise reduction model according to the speech related data of the target user A, so as to obtain the noise-reduced speech of the target user A Signals, including:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user A through the preprocessing module to obtain the first frequency domain signal of the first noisy speech signal and the target A ninth frequency domain signal of the VPU signal of user A;

merging the first frequency domain signal and the ninth frequency domain signal to obtain a second fused frequency domain signal;

sequentially processing the second fused frequency domain signal through the third encoding network, the GRU, and the second decoding network to obtain a mask of the tenth frequency domain signal of the voice signal of the target user A;

performing post-processing on the first frequency domain signal by the post-processing module according to the mask of the tenth frequency domain signal, to obtain the tenth frequency domain signal;

performing frequency-time transformation on the tenth frequency domain signal to obtain the noise-reduced speech signal of the target user A;

Wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.
The method according to claim 6, wherein the relevant data of the i-th target user among the M target users includes the VPU signal of the i-th target user, and the i is greater than 0 and less than or equal to an integer of M,

Time-frequency transformation is performed on the first noise signal and the VPU signal of the ith target user by the preprocessing module to obtain the eleventh frequency domain signal of the first noise signal and the ith target user The twelfth frequency domain signal of the VPU signal;

merging the eleventh frequency domain signal and the twelfth frequency domain signal to obtain a third fused frequency domain signal; wherein the first noise signal does not include the 1st to i-1 target users a first noisy speech signal of the speech signal;

Processing the third fused frequency domain signal successively through the third coding network, the GRU, and the second decoding network to obtain a mask of the thirteenth frequency domain signal of the voice signal of the i-th target user and a mask of the eleventh frequency domain signal;

The post-processing module performs post-processing on the eleventh frequency domain signal according to the mask of the thirteenth frequency domain signal and the mask of the eleventh frequency domain signal to obtain the thirteenth frequency domain signal. domain signal and a fourteenth frequency domain signal of the second noise signal;

performing frequency-time transformation on the thirteenth frequency domain signal and the fourteenth frequency domain signal to obtain the noise-reduced voice signal of the ith target user and the second noise signal, the second noise signal be the first noisy speech signal that does not contain the speech signals of the 1st to i target users;

Wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.
The method according to any one of claims 6, 7, 11, 12, and 16, wherein the noise-reduced speech signal of the target user is enhanced based on the speech enhancement coefficient of the target user, so as to Obtaining the enhanced voice signal of the target user includes:

For the target user A among the M target users, performing enhancement processing on the noise-reduced voice signal of the target user A based on the voice enhancement coefficient of the target user A, to obtain the enhanced voice signal of the target user A; The ratio of the amplitude of the enhanced voice signal of the target user A to the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

The merging of the interference noise suppression signal and the enhanced speech signal of the target user to obtain an output signal includes:

The enhanced speech signals of the M target users are fused with the interference noise suppression signal to obtain the output signal.
The method according to any one of claims 1-4, wherein the relevant data of the target user includes a VPU signal of the target user, and the method further includes: acquiring an in-ear sound signal of the target user ;

The step of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain a noise-reduced speech signal of the target user includes:

respectively performing time-frequency transformation on the first noisy speech signal and the in-ear sound signal to obtain a first frequency-domain signal of the first noisy speech signal and a fifth frequency-domain signal of the in-ear sound signal Signal;

obtaining the covariance matrix of the first noisy speech signal and the in-ear sound signal according to the target user's VPU signal, the first frequency domain signal, and the fifth frequency domain signal;

Obtaining the first minimum variance distortion-free response MVDR weight based on the covariance matrix;

Obtaining a sixth frequency domain signal of the first noisy speech signal and a seventh frequency domain signal of the in-ear sound signal based on the first MVDR weight, the first frequency domain signal and the fifth frequency domain signal Signal;

obtaining an eighth frequency domain signal of the noise-reduced speech signal according to the sixth frequency domain signal and the seventh frequency domain signal;

performing frequency-time transformation on the eighth frequency domain signal to obtain the noise-reduced speech signal.
The method according to claim 18, further comprising:

The interference noise signal is obtained according to the noise-reduced speech signal and the first noisy speech signal.
The method according to claim 5, wherein the relevant data of the target user A includes the VPU signal of the target user A, and the method further comprises: acquiring an in-ear sound signal of the target user A;

According to the voice-related data of the target user A, the noise reduction processing is performed on the first noisy voice signal through a voice noise reduction model, so as to obtain the noise-reduced voice signal of the target user A, including:

performing time-frequency transformation on the first noisy speech signal and the target user A's ear sound signal respectively, to obtain the first frequency domain signal of the first noisy speech signal and the target user A's ear sound signal a fifteenth frequency domain signal of the sound signal;

Obtaining the covariance matrix of the first noisy speech signal and the in-ear sound signal of the target user A according to the VPU signal of the target user A, the first frequency domain signal and the fifteenth frequency domain signal ;

obtaining a second MVDR weight based on the covariance matrix;

Obtaining the sixteenth frequency domain signal of the first noisy speech signal and the in-ear sound of the target user A based on the second MVDR weight, the first frequency domain signal and the fifteenth frequency domain signal The seventeenth frequency domain signal of the signal; the eighteenth frequency domain signal of the noise-reduced voice signal of the target user A is obtained according to the sixteenth frequency domain signal and the seventeenth frequency domain signal;

Perform frequency-time transformation on the eighteen frequency domain signals to obtain the noise-reduced speech signal of the target user A.
The method according to any one of claims 8-12, wherein the method further comprises:

Acquiring a first noise segment and a second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time;

Acquiring the signal-to-noise ratio SNR and sound pressure level SPL of the first noise segment;

If the SNR of the first noise segment is greater than a first threshold and the SPL of the first noise segment is greater than a second threshold, extracting a first temporary feature vector of the first noise segment;

performing noise reduction processing on the second noise segment based on the first temporary speech feature vector to obtain a second noise reduction noise segment;

performing damage assessment based on the second denoised noise segment and the second noise segment to obtain a first damage score;

If the first damage score is not greater than a third threshold, enter the PNR mode;

The acquisition of the first noisy speech signal includes:

determining said first noisy speech signal from a noise signal generated after said first noise segment;

The feature vector of the enrollment speech signal includes the first temporary feature vector.
The method according to claim 21, wherein if the first damage score is not greater than a third threshold, the method further comprises:

Sending first prompt information through the terminal device, where the first prompt information is used to prompt whether to enable the terminal device to enter the PNR mode;

The PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.
The method according to claim 21 or 22, further comprising:

When it is detected that the terminal device is used again, acquiring a second noisy voice signal;

When the SNR of the second noisy speech signal is lower than a fourth threshold, perform noise reduction processing on the second noisy speech signal according to the first temporary feature vector, so as to obtain the noise reduction of the current user voice signal;

performing impairment assessment based on the noise-reduced speech signal of the current user and the second noisy speech signal to obtain a second impairment score;

When the second damage score is not greater than the fifth threshold, sending the second prompt message through the terminal device, the second prompt message is used to prompt the current user that the terminal device can enter the PNR mode;

After detecting the operation instruction of the current user agreeing to enter the PNR mode, the terminal device enters the PNR mode to perform noise reduction processing on the third noisy voice signal, and the third noisy voice signal is in the PNR mode. acquired after the second noisy speech signal;

After detecting that the current user does not agree to enter the operation instruction of the PNR mode, the non-PNR mode is used to perform noise reduction processing on the third noisy speech signal.
The method according to claim 21 or 22, further comprising:

If the SNR of the first noise segment is not greater than the first threshold or the SPL of the first noise segment is not greater than the second threshold, and the terminal device has stored a reference temporary voiceprint feature vector, obtain the third noise fragments;

performing noise reduction processing on the third noise segment according to the reference temporary voiceprint feature vector to obtain a third noise reduction noise segment;

performing damage assessment according to the third noise segment and the third noise-reduced noise segment to obtain a third damage score;

If the third impairment score is greater than the sixth threshold and the SNR of the third noise segment is less than the seventh threshold, or the third impairment score is greater than the eighth threshold and the SNR of the third noise segment is not less than the first seven thresholds, the terminal device sends the third prompt information, and the third prompt information is used to prompt the current user that the terminal device can enter the PNR mode;

After detecting the operation instruction of the current user agreeing to enter the PNR mode, make the terminal device enter the PNR mode to perform noise reduction processing on the fourth noisy voice signal; After agreeing to enter the operation instruction of the PNR mode, use the non-PNR mode to perform noise reduction processing on the fourth noisy speech signal;

Wherein, the fourth noisy speech signal is determined from a noise signal generated after the third noise segment.
The method according to any one of claims 8-12, wherein the method further comprises:

Acquiring a first noise segment and a second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time;

Acquiring the signal collected by the microphone array of the auxiliary device of the terminal device for the environment where the terminal device is located; using the collected signal to calculate the signal arrival angle DOA and SPL of the first noise segment; if the second The DOA of a noise segment is greater than the ninth threshold and less than the tenth threshold, and the SPL of the first noise segment is greater than the eleventh threshold, then the second temporary feature vector of the first noise segment is extracted, based on the second Performing denoising processing on the second noise segment with the temporary feature vector to obtain a third denoising noise segment; performing damage assessment based on the third denoising noise segment and the second noise segment to obtain a fourth damage score ; If the fourth impairment score is greater than the twelfth threshold, enter the PNR mode;

The acquisition of the first noisy speech signal includes:

determining said first noisy speech signal from a noise signal generated after said first noise segment;

The feature vector of the enrollment speech signal includes the second temporary feature vector.
The method according to claim 25, wherein if the fourth damage score is not greater than the twelfth threshold, the method further comprises:

Sending fourth prompt information through the terminal device, where the fourth prompt information is used to prompt whether to make the terminal device enter the PNR mode;

The PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.
The method according to any one of claims 1-20, wherein the method further comprises:

When it is detected that the terminal device is in a handheld talking state, the PNR mode is not entered;

Entering the PNR mode when it is detected that the terminal device is in a hands-free call state, wherein the target user is the owner of the terminal device or the user who is using the terminal device;

When it is detected that the terminal device is in a video call state, enter the PNR mode, wherein the target user is the owner of the terminal device or the user closest to the terminal device;

When it is detected that the terminal device is connected to the earphone for conversation, enter the PNR mode, wherein the target user is a user wearing the earphone; the first noisy voice signal and the target voice-related data are collected by said headset; or

When it is detected that the terminal device is connected to a smart large-screen device, a smart watch or a vehicle-mounted device, enter the PNR mode, wherein the target user is the owner of the terminal device or the user who is using the terminal device, The first noisy voice signal and target voice-related data are collected by audio collection hardware of the smart large-screen device, the smart watch, or the vehicle-mounted device.
The method according to any one of claims 1-20, wherein the method further comprises:

Get the decibel value of the audio signal in the current environment;

If the decibel value of the audio signal in the current environment exceeds the preset decibel value, and the PNR function corresponding to the application program started by the terminal device is not enabled, then enable the PNR function corresponding to the application program started by the terminal device, and enter The PNR mode.
The method according to any one of claims 1-20, wherein the terminal device includes a display screen, and the display screen includes a plurality of display areas,

Wherein, each display area in the plurality of display areas displays a label and a corresponding function key, and the function key is used to control the function indicated by the corresponding label or the opening and closing of the PNR function of the application program.
The method according to any one of claims 1-20, wherein when voice data transmission is performed between the terminal device and another terminal device, the method further comprises:

receiving a voice enhancement request sent by the other terminal device, where the voice enhancement request is used to instruct the terminal device to enable the PNR function of the call function;

In response to the voice enhancement request, sending third prompt information through the terminal device, where the third prompt information is used to prompt whether to enable the terminal device to enable the PNR function of the call function;

After detecting the operation command for confirming to enable the PNR function of the call function, enable the PNR function of the call function, and enter the PNR mode;

Sending a speech enhancement response message to the other terminal device, where the speech enhancement response message is used to indicate that the terminal device has enabled the PNR function of the call function.
The method according to any one of claims 5-7, 10-12 and 17, wherein when the terminal device starts the video call or video recording function, the display interface of the terminal device includes the first area and the second area. Two areas, the first area is used to display video call content or video recording content, the second area is used to display M controls and corresponding M labels, and the M controls are related to the M target users In one-to-one correspondence, each control in the M controls includes a sliding button and a sliding bar, and by controlling the sliding button to slide on the sliding bar, the voice enhancement coefficient of the target user indicated by the label corresponding to the control is adjusted .
The method according to any one of claims 5-7, 10-12 and 17, wherein when the terminal device starts the video call or video recording function, the display interface of the terminal device includes a first area, so The above-mentioned first area is used to display the content of the video call or the content of the video recording;

When an operation on any object in the video call content or video recording content is detected, the control corresponding to the object is displayed in the first area, and the control includes a sliding button and a sliding bar. Slide on the slide bar to adjust the speech enhancement coefficient of the object.
The method according to any one of claims 1-4 and 8, wherein when the terminal device is an intelligent interactive device, the target voice-related data includes a voice signal containing a wake-up word, and the first band The noisy speech signal includes an audio signal containing command words.
A terminal device, characterized in that it includes:

An acquisition unit, configured to acquire a first noisy speech signal and target speech related data after the terminal device enters a person-specific noise reduction PNR mode, wherein the first noisy speech signal includes an interference noise signal and the target speech signal a user's voice signal, the target voice-related data is used to indicate the voice characteristics of the target user;

A noise reduction unit, configured to perform noise reduction processing on the first noisy speech signal according to the target speech-related data and a speech noise reduction model, to obtain a noise-reduced speech signal of the target user, wherein the speech reduction The noise model is realized based on neural network.
A terminal device according to claim 34, characterized in that,

The acquiring unit is further configured to acquire the speech enhancement coefficient of the target user;

The noise reduction unit is further configured to perform enhancement processing on the target user's noise-reduced speech signal based on the target user's speech enhancement coefficient to obtain the target user's enhanced speech signal, wherein the target user's The ratio of the amplitude of the enhanced voice signal to the amplitude of the noise-reduced voice signal of the target user is the voice enhancement coefficient of the target user.
The terminal device according to claim 35, characterized in that,

The obtaining unit is further configured to obtain the interference noise coefficient after the interference noise signal is also obtained through the noise reduction processing;

The noise reduction unit is further configured to suppress the interference noise signal based on the interference noise suppression coefficient to obtain an interference noise suppression signal, wherein the amplitude of the interference noise suppression signal is the same as the amplitude of the interference noise signal The ratio of the amplitude is the interference noise suppression coefficient; the interference noise suppression signal is fused with the enhanced speech signal of the target user to obtain an output signal.
The terminal device according to claim 34, characterized in that,

The acquisition unit is further configured to acquire an interference noise suppression coefficient after the interference noise signal is also obtained through the noise reduction processing;

The noise reduction unit is further configured to suppress the interference noise signal based on the interference noise suppression coefficient to obtain an interference noise suppression signal, wherein the amplitude of the interference noise suppression signal is the same as the amplitude of the interference noise signal The ratio of is the interference noise suppression coefficient; the interference noise suppression signal is fused with the noise-reduced speech signal of the target user to obtain an output signal.
The terminal device according to claim 35, wherein the target users include M, the target voice-related data include voice-related data of M target users, and the noise-reduced voice signals of the target users include the Noise-reducing voice signals of M target users, the voice enhancement coefficients of the target users include the voice enhancement coefficients of the M target users, and the M is an integer greater than 1;

In the aspect of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain a noise-reduced speech signal of the target user, the noise reduction unit is specifically used for:

For any target user A among the M target users, according to the voice-related data of the target user A, the noise reduction process is performed on the first noisy voice signal through the voice noise reduction model to obtain the target user A noise reduction voice signal;

In the aspect of performing enhancement processing on the target user's noise-reduced speech signal based on the target user's speech enhancement coefficient to obtain the target user's enhanced speech signal, the noise reduction unit is specifically used for:

Based on the speech enhancement coefficient of the target user A, the noise-reduced speech signal of the target user A is enhanced to obtain the enhanced speech signal of the target user A; the amplitude of the enhanced speech signal of the target user A is related to the The ratio of the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

The noise reduction unit is further configured to obtain an output signal based on the enhanced voice signals of the M target users.
The terminal device according to claim 36, wherein the target users include M, the target voice-related data include voice-related data of M target users, and the noise-reduced voice signals of the target users include the Noise-reduced voice signals of M target users, where M is an integer greater than 1;

In terms of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain the target user's noise-reduced speech signal and the interference noise signal, the noise reduction unit specifically Used for:

Perform noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech-related data of the first target user among the M target users, to obtain the noise-reduced speech of the first target user signal and the first noisy speech signal that does not contain the speech signal of the first target user;

performing noise reduction processing on the first noisy speech signal that does not include the speech signal of the first target user through the speech noise reduction model according to the speech-related data of the second target user among the M target users, Obtaining the noise-reduced speech signal of the 2nd target user and the first noisy speech signal that does not include the speech signal of the 1st target user and the speech signal of the 2nd target user;

Repeat the above process until the first noisy speech signal that does not contain the speech signals of the 1st to M-1 target users is subjected to noise reduction processing through the speech noise reduction model according to the speech related data of the Mth target user , to obtain the noise-reduced speech signal of the Mth target user and the interference noise signal.
The terminal device according to claim 36, wherein the target users include M, the target voice-related data include voice-related data of M target users, and the noise-reduced voice signals of the target users include the Noise-reduced voice signals of M target users, where M is an integer greater than 1;

In terms of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain the target user's noise-reduced speech signal and the interference noise signal, the noise reduction unit specifically Used for:

Perform noise reduction processing on the first noisy speech signal through the speech noise reduction model according to the speech-related data of the M target users, to obtain the noise-reduced speech signals of the M target users and the interference noise signal .
The terminal device according to any one of claims 34-37, wherein the target users include M, the relevant data of the target users include the registered voice signals of the target users, the registered voice signals of the target users The speech signal is the speech signal of the target user collected in an environment where the noise decibel value is lower than a preset value, and the speech noise reduction model includes a first coding network, a second coding network, a time convolutional network TCN and a first decoding network ;

In terms of obtaining the noise-reduced speech signal of the target user through the speech noise reduction model for the first noisy speech signal according to the target speech-related data, the noise reduction unit is specifically used for:

Using the first coding network and the second coding network to perform feature extraction on the registration speech signal of the target user and the first noisy speech signal respectively, so as to obtain a feature vector of the registration speech signal of the target user and the feature vector of the first noisy speech signal;

Obtaining a first eigenvector according to the eigenvector of the registration voice signal of the target user and the eigenvector of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

Obtaining the noise-reduced speech signal of the target user according to the first decoding network and the second feature vector.
The terminal device according to claim 41, wherein the noise reduction unit is also used for:

The interference noise signal is also obtained according to the first decoding network and the second eigenvector.
The terminal device according to claim 38, wherein the relevant data of the target user A includes the registered voice signal of the target user A, and the registered voice signal of the target user A is when the noise decibel value is lower than the preset The voice signal of the target user A collected under the setting environment, the voice noise reduction model includes the first encoding network, the second encoding network, TCN and the first decoding network,

In the aspect of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the speech-related data of the target user A, so as to obtain the noise-reduced speech signal of the target user A, the noise reduction The noise unit is specifically used for:

Using the first coding network and the second coding network to perform feature extraction on the registered speech signal of the target user A and the first noisy speech signal respectively, to obtain the features of the registered speech signal of the target user A vector and the feature vector of the first noisy speech signal;

Obtaining a first feature vector according to the feature vector of the registered voice signal of the target user A and the feature vector of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

The noise-reduced speech signal of the target user A is obtained according to the first decoding network and the second feature vector.
The terminal device according to claim 39, wherein the relevant data of the i-th target user among the M target users includes the registered voice signal of the i-th target user, and the i is greater than 0 and less than Or an integer equal to M, the speech noise reduction model includes a first coding network, a second coding network, a TCN and a first decoding network, and the noise reduction unit is specifically used for:

Using the first coding network and the second coding network to perform feature extraction on the registration speech signal of the target user and the first noise signal respectively, to obtain the feature vector and the registration speech signal of the i-th target user. The eigenvector of the first noise signal; wherein, the first noise signal is the first noisy speech signal that does not contain the speech signals of the 1st to i-1 target users;

Obtaining a first feature vector according to the feature vector of the i-th target user's registered voice signal and the feature vector of the first noise signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

According to the first decoding network and the second eigenvector, the noise-reduced speech signal and the second noise signal of the i-th target user are obtained, wherein the second noise signal does not contain the first to i-th targets A first noisy speech signal of the user's speech signal.
The terminal device according to claim 40, wherein, for the voice-related data of the M target users, the relevant data of each target user includes the registered voice signal of the target user, and the registered voice signal of the target user A The signal is the voice signal of the target user A collected in an environment where the noise decibel value is lower than the preset value, and the voice noise reduction model includes M first encoding networks, second encoding networks, TCN, first decoding networks and M a third decoding network,

Perform noise reduction processing on the noisy speech through the speech noise reduction model according to the speech-related data of the M target users, so as to obtain the noise-reduced speech signals of the M target users and the interference noise In terms of signals, the noise reduction unit is specifically used for:

Utilize the M first encoding networks to perform feature extraction on the registered voice signals of the M target users respectively, to obtain feature vectors of the registered voice signals of the M target users; use the second encoding network to extract the features of the first Performing feature extraction on the noisy speech signal to obtain a feature vector of the first noisy speech signal;

Obtaining a first eigenvector according to the eigenvectors of the registration voice signals of the M target users and the eigenvectors of the first noisy voice signal;

obtaining a second eigenvector according to the TCN and the first eigenvector;

According to each of the M third decoding networks, the second feature vector and the feature vector output by the first encoding network corresponding to the third decoding network, the noise-reduced voices of M target users are obtained. Signal;

The interference noise signal is obtained according to the first decoding network, the second feature vector and the feature vector of the first noisy speech signal.
The terminal device according to any one of claims 34-37, wherein the relevant data of the target user includes the target user's voice pickup VPU signal, and the voice noise reduction model includes a preprocessing module, a third An encoding network, a gated recurrent unit GRU, a second decoding network and a post-processing module,

In the aspect of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain the noise-reduced speech signal of the target user, the noise reduction unit specifically uses At:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user by the preprocessing module to obtain the first frequency domain signal of the first noisy speech signal and the VPU signal of the VPU signal. a second frequency domain signal;

merging the first frequency domain signal and the second frequency domain signal to obtain a first fused frequency domain signal;

sequentially processing the first fused frequency domain signal through the third encoding network, the GRU, and the second decoding network to obtain a mask of a third frequency domain signal of the speech signal of the target user;

performing post-processing on the first frequency-domain signal by the post-processing module according to the mask of the third frequency-domain signal, to obtain the third frequency-domain signal;

performing frequency-time transformation on the third frequency domain signal to obtain a noise-reduced voice signal of the target user;

Wherein, both the third encoding module and the second decoding module are implemented based on a convolutional layer and a frequency domain transformation module FTB.
The terminal device according to claim 46, wherein the noise reduction unit is specifically used for:

Processing the first fused frequency domain signal successively through the third encoding network, the GRU, and the second decoding network to obtain a mask of the first frequency domain signal;

performing post-processing on the first frequency-domain signal by the post-processing module according to the mask of the first frequency-domain signal, to obtain a fourth frequency-domain signal of the interference noise signal;

Perform frequency-time transformation on the fourth frequency domain signal to obtain the interference noise signal.
The terminal device according to claim 38, wherein the relevant data of the target user A includes the VPU signal of the target user A, and the speech noise reduction model includes a preprocessing module, a third coding network, a GRU, The second decoding network and the post-processing module, according to the speech-related data of the target user A, through the speech noise reduction model to the first noisy speech signal, to obtain the noise-reduced speech signal of the target user A In one aspect, the noise reduction unit is specifically used for:

Time-frequency transformation is performed on the first noisy speech signal and the VPU signal of the target user A through the preprocessing module to obtain the first frequency domain signal of the first noisy speech signal and the target A ninth frequency domain signal of the VPU signal of user A;

merging the first frequency domain signal and the ninth frequency domain signal to obtain a second fused frequency domain signal;

sequentially processing the second fused frequency domain signal through the third encoding network, the GRU, and the second decoding network to obtain a mask of the tenth frequency domain signal of the voice signal of the target user A;

performing post-processing on the first frequency domain signal by the post-processing module according to the mask of the tenth frequency domain signal, to obtain the tenth frequency domain signal;

performing frequency-time transformation on the tenth frequency domain signal to obtain the noise-reduced speech signal of the target user A;

Wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.
The terminal device according to claim 39, wherein the relevant data of the i-th target user among the M target users includes the VPU signal of the i-th target user, and the i is greater than 0 and less than or is equal to an integer of M, and the noise reduction unit is specifically used for:

Time-frequency transformation is performed on the first noise signal and the VPU signal of the ith target user by the preprocessing module to obtain the eleventh frequency domain signal of the first noise signal and the ith target user The twelfth frequency domain signal of the VPU signal;

merging the eleventh frequency domain signal and the twelfth frequency domain signal to obtain a third fused frequency domain signal; wherein the first noise signal does not include the 1st to i-1 target users a noisy speech signal of the speech signal;

Processing the third fused frequency domain signal successively through the third coding network, the GRU, and the second decoding network to obtain a mask of the thirteenth frequency domain signal of the voice signal of the i-th target user and a mask of the eleventh frequency domain signal;

The post-processing module performs post-processing on the eleventh frequency domain signal according to the mask of the thirteenth frequency domain signal and the mask of the eleventh frequency domain signal to obtain the thirteenth frequency domain signal. domain signal and a fourteenth frequency domain signal of the second noise signal;

performing frequency-time transformation on the thirteenth frequency domain signal and the fourteenth frequency domain signal to obtain the noise-reduced voice signal of the ith target user and the second noise signal, the second noise signal be the first noisy speech signal that does not contain the speech signals of the 1st to i target users;

Wherein, both the third encoding module and the second decoding module are implemented based on convolutional layers and FTB.
The terminal device according to any one of claims 39, 40, 44, 45, and 49, wherein the noise-reduced speech signal of the target user is enhanced during the speech enhancement coefficient based on the target user , to obtain the aspect of the enhanced speech signal of the target user, the noise reduction unit is specifically used for:

For any target user A among the M target users, the noise-reduced speech signal of the target user A is enhanced based on the speech enhancement coefficient of the target user A, so as to obtain the enhanced speech of the target user A signal; the ratio of the amplitude of the enhanced voice signal of the target user A to the amplitude of the noise-reduced voice signal of the target user A is the voice enhancement coefficient of the target user A;

In the aspect of fusing the interference noise suppression signal with the enhanced speech signal of the target user to obtain an output signal, the noise reduction unit is specifically used for:

The enhanced speech signals of the M target users are fused with the interference noise suppression signal to obtain the output signal.
The terminal device according to any one of claims 34-37, wherein the relevant data of the target user includes the VPU signal of the target user, and the acquisition unit is further configured to: acquire the target user's ear Internal sound signal;

In the aspect of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the target speech-related data to obtain the noise-reduced speech signal of the target user, the noise reduction unit specifically uses At:

respectively performing time-frequency transformation on the first noisy speech signal and the in-ear sound signal to obtain a first frequency-domain signal of the first noisy speech signal and a fifth frequency-domain signal of the in-ear sound signal Signal;

obtaining the covariance matrix of the first noisy speech signal and the in-ear sound signal according to the target user's VPU signal, the first frequency domain signal, and the fifth frequency domain signal;

Obtaining the first minimum variance distortion-free response MVDR weight based on the covariance matrix;

Obtaining a sixth frequency domain signal of the first noisy speech signal and a seventh frequency domain signal of the in-ear sound signal based on the first MVDR weight, the first frequency domain signal and the fifth frequency domain signal Signal;

obtaining an eighth frequency domain signal of the noise-reduced speech signal according to the sixth frequency domain signal and the seventh frequency domain signal;

performing frequency-time transformation on the eighth frequency domain signal to obtain the noise-reduced speech signal of the target user.
The terminal device according to claim 51, wherein the noise reduction unit is also used for:

The interference noise signal is obtained for the first noisy speech signal according to the noise-reduced speech signal of the target user.
The terminal device according to claim 38, wherein the relevant data of the target user A includes the VPU signal of the target user A, and the acquisition unit is further configured to acquire the in-ear sound of the target user A Signal;

In the aspect of performing noise reduction processing on the first noisy speech signal through a speech noise reduction model according to the speech-related data of the target user A to obtain the noise-reduced speech signal of the target user A, the noise reduction unit Specifically for:

performing time-frequency transformation on the first noisy speech signal and the target user A's ear sound signal respectively, to obtain the first frequency domain signal of the first noisy speech signal and the target user A's ear sound signal a fifteenth frequency domain signal of the sound signal;

According to the VPU signal of the target user A, the first frequency domain signal and the fifteenth frequency domain signal, the covariance matrix of the first noisy speech signal and the in-ear sound signal of the target user A is obtained ;

obtaining a second MVDR weight based on the covariance matrix;

Obtaining the sixteenth frequency domain signal of the first noisy speech signal and the in-ear sound of the target user A based on the second MVDR weight, the first frequency domain signal and the fifteenth frequency domain signal The seventeenth frequency domain signal of the signal; the eighteenth frequency domain signal of the noise-reduced voice signal of the target user A is obtained according to the sixteenth frequency domain signal and the seventeenth frequency domain signal;

Perform frequency-time transformation on the eighteenth frequency domain signal to obtain the noise-reduced speech signal of the target user A.
The terminal device according to any one of claims 41-45, wherein the acquiring unit is further configured to:

Obtaining a first noise segment and a second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time; obtaining the signal-to-noise ratio SNR and Sound pressure level SPL;

The terminal equipment also includes:

A determination unit, configured to extract a first temporary feature vector of the first noise segment if the SNR of the first noise segment is greater than a first threshold and the SPL of the first noise segment is greater than a second threshold; based on the The first temporary speech feature vector performs noise reduction processing on the second noise segment to obtain a second noise reduction noise segment; and performs damage assessment based on the second noise reduction noise segment and the second noise segment to obtain a second noise reduction segment. a damage score; if the first damage score is not greater than a third threshold, enter the PNR mode;

In the aspect of obtaining the first noisy speech signal, the obtaining unit is specifically used for:

The first noisy speech signal is determined from a noise signal generated after the first noise segment; the feature vector of the registration speech signal includes the first temporary feature vector.
The terminal device according to claim 54, wherein if the first damage score is not greater than a third threshold, the determining unit is further configured to:

Sending first prompt information through the terminal device, where the first prompt information is used to prompt whether to enable the terminal device to enter the PNR mode;

The PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.
The terminal device according to claim 54 or 55, characterized in that,

The obtaining unit is further configured to obtain a second noisy speech signal when it is detected that the terminal device is used again;

The noise reduction unit is further configured to: when the SNR of the second noisy speech signal is lower than a fourth threshold, perform noise reduction processing on the second noisy speech signal according to the first temporary feature vector, to obtain the noise-reduced voice signal of the current user;

The determining unit is further configured to perform damage assessment based on the noise-reduced speech signal of the current user and the second noisy speech signal to obtain a second damage score; when the second damage score is not greater than the fifth threshold, send the second prompt information through the terminal device, the second prompt information is used to prompt the current user that the terminal device can enter the PNR mode; when it is detected that the current user agrees to enter the After the operation command of the PNR mode, the terminal device enters the PNR mode to perform noise reduction processing on the third noisy speech signal, and the third noisy speech signal is obtained after the second noisy speech signal; After detecting that the current user does not agree to enter the operation instruction of the PNR mode, the non-PNR mode is used to perform noise reduction processing on the third noisy speech signal.
The terminal device according to claim 54 or 55, characterized in that,

The acquisition unit is further configured to: if the SNR of the first noise segment is not greater than the first threshold or the SPL of the first noise segment is not greater than the second threshold, and the terminal device has stored a reference temporary A voiceprint feature vector to obtain a third noise segment;

The noise reduction unit is further configured to perform noise reduction processing on the third noise segment according to the reference temporary voiceprint feature vector to obtain a third noise reduction noise segment;

The determining unit is further configured to perform damage assessment according to the third noise segment and the third noise-reduced noise segment to obtain a third damage score; if the third damage score is greater than a sixth threshold and the first The SNR of the three noise segments is less than the seventh threshold, or the third impairment score is greater than the eighth threshold and the SNR of the third noise segment is not less than the seventh threshold, then send the third prompt through the terminal device information, the third prompt information is used to prompt the current user that the terminal device can enter the PNR mode; after detecting the current user agrees to enter the operation instruction of the PNR mode, the terminal device enters the PNR mode Perform noise reduction processing on the fourth noisy speech signal; after detecting the current user's disagreement to enter the PNR mode operation instruction, use the non-PNR mode to perform noise reduction processing on the fourth noisy speech signal ; wherein the fourth noisy speech signal is determined from a noise signal generated after the third noise segment.
The terminal device according to any one of claims 41-45, characterized in that,

The acquiring unit is further configured to acquire a first noise segment and a second noise segment of the environment where the terminal device is located; the first noise segment and the second noise segment are continuous noise segments in time; the acquisition of the signals collected by the microphone array of the auxiliary device of the terminal device for the environment in which the terminal device is located;

The terminal equipment also includes:

A determining unit, configured to use the collected signal to calculate the signal arrival angle DOA and SPL of the first noise segment; if the DOA of the first noise segment is greater than the ninth threshold and less than the tenth threshold, and the first noise segment If the SPL of a noise segment is greater than the eleventh threshold, the second temporary feature vector of the first noise segment is extracted, and the noise reduction process is performed on the second noise segment based on the second temporary feature vector to obtain the third Noise reduction noise segment; perform damage assessment based on the third noise reduction noise segment and the second noise segment to obtain a fourth damage score; if the fourth damage score is greater than a twelfth threshold, enter the PNR model;

In the aspect of obtaining the first noisy speech signal, the obtaining unit is specifically used for:

determining said first noisy speech signal from a noise signal generated after said first noise segment;

The feature vector of the enrollment speech signal includes the second temporary feature vector.
The terminal device according to claim 58, wherein if the fourth damage score is not greater than the twelfth threshold, the determining unit is further configured to:

Sending fourth prompt information through the terminal device, where the fourth prompt information is used to prompt whether to make the terminal device enter the PNR mode;

The PNR mode is entered only after an operation instruction of the target user agreeing to enter the PNR mode is detected.
The terminal device according to any one of claims 34-53, wherein the terminal device further comprises:

A detection unit, configured to not enter the PNR mode when it is detected that the terminal device is in a hand-held talking state;

Entering the PNR mode when it is detected that the terminal device is in a hands-free call state, wherein the target user is the owner of the terminal device or the user who is using the terminal device;

When it is detected that the terminal device is in a video call state, enter the PNR mode, wherein the target user is the owner of the terminal device or the user closest to the terminal device;

When it is detected that the terminal device is connected to the earphone for conversation, enter the PNR mode, wherein the target user is a user wearing the earphone; the first noisy voice signal and the target voice-related data are collected by said headset; or

When it is detected that the terminal device is connected to a smart large-screen device, a smart watch or a vehicle-mounted device, enter the PNR mode, wherein the target user is the owner of the terminal device or the user who is using the terminal device, The first noisy voice signal and target voice-related data are collected by audio collection hardware of the smart large-screen device, the smart watch, or the vehicle-mounted device.
The terminal device according to any one of claims 34-53, wherein the acquisition unit is further configured to: acquire the decibel value of the audio signal of the current environment,

The terminal equipment also includes:

The control unit is configured to enable the PNR function corresponding to the application program started by the terminal device if the decibel value of the audio signal in the current environment exceeds the preset decibel value and the PNR function corresponding to the application program started by the terminal device is not enabled. PNR function and enter the PNR mode.
The terminal device according to any one of claims 34-53, wherein the terminal device includes a display screen, and the display screen includes a plurality of display areas,

Wherein, each display area of the plurality of display areas displays a label and a corresponding function key, and the function key is used to control the opening and closing of the PNR function of the application program indicated by the corresponding label.
The terminal device according to any one of claims 34-53, wherein when voice data transmission is performed between the terminal device and another terminal device, the terminal device further includes:

a receiving unit, configured to receive a voice enhancement request sent by the other terminal device, where the voice enhancement request is used to instruct the terminal device to enable the PNR function of the call function;

A control unit, configured to send third prompt information through the terminal device in response to the voice enhancement request, where the third prompt information is used to prompt whether to enable the terminal device to enable the PNR function of the call function; when detecting After the target user confirms that the PNR function of the call function is turned on for the terminal device, the PNR function of the call function is turned on, and enters the PNR mode;

A sending unit, configured to send a voice enhancement response message to the other terminal device, where the voice enhancement response message is used to indicate that the terminal device has enabled the PNR function of the call function.
The terminal device according to any one of claims 38-40, 43-45 and 50, wherein when the terminal device starts the video call or video recording function, the display interface of the terminal device includes the first area and In the second area, the first area is used to display video call content or video recording content, and the second area is used to display M controls and corresponding M labels, and the M controls are related to the M targets User one-to-one correspondence, each control in the M controls includes a sliding button and a sliding bar, by controlling the sliding button to slide on the sliding bar, to adjust the voice enhancement of the target user indicated by the label corresponding to the control coefficient.
The terminal device according to any one of claims 38-40, 43-45 and 50, wherein when the terminal device starts a video call or video recording function, the display interface of the terminal device includes a first area, The first area is used to display video call content or video recording content; the terminal device also includes:

A control unit, configured to display a control corresponding to the object in the first area when an operation on any object in the video call content or video recording content is detected, the control includes a sliding button and a sliding bar, and is controlled by The slide button slides on the slide bar to adjust the speech enhancement coefficient of the object.
The terminal device according to any one of claims 34-37 and 41, wherein when the terminal device is an intelligent interactive device, the target voice-related data includes a voice signal containing a wake-up word, and the first The noisy speech signal includes an audio signal containing command words.
A terminal device, characterized by comprising a processor and a memory, wherein the processor is connected to the memory, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute The method of any one of claims 1-33.
A chip system, characterized in that the chip system is applied to electronic equipment; the chip system includes one or more interface circuits, and one or more processors; the interface circuits and the processors are interconnected through lines The interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer When instructed, the electronic device executes the method according to any one of claims 1-33.
A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1-33.