WO2026012012A1 - 语音增强方法与相关方法、装置、设备、介质和程序产品 - Google Patents

语音增强方法与相关方法、装置、设备、介质和程序产品

Info

Publication number
WO2026012012A1
WO2026012012A1 PCT/CN2025/098737 CN2025098737W WO2026012012A1 WO 2026012012 A1 WO2026012012 A1 WO 2026012012A1 CN 2025098737 W CN2025098737 W CN 2025098737W WO 2026012012 A1 WO2026012012 A1 WO 2026012012A1
Authority
WO
WIPO (PCT)
Prior art keywords
voiceprint
speech
feature
features
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/098737
Other languages
English (en)
French (fr)
Inventor
朱唯鑫
饶为
王燕南
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2026012012A1 publication Critical patent/WO2026012012A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00—Speaker identification or verification techniques
    • G10L17/02—Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/022—Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/45—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of analysis window
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination

Definitions

  • This application relates to the field of speech processing technology, and more specifically, to speech enhancement.
  • Speech technology is a collective term for a series of technologies that utilize computers to analyze, process, and understand human speech. Its core includes speech recognition, speech synthesis, speech enhancement, and speech analysis. These technologies are widely used in daily life and various industries, significantly improving human-computer interaction. For example, in communication scenarios, speech enhancement is used to eliminate background noise and improve call quality.
  • speech enhancement technology has made significant progress.
  • Existing speech enhancement systems can process speech signals in real time in general acoustic environments and achieve speech enhancement functions in multi-voice, multi-device application scenarios.
  • These speech enhancement technologies not only improve personal communication and entertainment experiences but also provide effective solutions for audio processing in professional fields (audio and video calls, live streaming, and interviews, etc.).
  • This application provides a speech enhancement method, related methods, apparatus, devices, and storage media. These can improve the stability and accuracy of speech enhancement.
  • a speech enhancement method comprising: in a voice call state, acquiring reference speech of a target object, and extracting reference audio features of each frame of speech signal corresponding to the reference speech; performing voiceprint extraction on each reference audio feature based on a voiceprint extraction network to obtain frame-level voiceprint features corresponding to each reference audio feature; performing voiceprint clustering based on each frame-level voiceprint feature to obtain target voiceprint features corresponding to the target object; inputting the acquired audio features to be enhanced and the target voiceprint features of the speech to be enhanced into a speech enhancement network, outputting target enhanced audio features, and determining the target enhanced speech corresponding to the target enhanced audio features; wherein, the voiceprint extraction network and the speech enhancement network are trained based on a target loss determined by predicted enhanced audio features and clean audio features, the predicted enhanced audio features are obtained by the speech enhancement network through speech enhancement of the predicted voiceprint features corresponding to sample noisy speech and sample reference speech, and the sample noisy speech is obtained by speech mixing of sample clean speech matched with the voiceprint
  • a method for training a speech enhancement network includes: acquiring training samples, the training samples including clean speech samples, reference speech samples, and noisy speech samples, wherein the noisy speech samples are obtained by mixing environmental noise speech and the clean speech samples, and the clean speech samples and the reference speech samples in the training samples originate from the same sample object; extracting voiceprints from the reference audio features corresponding to the reference speech samples based on an initial voiceprint extraction network to obtain predicted voiceprint features; extracting features from the noisy speech samples to obtain corresponding noisy audio features; inputting the noisy audio features and the predicted voiceprint features into an initial speech enhancement network for speech enhancement, and outputting predicted enhanced audio features corresponding to the sample object; determining a target loss based on the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples; and iteratively training the initial voiceprint extraction network and the initial speech enhancement network based on the target loss until the training result conditions are met, thereby obtaining the voiceprint extraction network and the speech enhancement network.
  • a speech enhancement device comprising: a speech acquisition module, configured to acquire reference speech of a target object during a voice call, and extract reference audio features of each frame of speech signal corresponding to the reference speech; a voiceprint extraction module, configured to perform voiceprint extraction on each reference audio feature based on a voiceprint extraction network to obtain frame-level voiceprint features corresponding to each reference audio feature; a voiceprint clustering module, configured to perform voiceprint clustering based on each frame-level voiceprint feature to obtain target voiceprint features corresponding to the target object; and a speech enhancement module, configured to input the acquired audio features to be enhanced and the target voiceprint features into a speech enhancement network, output target enhanced audio features, and determine the target enhanced speech corresponding to the target enhanced audio features; wherein the voiceprint extraction network and the speech enhancement network are trained based on a target loss determined by predicted enhanced audio features and clean audio features, the predicted enhanced audio features are obtained by the speech enhancement network through speech enhancement of the predicted voiceprint features corresponding to sample noisy speech and sample reference speech,
  • a speech enhancement network training apparatus includes: a sample acquisition module for acquiring training samples, the training samples including clean sample speech, sample reference speech, and sample noisy sample speech, wherein the noisy sample speech is obtained by mixing environmental noise speech and the clean sample speech, and the clean sample speech and sample reference speech in the training samples originate from the same sample object; a feature prediction module for performing voiceprint extraction on sample reference audio features corresponding to the sample reference speech based on an initial voiceprint extraction network to obtain predicted voiceprint features; an audio extraction module for performing feature extraction on the sample noisy speech to obtain corresponding sample noisy frequency features; an audio prediction module for inputting the sample noisy frequency features and the predicted voiceprint features into an initial speech enhancement network for speech enhancement, and outputting predicted enhanced audio features corresponding to the sample object; and a network training module for determining a target loss based on the predicted enhanced audio features and the clean audio features corresponding to the clean sample speech, and iteratively training the initial voiceprint extraction network and the initial speech enhancement network based on the target loss until the training result conditions
  • a computer-readable storage medium that stores a computer program, wherein the above-described speech enhancement method and related methods are executed when the computer program is run by a processor.
  • a computer device including a processor and a memory, the memory storing a computer program, which, when invoked by the processor, executes the above-described speech enhancement method and related methods.
  • a computer program product which includes a computer program stored in a storage medium; a processor of a computer device reads the computer program from the storage medium and executes the computer program or instructions to cause the computer device to perform the above-described speech enhancement method and related methods.
  • the target voiceprint features corresponding to the target object can be quickly obtained.
  • the acquired audio features to be enhanced and the target voiceprint features of the speech to be enhanced are then input into the speech enhancement network, outputting the target enhanced audio features. Further, the target enhanced speech corresponding to the target enhanced audio features is determined. Therefore, compared to related technologies that utilize sensor-acquired reference speech for speech enhancement, this application can utilize the non-sensory, efficient, and stable extracted target voiceprint features to represent the unique speech characteristics of the target object. This allows for faster removal of other sound information unrelated to the unique speech characteristics of the target object from the audio features to be enhanced, including interfering human voice information and noise information, thereby improving the speed, stability, and accuracy of speech enhancement.
  • Figure 1 shows a schematic diagram of a system architecture provided in an embodiment of this application
  • Figure 2 shows a schematic diagram of a system deployment provided in an embodiment of this application
  • Figure 3 illustrates an application scenario of a speech enhancement method provided in an embodiment of this application.
  • Figure 4 illustrates an application scenario of another speech enhancement method provided in an embodiment of this application.
  • FIG. 5 shows a flowchart of a speech enhancement method provided in an embodiment of this application
  • Figure 6 shows a flowchart of a target voiceprint feature generation method provided in an embodiment of this application
  • FIG. 7 shows a flowchart of an audio feature extraction method provided in an embodiment of this application.
  • Figure 8 shows a flowchart of a speech enhancement method provided in an embodiment of this application.
  • Figure 9 shows a flowchart of a speech enhancement network training method provided in an embodiment of this application.
  • Figure 10 shows an architecture diagram of a speech enhancement network training method provided in an embodiment of this application
  • Figure 11 shows a framework diagram of a voiceprint extraction network provided in an embodiment of this application.
  • Figure 12 is a block diagram of a speech enhancement device provided in an embodiment of this application.
  • Figure 13 is a block diagram of a speech enhancement network training device provided in an embodiment of this application.
  • Figure 14 is a block diagram of a computer device provided in an embodiment of this application.
  • Figure 15 is a block diagram of a computer-readable storage medium provided in an embodiment of this application.
  • the speech enhancement method proposed in this application involves artificial intelligence (AI) technology.
  • Speech enhancement aims to improve the quality of speech signals, making them clearer and more intelligible in noisy and interfering environments.
  • These techniques encompass various processing methods, such as noise suppression, echo cancellation, and speech declarativeness.
  • Noise suppression uses algorithms to reduce the impact of environmental noise, resulting in a cleaner speech signal.
  • Echo cancellation focuses on removing echoes caused by reflections between devices or from the environment, ensuring natural and clear communication.
  • Speech declarativeness enhances the clarity and intelligibility of speech signals while preserving their naturalness.
  • Speech enhancement technologies play a crucial role in numerous fields. For example, the speech enhancement method proposed in this application can effectively eliminate background noise and interfering human voices, significantly improving speech quality.
  • Speech enhancement techniques can be categorized into several types based on their implementation and processing methods. These primarily include speech spectrum estimation methods based on spectral subtraction, noise estimation methods based on Gaussian Mixture Models (GMMs), and noise removal methods based on Deep Neural Networks (DNNs).
  • GMMs Gaussian Mixture Models
  • DNNs Deep Neural Networks
  • Spectral subtraction is a classic speech enhancement technique that improves speech signal quality by reducing the noise spectrum. Its basic principle is to subtract the noise spectrum from the speech spectrum to obtain the purified speech signal.
  • the noise spectrum is typically estimated based on noise samples collected during periods of silence (when there is no speech activity). While spectral subtraction is simple and computationally efficient, its effectiveness is limited when dealing with complex or non-stationary noise, and it is prone to introducing noise artifacts, making the enhanced speech sound unnatural.
  • GMMs Gaussian mixture models
  • speech enhancement GMMs estimate background noise and spectral subtraction coefficients, perform spectral subtraction on noisy speech, and recover clean speech.
  • the noisy speech is preprocessed to obtain its amplitude and phase; the amplitude is used for noise estimation and spectral subtraction, and the phase is used to recover the time-domain signal.
  • the GMM is used to estimate noise parameters and cepstral features of the clean speech in real time, and the spectral subtraction coefficient is calculated based on the estimated cepstral features.
  • Deep neural networks trained on large amounts of training data can learn complex features of speech and noise, thus achieving highly nonlinear mappings during the enhancement process.
  • Common deep neural network architectures include convolutional neural networks, recurrent neural networks, and self-attention models (e.g., Transformer). These networks can extract clear speech signals in noisy environments and handle various types of noise.
  • deep neural networks offer greater flexibility and adaptability, providing effective speech enhancement in different scenarios.
  • this approach often fails to effectively suppress background noise interference with human voices.
  • the technology still suffers from unsatisfactory speech enhancement results.
  • each training sample may include a sample reference speech, a sample clean speech, and a sample noisy speech.
  • the sample reference speech and sample clean speech in the same training sample originate from the same speaker.
  • the data acquisition device 110 can store the training data in a database 120.
  • the training device 130 can perform network training on a preset neural network based on the training data in the database 120 until the preset neural network meets the training termination condition, thus obtaining the trained target network 101.
  • the training termination condition can be: the total loss value of the target loss is less than a preset value, the total loss value of the target loss is within a threshold range, or the training iterations reach a preset number, etc.
  • the target network 101 can be a deep neural network or a network model composed of multiple neural networks. This target network 101 is also known as a voiceprint extraction network for extracting voiceprint features or a speech enhancement network for performing speech enhancement.
  • the voiceprint extraction network or the speech enhancement network can be a network framework composed of a Long Short-Term Memory (LSTM) network and a fully connected layer (FC), which is not limited here.
  • LSTM Long Short-Term Memory
  • FC fully connected layer
  • the training device 130 and execution device 140 mentioned above can be computer devices such as servers or terminals.
  • the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), blockchain, and big data and artificial intelligence platforms.
  • the terminal can be a smart mobile device, laptop, in-vehicle computer, desktop computer, etc.
  • the processing module 141 of the execution device 140 is used to perform calculations and other related processing.
  • the execution device 140 can call data, programs, etc. in the data storage system 160 for corresponding calculation processing, and store the processing results and instructions obtained from the calculation processing into the data storage system 150.
  • the execution device 140 can use the target network 101 (i.e., the trained speech enhancement network) to enhance the speech of the target object's conversation to obtain enhanced speech, and store the enhanced speech in the data storage system 160.
  • the client device 150 can obtain the enhanced speech from the data storage system 160 through the input/output (I/O) interface of the execution device 140 and play the enhanced speech for the user.
  • I/O input/output
  • the training data maintained in database 120 may not all come from data acquisition device 110; it may also be received from other devices.
  • client device 150 may also act as a data acquisition end, and the user's voice data acquired by client device 150, after relevant data processing, may be used as new training data and stored in database 120.
  • training device 130 may not necessarily train the preset neural network entirely based on the training data maintained in database 120; it may also train the preset neural network based on training data obtained from the cloud or other devices.
  • training device 130 may obtain voice datasets related to voice processing technology from an internet platform. The above description should not be construed as limiting the embodiments of this application.
  • Figure 1 is merely a schematic diagram of a system architecture provided in an embodiment of this application.
  • the system architecture and application scenarios described in this embodiment are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this embodiment.
  • the data storage system 160 in Figure 1 is an external memory relative to the execution device 150.
  • the data storage system 160 can also be placed inside the execution device 150.
  • the client device 150 can also be directly used as an execution device to deploy the trained target network for performing speech enhancement tasks.
  • FIG 2 illustrates a system deployment diagram.
  • the voice enhancement method and related methods provided in this application embodiment can also be deployed in the system shown in Figure 2.
  • This system includes a terminal 240, an internet connection 230, a gateway 220, a server 210, etc.
  • Terminal 240 may take various forms, including desktop computers, laptops, personal digital assistants (PDAs), smartphones, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. Terminal 240 can communicate with the Internet 230 via wired or wireless means to exchange data. For example, terminal 240 can communicate with the Internet 230 through a wireless router 250.
  • PDAs personal digital assistants
  • Terminal 240 can communicate with the Internet 230 through a wireless router 250.
  • Server 210 refers to a computer system capable of providing certain services to terminal 240. Compared to ordinary terminal 240, server 210 has higher requirements in terms of stability, security, and performance. Server 210 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).
  • Gateway 220 also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 240 to server 210 can be transmitted to the corresponding server 210 through gateway 220. Messages sent from server 210 to terminal 240 can also be transmitted to the corresponding terminal 240 through gateway 220.
  • the speech enhancement method and related methods in the embodiments of this application can be implemented entirely on the terminal 240, entirely on the server 210, or partially on the terminal 240 and partially on the server 210.
  • server 210 can serve as a training device to train a speaker extraction network and a speech enhancement network based on a training set.
  • Server 210 can also serve as an execution device to perform language enhancement on the speech to be enhanced using the speaker extraction network and the speech enhancement network, obtaining target enhanced speech for the target object.
  • server 210 can receive reference speech and speech to be enhanced from terminal 240. Then, server 210 can use the speaker extraction network to extract the speakerprint from the reference speech of the target object, obtaining corresponding speakerprint features. Further, the speech enhancement network performs language enhancement on the speech to be enhanced based on these speakerprint features, obtaining the target enhanced speech, and then returns the target enhanced speech to terminal 240.
  • the speech enhancement method and related methods can be implemented partly on the terminal 240 and partly on the server 210.
  • the server 210 can act as a training device to train a speaker extraction network and a speech enhancement network based on a training set.
  • the terminal 240 can act as an execution device to locally deploy the speaker extraction network and speech enhancement network trained by the server 210, and then use these networks to perform language enhancement on the speech to be enhanced, obtaining targeted enhanced speech for the target object.
  • Figure 2 is merely a schematic diagram of a system deployment provided in an embodiment of this application.
  • the system deployment scheme described in this embodiment is only for the purpose of more clearly illustrating the technical solution of this embodiment and does not constitute a limitation on the technical solution provided in this embodiment.
  • terminal 240 can refer to one of multiple terminals, and this embodiment only uses the three terminals shown in Figure 2 as an example.
  • the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.
  • Online conferencing is a remote conferencing method based on cloud computing technology. It connects multiple geographically dispersed participants via the internet, enabling real-time audio and video communication, file sharing, and collaboration. Cloud conferencing systems do not require expensive hardware or complex installation processes; participants can easily start and join meetings simply by installing the corresponding application on a device with an internet connection or by accessing it through a browser.
  • audio and video data are compressed and encrypted before being transmitted to each participant's device via a cloud server.
  • the efficient processing power and large-scale storage support of cloud computing enable cloud conferencing to support simultaneous communication among a large number of participants while ensuring high-quality audio and video transmission.
  • the speech enhancement method and related methods provided in this application can be applied to online conferencing scenarios to enhance the speech information in the meeting.
  • the online meeting service provided by the meeting service provider can be applied to server 210, terminal 241, and terminal 242.
  • Server 210 can include multiple high-performance cloud servers, such as training cloud servers, processing cloud servers, and client cloud servers.
  • the training cloud server can be used to train the speaker extraction network and speech enhancement network based on the training set.
  • the processing cloud server can be used to deploy the trained speaker extraction network and speech enhancement network to perform speech enhancement tasks.
  • the client cloud server can directly communicate with multiple terminals, such as terminal 241 and terminal 242.
  • the customer cloud server can receive account login requests and voice enhancement requests from terminals, and can also perform operations such as message distribution, such as returning the target enhanced voice obtained from the cloud server to the corresponding terminal.
  • Terminals 241 and 242 can install a conference client provided by the conference service provider, which can realize functions such as audio and video communication and screen sharing. When users use the conference client for voice communication, the terminal where the speaker is located can transmit the collected target voice to the server, which will then enhance the target voice and send the enhanced target voice to other participating terminals.
  • terminal 242 can automatically capture a segment of speech as reference speech and send it to server 210.
  • Server 210 uses a voiceprint extraction network to extract the voiceprint from the reference speech, obtaining the corresponding target voiceprint features.
  • terminal 242 can capture participant 260's speech in real time, i.e., the speech to be enhanced, and transmit it to server 210.
  • Server 210 can then use a voice enhancement network to enhance the speech based on the target voiceprint features, obtaining the enhanced target speech.
  • This enhanced target speech is then distributed to the terminals of other participants, such as terminal 241, allowing other participants to hear participant 260's speech after the voice enhancement.
  • Game voice chat plays a crucial role in modern multiplayer games, serving as a core tool for real-time communication and collaboration between players.
  • fast and clear voice communication directly impacts the gaming experience and player performance.
  • players can instantly share information, formulate tactics, coordinate actions, and even convey emotions and atmosphere, thereby improving the efficiency of teamwork and strategic execution.
  • technologies such as background noise cancellation and echo suppression makes the game voice environment clearer and purer, avoiding external interference.
  • the voice enhancement method and related methods provided in this application can be applied to game voice scenarios, providing voice enhancement services for voice communication between game players.
  • both game terminals 241 and 242 are equipped with game software, which includes a voice communication function developed based on the voice enhancement method and related methods of this application.
  • game software which includes a voice communication function developed based on the voice enhancement method and related methods of this application.
  • the player's terminal can collect the player's voice and enhance it locally before sending the enhanced voice to other players' terminals.
  • terminal 242 since terminal 242 is located in an online entertainment environment, the player's voice collected by terminal 242 often contains interference from other players. Therefore, when sending the player's voice to other players, terminal 242 performs voice enhancement locally on the collected player's voice.
  • terminal 242 can automatically collect a segment of voice as a reference voice and perform voiceprint extraction on the reference voice using a voiceprint extraction network to obtain the corresponding target voiceprint features. Furthermore, terminal 242 can collect the voice of the player speaking in real time, that is, the voice to be enhanced, and perform voice enhancement on the voice to be enhanced based on the target voiceprint features through the voice enhancement network to obtain the enhanced target voice, and distribute the enhanced target voice to the terminals of other game players.
  • Figures 3 and 4 are merely schematic diagrams illustrating two application scenarios provided in the embodiments of this application.
  • the application scenarios described in the embodiments of this application are only for more clearly illustrating the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application.
  • the speech enhancement method and related methods proposed in this application can also be used in other call scenarios, such as call noise reduction, speech recognition front-end, live streaming and video-on-demand applications, etc. Specifically:
  • a speech enhancement method is provided.
  • This speech enhancement method can be executed by a computer device (e.g., a server, edge computing device, or other terminal device with certain processing capabilities), which at least has storage, computing, and communication functions.
  • Figure 5 shows a flowchart of the speech enhancement method provided in this embodiment. As shown in Figure 5, the speech enhancement method specifically includes:
  • S110 In the state of voice call, acquire the reference speech of the target object, and extract the reference audio features of each frame of speech signal corresponding to the reference speech.
  • speech enhancement methods rely on signal processing or deep learning models. However, these methods often fail to effectively distinguish the voice features of different speakers in multi-speaker scenarios, potentially resulting in the enhanced speech containing the voices of other interfering speakers. Considering that each person's voiceprint features contain unique information about their voice, and that different people have unique voiceprint features, this uniqueness can be used to differentiate between different voices. Therefore, this application proposes speech enhancement based on non-intrusive reference speech, utilizing the voiceprint features extracted from the reference speech to achieve more accurate speech enhancement.
  • This application employs a seamless acquisition of reference speech because pre-recording reference speech places stringent requirements on the recording environment and is cumbersome. Furthermore, if the speaker corresponding to the speech to be enhanced changes, the speech enhancement will be ineffective, requiring re-recording of the reference speech, which inconveniences users during real-time calls.
  • the target object refers to the speaker whose voice is being collected.
  • the state of the voice call refers to the state in which the target object is engaged in a voice call with other objects within the call scenario.
  • the reference speech refers to the speech for which voiceprint feature extraction is to be performed; the duration of the reference speech can be preset.
  • a 20-second segment of the target's voice can be automatically collected as reference voice. Since the original voice signal is a complex, continuously changing time-domain signal, directly extracting voiceprint features from it often fails to reflect the actual content of the voice. Therefore, it is necessary to perform voice processing steps such as framing, windowing, and time-frequency conversion on the reference voice to ensure that representative multi-frame audio features corresponding to the frequency domain are extracted from the multi-frame time-domain signal of the reference voice—that is, the reference audio features—for subsequent voiceprint extraction.
  • speech processing may include steps such as framing, windowing, and time-frequency conversion on the speech signal of the reference speech, thereby capturing and analyzing useful reference audio features from the reference speech for speaker feature extraction. Specifically, this may include:
  • a sound quality test can be performed on the acquired reference speech. If the test result indicates that the reference speech meets the conditions for generating subsequent reference audio features, then subsequent speech processing is performed. Otherwise, the reference speech is re-acquired until it passes the sound quality test.
  • sound quality detection can be a signal-to-noise ratio (SNR) calculation of a reference speech signal.
  • SNR measures the relative strength of the speech signal and the noise signal.
  • signal analysis can be performed on the speech signal of the reference speech to obtain the noise signal within the speech signal.
  • the SNR of the speech signal and the noise signal can be calculated based on power or amplitude.
  • the calculation formula can be as follows:
  • SNR ⁇ sub> p ⁇ /sub> represents the signal-to-noise ratio calculated based on signal power.
  • P ⁇ sub> signal ⁇ /sub> represents the signal power corresponding to the speech signal
  • P ⁇ sub>noise ⁇ /sub> represents the signal power corresponding to the noise signal.
  • the signal-to-noise ratio can also be calculated using the amplitudes of the speech and noise signals, as shown in the following formula:
  • SNRm represents the signal-to-noise ratio calculated based on the signal amplitude.
  • M signal represents the mean square amplitude of the speech signal, and
  • P noise represents the mean square amplitude of the noise signal.
  • SNR dB 10log(SNR p or SNR m ).
  • the reference speech is divided into frames to obtain multiple speech frame signals.
  • the signal-to-noise ratio (SNR) threshold is a key value used to decide whether to perform the next step of speech processing or to reacquire the reference speech.
  • the SNR threshold can be determined based on SNR experiments conducted for specific use cases and the experimental values.
  • Framing refers to dividing the speech signal of the reference speech into several small segments, each called a frame.
  • a speech signal is a continuous time waveform, but its characteristics change over time.
  • long-duration speech signals can be divided into several frames, and the signal within each frame can be considered short-term stationary.
  • the reference speech can be re-acquired.
  • windowing can include Hamming windows, Hanning windows, and Blackman windows.
  • Windowing is to gradually attenuate the signal amplitude at the boundaries of the frame signal, thereby reducing the impact of boundary discontinuities on frequency domain analysis.
  • y[n] represents the windowed frame signal after windowing
  • x[n] represents the speech frame signal
  • w represents the window function
  • n represents the nth frame.
  • Time-frequency conversion refers to converting a windowed frame signal from a time-domain signal to a frequency-domain signal. Most information in a speech signal is frequency-related. Converting the time-domain signal to the frequency domain allows for more efficient analysis of the signal's spectral characteristics, yielding reference audio features, which are the signal's spectral features. Time-frequency conversion can include Fourier Transform and its variants, such as the Fast Fourier Transform.
  • a Fast Fourier Transform can be performed on each windowed frame signal to obtain the spectral characteristics of each windowed frame signal in the frequency domain as reference audio characteristics.
  • spectral characteristics may include power spectrum, phase spectrum, or Mel-Frequency Cepstral Coefficients (MFCCs), etc., and are not limited here.
  • the speaker extraction network refers to a deep neural network used for speaker extraction.
  • the input of the speaker extraction network is each reference audio feature, and the output is the frame-level speaker feature corresponding to each reference audio feature.
  • the speech enhancement network refers to a deep neural network used for speech enhancement.
  • the input of the speech enhancement network is the audio feature to be enhanced and the target speaker feature, and the output is the target enhanced audio feature.
  • the speaker extraction network and the speech enhancement network can be trained based on a target loss determined by predicting enhanced audio features and clean audio features.
  • the predicted enhanced audio features are obtained by the speech enhancement network through speech enhancement of the predicted speaker features corresponding to the sample noisy speech and the sample reference speech.
  • the sample noisy speech is obtained by speech mixing of the sample clean speech that matches the speaker pattern of the sample reference speech and the ambient noise speech.
  • the clean speech samples and the reference speech samples in the training samples originate from the same sample object. Therefore, the voiceprints of the clean speech samples and the reference speech samples are matched and originate from the same sample object.
  • the voiceprint extraction network can extract voiceprints from the reference audio features corresponding to the reference speech samples to obtain predicted voiceprint features.
  • Features can also be extracted from the noisy speech samples to obtain corresponding noisy frequency features.
  • These noisy frequency features and predicted voiceprint features are then input into the speech enhancement network for speech enhancement, outputting predicted enhanced audio features corresponding to the sample object.
  • the target loss is determined based on the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples.
  • the voiceprint extraction network and the speech enhancement network are then iteratively trained based on the target loss until the training result conditions are met, resulting in the voiceprint extraction network and the trained speech enhancement network.
  • the voiceprint extraction network can be used for voiceprint extraction
  • the trained speech enhancement network can be used for speech enhancement.
  • this application proposes to use a speaker extraction network to extract frame-level speaker features corresponding to each reference audio feature.
  • Each frame-level speaker feature is obtained by speaker extraction from the reference audio features corresponding to each frame signal in the frequency domain of the reference speech. This is because when using a sensor-based method to collect reference speech for speaker extraction, the speaker extraction network performs speaker extraction on a long speech segment, such as approximately 30 seconds of reference speech, which typically takes a considerable amount of time, such as 1 to 2 seconds. This results in high instantaneous CPU usage and consumes significant computational resources. Since CPU computational resources are limited, this can negatively impact the processing of other functions such as voice calls and network communications.
  • a speech signal is a continuous signal composed of a series of frames
  • voiceprint features can be extracted at each frame, and the final voiceprint features can be constructed by merging these frame-level features.
  • this application proposes a frame-level voiceprint extraction method, which extracts voiceprints from the reference audio features corresponding to each frame signal separately, transforming a segment-level voiceprint extraction into multiple frame-level voiceprint extractions. This avoids excessive instantaneous computational resource consumption during voiceprint extraction, which could disrupt the execution of other tasks such as calls and communications.
  • Voice enhancement is commonly used in real-time applications such as teleconferencing, VoIP, voice assistants, live streaming, and smart home devices. These applications require extremely low latency in processing voice signals to ensure that the target audience can communicate and interact instantly. Furthermore, voice signals often contain sensitive information about the target audience and the content of conversations. Therefore, voice enhancement can be performed locally on the terminal to avoid network-induced latency and maximize the protection of the target audience's privacy.
  • terminal devices typically have limited computing resources, especially compared to cloud servers. These devices have relatively limited CPU and GPU resources. Therefore, this application proposes using a lightweight neural network with lower computational requirements for voiceprint extraction.
  • the voiceprint extraction network can be a deep neural network consisting of an extraction layer, an expansion/compression layer, and a representation layer.
  • the extraction layer can consist of a standard 3 ⁇ 3 convolutional layer
  • the expansion/compression layer can consist of multiple inverse residual modules
  • the representation layer can consist of a global average pooling layer, a fully connected layer, and a final output layer.
  • the extraction layer can be the first convolutional layer of a convolutional neural network, used to downsample the input reference audio features to reduce the computational burden and memory requirements of subsequent layers.
  • the extraction layer can use the H-Swish activation function to improve nonlinearity and feature expressiveness, and finally output the reference voiceprint features corresponding to each reference audio feature.
  • multiple inverse residual modules of the extended compression layer are responsible for feature re-extraction and representation of each reference voiceprint feature.
  • efficient feature extraction is achieved with less computation, and finally the deep voiceprint features corresponding to each reference voiceprint feature are obtained.
  • the representation layer is used to generate frame-level audioprint features
  • the global average pooling layer can average each depth audioprint feature into a single value, convert it into a global feature vector, and then the fully connected layer maps the global feature vector to the representation space, and the output layer outputs the frame-level audioprint features corresponding to each reference audio feature.
  • the extraction layer When performing voiceprint feature extraction using a voiceprint extraction network composed of an extraction layer, an expansion compression layer, and a representation layer, the extraction layer significantly reduces the number of network parameters and computational complexity.
  • the expansion compression layer performs calculations by first expanding and then compressing features, which reduces computational overhead while maintaining efficient feature extraction. This results in a lightweight network architecture for the voiceprint extraction network, enabling terminal devices to complete voiceprint extraction tasks more efficiently.
  • using a lightweight voiceprint extraction network to extract voiceprint features frame by frame can distribute the computational burden across the feature extraction process of each frame. For example, it typically takes 1 to 2 seconds to extract voiceprints from a 30-second reference speech, which is computationally intensive, resulting in high instantaneous CPU occupancy and affecting other communication functions.
  • the voiceprint extraction network can operate within the limited computing resources, memory, and storage space of the terminal device, while meeting the requirements of real-time processing and low latency, and reducing dependence on network bandwidth and connection stability.
  • the voiceprint extraction network can also be a deep neural network composed of a long short-term memory network and fully connected layers. Speech features have short-term stationary temporal sequences, which aligns with the long short-term memory capability of the long short-term memory network, improving the quality of speech enhancement.
  • the long short-term memory network can also be a bidirectional long short-term memory network (Bi-LSTM), without limitation.
  • Average pooling can effectively capture voiceprint features over a longer time period, i.e., piecewise voiceprint features. For example, 40 frame-level voiceprint features of 25 milliseconds each can be transformed into a piecewise voiceprint feature representing 1 second. This piecewise voiceprint feature has the same dimension D as the frame-level voiceprint features, but it represents the voiceprint features of the entire 1 second, rather than the voiceprint features of a single frame. This preserves the key voiceprint information over this long time period, provides a consistent feature representation, and maintains a relatively consistent feature distribution across different piecewise voiceprint features. Furthermore, based on this consistent feature distribution, more accurate voiceprint clustering can be performed. To improve the generation efficiency of piecewise voiceprint features, max pooling can also be used.
  • max pooling can be performed on multiple frame-level voiceprint features in each voiceprint feature group, obtaining the feature values corresponding to each feature dimension in each frame-level voiceprint feature, and selecting the maximum and minimum values for the feature values corresponding to each feature dimension in different voiceprint feature groups to obtain the maximum feature value corresponding to each feature dimension of each voiceprint feature group. Furthermore, the maximum feature value corresponding to each feature dimension of each voiceprint feature group is used as the feature value of the slice-level voiceprint feature in the corresponding feature dimension of different voiceprint feature groups.
  • each voiceprint feature group contains N frame-level voiceprint features, and the dimension of each frame-level voiceprint feature is D.
  • the i-th frame-level voiceprint feature in a voiceprint feature group can be represented as [e ⁇ sub>i,1 ⁇ /sub> , e ⁇ sub>i,2 ⁇ /sub> , ..., e ⁇ sub>i,D ⁇ /sub> ]
  • Emax represents the slice-level voiceprint feature.
  • the feature value of each dimension of the slice-level voiceprint feature is the maximum feature value of all frame-level voiceprint features in the same dimension in its corresponding voiceprint feature group.
  • a slice-level voiceprint feature obtained by max pooling 10 frame-level voiceprint features has the feature value of the slice-level voiceprint feature in the first dimension as the maximum feature value of the first dimension of these 10 frame-level voiceprint features.
  • Max pooling can directly retain the most significant features or peaks in frame-level voiceprint features without averaging the feature values in each dimension, thereby improving the generation efficiency of frame-level voiceprint features. Furthermore, in voiceprint clustering, it can help to more accurately distinguish those with obvious voiceprint features, thus improving the accuracy of voiceprint clustering.
  • Max pooling is advantageous in highlighting significant voiceprint features and noise resistance, but it may lose global voiceprint information and be affected by extreme values.
  • Average pooling is better at preserving global voiceprint information and reducing noise, but it may not effectively highlight obvious voiceprint features. Therefore, hierarchical pooling combining average and max pooling can be used to generate slice-level voiceprint features, preserving both obvious voiceprint features and global voiceprint information, thus providing richer and more representative voiceprint features for subsequent voiceprint clustering tasks.
  • hierarchical pooling can be performed on multiple frame-level voiceprint features in each voiceprint feature group, obtaining the feature values corresponding to each feature dimension in each frame-level voiceprint feature. Further, the mean value of the feature values corresponding to each feature dimension in different voiceprint feature groups is calculated to obtain the average feature value of each voiceprint feature group in each feature dimension. Further, the average feature value of each voiceprint feature group in each feature dimension is used as the feature value of the intermediate slice-level voiceprint features of different voiceprint feature groups in the corresponding feature dimension.
  • the maximum and minimum values of the feature values corresponding to each feature dimension in the intermediate slice-level voiceprint features of different voiceprint feature groups are selected to obtain the maximum feature value of each intermediate slice-level voiceprint feature in each feature dimension, and the maximum feature value of each voiceprint feature group in each feature dimension is used as the feature value of the slice-level voiceprint features of different voiceprint feature groups in the corresponding feature dimension.
  • a voiceprint feature set contains 40 frame-level voiceprint features, each lasting 25ms.
  • Each frame-level voiceprint feature is a feature vector of dimension D.
  • average pooling is performed on the 40 frame-level voiceprint features to obtain a feature vector of dimension D, which is the intermediate slice-level voiceprint feature Emax .
  • max pooling is performed on the intermediate slice-level voiceprint feature Emax to obtain a feature vector of dimension D, which is the slice-level voiceprint feature E corresponding to this voiceprint feature set.
  • Hierarchical pooling first performs max pooling on multiple frame-level voiceprint features to capture the signal's peak features, i.e., prominent voiceprint features, thus reflecting the saliency of local voiceprint features. Then, average pooling is performed on the intermediate voiceprint features at the max-pooling level to obtain the final piece-level voiceprint features, reflecting the smoothness of the global voiceprint features. In this way, through layer-by-layer pooling, local and global voiceprint features can be gradually preserved at different scales, achieving a balance between local details and global trends in voiceprint features, preserving multi-level information of the voiceprint signal. This ensures that each pooling operation can capture important features at different levels, resulting in higher-quality piece-level voiceprint features for more accurate subsequent voiceprint clustering.
  • voiceprint clustering is performed to obtain multiple voiceprint clusters.
  • voiceprint clustering refers to the process of forming multiple different clusters based on the similarity of multiple slice-level voiceprint features.
  • Each cluster is a set of slice-level voiceprint features that are similar to each other within the same cluster and dissimilar to each other in other clusters.
  • Voiceprint clustering methods may include K-means clustering, partitioning around a center point clustering, and hierarchical clustering, etc., and are not limited thereto.
  • One implementation method involves bottom-up hierarchical clustering of multiple slice-level voiceprint features.
  • a suitable distance metric is chosen to measure the similarity between slice-level voiceprint features.
  • Commonly used distance metrics include Euclidean distance and cosine similarity. The choice of distance metric has a significant impact on the clustering results, and the metric that best reflects the similarity between features should be selected based on the application scenario.
  • Each slice-level voiceprint feature is treated as a separate cluster. Assuming there are n slice-level voiceprint features initially, there are n clusters initially, each containing only one slice-level voiceprint feature. Further, a distance matrix is calculated and constructed, where each element represents the distance (or similarity) between two clusters. The two closest clusters (i.e., the two clusters with the smallest distance) are found in the distance matrix and merged into a new cluster.
  • the set of clusters is updated, reducing the number of clusters from n to n-1.
  • the distance values in the distance matrix are updated. For example, a single linkage: the distance between the new cluster and other clusters is the minimum distance between any element in the new cluster and any element in any other cluster. A complete linkage: the distance between the new cluster and other clusters is the maximum distance between any element in the new cluster and any element in any other cluster.
  • each merging process is recorded to construct a tree structure, namely a clustering tree (or dendrogram).
  • the root node of the clustering tree represents a large cluster into which all piecewise voiceprint features are finally merged, while the leaf nodes represent each initial piecewise voiceprint feature.
  • Multiple voiceprint clusters are then obtained by segmenting the clustering tree.
  • the target voiceprint cluster By selecting the target voiceprint cluster from multiple voiceprint clusters, the target voiceprint features corresponding to the target object are determined.
  • the overall quality of the slice-level voiceprint features in the target voiceprint cluster satisfies the speech enhancement condition.
  • target voiceprint cluster selected from multiple voiceprint clusters is used to determine the target voiceprint features, i.e. the baseline features for subsequent speech enhancement, it is necessary to select the target voiceprint cluster that meets the speech enhancement conditions in order to ensure the quality of subsequent speech enhancement.
  • Speech enhancement conditions are used to measure whether speech enhancement is applicable.
  • the evaluation object is the overall slice-level voiceprint features in the voiceprint cluster, rather than individual slice-level voiceprint features. This can effectively screen out voiceprint clusters with high-quality voiceprint features and eliminate the influence of abnormal individual voiceprint features.
  • a target voiceprint cluster selected from multiple voiceprint clusters by means of clustering cluster attributes can be used to generate target voiceprint features corresponding to the target object based on the target voiceprint features.
  • Clustering cluster attributes may include at least cluster density and cluster radius, etc.
  • step (3) may include:
  • Cluster attributes refer to properties that can be used to describe and analyze clusters, such as cluster density, cluster radius, cluster diameter, cluster volume, or average distance within a cluster. Considering that higher cluster density means more data points within a cluster, resulting in greater sound energy, and that the sound of a target object often carries more energy than noise, and that a smaller cluster radius indicates a higher density of points within the cluster, suggesting stronger data consistency and a more concentrated set of voiceprint information belonging to the same speaker, which is beneficial for better clustering, cluster density and cluster radius are selected as the cluster attributes used to determine the target voiceprint clusters in this embodiment.
  • the target cluster can be determined by judging whether the distribution of clustering attributes is too balanced.
  • variance can be calculated based on the cluster density and cluster radius corresponding to each voiceprint cluster to obtain the density variance corresponding to the cluster density and the radius variance corresponding to the cluster radius.
  • the cluster with a cluster density greater than the cluster density of other clusters and a cluster radius smaller than the cluster radius of other clusters is selected as the target voiceprint cluster.
  • the reference speech of the target object is re-acquired until both the density variance and the radius variance are greater than a radius variance threshold, thus obtaining the target voiceprint cluster.
  • calculating the cluster radius corresponding to a voiceprint cluster is equivalent to calculating the maximum distance from the patch-level voiceprint features in the voiceprint cluster to the cluster center, or other statistical distances (such as the average distance).
  • d(x, ⁇ i ) represents the distance from the patch-level voiceprint feature x to the centroid ⁇ i .
  • the variance of the cluster density and cluster radius for each voiceprint cluster we can obtain the density variance corresponding to the cluster density and the radius variance corresponding to the cluster radius. Then, we compare the density variance with a density variance threshold and the radius variance with a radius variance threshold. When both the density variance and radius variance are greater than the density variance threshold, it indicates that the cluster distribution is not very uniform. Therefore, some clusters have clustering attributes that are significantly different from other voiceprint clusters. These voiceprint clusters can then be selected. From these selected clusters, those with a cluster density greater than the cluster density of other clusters and a cluster radius smaller than the cluster radius of other clusters are selected as target voiceprint clusters.
  • clusters with higher cluster density and smaller cluster radius usually mean higher similarity between points within the cluster, points within the cluster are closer to the center, and points within the cluster are distributed within a small spatial range. For voiceprint data, this means that the features of these voiceprints are very similar and belong to the same speaker, thus identifying a more accurate target voiceprint cluster, so as to generate more accurate target voiceprint features accordingly.
  • the average value of the feature vectors of multiple slice-level voiceprint features in the target voiceprint cluster is calculated. Specifically, the arithmetic mean of each component of each feature vector is calculated separately.
  • the target voiceprint cluster has n d-dimensional slice-level voiceprint features, denoted as ( v1 , v2 , ..., vn ).
  • the average value of multiple slice-level voiceprint features is also a d-dimensional slice-level voiceprint feature vavg , where each component is the average value of all slice-level voiceprint features in that dimension, that is:
  • Figure 6 illustrates a flowchart for generating target voiceprint features.
  • reference speech of the target object is acquired, and then sound quality detection is performed on the reference speech. If the sound quality detection passes, audio features are extracted from the reference speech, specifically as shown in the audio feature extraction flowchart in Figure 7.
  • the speech signal of the reference speech in the time domain is framed, windowed, and subjected to Fast Fourier Transform (FFT) to obtain multiple frame-level audio features in the frequency domain.
  • FFT Fast Fourier Transform
  • pooling operations are performed on the multiple frame-level audio features to obtain multiple slice-level audio features
  • voiceprint clustering is performed on the multiple slice-level audio features to obtain the target voiceprint features.
  • the target voiceprint features generated based on the frame-level voiceprint features corresponding to the reference speech are used to perform speech enhancement that suppresses interference with human voice, thereby improving the efficiency and accuracy of speech enhancement.
  • S140 Input the audio features to be enhanced and the target speaker features of the speech to be enhanced into the speech enhancement network, output the target enhanced audio features, and determine the target enhanced speech corresponding to the target enhanced audio features.
  • the speech to be enhanced refers to the speech delivered by the target subject during the voice call. That is, in the initial stage of a voice call, the target voiceprint features of the target subject can be determined seamlessly. Then, after obtaining the target voiceprint features, the speech to be enhanced delivered by the target subject during the subsequent time of the voice call can be enhanced using these features, thereby improving the speech quality of the target subject.
  • this embodiment of the application essentially divides the voice call into two stages.
  • the first stage is the initial period of the voice call mentioned in S110, which involves the aforementioned S110-S130.
  • This first stage is used to collect the voice of the target object to generate target voiceprint features, so as to enhance the voice of the target object in the voice call.
  • target voiceprint features are obtained through S130, it means that the first stage of the voice call ends and the second stage of the voice call begins.
  • the duration of the first stage is related to processing efficiency and accuracy requirements, for example, it can be 10 seconds.
  • the second stage of the voice call is the voice enhancement stage for the target's voice.
  • the voice enhancement is performed on the target's voice using the voiceprint features obtained in the first stage.
  • the voice of the target object is acquired imperceptibly from the start of the voice call to generate the target voiceprint feature.
  • the target voiceprint feature is generated, it takes 5 seconds.
  • the first stage mentioned above is from the 0th second to the 5th second of the voice call, and the second stage mentioned above is from the 6th second to the 20th minute.
  • the speech enhancement network is trained based on the target loss determined by the predicted enhanced audio features and clean audio features.
  • the predicted enhanced audio features are obtained by the speech enhancement network through speech enhancement of the predicted voiceprint features corresponding to the sample noisy speech and the sample reference speech.
  • the sample noisy speech is obtained by speech mixing of the sample clean speech matched with the voiceprint of the sample reference speech and the environmental noise speech.
  • the audio features to be enhanced in the speech to be enhanced can be extracted.
  • a speech enhancement network can be used to enhance the audio features to be enhanced based on the target speaker characteristics to generate target enhanced audio features that remove interfering speakers and other noise.
  • speech reconstruction can be performed on the target enhanced audio features to obtain the target enhanced speech corresponding to the target object. Specifically, it may include:
  • the speech signal to be enhanced in the time domain can be sequentially framed, windowed, and subjected to Fast Fourier Transform to obtain multiple audio features of the speech to be enhanced in the frequency domain. Furthermore, an audio feature vector is constructed from the multiple audio features as the audio feature to be enhanced.
  • the speech enhancement network enhances the audio features to be enhanced based on the target voiceprint features to obtain the target enhanced audio features.
  • the network architecture of the speech enhancement network can be the same as that of the voiceprint extraction network.
  • the speech enhancement network can consist of multiple long short-term memory networks and fully connected layers. The audio features to be enhanced and the target voiceprint features are input into the first long short-term memory network, and then the output of the long short-term memory network is used as the input of the next long short-term memory network until the fully connected layer outputs the target enhanced audio features.
  • the target-enhanced audio features are frequency domain features. Therefore, it is necessary to perform speech reconstruction on the target-enhanced audio features.
  • the inverse process of video transformation when extracting audio features e.g., inverse fast Fourier transform
  • the inverse process of video transformation when extracting audio features is used to perform speech reconstruction on the target-enhanced audio features to obtain the target-enhanced speech of the target object in the time domain.
  • Voiceprint features are individual-specific characteristics used to identify or verify a speaker's identity. They capture a speaker's unique speech characteristics, such as articulation patterns, tone, rhythm, and vocal cord features. Audio features, on the other hand, describe the physical and perceptual characteristics of audio signals. Speech enhancement networks use target voiceprint features to enhance audio features, resulting in target voiceprint features with the unique speech characteristics of the target object. Speech enhancement tailored to the target object is then obtained through speech reconstruction based on these target voiceprint features.
  • This embodiment can acquire reference speech of a target object during a voice call and extract reference audio features corresponding to each frame of the speech signal. Further, based on a voiceprint extraction network, voiceprint extraction is performed on each reference audio feature to obtain frame-level voiceprint features corresponding to each reference audio feature. Thus, by using frame-level voiceprint extraction on the non-intrusive acquired reference speech, voiceprint extraction can be performed on the reference audio features corresponding to each frame of the reference speech signal. By replacing segment-level voiceprint feature extraction with frame-level voiceprint feature extraction, the large computational resources required for extracting a single segment of voiceprint features can be evenly distributed across multiple frames of voiceprint feature extraction.
  • the target voiceprint features corresponding to the target object can be quickly obtained.
  • the acquired audio features to be enhanced and the target voiceprint features of the speech to be enhanced are then input into the speech enhancement network, which outputs the target enhanced audio features. Further, the target enhanced speech corresponding to the target enhanced audio features is determined.
  • the unique speech characteristics of the target object can be represented using the imperceptibly, efficiently, and stably extracted target voiceprint features. This allows for faster removal of other sound information unrelated to the unique speech characteristics of the target object from the audio features to be enhanced, including interfering sound information and noise information, thereby improving the speed, stability, and accuracy of speech enhancement.
  • a method for training a speech enhancement network is provided.
  • This speech enhancement network training method can be executed by a computer device (e.g., a server, edge computing device, or other terminal device with certain processing capabilities) before step 120 of the above embodiment.
  • This computer device must at least have storage, computing, and communication functions.
  • Figure 9 shows a flowchart of the speech enhancement network training method provided in this embodiment.
  • Figure 10 shows an architecture diagram of a speech enhancement network training method. The specific steps of this speech enhancement network training method will be described below with reference to Figures 9 and 10:
  • iterative grid training can be performed on the speaker extraction network and the speech enhancement network based on the target loss.
  • the speaker extraction network can extract speakerprints from the sample reference speech to obtain predicted speakerprint features.
  • Feature extraction is performed on the sample noisy speech to obtain the corresponding sample noisy frequency features.
  • These sample noisy frequency features and the predicted speakerprint features are then input into the speech enhancement network for speech enhancement, outputting predicted enhanced audio features.
  • the target loss is determined based on the predicted enhanced audio features and the clean audio features corresponding to the sample clean speech.
  • the sample noisy speech is obtained by mixing environmental noise speech and sample clean speech. Specifically:
  • S210 Obtain training samples, which include clean speech samples, reference speech samples, and noisy speech samples.
  • the speech to be enhanced often includes interfering human voices.
  • the voice of a student collected by an online classroom client may contain not only environmental noise but also the voices of other students, i.e., interfering human voices.
  • a voiceprint extraction network and a speech enhancement network can be trained based on sample reference speech with the same voiceprint information and sample noisy speech.
  • the goal of network training is for the speech enhancement network to enhance the sample noisy audio features based on the predicted voiceprint features generated by the voiceprint extraction network, and to extract clean speech from the noisy sample speech. In other words, the error between the predicted enhanced audio features and the clean audio features corresponding to the clean sample speech gradually decreases.
  • the noisy speech samples are obtained by mixing environmentally noisy speech with clean speech samples.
  • the clean speech samples and the reference speech samples in the training samples come from the same sample object. Clean speech samples refer to clear speech that is not affected by noise.
  • clean speech samples can be obtained through recording or from a pre-established speech database.
  • a clean speech sample is a pure sample; to obtain high-quality pure samples, background noise is typically avoided as much as possible. This can be achieved using a professional microphone while maintaining good sound quality and diversity of speech content.
  • ambient noise speech refers to speech interfered with by different types of noise.
  • This speech can be collected from the real world, such as by recording background noise in daily life using a microphone in different environments, or extracted from noise databases, such as simulated noise, in-vehicle noise, and ambient noise in a coffee shop.
  • the collected noisy speech should cover a variety of environments and noise types to enable the training model to have better generalization capabilities.
  • noisy sample speech is generated by mixing clean sample speech and ambient noisy speech. In other words, some noise is added to the clean sample speech, resulting in a noisy sample.
  • noisy sample speech can be used to simulate speech in real-world scenarios. In scenarios such as phone calls, video conferencing, speech recognition front-ends, and live streaming/on-demand applications, the speech acquired by devices is generally noisy. By mixing these noisy samples to form noisy sample speech, it can be used for subsequent network training.
  • ambient noise speech may include interfering human voices and background noise.
  • the noisy sample speech can be obtained by mixing interfering human voices, background noise, and clean sample speech in different proportions.
  • S ⁇ sub> mix ⁇ /sub> represents the noisy sample speech
  • S ⁇ sub>1 ⁇ /sub> represents the interfering human voice
  • S ⁇ sub>2 ⁇ /sub> represents the background noise
  • S ⁇ sub>3 ⁇ /sub> represents the clean sample speech.
  • the speech content of the reference speech and the clean speech can be different.
  • the reference speech and the clean speech may be two different news articles read by the same subject.
  • the speech content can include as many words as possible with different pronunciations, allowing the reference speech and the clean speech to cover more speech information, thus improving the accuracy and confidence of network training.
  • the speech content of the reference speech and the clean speech in the same training sample can also be the same.
  • the training set ⁇ x0 , x2 , ..., xn ⁇ can be obtained from the database. Where n>0 & n ⁇ N * , and n is the number of training samples.
  • the voiceprint is extracted from the sample reference audio features corresponding to the sample reference speech to obtain the predicted voiceprint features.
  • frame segmentation and windowing operations can be performed on the sample reference speech, followed by time-frequency conversion to obtain the corresponding frequency domain features.
  • the sample reference speech acquired by the microphone is sequentially framed and windowed to obtain the frame signal of the sample reference speech, and a Fast Fourier Transform is performed on the frame signal.
  • the frequency domain features after the Fast Fourier Transform are used as the sample reference audio features of the sample reference speech.
  • sample reference audio features are input into the speaker extraction network to extract speakerprints from the sample reference audio features corresponding to the sample reference speech.
  • Figure 11 shows the framework diagram of the speaker extraction network.
  • the speaker extraction network includes an extraction layer, an expansion and compression layer, and a representation layer.
  • Each sample reference audio feature is input into the extraction layer for speakerprint feature extraction, resulting in multiple sample reference speakerprint features.
  • Each sample reference speakerprint feature is then input into the expansion and compression layer for feature expansion and compression, resulting in multiple sample depth speakerprint features.
  • each sample depth speakerprint feature is input into the representation layer for speakerprint feature representation, resulting in predicted speakerprint features.
  • the noisy speech sample can be framed and windowed. Then, time-frequency conversion can be performed to obtain the corresponding frequency domain features as the noisy frequency features of the sample, such as logarithmic power spectrum (LPS) and Mel-frequency cepstral coefficients (MFCC), etc., which are not limited here.
  • LPS logarithmic power spectrum
  • MFCC Mel-frequency cepstral coefficients
  • speech data often cannot be directly input into a model for training like image data. Speech data lacks significant feature changes over long time, making it difficult to learn its features. Furthermore, speech temporal data typically consists of a 16K sampling rate (16,000 samples per second), and directly inputting these sampling points would result in excessive training data and make it difficult to achieve meaningful results. Therefore, in speech enhancement tasks, speech data can be transformed into acoustic features to serve as either the network's input or output.
  • S240 Input the sample noise frequency features and predicted speaker features into the initial speech enhancement network to perform speech enhancement, and output the predicted enhanced audio features corresponding to the sample object.
  • the network architecture of the speech enhancement network can be the same as that of the speaker extraction network.
  • the speech enhancement network includes an extraction layer, an expansion/compression layer, and a representation layer. Sample noisy frequency features and predicted speaker features are input into the speech enhancement network, and the speech is enhanced sequentially through the extraction layer, expansion/compression layer, and representation layer to obtain predicted enhanced audio features.
  • S250 Determine the target loss based on the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples. Iteratively train the initial speaker extraction network and the initial speech enhancement network based on the target loss until the training result conditions are met, thus obtaining the speaker extraction network and the speech enhancement network.
  • the training termination conditions may include: the target loss value is less than a preset value, the target loss value no longer changes, or the training iterations reach a preset number.
  • an optimizer can be used to optimize the target loss function, and the learning rate, batch size, and training epochs can be set based on experimental experience.
  • error calculation is performed on the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples.
  • the target loss is determined based on the calculation results. Further, the initial speaker extraction network and the initial speech enhancement network are iteratively trained based on the target loss until the training reaches the result conditions that the loss value of the target loss is less than a preset value, the loss value of the target loss is within a threshold range, or the number of training times reaches a preset number, thereby obtaining the speaker extraction network and the speech enhancement network.
  • the mean square error (MSE) between the predicted enhanced audio features and the clean audio features corresponding to the sample clean speech can be used as the target loss for the initial speaker extraction network and the initial speech enhancement network.
  • MSE mean square error
  • Loss( ⁇ ) represents the target loss
  • ⁇ represents the weight parameters of the initial speaker extraction network and the initial speech enhancement network
  • y ⁇ sub>i ⁇ /sub> represents the i-th training sample.
  • the corresponding predicted enhanced audio features This represents the clean audio features (ground truth) of the clean speech corresponding to the i-th training sample.
  • the target loss can also be determined based on the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples using other loss functions (e.g., cross-entropy loss function, absolute value loss function, etc.).
  • the loss under each loss function can be calculated separately based on the predicted enhanced audio features and the clean audio features using at least two different loss functions. Then, the losses under the at least two different loss functions are weighted, and the weighted result is used as the target loss.
  • this application provides a generative adversarial network-based approach to train both the initial speaker extraction network and the initial speech enhancement network, thereby improving the quality and naturalness of the enhanced speech.
  • the initial speaker extraction network and initial speech enhancement network can be used as generators, and a discriminator can be constructed.
  • the discriminator's network structure can be a convolutional neural network, a recurrent neural network, or a Transformer model, etc., without limitation.
  • a generative adversarial network is formed by the generator and discriminator. During the training of this GAN, the adversarial process between the generator and discriminator forces the generator's output to become increasingly realistic. The generator attempts to generate the most realistic enhanced predicted audio features possible, while the discriminator attempts to distinguish between the predicted enhanced audio features and the real clean audio features. This dynamic adversarial process drives the generator to gradually improve in quality and detail, going beyond simply relying on point-to-point error minimization based on MSE.
  • the discriminator's network parameters are iteratively updated based on a first loss determined by the predicted enhanced audio features and the clean audio features corresponding to the clean speech samples.
  • the generator's network parameters are iteratively updated based on a second loss determined by the estimated enhanced audio features.
  • the estimated enhanced audio features are the predicted enhanced audio features generated based on other training samples.
  • log D(1- yi ) represents the first discrimination result. This is represented as the second discrimination result.
  • log(D(y i )) represents the third discrimination result.
  • a speaker extraction network and a speech enhancement network are obtained.
  • a generative adversarial network GAN is used to train the speaker extraction and speech enhancement networks.
  • the generator composed of these two networks, aims to generate outputs that closely approximate realistic enhanced speech features, while the discriminator improves its discrimination ability by comparing the generated speech features with real speech features.
  • This method iteratively optimizes the generator and discriminator, enabling the generator to produce more realistic and high-quality enhanced speech.
  • the discriminator learns and focuses on the global structure and complex time-frequency patterns in the audio signal. This allows the generator to better learn richer and more expressive speech features when generating speech features, thus producing high-quality enhanced speech that better matches human perception. It is particularly suitable for speech enhancement tasks involving complex signals in diverse environments.
  • the trained speaker extraction network and speech enhancement network can be compared with existing deep neural networks for speech enhancement and speech enhancement methods based on consciously recorded reference speech.
  • the comparison metric can be the Noise Mean Option Score (NMOS).
  • NMOS Noise Mean Option Score
  • Table 1 The performance comparison results are shown in Table 1 below:
  • the speech enhancement method proposed in this application based on non-sensory acquisition of reference speech has a significant improvement over the speech enhancement method based on deep neural networks, and also shows a significant improvement over the speech enhancement method based on sensory recording of reference speech. This is because this application reduces the computational resources required to generate voiceprint vectors, and improves the quality of voiceprint features by eliminating interfering speech through clustering when generating voiceprint features. Thus, high-quality enhanced speech can be obtained by using these voiceprint features for efficient speech enhancement.
  • FIG. 12 shows a structural block diagram of a speech enhancement device 300 provided in an embodiment of this application.
  • the device 300 may include:
  • the voice acquisition module 310 is used to acquire the reference voice of the target object in the state of a voice call, and extract the reference audio features of each frame of voice signal corresponding to the reference voice.
  • the voiceprint extraction module 320 is used to extract voiceprints from each reference audio feature based on the voiceprint extraction network to obtain the frame-level voiceprint features corresponding to each reference audio feature.
  • the voiceprint clustering module 330 is used to perform voiceprint clustering based on each frame-level voiceprint feature to obtain the target voiceprint feature corresponding to the target object;
  • the speech enhancement module 340 is used to input the audio features to be enhanced and the target voiceprint features of the speech to be enhanced into the speech enhancement network, output the target enhanced audio features, and determine the target enhanced speech corresponding to the target enhanced audio features, wherein the speech to be enhanced is emitted by the target object in the voice call.
  • the voiceprint clustering module 330 may include a pooling operation unit, a voiceprint clustering unit, and a voiceprint determination unit.
  • the pooling operation unit is used to perform pooling operations based on each frame-level voiceprint feature to obtain multiple slice-level voiceprint features.
  • the voiceprint clustering unit is used to perform voiceprint clustering based on the multiple slice-level voiceprint features to obtain multiple voiceprint clusters;
  • the voiceprint determination unit is used to determine the target voiceprint features corresponding to the target object by selecting a target voiceprint cluster from the plurality of voiceprint clusters, wherein the overall quality of the slice-level voiceprint features in the target voiceprint cluster satisfies the speech enhancement condition.
  • the pooling operation unit may include a slice length subunit, a grouping subunit, and a pooling subunit.
  • the slice length subunit is used to obtain the pooled slice length
  • a grouping subunit is used to group each frame-level voiceprint feature based on the pooling slice length to obtain multiple voiceprint feature groups;
  • the pooling subunit is used to perform pooling operations on multiple frame-level voiceprint features in each voiceprint feature group to obtain the slice-level voiceprint features corresponding to each voiceprint feature group.
  • the pooling subunit may be specifically used to: for multiple frame-level voiceprint features in each voiceprint feature group, obtain the feature values corresponding to each feature dimension in each frame-level voiceprint feature; calculate the mean of the feature values corresponding to each feature dimension in different voiceprint feature groups to obtain the average feature value of each voiceprint feature group in each feature dimension; and use the average feature value of each voiceprint feature group in each feature dimension as the feature value of the slice-level voiceprint feature in the corresponding feature dimension of different voiceprint feature groups.
  • the pooling subunit may be specifically used to: for multiple frame-level voiceprint features in each voiceprint feature group, obtain the feature values corresponding to each feature dimension in each frame-level voiceprint feature; select the maximum and minimum values for the feature values corresponding to each feature dimension in different voiceprint feature groups respectively, and obtain the maximum feature value corresponding to each feature dimension of each voiceprint feature group; and use the maximum feature value corresponding to each feature dimension of each voiceprint feature group as the feature value of the slice-level voiceprint feature in the corresponding feature dimension of different voiceprint feature groups respectively.
  • the voiceprint determination unit may include an attribute subunit, a clustering subunit, and a voiceprint subunit.
  • the attribute sub-unit is used to obtain the cluster attributes corresponding to each voiceprint cluster
  • the clustering subunit is used to determine the target voiceprint cluster based on the clustering cluster attributes corresponding to each voiceprint cluster.
  • the voiceprint subunit is used to average multiple slice-level voiceprint features in the target voiceprint cluster to obtain the target voiceprint features corresponding to the target object.
  • the cluster attributes include at least cluster density and cluster radius.
  • the clustering subunit can be specifically used to: calculate the variance based on the cluster density and cluster radius corresponding to each voiceprint cluster to obtain the density variance corresponding to the cluster density and the radius variance corresponding to the cluster radius; when the density variance is greater than the density variance threshold and the radius variance is greater than the radius variance threshold, the cluster with a cluster density greater than the cluster density of other clusters and a cluster radius smaller than the cluster radius of other clusters is taken as the target voiceprint cluster; when the density variance is not greater than the density variance threshold or the radius variance is not greater than the radius variance threshold, the reference speech of the target object is re-acquired until the density variance is greater than the density variance threshold and the radius variance is greater than the radius variance threshold, thus obtaining the target voiceprint cluster.
  • the speech enhancement module 340 may be specifically used to: extract features from the acquired speech to be enhanced to obtain audio features to be enhanced; input the audio features to be enhanced and the target voiceprint features into a speech enhancement network, and have the speech enhancement network enhance the audio features to be enhanced based on the target voiceprint features to obtain target enhanced audio features; and reconstruct the speech from the target enhanced audio features to obtain target enhanced speech.
  • the voiceprint extraction network includes an extraction layer, an expansion compression layer, and a representation layer.
  • the voiceprint extraction module 320 can be specifically used to: input each reference audio feature into the extraction layer for voiceprint feature extraction to obtain multiple reference voiceprint features; input each reference voiceprint feature into the expansion compression layer for feature expansion and compression to obtain multiple depth voiceprint features; and input each depth voiceprint feature into the representation layer for voiceprint feature representation to obtain frame-level voiceprint features corresponding to each reference audio feature.
  • the speech acquisition module 310 may be specifically used to: perform sound quality detection on the reference speech to obtain the speech signal-to-noise ratio (SNR) of the reference speech; when the SNR is greater than the SNR threshold, perform frame segmentation on the reference speech to obtain multiple speech frame signals; perform windowing operation on each speech frame signal based on a window function to obtain a windowed frame signal corresponding to each speech frame signal; and perform time-frequency conversion on each windowed frame signal to obtain the reference audio features corresponding to each windowed frame signal in the frequency domain.
  • SNR speech signal-to-noise ratio
  • This embodiment can acquire reference speech of a target object during a voice call and extract reference audio features corresponding to each frame of the speech signal. Further, based on a voiceprint extraction network, voiceprint extraction is performed on each reference audio feature to obtain frame-level voiceprint features corresponding to each reference audio feature. Compared to related technologies that use sensor-acquired reference speech for speech enhancement, this application, during a voice call, utilizes a frame-level voiceprint extraction method on non-sensory acquired reference speech. This allows for voiceprint extraction of reference audio features corresponding to each frame of the reference speech signal. By replacing segment-level voiceprint feature extraction with frame-level voiceprint feature extraction, the large computational resources required for extracting a single segment of voiceprint features can be evenly distributed across multiple frames of voiceprint feature extraction.
  • the target voiceprint features corresponding to the target object can be quickly obtained.
  • the acquired audio features to be enhanced and the target voiceprint features of the speech to be enhanced are then input into the speech enhancement network, which outputs the target enhanced audio features. Further, the target enhanced speech corresponding to the target enhanced audio features is determined.
  • this application can utilize the non-sensory, efficient, and stable extracted target voiceprint features to represent the unique speech characteristics of the target object. This allows for faster removal of other sound information unrelated to the unique speech characteristics of the target object from the audio features to be enhanced, including interfering human voice information and noise information, thereby improving the speed, stability, and accuracy of speech enhancement.
  • FIG. 13 shows a structural block diagram of a speech enhancement network training device 400 provided in an embodiment of this application.
  • the device 400 may include:
  • the sample acquisition module 410 is used to acquire training samples, which include clean speech samples, reference speech samples, and noisy speech samples.
  • the noisy speech samples are obtained by mixing environmental noise speech and the clean speech samples.
  • the clean speech samples and reference speech samples in the training samples come from the same sample object.
  • Feature prediction module 420 is used to extract the voiceprint of the sample reference audio features corresponding to the sample reference speech based on the initial voiceprint extraction network to obtain predicted voiceprint features.
  • the audio extraction module 430 is used to extract features from the sample noisy speech to obtain the corresponding sample noisy audio features
  • the audio prediction module 440 is used to input the sample noise frequency features and the predicted voiceprint features into the initial speech enhancement network for speech enhancement, and output the predicted enhanced audio features corresponding to the sample object;
  • the network training module 450 is used to determine the target loss based on the predicted enhanced audio features and the clean audio features corresponding to the sample clean speech, and to iteratively train the speaker extraction network and the speech enhancement network based on the target loss until the training result conditions are met, thereby obtaining the speaker extraction network and the speech enhancement network.
  • the network training module 450 may be specifically used to: calculate the error between the predicted enhanced audio features and the clean audio features corresponding to the sample clean speech, determine the target loss based on the calculation result; and iteratively train the initial speaker extraction network and the initial speech enhancement network based on the target loss until the training reaches at least the result condition that the loss value of the target loss is less than a preset value, the loss value of the target loss is within a threshold range, or the number of training times reaches a preset number, thereby obtaining the speaker extraction network and the speech enhancement network.
  • the coupling between modules can be electrical, mechanical, or other forms of coupling.
  • the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
  • the integrated modules described above can be implemented in hardware or as software functional modules.
  • this embodiment of the application also provides a computer device 500, which includes a processor 510, a memory 520, a power supply 530, and an input unit 540.
  • the memory 520 stores a computer program, which, when called by the processor 510, can execute the various method steps provided in the above embodiments.
  • the structure of the computer device shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
  • Processor 510 may include one or more processing cores. Processor 510 connects to various parts of the entire battery management system using various interfaces and lines. It executes instructions, programs, instruction sets, or program sets stored in memory 520, calls data stored in memory 520, performs various functions and processes data within the battery management system, and performs various functions and processes data within the computer device, thereby providing overall control of the computer device.
  • processor 510 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA).
  • DSP Digital Signal Processing
  • FPGA Field-Programmable Gate Array
  • PDA Programmable Logic Array
  • Processor 510 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem.
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • modem modem
  • the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understandable that the aforementioned modem may not be integrated into the processor 510, but may be implemented using a separate communication chip.
  • the memory 520 may include random access memory (RAM) or read-only memory (ROM).
  • the memory 520 can be used to store instructions, programs, instruction sets, or program assemblies.
  • the memory 520 may include a program storage area and a data storage area.
  • the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above.
  • the data storage area may also store data created during the use of the computer device (such as phonebook and audio/video data). Accordingly, the memory 520 may also include a memory controller to provide the processor 510 with access to the memory 520.
  • the power supply 530 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.
  • the power supply 530 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
  • the input unit 540 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
  • the computer device 500 may also include an output unit, a display unit, etc., which will not be described in detail here.
  • the processor 510 of the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 520 according to the following instructions, and the processor 510 runs the data stored in the memory 520, such as telephone books and audio and video data, thereby implementing the various method steps provided in the foregoing embodiments.
  • this application embodiment also provides a computer-readable storage medium 600, which stores a computer program 610.
  • the computer program 610 can be invoked by the processor of a computer device to execute various method steps provided in this application embodiment.
  • Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM.
  • computer-readable storage media include non-volatile computer-readable storage medium.
  • Computer-readable storage medium 600 has storage space for computer programs that perform any of the method steps in the above embodiments. These computer programs can be read from or written to one or more computer program products. The computer programs can be compressed in an appropriate form.
  • module refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined objective, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof.
  • a processor or multiple processors or memory
  • each module or unit may be part of a larger module or unit that includes the functionality of that module or unit.
  • At least one (item) means one or more, and “more than” means two or more.
  • “And/or” is used to describe the relationship between related objects, indicating that three relationships can exist. For example, “A and/or B” can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character “/” generally indicates that the preceding and following related objects are in an “or” relationship. "At least one (item) of the following” or similar expressions refer to any combination of these items, including any combination of single or plural items.
  • At least one (item) of a, b, or c can represent: a, b, c, "a and b", “a and c", “b and c", or "a and b and c", where a, b, and c can be single or multiple.
  • the disclosed systems, apparatuses, and methods can be implemented in other ways.
  • the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods.
  • multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
  • the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
  • the units described as separate components may or may not be physically separate.
  • the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
  • the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
  • the integrated units described above can be implemented in hardware or as software functional units.
  • the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of this application in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.
  • This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.
  • a computer device which may be a personal computer, a server, or a network device, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Quality & Reliability (AREA)
  • Telephonic Communication Services (AREA)

Abstract

一种语音增强方法与语音增强网络训练方法、装置、设备及存储介质。包括：在语音通话的状态下，获取目标对象的参考语音，并提取参考语音对应每帧语音信号的参考音频特征（S110）；基于声纹提取网络对每个参考音频特征进行声纹提取，得到每个参考音频特征对应的帧级声纹特征（S120）；根据每个帧级声纹特征进行声纹聚类，得到目标对象对应的目标声纹特征（S130）；将待增强语音的待增强音频特征和目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定目标增强音频特征对应的目标增强语音（S140）。通过无感采集参考语音，对参考语音的每个帧信号进行帧级别的声纹提取，降低声纹提取的计算资源消耗，并利用目标声纹特征对待增强音频特征进行语音增强，提高语音增强的准确性和稳定性。方法可用于视频会议、音视频通话等各种需要语音增强的场景。

Description

语音增强方法与相关方法、装置、设备、介质和程序产品
本申请要求于2024年07月10日提交中国专利局、申请号为2024109239346、申请名称为“语音增强方法与相关方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及语音处理技术领域，更具体地，涉及语音增强。
背景技术
语音处理(Speech Technology)技术是利用计算机对人类语音进行分析、处理和理解的一系列技术的总称。其核心包括语音识别、语音合成、语音增强和语音分析。这些技术广泛应用于日常生活和各个行业，显著改善了人机交互的方式。例如，在通讯等场景中，通过语音增强消除背景噪音等以提升通话质量。
随着深度学习和人工智能的发展，语音增强技术取得了显著的进步。现有的语音增强系统能够在一般的声学环境中实时处理语音信号，并且在多语音、多设备的应用场景下实现语音增强功能。这些语音增强技术不仅提升个人通信和娱乐体验，也为专业领域(音视频通话、直播和采访等)的音频处理提供了有效的解决方案。
有关技术的语音增强方法基本都是围绕抑制干扰人声和背景噪声。例如，在传统方式中，基于谱减法的语音频谱估计和基于高斯混合模型的噪声估计，以及目前常用的基于深度神经网络的端到端噪声去除。然而，在复杂的语音采集环境下，通过这些语音增强方法得到的增强语音仍存在效果不佳的情况。
发明内容
本申请实施例提供一种语音增强方法与相关方法、装置、设备及存储介质。可以提高语音增强的稳定性和准确性。
根据本申请的一方面，提供了一种语音增强方法，该方法包括：在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征；基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征；根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征；将获取的待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音；其中，所述声纹提取网络和所述语音增强网络为根据预测增强音频特征和干净音频特征确定的目标损失训练得到，所述预测增强音频特征为语音增强网络对样本带噪语音和样本参考语音对应的预测声纹特征经语音增强得到，所述样本带噪语音由与所述样本参考语音的声纹匹配的样本干净语音及环境噪声语音经语音混合得到。
根据本申请的另一方面，提供了一种语音增强网络训练方法，该方法包括：获取训练样本，所述训练样本包括样本干净语音、样本参考语音和样本带噪语音，所述样本带噪语音是由环境噪声语音和所述样本干净语音经语音混合得到，其中，训练样本中的样本干净语音和样本参考语音来源于同一样本对象；基于初始声纹提取网络对所述样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征；对所述样本带噪语音进行特征提取，得到对应的样本带噪音频特征；将所述样本带噪音频特征和所述预测声纹特征输入至初始语音增强网络进行语音增强，输出所述样本对象对应的预测增强音频特征；根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
根据本申请的另一方面，提供了一种语音增强装置，该装置包括：语音获取模块，用于在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征；声纹提取模块，用于基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征；声纹聚类模块，用于根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征；语音增强模块，用于将获取的待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音；其中，所述声纹提取网络和所述语音增强网络为根据预测增强音频特征和干净音频特征确定的目标损失训练得到，所述预测增强音频特征为语音增强网络对样本带噪语音和样本参考语音对应的预测声纹特征经语音增强得到，所述样本带噪语音由与所述样本参考语音的声纹匹配的样本干净语音及环境噪声语音经语音混合得到。
根据本申请的另一方面，提供了一种语音增强网络训练装置，该装置包括：样本获取模块，用于获取训练样本，所述训练样本包括样本干净语音、样本参考语音和样本带噪语音，所述样本带噪语音是由环境噪声语音和所述样本干净语音经语音混合得到，其中，训练样本中的样本干净语音和样本参考语音来源于同一样本对象；特征预测模块，用于基于初始声纹提取网络对所述样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征；音频提取模块，用于对所述样本带噪语音进行特征提取，得到对应的样本带噪音频特征；音频预测模块，用于将所述样本带噪音频特征和所述预测声纹特征输入至初始语音增强网络进行语音增强，输出所述样本对象对应的预测增强音频特征；网络训练模块，用于根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
根据本申请的另一方面，提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，其中，在该计算机程序被处理器运行时执行上述的语音增强方法与相关方法。
根据本申请的另一方面，提供了一种计算机设备，该计算机设备包括处理器以及存储器，存储器存储有计算机程序，该计算机程序被处理器调用时执行上述的语音增强方法与相关方法。
根据本申请的另一方面，提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在存储介质中；计算机设备的处理器从存储介质读取该计算机程序，处理器执行该计算机程序或指令，使得计算机设备执行上述语音增强方法与相关方法。
本申请可以在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征，进一步地，基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征。如此，相对于有关技术利用有感采集的参考语音进行语音增强的方案而言，本申请在语音通话状态下，通过对无感采集的参考语音利用帧级声纹提取的方法，可以对参考语音的每个帧信号对应的参考音频特征分别进行声纹提取，通过将有关技术基于段级别的声纹特征提取，替换为基于帧级别的声纹特征提取，使得一段声纹特征提取所需的较大计算资源消耗，能够平均释放到多帧声纹特征提取的过程中，从而，避免因声纹提取的计算资源消耗过大，影响通话和通信等其他任务的执行，提高声纹特征提取的效率和稳定性。
进一步地，根据稳定且高效提取的每个帧级声纹特征进行声纹聚类，可以快速得到目标对象对应的目标声纹特征，并将获取的待增强语音的待增强音频特征和目标声纹特征输入至语音增强网络，输出目标增强音频特征，进一步地，确定目标增强音频特征对应的目标增强语音。由此，相对于有关技术利用有感采集的参考语音进行语音增强的方案而言，本申请能够利用无感、高效且稳定提取的目标声纹特征表示目标对象的独特语音特性，进而更加快速地从待增强音频特征中去除与目标对象的独特语音特性无关的其他声音信息，包括干扰人的声音信息和噪声信息等，从而提高语音增强的速度、稳定性和准确度。
本申请的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
图1示出了本申请实施例提供的一种系统架构示意图；
图2示出了本申请实施例提供的一种系统部署示意图；
图3示出了本申请实施例提供的一种语音增强方法的应用场景图；
图4示出了本申请实施例提供的另一种语音增强方法的应用场景图；
图5示出了本申请实施例提供的一种语音增强方法的流程示意图；
图6示出了本申请实施例提供的一种目标声纹特征的生成流程图；
图7示出了本申请实施例提供的一种音频特征提取流程图；
图8示出了本申请实施例提供的一种语音增强流程图；
图9示出了本申请实施例提供的一种语音增强网络训练方法的流程示意图；
图10示出了本申请实施例提供的一种语音增强网络训练的架构图；
图11示出了本申请实施例提供的一种声纹提取网络的框架图；
图12是本申请实施例提供的一种语音增强装置的模块框图；
图13是本申请实施例提供的一种语音增强网络训练装置的模块框图；
图14是本申请实施例提供的一种计算机设备的模块框图；
图15是本申请实施例提供的一种计算机可读存储介质的模块框图。
具体实施方式
为了使本技术领域的人员更好地理解本申请的方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整的描述。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。通过参考附图描述的实施方式是示例性地，仅用于解释本申请，而不能理解为对本申请的限制。基于本申请中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在说明书、权利要求书和上述附图所描述的一些流程中，包含了按照特定顺序出现的多个步骤，但应该清楚了解，这些步骤可以不按照其在本文中出现的顺序来执行或并行执行，步骤序号仅仅是用于区分开各个不同的步骤，序号本身不代表任何的执行顺序。此外，本文中的“第一”、“第二”或“目标”等描述，是用于区别类似的对象，而不必用于描述特定的顺序、先后次序或数量。
值得注意的是，在本申请的具体实施方式中，涉及到参考语音等相关数据，当本申请以上实施例运用到具体产品或技术中时，需要获得对象许可或者同意，且相关数据的收集、使用和处理需要遵守相关法律法规和标准。示例性地，当本申请实施例需要获取待增强语音时，可通过弹窗或者跳转到确认页面等方式来获得待增强语音的单独许可或者单独同意，在明确获得单独许可或单独同意之后，再获取用于使本申请实施例能够正常运行的必要的相关的数据。
对本申请实施例进行进一步详细说明之前，对本申请实施例中涉及的名词和术语进行说明，本申请实施例中涉及的名词和术语适用于如下的解释：
本申请提出的语音增强方法涉及人工智能(Artificial Intelligence,AI)技术。
其中，语音增强旨在改善语音信号的质量，使其在背景噪音和干扰环境中更加清晰和可懂。这些技术涵盖了多种处理方法，如噪音抑制、回声消除和语音清晰化。噪音抑制通过算法降低环境噪音的影响，使语音信号更为纯净。回声消除则专注于移除由于设备间或环境反射导致的回声，确保通信的自然和清晰。语音清晰化技术在保留语音自然性的同时，增强了语音信号的清晰度和可懂度。语音增强技术在众多领域发挥着关键作用。例如，本申请提出的语音增强方法能够有效消除背景噪音和干扰人声，显著语音质量。
语音增强的有关技术可以根据其实现方式和处理方法分为多种类别，主要包括基于谱减(Spectral Subtraction)的语音频谱评估方法、基于高斯混合模型(Gaussian Mixture Model，GMM)的噪声估计方法，以及基于深度神经网络(Deep Neural Network，DNN)的噪声去除方法。谱减法是一种经典的语音增强技术，主要通过减少噪声频谱来提升语音信号的质量。其基本原理是：将噪声和语音的频谱相减，从而得到净化后的语音信号。噪声频谱的估计通常基于在无语音活动时(静默期间)所采集的噪声样本。虽然谱减法简单且计算效率高，但它在处理复杂或非平稳噪声时效果有限，并且容易引入“噪声”伪影，使得增强后的语音听起来不自然。
高斯混合模型是一种统计模型，它通过多个高斯分布的线性组合来近似复杂的概率分布。在语音增强中，高斯混合模型估计背景噪声和谱减系数，对含噪语音进行谱减，恢复纯净语音。首先，含噪语音经过预处理得到含噪语音的幅度和相位，幅度用于噪声估计和谱减，相位用于恢复时域信号；然后利用高斯混合模型从含噪语音中实时估计噪声参数和纯净语音倒谱特征，并根据估得的纯净语音倒谱特征计算谱减系数；最后，对含噪语音的频谱进行谱减，恢复时域信号，并用重叠相加法得到增强后的语音。可以显著提高语音增强算法对非平稳噪声的抑制能力。这种方法的不足之处是不能有效抑制干扰人声。
通过大量训练数据训练得到的深度神经网络可以学习语音和噪声的复杂特征，从而在增强过程中实现高度非线性的映射。常见的深度神经网络的架构包括卷积神经网络、循环神经网络以及自注意力模型(如，Transformer)。这些网络能够在嘈杂环境中提取出清晰的语音信号，并且处理多种不同类型的噪声。与传统方法相比，深度神经网络具有更高的灵活性和适应性，能够在不同场景中提供有效的语音增强。然而，这种方法通常无法有效抑制背景干扰人声。总之，有关技术仍存在语音增强效果不佳的问题。
为了解决上述问题，发明人经过研究，提出了本申请实施例提供的语音增强方法以及语音增强网络训练方法。下面先对本申请所涉及到的上述方法的系统架构以及相关的应用场景进行说明。
请参阅图1，图1示出一种系统架构示意图。本申请实施例提供的上述方法可以应用在系统100中。其中，数据获取设备110用于获取训练集，训练集包括多个训练样本。针对本申请实施例中的语音增强网络训练方法来说，每个训练样本可以包括样本参考语音、样本干净语音以及样本带噪语音。同一训练样本中的样本参考语音和样本干净语音来源于同一样本发声对象。数据获取设备110在获取到训练数据之后，可以将训练数据存入数据库120，训练设备130可以基于数据库120中的训练数据对预设的神经网络进行网络训练，直至该预设的神经网络满足训练结束条件，得到训练后的目标网络101。
其中，训练结束条件可以为：目标损失的总损失值小于预设值、目标损失的总损失值处于阈值范围、或者训练次数达到预设次数等。目标网络101可以为深度神经网络或由多个神经网络组成的网络模型。该目标网络101，也即用于提取声纹特征的声纹提取网络或者用于执行语音增强的语音增强网络。示例性地，声纹提取网络或语音增强网络可以由长短期记忆网络(Long Short-Term Memory，LSTM)和全连接层(Fully Connected Layer，FC)构成的网络框架，在此不做限定。
上述训练设备130和执行设备140可以为服务器或者终端等计算机设备。其中，服务器可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(Content Delivery Network，CDN)、区块链以及大数据和人工智能平台等基础云计算服务的云服务器。终端可以是智能移动设备、笔记本电脑、车载电脑、台式计算机等。
执行设备140的处理模块141用于执行计算等相关的处理过程，执行设备140可以调用数据存储系统160中的数据、程序等以用于相应的计算处理，并将计算处理得到的处理结果等数据和指令存入数据存储系统150中。例如，执行设备140可以利用目标网络101(也即，训练后的语音增强网络)对目标对象的通话语音进行语音增强得到增强语音，并将增强语音存储至数据存储系统160，进而客户设备150可以由执行设备140的输入/输出(Input/Output，I/O)接口从该数据存储系统160中获取增强语音，以为用户播放增强语音。
可选地，数据库120中维护的训练数据不一定都来自于数据获取设备110，也可以从其他设备接收得到，例如，客户设备150也可以作为数据获取端，进而将客户设备150获取的用户的通过语音经相关的数据处理后作为新的训练数据，并存入数据库120。此外，训练设备130也不一定完全基于数据库120维护的训练数据对预设的神经网络进行训练，也有可能基于从云端或者其他设备获取的训练数据对预设的神经网络进行训练，例如，训练设备130可以从互联网平台获取语音处理技术相关的语音数据集，上述描述不应该作为对本申请实施例的限定。
需要说明的是，图1仅是本申请实施例提供的一种系统架构示意图，本申请实施例描述的系统架构以及应用场景是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定。例如，图1中的数据存储系统160相对执行设备150是外部存储器，在其它情况下，也可以将数据存储系统160置于执行设备150内，又例如，客户设备150也可以直接作为执行设备，部署训练后的目标网络用于执行语音增强的任务。
请参阅图2，图2示出一种系统部署示意图。示例性地，本申请实施例提供的语音增强方法与相关方法还可以部署在如图2所示的系统中。该系统包括终端240、互联网230、网关220、服务器210等。
终端240可以包括桌面电脑、膝上型电脑、个人数字助理(Personal Digital Assistant，PDA)、智能手机、专用终端等多种形式。另外，它可以是单台设备，也可以是多台设备组成的集合。终端240可以以有线或无线的方式与互联网230进行通信，交换数据。例如，终端240可以通过无线路由器250与互联网230进行通信。
服务器210是指能对终端240提供某些服务的计算机系统。相对于普通终端240来说，服务器210在稳定性、安全性、性能等方面都要求更高。服务器210可以是网络平台中的一台高性能计算机、多台高性能计算机的集群、一台高性能计算机中划出的一部分(例如，虚拟机)、多台高性能计算机中划出的一部分(例如，虚拟机)的组合等。
网关220又称网间连接器、协议转换器。网关在传输层上实现网络互连，是一种充当转换作用的计算机系统或设备。在使用不同的通信协议、数据格式或语言，甚至体系结构完全不同的两种系统之间，网关是一个翻译器。同时，网关也可以提供过滤和安全功能。终端240向服务器210发送的消息可以通过网关220传输到相应的服务器210。服务器210向终端240发送的消息也可以通过网关220传输到相应的终端240。
本申请实施例的语音增强方法与相关方法可以完全在终端240实现，也可以完全在服务器210实现，还可以一部分在终端240实现，另一部分在服务器210实现。
作为一种实施方式，语音增强方法与相关方法可以完全在终端240实现。示例性地，终端240可以作为训练设备基于训练集在本地训练出声纹提取网络和语音增强网络。终端240也可以作为执行设备在本地利用声纹提取网络和语音增强网络为待增强语音进行语言增强，得到针对目标对象的目标增强语音，具体地，终端240可以利用声纹提取网络对目标对象的参考语言进行声纹提取，得到对应的声纹特征，进一步地，通过语音增强网络基于该声纹特征对应待增强语音进行语言增强，得到目标增强语音。
作为另一种实施方式，语音增强方法与相关方法可以完全在服务器210实现。示例性地，服务器210可以作为训练设备基于训练集训练出声纹提取网络和语音增强网络。服务器210也可以作为执行设备利用声纹提取网络和语音增强网络为待增强语音进行语言增强，得到针对目标对象的目标增强语音，具体地，服务器210可以接收终端240发送的目标对象的参考语音和待增强语音，进而，服务器210可以利用声纹提取网络对目标对象的参考语言进行声纹提取，得到对应的声纹特征，进一步地，通过语音增强网络基于该声纹特征对应待增强语音进行语言增强，得到目标增强语音，并将目标增强语音返回终端240。
作为又一种实施方式，语音增强方法与相关方法可以一部分在终端240实现，另一部分在服务器210实现。示例性地，服务器210可以作为训练设备基于训练集训练出声纹提取网络和语音增强网络。终端240可以作为执行设备在本地部署服务器210训练出的声纹提取网络和语音增强网络，进而利用该声纹提取网络和语音增强网络为待增强语音进行语言增强，得到针对目标对象的目标增强语音。
需要说明的是，图2仅是本申请实施例提供的一种系统部署示意图。本申请实施例描述的系统部署方案仅是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定。例如，终端240可以泛指多个终端中的一个，本实施例仅以图2中示出的三个终端来举例说明。本领域普通技术人员可知，随着系统部署方案的演变，本发明实施例提供的技术方案对于类似的技术问题，同样适用。
本申请实施例可以应用在多种场景下，如图3所示的线上会议场景、图4所示的游戏场景等。
(1)线上会议场景
线上会议是一种基于云计算技术的远程会议方式，通过互联网将多个地理位置分散的参与者连接在一起，实现实时音视频交流、文件共享和协作。云会议系统不需要昂贵的硬件设备和复杂的安装过程，只需在具有互联网连接的设备上安装相应的应用程序或通过浏览器访问，就能轻松启动和参与会议。在线上会议中，音视频数据被压缩并加密后，通过云服务器传输到各个参与者的设备。云计算的高效处理能力和大规模存储支持，使得云会议能够在保证高质量的音视频传输的同时，支持大规模的参与者同步交流。本申请实施例提供的语音增强方法与相关方法可以应用在线上会议场景中，为会议中的语音信息进行语音增强。
如图3所示，在线上会议场景中，会议服务方所提供的线上会议服务可以应用在服务端210、终端241和终端242。其中，服务端210可以包括多台高性能的云服务器，例如，训练云服务器、处理云服务器和客户云服务器。训练云服务器可以用于根据训练集训练得到声纹提取网络和语音增强网络。处理云服务器可以用于部署训练后的声纹提取网络和语音增强网络，用于执行语音增强任务。客户云服务器可以直接与多个终端，如终端241和终端242进行网络通信。
示例性地，客户云服务器可以接收来自终端的账户登录请求，语音增强请求，还可以执行消息分发等操作，如将处理云服务器得到的目标增强语音返回给对应的终端。终端241和终端242可以安装由会议服务方所提供的会议客户端，会议客户端可以实现音视频交流、屏幕共享等功能。在用户使用会议客户端进行语音交流时，说话人所在终端可以将采集到的目标语音传送到服务端，由服务端对该目标语音进行语音增强，并将增强后的目标增强语音发送至参会的其他终端。
例如，参会者260在会议中使用终端242发言时，终端242可以自动采集一段语音作为参考语音，发送至服务端210，由服务端210基于声纹提取网络对应该参考语音进行声纹提取，得到对应的目标声纹特征。进一步地，终端242可以实时采集参会者260发言时的语音，也即待增强语音，并将待增强语音传送至服务端210，进而，服务端210可以通过语音增强网络基于目标声纹特征对待增强语音进行语音增强，得到增强后的目标增强语音，并将该目标增强语音分发至其他参会人的终端，例如，终端241，进而，其他参会人可以听到经语音增强后的参会者260的发言。
(2)游戏语音场景
游戏语音在现代多人游戏中扮演着至关重要的角色，是玩家之间进行实时沟通和协作的核心工具。在复杂的游戏环境中，尤其是团队竞技游戏、策略游戏和角色扮演游戏中，快速且清晰的语音交流直接影响到游戏体验和玩家的表现。通过游戏内语音，玩家能够即时分享信息、制定战术、协调动作，甚至传达情绪和氛围，从而提升团队合作和战略执行的效率。背景噪音消除和回声抑制等技术的应用，使得游戏语音环境更加清晰和纯净，避免外界干扰。随着游戏语音技术的不断发展，玩家的游戏体验得到了显著提升。本申请实施例提供的语音增强方法与相关方法可以应用在游戏语音场景中，为游戏玩家间的语音交流提供语音增强的服务。
如图4所示，游戏终端241和游戏终端242都部署了游戏软件，该游戏软件包括了基于本申请的语音增强方法与相关方法开发的语音交流功能。在游戏场景中，游戏玩家之间进行语音沟通时，玩家所在终端可以采集玩家发出的声音并在本地进行语音增强，之后，将增强后的语音发送至其他玩家的终端。示例性地，由于终端242所在环境为网络娱乐场景，通常，终端242采集到的玩家语音中会混合有其他人的干扰音，为此，终端242在将玩家语音发送给其他玩家时，会在本地对采集的玩家语音进行语音增强。例如，玩家使用终端242在游戏中发言时，终端242可以自动采集一段语音作为参考语音，并基于声纹提取网络对应该参考语音进行声纹提取，得到对应的目标声纹特征。进一步地，终端242可以实时采集玩家发言时的语音，也即待增强语音，并通过语音增强网络基于目标声纹特征对待增强语音进行语音增强，得到增强后的目标增强语音，并将该目标增强语音分发至其他游戏玩家的终端。
需要说明的是，图3和图4仅是本申请实施例提供的两种应用场景示意图。本申请实施例描述的应用场景仅是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定。本申请提出的语音增强方法与相关方法还可以用于其他的通话场景：如通话降噪、语音识别前端、直播点播应用等场景。具体的：
在通信降噪场景中，电话通话中可能存在噪声干扰，影响通话质量，通过训练后的声纹提取网络和语音增强网络，可以有效地减少噪声，提升语音的清晰度和可懂性，从而改善通话质量；在语音识别前端场景中，例如手机智能语音助手、车载语音助手等，语音前端的噪声处理是一个重要的环节，通过运用训练后的声纹提取网络和语音增强网络，可以减少噪声对语音识别的负面影响，提高语音识别的准确性和稳定性；在直播点播应用中，音频质量对用户体验至关重要，通过应用训练后的声纹提取网络和语音增强网络，可以提升音频的清晰度和质量，减少噪声干扰，使用户获得更好的听觉体验。本领域普通技术人员可知，随着应用场景的演变，本发明实施例提供的技术方案对于类似的技术问题，同样适用。
根据本申请的一个实施例，提供了一种语音增强方法。该语音增强方法可以由计算机设备(如，服务器、边缘计算设备或其他具备一定处理能力的终端设备等)执行，该计算机设备至少具有存储、计算和通信的功能。图5示出了本实施例提供的语音增强方法的流程示意图。如图5所示，该语音增强方法具体可以包括：
S110：在语音通话的状态下，获取目标对象的参考语音，并提取参考语音对应每帧语音信号的参考音频特征。
通常，有关技术的语音增强方法主要依赖于信号处理或深度学习模型等技术，这些方法在多说话人场景下无法有效区分不同说话人的声音特征，导致增强后的语音中可能包含其他干扰说话人的语音。考虑到每个人的声纹特征中隐含着自己说话声音的独特信息，不同人有自己唯一的声纹特征，利用声纹特征的唯一性可以对不同声音进行区分。为此，本申请提出基于无感采集的参考语音进行语音增强，以便利用由参考语音提取的声纹特征来对待增强语音进行更加精准地语音增强。
本申请之所以采用无感地采集参考语音，这是由于，提前录制参考语音对录制环境的要求较为苛刻且录制过程繁琐，并且一旦待增强语音对应的说话人发生改变时，针对待增强语音进行的语音增强就会无效，需要重新录制参考语音，这将给用户的实时通话带来不便。其中，目标对象是指被采集声音的说话人。语音通话的状态是指在通话场景中，目标对象与其他对象正在进行语音通话的状态。参考语音是指待进行声纹特征提取的语音，参考语音的时长可以预先设定。
例如，在目标对象利用社交软件的通话功能进行语音通话的场景中，若检测到对方已接听，并且检测到目标对象可以与对方进行语音交流，则可以自动采集目标对象的一段20秒的语音作为参考语音。由于，原始的语音信号是一个复杂的、连续变化的时域信号，直接从中提取声纹特征往往难以反映语音的实际内容，为此需要对参考语音进行分帧、加窗以及时频转换等语音处理的步骤，以确保从参考语音的多帧时域信号中提取出频域下对应的具有代表性的多帧的音频特征，也即参考音频特征，用于后续的声纹提取。
在一些实施例中，语音处理可以包括对参考语音的语音信号分别进行分帧、加窗和时频转换等处理步骤，进而，从参考语音中捕捉和分析出有用的参考音频特征，以便声纹特征的提取。具体可以包括：
(1)对参考语音进行音质检测，得到参考语音的语音信噪比。
考虑到如果通话环境很嘈杂会影响最终生成声纹特征的质量，所以可以对采集到的参考语音进行音质检测，当检测结果指示参考语音满足后续生成参考音频特征的条件时，再执行后续的语音处理。否则，重新采集参考语音，直至采集的参考语音能够通过音质检测。
作为一种实施方式，音质检测可以是对参考语音进行的信噪比(Signal to Noise Ratio，SNR)计算，信噪比用于衡量语音信号强度和噪声信号强度的相对大小。示例性地，可以对参考语音的语音信号进行信号分析，得到语音信号中的噪声信号，进一步地，基于功率或者基于幅度对语音信号和噪声信号进行信噪比计算，计算公式可以如下：
其中，SNRp表示基于信号功率计算的信噪比。Psignal表示语音信号对应的信号功率，Pnoise表示噪声信号对应的信号功率。可选地，还可以利用语音信号和噪声信号的幅度计算信噪比，计算公式可以如下：
其中，SNRm表示基于信号幅度计算的信噪比。Msignal表示语音信号对应的均方幅度，Pnoise表示噪声信号对应的均方幅度。在计算得到信噪比时，可以将信噪比用分贝(dB)表示，得到语音信噪比：SNRdB＝10log(SNRp or SNRm)。
(2)当语音信噪比大于信噪比阈值时，对参考语音进行分帧操作，得到多个语音帧信号。
其中，信噪比阈值用于决策下一步执行语音处理或者执行重新采集参考语音的关键值。信噪比阈值可以基于具体的使用场景进行信噪比实验，并根据实验值确定。分帧操作是指将参考语音的语音信号分成若干小段，每一段称为一个帧。通常，语音信号是一个连续的时间波形信号，但它的特性会随着时间变化。为了便于分析，可以将长时间的语音信号分成若干帧，而每个帧内的信号可以被认为是短时平稳的。
作为一种实施方式，当语音信噪比大于信噪比阈值时，可以将参考语音的语音信号分成预设长度的多个帧信号。相邻的帧信号通常会有一定的重叠(例如，50％～60％)，以确保帧与帧之间的连续性。这意味着在处理长时间信号时，每个帧之间会有一部分是共享的。例如，若采样频率为fs，每帧长度为T，则帧长度N＝fs×T采样点。帧长度可以为20毫秒或25毫秒。当语音信噪比不大于信噪比阈值时，可以重新采集参考语音。
(3)基于窗函数对每个语音帧信号分别进行加窗操作，得到每个语音帧信号对应的加窗帧信号。
由于直接截取语音信号的帧会导致频率失真，称为“频谱泄漏”(Spectral Leakage)。为了减少边缘效应，每个帧在进行频域转换前需要乘以一个窗函数，也即，加窗操作。其中，窗函数可以包括汉明窗(Hamming Window)、汉宁窗(Hanning Window)、黑曼窗(Blackman Window)等。加窗目的是在帧信号的边界逐渐衰减信号的幅度，以减少边界不连续性对频域分析的影响，计算公式可以如下：
y[n]＝x[n]·w
其中，y[n]表示加窗后的加窗帧信号，x[n]表示语音帧信号，w表示窗函数，n表示第n帧。
(4)对每个加窗帧信号分别进行时频转换，得到每个加窗帧信号在频域下对应的参考音频特征。
其中，时频转换是指将加窗帧信号由时域信号转换为频域信号，语音信号中的大多数信息都与频率有关。通过将时域信号转换到频域可以更有效地分析信号的频谱特性，得到参考音频特征，该参考音频特征即为信号的频谱特征。时频转换可以包括傅里叶变换(Fourier Transform)及其变体，例如快速傅里叶变换(Fourier Transform Fast)。
作为一种实施方式，可以对每个加窗帧信号分别进行快速傅里叶变换，得到每个加窗帧信号在频域下对应的频谱特征作为参考音频特征。其中，频谱特征可以包括功率谱(Power Spectrum)、相位谱(Phase Spectrum)或者梅尔频率倒谱系数(Mel-Frequency Cepstral Coefficients,MFCC)等，在此不做限定。
S120：基于声纹提取网络对每个参考音频特征进行声纹提取，得到每个参考音频特征对应的帧级声纹特征。
为了更好的说明本实施例，以下对网络训练过程进行示例性介绍：声纹提取网络是指用于进行声纹提取的深度神经网络。声纹提取网络的输入为每个参考音频特征，输出为每个参考音频特征对应的帧级声纹特征。语音增强网络是指用于进行语音增强的深度神经网络。语音增强网络的输入为待增强音频特征和目标声纹特征，输出为目标增强音频特征。声纹提取网络和语音增强网络可以为基于预测增强音频特征和干净音频特征确定的目标损失训练得到，该预测增强音频特征为语音增强网络对样本带噪语音和样本参考语音对应的预测声纹特征经语音增强得到，样本带噪语音由与样本参考语音的声纹匹配的样本干净语音及环境噪声语音经语音混合得到。
具体地，训练样本中的样本干净语音和样本参考语音来源于同一样本对象，所以，样本干净语音的声纹和样本参考语音的声纹是匹配的，来源于同一样本对象。在网络训练过程中，可以基于声纹提取网络对样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征，并对样本带噪语音进行特征提取，得到对应的样本带噪音频特征，进而将样本带噪音频特征和预测声纹特征输入至语音增强网络进行语音增强，输出样本对象对应的预测增强音频特征，并根据预测增强音频特征和样本干净语音对应的干净音频特征确定目标损失，进而基于目标损失对声纹提取网络和语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和训练后语音增强网络。以此，可以利用声纹提取网络进行声纹提取以及利用训练后语音增强网络进行语音增强。
为了保证语音增强的过程不会影响到语音通话，本申请提出利用声纹提取网络提取每个参考音频特征对应的帧级声纹特征。其中，每一个帧级声纹特征都是由参考语音在频域下每一个帧信号对应的参考音频特征经声纹提取得到。这是由于，利用有感方式采集参考语音进行声纹提取时，声纹提取网络是对一段长语音，如约30秒的参考语音进行的声纹提取，通常需要较长的时间，如1至2秒，使得瞬时的CPU占用比较高，很消耗计算资源，由于CPU的计算资源是有限的，进而会影响语音通话和网络通信等其他功能的处理。
考虑到语音信号是由一系列帧组成的连续信号，如此，声纹特征可以在每个帧上提取，并且最终的声纹特征可以通过合并这些帧级特征来构建。出于这一动机，本申请提出采用帧级声纹提取的方法，分别对每个帧信号对应的参考音频特征进行声纹提取，将一个段级别的声纹提取转换为多个帧级别的声纹提取，从而，避免因声纹提取的瞬时计算资源消耗过大，导致通话和通信等其他任务的执行。
由于，语音增强常用于实时应用，例如，电话会议、网络电话(VoIP)、语音助手、直播分享以及智能家居设备等。这些应用要求处理语音信号的延迟极低，以确保目标对象能够即时地交流和互动，并且，语音信号通常包含敏感的目标对象的信息和对话内容等，所以，语音增强可能在终端本地执行，从而避免网络引起的延迟，最大限度地保护目标对象的隐私。然而，终端设备通常拥有有限的计算资源，尤其是与云服务器相比。这些设备的CPU和GPU资源相对较少，因此，本申请提出使用计算要求较低的轻量级神经网络进行声纹提取。
作为一种实施方式，声纹提取网络可以是由提取层、扩展压缩层和表示层构成深度神经网络。其中，提取层可以由一个标准的3×3卷积层组成，扩展压缩层可以由多个反向残差模块组成，表示层可以由全局平均池化层、全连接层和最终的输出层组成。具体地：
(1)将每个参考音频特征输入至提取层进行声纹特征提取，得到多个参考声纹特征。
示例性地，提取层可以是由卷积神经网络的第一个卷积层，用于对输入的参考音频特征中进行下采样，以降低后续层的计算负担和内存需求，提取层可以使用H-Swish激活函数，以提高非线性能力和特征表达力，最终输出每个参考音频特征对应的参考声纹特征。
(2)将每个参考声纹特征输入至扩展压缩层进行特征扩展压缩，得到多个深度声纹特征。
示例性地，扩展压缩层的多个反向残差模块负责对每个参考声纹特征进行特征再提取和表示，通过扩展和压缩参考声纹特征的维度，利用较少的计算量来实现高效的特征提取，最终得到每个参考声纹特征对应的深度声纹特征。
(3)将每个深度声纹特征输入至表示层进行声纹特征表示，得到每个参考音频特征对应的帧级声纹特征。
示例性地，表示层用于生成帧级声纹特征，全局平均池化层可以将每个深度声纹特征平均为单个值，转换为一个全局特征向量，进而全连接层将全局特征向量映射到表示空间，并由输出层输出每个参考音频特征对应的帧级声纹特征。
通过由提取层、扩展压缩层和表示层构声纹提取网络成的声纹提取网络进行声纹特征提取时，由于提取层显著减少了网络参数数量和计算复杂度，扩展压缩层通过先扩展然后再压缩特征来进行计算，使得在保持高效特征提取的同时，减少了计算开销，使得声纹提取网络具有轻量级的网络架构，进而终端设备在处理声纹提取时能够更高效地完成声纹提取任务。进而，利用轻量级的声纹提取网络逐帧进行声纹特征提取可以将计算负担平摊到每一帧的特征提取过程中，例如，一般来说需要1至2秒钟的时间对一段约30秒的参考语音进行声纹提取，这种方式很耗计算资源，使得瞬时的中央处理器的占用率比较高，会影响通信的其他功能，而将一段约30秒的参考语音分成10帧3秒的帧级语音进行声纹提取，可以将30秒的计算量平摊到10帧3秒计算上，在降低计算时长的同时，使得瞬时的中央处理器的占用率大幅降低，如此，可以减少瞬时的计算负荷，优化终端设备的资源使用。从而声纹提取网络能够在终端设备的有限的计算资源、内存和存储空间下运行，同时满足实时处理和低延迟的要求，并且减少了对网络带宽和连接稳定性的依赖。
作为另一种实施方式，声纹提取网络还可以是由长短期记忆网络和全连接层组成的深度神经网络。语音特征具有短时平稳性的时序序列，与长短期记忆网络的长短期记忆能力相吻合，提高语音增强质量。可选地，长短期记忆网络也可以为双向长短期记忆网络(Bi-directional Long-Short Term Memory，Bi-LSTM)，在此不做限定。
S130：根据每个帧级声纹特征进行声纹聚类，得到目标对象对应的目标声纹特征。
考虑到采集的参考语音中除了目标对应的声音外，还可能包括其他干扰的噪声，例如，其他人说话的声音以及噪声等。为此，在得到帧级声纹特征之后，可以通过声纹聚类，从帧级声纹特征中除掉其他干扰的噪声，从而，提高生成目标对象对应的声纹特征的精确度。
本申请实施例基于声纹特征间的特征相似度，作为声纹聚类的依据，针对通过声纹聚类得到的多个聚类簇来说，处于同一个聚类簇中的声纹特征彼此相似，与其他聚类簇中的声纹特征相异。由于目标声纹特征是基于目标对象的参考语音得到的，通过声纹聚类得到的目标声纹特征可用于对目标对象发出的语音进行语音增强。而且，由于本申请实施例采用的是无感方式，在语音通话中采集参考语音来确定目标声纹特征，故目标声纹特征能够更好的对该语音通话中目标对象发出的语音(在得到目标声纹特征之后)进行语音增强，以此提升所输出目标对象的语音质量。
在本申请实施例中，可以先对所有的帧级声纹特征进行池化操作，得到多个片级声纹特征，进一步地，再对多个片级声纹特征进行声纹聚类，得到目标对象对应的目标声纹特征。具体地：
(1)根据每个帧级声纹特征进行池化操作，得到多个片级声纹特征。
其中，池化操作是指将每个较短时长的帧级声纹特征组合成多个相对于帧级声纹特征时长来说，具有中等时长的片级声纹特征。例如，将4个25毫秒的帧信息对应的帧级声纹特征，通过池化操作可以得到一个1秒的片级声纹特征。
这样做的目的是，通常一个较短时长的帧级声纹特征带有的声纹信息相对较少，而一个片级声纹特征可以捕捉到一个中等时间段内的声纹信息，从而提高声纹特征的聚类准确性。在一些实施例中，步骤(1)可以包括：
(1.1)获取池化片长；
(1.2)基于池化片长对每个帧级声纹特征进行分组，得到多个声纹特征组；
(1.3)对每个声纹特征组中的多个帧级声纹特征进行池化操作，得到每个声纹特征组对应的片级声纹特征。
其中，池化片长是指片级声纹特征对应的音频时长，池化片长可以根据实际场景需要，通过实验确定，例如，1秒。声纹特征组是经平均分组后，由多个帧级声纹特征构成的特征组。池化操作可以包括平均池化和最大池化。在一特殊场景下，由于总的帧级声纹特征的数量无法整除池化片长，也即，无法进行平均分组，此时，可以对最小特征数量对应的最小声纹特征组进行补充空特征，使得该最小声纹特征组中的帧级声纹特征数量与其他声纹特征组的帧级声纹特征数量相同，实现整除，空特征是指每个特征值均为零的特征。
作为一种实施方式，平均池化可以是针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值，并对不同声纹特征组中的各个特征维度对应的特征值分别进行均值计算，得到每个声纹特征组在每个特征维度对应的平均特征值，进一步地，将每个声纹特征组在每个特征维度对应的平均特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
示例性地，每个声纹特征组中有N个帧级声纹特征，每个帧级声纹特征的特征维度为D。若一个声纹特征组中的第i个帧级声纹特征可以表示为[ei,1,ei,2,…,ei,D]，则该声纹特征组对应的片级声纹特征可以表示为：
其中，表示片级声纹特征的第j个特征维度的特征值是其对应的声纹特征组中所有帧级声纹特征的第j个维度的特征值的平均值。
通过平均池化可以有效地捕捉到一个较长时间段内的声纹特征，也即片级声纹特征，例如，将40个25毫秒的帧级声纹特征变成了一个表示1秒的片级声纹特征，这个片级声纹特征具有与帧级声纹特征相同的维度D，但它代表了整个1秒钟的声纹特征，而不是单个帧的声纹特征，从而保存了这一长时间段内的关键的声纹信息，提供了一致的特征表示，能够在不同的片级声纹特征之间保持相对一致的特征分布。进而，基于一致的特征分布可以进行更准确的声纹聚类。为了提高片级声纹特征的生成效率，也可以采用最大池化的方式来生成片级声纹特征。
作为另一种实施方式，最大池化可以是针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值，并对不同声纹特征组中的各个特征维度对应的特征值分别进行最值选取，得到每个声纹特征组在每个特征维度对应的最大特征值，进一步地，将每个声纹特征组在每个特征维度对应的最大特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
示例性地，每个声纹特征组中有N个帧级声纹特征，每个帧级声纹特征的维度为D。若一个声纹特征组中的第i个帧级声纹特征可以表示为[ei,1,ei,2,…,ei,D]，则该声纹特征组对应的片级声纹特征可以表示为：
Emax＝[max(e1,1,e1,2,…,e1,D),max(e2,1,e2,2,…,e2,D),…,max(eN,1,eN,2,…,eN,D)]
其中，Emax表示片级声纹特征，该片级声纹特征中每个维度的特征值，都是其对应的声纹特征组中所有帧级声纹特征在相同维度上的特征值的最大值，例如，由10个帧级声纹特征经最大池化得到的一个片级声纹特征，该片级声纹特征在第一个维度上的特征值是这10个帧级声纹特征对应第一个维度的特征值中最大的特征值。
通过最大池化可以直接保留帧级声纹特征中最显著的特征或尖峰，无需对每个维度的特征值进行均值计算，从而提高片级声纹特征的生成效率，并且在声纹聚类中，可以有助于更准确地分辨出具有明显声纹特征，从而提高声纹聚类的准确性。
考虑到最大池化在突出显著声纹特征和抗噪性方面有优势，但可能会丢失全局声纹信息和受到极端值的影响。而平均池化在保留全局声纹信息和减少噪声影响方面更有优势，但可能无法有效突出明显的声纹特征。为此，可以利用结合平均池化和最大池化的层级池化生成片级声纹特征，以保留明显的声纹特征和全局声纹信息，从而为后续的声纹聚类任务提供更丰富和有代表性的声纹特征。
作为又一种实施方式，层级池化可以是针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值，进一步地，对不同声纹特征组中的各个特征维度对应的特征值分别进行均值计算，得到每个声纹特征组在每个特征维度对应的平均特征值，进一步地，将每个声纹特征组在每个特征维度对应的平均特征值，分别作为不同声纹特征组的中间片级声纹特征在对应特征维度上的特征值，进一步地，对不同声纹特征组的中间片级声纹特征中的各个特征维度对应的特征值分别进行最值选取，得到每个中间片级声纹特征在每个特征维度对应的最大特征值，并将每个声纹特征组在每个特征维度对应的最大特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
例如，一个声纹特征组中有40个25ms的帧级声纹特征，每个帧级声纹特征是一个维度为D的特征向量。先对40个帧级声纹特征进行平均池化，得到一个维度为D的特征向量，也即中间片级声纹特征Emax。进一步地，对中间片级声纹特征Emax进行最大池化，得到一个维度为D的特征向量，也即，该声纹特征组对应的片级声纹特征E。
通过层级池化，可以先对多个帧级声纹特征进行最大池化，以捕捉信号的尖峰特征，也即明显的声纹特征，从而反映局部声纹特征的显著性，进而再对最大池化的片级的中间声纹特征进行平均池化，得到最终的片级声纹特征，从而反映全局声纹特征的平滑性。如此，通过逐层池化，可以在不同尺度上逐步保留局部声纹特征和全局声纹特征，从而实现对声纹特征的局部细节和全局趋势的平衡，保留声纹信号的多级别信息。使得每一层的池化操作都能够捕捉到不同层次上的重要特征，得到更高质量的片级声纹特征，以便后续进行更准确的声纹聚类。
(2)基于多个片级声纹特征进行声纹聚类，得到多个声纹聚类簇。
在一些实施例中，声纹聚类是指将多个片级声纹特征基于声纹特征相似度组成的多个不同聚类簇的过程。每个聚类簇是一组片级声纹特征的集合，这些片级声纹特征与同一个簇中的片级声纹特征彼此相似，与其他簇中的片级声纹特征相异。声纹聚类的方法可以包括K-均值聚类、围绕中心点的划分(Partitioning Around Medoid)聚类以及层次聚类(Hierarchical Clustering)等，在此不做限定。
作为一种实施方式，可以多个片级声纹特征进行自底向上的层次聚类。具体地，选择一个合适的距离度量来衡量片级声纹特征之间的相似性。常用的距离度量包括欧氏距离(Euclidean distance)、余弦相似度(Cosine similarity)等。距离度量的选择对聚类结果有重要影响，需根据应用场景选择最能反映特征间相似性的度量。将每个片级声纹特征作为一个单独的簇。设初始状态下有n个片级声纹特征，则初始状态下有n个聚类簇，每个聚类簇只包含一个片级声纹特征。进一步地，计算并构建一个距离矩阵，其中每个元素表示两个聚类簇之间的距离(或相似度)。在距离矩阵中找到距离最近的两个聚类簇(即距离最小的两个聚类簇)，并合并这两个最近的聚类簇成一个新的聚类簇。
进一步地，更新簇的集合，从原来的n个聚类簇减少到n-1个聚类簇。根据新的聚类簇的结构，更新距离矩阵中的距离值。例如，单链接(single linkage):新聚类簇与其他聚类簇的距离为新聚类簇中任意一个元素与其他聚类簇中任意一个元素之间的最小距离。完全链接(complete linkage):新聚类簇与其他聚类簇的距离为新聚类簇中任意一个元素与其他聚类簇中任意一个元素之间的最大距离。
进一步地，重复寻找最近的聚类簇、合并最近的聚类簇以及更新距离矩阵的步骤，直到所有的聚类簇被合并成一个聚类簇，或达到预设的停止条件(例如聚类簇的数量达到预定值，或聚类簇的结构满足一定的相似度标准)，在上述层次聚类中，记录每一次合并的过程，构建一个树形结构，即聚类树(或称为树状图)。聚类树的根节点代表所有片级声纹特征最终被合并成的一个大簇，而叶节点代表初始的每个片级声纹特征。进而通过对聚类树进行分割，得到多个声纹聚类簇。
(3)通过从多个声纹聚类簇中选取的目标声纹聚类簇，确定目标对象对应的目标声纹特征。
其中，目标声纹聚类簇中的片级声纹特征的总体质量满足语音增强条件。
由于从多个声纹聚类簇中选取的目标声纹聚类簇是用于确定目标声纹特征，即后续进行语音增强的基准特征的，故在进行选取时，为了保证后续语音增强的质量，需要选择能够满足语音增强条件的作为目标声纹聚类簇。
语音增强条件用于衡量是否能够用于语音增强，而且评价的对象是声纹聚类簇中的整体片级声纹特征，而不是单个片级声纹特征，这样可以有效筛选出高质量声纹特征聚集的声纹聚类簇，排除单个声纹特征异常的影响。
在一些实施例中，可以已通过聚类簇属性从多个声纹聚类簇中选取的目标声纹聚类簇，以便基于目标声纹特征生成目标对象对应的目标声纹特征。聚类簇属性至少可以包括簇密度和簇半径等。具体地，步骤(3)可以包括：
(3.1)获取每个声纹聚类簇对应的聚类簇属性；
(3.1)根据每个声纹聚类簇对应的聚类簇属性，确定目标声纹聚类簇；
其中，聚类簇属性是指可用于描述和分析聚类簇的属性，例如，簇密度、簇半径、簇直径、簇体积或者簇内的平均距离等。考虑到较高的簇密度意味着簇内存在更多的数据点，声音能量越大，而目标对象的声音往往比噪音携带的能量大。并且较小的簇半径说明簇内点的紧密程度高，表示簇内的数据一致性较强，属于同一说话人的声纹信息较集中，有利更好聚类，所以，在本申请实施例中，选择簇密度和簇半径作为确定目标声纹聚类簇所使用的聚类簇属性。
由于目标聚类簇的聚类属性会显著区别于其他声纹聚类簇，而聚类属性的分布可以通过聚类属性的方差进行量化，具体地，方差越小说明聚类属性的分布越均匀，而聚类属性的分布不能太均匀，也即方差不能太小。因此，可以通过判断聚类属性的分布是否过于均衡，来确定目标聚类簇。
作为一种实施方式，可以基于每个声纹聚类簇对应的簇密度和簇半径进行方差计算，得到簇密度对应的密度方差和簇半径对应的半径方差。当密度方差大于密度方差阈值并且半径方差大于半径方差阈值时，将簇密度大于其他聚类簇的簇密度并且簇半径小于其他聚类簇的簇半径的聚类簇，作为目标声纹聚类簇。当密度方差不大于密度方差阈值或者半径方差不大于半径方差阈值时，重新获取目标对象的参考语音，直至密度方差大于密度方差阈值并且半径方差大于半径方差阈值，得到目标声纹聚类簇。
示例性地，计算声纹聚类簇对应的簇半径，即为计算声纹聚类簇中片级声纹特征到簇中心的最大距离，或其他统计距离(如平均距离)。例如，对于第i个声纹聚类簇Ci和其质心μi(簇中心)，其对应的簇半径为其中，d(x,μi)表示片级声纹特征x到质心μi的距离。计算声纹聚类簇对应的簇密度，即为片级声纹特征的数量与簇的体积或范围之间的比值。例如，先根据簇半径为Ri计算出第i个声纹聚类簇Ci的簇体积Vi，进一步地，簇密度ρi＝|Ci|/Vi，其中，|Ci|表示声纹聚类簇Ci中片级声纹特征的数量。
通过对每个声纹聚类簇对应的簇密度和簇半径进行方差计算，可以得到簇密度对应的密度方差和簇半径对应的半径方差，进而将密度方差和密度方差阈值进行比较，并将半径方差和半径方差阈值进行比较，当密度方差大于密度方差阈值且半径方差大于半径方差阈值时，表示聚类簇分布的不是太均匀，所以存在一些聚类簇的聚类属性会显著区别于其他声纹聚类簇，进而可以将聚类属性显著区别于其他声纹聚类簇的一些声纹聚类簇筛选出来，再从筛选出来这些声纹聚类簇中，将簇密度大于其他聚类簇的簇密度并且簇半径小于其他聚类簇的簇半径的聚类簇，作为目标声纹聚类簇。由于簇密度较大且簇半径较小的聚类簇，通常意味着簇内的点之间有较高的相似性，簇内的点距离中心较近，簇内的点分布在一个小的空间范围内。对声纹数据来说，这意味着这些声纹特征之间的特征非常相近，属于相同的说话者，进而确定出更为准确的目标声纹聚类簇，以便对应生成更加准确的目标声纹特征。
(3.2)对目标声纹聚类簇中的多个片级声纹特征进行平均计算，得到目标对象对应的目标声纹特征。
作为一种实施方式，在从多个声纹聚类簇中确定出目标声纹聚类簇之后，得到将目标声纹聚类簇中的多个片级声纹特征进行特征向量的平均值计算，具体地，通过对每个特征向量的各个分量分别计算其算术平均值。例如，目标声纹聚类簇中有n个d维度的片级声纹特征，记作(v1,v2,…,vn)，每个片级声纹特征可以表示为vi＝(vi1,vi2,…,vid)，多个片级声纹特征的平均值也是一个d维的片级声纹特征vavg，它的每个分量是所有片级声纹特征在该维度上的平均值，也即：
请参阅图6，图6示出一种目标声纹特征的生成流程图。在通话状态下，获取目标对象的参考语音，进而可以对参考语音进行音质检测，当音质检测通过时，对参考语音进行音频特征提取，具体地，如图7所示的音频特征提取流程图。可以先对参考语音在时域下的语音信号进行分帧、加窗和快速傅里叶变换(FFT)，得到参考语音在频域下的多个帧级音频特征。进一步地，对多个帧级音频特征进行池化操作得到多个片级音频特征，并对多个片级音频特征进行声纹聚类，得到目标声纹特征。如此，基于参考语音对应的帧级声纹特征生成的目标声纹特征，对待增强语音进行可抑制干扰人声的语音增强，以提高语音增强的效率和准确性。
S140：将待增强语音的待增强音频特征和目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定目标增强音频特征对应的目标增强语音。
其中，待增强语音为语音通话中目标对象发出的语音。也就是说，本申请实施例可以在语音通话状态的初期，通过无感方式先确定出本次语音通话中目标对象的目标声纹特征，然后在得到目标声纹特征后，在语音通话的后续时间内，通过目标声纹特征对目标对象继续说出的待增强语音进行语音增强，提高目标对象的语音质量。
也就是说，在一种可能的实现方式中，本申请实施例相当于将语音通话划分为两个阶段，第一阶段是S110中提及的语音通话刚开始的时段，该第一阶段涉及前述S110-S130，用于采集目标对象的语音来生成目标声纹特征，以便为该语音通话中目标对象的语音进行语音增强。当通过S130得到目标声纹特征时，意味着该语音通话的第一阶段结束，该语音通话的第二阶段开始。其中，第一阶段的时长与处理效率、精度要求相关，例如可以是10s。
该语音通话的第二阶段是对目标对象的语音进行语音增强的阶段，通过在第一阶段获取的目标声纹特征，对该语音通话中目标对象的语音进行语音增强。
示例性的，假设该语音通话有20分钟，从语音通话开始时就无感的获取目标对象的语音来生成目标声纹特征，当目标声纹特征生成时，耗时5秒，那么相当于从该语音通话的第0秒到第5秒，为前述的第一阶段，从第6秒到第20分钟，为前述的第二阶段。
可选的，语音增强网络为根据预测增强音频特征和干净音频特征确定的目标损失训练得到，预测增强音频特征为语音增强网络对样本带噪语音和样本参考语音对应的预测声纹特征经语音增强得到，样本带噪语音由与样本参考语音的声纹匹配的样本干净语音及环境噪声语音经语音混合得到。
在一些实施例中，如图8所示的语音增强流程图，可以提取待增强语音的待增强音频特征，进而通过语音增强网络基于目标声纹特征对待增强音频特征进行语音增强，以生成去除干扰人和其他噪声的目标增强音频特征，进而对目标增强音频特征进行语音重构，得到目标对象对应的目标增强语音。具体可以包括：
(1)对获取的待增强语音进行特征提取，得到待增强音频特征。
作为一种实施方式，可以对待增强语音在时域下的语音信号依次进行分帧、加窗和快速傅里叶变换，得到待增强语音在频域下的多个音频特征，进一步地，由多个音频特征构建一个音频特征向量作为待增强音频特征。
(2)将待增强音频特征和目标声纹特征输入至语音增强网络，由语音增强网络基于目标声纹特征对待增强音频特征进行语音增强，得到目标增强音频特征。
作为一种实施方式，语音增强网络的网络架构可以和声纹提取网络的网络架构相同，例如，语音增强网络可以由多个长短期记忆网络和全连接层组成，将待增强音频特征和目标声纹特征输入至第一个长短期记忆网络，进而长短期记忆网络的输出作为下一长短期记忆网络输入，直至全连接层输出目标增强音频特征。
(3)对目标增强音频特征进行语音重构，得到目标增强语音。
作为一种实施方式，目标增强音频特征是频域特征，所以，需要对目标增强音频特征进行语音重构，示例性地，利用提取音频特征时的视频转换的逆过程(如，逆快速傅里叶变换)对目标增强音频特征进行语音重构，得到时域下的目标对象的目标增强语音。
由于声纹特征是特定于个体的特征，用于识别或验证说话者的身份。它捕捉了说话者的独特语音特性，如发音方式、声调、节奏、声带特征等。而音频特征是描述音频信号的物理和感知特征。语音增强网络通过目标声纹特征对待增强音频特征进行语音增强，得到具有目标对象的独特语音特性的目标声纹特征，通过对目标声纹特征进行语音重构得到针对目标对象的语音增强。
本实施例可以在语音通话状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征，进一步地，基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征。如此，通过对无感采集的参考语音利用帧级声纹提取的方法，可以对参考语音的每个帧信号对应的参考音频特征分别进行声纹提取，通过将有关技术基于段级别的声纹特征提取，替换为基于帧级别的声纹特征提取，使得一段声纹特征提取所需的较大计算资源消耗，能够平均释放到多帧声纹特征提取的过程中，从而，避免因声纹提取的计算资源消耗过大，影响通话和通信等其他任务的执行，提高声纹特征提取的效率和稳定性。进一步地，根据稳定且高效提取的每个帧级声纹特征进行声纹聚类，可以快速得到目标对象对应的目标声纹特征，并将获取的待增强语音的待增强音频特征和目标声纹特征输入至语音增强网络，输出目标增强音频特征，进一步地，确定目标增强音频特征对应的目标增强语音。由此，能够利用无感、高效且稳定提取的目标声纹特征表示目标对象的独特语音特性，进而更加快速地从待增强音频特征中去除与目标对象的独特语音特性无关的其他声音信息，包括干扰人的声音信息和噪声信息等，从而提高语音增强的速度、稳定性和准确度。
根据本申请的另一个实施例，提供了一种语音增强网络训练方法。该语音增强网络训练方法可以在上述实施例的步骤120之前，由计算机设备(如，服务器、边缘计算设备或其他具备一定处理能力的终端设备等)执行，该计算机设备至少具有存储、计算和通信的功能。请参阅图9和图10，图9示出了本实施例提供的语音增强网络训练方法的流程示意图。图10示出一种语音增强网络训练的架构图。下面将结合图9与图10对该语音增强网络训练方法的具体步骤进行说明：
如图10所示，基于目标损失可以对声纹提取网络和语音增强网络进行迭代网格训练。基于声纹提取网络可以对样本参考语音进行声纹提取，得到预测声纹特征。对样本带噪语音进行特征提取，得到对应的样本带噪音频特征，并将样本带噪音频特征和预测声纹特征输入至语音增强网络进行语音增强，输出预测增强音频特征，进而，根据预测增强音频特征和样本干净语音对应的干净音频特征确定目标损失，样本带噪语音是由环境噪声语音和样本干净语音经语音混合得到。具体地：
S210：获取训练样本，训练样本包括样本干净语音、样本参考语音和样本带噪语音。
在语音增强技术的使用场景中，需要进行语音增强的待增强语音通常包括干扰人声。例如，网络课堂客户端采集的某位同学的说话声音中，除了环境噪声之外，还会包括其他同学的说话声音，也即干扰人声。为此，可以基于带有同一声纹信息的样本参考语音和样本带噪语音来训练声纹提取网络和语音增强网络。网络训练的目标是语音增强网络可以基于声纹提取网络生成的预测声纹特征对样本带噪音频特征进行语音增强，能够从样本带噪语音抽离出样本干净语音，也即，预测增强音频特征和样本干净语音对应的干净音频特征之间的误差越来越小。
其中，样本带噪语音是由环境噪声语音和样本干净语音经语音混合得到，训练样本中的样本干净语音和样本参考语音来源于同一样本对象。样本干净语音是指没有受到噪声干扰的清晰语音。
作为一种实施方式，样本干净语音可以是录音得到或者从预先建立的语音数据库中获取得到的。样本干净语音是一个纯净样本，为了获取高质量的纯净样本，通常会尽可能避免背景噪声的存在，可以使用专业的麦克风进行录制，同时保持良好的声音质量和语音内容的多样性。
作为一种实施方式，环境噪声语音是指受到不同类型噪声干扰的语音。这些语音可以从真实世界中采集的，例如通过在不同环境下使用麦克风记录日常生活中的背景噪声，也可以从噪声数据库中提取，如模拟噪声、车内噪声、咖啡厅环境噪声等环境噪音。进一步的，收集的噪声语音应该涵盖多样的环境和噪声类型，以便训练模型具有更好的泛化能力。
作为一种实施方式，样本带噪语音是在得到样本干净语音和环境噪声语音后，将二者混合之后生成的语音，也就是在样本干净语音的基础上，混合了一些噪声，从而使得纯净样本带噪。样本带噪语音可以用于模拟实际场景中的语音，在通话、视频会议、语音识别前端、直播点播应用等场景中，设备获取的语音一般是带有噪声的，通过混合形成样本带噪语音，可以用于后续的网络训练。
示例性地，环境噪声语音可以包括干扰人声和背景噪声。样本带噪语音可以是对干扰人声、背景噪声和样本干净语音进行不同比例的混合得到，可选地，样本带噪语音可以按照如下公式如下：
Smix＝αS1+βS2+γS3
其中，Smix表示样本带噪语音，S1表示干扰人声，S2表示背景噪声，S3表示样本干净语音。α,β,γ是比例参数，α,β,γ∈(0,1)且α+β+γ＝1。值得注意的是，在同一个训练样本中，样本参考语音的样本对象和样本干净语音的样本对象是同一个人，样本参考语音和样本干净语音的对象和干扰人声的样本对象是不同的人。
样本参考语音和样本干净语音的语音内容是可以不同的，例如，样本参考语音和样本干净语音的内容是同一样本对象阅读的两篇不同的新闻。语音内容可以仅可能多的包括不同发音的字词，使得样本参考语音和样本干净语音覆盖更多的语音信息，提高网络训练的准确率和置信度。当然，在其他实施例中，同一训练样本中的样本参考语音和样本干净语音的语音内容也可以是相同的。对声纹提取网络和语音增强网络进行训练时，可以从数据库中获取训练集{x0,x2,…,xn}。其中，n>0&n∈N*。n为训练样本个数。
S220：基于初始声纹提取网络对样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征。
作为一种实施方式，在获取样本参考语音后，可以对样本参考语音进行分帧操作和加窗操作，之后，进行时频转换，得到对应的频域特征。具体地，依次对麦克风采集的样本参考语音进行分帧和加窗，得到样本参考语音的帧信号，并对帧信号进行快速傅里叶变换，并将快速傅里叶变换之后的频域特征作为样本参考语音的样本参考音频特征。
进一步地，将样本参考音频特征输入至声纹提取网络对样本参考语音对应的样本参考音频特征进行声纹提取。如图11所示的声纹提取网络的框架图。声纹提取网络包括提取层、扩展压缩层和表示层。将每个样本参考音频特征输入至提取层进行声纹特征提取，得到多个样本参考声纹特征，并将每个样本参考声纹特征输入至扩展压缩层进行特征扩展压缩，得到多个样本深度声纹特征，进而将每个样本深度声纹特征输入至表示层进行声纹特征表示，得到预测声纹特征。
S230：对样本带噪语音进行特征提取，得到对应的样本带噪音频特征。
作为一种实施方式，可以对样本带噪语音进行分帧操作和加窗操作，之后，进行时频转换，得到对应的频域特征作为样本带噪音频特征，例如，对数功率谱(Logarithmic Power Spectrum，LPS)和梅尔频率倒谱系数(Mel-Frequency Cepstral Coefficients，MFCC)等，在此不做限定。
这是由于语音数据往往不能像图像数据那样直接输入到模型中训练，其在长时域上没有明显的特征变化，所以很难学习到语音数据的特征，加之语音的时域数据通常由16K采样率构成，即1秒16000个采样点，直接输入时域采样点会导致训练数据量过大且很难训练出具有实际意义的效果。因此，在语音增强任务中，可以将语音数据转化为声学特征作为网络的输入或者输出。
S240：将样本带噪音频特征和预测声纹特征输入至初始语音增强网络进行语音增强，输出样本对象对应的预测增强音频特征。
作为一种实施方式，语音增强网络的网络架构可以和声纹提取网络的网络架构相同。示例性地，语音增强网络包括提取层、扩展压缩层和表示层。将样本带噪音频特征和预测声纹特征输入至语音增强网络，依次经过提取层、扩展压缩层和表示层进行语音增强，得到预测增强音频特征。
S250：根据预测增强音频特征和样本干净语音对应的干净音频特征确定目标损失，基于目标损失对初始声纹提取网络和初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
其中，训练结束条件可以包括：目标损失的损失值小于预设值、目标损失的损失值不再变化、或者训练次数达到预设次数等。可选的，可以采用优化器去优化目标损失函数，基于实验经验设置学习率、训练时的Batch Size(批量大小)训练的Epoch(时期)。
在一些实施例中，对预测增强音频特征和样本干净语音对应的干净音频特征进行误差计算，根据计算结果确定目标损失，进一步地，基于目标损失对初始声纹提取网络和初始语音增强网络进行迭代训练，直至训练至少达到目标损失的损失值小于预设值、目标损失的损失值处于阈值范围或者训练次数达到预设次数的结果条件，得到声纹提取网络和语音增强网络。
作为一种实施方式，可以通过计算预测增强音频特征和样本干净语音对应的干净音频特征的均方误差(Mean Square Error，MSE)来作为初始声纹提取网络和初始语音增强网络的目标损失，计算公式如下：
其中，Loss(τ)表示目标损失，τ表示初始声纹提取网络和初始语音增强网络的权重参数，yi表示第i个训练样本对应的预测增强音频特征，表示第i个训练样本对应的样本干净语音的干净音频特征(ground truth)。在其他实施例中，还可以通过其他损失函数(例如交叉熵损失函数、绝对值损失函数等)来基于预测增强音频特征和样本干净语音对应的干净音频特征，确定目标损失。可选地，可以通过至少两种不同的损失函数来分别根据预测增强音频特征和干净音频特征来分别计算在每一种损失函数下的损失，之后，将在至少两种不同的损失函数下的损失进行加权处理，将加权处理结果作为目标损失。
考虑到直接预测增强音频特征和干净音频特征的误差构建目标损失来训练网络，这往往导致训练过程生成的预测增强音频特征过于平滑，缺乏细节。虽然这种方法可以有效地减少整体误差，但它在捕捉语音感知上重要的细节和质感方面表现较弱。并且，由于过分关注局部的点对点误差，例如，MSE损失容易导致增强后的语音中产生伪影，这些伪影可能在频谱中表现为不自然的噪声或失真。为此，本申请提供基于生成对抗网络来训练初始声纹提取网络和初始语音增强网络，从而，提升增强语音的质量和自然度。
作为另一种实施方式，可以将初始声纹提取网络和初始语音增强网络作为生成器(Generator)，并构建一个判别器(Discriminator)，该判别器的网络结构可以为卷积神经网络、循环神经网络或者Transformer模型等，在此不做限定。进一步地，由生成器和判别器组成生成对抗网络，在训练该生成对抗网络的过程中，通过生成器和判别器之间的对抗过程，迫使生成器生成的输出越来越真实。生成器试图生成尽可能真实的增强后的预测增强音频特征，而判别器则试图区分预测增强音频特征和真实的干净音频特征。这种动态对抗过程促使生成器在质量和细节上逐步提升，超越单纯依赖MSE的点对点误差最小化。
具体地，固定生成器的网络参数，基于预测增强音频特征和样本干净语音对应的干净音频特征确定的第一损失，对判别器进行网络参数迭代更新，固定判别器的网络参数，基于估测增强音频特征确定的第二损失，对生成器进行网络参数迭代更新。其中，估测增强音频特征为基于其他训练样本，生成的预测增强音频特征。
示例性地，将预测增强音频特征输入至判别器进行真实性判定，得到预测增强音频特征对应的第一判别结果，并将样本干净语音对应的干净音频特征输入至语音判别器进行真实性判定，得到干净音频特征对应的第二判别结果，进而根据第一判别结果和第二判别结果，确定第一损失函数，并固定生成器的网络参数，基于第一损失函数更新语音判别器的网络参数。
例如，干净音频特征表示为预测增强音频特征表示yi＝G(xi,ei)，其中，xi表示样本带噪音频特征，ei表示预测声纹特征，G(·)表示由初始声纹提取网络和初始语音增强网络构成的生成器，D(·)表示判别器。第一损失函数可以为：
其中，log D(1-yi)表示为第一判别结果，表示为第二判别结果。
进一步地，基于生成器的初始声纹提取网络提取新的样本参考语音对应的估测声纹特征，并将新的样本带噪语音的样本带噪音频特征和估测声纹特征输入至生成器的初始语音增强网络进行语音增强，输出估测增强音频特征。进而，将估测增强音频特征输入至判别器进行真实性判定，得到估测增强音频特征对应的第三判别结果，基于第三判别结果，确定第二损失函数。固定语音判别器的网络参数，基于第二损失函数更新初始声纹提取网络和初始语音增强网络的网络参数。第二损失函数可以为：
其中，log(D(yi))表示为第三判别结果。
通过先固定生成器的网络参数，基于第一损失对判别器进行网络参数更新，再固定判别器的网络参数，基于第二损失对生成器进行网络参数更新，经过多次迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。如此，使用生成对抗网络的方法训练声纹提取网络和语音增强网络，由声纹提取网络和语音增强网络组成的生成器目标在于生成接近真实增强目标语音特征的输出，而判别器则通过对比生成语音特征和真实语音特征来提高其判别能力。
这种方法不断迭代优化生成器和判别器，从而使生成器能够生成更加逼真和高质量的增强语音。通过这种生成对抗式的迭代训练，判别器可以学习并关注音频信号中的全局结构和复杂的时频模式，这使得生成器在生成语音特征时，能够更好地学习到更丰富和表达力更强的语音特征，从而生成更符合人耳感知的高质量的目标增强语音。尤其适用于对多样化环境下复杂信号的语音增强任务。可选地，在网络训练结束时，可以将训练得到的声纹提取网络和语音增强网络与现有的用于语音增强的深度神经网络，以及有感录制参考语音进行的语音增强方法进行性能比较。
比较指标可以使用噪声平均选项得分(Noise Mean Option Score，NMOS)。NMOS的值越大，表明语音增强的性能越好。如下表1所示的性能比较结果：
表1
由上表1观察可知，本申请提出的基于无感采集参考语音进行语音增强的方法，与基于深度神经网络进行语音增强的方法相比，语音增强算法有大幅提升，和基于有感声录制参考语音进行语音增强的方法相比，语音增强算法也有明显的提升，这是由于本申请降低了生成声纹向量所需的计算资源，并在生成声纹特征时，通过聚类方式，剔除干扰语音，提高了声纹特征的质量，从而，在利用该声纹特征进行高效地语音增强，得到高质量的增强语音。
请参阅图12，其示出了本申请实施例提供的一种语音增强装置300的结构框图。该装置300可以包括：
语音获取模块310，用于在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征；
声纹提取模块320，用于基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征；
声纹聚类模块330，用于根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征；
语音增强模块340，用于将待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音，所述待增强语音为所述语音通话中目标对象发出的。
在一些实施例中，声纹聚类模块330可以包括池化操作单元、声纹聚类单元和声纹确定单元。
池化操作单元，用于根据每个帧级声纹特征进行池化操作，得到多个片级声纹特征；
声纹聚类单元，用于基于所述多个片级声纹特征进行声纹聚类，得到多个声纹聚类簇；
声纹确定单元，用于通过从所述多个声纹聚类簇中选取的目标声纹聚类簇，确定所述目标对象对应的目标声纹特征，所述目标声纹聚类簇中的片级声纹特征的总体质量满足语音增强条件。
在一些实施例中，池化操作单元可以包括片长子单元、分组子单元和池化子单元。
片长子单元，用于获取池化片长；
分组子单元，用于基于所述池化片长对所述每个帧级声纹特征进行分组，得到多个声纹特征组；
池化子单元，用于对每个声纹特征组中的多个帧级声纹特征进行池化操作，得到所述每个声纹特征组对应的片级声纹特征。
在一些实施例中，池化子单元可以具体用于：针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值；对不同声纹特征组中的各个特征维度对应的特征值分别进行均值计算，得到每个声纹特征组在每个特征维度对应的平均特征值；将每个声纹特征组在每个特征维度对应的平均特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
在一些实施例中，池化子单元还可以具体用于：针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值；对不同声纹特征组中的各个特征维度对应的特征值分别进行最值选取，得到每个声纹特征组在每个特征维度对应的最大特征值；将每个声纹特征组在每个特征维度对应的最大特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
在一些实施例中，声纹确定单元可以包括属性子单元、聚类子单元和声纹子单元。
属性子单元，用于获取每个声纹聚类簇对应的聚类簇属性；
聚类子单元，用于根据所述每个声纹聚类簇对应的聚类簇属性，确定目标声纹聚类簇；
声纹子单元，用于对所述目标声纹聚类簇中的多个片级声纹特征进行平均计算，得到所述目标对象对应的目标声纹特征。
在一些实施例中，所述聚类簇属性至少包括簇密度和簇半径，聚类子单元可以具体用于：基于所述每个声纹聚类簇对应的簇密度和簇半径进行方差计算，得到簇密度对应的密度方差和簇半径对应的半径方差；当所述密度方差大于密度方差阈值并且所述半径方差大于半径方差阈值时，将簇密度大于其他聚类簇的簇密度并且簇半径小于其他聚类簇的簇半径的聚类簇，作为目标声纹聚类簇；当所述密度方差不大于密度方差阈值或者所述半径方差不大于半径方差阈值时，重新获取目标对象的参考语音，直至所述密度方差大于密度方差阈值并且所述半径方差大于半径方差阈值，得到目标声纹聚类簇。
在一些实施例中，语音增强模块340可以具体用于：对获取的待增强语音进行特征提取，得到待增强音频特征；将所述待增强音频特征和所述目标声纹特征输入至语音增强网络，由所述语音增强网络基于所述目标声纹特征对所述待增强音频特征进行语音增强，得到目标增强音频特征；对所述目标增强音频特征进行语音重构，得到目标增强语音。
在一些实施例中，所述声纹提取网络包括提取层、扩展压缩层和表示层，所述声纹提取模块320可以具体用于：将所述每个参考音频特征输入至所述提取层进行声纹特征提取，得到多个参考声纹特征；将每个参考声纹特征输入至所述扩展压缩层进行特征扩展压缩，得到多个深度声纹特征；将每个深度声纹特征输入至所述表示层进行声纹特征表示，得到所述每个参考音频特征对应的帧级声纹特征。
在一些实施例中，语音获取模块310可以具体用于：对所述参考语音进行音质检测，得到所述参考语音的语音信噪比；当所述语音信噪比大于信噪比阈值时，对所述参考语音进行分帧操作，得到多个语音帧信号；基于窗函数对每个语音帧信号分别进行加窗操作，得到每个语音帧信号对应的加窗帧信号；对每个加窗帧信号分别进行时频转换，得到每个加窗帧信号在频域下对应的参考音频特征。
本实施例可以在语音通话状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征，进一步地，基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征。相对于有关技术利用有感采集的参考语音进行语音增强的方案而言，本申请在语音通话状态下，通过对无感采集的参考语音利用帧级声纹提取的方法，可以对参考语音的每个帧信号对应的参考音频特征分别进行声纹提取，通过将有关技术基于段级别的声纹特征提取，替换为基于帧级别的声纹特征提取，使得一段声纹特征提取所需的较大计算资源消耗，能够平均释放到多帧声纹特征提取的过程中，从而，避免因声纹提取的计算资源消耗过大，影响通话和通信等其他任务的执行，提高声纹特征提取的效率和稳定性。进一步地，根据稳定且高效提取的每个帧级声纹特征进行声纹聚类，可以快速得到目标对象对应的目标声纹特征，并将获取的待增强语音的待增强音频特征和目标声纹特征输入至语音增强网络，输出目标增强音频特征，进一步地，确定目标增强音频特征对应的目标增强语音。相对于有关技术利用有感采集的参考语音进行语音增强的方案而言，本申请能够利用无感、高效且稳定提取的目标声纹特征表示目标对象的独特语音特性，进而更加快速地从待增强音频特征中去除与目标对象的独特语音特性无关的其他声音信息，包括干扰人的声音信息和噪声信息等，从而提高语音增强的速度、稳定性和准确度。
请参阅图13，其示出了本申请实施例提供的一种语音增强网络训练装置400的结构框图。该装置400可以包括：
样本获取模块410，用于获取训练样本，所述训练样本包括样本干净语音、样本参考语音和样本带噪语音，所述样本带噪语音是由环境噪声语音和所述样本干净语音经语音混合得到，其中，训练样本中的样本干净语音和样本参考语音来源于同一样本对象；
特征预测模块420，用于基于初始声纹提取网络对所述样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征；
音频提取模块430，用于对所述样本带噪语音进行特征提取，得到对应的样本带噪音频特征；
音频预测模块440，用于将所述样本带噪音频特征和所述预测声纹特征输入至初始语音增强网络进行语音增强，输出所述样本对象对应的预测增强音频特征；
网络训练模块450，用于根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述声纹提取网络和所述语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
在一些实施例中，网络训练模块450可以具体用于：对所述预测增强音频特征和所述样本干净语音对应的干净音频特征进行误差计算，根据计算结果确定目标损失；基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至训练至少达到目标损失的损失值小于预设值、目标损失的损失值处于阈值范围或者训练次数达到预设次数的结果条件，得到声纹提取网络和语音增强网络。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述装置、模块等的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，模块相互之间的耦合可以是电性，机械或者其它形式的耦合。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
如图14所示，本申请实施例还提供一种计算机设备500，该计算机设备500包括处理器510、存储器520、电源530和输入单元540，存储器520存储有计算机程序，该计算机程序被处理器510调用时，可实执行上述实施例提供的各种方法步骤。本领域技术人员可以理解，图中示出的计算机设备的结构并不构成对计算机设备的限定，可以包括比图示更多或者更少的部件，或者组合某些部件，或者不同的部件布置。其中：
处理器510可以包括一个或多个处理核。处理器510利用各种接口和线路连接整个电池管理系统内的各种部分，通过运行或执行存储在存储器520内的指令、程序、指令集或程序集，调用存储在存储器520内的数据，执行电池管理系统的各种功能和处理数据，以及执行计算机设备的各种功能和处理数据，从而对计算机设备进行整体控制。可选地，处理器510可以采用数字信号处理(Digital Signal Processing，DSP)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)、可编程逻辑阵列(Programmable Logic Array，PLA)中的至少一种硬件形式来实现。处理器510可集成中央处理器(Central Processing Unit，CPU)、图像处理器(Graphics Processing Unit，GPU)和调制解调器等中的一种或几种的组合。其中，CPU主要处理操作系统、用户界面和应用程序等；GPU用于负责显示内容的渲染和绘制；调制解调器用于处理无线通信。可以理解的是，上述调制解调器也可以不集成到处理器510中，单独通过一块通信芯片进行实现。
存储器520可以包括随机存储器(Random Access Memory，RAM)，也可以包括只读存储器(Read-Only Memory)。存储器520可用于存储指令、程序、指令集或程序集。存储器520可包括存储程序区和存储数据区，其中，存储程序区可存储用于实现操作系统的指令、用于实现至少一个功能的指令(比如触控功能、声音播放功能、图像播放功能等)、用于实现上述各种方法实施例的指令等。存储数据区还可以存储计算机设备在使用中所创建的数据(比如电话本和音视频数据)等。相应地，存储器520还可以包括存储器控制器，以提供处理器510对存储器520的访问。
电源530可以通过电源管理系统与处理器510逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。电源530还可以包括一个或一个以上的直流或交流电源、再充电系统、电源故障检测电路、电源转换器或者逆变器、电源状态指示器等任意组件。
输入单元540，该输入单元540可用于接收输入的数字或字符信息，以及产生与用户设置以及功能控制有关的键盘、鼠标、操作杆、光学或者轨迹球信号输入。
尽管未示出，计算机设备500还可以包括输出单元、显示单元等，在此不再赘述。具体在本申请实施例中，计算机设备的处理器510会按照如下的指令，将一个或一个以上的计算机程序的进程对应的可执行文件加载到存储器520中，并由处理器510来运行存储在存储器520中的比如电话本和音视频数据，从而实现前述实施例提供的各种方法步骤。
如图15所示，本申请实施例还提供一种计算机可读存储介质600，该计算机可读存储介质600中存储有计算机程序610，计算机程序610可被计算机设备的处理器调用于执行本申请实施例提供的各种方法步骤。
计算机可读存储介质可以是诸如闪存、EEPROM(电可擦除可编程只读存储器)、EPROM、硬盘或者ROM之类的电子存储器。可选地，计算机可读存储介质包括非易失性计算机可读存储介质(Non-Transitory Computer-Readable Storage Medium)。计算机可读存储介质600具有执行上述实施例中任何方法步骤的计算机程序的存储空间。这些计算机程序可以从一个或者多个计算机程序产品中读出或者写入到这一个或者多个计算机程序产品中。计算机程序能够以适当形式进行压缩。
根据本申请的一个方面，提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序被存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机程序，处理器执行该计算机程序，使得该计算机设备执行上述实施例提供的各种方法步骤。
本申请的说明书及上述附图中的术语“包括”和“包含”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或装置不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或装置固有的其它步骤或单元。术语“模块”或“单元”是指有预定功能的计算机程序或计算机程序的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
应了解，在本申请实施例的描述中，多个(或多项)的含义是两个以上，大于、小于、超过等理解为不包括本数，以上、以下、以内等理解为包括本数。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机装置(可以是个人计算机，服务器，或者网络装置等)执行本申请各个实施例方法的全部或部分步骤。还应了解，本申请实施例提供的各种实施方式可以任意进行组合，以实现不同的技术效果。
以上，仅是本申请的较佳实施例而已，并非对本申请作任何形式上的限制，虽然本申请已以较佳实施例揭示如上，然而并非用以限定本申请，任何本领域技术人员，在不脱离本申请技术方案范围内，当可利用上述揭示的技术内容做出些许更动或修饰为等同变化的等效实施例，但凡是未脱离本申请技术方案内容，依据本申请的技术实质对以上实施例所作的任何简介修改、等同变化与修饰，均仍属于本申请技术方案的范围内。

Claims (17)

  1. 一种语音增强方法，所述方法由计算机设备执行，所述方法包括：
    在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征；
    基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征；
    根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征，所述目标声纹特征用于对所述目标对象发出的语音进行语音增强；
    将待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音，所述待增强语音为所述语音通话中所述目标对象发出的。
  2. 根据权利要求1所述的方法，所述根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征，包括：
    根据每个帧级声纹特征进行池化操作，得到多个片级声纹特征；
    基于所述多个片级声纹特征进行声纹聚类，得到多个声纹聚类簇；
    通过从所述多个声纹聚类簇中选取的目标声纹聚类簇，确定所述目标对象对应的目标声纹特征，所述目标声纹聚类簇中的片级声纹特征的总体质量满足语音增强条件。
  3. 根据权利要求2所述的方法，所述根据每个帧级声纹特征进行池化操作，得到多个片级声纹特征，包括：
    获取池化片长；
    基于所述池化片长对所述每个帧级声纹特征进行分组，得到多个声纹特征组；
    对每个声纹特征组中的多个帧级声纹特征进行池化操作，得到所述每个声纹特征组对应的片级声纹特征。
  4. 根据权利要求3所述的方法，所述对每个声纹特征组中的多个帧级声纹特征进行池化操作，得到所述每个声纹特征组对应的片级声纹特征，包括：
    针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值；
    对不同声纹特征组中的各个特征维度对应的特征值分别进行均值计算，得到每个声纹特征组在每个特征维度对应的平均特征值；
    将每个声纹特征组在每个特征维度对应的平均特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
  5. 根据权利要求3所述的方法，所述对每个声纹特征组中的多个帧级声纹特征进行池化操作，得到所述每个声纹特征组对应的片级声纹特征，包括：
    针对每个声纹特征组中的多个帧级声纹特征，获取每个帧级声纹特征中各个特征维度对应的特征值；
    对不同声纹特征组中的各个特征维度对应的特征值分别进行最值选取，得到每个声纹特征组在每个特征维度对应的最大特征值；
    将每个声纹特征组在每个特征维度对应的最大特征值，分别作为不同声纹特征组的片级声纹特征在对应特征维度上的特征值。
  6. 根据权利要求2至5任一项所述的方法，所述通过从所述多个声纹聚类簇中选取的目标声纹聚类簇，确定所述目标对象对应的目标声纹特征，包括：
    获取每个声纹聚类簇对应的聚类簇属性；
    根据所述每个声纹聚类簇对应的聚类簇属性，确定目标声纹聚类簇；
    对所述目标声纹聚类簇中的多个片级声纹特征进行平均计算，得到所述目标对象对应的目标声纹特征。
  7. 根据权利要求6所述的方法，所述聚类簇属性至少包括簇密度和簇半径，所述根据所述每个声纹聚类簇对应的聚类簇属性，确定目标声纹聚类簇，包括：
    基于所述每个声纹聚类簇对应的簇密度和簇半径进行方差计算，得到簇密度对应的密度方差和簇半径对应的半径方差；
    当所述密度方差大于密度方差阈值并且所述半径方差大于半径方差阈值时，将簇密度大于其他聚类簇的簇密度并且簇半径小于其他聚类簇的簇半径的聚类簇，作为目标声纹聚类簇；
    当所述密度方差不大于密度方差阈值或者所述半径方差不大于半径方差阈值时，重新获取目标对象的参考语音，直至所述密度方差大于密度方差阈值并且所述半径方差大于半径方差阈值，得到目标声纹聚类簇。
  8. 根据权利要求1-7任意一项所述的方法，所述将获取的待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音，包括：
    对获取的待增强语音进行特征提取，得到待增强音频特征；
    将所述待增强音频特征和所述目标声纹特征输入至语音增强网络，由所述语音增强网络基于所述目标声纹特征对所述待增强音频特征进行语音增强，得到目标增强音频特征；
    对所述目标增强音频特征进行语音重构，得到目标增强语音。
  9. 根据权利要求1-8任意一项所述的方法，所述声纹提取网络包括提取层、扩展压缩层和表示层，所述基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征，包括：
    将所述每个参考音频特征输入至所述提取层进行声纹特征提取，得到多个参考声纹特征；
    将每个参考声纹特征输入至所述扩展压缩层进行特征扩展压缩，得到多个深度声纹特征；
    将每个深度声纹特征输入至所述表示层进行声纹特征表示，得到所述每个参考音频特征对应的帧级声纹特征。
  10. 根据权利要求1-9任意一项所述的方法，所述提取所述参考语音对应每帧语音信号的参考音频特征，包括：
    对所述参考语音进行音质检测，得到所述参考语音的语音信噪比；
    当所述语音信噪比大于信噪比阈值时，对所述参考语音进行分帧操作，得到多个语音帧信号；
    基于窗函数对每个语音帧信号分别进行加窗操作，得到每个语音帧信号对应的加窗帧信号；
    对每个加窗帧信号分别进行时频转换，得到每个加窗帧信号在频域下对应的参考音频特征。
  11. 一种语音增强网络训练方法，所述方法包括：
    获取训练样本，所述训练样本包括样本干净语音、样本参考语音和样本带噪语音，所述样本带噪语音是由环境噪声语音和所述样本干净语音经语音混合得到，其中，训练样本中的样本干净语音和样本参考语音来源于同一样本对象；
    基于初始声纹提取网络对所述样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征；
    对所述样本带噪语音进行特征提取，得到对应的样本带噪音频特征；
    将所述样本带噪音频特征和所述预测声纹特征输入至初始语音增强网络进行语音增强，输出所述样本对象对应的预测增强音频特征；
    根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
  12. 根据权利要求11所述的方法，所述根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络，包括：
    对所述预测增强音频特征和所述样本干净语音对应的干净音频特征进行误差计算，根据计算结果确定目标损失；
    基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至训练至少达到目标损失的损失值小于预设值、目标损失的损失值处于阈值范围或者训练次数达到预设次数的结果条件，得到声纹提取网络和语音增强网络。
  13. 一种语音增强装置，所述装置包括：
    语音获取模块，用于在语音通话的状态下，获取目标对象的参考语音，并提取所述参考语音对应每帧语音信号的参考音频特征；
    声纹提取模块，用于基于声纹提取网络对每个参考音频特征进行声纹提取，得到所述每个参考音频特征对应的帧级声纹特征；
    声纹聚类模块，用于根据每个帧级声纹特征进行声纹聚类，得到所述目标对象对应的目标声纹特征，所述目标声纹特征用于对所述目标对象发出的语音进行语音增强；
    语音增强模块，用于将待增强语音的待增强音频特征和所述目标声纹特征输入至语音增强网络，输出目标增强音频特征，并确定所述目标增强音频特征对应的目标增强语音，所述待增强语音为所述语音通话中所述目标对象发出的。
  14. 一种语音增强网络训练装置，所述装置包括：
    样本获取模块，用于获取训练样本，所述训练样本包括样本干净语音、样本参考语音和样本带噪语音，所述样本带噪语音是由环境噪声语音和所述样本干净语音经语音混合得到，其中，训练样本中的样本干净语音和样本参考语音来源于同一样本对象；
    特征预测模块，用于基于初始声纹提取网络对所述样本参考语音对应的样本参考音频特征进行声纹提取，得到预测声纹特征；
    音频提取模块，用于对所述样本带噪语音进行特征提取，得到对应的样本带噪音频特征；
    音频预测模块，用于将所述样本带噪音频特征和所述预测声纹特征输入至初始语音增强网络进行语音增强，输出所述样本对象对应的预测增强音频特征；
    网络训练模块，用于根据所述预测增强音频特征和所述样本干净语音对应的干净音频特征确定目标损失，基于所述目标损失对所述初始声纹提取网络和所述初始语音增强网络进行迭代训练，直至达到训练结果条件，得到声纹提取网络和语音增强网络。
  15. 一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，当所述计算机程序被处理器执行时，实现如权利要求1至10任一项所述的语音增强方法，或者如权利要求11至12任一项所述的语音增强网络训练方法。
  16. 一种计算机设备，包括：
    存储器；
    处理器，所述存储器上存储有计算机程序，所述计算机程序被所述处理器执行时，实现如权利要求1至10任一项所述的语音增强方法，或者如权利要求11至12任一项所述的语音增强网络训练方法。
  17. 一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时，实现如权利要求1至10任一项所述的语音增强方法，或者如权利要求11至12任一项所述的语音增强网络训练方法。
PCT/CN2025/098737 2024-07-10 2025-06-03 语音增强方法与相关方法、装置、设备、介质和程序产品 Pending WO2026012012A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410923934.6 2024-07-10
CN202410923934.6A CN121331149A (zh) 2024-07-10 2024-07-10 语音增强方法与相关方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2026012012A1 true WO2026012012A1 (zh) 2026-01-15

Family

ID=98337034

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/098737 Pending WO2026012012A1 (zh) 2024-07-10 2025-06-03 语音增强方法与相关方法、装置、设备、介质和程序产品

Country Status (2)

Country Link
CN (1) CN121331149A (zh)
WO (1) WO2026012012A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112435684A (zh) * 2020-11-03 2021-03-02 中电金信软件有限公司 语音分离方法、装置、计算机设备和存储介质
CN112687276A (zh) * 2021-03-11 2021-04-20 北京世纪好未来教育科技有限公司 一种音频信号处理方法、装置及存储介质
CN113851136A (zh) * 2021-09-26 2021-12-28 平安科技（深圳）有限公司 基于聚类的说话人识别方法、装置、设备及存储介质
US20220084509A1 (en) * 2020-09-14 2022-03-17 Pindrop Security, Inc. Speaker specific speech enhancement
CN116153328A (zh) * 2023-02-10 2023-05-23 京东科技信息技术有限公司 一种音频数据处理方法、系统、存储介质及电子设备
CN116741193A (zh) * 2023-08-09 2023-09-12 腾讯科技（深圳）有限公司 语音增强网络的训练方法、装置、存储介质及计算机设备
CN117059068A (zh) * 2022-05-07 2023-11-14 腾讯科技（深圳）有限公司 语音处理方法、装置、存储介质及计算机设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220084509A1 (en) * 2020-09-14 2022-03-17 Pindrop Security, Inc. Speaker specific speech enhancement
CN112435684A (zh) * 2020-11-03 2021-03-02 中电金信软件有限公司 语音分离方法、装置、计算机设备和存储介质
CN112687276A (zh) * 2021-03-11 2021-04-20 北京世纪好未来教育科技有限公司 一种音频信号处理方法、装置及存储介质
CN113851136A (zh) * 2021-09-26 2021-12-28 平安科技（深圳）有限公司 基于聚类的说话人识别方法、装置、设备及存储介质
CN117059068A (zh) * 2022-05-07 2023-11-14 腾讯科技（深圳）有限公司 语音处理方法、装置、存储介质及计算机设备
CN116153328A (zh) * 2023-02-10 2023-05-23 京东科技信息技术有限公司 一种音频数据处理方法、系统、存储介质及电子设备
CN116741193A (zh) * 2023-08-09 2023-09-12 腾讯科技（深圳）有限公司 语音增强网络的训练方法、装置、存储介质及计算机设备

Also Published As

Publication number Publication date
CN121331149A (zh) 2026-01-13

Similar Documents

Publication Publication Date Title
US11894014B2 (en) Audio-visual speech separation
US12293770B2 (en) Voice signal dereverberation processing method and apparatus, computer device and storage medium
CN108899044B (zh) 语音信号处理方法及装置
US9640194B1 (en) Noise suppression for speech processing based on machine-learning mask estimation
CN112309365B (zh) 语音合成模型的训练方法、装置、存储介质以及电子设备
US20240290338A1 (en) Speech processing
US20260004788A1 (en) Method and apparatus for performing speech enhancement, storage medium, device, and product
CN114338623B (zh) 音频的处理方法、装置、设备及介质
CN111951823A (zh) 一种音频处理方法、装置、设备及介质
US20240005908A1 (en) Acoustic environment profile estimation
CN111722696B (zh) 用于低功耗设备的语音数据处理方法和装置
CN114333912B (zh) 语音激活检测方法、装置、电子设备和存储介质
CN115083440A (zh) 音频信号降噪方法、电子设备和存储介质
Chen et al. Conan's Bow Tie: A Streaming Voice Conversion for Real-Time VTuber Livestreaming
CN118609608A (zh) 音频处理系统和应用中使用话音活动检测的降噪
CN113571075A (zh) 音频处理的方法、装置、电子设备和存储介质
CN113571082A (zh) 语音通话的控制方法、装置、计算机可读介质及电子设备
CN115064177A (zh) 基于声纹编码器的语音转换方法、装置、设备及介质
HK40092286A (zh) 语音增强网络的训练方法、装置、存储介质及计算机设备
HK40092286B (zh) 语音增强网络的训练方法、装置、存储介质及计算机设备
CN121331149A (zh) 语音增强方法与相关方法、装置、设备及存储介质
EP4552120B1 (en) Selective noise suppression for speech data in device communication
HK40069959B (zh) 音频的处理方法、装置、设备及介质
Debes et al. CRUSE-cMGAN: a Conditional Metric GAN Framework for Real-Time Speech Enhancement Algorithms
CN112750456A (zh) 即时通信应用中的语音数据处理方法、装置及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25835915

Country of ref document: EP

Kind code of ref document: A1