WO2022012195A1 - 音频信号处理方法和相关装置 - Google Patents

音频信号处理方法和相关装置 Download PDF

Info

Publication number
WO2022012195A1
WO2022012195A1 PCT/CN2021/097663 CN2021097663W WO2022012195A1 WO 2022012195 A1 WO2022012195 A1 WO 2022012195A1 CN 2021097663 W CN2021097663 W CN 2021097663W WO 2022012195 A1 WO2022012195 A1 WO 2022012195A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio signal
signal
frequency domain
audio
domain signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/097663
Other languages
English (en)
French (fr)
Inventor
梁俊斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Publication of WO2022012195A1 publication Critical patent/WO2022012195A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters

Definitions

  • the embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to audio signal processing.
  • the voice receiving device needs to perform voice enhancement processing on the received voice signal.
  • the algorithms for voice enhancement processing for voice receiving devices of different manufacturers, different kernel versions, and different application software may be different.
  • the speech signal after speech enhancement processing may have different degrees of speech impairment.
  • the compensation strategy is set based on the measured distortion results, and then on the equipment corresponding to the device model and software version, the spectrum gain compensation is performed through the set compensation strategy.
  • an embodiment of the present application provides an audio signal processing method, the method includes:
  • the first audio signal is processed through a spectrum compensation model to obtain a prediction result of predictive compensation for the distorted spectrum in the first audio signal;
  • the neural network model obtained by training the original audio samples corresponding to the audio samples;
  • the first audio signal is reconstructed to obtain a target audio signal after repairing the distorted spectrum in the first audio signal.
  • an embodiment of the present application provides an audio signal processing apparatus, and the apparatus includes:
  • a signal acquisition module for acquiring the first audio signal
  • a result acquisition module configured to process the first audio signal through a spectrum compensation model to obtain a prediction result of performing prediction compensation on the distorted spectrum in the first audio signal;
  • the spectrum compensation model is to use the spectrum distorted audio sample , and the neural network model obtained by training the original audio samples corresponding to the spectrally distorted audio samples;
  • a target acquisition module configured to reconstruct the first audio signal according to the prediction result, and obtain a target audio signal after repairing the distorted spectrum in the first audio signal.
  • an embodiment of the present application provides a computer device, the computer device includes a processor and a memory, and the memory stores at least one instruction, at least a piece of program, code set or instruction set, so The at least one instruction, the at least one piece of program, the code set or the instruction set are loaded and executed by the processor to implement the audio signal processing method of the above aspect.
  • an embodiment of the present application provides a computer-readable storage medium, where the storage medium is used for storing a computer program, and the computer program is used for executing the audio signal processing method in the above aspect.
  • a computer program product or computer program where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, and causes the computer device to perform the audio signal processing method of the above-mentioned aspect.
  • FIG. 1 is a framework diagram of model training and prediction compensation according to an exemplary embodiment
  • FIG. 2 is a model architecture diagram of a machine learning model according to an exemplary embodiment embodiment
  • FIG. 3 is a schematic diagram of an audio signal processing method according to an exemplary embodiment
  • FIG. 4 is a schematic diagram of an audio signal processing method according to an exemplary embodiment
  • FIG. 5 is an architectural diagram of an audio signal processing method according to an exemplary embodiment
  • FIG. 6 is a schematic diagram of a voice restoration applied in a voice call system according to an exemplary embodiment
  • FIG. 7 is a schematic diagram of another voice restoration applied in a voice call system according to an exemplary embodiment
  • FIG. 8 is a structural block diagram of an audio signal processing method according to an exemplary embodiment
  • Fig. 9 is a schematic structural diagram of a computer device according to an exemplary embodiment.
  • the spectral compensation model is obtained by training the spectrally distorted audio samples and the original audio samples corresponding to the spectrally distorted audio samples.
  • Speech enhancement processing In the application scenario of a voice call, the recording signal collected by the device needs to undergo voice enhancement processing, which mainly includes: echo cancellation, noise suppression, volume self-adjustment, frequency response equalization, etc.
  • the voice enhancement processing can be implemented through hardware, for example, voice enhancement processing can be implemented through some audio chips, or a respective voice enhancement processing function module can be added at the application layer.
  • the embodiments of the present application involve the application of the speech enhancement processing technology, for example, the first audio signal is obtained through the speech enhancement processing technology.
  • the type of damage can include high-frequency damage.
  • the high-frequency damage can be the high-frequency damage of the voice signal caused by the noise reduction or echo cancellation algorithm, which is manifested in that the original high-frequency information in the voice signal is significantly weakened, which causes the sound corresponding to the voice signal to become dull and unclear;
  • the type of damage may include frequency band damage, which means that signals in certain fixed frequency bands are fixedly attenuated, which may be caused by poor equalization processing, resulting in obvious distortion of the sound corresponding to the speech signal.
  • FIG. 1 is a framework diagram of model training and prediction compensation according to an exemplary embodiment.
  • the model training device 110 trains an end-to-end machine learning model through a pre-prepared sample set including original audio samples and their corresponding spectrally distorted audio samples.
  • the prediction compensation stage predicting The device 120 directly predicts, according to the trained machine learning model and the input first audio signal, a prediction result of performing prediction compensation on the distortion spectrum in the first audio signal.
  • the above-mentioned model training device 110 and prediction device 120 may be computer devices with machine learning capabilities.
  • the computer devices may be stationary computer devices such as personal computers and servers, or the computer devices may also be tablet computers or electronic devices.
  • Mobile computer equipment such as book readers.
  • the model training device 110 and the prediction device 120 may be the same device, or the model training device 110 and the prediction device 120 may also be different devices.
  • the model training device 110 and the prediction device 120 may be the same type of device, for example, the model training device 110 and the prediction device 120 may both be personal computers;
  • the training device 110 and the prediction device 120 may also be different types of devices, for example, the model training device 110 may be a server, and the prediction device 120 may be a mobile terminal device or the like.
  • the embodiments of the present application do not limit the specific types of the model training device 110 and the prediction device 120 .
  • Fig. 2 is a model architecture diagram of a machine learning model according to an exemplary embodiment.
  • the machine learning model 20 in this embodiment of the present application may include a sample set for generating samples and storing samples, and a spectral compensation model 210 part, wherein the sample set stores the original audio samples obtained through collection and the corresponding Spectrally distorted audio samples, or original audio samples and corresponding spectrally distorted audio samples generated by artificial construction.
  • Each spectrally distorted audio sample stored in the sample set is input into the spectral compensation model 210, and the corresponding original audio sample is used as the output target to perform model training on the spectral compensation model 210.
  • an audio signal, and outputting a prediction result that is, the prediction spectrum restoration audio signal corresponding to the first audio signal.
  • FIG. 3 is a schematic diagram of an audio signal processing method according to an exemplary embodiment, and the audio signal processing method can be executed by an audio processing device.
  • the above audio processing device may be the prediction device 120 in the system shown in FIG. 1 above.
  • the audio signal processing method may include the following steps:
  • Step 301 acquiring a first audio signal.
  • the first audio signal is an original audio signal collected by an audio processing device, an audio signal obtained after voice enhancement processing in the inherent hardware or software of the audio processing device, or the first audio signal It is an audio signal collected by an audio processing device that has undergone speech enhancement processing.
  • the first audio signal will be distorted in different situations due to speech enhancement processing performed by different algorithms.
  • the algorithms used by the audio processing device to perform speech enhancement processing on the audio signal through inherent hardware or software may vary according to the manufacturer of the audio processing device, the kernel version and the type of application software. As a result, the obtained first audio signals have different distortions.
  • the voice enhancement processing can include echo cancellation processing, noise suppression processing, volume self-adjustment processing, and frequency response equalization processing.
  • Audio processing equipment of different manufacturers or different kernel versions and different types of application software can perform voice enhancement processing. The degree of emphasis on each aspect is different, so that the distortion in the first audio signal obtained after the speech enhancement processing is different.
  • the distortion in the first audio signal can be expressed as high-frequency damage
  • the high-frequency damage may be high-frequency damage caused by noise suppression processing or echo cancellation processing, that is, the original high-frequency information in the audio signal is significantly weakened, resulting in sound It becomes dull and unclear; it can also be frequency band damage, that is, some fixed frequency band signals are fixedly attenuated, which may be due to the fact that the frequency response equalization process has not been processed well, resulting in obvious distortion of the audio frequency.
  • Step 302 Process the first audio signal by using a spectrum compensation model to obtain a prediction result of performing prediction compensation on the distorted spectrum in the first audio signal; A neural network model trained on audio samples.
  • the first audio signal is input into a spectrum compensation model, and a situation where the predicted first audio signal is undistorted is output as a prediction result.
  • the spectrum compensation model is a neural network model obtained by updating relevant parameters by training the spectrum-distorted audio samples and the original audio samples corresponding to the spectrum-distorted audio samples.
  • Step 303 reconstruct the first audio signal according to the prediction result, and obtain a target audio signal after repairing the distorted spectrum in the first audio signal.
  • the first audio signal can be reconstructed through the prediction result of the first audio signal output by the neural network model, and an audio signal after the first audio signal has been repaired is generated as the target audio signal.
  • the target audio signal is used as the audio signal after repairing the first audio signal, which can solve the problem of audio distortion in practical applications.
  • the terminal on the side sending the voice receives the user's voice signal
  • the received voice signal needs to be processed by voice enhancement first, and then the voice signal after the voice enhancement processing is reconstructed and repaired , send the repaired voice signal to the terminal on the voice receiving side, and play it, so that the user on the voice receiving side can clearly receive the voice content.
  • the reconstructed and repaired audio signal can also be used to process recorded audio, optimize audio during live broadcast, process audio with damaged sound quality by music playback software, and optimize audio by video playback software.
  • the neural network model obtained through model training can make compensation predictions for different equipment models and software versions uniformly, which solves the problem of limitations in the application scenarios of speech signal repair, thereby improving the generality of speech signal repair. sex.
  • FIG. 4 is a schematic diagram of an audio signal processing method according to an exemplary embodiment, and the audio signal processing method can be executed by an audio processing device.
  • the above audio processing device may be the model training device 110 and the prediction device 120 in the system shown in FIG. 1 .
  • the audio signal processing method may include two stages: a model training stage and a model application stage, which may be performed by the model training device 110 during the model training stage, wherein the model offline training stage may include the following steps:
  • step 401 raw audio samples are obtained.
  • the acquired original audio samples may be collected from the outside world, or obtained by artificial construction.
  • the original audio samples may be pre-stored in the model training device, or collected and stored by the model training device.
  • the prepared different types of noise sequences may be babble (speech) noise, street (street) noise, office (office) noise, or white (white) noise, and the like.
  • step 402 a power spectrum value suppression process is performed on a part of the frequency band in the frequency domain signal corresponding to the original audio sample to obtain a spectrally distorted audio sample corresponding to the original audio sample.
  • part of frequency bands on the frequency-domain signals are randomly selected for suppression processing.
  • a number of spectrally distorted audio samples corresponding to the original audio samples can be obtained by multiplying the power spectrum value corresponding to a part of the frequency band randomly selected on the frequency domain signal by a random value less than or equal to 1.
  • the partial frequency band on the frequency domain signal can also be selected according to the actual application.
  • the original audio sample is an audio signal with an 8khz spectrum
  • the part of the signal whose frequency is below 2khz is relatively less damaged, and the part of the signal above 4khz is more seriously damaged, so it can be distributed according to the frequency band damage probability. frequency band extraction.
  • step 403 using the spectrally distorted audio samples as input, and using the original audio samples as the training target, machine learning training is performed to obtain a spectral compensation model.
  • the original audio samples and the corresponding spectrally distorted audio samples obtained by the above-mentioned methods can be used to train a spectral compensation model and update the parameters of the model.
  • the spectral compensation model is a neural network model obtained by training the spectrally distorted audio samples and the original audio samples corresponding to the spectrally distorted audio samples.
  • the spectral compensation model is an RNN (Recurrent Neural Network, Recurrent Neural Network) model or an LSTM (Long Short-Term Memory, Long Short-Term Memory Network) model.
  • logarithmic processing is required between the spectrally distorted audio samples on the input side used for training the spectral compensation model and the original audio samples on the target output side.
  • the power spectrum value corresponding to the spectrally distorted audio sample on the input side and the power spectrum value corresponding to the original audio sample need to be logarithmically processed, and the calculation formula of the logarithmic processing can be as follows:
  • S dB (i, k) is the logarithmic value of the power spectrum, i is the corresponding frame number, and k is the corresponding frequency index value.
  • the spectral compensation model obtained by training the sample set obtained after logarithmic processing still has logarithmic values on the input side and the output side in practical applications.
  • the model training device can complete the training and update of the spectrum compensation model.
  • model application phase may be performed by the prediction device 120, wherein the model application phase may include the following steps:
  • step 404 a first audio signal is acquired.
  • the first audio signal after audio enhancement processing is acquired.
  • the first audio signal may be a time-domain signal with partial distortion.
  • step 405 the first audio signal is converted into a corresponding frequency domain signal.
  • the first audio signal is a time-frequency signal, and the time-frequency signal is converted into a corresponding frequency-domain signal through an operation for subsequent calculation.
  • the first audio signal needs to be framed and windowed to achieve the purpose of truncating the sampling time and processing the limited signal again.
  • the function window may include a rectangular window, a triangular window, a Haining window, a Hamming window, a Kaiser window, and the like. Windowing of the first audio signal may use a squaring of the Hamming window.
  • the window function corresponding to the square root of the Hamming window used can be as follows:
  • n takes the integer value in [0, N-1], and N is the length of the window sample point of 20ms.
  • the windowed first audio signal can be obtained by multiplying the first audio signal by a window function, and the specific formula is as follows:
  • x in is composed of the time domain signal including the 10ms of the previous frame and the time domain signal of the current frame of 10ms.
  • frequency domain conversion is performed on the processed time domain signal to obtain a corresponding frequency domain signal.
  • the manner of converting the time domain signal into the frequency domain signal may include different algorithms.
  • a corresponding frequency-domain signal is obtained by performing discrete Fourier transform (Discrete Fourier Transformation, DFT) on the processed time-domain signal.
  • DFT discrete Fourier Transformation
  • the amplitude X(i, k) of each frequency point in the frequency domain signal can be obtained by discrete Fourier transform.
  • a corresponding frequency-domain signal is obtained by performing an improved discrete cosine transform on the processed time-domain signal.
  • the modified discrete cosine transform (Modified Discrete Cosine Transform, MDCT) is a transform related to the Fourier transform, based on the fourth type discrete cosine transform (DCT-IV).
  • MDCT Modified Discrete Cosine Transform
  • DCT-IV fourth type discrete cosine transform
  • the modified discrete cosine transform is similar to the discrete Fourier transform, but only uses real numbers.
  • the calculation method is similar to the algorithm of discrete Fourier transform.
  • step 406 the frequency domain signal is divided into at least one subband frequency domain signal.
  • the entire frequency domain signal corresponding to the first audio signal is divided into at least one subband frequency domain signal.
  • dividing the frequency domain signal into at least one subband may include dividing during linear frequency domain transform and dividing during nonlinear frequency domain transform.
  • the frequency domain signal when performing linear frequency domain transformation, is divided into at least one subband frequency domain signal, and the frequency domain signal can be divided into at least one subband frequency domain signal by averaging.
  • the frequency domain signal when performing nonlinear frequency domain transformation, is divided into at least one subband frequency domain signal, and the Bark domain can be used as a scale to divide the frequency domain signal into at least one subband. frequency domain signal.
  • the Bark domain is the 24 critical frequency bands that the auditory filter simulates.
  • the Bark domain can be used to describe the signal.
  • the frequency domain signal is divided into at least one subband frequency domain signal through the Bark domain, and each subband frequency domain signal is uneven. of.
  • the method for dividing the frequency domain signal into at least one subband frequency domain signal may be performed on the basis of dividing the frequency domain signal into at least one subband frequency domain signal equally divided.
  • the self-contained serial number corresponding to each frequency point is obtained.
  • step 407 frequency bin amplitudes in at least one subband frequency domain signal are determined.
  • the amplitude of each frequency point included in each subband frequency signal is determined.
  • the frequency point sequence number included in each subband frequency signal is determined according to the division of at least one subband frequency signal.
  • the corresponding amplitude of each frequency point can be determined according to the frequency point serial number.
  • the frequency point amplitude can be obtained by calculation through the formula shown in step 402 .
  • the uniformly divided subband is converted into subband division based on the scale of the Bark domain, and the amplitude of the frequency point of the 0th subband can be
  • step 408 the power spectrum value of the frequency domain signal of at least one subband is determined according to the frequency bin amplitude.
  • the power spectrum value corresponding to each subband frequency domain signal is determined by acquiring the amplitude value of each frequency point included in each subband frequency domain signal.
  • the amplitude of each frequency point obtained after the Fourier transform can be further calculated as the square of the amplitude of each frequency point, that is, the corresponding power spectrum value.
  • calculation formula for calculating the corresponding power spectrum value through the amplitude value of each frequency point may be as follows:
  • i is the corresponding frame sequence number
  • k is the corresponding frequency point index value
  • the amplitude of each frequency point obtained after the improved discrete cosine transform can also be calculated as the square of the amplitude of each frequency point as the corresponding power spectrum value.
  • step 409 the power spectrum value of at least one subband frequency domain signal is input into the spectrum compensation model, and the subband prediction results corresponding to each subband frequency domain signal respectively are obtained.
  • the power spectrum value corresponding to the frequency domain signal of each subband is input into the spectrum compensation model, and the predicted power spectrum value corresponding to the frequency domain signal of each subband is obtained as the subband of the frequency domain signal of each subband forecast result.
  • a prediction result of performing prediction compensation on the distortion spectrum in the first audio signal is determined according to the subband prediction result.
  • logarithmization is performed on the power spectrum values corresponding to the frequency domain signals of each subband, and the logarithmized power spectrum values are input into the spectrum compensation model to obtain the predicted power spectrum values.
  • the logarithmic value of as the prediction result.
  • step 410 according to the prediction result, the reconstructed power spectrum value of the first audio signal is obtained.
  • the reconstructed power spectrum value corresponding to each subband frequency domain signal is obtained according to the prediction result corresponding to each subband frequency domain signal output by the spectrum compensation model.
  • the reconstructed power spectrum value may be a logarithmic value.
  • the reconstructed power spectrum value of the first audio signal may be directly obtained according to the prediction result.
  • the predicted power spectrum value corresponding to at least one subband frequency domain signal is directly used as the reconstructed power spectrum value.
  • the logarithm of the power spectrum is can directly as the reconstructed power spectrum value.
  • the reconstructed power spectrum value is determined according to the prediction result and the power spectrum value corresponding to each input subband frequency domain signal.
  • the sum of the power spectrum value corresponding to the first audio signal and the frequency band impairment rate may be used as the reconstructed power spectrum value.
  • the frequency band impairment rate is a historical smoothed value of the difference between the predicted power spectrum value corresponding to the frequency domain signal of at least one subband and the power spectrum value corresponding to the first audio signal.
  • the logarithm of the power spectrum is The difference between the actual logarithm value of the power spectrum and the predicted logarithm value of the power spectrum for each frequency point is:
  • the historical smoothed value can be calculated as follows Formula to calculate:
  • ⁇ is a parameter whose value range is (0,1).
  • the difference between the actual logarithm value of the power spectrum and the predicted logarithm value of the power spectrum corresponding to the frequency point is used as the frequency point.
  • Gain to determine the frequency bin amplitude after reconstruction.
  • step 411 a target audio signal corresponding to the reconstructed power spectrum value is generated.
  • the generated logarithm of the reconstructed power spectrum value is converted into a linear value, the frequency point amplitude corresponding to the power spectrum value is determined, and the corresponding time domain signal is generated according to the frequency point amplitude as a target audio signal.
  • the respective predictions corresponding to at least one subband frequency domain signal as the reconstructed power spectrum value is converted into a linear value, and the linear value is squared to determine the amplitude of each frequency point.
  • the power spectrum corresponding to each subband frequency domain signal is The value is added to the frequency band impairment rate to obtain the reconstructed power spectrum value.
  • the reconstructed power spectrum value is converted into a linear value, the linear value is squared, and the amplitude of each frequency point is determined.
  • the formula for calculating the reconstructed power spectrum value can be as follows:
  • the calculation formula for converting the logarithmic value of the power spectrum to a linear value can be as follows:
  • the amplitude of each frequency point in the discrete Fourier transform domain can be obtained by Calculation.
  • the time domain transform is performed on the frequency domain signal corresponding to the reconstructed power spectrum value to obtain the target audio signal.
  • the conversion from the frequency domain signal to the time domain signal may be performed by inverse discrete Fourier transform or improved inverse discrete cosine transform.
  • x out can be 20ms data.
  • the output of the current frame is the second half of the 10ms data in the 20ms data converted from the previous frequency domain signal to the time domain signal and the first half of the current frame. 10ms data The result of adding the result is used as the target audio signal outputted in the current frame.
  • the prediction device can complete the compensation and reconstruction of the first audio signal through the spectrum compensation model to obtain the target audio signal.
  • the neural network model obtained through model training can make compensation predictions for different equipment models and software versions uniformly, which solves the problem of limitations in the application scenarios of speech signal repair, thereby improving the generality of speech signal repair. sex.
  • FIG. 5 is a structural diagram of an audio signal processing method according to an exemplary embodiment, and the audio signal processing method can be executed by an audio processing device.
  • the above audio processing device may be the model training device 110 and the prediction device 120 in the system shown in FIG. 1 .
  • the audio signal processing method may include two stages, namely a model training stage and a model application stage.
  • the model training phase may be performed by offline training performed by the model training device, and the model training phase may include the following steps 51:
  • step 51 taking the spectrally damaged audio as the input and the corresponding original audio as the output target, offline training is performed on the deep neural network model, and the trained neural network model is updated.
  • the spectrum-impaired audio and the corresponding original audio may be sample data pre-stored in the sample set.
  • the model training device uses the spectrum-impaired audio as the input and the corresponding original audio as the output target.
  • the pair may be a recurrent neural network model RNN or a long
  • the spectral compensation model of the short-term memory network model LSTM is trained and updated.
  • the model application stage may be performed online by the prediction device, and the model application stage may include the following steps 52 to 59:
  • step 52 the prediction device collects the external audio signal through the audio collection function.
  • the module responsible for audio collection in the audio processing device may, for example, collect external audio through a microphone module to obtain audio signals.
  • step 53 the prediction device generates the audio signal after the speech enhancement process by subjecting the acquired audio signal to the software and hardware speech enhancement processing inherent in the prediction device.
  • the inherent software and hardware speech enhancement processing may include applying different speech enhancement algorithms to process the audio signal, and the generated audio signal after speech enhancement processing may have different degrees of signal damage.
  • step 54 the audio signal after the speech enhancement processing is a time-domain signal, and the time-domain signal is transformed into a frequency-domain signal through an operation.
  • the operation method for transforming the time domain signal into the frequency domain signal may be discrete Fourier transform or discrete cosine transform.
  • step 55 the frequency domain signal is input into a neural network model for prediction.
  • the samples at the input end and the target output end are the logarithm of the power spectrum, so it is necessary to calculate the logarithm value of the power spectrum corresponding to the frequency domain signal, and input the logarithm value of the power spectrum into the spectrum compensation model. forecast in.
  • the prediction result output by the neural network model is the predicted audio signal of the predicted repaired speech enhancement distorted signal, and the obtained predicted audio signal and the previously input frequency domain signal are subjected to sub-band damage rate analysis to obtain the current The subband frequency domain signal impairment rate of the signal.
  • the prediction result output by the spectrum compensation model is the logarithm value of the power spectrum corresponding to the predicted audio signal, and the subband damage rate can be analyzed by the logarithm value of the power spectrum and the logarithm value of the power spectrum corresponding to the input frequency domain signal.
  • the ratio is the historical smoothing value of the difference between the logarithmic value of the power spectrum and the logarithmic value of the power spectrum corresponding to the input frequency domain signal.
  • this step can be omitted, that is, the directly obtained predicted audio signal is used as a basis to perform the next step.
  • step 57 the damaged frequency band is reconstructed to generate a reconstructed frequency domain signal.
  • each input subband frequency domain signal is a damaged frequency band
  • the logarithm value of the power spectrum of the reconstructed frequency domain signal will be calculated and determined by means of the subband damage rate, or, when step 56 is omitted, the directly obtained
  • the logarithm of the power spectrum of the predicted audio signal is taken as the logarithm of the power spectrum of the reconstructed frequency domain signal.
  • step 58 the reconstructed frequency domain signal is converted into a time domain signal.
  • step 54 when the operation method for transforming the time-domain signal into the frequency-domain signal is discrete Fourier transform, the calculation method for converting the reconstructed frequency-domain signal into the time-domain signal may use inverse discrete Fourier transform If in step 54, the operation method for transforming the time domain signal into the frequency domain signal is discrete cosine transform, the calculation method for converting the reconstructed frequency domain signal into the time domain signal can use the inverse discrete cosine transform.
  • step 59 the time domain signal is output as the repaired target audio.
  • the time-domain signal converted from the reconstructed frequency-domain signal is output as the target audio signal, which may be played by the audio playback module of the prediction device, for example, by the speaker module to play the target audio.
  • the neural network model obtained through model training can make compensation predictions for different equipment models and software versions uniformly, which solves the problem of limitations in the application scenarios of speech signal repair, thereby improving the generality of speech signal repair. sex.
  • FIG. 6 is a schematic diagram of a voice restoration applied to a voice call system according to an exemplary embodiment.
  • the voice call system may include a voice transmitter 620 , a voice receiver 640 and a server for data transmission. 630.
  • the process of voice restoration in the voice call system may be as follows:
  • the voice transmitter 620 collects voice signals from the outside world, and performs inherent software and hardware voice enhancement processing on the collected voice signals in the voice transmitter 620. After the enhancement processing, the voice signal with certain damage is repaired, and the repaired enhanced voice signal is sent to the voice receiving end 640 through the server end 630, and the voice receiving end 640 uses its own voice playback module. play.
  • the spectrum compensation model in the voice transmitter 620 is obtained through offline model training by the model training device 610, and the voice transmitter 620 can download, install and update the trained spectrum compensation model in the application.
  • FIG. 7 is a schematic diagram of another voice restoration applied in a voice communication system according to an exemplary embodiment.
  • the voice communication system may include a voice transmitter 720 , a voice receiver 740 and a server for data transmission. end 730.
  • the process of voice restoration in the voice communication system may be as follows:
  • the voice transmitting end 720 collects the voice signal from the outside world, and performs inherent software and hardware voice enhancement processing on the collected voice signal in the voice transmitting end 720, and then enhances the voice signal with certain damage after the voice enhancement processing.
  • the server end 730 performs speech restoration processing on the speech signal with certain damage after the speech enhancement processing through the spectrum compensation model in the above-mentioned embodiment, and then sends the repaired enhanced speech signal to the speech receiving end 740.
  • the voice receiving end 640 plays the repaired enhanced voice signal through its own voice playing module.
  • the spectrum compensation model in the server end 730 is obtained through offline model training by the model training device 710, and the server end 730 can acquire the spectrum compensation model completed by the training to perform a certain damage on the speech signal after the speech enhancement processing. Speech repair processing.
  • the neural network model obtained through model training can make compensation predictions for different equipment models and software versions uniformly, which solves the problem of limitations in the application scenarios of speech signal repair, thereby improving the generality of speech signal repair. sex.
  • Fig. 8 is a structural block diagram of an audio signal processing apparatus according to an exemplary embodiment.
  • the audio signal processing method may be executed by an audio processing device to execute all or part of the steps in the method shown in the corresponding embodiment of FIG. 3 or FIG. 4 .
  • the audio signal processing device may include:
  • a signal acquisition module 810 configured to acquire a first audio signal
  • a result obtaining module 820 configured to process the first audio signal through a spectrum compensation model to obtain a prediction result of performing prediction compensation on the distorted spectrum in the first audio signal;
  • the spectrum compensation model is to use a spectrum distorted audio samples, and the neural network model obtained by training the original audio samples corresponding to the spectrally distorted audio samples;
  • the target obtaining module 830 is configured to reconstruct the first audio signal according to the prediction result, and obtain a target audio signal after repairing the distorted spectrum in the first audio signal.
  • the result obtaining module 820 includes:
  • a frequency-domain conversion submodule for converting the first audio signal into a corresponding frequency-domain signal
  • a subband dividing submodule configured to divide the frequency domain signal into at least one subband frequency domain signal
  • an amplitude determination submodule configured to determine the frequency point amplitude in the at least one subband frequency domain signal
  • a power spectrum value determination submodule configured to determine the power spectrum value of the at least one subband frequency domain signal according to the frequency point amplitude
  • the sub-band result obtaining sub-module is configured to input the power spectrum value of the at least one sub-band frequency domain signal into the spectrum compensation model, and obtain the sub-band prediction results corresponding to the at least one sub-band frequency domain signal respectively.
  • a prediction result of performing prediction compensation on the distortion spectrum in the first audio signal is determined according to the subband prediction result.
  • the frequency domain conversion submodule includes:
  • a windowing processing unit configured to perform frame-by-frame windowing processing on the first audio signal, and determine the time-domain signal processed by the first audio signal
  • a signal obtaining unit configured to perform frequency domain conversion on the processed time domain signal to obtain the corresponding frequency domain signal.
  • the signal acquisition unit is configured to:
  • the corresponding frequency-domain signal is obtained by performing an improved discrete cosine transform on the processed time-domain signal.
  • the sub-band division sub-module includes:
  • a subband dividing unit configured to divide the frequency domain signal into the at least one subband frequency domain signal with the Bark domain as the granularity.
  • the sub-band result acquisition sub-module includes:
  • a subband result acquisition unit configured to input the power spectrum value corresponding to the at least one subband frequency domain signal into the spectrum compensation model, and obtain the predicted power spectrum value corresponding to the at least one subband frequency domain signal, as a subband prediction result of the at least one subband frequency domain signal.
  • the target acquisition module 830 includes:
  • a power spectrum value acquisition sub-module configured to acquire the reconstructed power spectrum value of the first audio signal according to the prediction result
  • a target generation sub-module configured to generate the target audio signal corresponding to the reconstructed power spectrum value.
  • the target acquisition module 830 includes:
  • a power spectrum value generation sub-module configured to use the sum of the power spectrum value corresponding to the first audio signal and the frequency band impairment rate as the reconstructed power spectrum value;
  • the frequency band impairment rate is the The predicted power spectrum value corresponding to the frequency domain signal respectively, and the historical smoothing value of the difference between the power spectrum value corresponding to the first audio signal;
  • the power spectrum value determination submodule is configured to use the predicted power spectrum values corresponding to the at least one subband frequency domain signal respectively as the reconstructed power spectrum value.
  • the target generation submodule includes:
  • a time domain conversion unit configured to perform time domain transformation on the frequency domain signal corresponding to the reconstructed power spectrum value to obtain the target audio signal.
  • the signal acquisition module 810 includes:
  • the signal acquisition sub-module is used for acquiring the first audio signal after audio enhancement processing.
  • the apparatus further includes:
  • a sample acquisition module configured to process the first audio signal through a spectrum compensation model, and acquire the original audio sample before obtaining a prediction result of performing prediction compensation on the distortion spectrum in the first audio signal;
  • a distortion sample acquisition module configured to perform power spectrum value suppression processing on a part of the frequency band in the frequency domain signal corresponding to the original audio sample, to obtain a spectrally distorted audio sample corresponding to the original audio sample;
  • a model acquisition module configured to take the spectrally distorted audio samples as input, and use the original audio samples as training targets to perform machine learning training to obtain the spectral compensation model.
  • the spectral compensation model is RNN or LSTM.
  • the neural network model obtained through model training can make compensation predictions for different equipment models and software versions uniformly, which solves the problem of limitations in the application scenarios of speech signal repair, thereby improving the generality of speech signal repair. sex.
  • Fig. 9 is a schematic structural diagram of a computer device according to an exemplary embodiment.
  • the computer device may be implemented as an audio processing device.
  • the audio processing device may include the model training device 110 and the prediction device 120 shown in FIG. 1 .
  • the computer device 900 includes a central processing unit (Central Processing Unit, CPU) 901, a system memory 904 including a random access memory (Random Access Memory, RAM) 902 and a read-only memory (Read-Only Memory, ROM) 903, and A system bus 905 that connects the system memory 904 and the central processing unit 901 .
  • CPU Central Processing Unit
  • RAM random access memory
  • ROM Read-Only Memory
  • the computer device 900 also includes a basic input/output system (Input/Output, I/O system) 906 that helps to transfer information between various devices in the computer, and is used to store an operating system 913, application programs 914 and other program modules 915 of the mass storage device 907.
  • I/O system input/output system
  • the mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905 .
  • the mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900 . That is, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.
  • a computer-readable medium such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.
  • the computer-readable media can include computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, cassette, magnetic tape, magnetic disk storage or other magnetic storage device.
  • EPROM Erasable Programmable Read Only Memory
  • EEPROM Electrically Erasable Programmable Read-Only Memory
  • CD-ROM Compact Disc
  • DVD Digital Video Disc
  • the computer device 900 can be connected to the Internet or other network devices through a network interface unit 911 connected to the system bus 905 .
  • the memory also includes one or more programs, the one or more programs are stored in the memory, and the central processing unit 901 implements all or all of the methods shown in FIG. 3 or FIG. 4 by executing the one or more programs. some steps.
  • non-transitory computer-readable storage medium including instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to complete the present application All or part of the steps of the method shown in each embodiment.
  • the non-transitory computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), magnetic tapes, floppy disks, and optical data storage devices, etc.
  • a computer program product or computer program comprising computer instructions stored in a computer readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the audio signal processing methods provided in various optional implementations of the above aspects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Circuit For Audible Band Transducer (AREA)

Abstract

一种音频信号处理方法、装置、计算机设备、计算机可读存储介质及计算机程序产品，涉及人工智能技术领域。该方法包括：获取第一音频信号（301）；通过频谱补偿模型对第一音频信号进行处理，获得对第一音频信号中的失真频谱进行预测补偿的预测结果（302）；根据预测结果，对第一音频信号进行重构，获得对第一音频信号中的失真频谱进行修复后的目标音频信号（303）。基于人工智能AI的思想，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。

Description

音频信号处理方法和相关装置
本申请要求于2020年07月13日提交中国专利局、申请号为202010670626.9、申请名称为“音频信号处理方法、装置、计算机设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及人工智能技术领域，特别涉及音频信号处理。
背景技术
如今，在语音通话的应用中，语音接收设备需要对接收到的语音信号进行语音增强处理，针对于不同厂家、不同内核版本、不同应用软件的语音接收设备进行语音增强处理的算法可以是不相同的，并且进行语音增强处理后的语音信号可能出现不同程度的语音损伤的问题。
在相关技术中，在对通过语音增强处理造成语音损伤之后，需要针对进行语音增强处理的设备机型以及语音增强处理软件版本，对损伤的语音信号进行离线测定和频谱分析，得到损伤的语音信号对应的频谱失真分布和失真量化值，通过这些测定的失真结果设定补偿策略，后续在该设备机型以及软件版本对应的设备上，通过设定好的补偿策略进行频谱增益补偿。
发明内容
一方面，本申请实施例提供了一种音频信号处理方法，所述方法包括：
获取第一音频信号；
通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果；所述频谱补偿模型是通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到的神经网络模型；
根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号。
另一方面，本申请实施例提供了一种音频信号处理装置，所述装置包括：
信号获取模块，用于获取第一音频信号；
结果获取模块，用于通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果；所述频谱补偿 模型是通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到的神经网络模型；
目标获取模块，用于根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号。
子带频域另一方面，本申请实施例提供了一种计算机设备，所述计算机设备包含处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现以上方面的音频信号处理方法。
另一方面，本申请实施例提供了一种计算机可读存储介质，所述存储介质用于存储计算机程序，所述计算机程序用于执行以上方面的音频信号处理方法。
根据本申请实施例的一个方面，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述方面的音频信号处理方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
图1是根据一示例性实施例示出的一种模型训练及预测补偿的框架图；
图2是根据一示例性实施例实施例示出的一种机器学习模型的模型架构图；
图3是根据一示例性实施例示出的一种音频信号处理方法的示意图；
图4是根据一示例性实施例示出的一种音频信号处理方法的示意图；
图5是根据一示例性实施例示出的一种音频信号处理方法的架构图；
图6是根据一示例性实施例示出的一种应用在语音通话系统中的语音修复示意图；
图7是根据一示例性实施例示出的另一种应用在语音通话系统中的语音修复示意图；
图8是根据一示例性实施例示出的一种音频信号处理方法的结构方框图；
图9是根据一示例性实施例示出的一种计算机设备的结构示意图。
具体实施方式
本申请提供的方案涉及人工智能的机器学习等技术。例如通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到频谱补偿模型。
语音增强处理。在语音通话的应用场景中，对于设备采集到的录音信号需要经过语音增强处理，其中，语音增强处理主要包括：回声消除、噪声抑制、音量自调节、频响均衡等。
语音增强处理可以通过硬件进行实现，例如，通过一些音频芯片实现语音增强处理，或者也可以在应用层加入各自的语音增强处理功能模块。
本申请实施例中涉及了对语音增强处理技术的应用，例如通过语音增强处理技术得到第一音频信号。
在实际应用中可以发现不同厂家、不同内核版本、不同应用软件的语音增强处理算法表现不一，其中的部分算法会对采集到的录音信号有明显损伤，例如，损伤的类型可以包括高频损伤，高频损伤可以是降噪或者回声消除算法导致的语音信号的高频损伤，表现为语音信号中原有的高频信息被明显衰弱，这导致语音信号对应的声音变得发闷、不清晰；另外，损伤的类型可以包括频带损伤，表现为某些固定频带的信号被固定衰减，可能是均衡处理没有做好引起的，导致语音信号对应的声音听感上有明显失真。
本申请实施例的方案包括模型训练阶段和预测阶段。图1是根据一示例性实施例示出的一种模型训练及预测补偿的框架图。如图1所示，在模型训练阶段，模型训练设备110通过预先准备好的包括原始音频样本以及其对应的频谱失真音频样本的样本集训练出端到端的机器学习模型，在预测补偿阶段，预测设备120根据训练好的机器学习模型以及输入的第一音频信号直接预测出第一音频信号中失真频谱进行预测补偿的预测结果。
其中，上述模型训练设备110和预测设备120可以是具有机器学习能力的计算机设备，比如，该计算机设备可以是个人电脑以及服务器等固定式计算机设备，或者，该计算机设备也可以是平板电脑或者电子书阅读器等移动式计算机 设备。
可选的，上述模型训练设备110和预测设备120可以是同一个设备，或者，模型训练设备110和预测设备120也可以是不同的设备。并且，当模型训练设备110和预测设备120是不同的设备时，模型训练设备110和预测设备120可以是同一类型的设备，比如模型训练设备110和预测设备120可以都是个人电脑；或者，模型训练设备110和预测设备120也可以是不同类型的设备，比如模型训练设备110可以是服务器，而预测设备120可以是移动终端设备等。本申请实施例对于模型训练设备110和预测设备120的具体类型不做限定。
图2是根据一示例性实施例示出的一种机器学习模型的模型架构图。如图2所示，本申请实施例中的机器学习模型20可以包含生成样本以及存储样本的样本集，和一个频谱补偿模型210部分，其中样本集中存储有通过采集获取的原始音频样本以及对应的频谱失真音频样本，或者通过人工构建生成的原始音频样本以及对应的频谱失真音频样本。将样本集中存储的各个频谱失真音频样本输入到频谱补偿模型210中，并且以对应的原始音频样本作为输出目标，对频谱补偿模型210进行模型训练，该频谱补偿模型210，用于根据输入的第一音频信号，输出预测结果，即第一音频信号对应的预测频谱修复音频信号。
请参考图3，其是根据一示例性实施例示出的一种音频信号处理方法的示意图，该音频信号处理方法可以由音频处理设备执行。其中，上述音频处理设备可以是上述图1所示系统中的预测设备120。如图3所示，该音频信号处理方法可以包括以下步骤：
步骤301，获取第一音频信号。
在一种可能的实现方式中，第一音频信号是音频处理设备采集的原始音频信号，经过该音频处理设备固有的硬件或者软件中的语音增强处理后获取到的音频信号，或者第一音频信号是由音频处理设备采集得到的已经过语音增强处理后的音频信号，第一音频信号由于经过不同算法的语音增强处理会发生不同情况的失真。
其中，音频处理设备通过固有的硬件或者软件对音频信号进行语音增强处理所用的算法，可以根据音频处理设备的厂家、内核版本以及应用软件的种类而各不相同。从而导致获取到的第一音频信号中的失真情况各不相同。
其中，语音增强处理可以包含回声消除处理、噪声抑制处理、音量自调节处理以及频响均衡处理等多个方面，不同厂家或者不同内核版本的音频处理设备以及不同种类的应用软件，进行语音增强处理时对各个方面的偏重程度各不相同，从而导致语音增强处理后获得的第一音频信号中的失真情况各不相同。
比如，第一音频信号中的失真情况可以表现为高频损伤，高频损伤可能是噪声抑制处理或者回声消除处理导致的高频损伤，即音频信号中原有的高频信息被明显衰弱，导致声音变得发闷、不清晰；也可以是频带损伤，即某些固定频带的信号被固定衰减，可能是由于频响均衡处理没有处理好，导致音频在听感上有明显失真。
步骤302，通过频谱补偿模型对第一音频信号进行处理，获得对第一音频信号中的失真频谱进行预测补偿的预测结果；频谱补偿模型是通过频谱失真音频样本，以及频谱失真音频样本对应的原始音频样本训练得到的神经网络模型。
在一种可能的实现方式中，将第一音频信号输入频谱补偿模型，输出得到预测的第一音频信号未失真的情况作为预测结果。
其中，频谱补偿模型是通过频谱失真音频样本，以及频谱失真音频样本对应的原始音频样本进行训练，更新相关参数得到的神经网络模型。
步骤303，根据预测结果，对第一音频信号进行重构，获得对第一音频信号中的失真频谱进行修复后的目标音频信号。
在一种可能的实现方式中，通过神经网络模型输出的对第一音频信号的预测结果，可以对第一音频信号进行重构，生成第一音频信号进行修复后的音频信号作为目标音频信号。
其中，目标音频信号作为对第一音频信号进行修复后的音频信号，可以解决在实际应用中的音频失真的问题。
比如，在运用社交软件进行语音通话的过程中，发送语音侧的终端接收到用户的语音信号，接收到的语音信号需要先进行语音增强处理，然后对语音增强处理后的语音信号进行重构修复，将修复后的语音信号发送给接收语音侧的终端，并且进行播放，使接收语音侧的用户可以清晰的接收语音内容。或者，重构修复音频信号也可以用作对录制音频的处理、直播过程中对音频的优化处理、音乐播放软件对音质受损的音频的处理以及视频播放软件对音频的优化处 理等。
综上所述，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。
请参考图4，其是根据一示例性实施例示出的一种音频信号处理方法的示意图，该音频信号处理方法可以由音频处理设备执行。其中，上述音频处理设备可以是上述图1所示系统中的模型训练设备110和预测设备120。如图4所示，该音频信号处理方法可以包括模型训练阶段以及模型应用阶段这两个阶段，在模型训练阶段可以由模型训练设备110执行，其中，模型离线训练阶段可以包括以下步骤：
在步骤401中，获取原始音频样本。
在本申请实施例中，获取的原始音频样本可以是从外界进行采集的，或者是进行人工构造获得的。
在一种可能的实现方式中，原始音频样本可以预先存储在模型训练设备中，或者通过模型训练设备进行采集存储。
在一种可能的实现方式中，通过人工构造原始音频样本，首先，准备一批不带噪声的音频信号以及一批不同类型的噪声序列，然后通过配置不同的SNR(信噪比)组合，将不带噪声的音频信号与噪声序列按照配置的不同信噪比进行线性叠加，生成带噪音的原始音频信号，同时为了使原始音频样本在能量级别上的覆盖更广，将带噪音的原始音频信号整体乘以不同的增益值。
其中，准备的不同类型的噪声序列可以是babble(语音)噪声、street(街道)噪声、office(办公)噪声或者white(白)噪声等。
在步骤402中，对原始音频样本对应的频域信号中的部分频带进行功率谱值的抑制处理，得到原始音频样本对应的频谱失真音频样本。
在一种可能的实现方式中，将原始音频样本由时域信号转换成频域信号后，随机抽取频域信号上的部分频带进行抑制处理。
其中，可以通过将随机抽取频域信号上的部分频带对应的功率谱值乘以一个小于等于1的随机值，得到原始音频样本对应的若干个频谱失真音频样本。
在一种可能的实现方式中，抽取频域信号上的部分频带也可以依据实际应 用中的情况进行选取。
比如，当原始音频样本为一个8khz频谱的音频信号，该信号中的频率在2khz以下的部分信号损伤相对较少，并且位于4khz以上的部分信号损伤比较严重，所以可以按照频带受损概率的分布情况进行频带抽取。
其中，选取的样本中要确保各个不同频带损伤构造的样本都具有一定的数量。以确保训练的准确性。
在步骤403中，以频谱失真音频样本为输入，并以原始音频样本为训练目标进行机器学习训练，获得频谱补偿模型。
在本申请实施例中，通过上述获得的原始音频样本以及对应的频谱失真音频样本，可以用于训练频谱补偿模型并且进行模型的参数更新。
在一种可能的实现方式中，频谱补偿模型是通过频谱失真音频样本，以及频谱失真音频样本对应的原始音频样本训练得到的神经网络模型。
其中，频谱补偿模型是RNN(Recurrent Neural Network，循环神经网络)模型或者LSTM(Long Short-Term Memory，长短期记忆网络)模型。
在一种可能的实现方式中，用于训练频谱补偿模型的输入侧的频谱失真音频样本与目标输出侧的原始音频样本需要进行对数化处理。
其中，输入侧的频谱失真音频样本对应的功率谱值以及原始音频样本对应的功率频谱值，需要进行对数化处理，对数化处理的计算公式可以如下所示：
S dB(i,k)＝20log(S(i,k))
其中，S dB(i,k)为功率谱对数值，i为对应的帧序号，k为对应的频点索引值。
在一种可能的实现方式中，通过对数化处理后获得的样本集训练得到的频谱补偿模型，在实际应用中的输入侧与输出侧仍为对数值。
通过步骤401到步骤403，模型训练设备可以完成对频谱补偿模型的训练与更新。
接下来，在模型应用阶段可以由预测设备120执行，其中，模型应用阶段可以包括以下步骤：
在步骤404中，获取第一音频信号。
在一种可能的实现方式中，获取经过音频增强处理后的第一音频信号。
其中，第一音频信号可以是具有部分失真的时域信号。
在步骤405中，将第一音频信号转换为对应的频域信号。
在本申请实施例中，第一音频信号是时频信号，将该时频信号通过运算转换为对应的频域信号，以进行后续的计算。
在一种可能的实现方式中，在进行运算转换为对应的频域信号之前，需要对第一音频信号进行分帧加窗处理，确定处理后的时域信号。
其中，在信号处理的过程中，由于计算机设备只能处理有限长度的信号，所以需要将第一音频信号进行分帧加窗处理，达到以采样时间进行截断，对有限的信号再进行处理的目的。
其中，函数窗可以包括矩形窗、三角窗、海宁窗、汉明窗以及凯瑟窗等。对第一音频信号进行加窗处理可以使用汉明窗的开方。
比如，当窗长度为20ms，一帧数据长度为10ms时，所使用的汉明窗的开方对应的窗函数可以如下所示：
Figure PCTCN2021097663-appb-000001
其中，n取[0，N-1]中的整数值，N为20ms的窗样点长度。
其中，加窗处理后的第一音频信号可以通过第一音频信号与窗函数相乘得到，具体公式如下：
x w(n)＝x in(n)*win(n)
其中，x in是由包含上一个帧10ms的时域信号以及当前帧10ms的时域信号构成的。
在一种可能的实现方式中，在对第一音频信号进行加窗处理之后，对处理后的时域信号进行频域转换，获得对应的频域信号。
其中，将时域信号转换成频域信号的方式可以包括不同的算法。
在一种可能的实现方式中，通过对处理后的时域信号进行离散傅里叶变换(Discrete Fourier Transformation，DFT)，获得对应的频域信号。
其中，离散傅里叶变换的计算公式如下所示：
Figure PCTCN2021097663-appb-000002
其中，通过离散傅里叶变换可以获得频域信号中的各频点幅值X(i,k)。
在另一种可能的实现方式中，通过对处理后的时域信号进行改进离散余弦变换，获得对应的频域信号。
其中，改进的离散余弦变换(Modified Discrete Cosine Transform,MDCT)是一种与傅里叶变换相关的变换，以第四型离散余弦变换(DCT-IV)为基础。改进的离散余弦变换类似于离散傅里叶变换，但只使用实数。计算方式与离散傅里叶变换的算法相似。
在步骤406中，将频域信号划分为至少一个子带频域信号。
在本申请实施例中，将第一音频信号对应的整段频域信号划分为至少一个子带频域信号。
其中，划分为至少一个子带频域信号可以包括在线性频域变换时进行划分，以及在进行非线性频域变换时进行划分两种情况。
在一种可能的实现方式中，在进行线性频域变换时，将频域信号划分为至少一个子带频域信号，可以通过对频域信号进行均分为至少一个子带频域信号。
在另一种可能的实现方式中，在进行非线性频域变换时，将频域信号划分为至少一个子带频域信号，可以以Bark域为尺度，将频域信号划分为至少一个子带频域信号。
其中，Bark域是听觉滤波器模拟出听觉的24个临界频带，用Bark域可以描述信号，通过Bark域对频域信号划分为至少一个子带频域信号，各个子带频域信号是不均匀的。
其中，以Bark域为尺度，将频域信号划分为至少一个子带频域信号的方法可以在划分为均分的至少一个子带频域信号的基础上进行。通过将各个均匀划分的子带频域信号中的频点映射到Bark域的子带频域信号中，获取各个频点对应的自带序号。
在步骤407中，确定至少一个子带频域信号中的频点幅值。
在本申请实施例中，按照至少一个子带频率信号的划分情况，确定各个子带频率信号中包含的各个频点幅值。
在一种可能的实现方式中，按照至少一个子带频率信号的划分情况，确定各个子带频率信号中包含的频点序号。可以根据频点序号确定对应的各个频点幅值。
其中，频点幅值可以通过步骤402所示的公式进行计算获得。
比如，若Bark域的第0子带对应均匀划分的第0～2个频点，将均匀划分的子带转换为以Bark域为尺度进行子带划分，第0子带的频点幅值可以映射为X b[0]＝1/3*(x[0]+x[1]+x[2])。
在步骤408中，根据频点幅值，确定至少一个子带频域信号的功率谱值。
在本申请实施例中，通过获取各个子带频域信号中包含的各个频点幅值，确定各个子带频域信号对应的功率谱值。
在一种可能的实现方式中，经过傅里叶变换后的获得的各个频点幅值，可以进一步计算各个频点幅值的平方即为对应的功率谱值。
其中，通过各个频点幅值计算对应的功率谱值的计算公式可以是如下所示：
S(i,k)＝|X(i,k)| 2k＝1,2,3…,N
其中，i为对应的帧序号，k为对应的频点索引值。
在一种可能的实现方式中，经过改进的离散余弦变换后获得的各个频点幅值，也可以通过计算各个频点幅值的平方作为对应的功率谱值。
在步骤409中，将至少一个子带频域信号的功率谱值输入到频谱补偿模型中，获取各个子带频域信号分别对应的子带预测结果。
在本申请实施例中，将各个子带频域信号对应的功率谱值输入到频谱补偿模型中，获得各个子带频域信号对应的预测功率谱值，作为各个子带频域信号的子带预测结果。根据所述子带预测结果确定对所述第一音频信号中的失真频谱进行预测补偿的预测结果。
在一种可能的实现方式中，将各个子带频域信号对应的功率谱值进行对数化处理，将对数化处理后的功率谱值输入到频谱补偿模型中，获得预测的功率谱值的对数值作为预测结果。
在步骤410中，根据预测结果，获取对第一音频信号重构后的功率谱值。
在本申请实施例中，根据频谱补偿模型输出的各个子带频域信号对应的预测结果，获取各个子带频域信号对应的重构后的功率谱值。
其中，重构后的功率谱值可以是对数值。
在一种可能的实现方式中，根据预测结果可以直接获取对第一音频信号重构后的功率谱值。
其中，将至少一个子带频域信号各自对应的预测功率谱值，直接作为重构后的功率谱值。
比如，由频谱补偿模型输出的预测结果，功率谱对数值为
Figure PCTCN2021097663-appb-000003
可以将
Figure PCTCN2021097663-appb-000004
直接作为重构后的功率谱值。
在另一种可能的实现方式中，根据预测结果与输入的各个子带频域信号对应的功率谱值，确定重构后的功率谱值。
其中，可以将第一音频信号对应的功率谱值与频带损伤率之和，作为重构后的功率谱值。
其中，频带损伤率是至少一个子带频域信号各自对应的预测功率谱值，与第一音频信号对应的功率谱值之差的历史平滑值。
比如，由频谱补偿模型输出的预测结果，功率谱对数值为
Figure PCTCN2021097663-appb-000005
各个频点的对应的实际功率谱对数值和预测功率谱对数值之间的差值为：
Figure PCTCN2021097663-appb-000006
然后，通过各个频点对应的该差值，计算实际功率谱对数值和预测功率谱对数值之间差值的历史平滑值，将该历史平滑值确定为频带损伤率，历史平滑值可以通过如下公式进行计算：
Figure PCTCN2021097663-appb-000007
其中，α为取值范围为(0,1)的参数。
在一种可能的实现方式中，若通过非线性频域变换划分子带频域信号，则将频点对应的实际功率谱对数值和预测功率谱对数值之间的差值作为该频点的增益，确定重构后的频点幅值。
比如，若X b[0]通过神经网络模型预测得到对应的预测结果为X′ b[0]，将X′ b[0]-X b[0]得到Bark域的第0子带的增益为gain[0]，则重构后的频点为X′[0]＝X[0]+gain[0]；X′[1]＝X[1]+gain[0]；X′[2]＝X[2]+gain[0]。
在步骤411中，生成重构后的功率谱值对应的目标音频信号。
在本申请实施例中，将生成的重构后的功率谱值的对数值转换为线性值， 确定该功率谱值对应的频点幅值，根据频点幅值生成对应的时域信号作为目标音频信号。
在一种可能的实现方式中，当根据预测结果直接获取对第一音频信号重构后的功率谱值时，将作为重构后的功率谱值的至少一个子带频域信号各自对应的预测功率谱值转换为线性值，对该线性值进行开方，确定各个频点幅值。
比如，可以将
Figure PCTCN2021097663-appb-000008
直接作为重构后的功率谱值S′ dB(i,k)。
在另一种可能的实现方式中，当根据预测结果与输入的各个子带频域信号对应的功率谱值确定重构后的功率谱值时，通过将各个子带频域信号对应的功率谱值与频带损伤率相加，得到重构后的功率谱值。将重构后的功率谱值转换为线性值，对该线性值进行开方，确定各个频点幅值。
其中，计算重构后的功率谱值的公式可以如下所示：
Figure PCTCN2021097663-appb-000009
其中，将功率谱对数值转换为线性值的计算公式可以如下所示：
S′(i,k)＝power(10,0.05*S′ dB(i,k))
其中，离散傅里叶变换域中的各个频点幅值可以通过
Figure PCTCN2021097663-appb-000010
进行计算。
在一种可能的实现方式中，对重构后的功率谱值对应的频域信号进行时域变换，获得目标音频信号。
其中，由频域信号转换为时域信号可以通过反离散傅里叶变换或者改进的反离散余弦变换。
其中，反离散傅里叶变换的计算公式如下所示：
Figure PCTCN2021097663-appb-000011
然后，对经过反离散傅里叶变换得到的时域信号进行加窗处理，计算公式如下：
x out(n)＝x idft(n)*win(n)
比如，x out可以是20ms的数据，在当前帧位于10ms的位置，当前帧的输出为前一次频域信号转换成时域信号的20ms数据中的后半10ms数据结果与当前帧的前半10ms数据结果进行相加的结果，将相加结果作为当前帧输出的目标音频信号。
通过步骤404到步骤411，预测设备可以完成通过频谱补偿模型对第一音频信号进行补偿重构，获得目标音频信号。
综上所述，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。
请参考图5，其是根据一示例性实施例示出的一种音频信号处理方法的架构图，该音频信号处理方法可以由音频处理设备执行。其中，上述音频处理设备可以是上述图1所示系统中的模型训练设备110和预测设备120。如图5所示，该音频信号处理方法可以包括两个阶段，分别是模型训练阶段以及模型应用阶段。
模型训练阶段可以是由模型训练设备进行离线训练进行的，模型训练阶段可以包括以下步骤51：
在步骤51中，以频谱损伤音频作为输入端，以及对应的原始音频作为输出目标，对深度神经网络模型进行离线训练，将训练后的神经网络模型进行模型更新。
其中，频谱损伤音频以及对应的原始音频可以是预先存储在样本集中的样本数据，模型训练设备将频谱损伤音频作为输入端，对应的原始音频作为输出目标，对可以是循环神经网络模型RNN或者长短期记忆网络模型LSTM的频谱补偿模型进行训练及更新。
模型应用阶段可以是由预测设备进行在线执行，模型应用阶段可以包括以下步骤52到步骤59：
在步骤52中，预测设备通过音频采集功能对外界的音频信号进行采集。
其中，音频处理设备中的负责音频采集的模块，比如可以通过麦克风模块对外界的音频进行采集，获得音频信号。
在步骤53中，预测设备将采集获得的音频信号通过该预测设备中固有的软硬件语音增强处理，生成语音增强处理后的音频信号。
其中，固有的软硬件语音增强处理可以包括应用不同的语音增强算法对音频信号进行处理，生成的语音增强处理后的音频信号可以具有不同程度的信号损伤。
在步骤54中，语音增强处理后的音频信号为一种时域信号，将该时域信号经过运算变换为频域信号。
其中，时域信号变换为频域信号的运算方式可以是离散傅里叶变换或者离散余弦变换。
在步骤55中，将该频域信号输入到神经网络模型中进行预测。
其中，由于频谱补偿模型进行离线训练时，输入端与目标输出端的样本为功率谱的对数值，所以需要计算该频域信号对应的功率谱对数值，将该功率谱对数值输入到频谱补偿模型中进行预测。
在步骤56中，神经网络模型输出的预测结果为预测的修复语音增强处理后失真信号的预测音频信号，通过获取到的预测音频信号与之前输入的频域信号进行子带损伤率分析，获取当前信号的子带频域信号损伤率。
其中，频谱补偿模型输出的预测结果为预测音频信号对应的功率谱对数值，可以通过将该功率谱对数值与输入的频域信号对应的功率谱对数值进行子带损伤率分析，子带损伤率即为该功率谱对数值与输入的频域信号对应的功率谱对数值差值的历史平滑值。
其中，该步骤可以省略，即将直接获取的预测音频信号作为依据执行接下来的步骤。
在步骤57中，对受损频带进行重构，生成重构后的频域信号。
其中，输入的各个子带频域信号为受损频带，将通过子带损伤率的方式计算确定重构后的频域信号的功率谱对数值，或者，当省略步骤56时，将直接获取的预测音频信号的功率谱对数值作为重构后的频域信号的功率谱对数值。
在步骤58中，将重构后的频域信号转换为时域信号。
其中，若在步骤54中，时域信号变换为频域信号的运算方式为离散傅里叶变换时，将重构后的频域信号转换为时域信号的计算方式可以运用反离散傅里叶变换，或者若在步骤54中，时域信号变换为频域信号的运算方式为离散余弦变换时，将重构后的频域信号转换为时域信号的计算方式可以运用反离散余弦变换。
在步骤59中，将该时域信号作为修复完成的目标音频进行输出。
其中，将重构后的频域信号转换成的时域信号作为目标音频信号进行输出， 可以通过预测设备的音频播放模块进行播放，比如，通过扬声器模块进行目标音频的播放。
综上所述，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。
请参考图6，其是根据一示例性实施例示出的一种应用在语音通话系统中的语音修复示意图，该语音通话系统可以包括语音发送端620、语音接收端640以及进行数据传输的服务器端630。其中，如图6所示，在语音通话系统中进行语音修复的过程可以如下：
在通话过程中，语音发送端620从外界采集到语音信号，并且在语音发送端620中对采集到的语音信号进行固有的软硬件语音增强处理，然后通过上述实施例中的频谱补偿模型对语音增强处理后有一定损伤的语音信号进行语音修复，将修复后的增强语音信号通过服务器端630发送到语音接收端640中，语音接收端640通过本身的语音播放模块对修复后的增强语音信号进行播放。
其中，在语音发送端620中的频谱补偿模型是经过模型训练设备610进行离线模型训练获得的，语音发送端620可以通过在应用程序中下载安装并且更新该训练完成的频谱补偿模型。
请参考图7，其是根据一示例性实施例示出的另一种应用在语音通话系统中的语音修复示意图，该语音通话系统可以包括语音发送端720、语音接收端740以及进行数据传输的服务器端730。其中，如图7所示，在语音通话系统中进行语音修复的过程可以如下：
在通话过程中，语音发送端720从外界采集到语音信号，并且在语音发送端720中对采集到的语音信号进行固有的软硬件语音增强处理，然后将语音增强处理后有一定损伤的语音信号发送到服务器端730，在服务器端730通过上述实施例中的频谱补偿模型对语音增强处理后有一定损伤的语音信号进行语音修复处理，然后将修复后的增强语音信号发送到语音接收端740中，语音接收端640通过本身的语音播放模块对修复后的增强语音信号进行播放。
其中，在服务器端730中的频谱补偿模型是经过模型训练设备710进行离线模型训练获得的，服务器端730可以通过获取该训练完成的频谱补偿模型，对 语音增强处理后有一定损伤的语音信号进行语音修复处理。
综上所述，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。
图8是根据一示例性实施例示出的一种音频信号处理装置的结构方框图。该音频信号处理方法可以由音频处理设备执行，以执行图3或图4对应实施例所示的方法中的全部或者部分步骤。该音频信号处理装置可以包括：
信号获取模块810，用于获取第一音频信号；
结果获取模块820，用于通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果；所述频谱补偿模型是通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到的神经网络模型；
目标获取模块830，用于根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号。
在一种可能实现的方法中，所述结果获取模块820，包括：
频域转换子模块，用于将所述第一音频信号转换为对应的频域信号；
子带划分子模块，用于将所述频域信号划分为至少一个子带频域信号；
幅值确定子模块，用于确定所述至少一个子带频域信号中的频点幅值；
功率谱值确定子模块，用于根据所述频点幅值，确定所述至少一个子带频域信号的功率谱值；
子带结果获取子模块，用于将所述至少一个子带频域信号的功率谱值输入到所述频谱补偿模型中，获取所述至少一个子带频域信号分别对应的子带预测结果。根据所述子带预测结果确定对所述第一音频信号中的失真频谱进行预测补偿的预测结果。
在一种可能的实现方式中，所述频域转换子模块，包括：
加窗处理单元，用于对所述第一音频信号进行分帧加窗处理，确定所述第一音频信号处理后的时域信号；
信号获取单元，用于对所述处理后的时域信号进行频域转换，获得对应的所述频域信号。
在一种可能的实现方式中，所述信号获取单元，用于，
通过对所述处理后的时域信号进行离散傅里叶变换，获得对应的所述频域信号；
或者，通过对所述处理后的时域信号进行改进离散余弦变换，获得对应的所述频域信号。
在一种可能的实现方式中，所述子带划分子模块，包括：
子带划分单元，用于以Bark域为粒度，将所述频域信号划分为所述至少一个子带频域信号。
在一种可能的实现方式中，所述子带结果获取子模块，包括：
子带结果获取单元，用于将所述至少一个子带频域信号对应的功率谱值输入到所述频谱补偿模型中，将获得所述至少一个子带频域信号对应的预测功率谱值，作为所述至少一个子带频域信号的子带预测结果。
在一种可能的实现方式中，所述目标获取模块830，包括：
功率谱值获取子模块，用于根据所述预测结果，获取对所述第一音频信号重构后的功率谱值；
目标生成子模块，用于生成所述重构后的功率谱值对应的所述目标音频信号。
在一种可能的实现方式中，所述目标获取模块830，包括：
功率谱值生成子模块，用于将所述第一音频信号对应的功率谱值与频带损伤率之和，作为所述重构后的功率谱值；所述频带损伤率是所述至少一个子带频域信号分别对应的预测功率谱值，与所述第一音频信号对应的功率谱值之差的历史平滑值；
或者，
功率谱值确定子模块，用于将所述至少一个子带频域信号分别对应的预测功率谱值，作为所述重构后的功率谱值。
在一种可能的实现方式中，所述目标生成子模块，包括：
时域转换单元，用于对所述重构后的功率谱值对应的频域信号进行时域变换，获得所述目标音频信号。
在一种可能的实现方式中，所述信号获取模块810，包括：
信号获取子模块，用于获取经过音频增强处理后的所述第一音频信号。
在一种可能的实现方式中，所述装置还包括：
样本获取模块，用于通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果之前，获取所述原始音频样本；
失真样本获取模块，用于对所述原始音频样本对应的频域信号中的部分频带进行功率谱值的抑制处理，得到所述原始音频样本对应的频谱失真音频样本；
模型获取模块，用于以所述频谱失真音频样本为输入，并以所述原始音频样本为训练目标进行机器学习训练，获得所述频谱补偿模型。
在一种可能的实现方式中，所述频谱补偿模型是RNN或者LSTM。
综上所述，通过模型训练得到的神经网络模型可以针对不同的设备机型和软件版本统一进行补偿预测，解决了语音信号修复的应用场景存在局限性的问题，从而提高了语音信号修复的通用性。
图9是根据一示例性实施例示出的一种计算机设备的结构示意图。该计算机设备可以实现为音频处理设备。该音频处理设备可以包含图1所示的模型训练设备110以及预测设备120。所述计算机设备900包括中央处理单元(Central Processing Unit，CPU)901、包括随机存取存储器(Random Access Memory，RAM)902和只读存储器(Read-Only Memory，ROM)903的系统存储器904，以及连接系统存储器904和中央处理单元901的系统总线905。所述计算机设备900还包括帮助计算机内的各个器件之间传输信息的基本输入/输出系统(Input/Output，I/O系统)906，和用于存储操作系统913、应用程序914和其他程序模块915的大容量存储设备907。
所述大容量存储设备907通过连接到系统总线905的大容量存储控制器(未示出)连接到中央处理单元901。所述大容量存储设备907及其相关联的计算机可读介质为计算机设备900提供非易失性存储。也就是说，所述大容量存储设备907可以包括诸如硬盘或者只读光盘(Compact Disc Read-Only Memory，CD-ROM)驱动器之类的计算机可读介质(未示出)。
不失一般性，所述计算机可读介质可以包括计算机存储介质和通信介质。计算机存储介质包括以用于存储诸如计算机可读指令、数据结构、程序模块或 其他数据等信息的任何方法或技术实现的易失性和非易失性、可移动和不可移动介质。计算机存储介质包括RAM、ROM、可擦除可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、带电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、CD-ROM、数字视频光盘(Digital Video Disc，DVD)或其他光学存储、磁带盒、磁带、磁盘存储或其他磁性存储设备。当然，本领域技术人员可知所述计算机存储介质不局限于上述几种。上述的系统存储器904和大容量存储设备907可以统称为存储器。
计算机设备900可以通过连接在所述系统总线905上的网络接口单元911连接到互联网或者其它网络设备。
所述存储器还包括一个或者一个以上的程序，所述一个或者一个以上程序存储于存储器中，中央处理器901通过执行该一个或一个以上程序来实现图3或图4所示的方法的全部或者部分步骤。
在示例性实施例中，还提供了一种包括指令的非临时性计算机可读存储介质，例如包括计算机程序(指令)的存储器，上述程序(指令)可由计算机设备的处理器执行以完成本申请各个实施例所示的方法的全部或者部分步骤。例如，所述非临时性计算机可读存储介质可以是只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、光盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、磁带、软盘和光数据存储设备等。
根据本申请的一个方面，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述方面的各种可选实现方式中提供的音频信号处理方法。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (16)

  1. 一种音频信号处理方法，所述方法由音频处理设备执行，所述方法包括：
    获取第一音频信号；
    通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果；所述频谱补偿模型是通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到的神经网络模型；
    根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号。
  2. 根据权利要求1所述的方法，所述通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果，包括：
    将所述第一音频信号转换为对应的频域信号；
    将所述频域信号划分为至少一个子带频域信号；
    确定所述至少一个子带频域信号中的频点幅值；
    根据所述频点幅值，确定所述至少一个子带频域信号的功率谱值；
    将所述至少一个子带频域信号的功率谱值输入到所述频谱补偿模型中，获取所述至少一个子带频域信号分别对应的子带预测结果；
    根据所述子带预测结果确定对所述第一音频信号中的失真频谱进行预测补偿的预测结果。
  3. 根据权利要求2所述的方法，所述将所述第一音频信号转换为对应的频域信号，包括：
    对所述第一音频信号进行分帧加窗处理，确定所述第一音频信号处理后的时域信号；
    对所述处理后的时域信号进行频域转换，获得对应的所述频域信号。
  4. 根据权利要求3所述的方法，所述对所述处理后的时域信号进行频域转换，获得对应的所述频域信号，包括：
    通过对所述处理后的时域信号进行离散傅里叶变换，获得对应的所述频域 信号；
    或者，
    通过对所述处理后的时域信号进行改进离散余弦变换，获得对应的所述频域信号。
  5. 根据权利要求2所述的方法，所述将所述频域信号划分为至少一个子带频域信号，包括：
    以Bark域为粒度，将所述频域信号划分为所述至少一个子带频域信号。
  6. 根据权利要求2所述的方法，所述将所述至少一个子带频域信号的功率谱值输入到所述频谱补偿模型中，获取所述至少一个子带频域信号分别对应的子带预测结果，包括：
    将所述至少一个子带频域信号对应的功率谱值输入到所述频谱补偿模型中，将获得所述至少一个子带频域信号对应的预测功率谱值，作为所述至少一个子带频域信号的子带预测结果。
  7. 根据权利要求6所述的方法，所述根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号，包括：
    根据所述预测结果，获取对所述第一音频信号重构后的功率谱值；
    生成所述重构后的功率谱值对应的所述目标音频信号。
  8. 根据权利要求7所述的方法，所述根据所述预测结果，获取对所述第一音频信号重构后的功率谱值，包括：
    将所述第一音频信号对应的功率谱值与频带损伤率之和，作为所述重构后的功率谱值；所述频带损伤率是所述至少一个子带频域信号分别对应的预测功率谱值，与所述第一音频信号对应的功率谱值之差的历史平滑值；
    或者，
    将所述至少一个子带频域信号分别对应的预测功率谱值，作为所述重构后的功率谱值。
  9. 根据权利要求7所述的方法，所述生成所述重构后的功率谱值对应的所述目标音频信号，包括：
    对所述重构后的功率谱值对应的频域信号进行时域变换，获得所述目标音 频信号。
  10. 根据权利要求1所述的方法，所述获取第一音频信号，包括：
    获取经过音频增强处理后的所述第一音频信号。
  11. 根据权利要求1所述的方法，所述方法还包括：
    获取所述原始音频样本；
    对所述原始音频样本对应的频域信号中的部分频带进行功率谱值的抑制处理，得到所述原始音频样本对应的所述频谱失真音频样本；
    以所述频谱失真音频样本为输入，并以所述原始音频样本为训练目标进行机器学习训练，获得所述频谱补偿模型。
  12. 根据权利要求1至11任一所述的方法，所述频谱补偿模型是循环神经网络模型RNN或者长短期记忆网络模型LSTM。
  13. 一种音频信号处理装置，所述装置包括：
    信号获取模块，用于获取第一音频信号；
    结果获取模块，用于通过频谱补偿模型对所述第一音频信号进行处理，获得对所述第一音频信号中的失真频谱进行预测补偿的预测结果；所述频谱补偿模型是通过频谱失真音频样本，以及所述频谱失真音频样本对应的原始音频样本训练得到的神经网络模型；
    目标获取模块，用于根据所述预测结果，对所述第一音频信号进行重构，获得对所述第一音频信号中的失真频谱进行修复后的目标音频信号。
  14. 一种计算机设备，所述计算机设备包含处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现如权利要求1至12任一所述的音频信号处理方法。
  15. 一种计算机可读存储介质，所述存储介质用于存储计算机程序，所述计算机程序用于执行权利要求1至12任一所述的音频信号处理方法。
  16. 一种包括指令的计算机程序产品，当其在计算机上运行时，使得所述计算机执行权利要求1至12任一所述的音频信号处理方法。
PCT/CN2021/097663 2020-07-13 2021-06-01 音频信号处理方法和相关装置 Ceased WO2022012195A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010670626.9A CN112820315B (zh) 2020-07-13 2020-07-13 音频信号处理方法、装置、计算机设备及存储介质
CN202010670626.9 2020-07-13

Publications (1)

Publication Number Publication Date
WO2022012195A1 true WO2022012195A1 (zh) 2022-01-20

Family

ID=75853211

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/097663 Ceased WO2022012195A1 (zh) 2020-07-13 2021-06-01 音频信号处理方法和相关装置

Country Status (2)

Country Link
CN (1) CN112820315B (zh)
WO (1) WO2022012195A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114866856A (zh) * 2022-05-06 2022-08-05 北京达佳互联信息技术有限公司 音频信号的处理方法、音频生成模型的训练方法及装置
CN114974299A (zh) * 2022-08-01 2022-08-30 腾讯科技（深圳）有限公司 语音增强模型的训练、增强方法、装置、设备、介质
CN115424630A (zh) * 2022-06-14 2022-12-02 青岛信芯微电子科技股份有限公司 一种目标端到端模型的训练方法、混合音频信号处理方法
CN116248229A (zh) * 2022-12-08 2023-06-09 南京龙垣信息科技有限公司 一种面向实时语音通讯的丢包补偿方法
CN116312571A (zh) * 2023-02-01 2023-06-23 深圳大学 丢包补偿模型训练方法、丢包补偿方法及装置
CN116741191A (zh) * 2023-07-06 2023-09-12 北京达佳互联信息技术有限公司 音频信号处理方法、装置、电子设备及存储介质
CN117395181A (zh) * 2023-12-12 2024-01-12 方图智能(深圳)科技集团股份有限公司 基于物联网的低延时多媒体音频传输检测方法及系统
CN118918882A (zh) * 2024-08-23 2024-11-08 美的集团(上海)有限公司 模型训练方法、装置、电子设备和非易失性可读存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112820315B (zh) * 2020-07-13 2023-01-06 腾讯科技（深圳）有限公司 音频信号处理方法、装置、计算机设备及存储介质
CN113612808B (zh) * 2021-10-09 2022-01-25 腾讯科技（深圳）有限公司 音频处理方法、相关设备、存储介质及程序产品
CN116052712B (zh) * 2021-10-28 2026-02-13 腾讯科技（深圳）有限公司 语音信号处理方法、装置、计算设备及存储介质
CN114360564A (zh) * 2021-12-30 2022-04-15 西安讯飞超脑信息科技有限公司 语音增强方法、相关设备及可读存储介质
CN114822567B (zh) * 2022-06-22 2022-09-27 天津大学 一种基于能量算子的病理嗓音频谱重构方法
CN116343814A (zh) * 2023-03-27 2023-06-27 美的集团(上海)有限公司 频响均衡模型的训练方法、语音合成方法和可读存储介质
CN117912477A (zh) * 2023-11-20 2024-04-19 广州深空智能科技有限公司 一种通过声纹补偿失真语音的系统及方法
CN119905110B (zh) * 2025-01-27 2025-12-16 北京华控智加科技有限公司 一种基于预训练神经网络的任意采样率声音分析方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101512938A (zh) * 2006-08-01 2009-08-19 Dts（英属维尔京群岛）有限公司 用于补偿音频变换器的线性和非-线性失真的神经网络滤波技术
CN107112025A (zh) * 2014-09-12 2017-08-29 美商楼氏电子有限公司 用于恢复语音分量的系统和方法
CN107895571A (zh) * 2016-09-29 2018-04-10 亿览在线网络技术（北京）有限公司 无损音频文件识别方法及装置
CN109147805A (zh) * 2018-06-05 2019-01-04 安克创新科技股份有限公司 基于深度学习的音频音质增强
CN109686381A (zh) * 2017-10-19 2019-04-26 恩智浦有限公司 用于信号增强的信号处理器和相关方法
US20200090676A1 (en) * 2018-09-17 2020-03-19 Honeywell International Inc. System and method for audio noise reduction
CN112820315A (zh) * 2020-07-13 2021-05-18 腾讯科技（深圳）有限公司 音频信号处理方法、装置、计算机设备及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101512938A (zh) * 2006-08-01 2009-08-19 Dts（英属维尔京群岛）有限公司 用于补偿音频变换器的线性和非-线性失真的神经网络滤波技术
CN107112025A (zh) * 2014-09-12 2017-08-29 美商楼氏电子有限公司 用于恢复语音分量的系统和方法
CN107895571A (zh) * 2016-09-29 2018-04-10 亿览在线网络技术（北京）有限公司 无损音频文件识别方法及装置
CN109686381A (zh) * 2017-10-19 2019-04-26 恩智浦有限公司 用于信号增强的信号处理器和相关方法
CN109147805A (zh) * 2018-06-05 2019-01-04 安克创新科技股份有限公司 基于深度学习的音频音质增强
US20200090676A1 (en) * 2018-09-17 2020-03-19 Honeywell International Inc. System and method for audio noise reduction
CN112820315A (zh) * 2020-07-13 2021-05-18 腾讯科技（深圳）有限公司 音频信号处理方法、装置、计算机设备及存储介质

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114866856A (zh) * 2022-05-06 2022-08-05 北京达佳互联信息技术有限公司 音频信号的处理方法、音频生成模型的训练方法及装置
CN114866856B (zh) * 2022-05-06 2024-01-02 北京达佳互联信息技术有限公司 音频信号的处理方法、音频生成模型的训练方法及装置
CN115424630A (zh) * 2022-06-14 2022-12-02 青岛信芯微电子科技股份有限公司 一种目标端到端模型的训练方法、混合音频信号处理方法
WO2024027295A1 (zh) * 2022-08-01 2024-02-08 腾讯科技（深圳）有限公司 语音增强模型的训练、增强方法、装置、电子设备、存储介质及程序产品
CN114974299A (zh) * 2022-08-01 2022-08-30 腾讯科技（深圳）有限公司 语音增强模型的训练、增强方法、装置、设备、介质
CN114974299B (zh) * 2022-08-01 2022-10-21 腾讯科技（深圳）有限公司 语音增强模型的训练、增强方法、装置、设备、介质
CN116248229A (zh) * 2022-12-08 2023-06-09 南京龙垣信息科技有限公司 一种面向实时语音通讯的丢包补偿方法
CN116248229B (zh) * 2022-12-08 2023-12-01 南京龙垣信息科技有限公司 一种面向实时语音通讯的丢包补偿方法
CN116312571A (zh) * 2023-02-01 2023-06-23 深圳大学 丢包补偿模型训练方法、丢包补偿方法及装置
CN116741191A (zh) * 2023-07-06 2023-09-12 北京达佳互联信息技术有限公司 音频信号处理方法、装置、电子设备及存储介质
CN117395181A (zh) * 2023-12-12 2024-01-12 方图智能(深圳)科技集团股份有限公司 基于物联网的低延时多媒体音频传输检测方法及系统
CN117395181B (zh) * 2023-12-12 2024-02-13 方图智能(深圳)科技集团股份有限公司 基于物联网的低延时多媒体音频传输检测方法及系统
CN118918882A (zh) * 2024-08-23 2024-11-08 美的集团(上海)有限公司 模型训练方法、装置、电子设备和非易失性可读存储介质

Also Published As

Publication number Publication date
CN112820315A (zh) 2021-05-18
CN112820315B (zh) 2023-01-06

Similar Documents

Publication Publication Date Title
WO2022012195A1 (zh) 音频信号处理方法和相关装置
CN110322891B (zh) 一种语音信号的处理方法、装置、终端及存储介质
JP7636088B2 (ja) 音声強調方法、装置、機器及びコンピュータプログラム
KR20230013054A (ko) 심층 신경망을 사용하는 시변 및 비선형 오디오 처리
JP7615510B2 (ja) 音声強調方法、音声強調装置、電子機器、及びコンピュータプログラム
CN112233693B (zh) 一种音质评估方法、装置和设备
JP6827908B2 (ja) 音源強調装置、音源強調学習装置、音源強調方法、プログラム
CN113990343B (zh) 语音降噪模型的训练方法和装置及语音降噪方法和装置
CN114333892A (zh) 一种语音处理方法、装置、电子设备和可读介质
CN111326166B (zh) 语音处理方法及装置、计算机可读存储介质、电子设备
CN114333893A (zh) 一种语音处理方法、装置、电子设备和可读介质
CN114333891A (zh) 一种语音处理方法、装置、电子设备和可读介质
CN115240701B (zh) 降噪模型的训练方法、语音降噪方法、装置和电子设备
JP2000069597A (ja) インパルス応答測定方法
CN113593604A (zh) 检测音频质量方法、装置及存储介质
CN110875037A (zh) 语音数据处理方法、装置及电子设备
CN113571081B (zh) 语音增强方法、装置、设备及存储介质
CN118197350A (zh) 一种噪声提取模型的训练、噪声提取方法及装置和模型
CN115101084A (zh) 模型训练方法、音频处理方法、装置、音箱、设备及介质
Campbell et al. Feature extraction of automatic speaker recognition, analysis and evaluation in real environment
HK40043824A (zh) 音频信号处理方法、装置、计算机设备及存储介质
HK40043824B (zh) 音频信号处理方法、装置、计算机设备及存储介质
CN114512141B (zh) 音频分离的方法、装置、设备、存储介质和程序产品
HK40052886B (zh) 语音增强方法、装置、设备及存储介质
US20250029623A1 (en) Electronic apparatus and controlling method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21843222

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21843222

Country of ref document: EP

Kind code of ref document: A1