WO2025007868A1 - 音频信号恢复方法、装置、电子设备及可读存储介质 - Google Patents

音频信号恢复方法、装置、电子设备及可读存储介质 Download PDF

Info

Publication number
WO2025007868A1
WO2025007868A1 PCT/CN2024/103191 CN2024103191W WO2025007868A1 WO 2025007868 A1 WO2025007868 A1 WO 2025007868A1 CN 2024103191 W CN2024103191 W CN 2024103191W WO 2025007868 A1 WO2025007868 A1 WO 2025007868A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
time domain
feature information
domain feature
residual network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/103191
Other languages
English (en)
French (fr)
Inventor
张勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vivo Mobile Communication Co Ltd
Original Assignee
Vivo Mobile Communication Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vivo Mobile Communication Co Ltd filed Critical Vivo Mobile Communication Co Ltd
Publication of WO2025007868A1 publication Critical patent/WO2025007868A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/005—Correction of errors induced by the transmission channel, if related to the coding algorithm
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/18—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being spectral information of each sub-band
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
    • G10L25/21—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters the extracted parameters being power information
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/764—Media network packet handling at the destination 

Definitions

  • the present application belongs to the field of signal processing, and specifically relates to an audio signal recovery method, device, electronic device and readable storage medium.
  • IP Internet Protocol
  • the purpose of the embodiments of the present application is to provide an audio signal recovery method, device, electronic device and readable storage medium, which can improve the quality of voice calls.
  • an embodiment of the present application provides an audio signal recovery method, the method comprising: when a first voice signal is lost, inputting a cached second voice signal into a first audio signal recovery model, the first audio signal recovery model comprising an encoding module and a decoding module, the encoding module comprising M layers of first residual network layers, and the decoding module comprising M layers of second residual network layers; processing the second voice signal sequentially through the M layers of first residual network layers to obtain first time domain feature information; processing the first time domain feature information sequentially through the M layers of second residual network layers to obtain a target voice signal; wherein the second voice signal comprises at least one of the following: X frames of voice signal before the first voice signal, Y frames of voice signal after the first voice signal, and M, X, and Y are positive integers.
  • an embodiment of the present application provides an audio signal recovery device, which includes: an input module and a processing module; the input module is used to input the cached second voice signal into a first audio signal recovery model when the first voice signal is lost, and the first audio signal recovery model includes an encoding module and a decoding module, the encoding module includes M layers of first residual network layers, and the decoding module includes M layers of second residual network layers; the processing module is used to process the second voice signal through M layers of first residual network layers in sequence to obtain first time domain feature information; the processing module is also used to process the first time domain feature information through M layers of second residual network layers in sequence to obtain a target voice signal; wherein the second voice signal includes at least one of the following: X frames of voice signal before the first voice signal, Y frames of voice signal after the first voice signal, and M, X, and Y are positive integers.
  • an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
  • an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
  • an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
  • an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
  • the cached second voice signal is input into the first audio signal recovery model
  • the first audio signal recovery model includes an encoding module and a decoding module
  • the encoding module includes M layers of first residual network layers
  • the decoding module includes M layers of second residual network layers
  • the second voice signal is sequentially processed by the M layers of first residual network layers to obtain first time domain feature information
  • the first time domain feature information is sequentially processed by the M layers of second residual network layers to obtain a target voice signal
  • the second voice signal includes at least one of the following: X frames of voice signal before the first voice signal, Y frames of voice signal after the first voice signal, M, X, Y are positive integers.
  • the residual network layer of the encoding module in the audio signal recovery model extracts the time domain feature information of the cached voice signal before or after the lost voice signal, and synthesizes the time domain feature information extracted by the encoding module through the residual network layer of the decoding module in the audio signal recovery model to recover the lost voice signal.
  • the lost voice signal can be recovered by inputting the cached voice signal into the audio signal recovery model when the voice signal is lost, the lost voice signal can be recovered through the audio signal recovery model during the transmission of the voice signal, thereby avoiding the user's perception of the loss of the voice data packet, reducing the negative impact of the loss of the voice data packet, and thus effectively improving the quality of the voice call.
  • FIG1 is a schematic structural diagram of an AMR-WB decoder principle provided in an embodiment of the present application.
  • FIG2 is a flowchart of an example of packet loss concealment of an AMR-WB decoder provided in an embodiment of the present application
  • FIG. 3 is a schematic diagram of an example of an AMR-WB decoder packet loss concealment state machine judging error frames and normal frames provided by an embodiment of the present application;
  • FIG4 is a flow chart of a DNN-based packet loss concealment method provided in an embodiment of the present application.
  • FIG5 is a flow chart of a feature extraction based on DNN provided in an embodiment of the present application.
  • FIG6 is a flowchart of a DNN-based speech time domain signal waveform reconstruction provided in an embodiment of the present application.
  • FIG7 is a flow chart of an audio signal recovery method provided in an embodiment of the present application.
  • FIG8 is a schematic diagram of the structure of a residual network provided in an embodiment of the present application.
  • FIG9 is a schematic diagram of a network structure of an encoding module provided in an embodiment of the present application.
  • FIG10 is a flow chart of recovering a lost voice signal provided by an embodiment of the present application.
  • FIG11 is a schematic diagram of a network structure of a decoding module provided in an embodiment of the present application.
  • FIG12 is a flow chart of a training GAN network model provided in an embodiment of the present application.
  • FIG. 13 is a training flowchart of a second audio signal recovery model provided in an embodiment of the present application.
  • FIG14 is a schematic diagram of the structure of an MS-STFT discriminator provided in an embodiment of the present application.
  • FIG15 is a schematic diagram of an example flow chart of an audio signal recovery method provided in an embodiment of the present application.
  • FIG16 is a schematic diagram of the structure of an audio signal recovery device provided in an embodiment of the present application.
  • FIG17 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application.
  • FIG. 18 is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
  • first, second, etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first”, “second”, etc. are generally of one type, and the number of objects is not limited.
  • the first object can be one or more.
  • “and/or” in the specification and claims represents at least one of the connected objects, and the character “/" generally indicates that the objects associated with each other are in an "or” relationship.
  • the audio signal recovery method, device, electronic device and readable storage medium provided in the embodiments of the present application are suitable for scenarios in which voice calls are conducted based on IP.
  • voice is usually divided into smaller packets or frames and transmitted in a certain time sequence.
  • voice packet loss will inevitably occur due to network congestion, jitter delay and other reasons, which will greatly affect the quality of voice calls.
  • voice data packet loss in order to prevent users from feeling the occurrence of packet loss, when packet loss is detected, it is necessary to hide its errors in order to effectively improve the quality of the final voice call.
  • AMR-WB adaptive multi-rate wideband
  • LPC linear predictive coding
  • the AMR-WB decoder first restores the received coded bitstream into parameter coding, and obtains the coding parameters corresponding to the 20ms speech frame after decoding. These parameters are: ISF parameters, 4 pitch periods, 4 fixed codebook vectors, 4 adaptive codebook gains, and 4 fixed codebook gains.
  • the ISF parameters of each frame are converted into ISP parameters, and then interpolated and converted to obtain the LPC synthesis filter coefficients of 4 subframes, each of which contains a 5ms speech frame.
  • the principle block diagram of the AMR-WB decoder is shown in Figure 1.
  • the adaptive codebook excitation v(n) and the fixed codebook excitation c(n) are added to obtain the total excitation signal u(n), and the synthesized speech is obtained through the LPC synthesis filter.
  • the block diagram of the AMR-WB decoder speech coding packet loss concealment system is shown in Figure 2.
  • the decoder detects the packet loss and calls the packet loss concealment module for error recovery.
  • the AMR-WB packet loss concealment algorithm uses a state machine to determine the state of the error frame and restores the coding parameters according to the state of the state machine, including ISF parameters, pitch period, fixed codebook vector, adaptive codebook gain, and fixed codebook gain. After obtaining the restored coding parameters, the AMR-WB decoder is used to synthesize speech.
  • the schematic diagram of the example of the AMR-WB decoder packet loss concealment state machine judging error frames and normal frames is shown in Figure 3.
  • the Bad Frame Indicator (BFI) of the current frame and the Bad Frame Indicator PrevBFI of the previous frame are used as input quantities at the same time.
  • the state of the current frame in the state machine model is determined based on the two input quantities.
  • the solid line in Figure 3 represents the change path of the error frame, and the dotted line represents the change path of the correct frame.
  • the system starts from state 0. Each time an error speech frame is detected, the state counter is increased by 1, and it is saturated when it reaches 6. Each time a correct speech frame is detected, the state counter is shifted right by 1.
  • the state represents the quality of the channel. It can be understood that the larger the value of the state counter, the worse the channel quality.
  • PrevBFI indicates an error frame indication of the previous frame.
  • the fixed codebook gain g c (n) parameter of the currently correctly received frame can be modified by the following formula (1), and it is limited to be less than the fixed codebook gain g c (n-1) value used by the last subframe of the last correctly received frame.
  • g c (n-1) is the fixed codebook gain of the last subframe of the last correctly received frame
  • g c (n) is the fixed codebook gain of the current frame
  • 1.25 is the gain constant. It can be understood that the remaining coding parameters of the current frame can be decoded normally and used for synthesizing speech.
  • the size of the gain factor used in the recovery of the adaptive codebook gain and the fixed codebook gain can be determined according to the state of the state machine. The larger the state value, the worse the channel quality and the larger the attenuation factor.
  • the adaptive codebook gain gp (n) and the fixed codebook gain gc (n) can be replaced by the attenuation value of the previous subframe.
  • the adaptive codebook gain gp (n) can be calculated by the following formula (2):
  • gp (n) is the adaptive codebook gain of the current frame
  • ( gp (n-1), ..., gp (n-5)) is the adaptive codebook gain of the past five subframes
  • median5( gp (n-1), ..., gp (n-5)) represents the median value of the past five adaptive codebook gains gp (n).
  • Pp (state) is the suppression factor
  • the fixed codebook gain g c (n) can be calculated by the following formula (3):
  • gc (n) is the fixed codebook gain of the current frame
  • gc (n-1), ..., gc (n-5) are the fixed codebook gains of the past five subframes
  • median5( gc (n-1), ..., gc (n-5)) represents the median value of the past five fixed codebook gains gc (n).
  • Pc (state) is the suppression factor
  • STATE is the state of the current state machine model
  • Pc (1) 0.50
  • Pc (2) 0.25
  • Pc (3) 0.25
  • Pc (4) 0.25
  • Pc (5) 0.15
  • Pc (6) 0.01. It can be understood that the higher the state value, the more the gain is attenuated.
  • the spectrum coefficient used by the AMR-WB standard is the 16th order ISF parameter.
  • the estimated value of the ISF parameter of the error frame is obtained by adaptive mean value interpolation of the ISF parameter, which can be calculated by the following formula (4):
  • ISF q (i) ⁇ past_ISF q (i)+(1- ⁇ ) ⁇ ISF mean (i)
  • ISF q (i) is the ISF vector parameter of the current frame
  • past_ISF q (i) is the ISF vector parameter of the previous frame
  • ISF mean (i) represents the weighted value of the adaptive mean and the constant ISF, which can be calculated by formula (5):
  • ISF mean (i) ⁇ ISF const_mean (i)+(1- ⁇ ) ⁇ ISF adaptive_mean (i) Formula (5)
  • ISF const_mean (i) is a vector containing the long-term average ISF.
  • the pitch period and adaptive codebook gain of the last 5 correctly received speech frames can be stored in the buffers, which are T buffer and g p buffer respectively. They will be used to find the best pitch period for the current frame.
  • the pitch period recovery algorithm first determines the availability of the pitch period of the past subframe. If the voicing and stability are strong, it means that the error frame has not changed much compared with the past frame, and the pitch period of the lost frame can be replaced by the pitch period of the past frame; otherwise, the pitch period value of the error frame is randomly generated within a certain range.
  • the pitch period parameter recovery algorithm is divided into the following steps 1 and 2:
  • Step 1 Calculate the past pitch cycle availability factor Q lag_t-1 , which can be calculated by formula (7):
  • gp (n-1) represents the adaptive codebook gain of the previous correctly received frame
  • gp (n-2) is the adaptive codebook gain of the correct frame before the previous correctly received frame.
  • Step 2 Estimate the current frame pitch period T. After obtaining Q lag_t-1 in step 1, the current frame pitch period T can be calculated by the following formula (8):
  • T(n-1) is the pitch period of the previous correctly received frame
  • T max is the maximum value in the pitch period buffer
  • T max-1 is the second largest value in the pitch period buffer
  • T max-2 is the third largest value in the pitch period buffer.
  • RND(T max -T max-2 ) is a random number generation function.
  • the fixed codebook vector of the error frame can be replaced by a randomly generated sequence.
  • speech coding for packet loss error concealment methods is usually based on the Code Excited Linear Prediction (CELP) model/algebraic code excited linear prediction (ACELP).
  • CELP Code Excited Linear Prediction
  • ACELP algebraic code excited linear prediction
  • the loss of a complete speech frame information will not only seriously affect the quality of the synthesized speech of the current lost frame, but also affect the subsequent consecutive frames or even dozens of frames.
  • the restored coding parameters of the current packet loss concealment method are inaccurate, which seriously reduces the quality of the synthesized speech.
  • the existing method can only restore the acoustic feature parameters of the lost frame speech, try to maintain the continuity of the restored speech waveform, and cannot restore the semantic information of the lost frame speech.
  • the log power spectrum (LPS) and spectrum phase are used.
  • the flow chart of the PLC method of DNN is shown in FIG4 , where ⁇ Y l and ⁇ Y f represent the LPS characteristics and spectrum phase of the past P frame signal, respectively. and They represent the LPS features and spectral phase predicted by the DNN network respectively.
  • this method uses clean speech to train two DNN networks (both are fully connected networks with 4 hidden layers) to predict the LPS and spectral phase of lost speech frames respectively.
  • the packet loss recovery phase first determine whether the current received frame is a lost signal frame. If it is not a lost frame, perform normal decoding. If it is a lost frame, call the packet loss concealment method for recovery.
  • Y t represents the input time domain signal
  • Y f represents the spectrum obtained after performing Discrete Fourier Transform (DFT) on the input time domain signal
  • 2 and Log( ⁇ ) represent square operation and logarithmic operation respectively.
  • the output of the feature extraction module is the LPS feature Y l and the spectrum phase ⁇ Yf. and Represent the LPS feature and the spectral phase predicted by the DNN network, respectively, and exp( ⁇ ) represents the exponential operation.
  • the output of the waveform reconstruction module is the reconstructed time domain signal
  • the structure of the training set data itself is very important for the accuracy of network training. From the perspective of probability, the structure means that the data obeys a certain probability distribution.
  • the speech signal presents a complex Gaussian distribution in the frequency domain. Under this condition, it can be proved that the amplitude of the speech signal spectrum obeys the Rayleigh distribution, and the phase of the speech signal spectrum obeys the uniform distribution.
  • LPS is obtained by transforming the amplitude value, retaining the structure, and can be learned by the DNN network.
  • the phase of the speech signal spectrum obeys the uniform distribution, which is highly random and difficult to be learned by the DNN network. Therefore, the spectrum phase predicted by the DNN network is not accurate, which reduces the quality of the synthesized speech signal.
  • the residual network layer of the encoding module in the audio signal recovery model can be used to extract the time domain feature information of the cached voice signal before or after the lost voice signal, and the residual network layer of the decoding module in the audio signal recovery model can be used to synthesize the time domain feature information extracted by the encoding module to recover the lost voice signal.
  • the lost voice signal can be recovered by inputting the cached voice signal into the audio signal recovery model when the voice signal is lost, the lost voice signal can be recovered by the audio signal recovery model during the transmission of the voice signal, thereby avoiding the user's perception of the loss of the voice data packet, reducing the negative impact of the loss of the voice data packet, and effectively improving the quality of the voice call.
  • the audio signal recovery method provided in the embodiment of the present application may be executed by an audio signal recovery device.
  • the audio signal recovery device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip.
  • the audio signal recovery method provided in the embodiment of the present application will be described below by taking the audio signal recovery device as an example.
  • FIG7 shows a flowchart of an audio signal recovery method provided by the present application embodiment.
  • the method can be applied to an audio signal recovery device.
  • the audio signal recovery method provided by the present application embodiment can include the following steps 101 to 103.
  • Step 101 When a first voice signal is lost, the audio signal recovery device inputs a cached second voice signal into a first audio signal recovery model.
  • the second voice signal includes at least one of the following: X frames of voice signal before the first voice signal, and Y frames of voice signal after the first voice signal, where X and Y are positive integers.
  • the first voice signal may be one or more frames of voice signals in a current voice call, or one or more frames of audio signals in a recorded audio.
  • the first voice signal loss may indicate that the audio signal receiving end receives
  • the speech signal is not a continuous speech signal, that is, there are missing speech signal frames in the speech signal received by the audio signal receiving end.
  • the audio signal transmitter may send a 10-frame voice signal to the audio signal receiver, and the audio signal receiver only receives the voice signals of frames 1 to 8 and frame 10, but does not receive the voice signal of frame 9.
  • the voice signal of frame 9 is the lost voice signal, i.e., the first voice signal.
  • the audio signal recovery device may cache the first few frames of the first voice signal, or may cache the last few frames of the first voice signal for subsequent use. Specifically, the audio signal recovery device caches at least one of the X frames of voice signal before the first voice signal and the Y frames of voice signal after the first voice signal.
  • the audio signal recovery device can cache the 1st to 3rd frames of voice signals, or the 5th frame of voice signals, or the 1st to 3rd frames of voice signals and the 5th frame of voice signals.
  • the audio signal recovery device caches at least X frames of voice signals before the first voice signal; in non-real-time audio, such as voice messages, the audio signal recovery device caches at least X frames of voice signals before the first voice signal and Y frames of voice signals after the first voice signal.
  • the first audio signal recovery model includes an encoding module and a decoding module, wherein the encoding module includes M layers of first residual network layers, and the decoding module includes M layers of second residual network layers, where M is a positive integer.
  • both the encoding module and the decoding module in the above-mentioned first audio signal recovery model can use a convolutional encoder-decoder network (CEDN) to convolve the time domain feature information of the speech signal.
  • CEDN convolutional encoder-decoder network
  • the CEDN network is a symmetric network consisting of a pre-convolution layer (pre-conv), a residual network layer, a sampling layer and a post-convolution layer (post-conv).
  • pre-conv pre-convolution layer
  • post-conv post-convolution layer
  • the pre-convolution layer and the post-convolution layer can be used to change the number of channels of the signal input into the first audio signal recovery model, so that the time domain feature information outputted therefrom can be input into other network layers;
  • the sampling layer can include an upsampling layer or a downsampling layer to increase or decrease the dimension of the time domain feature information and retain valid information;
  • the residual network layer can be used to extract the time domain feature information.
  • the audio signal recovery device inputs any layer in the audio signal recovery model and obtains the corresponding time domain feature information, it can also normalize the obtained time domain feature information to improve the convergence of the network trained using the back propagation algorithm.
  • Step 102 The audio signal recovery device processes the second speech signal through M first residual network layers in sequence to obtain first time domain feature information.
  • the first time domain feature information may include feature information of the time domain waveform of the second speech signal, such as the period and amplitude of the time domain waveform. Further, the time domain feature information may be transmitted between the M-layer first residual network layers in the form of the time domain waveform feature map of the second speech signal.
  • a detailed description of the second speech signal being sequentially processed by the M-layer first residual network layer can be found in the relevant descriptions in the following steps 102b1 and 102b2, which will not be repeated here.
  • the above step 102 may include the following steps 102a and 102b.
  • Step 102a The audio signal recovery device extracts second time domain feature information of the second speech signal through the pre-convolution layer in the encoding module.
  • the audio signal recovery device can change the number of channels of the second speech signal through the pre-convolution layer in the encoding module, thereby converting the second speech signal into multi-dimensional time domain feature information, namely, the second time domain feature information.
  • the audio signal recovery device can input the second speech signal into the pre-convolution layer to obtain the second time domain feature information with a channel number of 32. That is, the audio signal recovery device converts the second speech signal into 32-dimensional time domain feature information, i.e., the second time domain feature information, through the pre-convolution layer in the encoding module.
  • Step 102b The audio signal recovery device processes the second time domain feature information through M layers of first residual network layers in sequence to obtain first time domain feature information.
  • the first time domain feature information has a greater Multi-dimensional time domain feature information containing higher semantics.
  • the audio signal restoration device can perform feature extraction processing on the second time domain feature information through the M-layer first residual network layer in sequence to obtain the first time domain feature information.
  • the audio signal recovery device can first extract the second time domain feature information of the second voice signal through the pre-convolution layer in the encoding module, and then obtain the first time domain feature information by sequentially processing the second time domain feature information through the M-layer first residual network, the audio signal recovery device can finally obtain time domain feature information with more dimensions and higher semantics by extracting the time domain feature information of the voice signal layer by layer, so that the lost voice signal can be recovered through the time domain feature information containing higher semantics, thereby improving the quality of voice communication.
  • the above step 102b may include the following steps 102b1 and 102b2.
  • Step 102b1 the audio signal recovery device inputs the time domain feature information of the i-th first residual network layer, processes it through the first convolutional layer of the i-th first residual network layer, obtains the third time domain feature information, and processes it through the second convolutional layer and the third convolutional layer of the i-th first residual network layer to obtain the fourth time domain feature information.
  • the first convolution layer, the second convolution layer and the third convolution layer are all one-dimensional convolution layers, the number of input channels and the number of output channels of the first convolution layer are the same, the convolution kernel of the second convolution layer is different from the convolution kernel of the first convolution layer, and the convolution kernel of the third convolution layer is the same as the convolution kernel of the first convolution layer, i ⁇ 1,...,M ⁇ .
  • the third time domain feature information is time domain feature information of an information scale including the convolution kernel size of the first convolution layer
  • the fourth time domain feature information is time domain feature information of an information scale including the convolution kernel size of the second convolution layer. That is, the third time domain feature information and the fourth time domain feature information are time domain feature information of different scales.
  • the i-th first residual network layer can be any first residual network layer among the M first residual network layers.
  • the i-th first residual network layer may be the first first residual network layer among the M first residual network layers, or may be the last first residual network layer, or may be any first residual network layer in the middle.
  • the traditional classical convolutional neural network in order to improve the performance of the neural network, the traditional classical convolutional neural network usually extracts features better by increasing the number of layers of the neural network.
  • blindly increasing the number of layers of the neural network will not only not improve the performance of the neural network, but will also cause the gradient vanishing problem, and the neural network will inevitably degenerate, thereby hindering the training of the neural network model.
  • the neural network with fewer layers is better than the network with more layers in performance.
  • the deepening of the number of layers of the neural network model usually increases the complexity of the neural network model, which easily leads to overfitting of the neural network. Therefore, the audio signal recovery method provided in the embodiment of the present application also provides a residual network (Modified-Resnet) layer, and its structural schematic diagram is shown in Figure 8.
  • the residual network layer may be composed of two parallel branches.
  • One branch includes a convolution layer 81 and a normalization layer corresponding thereto; the other branch includes a convolution layer 82 and a normalization layer corresponding thereto, an activation function, and a convolution layer 83 and a normalization layer corresponding thereto, an activation function.
  • convolution layer 81, convolution layer 82, and convolution layer 83 are all one-dimensional convolution layers, the number of input channels of convolution layer 81 is the same as the number of output channels, the convolution kernel of convolution layer 82 is different from the convolution kernel of convolution layer 81, and the convolution kernel of convolution layer 83 is the same as the convolution kernel of convolution layer 81.
  • the audio signal recovery device can increase the information interaction between channels through multiple one-dimensional convolutional layers, and at the same time can adopt a two-branch parallel structure, by using convolutional layers with different convolution kernel sizes to convolve the input features, and fuse the obtained time domain feature information of different scales, so that each layer of the first residual network can extract richer time domain feature information.
  • the first residual network layer may include two parallel branches, one branch including the first convolutional layer, and the other branch including the second convolutional layer and the third convolutional layer. That is, after the audio signal recovery device inputs the time domain feature information of the i-th first residual network layer into the i-th first residual network layer, the third time domain feature information and the fourth time domain feature information can be obtained simultaneously through the two branches.
  • the convolution layer 81 is the first convolution layer of the first residual network layer of the i-th layer
  • the convolution layer 82 is the second convolution layer of the first residual network layer of the i-th layer
  • the convolution layer 83 is the third convolution layer of the first residual network layer of the i-th layer.
  • the audio signal recovery device can process the time domain feature information of the first residual network layer of the i-th layer through the convolution layer 81 and the normalization layer to obtain the third time domain feature information; and through the convolution layer 82 of the first residual network layer of the i-th layer and its corresponding normalization layer, the convolution layer 83 and its corresponding normalization layer to obtain the fourth time domain feature information.
  • the audio signal recovery device can first process the time domain feature information of the first residual network layer of the i-th layer input through the activation function layer, so that the time domain feature information of the first residual network layer of the i-th layer input is added with nonlinear features, so that the second convolution layer and the third convolution layer can better extract the features of the time domain feature information of the first residual network layer of the i-th layer input.
  • an Exponential Linear Unit (ELU) activation function and weight normalization may be used to ensure the stability of the first audio signal recovery model.
  • ELU Exponential Linear Unit
  • other activation functions and normalization methods may also be used to ensure the stability of the first audio signal recovery model.
  • the embodiments of the present application are not specifically limited.
  • Step 102b2 The audio signal recovery device obtains the time domain feature information output by the i-th first residual network layer based on the third time domain feature information and the fourth time domain feature information.
  • the audio signal recovery device may add time domain feature information of different scales in the third time domain feature information and the fourth time domain feature information to obtain the time domain feature information output by the i-th first residual network layer.
  • time domain feature information output by the first residual network layer of the Mth layer is the first time domain feature information.
  • the audio signal restoration device can perform the above steps 102b1 and 102b2 on it, and finally obtain the first time domain feature information.
  • different numbers of input channels, output channels, and convolution kernels may be set for the second convolutional layer and the third convolutional layer, so that the number of channels of the obtained fourth time domain feature information may be the same as the number of channels of the third time domain feature information but at a different scale, so that the third time domain feature information may be added to the fourth time domain feature information to obtain richer time domain feature information.
  • the audio signal recovery device can set the number of input channels of the convolution layer 82 to twice the number of output channels, set the convolution kernel to 3, and then set the number of output channels of the convolution layer 83 to twice the number of input channels, and set the convolution kernel to 1, so that the number of channels of the time domain feature information output after processing by the convolution layers 82 and 83 is the same as the number of channels of the time domain feature information output after processing by the convolution layer 81, but the scale is different.
  • the number of channels of the time domain feature information input to the first residual network layer of the i-th layer is 128, the number of input channels of the first convolutional layer is 128, the number of output channels is 128, and the convolution kernel is 1; the number of input channels of the second convolutional layer is 128, the number of output channels is 64, and the convolution kernel is 3; the number of input channels of the third convolutional layer is 64, the number of output channels is 128, and the convolution kernel is 1.
  • the time domain feature information input to the first residual network layer of the i-th layer is processed by the first convolutional layer to obtain the third time domain feature information with the number of channels of 128 and the convolution kernel of 1; and is processed by the second convolutional layer and the third convolutional layer of the first residual network layer of the i-th layer to obtain the fourth time domain feature information with the number of channels of 128 and the convolution kernel of 3.
  • the audio signal recovery device can obtain time domain feature information including two scales of convolution kernel 1 and convolution kernel 3.
  • convolution kernels of other sizes and numbers of input and output channels may be set to obtain time domain feature information of different scales. This embodiment of the present application is not specifically limited.
  • each layer of the first residual network can extract richer time domain feature information.
  • the above-mentioned encoding module may further include M downsampling layers, and the M downsampling layers are arranged alternately with the M first residual network layers; the number of output channels of the M downsampling layers increases from high to low according to the level.
  • step 102b may include the following steps 102b3 and 102b4.
  • Step 102b3 The audio signal recovery device processes the time domain feature information of the x-th first residual network layer through the x-th first residual network layer to obtain fifth time domain feature information.
  • Step 102b4 The audio signal recovery device outputs the fifth time domain feature information through the x-th downsampling layer connected to the x-th first residual network layer.
  • the downsampling layer may be used to increase the number of channels of the input time domain feature information, so as to reduce the dimension of the time domain feature information and retain valid information, thereby preventing the loss of time domain feature information.
  • the number of channels of the x-th first residual network layer is 32
  • the number of input channels of the x-th downsampling layer is 32
  • the audio signal recovery device can set the number of output channels of the x-th downsampling layer to 62.
  • the x-th residual network extracts the time domain feature information
  • the number of channels of the time domain feature information is increased to prevent the time domain feature from being lost. Loss of information.
  • the number of output channels of the downsampling layer in the encoding module may increase from high to low according to the level.
  • the number of output channels of the M-layer downsampling layer may be preset or may increase from high to low according to the level according to a fixed value.
  • the encoding module includes 4 downsampling layers, and the number of output channels of the 4 downsampling layers is preset.
  • the number of input channels of the first downsampling layer can be preset to 32, and the number of output channels can be preset to 64; the number of input channels of the second downsampling layer can be preset to 64, and the number of output channels can be preset to 128; the number of input channels of the third downsampling layer can be preset to 128, and the number of output channels can be preset to 256; the number of input channels of the fourth downsampling layer can be preset to 256, and the number of output channels can be preset to 512.
  • the encoding module includes 4 downsampling layers, and the number of output channels of the 4 downsampling layers increases from high to low according to a fixed value.
  • the fixed value is 64
  • the number of input channels of the first downsampling layer can be 32, and the number of output channels can be 96
  • the number of input channels of the second downsampling layer can be 96
  • the number of output channels can be 160
  • the number of input channels of the third downsampling layer can be 160
  • the number of output channels can be 224
  • the number of input channels of the fourth downsampling layer can be 224
  • the number of output channels can be 288.
  • the number of input channels of the M-layer downsampling layer gradually increases from high to low with the level of the layer because the number of output channels of the M-layer downsampling layer is increasing. In order to enable the connection between the downsampling layers and the transmission of the time domain feature signals, the number of input channels of the M-layer downsampling layer will also increase accordingly.
  • the audio signal recovery device can downsample the time domain feature information layer by layer through M downsampling layers. Specifically, the audio signal recovery device can downsample the time domain feature information layer by layer by using progressive sampling and filtering with filters with different receptive fields, thereby effectively alleviating the loss of waveform details of the time domain feature graph caused by large-scale direct downsampling, and can increase the representativeness of the time domain feature information by adjusting the number of channels of time domain feature information of different scales.
  • the level of the network layer can be determined according to the position of the network layer in the first audio restoration model. Specifically, the network layer at the top in the first audio restoration model has the highest level, the network layer at the middle position in the first audio restoration model decreases as the position moves downward, and the network layer at the bottom in the first audio restoration model has the lowest level.
  • the encoding module of the first audio signal recovery model may include M first residual network layers and M downsampling layers, and each network layer in the encoding module may be arranged alternately by the first residual network layer and the downsampling layer. That is, after the time domain feature information input to the x-th first residual network layer is extracted through the x-th first residual network layer, the number of channels of the time domain feature information can be increased immediately through the x-th downsampling layer to prevent the loss of the time domain feature information.
  • the network structure of the encoding module is shown in Figure 9.
  • the audio signal recovery device can input the second speech signal into the encoding module, and after being processed by the pre-convolution layer 901, the second speech signal first passes through a one-dimensional convolution with an input channel number of 1, an output channel number of 32, a convolution kernel size of 7, and a step size of 1, and the time domain feature information output by the pre-convolution layer 901 is processed by a normalization function to obtain the second time domain feature information of the second speech signal; then the second time domain feature information is processed by the first residual network layer 902, that is, the first residual network layer of the first layer, to obtain the time domain feature information with a channel number of 32 output by the first residual network layer.
  • the time domain feature information output by the first residual network layer 902 is input into the downsampling layer 903, i.e., the first downsampling layer, and firstly passes through a one-dimensional convolution with an input channel number of 32, an output channel number of 64, a convolution kernel size of 4, and a step size of 2, and the time domain feature information output by the one-dimensional convolution is processed by a normalization function to obtain the time domain feature information with a channel number of 64 output by the downsampling layer 903; the time domain feature information output by the downsampling layer 903 is then processed by the first residual network layer 904, i.e., the second first residual network layer, and the channel number of the first residual network layer 904 output is 64; then the time domain feature information with the number of channels of 64 is input to the downsampling layer 905, and a one-dimensional convolution with the number of input channels of 64, the number of output channels of 128, the convolution kernel size of 8, and the step size of 4 is performed, and the time domain feature
  • the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain time domain feature information with a channel number of 256; the time domain feature information with a channel number of 256 is then input to the first residual network layer 908, i.e., the fourth first residual network layer, to obtain higher semantic time domain feature information with a channel number of 256; then the first residual network layer 908 outputs
  • the time domain feature information is input into the downsampling layer 909, i.e., the fourth downsampling layer, and a one-dimensional convolution with an input channel number of 256, an output channel number of 512, a convolution kernel size of 16, and a step size of 8 is passed through, and the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain the time domain feature information with a channel number of 512; finally, the time domain feature information with a channel number of 512 is input into the post-convolution layer 910, first through an activation function, and then through a
  • the audio signal recovery device can extract and downsample the time domain feature information of the speech signal layer by layer through M downsampling layers, it is possible to increase the representativeness of the extracted time domain feature information while avoiding the loss of time domain feature information.
  • Step 103 The audio signal recovery device processes the first time domain feature information through M layers of second residual network layers in sequence to obtain a target speech signal.
  • the process in which the audio signal recovery device sequentially processes the first time domain feature information through the M-layer second residual network layer is actually a process of recovering the first speech signal based on the first time domain feature information to obtain the target speech signal.
  • the audio signal restoration device may output the target voice signal as the restored first voice signal.
  • the audio signal recovery device caches the speech signals of the 1st, 2nd and 3rd frames, that is, the second speech signal, when the speech signal of the 4th frame is lost, that is, when the first speech signal is lost
  • the audio signal recovery device can input the speech signals of the 1st, 2nd and 3rd frames into the first audio signal recovery model, and map the speech signals of the 1st, 2nd and 3rd frames into 1 frame of speech signal through the preprocessing module in the first audio signal recovery model, and extract the corresponding time domain feature information from the 1 frame of speech signal through the encoding module; and generate 1 frame of speech signal, that is, the target speech signal, through the decoding module based on the extracted time domain feature information, and the generated 1 frame of speech signal is the recovered 4th frame of speech signal.
  • the audio signal recovery device may update the buffered voice signal after completing the recovery of the first voice signal.
  • the audio signal recovery device can cache the three frames of voice signals before the first voice signal, that is, cache the voice signals of the 1st, 2nd, and 3rd frames. After the audio signal recovery device completes the recovery of the 4th frame of voice signal, the 4th frame of voice signal can be cached, so that the voice signals cached in the audio signal recovery device are updated to the 2nd, 3rd, and 4th frames of voice signals.
  • the cache when the audio signal recovery device updates the cached voice signal, the cache may be updated by overwriting the previous voice signal with each voice frame, or the cache may be updated by overwriting the previous voice signal with the last voice frame.
  • the embodiments of the present application are not specifically limited.
  • the audio signal recovery device when the first voice signal is not lost, can decode it normally and update the cache.
  • the audio signal recovery device buffers the first frame of voice signal X1 , the second frame of voice signal X2 , the third frame of voice signal X3 , the fourth frame of voice signal X4 , the fifth frame of voice signal X5 , the sixth frame of voice signal X6 , the seventh frame of voice signal X7 , the eighth frame of voice signal X8 , the ninth frame of voice signal X9 , and the tenth frame of voice signal X10 , and is receiving the eleventh frame of voice signal X11 .
  • FIG10 it is a flow chart of recovering the lost voice signal when the audio signal recovery device receives the voice signal. The audio signal recovery device first determines whether the eleventh frame of voice signal X11 is lost.
  • the audio signal recovery device can decode it normally, output the 11th frame voice signal X11 and update the cache; in the case that the 11th frame voice signal X11 is lost, the audio signal recovery device can input the cached 1st frame voice signal X1 , 2nd frame voice signal X2 , 3rd frame voice signal X3 , 4th frame voice signal X4 , 5th frame voice signal X5 , 6th frame voice signal X6 , 7th frame voice signal X7 , 8th frame voice signal X8 , 9th frame voice signal X9 and 10th frame voice signal X10 into the first audio signal recovery model to generate a target voice signal, output the target voice signal as the recovered 11th frame voice signal X11 and update the cache.
  • the updated buffer includes the 2nd frame voice signal X2 , the 3rd frame voice signal X3 , the 4th frame voice signal X4 , the 5th frame voice signal X5 , the 6th frame voice signal X6 , the 7th frame voice signal X7 , the 8th frame voice signal X8 , the 9th frame voice signal X9 , the 10th frame voice signal X10 , and the 11th frame voice signal X11 .
  • the above step 103 may include the following steps 103a and 103b.
  • Step 103a The audio signal recovery device processes the first time domain feature information through M layers of second residual network layers in sequence to obtain sixth time domain feature information.
  • the first time domain feature information is time domain feature information with more dimensions and higher semantics than the sixth time domain feature information.
  • the first time domain feature information and the sixth time domain feature information are time domain feature information of different scales.
  • the above step 103a may include the following steps 103a1 and 103a2.
  • Step 103a1 the audio signal recovery device inputs the time domain feature information of the j-th second residual network layer, processes it through the fourth convolutional layer of the j-th second residual network layer, obtains the seventh time domain feature information, and processes it through the fifth convolutional layer and the sixth convolutional layer of the j-th second residual network layer to obtain the eighth time domain feature information.
  • the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all one-dimensional convolution layers, the number of input channels and the number of output channels of the fourth convolution layer are the same, the convolution kernel of the fifth convolution layer is different from the convolution kernel of the fourth convolution layer, and the convolution kernel of the sixth convolution layer is the same as the convolution kernel of the fourth convolution layer, j ⁇ 1,...,M ⁇ .
  • Step 103a2 The audio signal recovery device obtains the time domain feature information output by the j-th second residual network layer based on the seventh time domain feature information and the eighth time domain feature information.
  • each layer of the first residual network can extract richer time domain feature information.
  • the above-mentioned decoding module may also include M upsampling layers, and the M upsampling layers are arranged alternately with the M second residual network layers; the number of output channels of the M upsampling layers decreases from high to low according to the level.
  • step 103a may further include the following step 103a3.
  • Step 103a3 The audio signal recovery device processes the time domain feature information output by the yth upsampling layer through the yth second residual network layer connected to the yth upsampling layer to obtain the time domain feature information output by the yth second residual network layer.
  • the upsampling layer can be used to reduce the number of channels of the input time domain feature information, so as to reconstruct the time domain feature information to its original size by upsampling the first time domain feature information layer by layer, increase the dimension of the time domain feature information, reduce the scale of the time domain feature information, and realize the recovery of the speech signal.
  • the audio signal recovery device can set the number of output channels of the y-th layer downsampling layer to 62.
  • the number of channels of the time domain feature information is reduced, the dimension of the time domain feature information is reduced, and the recovery of the speech signal is achieved.
  • the input time domain feature information may be reduced in order from high to low by the number of output channels of the upsampling layer in the decoding module.
  • the number of output channels of the M-layer upsampling layer may be pre-set or may be reduced in order from high to low by a fixed value.
  • the decoding module includes 4 upsampling layers, and the number of output channels of the 4 upsampling layers is preset.
  • the number of input channels of the first upsampling layer can be preset to 512, and the number of output channels can be preset to 218;
  • the number of input channels of the second upsampling layer can be preset to 218, and the number of output channels can be preset to 128;
  • the number of input channels of the third upsampling layer can be preset to 128, and the number of output channels can be preset to 64;
  • the number of input channels of the fourth upsampling layer can be preset to 64, and the number of output channels can be preset to 32.
  • the encoding module includes 4 upsampling layers, and the number of output channels of the 4 upsampling layers is reduced from high to low according to a fixed value.
  • the fixed value is 64
  • the number of input channels of the first upsampling layer can be 288, and the number of output channels can be 224;
  • the number of input channels of the second upsampling layer can be 224, and the number of output channels can be 160;
  • the number of input channels of the third upsampling layer can be 160, and the number of output channels can be 96;
  • the number of input channels of the fourth upsampling layer can be 96, and the number of output channels can be 32.
  • the number of input channels of the M-layer upsampling layer gradually decreases from high to low with the level of the layer because the number of output channels of the M-layer upsampling layer is decreasing. In order for the time domain feature signal to be transmitted between the two upsampling layers, the number of input channels of the M-layer upsampling layer will also decrease accordingly.
  • the audio signal recovery device can upsample the time domain feature information layer by layer through M upsampling layers, and can effectively alleviate the loss of waveform details of the time domain feature map caused by large-scale direct upsampling by using progressive sampling and filtering with filters with different receptive fields, and can also effectively alleviate the loss of waveform details of the time domain feature map caused by large-scale direct upsampling by using different upsampling layers.
  • the number of channels of the time domain feature information of the scale is adjusted to realize the recovery of the speech signal.
  • the decoding module of the first audio signal recovery model may include M upsampling layers and M second residual network layers, and each network layer in the decoding module may be alternately arranged by upsampling layers and second residual network layers. That is, after the time domain feature information input to the i-th upsampling layer is reduced in the number of channels of the time domain feature information through the i-th upsampling layer, the time domain feature information may be immediately extracted through the i-th second residual network layer to restore the speech signal.
  • the audio signal recovery device can first input the first time domain feature information into the decoding module, and after processing by the pre-convolution layer 1101, the first time domain feature information is first passed through a one-dimensional convolution with an input channel number of 128, an output channel number of 512, a convolution kernel size of 7, and a step size of 1, and the time domain feature information output by the pre-convolution layer 1101 is processed by a normalization function to obtain time domain feature information with a channel number of 32; then the time domain feature information with a channel number of 32 is input into the upsampling layer 1102, that is, the first upsampling layer, through a one-dimensional convolution with an input channel number of 512, an output channel number of 256, a convolution kernel size of 16, and a step size of 8, and the time domain feature information output by the
  • the time domain feature information with the number of channels being 256 is input into the second residual network layer 1103, i.e., the first second residual network layer, to obtain the time domain feature information with the number of channels being 256; the time domain feature information with the number of channels being 256 is then input into the upsampling layer 1104, i.e., the second upsampling layer, through a one-dimensional convolution with the number of input channels being 256, the number of output channels being 128, the convolution kernel size being 10, and the step size being 5, and the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain the time domain feature information with the number of channels being 128; the time domain feature information with the number of channels being 128 is then input into the second residual network layer 1105, i.e., the second second residual network layer, to obtain the time domain feature information with the number of channels being 128.
  • the second residual network layer 1105 i.e., the second second residual network layer
  • the time domain feature information with the number of channels being 128 is then input into the upsampling layer 1106, i.e., the third upsampling layer, and a one-dimensional convolution is performed with the number of input channels being 128, the number of output channels being 64, the convolution kernel size being 8, and the step size being 4, and the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain the time domain feature information with the number of channels being 64; the time domain feature information with the number of channels being 64 is then input into the second residual network layer 1107, i.e., the third second residual network layer, to obtain the time domain feature information with the number of channels being 64; the time domain feature information with the number of channels being 64 is then input into the upsampling layer 1108, i.e., the fourth upsampling layer, and a one-dimensional convolution is performed with the number of input channels being 64, the number of output channels being 64, and the convolution kernel size being 8.
  • a one-dimensional convolution with a channel number of 32, a convolution kernel size of 4, and a step size of 2 is performed, and the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain time domain feature information with a channel number of 32; the time domain feature information with a channel number of 32 is then input into the second residual network layer 1109, that is, the fourth second residual network layer, to obtain time domain feature information with a channel number of 32; finally, the time domain feature information with a channel number of 32 is input into the post-convolution layer 1110, through a one-dimensional convolution with an input channel number of 32, an output channel number of 1, a convolution kernel size of 7, and a step size of 1, and the time domain feature information output by the one-dimensional convolution is passed through a normalization function to obtain a target speech signal with a channel number of 1, and the target speech signal is output from the decoding module.
  • the audio signal recovery device can upsample and extract features of the time domain feature information of the speech signal layer by layer through M upsampling layers, the lost speech signal can be restored according to the time domain feature information extracted by the encoding module.
  • Step 103b The audio signal recovery device inputs the sixth time domain feature information into the post-convolution layer of the decoding module for processing to obtain a target speech signal.
  • the audio signal recovery device can change the number of channels of the sixth time domain feature information through the post-convolution layer in the decoding module, thereby synthesizing the sixth time domain feature information to obtain the target speech signal.
  • the audio signal recovery device can input the sixth time domain feature information into the post-convolution layer, combine the 32 channels of the sixth time domain feature information into 1 channel, and thus obtain the target speech signal. It can be understood that the audio signal recovery device can convert the time domain feature information into speech information through the post-convolution layer in the decoding module.
  • the audio signal recovery device can first process the first time domain feature information through the M-layer second residual network layer in sequence to obtain the sixth time domain feature information, and then obtain the target speech signal through the post-convolution layer in the decoding module. Therefore, the audio signal recovery device can restore the time domain feature information of the lost speech signal layer by layer through the M-layer second residual network, and convert the time domain feature information into a speech signal through the post-convolution layer in the decoding module, and finally recover the lost speech signal.
  • the time domain feature information of the cached voice signal before or after the lost voice signal can be extracted through the residual network layer of the encoding module in the audio signal recovery model, and the time domain feature information of the voice signal before or after the lost voice signal is recovered through the residual network layer of the encoding module in the audio signal recovery model.
  • the residual network layer of the decoding module synthesizes the time domain feature information extracted by the encoding module to restore the lost voice signal. In this way, when the voice signal is lost, the lost voice signal can be restored by inputting the cached voice signal into the audio signal recovery model.
  • the lost voice signal can be restored by the audio signal recovery model, thereby avoiding the user's perception of the loss of voice data packets, reducing the negative impact of the loss of voice data packets, and effectively improving the quality of voice calls.
  • the audio signal recovery method provided in the embodiment of the present application may further include the following step 104.
  • the above step 102 may include the following step 102c.
  • Step 104 After inputting the multiple frames of speech signals into the first audio signal recovery model, the audio signal recovery device maps the multiple frames of speech signals into one frame of speech signal.
  • the multi-frame voice signal when the second voice signal is a multi-frame voice signal, may include at least two frames of voice signals before the first voice signal, may also include at least two frames of voice signals after the first voice signal, may also include at least one frame of voice signal before the first voice signal and at least one frame of voice signal after the first voice signal.
  • the embodiments of the present application are not specifically limited.
  • the first audio signal restoration model may further include a preprocessing module, and the audio signal restoration device may map multiple frames of speech signals into one frame of speech signal through the preprocessing module.
  • the audio signal restoration device may extract the time domain feature information of the multiple frames of speech signals and map it to one frame of speech signals.
  • Step 102c The audio signal recovery device processes a frame of speech signal through M first residual network layers in sequence to obtain first time domain feature information.
  • the frequency signal recovery device can obtain the first time domain feature information by mapping the multi-frame voice signal into one-frame voice signal when the second voice signal is a multi-frame voice signal, and sequentially processing the one-frame voice signal through the M-layer first residual network layer, only the useful part of the multi-frame voice signal can be processed, avoiding the processing of the useless part of the multi-frame voice signal, and saving the resources of the frequency signal recovery model.
  • the audio signal restoration device provided by the embodiment of the present application may further include the following steps 105 and 106.
  • Step 105 The audio signal recovery device obtains at least one set of training speech samples.
  • Each group of training speech samples includes speech packets marked as correctly received and speech packets marked as lost.
  • the voice packets marked as lost in each group of training voice samples can be obtained by simulating voice packet loss based on real voice. It is understandable that in actual implementation, a complete voice packet can include the correctly received voice packets and the voice packets marked as lost.
  • the speech packet including the lost speech signal may include the 1st to 3rd frames of the lost 4th frame of speech signal and the 5th frame of speech signal.
  • Step 106 The audio signal recovery device trains the second audio signal recovery model based on at least one set of training speech samples to obtain the first audio signal recovery model.
  • the loss function in the second audio signal restoration model includes at least two of the following: an adversarial loss parameter, a multi-resolution time-frequency domain signal loss parameter, and a relative feature matching loss parameter.
  • the second audio signal restoration model is an original model without audio signal restoration function, that is, the second audio signal restoration model is an untrained first audio signal restoration model.
  • the second audio signal recovery model may be a deep learning network model based on a generative adversarial network (GAN).
  • GAN generative adversarial network
  • a GAN network is usually composed of two networks that are trained simultaneously and compete with each other, namely a generator and a discriminator.
  • the input of the discriminator consists of two parts, namely, the real sample data and the data generated by the generator, and its output is usually a probability value, which indicates the distribution probability that the discriminator determines that the input is real data. If the input comes from real data, the output is 1, otherwise the output is 0.
  • the generator can receive a random variable to generate fake sample data, and the discriminator determines the distribution probability of the fake sample data as real data; and the output of the discriminator will also be fed back to the generator to guide the training of the generator.
  • the discriminator cannot distinguish whether the input data comes from real sample data or data generated by the generator, that is, the output probability of the discriminator is 0.5 each time, at which time the data generated by the generator is infinitely close to the real sample data, and the model is optimal.
  • generators and discriminators are usually implemented using deep neural networks.
  • the GAN network has basically no restrictions on the random variables input to the generator.
  • the random variables can be random noise or variables that conform to a certain distribution.
  • the generator can gradually learn any probability distribution.
  • the trained generator can generate data that is infinitely close to the real data but not exactly the same as the real data, that is, the generator actually learns an approximate distribution of the training data.
  • the main purpose of the discriminator is to determine whether the input data is real sample data and provide feedback to guide the generator for training.
  • the generator and the discriminator will each update their own parameters to minimize the loss, and through continuous iterative optimization, they will eventually reach a Nash equilibrium state. It can be understood that the above-mentioned second audio signal recovery model is the generator in the GAN network.
  • the audio signal recovery device can input the voice package containing the lost voice signal in the acquired training voice sample into the second audio signal recovery model to generate a voice signal; the audio signal recovery device can judge the similarity between the generated voice signal and the real voice through the discriminator, judge whether the second audio signal recovery model successfully recovers the lost voice signal, and adjust the generator model parameters according to the judgment result of the discriminator to improve the similarity between the generated voice signal and the real voice.
  • the audio signal recovery device can cache the voice signal that is not lost in the voice packet.
  • the second audio signal recovery model can extract the time domain feature information of the cached voice signal through the encoding module, and generate a voice signal according to the extracted time domain feature information through the decoding module.
  • the audio signal recovery device can judge the similarity between the generated voice signal and the real voice through the discriminator to judge whether the second audio signal recovery model successfully recovers the lost voice signal.
  • the audio signal recovery device can judge that the second audio signal recovery model successfully recovers the lost voice signal when the discriminator judges that the similarity between the generated voice signal and the real voice is greater than or equal to the similarity threshold; when the discriminator judges that the similarity between the generated voice signal and the real voice is less than the similarity threshold, it is judged that the second audio signal recovery model fails to recover the lost voice signal.
  • the audio signal recovery device can adjust the generator model parameters according to the judgment result of the discriminator to improve the similarity between the regenerated voice information and the real voice.
  • FIG13 shows a training flowchart of a packet loss concealment method based on a GAN network, that is, a training flowchart of a second audio signal recovery model.
  • the generator is composed of a preprocessing module, an encoding module, and a decoding module, wherein the encoding module is used to extract the time domain feature information of the speech signal, and the decoding module synthesizes the speech signal through the time domain feature information extracted by the encoding module.
  • the generator After the generator generates a speech signal, it can be input into the discriminator together with the real speech signal.
  • the discriminator makes a judgment on the input speech signal and feeds the judgment result back to the generator.
  • the generator updates its model parameters according to the feedback result, and gradually makes the generated speech signal close to the real speech signal.
  • the discriminator will also update the network parameters of the discriminator according to the discriminator loss function. While the generator improves the quality of speech signal generation, the discriminator can also improve its discrimination ability at the same time.
  • the audio signal recovery device can use a multi-scale short-time Fourier transform (MS-STFT) discriminator to distinguish between the speech signal generated by the generator and the real speech signal.
  • MS-STFT multi-scale short-time Fourier transform
  • a discriminator such as a Multi-Period Discriminator and a Multi-Scale Discriminator can also be used to distinguish between the speech signal generated by the generator and the real speech signal.
  • the embodiments of the present application are not specifically limited.
  • the MS-STFT discriminator includes 5 short-time Fourier transform (STFT) discriminators.
  • STFT short-time Fourier transform
  • the 5 STFT discriminators When discriminating the input speech signal, the 5 STFT discriminators perform FFT transform on the input speech signal respectively.
  • the frame length of the FFT transform performed on the input speech signal is: [128, 256, 512, 1024, 2048], and the corresponding frame shift is: [32, 64, 128, 256, 512], and the spectrum data corresponding to the input speech signal is obtained.
  • the spectrum data can then be input into the discriminator for processing.
  • Discriminators of different scales can learn features from different levels of speech structure, thereby prompting the generator to improve the quality of synthesized speech.
  • the STFT discriminator usually selects the complex spectrum as input, which helps the discriminator learn finer details in speech.
  • a dilated convolution is added to the STFT discriminator to improve the efficiency of the convolution layer in extracting speech context information.
  • the dilated convolution allows the convolution to increase the receptive field size to capture more context information, and different receptive fields can complement each other in sequence modeling, thereby improving the discriminator performance.
  • the discriminator loss function can be calculated by formula (9):
  • LD is the discriminator loss function
  • K is the number of discriminators
  • the speech signal generated by the generator is the discriminant output of the kth discriminator when the generated speech signal is input
  • x is the real speech
  • D k (x) is the discriminant output of the kth discriminator when the real speech is input
  • max( ⁇ ) is the maximum value operation.
  • the network parameters of the discriminator can be updated by minimizing the discriminator loss function LD .
  • the loss function in the second audio signal recovery model may be a loss function composed of multiple constraint items.
  • the quality of the generated voice signal may be improved by the loss function composed of multiple constraint items, thereby more accurately recovering the lost voice signal and improving the quality of the voice call.
  • the loss function composed of the above-mentioned multiple constraints may include at least two of the following: adversarial loss, multi-resolution time-frequency domain signal loss, and relative feature matching loss.
  • the above-mentioned anti-loss parameter can be calculated by the following formula (10):
  • Formula (11) calculates:
  • S(n, k) represents the spectrum amplitude of the kth frequency point in the nth frame after the real speech signal is transformed by Fast Fourier Transform (FFT), and its FFT transform length is K, where K ⁇ 2 6 , 2 7 , 2 8 , 2 9 , 2 10 , 2 11 ⁇ , and its frame shift is K/4.
  • FFT Fast Fourier Transform
  • K ⁇ 2 6 , 2 7 , 2 8 , 2 9 , 2 10 , 2 11 ⁇ represents the spectrum amplitude of the kth frequency point in the nth frame after the FFT transformation of the generated synthetic speech signal
  • ⁇ tf is the weighting coefficient
  • It is used to balance the proportion of 1-norm distortion and 2-norm distortion of the real speech spectrum and the synthesized speech spectrum.
  • ⁇ 1 represents the calculation of the 1-norm of the input variable
  • ⁇ 2 represents the calculation of the 2-norm of the input variable.
  • the above relative feature matching loss parameter can be calculated by the following formula (12):
  • the resulting speech signal is the output of the lth layer of the kth discriminator.
  • mean( ⁇ ) represents the mean operation.
  • the loss function in the second audio signal restoration model can be calculated by the following formula (13):
  • the weighted coefficients of the loss, multi-resolution time-frequency domain signal loss, and relative feature matching loss are all constants.
  • the audio signal restoration device can update the model parameters of the second audio signal restoration model by minimizing the second audio signal restoration model loss function LG to improve the quality of the generated speech signal.
  • the second audio signal recovery model is trained through a multi-constraint loss function including adversarial loss, multi-resolution time-frequency domain signal loss, and relative feature matching loss to obtain the first audio signal recovery model. Therefore, the trained first audio signal recovery model can generate a speech signal that is closer to the real speech, thereby This makes the trained first audio signal restoration model have better audio signal restoration capability.
  • the trained first audio signal recovery model can be integrated into the AMR-WB decoder.
  • the AMR-WB decoder can restore the lost voice signal with high quality through the audio signal recovery method provided in the embodiment of the present application.
  • a 16kHz sampled wideband speech signal is encoded by an AMR-WB encoder, and the encoding rate of the AMR-WB encoder is set to 12.65kbit/s.
  • an IP data packet stores only one frame of encoded speech signal, and the decoding end has correctly received the 1st, 2nd, ..., 100th frames, and the 101st frame is lost. At this time, the decoder can recover the 101st frame of speech signal.
  • the AMR-WB codec algorithm has a frame length of 20ms, and a frame of signal contains 320 sample points.
  • the cache size is 5 frames, that is, the decoded speech signals of the past 5 frames are stored, including: 96th frame, 97th frame, 98th frame, 99th frame, 100th frame.
  • the audio signal restoration method may include the following steps 201 to 203 .
  • Step 201 The audio signal recovery device maps an input speech signal of 5 frames in length into a speech signal of 1 frame of fixed length through a preprocessing module in a first audio signal recovery model.
  • the preprocessing module in the first audio signal recovery model may be composed of a fully connected layer network and a "Tanh()" activation function, and the input signal may pass through the fully connected layer network layer and the nonlinear activation function layer in sequence, and then obtain the output signal, which is defined as:
  • the input signal of 5 frames can be mapped to a signal of 1 frame of fixed length.
  • its output tensor be x1, and its dimension is torch.size([1,1,320]).
  • Step 202 The audio signal recovery device inputs a speech signal of a fixed length of one frame into a first audio signal recovery model, and generates a target speech signal through an encoding module in the first audio signal recovery model and a decoding module in the first audio signal recovery model.
  • the audio signal recovery device can process the encoding module output tensor x1 (whose dimension is torch.size([1,1,320])) through the encoding module, and make its output feature z, whose dimension is torch.size([1,128,1]). Then, the feature z output by the encoding module is input to the decoding module for processing, and its output tensor is x2, whose dimension is torch.size([1,1,320]). The 320 sample points output by the decoding module are the restored 101st frame signal.
  • Step 203 The audio signal recovery device updates the cache.
  • the audio signal recovery device may remove the 320 sample points of the 96th frame stored from the cache, and then store the newly generated 320 sample points of the 101st frame into the cache.
  • the 5 frames of signals stored in the cache buffer are 97 frames, 98 frames, 99 frames, 100 frames, and 101 frames.
  • the audio signal recovery device may continue to start decoding the 102th frame, and cycle in sequence until all speech signals are decoded.
  • the preprocessing module in the audio signal recovery model can be used to map at least one frame of the cached voice signal to a voice signal of fixed length, and then the encoding module and the decoding module in the audio signal recovery model can be used to recover the lost voice signal.
  • the lost voice signal can be recovered by the audio signal recovery model, thereby avoiding the user's perception of the loss of the voice data packet and reducing the negative impact of the loss of the voice data packet.
  • the recovered voice signal can be cached so that the lost voice signal can be recovered, thereby improving the quality of the voice call.
  • the audio signal recovery method provided in the embodiment of the present application may be executed by an audio signal recovery device.
  • the audio signal recovery device executing the audio signal recovery method is taken as an example to illustrate the audio signal recovery device provided in the embodiment of the present application.
  • Fig. 16 shows a possible structural diagram of an audio signal recovery device involved in an embodiment of the present application.
  • the audio signal recovery device 160 may include: an input module 161 and a processing module 162 .
  • the input module 161 is used to replace the cached second voice signal when the first voice signal is lost.
  • the signal is input into the first audio signal recovery model, the first audio signal recovery model includes an encoding module and a decoding module, the encoding module includes M layers of first residual network layers, and the decoding module includes M layers of second residual network layers;
  • a processing module 162 is used to process the second speech signal through the M layers of first residual network layers in sequence to obtain first time domain feature information; and is used to process the first time domain feature information through the M layers of second residual network layers in sequence to obtain a target speech signal;
  • the second speech signal includes at least one of the following: X frames of speech signal before the first speech signal, and Y frames of speech signal after the first speech signal, and M, X, and Y are positive integers.
  • processing module 162 is specifically configured to:
  • the second time domain feature information is processed sequentially by M first residual network layers to obtain the first time domain feature information.
  • processing module 162 is specifically configured to:
  • the time domain feature information input into the first residual network layer of the i-th layer is processed by the first convolution layer of the i-th layer to obtain the third time domain feature information, and is processed by the second convolution layer and the third convolution layer of the i-th layer to obtain the fourth time domain feature information, i ⁇ 1,...,M ⁇ ;
  • the first convolution layer, the second convolution layer and the third convolution layer are all one-dimensional convolution layers;
  • the number of input channels and output channels of the first convolutional layer are the same;
  • the convolution kernel of the second convolution layer is different from the convolution kernel of the first convolution layer, and the convolution kernel of the third convolution layer is the same as the convolution kernel of the first convolution layer.
  • the encoding module further includes M downsampling layers, and the M downsampling layers are alternately arranged with the M first residual network layers; the number of output channels of the M downsampling layers increases from high to low according to the level;
  • the processing module 162 is specifically used for:
  • the time domain feature information input into the first residual network layer of the xth layer is processed by the first residual network layer of the xth layer to obtain the fifth time domain feature information;
  • the fifth time domain feature information is output through the xth downsampling layer connected to the xth first residual network layer, x ⁇ 1,...,M ⁇ .
  • processing module 162 is specifically configured to:
  • the first time domain feature information is processed sequentially by M layers of second residual network layers to obtain sixth time domain feature information;
  • the sixth time domain feature information is input into the post-convolution layer of the decoding module for processing to obtain the target speech signal.
  • processing module 162 is specifically configured to:
  • the time domain feature information input into the j-th second residual network layer is processed by the fourth convolutional layer of the j-th second residual network layer to obtain the seventh time domain feature information, and is processed by the fifth convolutional layer and the sixth convolutional layer of the j-th second residual network layer to obtain the eighth time domain feature information, j ⁇ 1,...,M ⁇ ;
  • the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all one-dimensional convolution layers;
  • the number of input channels of the fourth convolutional layer is the same as the number of input channels
  • the convolution kernel of the fifth convolution layer is different from the convolution kernel of the fourth convolution layer, and the convolution kernel of the sixth convolution layer is the same as the convolution kernel of the fourth convolution layer.
  • the decoding module further includes M upsampling layers, and the M upsampling layers and the M second residual network layers are alternately arranged; the number of output channels of the M upsampling layers decreases from high to low according to the level;
  • the processing module 162 is specifically used for:
  • the time domain feature information output by the yth upsampling layer is processed by the yth second residual network layer connected to the yth upsampling layer to obtain the time domain feature information output by the yth second residual network layer, y ⁇ 1,...,M-1 ⁇ .
  • the processing module 162 is further configured to, when the second speech signal is a multi-frame speech signal, map the multi-frame speech signal into a single-frame speech signal after inputting the multi-frame speech signal into the first audio signal recovery model before sequentially processing the second speech signal through the M-layer first residual network layer to obtain the first time domain feature information;
  • the processing module 162 is specifically used to process a frame of speech signal in sequence through the M-layer first residual network layer.
  • the first time domain feature information is obtained.
  • the above device further includes: an acquisition module;
  • An acquisition module used for acquiring at least one group of training voice samples before the input module 161 inputs the cached second voice signal into the first audio signal recovery model when the first voice signal is lost, each group of training voice samples includes voice packets marked as correctly received and voice packets marked as lost;
  • the processing module 162 is further configured to train the second audio signal recovery model based on at least one set of training speech samples obtained by the acquisition module to obtain the first audio signal recovery model;
  • the loss function in the second audio signal restoration model includes at least two of the following: an adversarial loss parameter, a multi-resolution time-frequency domain signal loss parameter, and a relative feature matching loss parameter.
  • the embodiment of the present application provides an audio signal recovery device.
  • the time domain feature information of the cached voice signal before or after the lost voice signal can be extracted through the residual network layer of the encoding module in the audio signal recovery model, and the time domain feature information extracted by the encoding module can be synthesized through the residual network layer of the decoding module in the audio signal recovery model to recover the lost voice signal.
  • the lost voice signal can be recovered by inputting the cached voice signal into the audio signal recovery model when the voice signal is lost, the lost voice signal can be recovered through the audio signal recovery model during the transmission of the voice signal, thereby avoiding the user's perception of the loss of the voice data packet, reducing the negative impact of the loss of the voice data packet, and thus effectively improving the quality of the voice call.
  • the audio signal recovery device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip.
  • the electronic device can be a terminal or other devices other than a terminal.
  • the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc.
  • NAS Network Attached Storage
  • PC personal computer
  • TV television
  • teller machine a self-service machine
  • the audio signal recovery device in the embodiment of the present application may be a device having an operating system.
  • the operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
  • the audio signal recovery device provided in the embodiment of the present application can implement the various processes implemented in the above-mentioned audio signal recovery method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.
  • an embodiment of the present application also provides an electronic device 1700, including a processor 1701 and a memory 1702, and the memory 1702 stores a program or instruction that can be executed on the processor 1701.
  • the program or instruction is executed by the processor 1701, the various steps of the above-mentioned audio signal recovery method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
  • the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
  • FIG. 18 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.
  • the electronic device 1800 includes but is not limited to: a radio frequency unit 1801, a network module 1802, an audio output unit 1803, an input unit 1804, a sensor 1805, a display unit 1806, a user input unit 1807, an interface unit 1808, a memory 1809, and a processor 1810 and other components.
  • the electronic device 1800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1810 through a power management system, so that the power management system can manage charging, discharging, and power consumption.
  • a power source such as a battery
  • the electronic device structure shown in FIG18 does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently, which will not be described in detail here.
  • the processor 1810 is used to input the cached second voice signal into the first audio signal recovery model when the first voice signal is lost, the first audio signal recovery model includes an encoding module and a decoding module, the encoding module includes M layers of first residual network layers, and the decoding module includes M layers of second residual network layers; and is used to process the second voice signal through the M layers of first residual network layers in sequence to obtain first time domain feature information; and is used to process the first time domain feature information through the M layers of second residual network layers in sequence to obtain a target voice signal; wherein the second voice signal includes at least one of the following: X frames of voice signal before the first voice signal, Y frames of voice signal after the first voice signal, and M, X, and Y are positive integers.
  • the processor 1810 is specifically configured to:
  • the second time domain feature information is processed sequentially by M first residual network layers to obtain the first time domain feature information.
  • the processor 1810 is specifically configured to:
  • the time domain feature information input into the first residual network layer of the i-th layer is processed by the first convolution layer of the i-th layer to obtain the third time domain feature information, and is processed by the second convolution layer and the third convolution layer of the i-th layer to obtain the fourth time domain feature information, i ⁇ 1,...,M ⁇ ;
  • the first convolution layer, the second convolution layer and the third convolution layer are all one-dimensional convolution layers;
  • the number of input channels and output channels of the first convolutional layer are the same;
  • the convolution kernel of the second convolution layer is different from the convolution kernel of the first convolution layer, and the convolution kernel of the third convolution layer is the same as the convolution kernel of the first convolution layer.
  • the encoding module further includes M downsampling layers, and the M downsampling layers are alternately arranged with the M first residual network layers; the number of output channels of the M downsampling layers increases from high to low according to the level;
  • the processor 1810 is specifically configured to:
  • the time domain feature information input into the first residual network layer of the xth layer is processed by the first residual network layer of the xth layer to obtain the fifth time domain feature information;
  • the fifth time domain feature information is output through the xth downsampling layer connected to the xth first residual network layer, x ⁇ 1,...,M ⁇ .
  • the processor 1810 is specifically configured to:
  • the first time domain feature information is processed sequentially by M layers of second residual network layers to obtain sixth time domain feature information;
  • the sixth time domain feature information is input into the post-convolution layer of the decoding module for processing to obtain the target speech signal.
  • the processor 1810 is specifically configured to:
  • the time domain feature information input into the j-th second residual network layer is processed by the fourth convolutional layer of the j-th second residual network layer to obtain the seventh time domain feature information, and is processed by the fifth convolutional layer and the sixth convolutional layer of the j-th second residual network layer to obtain the eighth time domain feature information, j ⁇ 1,...,M ⁇ ;
  • the fourth convolution layer, the fifth convolution layer and the sixth convolution layer are all one-dimensional convolution layers;
  • the number of input channels of the fourth convolutional layer is the same as the number of input channels
  • the convolution kernel of the fifth convolution layer is different from the convolution kernel of the fourth convolution layer, and the convolution kernel of the sixth convolution layer is the same as the convolution kernel of the fourth convolution layer.
  • the decoding module further includes M upsampling layers, and the M upsampling layers and the M second residual network layers are alternately arranged; the number of output channels of the M upsampling layers decreases from high to low according to the level;
  • the processor 1810 is specifically configured to:
  • the time domain feature information output by the yth upsampling layer is processed by the yth second residual network layer connected to the yth upsampling layer to obtain the time domain feature information output by the yth second residual network layer, y ⁇ 1,...,M-1 ⁇ .
  • the processor 1810 is further configured to, when the second speech signal is a multi-frame speech signal, before sequentially processing the second speech signal through the M-layer first residual network layer to obtain the first time domain feature information, input the multi-frame speech signal into the first audio signal recovery model, and then map the multi-frame speech signal into one frame speech signal;
  • the processor 1810 is specifically used to process a frame of speech signal through M first residual network layers in sequence to obtain first time domain feature information.
  • the processor 1810 is further configured to, when the first voice signal is lost, before inputting the cached second voice signal into the first audio signal recovery model, obtain at least one group of training voice samples, each group of training voice samples including voice packets marked as correctly received and voice packets marked as lost; and to train the second audio signal recovery model based on the at least one group of training voice samples to obtain the first audio signal recovery model;
  • the loss function in the second audio signal restoration model includes at least two items: adversarial loss parameter, Multi-resolution time-frequency domain signal loss parameters and relative feature matching loss parameters.
  • An embodiment of the present application provides an electronic device.
  • the residual network layer of the encoding module in the audio signal recovery model can be used to extract the time domain feature information of the cached voice signal before or after the lost voice signal, and the residual network layer of the decoding module in the audio signal recovery model can be used to synthesize the time domain feature information extracted by the encoding module to recover the lost voice signal.
  • the lost voice signal can be recovered by inputting the cached voice signal into the audio signal recovery model when the voice signal is lost, the lost voice signal can be recovered by the audio signal recovery model during the transmission of the voice signal, thereby avoiding the user's perception of the loss of the voice data packet, reducing the negative impact of the loss of the voice data packet, and thus effectively improving the quality of the voice call.
  • the input unit 1804 may include a graphics processing unit (GPU) 18041 and a microphone 18042, and the graphics processor 18041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode.
  • the display unit 1806 may include a display panel 18061, and the display panel 18061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc.
  • the user input unit 1807 includes a touch panel 18071 and at least one of other input devices 18072.
  • the touch panel 18071 is also called a touch screen.
  • the touch panel 18071 may include two parts: a touch detection device and a touch controller.
  • Other input devices 18072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
  • the memory 1809 can be used to store software programs and various data.
  • the memory 1809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc.
  • the memory 1809 may include a volatile memory or a non-volatile memory, or the memory 1809 may include both volatile and non-volatile memories.
  • the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (ELPROM).
  • ROM read-only memory
  • PROM programmable read-only memory
  • EPROM erasable programmable read-only memory
  • EPROM electrically erasable programmable read-only memory
  • ELPROM electrically erasable programmable read-only memory
  • the volatile memory can be a random access memory (Random Access Memory, RAM), a static random access memory (Static RAM, SRAM), a dynamic random access memory (Dynamic RAM, DRAM), a synchronous dynamic random access memory (Synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDRSDRAM), an enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), a synchronous connection dynamic random access memory (Synch link DRAM, SLDRAM) and a direct memory bus random access memory (Direct Rambus RAM, DRRAM).
  • the memory 1809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
  • the processor 1810 may include one or more processing units; optionally, the processor 1810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1810.
  • An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored.
  • a program or instruction is stored.
  • each process of the above-mentioned audio signal recovery method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
  • the processor is the processor in the electronic device described in the above embodiment.
  • the readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
  • An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio signal recovery method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
  • An embodiment of the present application provides a computer program product, which is stored in a storage medium.
  • the program product is executed by at least one processor to implement the various processes of the above-mentioned audio signal recovery method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the technical solution of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
  • a storage medium such as ROM/RAM, a disk, or an optical disk
  • a terminal which can be a mobile phone, a computer, a server, or a network device, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Transmission Systems Not Characterized By The Medium Used For Transmission (AREA)

Abstract

一种音频信号恢复方法、装置、电子设备及可读存储介质，属于信号处理领域，其中该方法包括：在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层（101）；将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息（102）；将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数（103）。

Description

音频信号恢复方法、装置、电子设备及可读存储介质
相关申请的交叉引用
本申请主张在2023年07月05日在中国提交的中国专利申请号202310823707.1的优先权，其全部内容通过引用包含于此。
技术领域
本申请属于信号处理领域，具体涉及一种音频信号恢复方法、装置、电子设备及可读存储介质。
背景技术
目前，在基于网际协议(Internet Protocol，IP)的语音通信中，语音通常会被分割成较小的语音数据包，并按一定时序进行传输。
然而，在相关技术中，在实际通过通信信道传输语音数据包的过程中，由于网络拥塞、抖动延迟等原因，经常会导致在传输过程中出现语音丢包现象。如此，容易导致语音通话质量较差。
发明内容
本申请实施例的目的是提供一种音频信号恢复方法、装置、电子设备及可读存储介质，能够提升语音通话的质量。
第一方面，本申请实施例提供了一种音频信号恢复方法，该方法包括：在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层；将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息；将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
第二方面，本申请实施例提供了一种音频信号恢复装置，该装置包括：输入模块和处理模块；输入模块，用于在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层；处理模块，用于将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息；处理模块，还用于将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
第三方面，本申请实施例提供了一种电子设备，该电子设备包括处理器和存储器，所述存储器存储可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如第一方面所述的方法的步骤。
第四方面，本申请实施例提供了一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如第一方面所述的方法的步骤。
第五方面，本申请实施例提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第一方面所述的方法。
第六方面，本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如第一方面所述的方法。
在本申请实施例中，在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层；将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息；将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数。通过该方案，在语音通话时，若丢失了某个语音信号，则可以通过 音频信号恢复模型中的编码模块的残差网络层提取缓存的丢失语音信号前或丢失语音信号后的语音信号的时域特征信息，并通过音频信号恢复模型中的解码模块的残差网络层合成编码模块提取的时域特征信息，恢复出丢失的语音信号。如此，由于可以在语音信号丢失的情况下，通过将缓存的语音信号输入音频信号恢复模型恢复出丢失的语音信号，因此在语音信号的传输过程中，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响，进而可以有效地提高语音通话的质量。
附图说明
图1是本申请实施例提供的一种AMR-WB解码器原理的结构示意图；
图2是本申请实施例提供的一种AMR-WB解码器丢包隐藏的实例流程图；
图3是本申请实施例提供的一种AMR-WB解码器丢包隐藏状态机判断错误帧与正常帧的实例示意图；
图4是本申请实施例提供的一种基于DNN的丢包隐藏方法的流程图；
图5是本申请实施例提供的一种基于DNN的特征提取的流程图；
图6是本申请实施例提供的一种基于DNN的语音时域信号波形重建的流程图；
图7是本申请实施例提供的一种音频信号恢复方法的流程图；
图8是本申请实施例提供的一种残差网络的结构示意图；
图9是本申请实施例提供的一种编码模块的网络结构示意图；
图10是本申请实施例提供的一种对丢失的语音信号进行恢复的流程图；
图11是本申请实施例提供的一种解码模块的网络结构示意图；
图12是本申请实施例提供的一种训练GAN网络模型的流程图；
图13是本申请实施例提供的一种第二音频信号恢复模型的训练流程图；
图14是本申请实施例提供的一种MS-STFT判别器的结构示意图；
图15是本申请实施例提供的一种音频信号恢复方法的实例流程示意图；
图16是本申请实施例提供的一种音频信号恢复装置的结构示意图；
图17是本申请实施例提供的一种电子设备的硬件结构示意图之一；
图18是本申请实施例提供的一种电子设备的硬件结构示意图之二。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书中的术语“第一”、“第二”等是用于区别类似的对象，而不用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施，且“第一”、“第二”等所区分的对象通常为一类，并不限定对象的个数，例如第一对象可以是一个，也可以是多个。此外，说明书以及权利要求中“和/或”表示所连接对象的至少其中之一，字符“/”，一般表示前后关联对象是一种“或”的关系。
下面结合附图，通过具体的实施例及其应用场景对本申请实施例提供的音频信号恢复方法、装置、电子设备及可读存储介质进行详细地说明。
本申请实施例提供的音频信号恢复方法、装置、电子设备及可读存储介质适用于基于IP进行语音通话的场景中。
目前，当进行基于IP的语音通信时，语音通常会被分割成较小的分组包或帧，并按一定时序进行传输。但是，在实际通信信道下，由于网络拥塞、抖动延迟等原因，必然会出现语音丢包现象，从而在很大程度上影响语音通话质量。在出现语音数据包丢失情况下，为了不使用户感觉到有丢包现象的发生，当检测到丢包后，需要对其进行错误隐藏，以便有效地提高最终语音通话的质量。
在相关技术中，针对丢包的错误隐藏方法有两种。一种为基于自适应多速率宽带编码(Adaptive Multi-Rate-Wideband，AMR-WB)解码器的丢包隐藏方法，另一种为基于深度神经网络(Deep Neural Network，DNN)的线性预测编码(Linear Predictive Coding，LPC)方法。
对于基于AMR-WB解码器的错误隐藏方法，AMR-WB解码器首先把接收到的编码码流恢复成参数编码，解码后得到20ms语音帧对应的编码参数。这些参数为：ISF参数、4个基音周期、4个固定码本矢量、4个自适应码本增益、4个固定码本增益。每帧ISF参数被转换为ISP参数，然后内插并转换得到4个子帧的LPC合成滤波器系数，每个子帧包含5ms的语音帧。AMR-WB解码器原理框图如图1所示，对于每个子帧，其可以在基音周期处内插过去的激励而得到自适应码本，并将自适应码本乘以自适应码本增益gp得到自适应码本激励v(n)；并将固定码本乘以固定码本增益gc得到固定码本激励c(n)；最后将自适应码本激励v(n)和固定码本激励c(n)相加得到总激励信号u(n)，通过LPC合成滤波器得到合成语音。
AMR-WB解码器语音编码丢包隐藏系统框图如图2所示，当编码码流在传输中发生了丢包后，解码端检测到丢包后调用丢包隐藏模块进行错误恢复。AMR-WB丢包隐藏算法使用状态机来确定错误帧的状态，并根据状态机状态来恢复编码参数，包括ISF参数、基音周期、固定码本矢量、自适应码本增益、固定码本增益。得到恢复的编码参数后，再使用AMR-WB解码器来合成语音。
AMR-WB解码器丢包隐藏状态机判断错误帧与正常帧的实例示意图如图3所示，当前帧的错帧指示符(Bad Frame Indicator，BFI)和前一帧的错帧指示符PrevBFI同时作为输入量，根据两个输入量确定当前帧处于状态机模型中的何种状态。图3中实线表示错误帧的变化路径，虚线表示正确帧的变化路径。系统从状态0开始。每次检测到一个错误语音帧，状态计数器就加1，当它达到6时饱和。每次检测到一个正确语音帧，状态计数器就右移1。状态表示信道的质量。可以理解，状态计数器的值越大，信道质量越差。
状态机的控制流程可以用下面的代码来描述，“BFI＝”为当前帧错帧指示，
“BFI＝1”为错误帧，“BFI＝0”为正确帧，“STATE＝”为状态变量，“PrevBFI＝”为前一帧错帧指示。
对于图3中STATE＝0或1、BFI＝0且PrevBFI＝0的情况，表示当前接收或先前接收的语音帧中没有检测到错误。换句话说，当前接收的正确语音帧将会被正常解码。
对于图3中STATE＝0至3、BFI＝0且PrevBFI＝1的情况，表示当前接收到的语音帧中没有检测到错误，但之前接收到的语音帧是错误的。那么可以通过下述公式(1)对当前正确接收帧的固定码本增益gc(n)参数进行修改，且其被限制在最后一个正确接收帧的最后一个子帧所使用的固定码本增益gc(n-1)值以下。
其中，为当前正确接收帧解码的固定码本增益，gc(n-1)为最后一个正确接收帧的最后一个子帧的固定码本增益，gc(n)为当前帧的固定码本增益，1.25为增益常数。可以理解，当前帧的其余编码参数可以正常解码并用于合成语音。
对于图3中STATE＝1至6、BFI＝1且PrevBFI＝0或1的情况，表示在当前接收到的语音帧中检测到错误，可以调用错误隐藏恢复编码参数。
对于自适应码本增益和固定码本增益的恢复，自适应码本增益和固定码本增益的恢复中使用的增益因子的大小可以根据状态机状态确定，状态值越大，表示信道质量越差，衰减因子也越大。在AMR-WB丢包隐藏算法中，自适应码本增益gp(n)和固定码本增益gc(n)可以由先前子帧的衰减值代替。自适应码本增益gp(n)可以通过下述的公式(2)进行计算：
gp(n)＝Pp(state)×median5(gp(n-1)，……，gp(n-5))   公式(2)
其中，gp(n)为当前帧的自适应码本增益，(gp(n-1)，……，gp(n-5))为过去5个子帧的自适应码本增益，median5(gp(n-1)，……，gp(n-5))表示过去5个自适应码本增益gp(n)数值的中间值。Pp(state)为抑制因子，STATE为当前状态机模 型的状态，Pp(1)＝0.98，Pp(2)＝0.96，Pp(3)＝0.75，Pp(4)＝0.23，Pp(5)＝0.05，Pp(6)＝0.01。可以理解，状态值越高，增益衰减得越多。
固定码本增益gc(n)可以通过下述的公式(3)进行计算：
其中，gc(n)为当前帧的固定码本增益，gc(n-1)，……，gc(n-5)为过去5个子帧的固定码本增益，median5(gc(n-1)，……，gc(n-5))表示过去5个固定码本增益gc(n)数值的中间值。VAD_HIST是连续VAD＝0判决的数量，VAD＝0表示当前帧为非语音帧。Pc(state)为抑制因子，STATE为当前状态机模型的状态，Pc(1)＝0.50，Pc(2)＝0.25，Pc(3)＝0.25，Pc(4)＝0.25，Pc(5)＝0.15，Pc(6)＝0.01。可以理解，状态值越高，增益衰减得越多。
对于ISF矢量参数的恢复，AMR-WB标准使用的谱系数是16阶ISF参数。在发生帧错误时，错误帧ISF参数的估计值通过ISF参数的自适应平均值插值得到，其可以通过下述公式(4)计算得到：
ISFq(i)＝α×past_ISFq(i)+(1-α)×ISFmean(i)   公式(4)
其中，i为帧索引，i＝0至15，α＝0.9，ISFq(i)为当前帧的ISF矢量参数，past_ISFq(i)为前一帧的ISF矢量参数，ISFmean(i)表示自适应均值和常量ISF的加权值，其可以通过公式(5)计算得到：
ISFmean(i)＝β×ISFconst_mean(i)+(1-β)×ISFadaptive_mean(i) 公式(5)
其中，β＝0.75，ISFconst_mean(i)是一个包含长时间平均ISF的矢量。
ISFadaptive_mean(i)是最近3个正确接收帧的past_ISFq(i)的平均值，只有当BFI＝0时(即正确接收帧)，ISFadaptive_mean(i)才进行更新。ISFadaptive_mean(i)可以通过公式(6)计算得到：
对于基因周期参数的恢复，可以将5个最后正确接收语音帧的基音周期和自适应码本增益分别被存入缓冲buffer，令其分别为Tbuffer和gp buffer，它们将用于寻找当前帧最佳的基音周期。基音周期恢复算法首先对过去子帧基音周期的可用性进行判断，如果浊音性和稳定性强，说明该错误帧和过去帧比较变化不大，丢失帧的基音周期可以用过去一帧的代替；否则，该错误帧的基音周期值在一定范围内随机产生。基音周期参数恢复算法分为下述的步骤1和步骤2：
步骤1：计算过去基音周期可用性因子Qlag_t-1，其可以通过公式(7)计算得到：
其中，表示自适应码本增益缓冲buffer中的最小值，gp(n-1)表示前一个正确接收帧的自适应码本增益，gp(n-2)是前一个正确接收帧之前的正确帧的自适应码本增益。
步骤2：估计当前帧基音周期T。步骤1得到Qlag_t-1后，当前帧基音周期T可以通过下述的公式(8)计算得到：
其中，T(n-1)为前一个正确接收帧的基音周期，Tmax为基音周期缓冲buffer中的最大值，Tmax-1为基音周期缓冲buffer中的第二大值，Tmax-2为基音周期缓冲buffer中的第三大值。RND(Tmax-Tmax-2)为一个随机数生成函数。
对于固定码本矢量的恢复，错误帧的固定码本矢量可以由随机产生的序列代替。
由于语音信号是一种短时平稳的准周期信号，相邻的语音帧之间存在着一定的相关性，因此现有的丢包隐藏方法通常利用编码参数的帧内与帧间的相关性来对丢包的语音编码参数进行恢复。在相关技术中，针对丢包的错误隐藏方法的语音编码通常基于码激励线性预测模型(Code Excited Linear Prediction，CELP)/代数码激励线性预 测(Algebraic Code Excited Linear Prediction，ACELP)，在编码过程中通常提取丢包隐藏(Packet Loss Concealment，PLC)参数、基音周期、自适应码本及其增益、固定码本及其增益等参数将其编码发送到解码端，解码端基于上述参数合成语音。
然而，由于CELP/ACELP编码方法固有的较紧密的帧间相关性，使得一个完整语音帧信息的丢失不仅会对当前丢失帧的合成语音质量造成严重影响，还会影响其后连续的几帧甚至十几帧。现在的丢包隐藏方法在连续丢帧时，其恢复的编码参数不准确，严重降低了合成语音的质量。同时，现有方法只能实现丢帧语音的声学特征参数恢复，尽量保持恢复语音波形的连续性，无法恢复丢帧语音的语义信息。
对于DNN的PLC方法，随着深度学习技术的发展，利用对数功率谱(Log Power Spectrum，LPS)和频谱相位。在相关技术中，DNN的PLC方法的流程图如图4所示，图4中∠Yl和∠Yf分别代表过去P帧信号的LPS特征和频谱相位，和分别代表DNN网络预测的LPS特征和频谱相位。在训练阶段，该方法利用干净语音训练2个DNN网络(均为包含4层隐藏层的全连接网络)，分别用于预测丢失语音帧的LPS和频谱相位。在丢包恢复阶段，首先判断当前接收帧是否为丢失信号帧，如果不是丢失帧，则进行正常解码。如果是丢失帧，则调用丢包隐藏方法进行恢复。
在执行丢包恢复时，其首先提取历史buffer中缓存的过去P帧信号，P为正整数。然后计算每帧信号的LPS和频谱相位(如图5所示)，并将过去P帧信号的LPS特征和频谱相位分别输入对应的DNN网络，分别预测当前丢失语音帧的LPS特征和频谱相位，得到预测的LPS特征和频谱相位后执行波形重建(如图6所示)，即可得到重建的时域信号帧。
其中，图5中Yt表示输入的时域信号，Yf表示对输入时域信号进行离散傅里叶变换(Disctete Fourier Transform，DFT)后得到的频谱，|·|2和Log(·)分别代表平方运算和对数运算。特征提取模块的输出为LPS特征Yl和频谱相位∠Yf。图6中和分别代表LPS特征和DNN网络预测的频谱相位，exp(·)代表指数运算。波形重建模块的输出为重建的时域信号
对于深度学习而言，训练集数据本身的结构性对网络训练的准确性十分重要，结构性在概率的角度分析就是数据服从某种概率分布。语音信号在频域中呈现复高斯分布，在该条件下，可以证明语音信号频谱的幅度服从瑞利分布，语音信号频谱的相位服从均匀分布。LPS通过幅度值变换而来，保留了结构性，可以被DNN网络学习。语音信号频谱的相位服从均匀分布，其随机性较强，很难被DNN网络学习。因此，通过DNN网络预测的频谱相位并不准确，其降低了合成语音信号的质量。
而本申请实施例提供的音频信号恢复方法、装置、电子设备及可读存储介质中，在语音通话时，若丢失了某个语音信号，则可以通过音频信号恢复模型中的编码模块的残差网络层提取缓存的丢失语音信号前或丢失语音信号后的语音信号的时域特征信息，并通过音频信号恢复模型中的解码模块的残差网络层合成编码模块提取的时域特征信息，恢复出丢失的语音信号。如此，由于可以在语音信号丢失的情况下，通过将缓存的语音信号输入音频信号恢复模型恢复出丢失的语音信号，因此在语音信号的传输过程中，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响，进而可以有效地提高语音通话的质量。
本申请实施例提供的音频信号恢复方法的执行主体可以为音频信号恢复装置。示例性地，该音频信号恢复装置可以为电子设备，也可以为该电子设备中的部件，例如集成电路或芯片。以下将以音频信号恢复装置为例对本申请实施例提供的音频信号恢复方法进行示例性说明。
本申请实施例提供一种音频信号恢复方法，图7示出了本申请实施例提供的一种音频信号恢复方法的流程图，该方法可以应用于音频信号恢复装置。如图7所示，本申请实施例提供的音频信号恢复方法可以包括下述的步骤101至步骤103。
步骤101、音频信号恢复装置在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型。
其中，上述第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，X，Y为正整数。
在本申请的一些实施例中，第一语音信号可以为当前语音通话中的一帧或多帧语音信号，也可以为一段录制好的音频中的一帧或多帧音频信号。
在本申请的一些实施例中，第一语音信号丢失可以表示音频信号接收端接收到的 语音信号不是连续的语音信号，即音频信号接收端接收到的语音信号中存在缺失的语音信号帧。
示例性地，音频信号发送端可以发送一段包含10帧的语音信号给音频信号接收端，音频信号接收端只接收到了第1-8帧和第10帧的语音信号，没有接收到第9帧语音信号。则第9帧语音信号为丢失的语音信号，即第一语音信号。
在本申请的一些实施例中，音频信号恢复装置可以缓存第一语音信号的前几帧语音信号，也可以缓存第一语音信号的后几帧语音信号，以便后续使用。具体的，音频信号恢复装置至少缓存了第一语音信号前的X帧语音信号和第一语音信号之后的Y帧语音信号中的至少一项。
示例性地，假设第一语音信号为5帧语音信号中的第4帧语音信号，则音频信号恢复装置可以缓存第1-3帧语音信号，也可以缓存第5帧语音信号，还可以缓存第1-3帧语音信号和第5帧语音信号。
可以理解，在实时的语音通话中，音频信号恢复装置至少缓存了第一语音信号前的X帧语音信号；在非实时的音频中，如语音留言，音频信号恢复装置至少缓存了第一语音信号前的X帧语音信号和第一语音信号之后的Y帧语音信号。
本申请的一些实施例中，上述第一音频信号恢复模型包括编码模块和解码模块，其中，上述编码模块包括M层第一残差网络层，上述解码模块包括M层第二残差网络层，M为正整数。
在本申请的一些实施例中，上述第一音频信号恢复模型中的编码模块和解码模块均可以采用卷积编解码网络(Convolutional Encoder-Decoder Network，CEDN)对语音信号的时域特征信息进行卷积。如此，可以仅基于语音信号的时域特征信息对丢失的语音信号进行恢复，而无需进行时频变换，从而可以避免频域处理过程中由于相位预测不准确而造成的音频失真，进而可以恢复出符合语音特征、与前后文语音连贯的高质量语音。
在本申请的一些实施例中，CEDN网络是一种对称性网络，由预卷积层(pre-conv)、残差网络层、采样层和后卷积层(post-conv)组成。
其中，预卷积层和后卷积层可以用来改变输入第一音频信号恢复模型的信号的通道数，从而使得其输出的时域特征信息可以输入其他的网络层；采样层可以包括上采样层或下采样层，用来升高或降低时域特征信息的维度并保留有效信息；残差网络层可以用于提取时域特征信息。
需要说明的是，音频信号恢复装置在对音频信号恢复模型中的任一层进行输入，得到对应的时域特征信息后，还可以将得到的时域特征信息经过归一化处理，以提升采用反向传播算法训练的网络的收敛性。
步骤102、音频信号恢复装置将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息。
在本申请的一些实施例中，上述第一时域特征信息可以包括第二语音信号的时域波形图的特征信息，如时域波形图的周期、幅度。进一步地，时域特征信息可以以第二语音信号的时域波形特征图的形式在M层第一残差网络层间进行传输。
在本申请的一些实施例中，第二语音信号经M层第一残差网络层依次处理的详细描述，可以参见下述步骤102b1和步骤102b2中的相关描述，此处不予赘述。
可选地，在本申请的一些实施例中，上述步骤102可以包括下述的步骤102a和步骤102b。
步骤102a、音频信号恢复装置通过编码模块中的预卷积层，提取第二语音信号的第二时域特征信息。
在本申请的一些实施例中，音频信号恢复装置可以通过编码模块中的预卷积层，改变第二语音信号的通道数，从而将第二语音信号转换成多维度的时域特征信息，即第二时域特征信息。
示例性地，假设预卷积层的输入通道数为1，输出通道数为32，那么音频信号恢复装置可以将第二语音信号输入预卷积层，得到通道数为32的第二时域特征信息。也就是说，音频信号恢复装置通过编码模块中的预卷积层，将第二语音信号转换成32维时域特征信息，即第二时域特征信息。
步骤102b、音频信号恢复装置将第二时域特征信息经M层第一残差网络层依次处理，得到第一时域特征信息。
在本申请的一些实施例中，第一时域特征信息相较于第二时域特征信息为具有更 多维度且包含更高语义的时域特征信息。如此，音频信号恢复装置可以将第二时域特征信息经M层第一残差网络层依次进行特征提取处理，得到第一时域特征信息。
如此，由于音频信号恢复装置可以先通过编码模块中的预卷积层，提取出第二语音信号的第二时域特征信息，再通过将该第二时域特征信息经M层第一残差网络依次处理，得到第一时域特征信息。因此，音频信号恢复装置可以通过逐层提取语音信号的时域特征信息，最终得到具有更多维度且包含更高语义的时域特征信息，从而可以通过该包含更高语义的时域特征信息完成对丢失语音信号的恢复，提高语音通信的质量。
可选地，在本申请的一些实施例中，上述步骤102b可以包括下述的步骤102b1和步骤102b2。
步骤102b1、音频信号恢复装置将输入第i层第一残差网络层的时域特征信息，通过第i层第一残差网络层的第一卷积层处理，得到第三时域特征信息，并通过第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到第四时域特征信息。
其中，上述第一卷积层、第二卷积层以及第三卷积层均为一维卷积层，第一卷积层的输入通道数和输出通道数相同，第二卷积层的卷积核与第一卷积层的卷积核不同、且第三卷积层的卷积核与第一卷积层的卷积核相同，i∈{1，……，M}。
在本申请的一些实施例中，第三时域特征信息为包含第一卷积层的卷积核大小的信息尺度的时域特征信息，第四时域特征信息为包含第二卷积层的卷积核大小的信息尺度的时域特征信息。即，第三时域特征信息和第四时域特征信息为尺度不同的时域特征信息。
可以理解，第i层第一残差网络层可以为M层第一残差网络层中的任一层第一残差网络层。
示例性地，第i层第一残差网络层可以为M层第一残差网络层中的第一层第一残差网络层，也可以为最后一层第一残差网络层，还可以为中间的任一层第一残差网络层。
在相关技术中，为了提高神经网络的性能，传统的经典卷积神经网络通常是通过增加神经网络的层数来更好的提取特征。然而，一味的增加神经网络的层数不仅不会提高神经网络的性能，反而会出现梯度消失问题，神经网络也会不可避免地发生退化从而阻碍神经网络模型的训练，此时层数少的神经网络在性能上反而优于层数多的网络。此外，神经网络模型层数的加深通常会使得神经网络模型的复杂度变高，容易导致神经网络发生过拟合。因此，本申请实施例提供的音频信号恢复方法还提供了一种残差网络(Modified-Resnet)层，其结构示意图如图8所示。
可选地，本申请的一些实施例中，如图8所示，上述残差网络层可以由并行的两条支路构成。其中一条支路包含卷积层81和其所对应的归一化层；另一条支路包含卷积层82和其所对应的归一化层、激活函数，卷积层83和其所对应的归一化层、激活函数。其中，卷积层81、卷积层82和卷积层83均为一维卷积层，卷积层81的输入通道数与输出通道数相同，卷积层82的卷积核与卷积层81的卷积核不同、且卷积层83的卷积核与卷积层81的卷积核相同。如此，音频信号恢复装置可以通过多个一维卷积层增加通道间的信息交互，同时可以采用两条支路并行的结构，通过分别使用卷积核大小不同的卷积层对输入的特征进行卷积，并将得到的不同尺度的时域特征信息进行融合，因此可以使得每一层第一残差网络都可以提取到更加丰富的时域特征信息。
可选地，在本申请的一些实施例中，上述第一残差网络层可以包括并行的两条支路，一条支路包括第一卷积层，另一条支路包括第二卷积层和第三卷积层。也就是说，音频信号恢复装置在将输入第i层第一残差网络层的时域特征信息输入第i层第一残差网络层之后，可以通过该两条支路同时得到第三时域特征信息和第四时域特征信息。
可选地，在本申请的一些实施例中，以图8所示残差网络层的结构为例，卷积层81为第i层第一残差网络层的第一卷积层，卷积层82为第i层第一残差网络层的第二卷积层，卷积层83为第i层第一残差网络层的第三卷积层。音频信号恢复装置可以将输入第i层第一残差网络层的时域特征信息，通过卷积层81和归一化层处理，得到第三时域特征信息；并通过第i层第一残差网络层的卷积层82及其对应的归一化层、卷积层83及其对应的归一化层处理，得到第四时域特征信息。
需要说明的是，对于第i层第一残差网络层的第二卷积层和第三卷积层，音频信 号恢复装置可以先将输入第i层第一残差网络层的时域特征信息，通过激活函数层处理，使得输入第i层第一残差网络层的时域特征信息增加非线性特征，从而使得第二卷积层和第三卷积层可以更好的对输入第i层第一残差网络层的时域特征信息进行特征提取。
在本申请的一些实施例中，可以采用指数线性单元(Exponential Linear Unit，ELU)激活函数以及权重归一化来保证第一音频信号恢复模型的稳定性。在实际实现中，还可以通过其他的激活函数和归一化方法来保证第一音频信号恢复模型的稳定性。本申请实施例不作具体限定。
步骤102b2、音频信号恢复装置基于第三时域特征信息和第四时域特征信息，得到第i层第一残差网络层输出的时域特征信息。
在本申请的一些实施例中，音频信号恢复装置可以将第三时域特征信息和第四时域特征信息中不同尺度的时域特征信息相加，得到第i层第一残差网络层输出的时域特征信息。
可以理解，第M层第一残差网络层输出的时域特征信息则为第一时域特征信息。
需要说明的是，对于M层第一残差网络层中的每层第一残差网络，音频信号恢复装置均可以对其执行上述步骤102b1和步骤102b2，最终得到第一时域特征信息。
在本申请的一些实施例中，可以为第二卷积层和第三卷积层设置不同的输入通道数、输出通道数以及卷积核，以使得得到的第四时域特征信息的通道数可以与第三时域特征信息的通道数相同但尺度不同，从而使得第三时域特征信息可以与第四时域特征信息相加可以得到更加丰富的时域特征信息。
示例性地，如图8所示，音频信号恢复装置可以在卷积层82的输入通道数设置为输出通道数的2倍，卷积核设置为3，再将卷积层83的输出通道数设置为输入通道数的2倍，卷积核设置为1，以使得经卷积层82和卷积层83处理后输出的时域特征信息的通道数，与经卷积层81处理后输出的时域特征信息的通道数相同但尺度不同。
例如，假设输入第i层第一残差网络层的时域特征信息的通道数为128，则第一卷积层的输入通道数为128，输出通道数为128，卷积核为1；第二卷积层的输入通道数为128，输出通道数为64，卷积核为3；第三卷积层的输入通道数为64，输出通道数为128，卷积核为1。如此，输入第i层第一残差网络层的时域特征信息通过第一卷积层处理，可以得到通道数为128，卷积核为1的第三时域特征信息；并通过第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到通道数为128，卷积核为3的第四时域特征信息。音频信号恢复装置在将第三时域特征信息与第四时域特征信息相加之后，可以得到包含卷积核为1和卷积核为3两个尺度的时域特征信息。
在实际实现中，可以设置其他大小的卷积核和输入输出通道数，以得到不同尺度的时域特征信息。本申请实施例不作具体限定。
如此，由于第一残差网络层可以采用两条支路并行的结构，通过分别使用卷积核大小不同的卷积层对输入的特征进行卷积，并将得到的不同尺度的时域特征信息进行融合，因此可以使得每一层第一残差网络都可以提取到更加丰富的时域特征信息。
可选地，在本申请的一些实施例中，上述编码模块还可以包括M层下采样层，M层下采样层与M个第一残差网络层交替排列；M层下采样层的输出通道数按照层级从高至低依次增大。
示例性地，上述步骤102b可以包括下述的步骤102b3和步骤102b4。
步骤102b3、音频信号恢复装置将输入第x层第一残差网络层的时域特征信息，通过第x层第一残差网络层处理，得到第五时域特征信息。
步骤102b4、音频信号恢复装置将第五时域特征信息通过与第x层第一残差网络层连接的第x层下采样层输出。
其中，x∈{1，……，M}。
在本申请的一些实施例中，下采样层可以用来增加输入的时域特征信息的通道数，以降低时域特征信息的维度并保留有效信息，防止时域特征信息丢失现象的发生。
示例性地，假设第x层第一残差网络层的通道数为32，则第x层下采样层的输入通道数为32，音频信号恢复装置可以设置第x层下采样层的输出通道数为62，在经过第x层残差网络提取时域特征信息后，增加时域特征信息的通道数，防止时域特 征信息的丢失。
在本申请的一些实施例中，编码模块中的下采样层的输出通道数可以按照层级从高至低依次增大。其中，M层下采样层的输出通道数可以为预先设置好的，也可以为按照固定数值按照层级从高至低依次增大。
示例性地，以编码模块包括4层下采样层，且该4层下采样层的输出通道数为预先设置好的为例。第一层下采样层的输入通道数可以预先设置为32，输出通道数可以预先设置为64；第二层下采样层的输入通道数可以预先设置为64，输出通道数可以预先设置为128；第三层下采样层的输入通道数可以预先设置为128，输出通道数可以预先设置为256；第四层下采样层的输入通道数可以预先设置为256，输出通道数可以预先设置为512。
示例性地，以编码模块包括4层下采样层，且该4层下采样层的输出通道数按照固定数值按照层级从高至低依次增大为例。假设固定数值为64，则第一层下采样层的输入通道数可以为32，输出通道数可以为96；第二层下采样层的输入通道数可以为96，输出通道数可以为160；第三层下采样层的输入通道数可以为160，输出通道数可以为224；第四层下采样层的输入通道数可以为224，输出通道数可以为288。
需要说明的是，M层下采样层的输入通道数随层级高低依次从高至低逐渐增大是因为M层下采样层的输出通道数在增大，为了可以下采样层之间的连接与时域特征信号的传输，因此M层下采样层的输入通道数也会随之增大。
在本申请的一些实施例中，音频信号恢复装置可以通过M层下采样层对时域特征信息进行逐层下采样。具体地，音频信号恢复装置可以通过采用渐进式采样和具有不同感受野的滤波器滤波对时域特征信息进行逐层下采样，从而可以有效缓解大范围直接下采样造成的时域特征图波形细节的损失，并且可以通过对对不同尺度的时域特征信息的通道数进行调整，增加时域特征信息的表征性。
本申请的一些实施例中，网络层的层级高低可以根据该网络层在第一音频恢复模型中的位置确定。具体地，第一音频恢复模型中最上层的网络层层级最高，第一音频恢复模型中处于中间位置的网络层层级根据位置的下移而降低，第一音频恢复模型中最下层的网络层层级最低。
在本申请的一些实施例中，第一音频信号恢复模型的编码模块中可以包含M层第一残差网络层和M层下采样层，编码模块中各网络层可以由第一残差网络层和下采样层交替进行排布。即，输入第x层第一残差网络层的时域特征信息在通过第x层第一残差网络层进行时域特征信息提取之后，可以随即通过第x层下采样层增加时域特征信息的通道数，防止时域特征信息丢失。
示例性地，假设编码模块包含一层预卷积层、一层后卷积层、4层第一残差网络层以及4层下采样层，编码模块的网络结构如图9所示。音频信号恢复装置可以将第二语音信号输入编码模块，经预卷积层901处理，将第二语音信号先通过一个输入通道数为1、输出通道数为32、卷积核大小为7、步长为1的一维卷积，并将预卷积层901输出的时域特征信息通过归一化函数处理，得到第二语音信号的第二时域特征信息；然后将第二时域特征信息通过第一残差网络层902处理，即第一层第一残差网络层，得到第一层第一残差网络层输出的通道数为32的时域特征信息；将第一残差网络层902输出的时域特征信息输入下采样层903，即第一层下采样层，先通过一个输入通道数为32、输出通道数为64、卷积核大小为4、步长为2的一维卷积，并将一维卷积输出的时域特征信息经归一化函数处理，得到下采样层903输出的通道数为64的时域特征信息；再将下采样层903输出的时域特征信息经第一残差网络层904处理，即第二层第一残差网络层，得到第一残差网络层904输出的通道数为64的时域特征信息；然后将该通道数为64的时域特征信息输入下采样层905，通过一个输入通道数为64、输出通道数为128、卷积核大小为8、步长为4的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数处理，得到通道数为128的时域特征信息；再将该通道数为128的时域特征信息输入第一残差网络层906，即第三层第一残差网络层，得到更高语义的通道数为128的时域特征信息；然后将第一残差网络层906输出的时域特征信息输入下采样层907，即第三层下采样层，通过一个输入通道数为128、输出通道数为256、卷积核大小为10、步长为5的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为256的时域特征信息；再将该通道数为256的时域特征信息输入第一残差网络层908，即第四层第一残差网络层，得到更高语义的通道数为256的时域特征信息；然后将第一残差网络层908输出 的时域特征信息输入下采样层909，即第四层下采样层，通过一个输入通道数为256、输出通道数为512、卷积核大小为16、步长为8的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为512的时域特征信息；最后将该通道数为512的时域特征信息输入后卷积层910，先通过激活函数，再通过一个输入通道数为512、输出通道数为128、卷积核大小为7、步长为1的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为128的第一时域特征信息，并将该第一时域特征信息从编码模块中输出。
如此，由于音频信号恢复装置可以通过M层下采样层，对语音信号的时域特征信息进行逐层的特征提取和下采样，因此可以在增加提取到的时域特征信息的表征性的同时，避免时域特征信息的损失。
步骤103、音频信号恢复装置将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号。
在本申请的一些实施例中，音频信号恢复装置将第一时域特征信息经M层第二残差网络层依次处理的过程，实际上是根据第一时域特征信息恢复第一语音信号，得到目标语音信号的过程。
在本申请的一些实施例中，上述音频信号恢复装置在得到目标语音信号之后，可以将目标语音信号作为恢复的第一语音信号进行输出。
示例性地，假设音频信号恢复装置缓存有第1、2、3帧的语音信号，即第二语音信号，在第4帧语音信号丢失的情况下，即第一语音信号丢失的情况下，音频信号恢复装置可以将第1、2、3帧的语音信号输入第一音频信号恢复模型中，通过第一音频信号恢复模型中的预处理模块将第1、2、3帧的语音信号映射为1帧语音信号，将该1帧语音信号经编码模块提取出其所对应的时域特征信息；并将提取出的时域特征信息经解码模块生成1帧语音信号，即目标语音信号，该生成的1帧语音信号即为恢复的第4帧语音信号。
在本申请的一些实施例中，音频信号恢复装置是完成对第一语音信号的恢复之后，可以对缓存的语音信号进行更新。
示例性地，音频信号恢复装置可以缓存第一语音信号前的3帧语音信号，即缓存有第1、2、3帧的语音信号，在音频信号恢复装置完成对第4帧语音信号的恢复之后，可以对第4帧语音信号进行缓存，使得音频信号恢复装置中缓存的语音信号更新为第2、3、4帧语音信号。
在本申请的一些实施例中，音频信号恢复装置对缓存的语音信号进行更新时，可以以将每一帧语音信号覆盖前一帧语音信号的方式更新缓存，也可以以最后一帧语音信号覆盖最前一帧语音信号的方式跟新缓存。本申请实施例不作具体限定。
在本申请的一些实施例中，在上述第一语音信号没有丢失的情况下，音频信号恢复装置可以对其进行正常解码并更新缓存。
示例性地，以音频信号恢复装置缓存有第1帧语音信号X1、第2帧语音信号X2、第3帧语音信号X3、第4帧语音信号X4、第5帧语音信号X5、第6帧语音信号X6、第7帧语音信号X7、第8帧语音信号X8、第9帧语音信号X9、第10帧语音信号X10，正在接收第11帧语音信号X11为例。如图10所示，为音频信号恢复装置接收语音信号时对丢失的语音信号进行恢复的流程图。音频信号恢复装置首先判断第11帧语音信号X11是否丢失。在第11帧语音信号X11没有丢失的情况下，音频信号恢复装置可以对其进行正常解码，输出第11帧语音信号X11并更新缓存；在第11帧语音信号X11丢失的情况下，音频信号恢复装置可以将缓存的第1帧语音信号X1、第2帧语音信号X2、第3帧语音信号X3、第4帧语音信号X4、第5帧语音信号X5、第6帧语音信号X6、第7帧语音信号X7、第8帧语音信号X8、第9帧语音信号X9、第10帧语音信号X10，输入第一音频信号恢复模型，生成目标语音信号，将目标语音信号作为恢复的第11帧语音信号X11进行输出并更新缓存。可以理解，更新后的缓存包含第2帧语音信号X2、第3帧语音信号X3、第4帧语音信号X4、第5帧语音信号X5、第6帧语音信号X6、第7帧语音信号X7、第8帧语音信号X8、第9帧语音信号X9、第10帧语音信号X10、第11帧语音信号X11。
可选地，在本申请的一些实施例中，上述步骤103可以包括下述的步骤103a和步骤103b。
步骤103a、音频信号恢复装置将第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息。
在本申请的一些实施例中，第一时域特征信息相较于第六时域特征信息为具有更多维度且包含更高语义的时域特征信息。换句话说，第一时域特征信息与第六时域特征信息为不同尺度的时域特征信息。
进一步可选地，在本申请的一些实施例中，上述步骤103a可以包括下述的步骤103a1和步骤103a2。
步骤103a1、音频信号恢复装置将输入第j层第二残差网络层的时域特征信息，通过第j层第二残差网络层的第四卷积层处理，得到第七时域特征信息，并通过第j层第二残差网络层的第五卷积层以及第六卷积层进行处理，得到第八时域特征信息。
其中，第四卷积层、第五卷积层以及第六卷积层均为一维卷积层，第四卷积层的输入通道数和输出通道数相同，第五卷积层的卷积核与第四卷积层的卷积核不同、且第六卷积层的卷积核与第四卷积层的卷积核相同，j∈{1，……，M}。
步骤103a2、音频信号恢复装置基于第七时域特征信息和第八时域特征信息，得到第j层第二残差网络层输出的时域特征信息。
对于第二残差网络层的第四卷积层、第五卷积层和第六卷积层的详细描述，可以参见上述步骤102b1和步骤102b2中关于第一卷积层、第二卷积层和第三卷积层的相关描述，此处不再赘述。
如此，由于第二残差网络层可以采用两条支路并行的结构，通过分别使用卷积核大小不同的卷积层对输入的特征进行卷积，并将得到的不同尺度的时域特征信息进行融合，因此可以使得每一层第一残差网络都可以提取到更加丰富的时域特征信息。
可选地，在本申请的一些实施例中，上述解码模块还可以包括M层上采样层，M层上采样层与M个第二残差网络层交替排列；M层上采样层的输出通道数按照层级从高至低依次降低。
示例性地，上述步骤103a还可以包括下述的步骤103a3。
步骤103a3、音频信号恢复装置将第y层上采样层输出的时域特征信息，通过与第y层上采样层连接的第y层第二残差网络层处理，得到第y层第二残差网络层输出的时域特征信息。
其中，y∈{1，……，M}。
在本申请的一些实施例中，上采样层可以用来减少输入的时域特征信息的通道数，以通过对第一时域特征信息逐层上采样，将时域特征信息重建为原始大小，并增加时域特征信息的维度，降低时域特征信息的尺度，实现语音信号的恢复。
示例性地，假设第y层第二残差网络层的通道数为128，则与第y层第二残差网络层连接的第y层下采样层的输入通道数为128，音频信号恢复装置可以设置第y层下采样层的输出通道数为62，在经过第y层残差网络提取时域特征信息前，减少时域特征信息的通道数，降低时域特征信息的维度，实现语音信号的恢复。
在本申请的一些实施例中，输入的时域特征信息可以被解码模块中的上采样层的输出通道数按照层级从高至低依次降低。其中，M层上采样层的输出通道数可以为预先设置好的，也可以为按照固定数值按照层级从高至低依次降低。
示例性地，以解码模块包括4层上采样层，且该4层上采样层的输出通道数为预先设置好的为例。第一层上采样层的输入通道数可以预先设置为512，输出通道数可以预先设置为218；第二层上采样层的输入通道数可以预先设置为218，输出通道数可以预先设置为128；第三层上采样层的输入通道数可以预先设置为128，输出通道数可以预先设置为64；第四层上采样层的输入通道数可以预先设置为64，输出通道数可以预先设置为32。
示例性地，以编码模块包括4层上采样层，且该4层上采样层的输出通道数按照固定数值按照层级从高至低依次降低为例。假设固定数值为64，则第一层上采样层的输入通道数可以为288，输出通道数可以为224；第二层上采样层的输入通道数可以为224，输出通道数可以为160；第三层上采样层的输入通道数可以为160，输出通道数可以为96；第四层上采样层的输入通道数可以为96，输出通道数可以为32。
需要说明的是，M层上采样层的输入通道数随层级高低依次从高至低逐渐降低是因为M层上采样层的输出通道数在降低，为了时域特征信号能在两个上采样层之间传输，因此M层上采样层的输入通道数也会随之降低。
在本申请的一些实施例中，音频信号恢复装置可以通过M层上采样层对时域特征信息进行逐层上采样，通过采用渐进式采样和具有不同感受野的滤波器滤波可以有效缓解大范围直接上采样造成的时域特征图波形细节的损失，并且可以通过对对不同 尺度的时域特征信息的通道数进行调整，实现语音信号的恢复。
在本申请的一些实施例中，第一音频信号恢复模型的解码模块中可以包含M层上采样层和M层第二残差网络层，解码模块中各网络层可以由上采样层和第二残差网络层交替进行排布。即，输入第i层上采样层的时域特征信息在通过第i层上采样层减少时域特征信息的通道数之后，可以随即通过第i层第二残差网络层进行时域特征信息的特征提取，以恢复语音信号。
示例性地，假设解码模块包含一层预卷积层、一层后卷积层、4层上采样层以及4层第二残差网络层，解码模块的网络结构如图11所示。音频信号恢复装置可以先将第一时域特征信息输入解码模块，经预卷积层1101处理，将第一时域特征信息先通过一个输入通道数为128、输出通道数为512、卷积核大小为7、步长为1的一维卷积，并将预卷积层1101输出的时域特征信息通过归一化函数处理，得到通道数为32的时域特征信息；然后将该通道数为32的时域特征信息输入上采样层1102，即第一层上采样层，通过一个输入通道数为512、输出通道数为256、卷积核大小为16、步长为8的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为256的时域特征信息；再将该通道数为256的时域特征信息输入第二残差网络层1103，即第一层第二残差网络层，得到通道数为256的时域特征信息；然后将该通道数为256的时域特征信息输入上采样层1104，即第二层上采样层，通过一个输入通道数为256、输出通道数为128、卷积核大小为10、步长为5的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为128的时域特征信息；再将该通道数为128的时域特征信息输入第二残差网络层1105，即第二层第二残差网络层，得到通道数为128的时域特征信息；然后将该通道数为128的时域特征信息输入上采样层1106，即第三层上采样层，通过一个输入通道数为128、输出通道数为64、卷积核大小为8、步长为4的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为64的时域特征信息；再将该通道数为64的时域特征信息输入第二残差网络层1107，即第三层第二残差网络层，得到通道数为64的时域特征信息；然后将该通道数为64的时域特征信息输入上采样层1108，即第四层上采样层，通过一个输入通道数为64、输出通道数为32、卷积核大小为4、步长为2的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为32的时域特征信息；再将该通道数为32的时域特征信息输入第二残差网络层1109，即第四层第二残差网络层，得到通道数为32的时域特征信息；最后将该通道数为32的时域特征信息输入后卷积层1110，通过一个输入通道数为32、输出通道数为1、卷积核大小为7、步长为1的一维卷积，并将该一维卷积输出的时域特征信息通过归一化函数，得到通道数为1的目标语音信号，并将该目标语音信号从解码模块中输出。
如此，由于音频信号恢复装置可以通过M层上采样层，对语音信号的时域特征信息进行逐层的上采样和特征提取，因此可以根据编码模块提取到的时域特征信息，恢复出丢失的语音信号。
步骤103b、音频信号恢复装置将第六时域特征信息输入解码模块的后卷积层处理，得到目标语音信号。
在本申请的一些实施例中，音频信号恢复装置可以通过解码模块中的后卷积层，改变第六时域特征信息的通道数，从而将第六时域特征信息进行合成，得到目标语音信号。
示例性地，假设后卷积层的输入通道数为32，输出通道数为1，那么音频信号恢复装置可以将第六时域特征信息输入后卷积层，将第六时域特征信息的32个通道合为1个通道，从而得到目标语音信号。可以理解，音频信号恢复装置可以通过解码模块中的后卷积层，将时域特征信息转换成语音信息。
如此，由于音频信号恢复装置可以先将第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息，再通过解码模块中的后卷积层，得到目标语音信号。因此，音频信号恢复装置可以通过经M层第二残差网络逐层还原出丢失语音信号的时域特征信息，并通过解码模块中的后卷积层将时域特征信息转换为语音信号，最终会恢复出丢失的语音信号。
在本申请实施例提供的音频信号恢复方法中，在语音通话时，若丢失了某个语音信号，则可以通过音频信号恢复模型中的编码模块的残差网络层提取缓存的丢失语音信号前或丢失语音信号后的语音信号的时域特征信息，并通过音频信号恢复模型中的 解码模块的残差网络层合成编码模块提取的时域特征信息，恢复出丢失的语音信号。如此，由于可以在语音信号丢失的情况下，通过将缓存的语音信号输入音频信号恢复模型恢复出丢失的语音信号，因此在语音信号的传输过程中，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响，进而可以有效地提高语音通话的质量。
可选地，在本申请的一些实施例中，在第二语音信号为多帧语音信号的情况下，在上述步骤102之前，本申请实施例提供的音频信号恢复方法还可以包括下述的步骤104，结合步骤104，上述步骤102可以包括下述的步骤102c。
步骤104、音频信号恢复装置将多帧语音信号输入第一音频信号恢复模型之后，将多帧语音信号映射为一帧语音信号。
在本申请的一些实施例中，在第二语音信号为多帧语音信号的情况下，该多帧语音信号可以包括第一语音信号前的至少两帧语音信号，也可以包含第一语音信号之后的至少两帧语音信号，还可以包含第一语音信号前的至少一帧语音信号和第一语音信号之后的至少一帧语音信号。本申请实施例不作具体限定。
在本申请的一些实施例中，上述第一音频信号恢复模型还可以包括预处理模块，音频信号恢复装置可以通过预处理模块将多帧语音信号映射为一帧语音信号。
示例性地，音频信号恢复装置可以提取该多帧语音信号的时域特征信息，映射至一帧语音信号中。
步骤102c、音频信号恢复装置将一帧语音信号经M层第一残差网络层依次处理，得到第一时域特征信息。
对于音频信号恢复装置将一帧语音信号经M层第一残差网络层依次处理的过程的描述，可以参见上述步骤102中音频信号恢复装置将第二语音信号经M层第一残差网络层依次处理的过程的相关描述，此处不再赘述。
如此，由于频信号恢复装置可以在第二语音信号为多帧语音信号的情况下，通过将多帧语音信号映射为一帧语音信号，对该一帧语音信号经M层第一残差网络层依次处理，得到第一时域特征信息。因此可以只对该多帧语音信号中有用的部分进行处理，避免对多帧语音信号中无用的部分的处理，节省的频信号恢复模型的资源。
在本申请的一些实施例中，在上述步骤101之前，本申请实施例提供的音频信号恢复装置还可以包括下述的步骤105和步骤106。
步骤105、音频信号恢复装置获取至少一组训练语音样本。
其中，每组训练语音样本包括标记为正确接收的语音包和标记为丢失的语音包。
在本申请的一些实施例中，上述每组训练语音样本中标记为丢失的语音包可以为根据真实语音模拟语音丢包得到的。可以理解的是，实际实现中，一个完整的语音包可以包括上述正确接收的语音包和标记为丢失的语音包。
示例性地，假设真实语音包含5帧语音信号，则包含丢失语音信号的语音包可以包含丢失第4帧语音信号的第1-3帧和第5帧语音信号。
步骤106、音频信号恢复装置基于至少一组训练语音样本，对第二音频信号恢复模型训练，得到第一音频信号恢复模型。
其中，第二音频信号恢复模型中的损失函数包括以下至少两项：对抗损失参数、多分辨率时频域信号损失参数以及相对特征匹配损失参数。
可以理解，上述第二音频信号恢复模型为不具备音频信号恢复功能的原始模型。即，第二音频信号恢复模型为未经训练的第一音频信号恢复模型。
在本申请的一些实施例中，上述第二音频信号恢复模型可以为基于生成对抗网络(Generative Adversarial Network，GAN)的深度学习网络模型。其中，GAN网络通常由两个同时训练、相互竞争的网络组成，即生成器(Generator)和判别器(Discriminator)。
示例性地，在GAN网络的训练过程中，如图12所示，判别器的输入由两部分组成，分别是真实样本数据和生成器生成的数据，其输出通常是一个概率值，表示判别器认定输入是真实数据的分布概率。若输入来自真实数据，则输出为1，否则输出0。生成器可以接收一个随机变量，生成假样本数据，判别器判断该假样本数据为真实数据的分布概率；并且判别器的输出还会反馈给生成器，用于指导生成器的训练。在理想情况下，判别器无法判别输入数据是来自真实样本数据还是生成器生成的数据，即判别器每次的输出概率都为0.5，此时生成器生成的数据无限接近真实样本数据，模型达到最优。在实际应用中，生成器和判别器通常采用深度神经网络来实现。
在实际实现中，GAN网络对生成器输入的随机变量基本没有限制，随机变量可以为随机噪声或者为符合某种分布的变量。生成器理论上可以逐渐学习任何概率分布，训练后的生成器可以生成和真实数据无限接近的、但又不是和真实数据完全一样的数据，即生成器实际上是学习了训练数据的一个近似分布。判别器的主要目的是判断输入的数据是否为真实的样本数据，并提供反馈以指导生成器进行训练。在GAN网络的训练过程中，生成器和判别器会各自更新自身的参数使得损失最小，通过不断迭代优化，最终达到纳什均衡状态。可以理解，上述第二音频信号恢复模型为GAN网络中的生成器。
在本申请的一些实施例中，音频信号恢复装置可以将获取到的训练语音样本中包含丢失语音信号的语音包，输入第二音频信号恢复模型，生成一个语音信号；音频信号恢复装置可以通过判别器判断生成的语音信号与真实语音的相似度，判断第二音频信号恢复模型恢复丢失语音信号是否成功，并根据判别器的判断结果对生成器模型参数进行调整，以提升生成的语音信号与真实语音的相似度。
具体地，音频信号恢复装置可以缓存语音包中未丢失的语音信号。第二音频信号恢复模型可以通过编码模块提取缓存的语音信号的时域特征信息，并通过解码模块根据提取的时域特征信息，生成语音信号。音频信号恢复装置可以通过判别器判断生成的语音信号与真实语音的相似度，判断第二音频信号恢复模型恢复丢失语音信号是否成功。音频信号恢复装置可以在判别器判断生成的语音信号与真实语音的相似度大于或等于相似度阈值的情况下，判断第二音频信号恢复模型成功恢复丢失的语音信号；在判别器判断生成的语音信号与真实语音的相似度小于相似度阈值的情况下，判断第二音频信号恢复模型恢复丢失的语音信号失败，音频信号恢复装置可以根据判别器的判断结果调整生成器模型参数，以提高再次生成的语音信息与真实语音的相似度。
图13示出了基于GAN网络的丢包隐藏方法训练流程图，即第二音频信号恢复模型的训练流程图。生成器由预处理模块、编码模块、解码模块组成，其中，编码模块用于提取语音信号的时域特征信息，解码模块通过编码模块提取的时域特征信息合成语音信号。在训练过程中，生成器生成语音信号后，可以将其与真实语音信号一起输入到判别器。判别器对输入的语音信号进行判决，并将判别结果反馈给生成器。生成器根据反馈结果对其模型参数进行更新，逐渐使其生成的语音信号接近真实语音信号。在训练过程中，判别器也会根据判别器损失函数对判别器的网络参数进行更新，在生成器提高语音信号生成质量的同时，判别器也可以同时提高其判别能力。
在本申请的一些实施例中，音频信号恢复装置可以使用多尺度短时傅里叶变换(Multi-Scale Short Time Fourier Transform，MS-STFT)判别器对生成器生成的语音信号和真实语音信号进行判别。实际实现中，还可以使用Multi-Period Discriminator判别器、Multi-Scale Discriminator判别器等判别器对生成器生成的语音信号和真实语音信号进行判别。本申请实施例不作具体限定。
示例性地，以采用MS-STFT判别器进行判别为例。如图14所示，MS-STFT判别器包括5个短时傅里叶变换(Short Time Fourier Transform，STFT)鉴别器。在对输入语音信号进行判别时，5个STFT鉴别器分别对输入的语音信号进行FFT变换，其对输入语音信号执行FFT变换的帧长为：[128,256,512,1024,2048]，对应的帧移为：[32,64,128,256,512]，得到输入的语音信号对应的频谱数据。然后可以将频谱数据输入判别器中进行处理，不同尺度的判别器可以从不同层次的语音结构中学习特征，从而促使生成器提高合成语音质量。
在实际应用中，STFT鉴别器通常选择复数谱作为输入，其有助于判别器学习语音中更精细的细节特征。同时，由于语音的上下文信息对丢帧语音的恢复是极其重要的，因此在STFT鉴别器中增加了空洞卷积(Dilated Convolution)来提高卷积层对语音上下文信息的提取效率，空洞卷积允许卷积增加感受野大小从而捕获更大的上下文信息，并且不同的感受野可以在序列建模中相互补充，从而提高判别器性能。
在GAN网络训练时，判别器损失函数可以通过公式(9)计算得到：
其中，LD为判别器损失函数，K为判别器个数，为生成器生成的语音信号，为输入生成的语音信号时第k个判别器的判别输出，x为真实语音，Dk(x)为输入真实语音时第k个判别器的判别输出，max(·)为取最大值操作。在GAN训练时， 可以通过最小化判别器损失函数LD来更新判别器的网络参数。
在本申请的一些实施例中，第二音频信号恢复模型中的损失函数可以为一个由多约束项构成的损失函数。在第二音频信号恢复模型训练过程中，可以通过该多约束项构成的损失函数，提高生成语音信号的质量，从而更准确的对丢失语音信号进行恢复，提高语音通话的质量。
在本申请的一些实施例中，上述多约束项构成的损失函数可以包括以下至少两项：对抗损失、多分辨率时频域信号损失、相对特征匹配损失。
在本申请的一些实施例中，上述对抗损失参数可以通过下述的公式(10)计算得到：


式(11)计算得到：
其中，为多分辨率时频域信号损失，S(n，k)表示对真实语音信号做快速傅里叶变换(Fast Fourier Transform，FFT)变换后第n帧第k个频点的频谱幅度，其FFT变换长度为K，其中K∈{26，27，28，29，210，211}，其帧移为K/4，表示对生成的合成语音信号做FFT变换后第n帧第k个频点的频谱幅度，αt-f为加权系数，其用于平衡真实语谱和合成语音谱的1-范数失真和2-范数失真的比重，‖·‖1表示计算输入变量的1-范数，‖·‖2表示计算输入变量的2-范数。
在本申请的一些实施例中，上述相对特征匹配损失参数可以通过下述的公式(12)计算得到：


成的语音信号时第k个判别器第l层的输出。mean(·)表示取均值操作。
在本申请的一些实施例中，在训练过程中，第二音频信号恢复模型中的损失函数可以通过下述的公式(13)计算得到：

失、多分辨率时频域信号损失、相对特征匹配损失的加权系数，均为常数。在第二音频信号恢复模型的训练过程中，音频信号恢复装置可以通过最小化第二音频信号恢复模型损失函数LG来更新第二音频信号恢复模型的模型参数，以提高生成的语音信号的质量。
如此，由于可以采用包含标记为正确接收的语音包和标记为丢失的语音包的训练语音样本，通过包含对抗损失、多分辨率时频域信号损失、相对特征匹配损失的多约束项损失函数，对第二音频信号恢复模型训练，得到第一音频信号恢复模型，因此可以使得训练后的第一音频信号恢复模型可以生成与真实语音较为接近的语音信号，从 而使得训练后的第一音频信号恢复模型具有较好的音频信号恢复能力。
以下以具体示例来对本申请实施例提供的音频信号恢复方法进行示例性说明。
在实际应用中，在完成对第二音频信号恢复模型的训练，得到第一音频信号恢复模型之后，可以将训练好的第一音频信号恢复模型集成到AMR-WB解码器中。当语音数据包传输发生丢包时，AMR-WB解码器可以通过本申请实施例提供的音频信号恢复方法高质量地恢复丢失的语音信号。
假设一段16kHz采样宽带语音信号通过AMR-WB编码器编码，AMR-WB编码器的编码码率设定为12.65kbit/s。假定一个IP数据包只存储1帧编码语音信号，解码端已正确接收第1、2、……、100帧，第101帧丢失。此时，解码器可以对第101帧语音信号进行恢复。AMR-WB编解码算法一帧长度为20ms，一帧信号包含320个样本点。假设缓存的大小为5帧，即存储过去5帧的解码语音信号，包括：96帧、97帧、98帧、99帧、100帧。
如图15所示，该音频信号恢复方法可以包括下述的步骤201至步骤203。
步骤201、音频信号恢复装置通过第一音频信号恢复模型中的预处理模块，将输入的5帧长度的语音信号映射为1帧固定长度的语音信号。
在本申请的一些实施例中，频信号恢复装置可以先从缓存中取出过去5帧的解码语音信号，组成一个输入张量x，其维度为torch.size([1,1,1600])，其中第1个维度为batch大小，第2个维度为channel大小，第3个维度为信号长度(1600＝320*5)。
在本申请的一些实施例中，第一音频信号恢复模型中的预处理模块可以由一个全连接层网络和“Tanh()”激活函数组成，输入信号可以依次通过全连接层网络层和非线性激活函数层，然后得到输出信号，其定义为：
nn.Linear(1600,320)
nn.Tanh()
将输入张量x通过上述预处理模块，可以将输入的5帧长度的信号映射为1帧固定长度的信号。令其输出张量为x1，其维度为torch.size([1,1,320])。
步骤202、音频信号恢复装置将1帧固定长度的语音信号输入第一音频信号恢复模型，通过第一音频信号恢复模型中的编码模块和第一音频信号恢复模型中的解码模块生成目标语音信号。
在本申请的一些实施例中，音频信号恢复装置可以将编码模块输出张量x1(其维度为torch.size([1,1,320]))通过编码模块进行处理，令其输出特征为z，其维度为torch.size([1,128,1])。然后将编码模块输出的特征z输入到解码模块进行处理，令其输出张量为x2，其维度为torch.size([1,1,320])，解码模块输出的320个样本点即为恢复的第101帧信号。
步骤203、音频信号恢复装置更新缓存。
在本申请的一些实施例中，音频信号恢复装置可以将存储的第96帧的320个样本点从缓存中移除，然后将新生成的第101帧的320个样本点存入缓存，此时缓存buffer中存储的5帧信号为97帧、98帧、99帧、100帧、101帧。
在本申请的一些实施例中，音频信号恢复装置在更新完缓存之后，可以继续开始第102帧的解码，依次循环直至所有语音信号均解码完成。
在本申请实施例提供的音频信号恢复方法中，在传输语音信号的过程中，若丢失了某段语音信号，则可以通过音频信号恢复模型中的预处理模块将缓存的至少一帧语音信号映射为固定长度的语音信号，然后通过音频信号恢复模型中的编码模块和解码模块恢复出丢失的语音信号。如此，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响。同时，还可以在完成对丢失语音信号的恢复之后，缓存恢复得到的语音信号，以使得恢复之后丢失的语音信号，从而可以提高语音通话的质量。
本申请实施例提供的音频信号恢复方法，执行主体可以为音频信号恢复装置。本申请实施例中以音频信号恢复装置执行音频信号恢复方法为例，说明本申请实施例提供的音频信号恢复装置。
图16示出了本申请实施例中涉及的音频信号恢复装置的一种可能的结构示意图。如图16所示，该音频信号恢复装置160可以包括：输入模块161和处理模块162。
其中，输入模块161，用于在第一语音信号丢失的情况下，将缓存的第二语音信 号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层；处理模块162，用于将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息；以及用于将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
在一种可能的实现方式中，上述处理模块162，具体用于：
通过编码模块中的预卷积层，提取第二语音信号的第二时域特征信息；
将第二时域特征信息经M层第一残差网络层依次处理，得到第一时域特征信息。
在一种可能的实现方式中，上述处理模块162，具体用于：
将输入第i层第一残差网络层的时域特征信息，通过第i层第一残差网络层的第一卷积层处理，得到第三时域特征信息，并通过第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到第四时域特征信息，i∈{1，……，M}；
基于第三时域特征信息和第四时域特征信息，得到第i层第一残差网络层输出的时域特征信息；
其中，第一卷积层、第二卷积层以及第三卷积层均为一维卷积层；
第一卷积层的输入通道数和输出通道数相同；
第二卷积层的卷积核与第一卷积层的卷积核不同、且第三卷积层的卷积核与第一卷积层的卷积核相同。
在一种可能的实现方式中，上述编码模块还包括M层下采样层，M层下采样层与M个第一残差网络层交替排列；M层下采样层的输出通道数按照层级从高至低依次增大；
上述处理模块162，具体用于：
将输入第x层第一残差网络层的时域特征信息，通过第x层第一残差网络层处理，得到第五时域特征信息；
将所述第五时域特征信息通过与所述第x层第一残差网络层连接的第x层下采样层输出，x∈{1，……，M}。
在一种可能的实现方式中，上述处理模块162，具体用于：
将第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息；
将第六时域特征信息输入解码模块的后卷积层处理，得到目标语音信号。
在一种可能的实现方式中，上述处理模块162，具体用于：
将输入第j层第二残差网络层的时域特征信息，通过第j层第二残差网络层的第四卷积层处理，得到第七时域特征信息，并通过第j层第二残差网络层的第五卷积层以及第六卷积层进行处理，得到第八时域特征信息，j∈{1，……，M}；
基于第七时域特征信息和第八时域特征信息，得到第j层第二残差网络输出的第六时域特征信息；
其中，第四卷积层、第五卷积层以及第六卷积层均为一维卷积层；
第四卷积层的输入通道数和输入通道数相同；
第五卷积层的卷积核与第四卷积层的卷积核不同、且第六卷积层的卷积核与第四卷积层的卷积核相同。
在一种可能的实现方式中，上述解码模块还包括M层上采样层，M层上采样层与M个第二残差网络层交替排列；M层上采样层的输出通道数按照层级从高至低依次降低；
上述处理模块162，具体用于：
将第y层上采样层输出的时域特征信息，通过与第y层上采样层连接的第y层第二残差网络层处理，得到第y层第二残差网络层输出的时域特征信息，y∈{1，……，M-1}。
在一种可能的实现方式中，上述处理模块162，还用于在第二语音信号为多帧语音信号的情况下，再将第二语音信号经M层第一残差网络层依次处理得到第一时域特征信息之前，将多帧语音信号输入第一音频信号恢复模型之后，将多帧语音信号映射为一帧语音信号；
上述处理模块162，具体用于将一帧语音信号经M层第一残差网络层依次处理， 得到第一时域特征信息。
在一种可能的实现方式中，上述装置还包括：获取模块；
获取模块，用于在输入模块161在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型之前，获取至少一组训练语音样本，每组训练语音样本包括标记为正确接收的语音包和标记为丢失的语音包；
上述处理模块162，还用于基于获取模块获取的至少一组训练语音样本，对第二音频信号恢复模型训练，得到第一音频信号恢复模型；
其中，第二音频信号恢复模型中的损失函数包括以下至少两项：对抗损失参数、多分辨率时频域信号损失参数以及相对特征匹配损失参数。
本申请实施例提供一种音频信号恢复装置，在语音通话时，若丢失了某个语音信号，则可以通过音频信号恢复模型中的编码模块的残差网络层提取缓存的丢失语音信号前或丢失语音信号后的语音信号的时域特征信息，并通过音频信号恢复模型中的解码模块的残差网络层合成编码模块提取的时域特征信息，恢复出丢失的语音信号。如此，由于可以在语音信号丢失的情况下，通过将缓存的语音信号输入音频信号恢复模型恢复出丢失的语音信号，因此在语音信号的传输过程中，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响，进而可以有效地提高语音通话的质量。
本申请实施例中的音频信号恢复装置可以是电子设备，也可以是电子设备中的部件，例如集成电路或芯片。该电子设备可以是终端，也可以为除终端之外的其他设备。示例性的，电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、移动上网装置(Mobile Internet Device，MID)、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备、机器人、可穿戴设备、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本或者个人数字助理(personal digital assistant，PDA)等，还可以为服务器、网络附属存储器(Network Attached Storage，NAS)、个人计算机(personal computer，PC)、电视机(television，TV)、柜员机或者自助机等，本申请实施例不作具体限定。
本申请实施例中的音频信号恢复装置可以为具有操作系统的装置。该操作系统可以为安卓(Android)操作系统，可以为ios操作系统，还可以为其他可能的操作系统，本申请实施例不作具体限定。
本申请实施例提供的音频信号恢复装置能够实现如上述音频信号恢复方法实施例实现的各个过程，达到相同的技术效果，为避免重复，这里不再赘述。
可选地，如图17所示，本申请实施例还提供一种电子设备1700，包括处理器1701和存储器1702，存储器1702上存储有可在所述处理器1701上运行的程序或指令，该程序或指令被处理器1701执行时实现上述音频信号恢复方法实施例的各个步骤，且能达到相同的技术效果，为避免重复，这里不再赘述。
需要说明的是，本申请实施例中的电子设备包括上述所述的移动电子设备和非移动电子设备。
图18为实现本申请实施例的一种电子设备的硬件结构示意图。
该电子设备1800包括但不限于：射频单元1801、网络模块1802、音频输出单元1803、输入单元1804、传感器1805、显示单元1806、用户输入单元1807、接口单元1808、存储器1809、以及处理器1810等部件。
本领域技术人员可以理解，电子设备1800还可以包括给各个部件供电的电源(比如电池)，电源可以通过电源管理系统与处理器1810逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。图18中示出的电子设备结构并不构成对电子设备的限定，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置，在此不再赘述。
其中，处理器1810，用于在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，第一音频信号恢复模型包括编码模块和解码模块，编码模块包括M层第一残差网络层，解码模块包括M层第二残差网络层；以及，用于将第二语音信号经M层第一残差网络层依次处理，得到第一时域特征信息；并且用于将第一时域特征信息经M层第二残差网络层依次处理，得到目标语音信号；其中，第二语音信号包括以下至少之一：第一语音信号前的X帧语音信号，第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
在一种可能的实现方式中，上述处理器1810，具体用于：
通过编码模块中的预卷积层，提取第二语音信号的第二时域特征信息；
将第二时域特征信息经M层第一残差网络层依次处理，得到第一时域特征信息。
在一种可能的实现方式中，上述处理器1810，具体用于：
将输入第i层第一残差网络层的时域特征信息，通过第i层第一残差网络层的第一卷积层处理，得到第三时域特征信息，并通过第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到第四时域特征信息，i∈{1，……，M}；
基于第三时域特征信息和第四时域特征信息，得到第i层第一残差网络层输出的时域特征信息；
其中，第一卷积层、第二卷积层以及第三卷积层均为一维卷积层；
第一卷积层的输入通道数和输出通道数相同；
第二卷积层的卷积核与第一卷积层的卷积核不同、且第三卷积层的卷积核与第一卷积层的卷积核相同。
在一种可能的实现方式中，上述编码模块还包括M层下采样层，M层下采样层与M个第一残差网络层交替排列；M层下采样层的输出通道数按照层级从高至低依次增大；
上述处理器1810，具体用于：
将输入第x层第一残差网络层的时域特征信息，通过第x层第一残差网络层处理，得到第五时域特征信息；
将第五时域特征信息通过与第x层第一残差网络层连接的第x层下采样层输出，x∈{1，……，M}。
在一种可能的实现方式中，上述处理器1810，具体用于：
将第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息；
将第六时域特征信息输入解码模块的后卷积层处理，得到目标语音信号。
在一种可能的实现方式中，上述处理器1810，具体用于：
将输入第j层第二残差网络层的时域特征信息，通过第j层第二残差网络层的第四卷积层处理，得到第七时域特征信息，并通过第j层第二残差网络层的第五卷积层以及第六卷积层进行处理，得到第八时域特征信息，j∈{1，……，M}；
基于第七时域特征信息和第八时域特征信息，得到第j层第二残差网络输出的第六时域特征信息；
其中，第四卷积层、第五卷积层以及第六卷积层均为一维卷积层；
第四卷积层的输入通道数和输入通道数相同；
第五卷积层的卷积核与第四卷积层的卷积核不同、且第六卷积层的卷积核与第四卷积层的卷积核相同。
在一种可能的实现方式中，上述解码模块还包括M层上采样层，M层上采样层与M个第二残差网络层交替排列；M层上采样层的输出通道数按照层级从高至低依次降低；
上述处理器1810，具体用于：
将第y层上采样层输出的时域特征信息，通过与第y层上采样层连接的第y层第二残差网络层处理，得到第y层第二残差网络层输出的时域特征信息，y∈{1，……，M-1}。
在一种可能的实现方式中，上述处理器1810，还用于在第二语音信号为多帧语音信号的情况下，在将第二语音信号经M层第一残差网络层依次处理得到第一时域特征信息之前，将多帧语音信号输入第一音频信号恢复模型之后，将多帧语音信号映射为一帧语音信号；
上述处理器1810，具体用于将一帧语音信号经M层第一残差网络层依次处理，得到第一时域特征信息。
在一种可能的实现方式中，上述处理器1810，还用于在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型之前，获取至少一组训练语音样本，每组训练语音样本包括标记为正确接收的语音包和标记为丢失的语音包；以及用于基于至少一组训练语音样本，对第二音频信号恢复模型训练，得到第一音频信号恢复模型；
其中，第二音频信号恢复模型中的损失函数包括以下至少两项：对抗损失参数、 多分辨率时频域信号损失参数以及相对特征匹配损失参数。
本申请实施例提供一种电子设备，在语音通话时，若丢失了某段语音信号，则可以通过音频信号恢复模型中的编码模块的残差网络层提取缓存的丢失语音信号前或丢失语音信号后的语音信号的时域特征信息，并通过音频信号恢复模型中的解码模块的残差网络层合成编码模块提取的时域特征信息，恢复出丢失的语音信号。如此，由于可以在语音信号丢失的情况下，通过将缓存的语音信号输入音频信号恢复模型恢复出丢失的语音信号，因此在语音信号的传输过程中，可以通过音频信号恢复模型恢复出丢失的语音信号，从而避免用户感知到语音数据包的丢失，降低语音数据包丢失所带来的负面影响，进而可以有效地提高语音通话的质量。
应理解的是，本申请实施例中，输入单元1804可以包括图形处理器(Graphics Processing Unit，GPU)18041和麦克风18042，图形处理器18041对在视频捕获模式或图像捕获模式中由图像捕获装置(如摄像头)获得的静态图片或视频的图像数据进行处理。显示单元1806可包括显示面板18061，可以采用液晶显示器、有机发光二极管等形式来配置显示面板18061。用户输入单元1807包括触控面板18071以及其他输入设备18072中的至少一种。触控面板18071，也称为触摸屏。触控面板18071可包括触摸检测装置和触摸控制器两个部分。其他输入设备18072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆，在此不再赘述。
存储器1809可用于存储软件程序以及各种数据。存储器1809可主要包括存储程序或指令的第一存储区和存储数据的第二存储区，其中，第一存储区可存储操作系统、至少一个功能所需的应用程序或指令(比如声音播放功能、图像播放功能等)等。此外，存储器1809可以包括易失性存储器或非易失性存储器，或者，存储器1809可以包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器
(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本申请实施例中的存储器1809包括但不限于这些和任意其它适合类型的存储器。
处理器1810可包括一个或多个处理单元；可选的，处理器1810集成应用处理器和调制解调处理器，其中，应用处理器主要处理涉及操作系统、用户界面和应用程序等的操作，调制解调处理器主要处理无线通信信号，如基带处理器。可以理解的是，上述调制解调处理器也可以不集成到处理器1810中。
本申请实施例还提供一种可读存储介质，所述可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述音频信号恢复方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
其中，所述处理器为上述实施例中所述的电子设备中的处理器。所述可读存储介质，包括计算机可读存储介质，如计算机只读存储器ROM、随机存取存储器RAM、磁碟或者光盘等。
本申请实施例另提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现上述音频信号恢复方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
应理解，本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。
本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如上述音频信号恢复方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、 物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。

Claims (23)

  1. 一种音频信号恢复方法，所述方法包括：
    在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，所述第一音频信号恢复模型包括编码模块和解码模块，所述编码模块包括M层第一残差网络层，所述解码模块包括M层第二残差网络层；
    将所述第二语音信号经所述M层第一残差网络层依次处理，得到第一时域特征信息；
    将所述第一时域特征信息经所述M层第二残差网络层依次处理，得到目标语音信号；
    其中，所述第二语音信号包括以下至少之一：所述第一语音信号前的X帧语音信号，所述第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
  2. 根据权利要求1所述的方法，其中，所述将所述第二语音信号经所述M层第一残差网络层依次处理，得到第一时域特征信息，包括：
    通过所述编码模块中的预卷积层，提取所述第二语音信号的第二时域特征信息；
    将所述第二时域特征信息经所述M层第一残差网络层依次处理，得到所述第一时域特征信息。
  3. 根据权利要求2所述的方法，其中，所述将所述第二时域特征信息经所述M层第一残差网络层依次处理，得到所述第一时域特征信息，包括：
    将输入第i层第一残差网络层的时域特征信息，通过所述第i层第一残差网络层的第一卷积层处理，得到第三时域特征信息，并通过所述第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到第四时域特征信息，i∈{1，……，M}；
    基于所述第三时域特征信息和所述第四时域特征信息，得到所述第i层第一残差网络层输出的时域特征信息；
    其中，所述第一卷积层、所述第二卷积层以及所述第三卷积层均为一维卷积层；
    所述第一卷积层的输入通道数和输出通道数相同；
    所述第二卷积层的卷积核与所述第一卷积层的卷积核不同、且所述第三卷积层的卷积核与所述第一卷积层的卷积核相同。
  4. 根据权利要求2或3所述的方法，其中，所述编码模块还包括M层下采样层，所述M层下采样层与所述M个第一残差网络层交替排列；
    所述M层下采样层的输出通道数按照层级从高至低依次增大；
    所述将所述第二时域特征信息经所述M层第一残差网络层依次处理，得到所述第一时域特征信息，包括：
    将输入第x层第一残差网络层的时域特征信息，通过所述第x层第一残差网络层处理，得到第五时域特征信息；
    将所述第五时域特征信息通过与所述第x层第一残差网络层连接的第x层下采样层输出，x∈{1，……，M}。
  5. 根据权利要求1所述的方法，其中，所述将所述第一时域特征信息经所述M层第二残差网络层依次处理，得到目标语音信号，包括：
    将所述第一时域特征信息经所述M层第二残差网络层依次处理，得到第六时域特征信息；
    将所述第六时域特征信息输入所述解码模块的后卷积层处理，得到所述目标语音信号。
  6. 根据权利要求5所述的方法，其中，所述将所述第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息，包括：
    将输入第j层第二残差网络层的时域特征信息，通过所述第j层第二残差网络层的第四卷积层处理，得到第七时域特征信息，并通过所述第j层第二残差网络层的第五卷积层以及第六卷积层进行处理，得到第八时域特征信息，j∈{1，……，M}；
    基于所述第七时域特征信息和所述第八时域特征信息，得到所述第j层第二残差网络层输出的时域特征信息；
    其中，所述第四卷积层、所述第五卷积层以及所述第六卷积层均为一维卷积层；
    所述第四卷积层的输入通道数和输出通道数相同；
    所述第五卷积层的卷积核与所述第四卷积层的卷积核不同、且所述第六卷积层的卷积核与所述第四卷积层的卷积核相同。
  7. 根据权利要求5或6所述的方法，其中，所述解码模块还包括M层上采样层，所述M层上采样层与所述M个第二残差网络层交替排列；
    所述M层上采样层的输出通道数按照层级从高至低依次降低；
    所述将所述第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息，包括：
    将第y层上采样层输出的时域特征信息，通过与所述第y层上采样层连接的所述第y层第二残差网络层处理，得到所述第y层第二残差网络层输出的时域特征信息，y∈{1，……，M}。
  8. 根据权利要求1所述的方法，其中，在所述第二语音信号为多帧语音信号的情况下，所述将所述第二语音信号经所述M层第一残差网络层依次处理，得到第一时域特征信息之前，所述方法还包括：
    将所述多帧语音信号输入所述第一音频信号恢复模型之后，将所述多帧第二语音信号映射为一帧语音信号；
    所述将所述第二语音信号经所述M层第一残差网络层依次处理，得到第一时域特征信息，包括：
    将所述一帧语音信号经所述M层第一残差网络层依次处理，得到所述第一时域特征信息。
  9. 根据权利要求1所述的方法，其中，所述在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型之前，所述方法还包括：
    获取至少一组训练语音样本，每组训练语音样本包括标记为正确接收的语音包和标记为丢失的语音包；
    基于所述至少一组训练语音样本，对第二音频信号恢复模型训练，得到所述第一音频信号恢复模型；
    其中，所述第二音频信号恢复模型中的损失函数包括以下至少两项：对抗损失参数、多分辨率时频域信号损失参数以及相对特征匹配损失参数。
  10. 一种音频信号恢复的装置，所述装置包括：输入模块和处理模块；
    所述输入模块，用于在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型，所述第一音频信号恢复模型包括编码模块和解码模块，所述编码模块包括M层第一残差网络层，所述解码模块包括M层第二残差网络层；
    所述处理模块，用于将所述第二语音信号经所述M层第一残差网络层依次处理，得到第一时域特征信息；
    所述处理模块，还用于将所述第一时域特征信息经所述M层第二残差网络层依次处理，得到目标语音信号；
    其中，所述第二语音信号包括以下至少之一：所述第一语音信号前的X帧语音信号，所述第一语音信号之后的Y帧语音信号，M，X，Y为正整数。
  11. 根据权利要求10所述的装置，其中，所述处理模块，具体用于：
    通过所述编码模块中的预卷积层，提取所述第二语音信号的第二时域特征信息；
    将所述第二时域特征信息经所述M层第一残差网络层依次处理，得到所述第一时域特征信息。
  12. 根据权利要求11所述的装置，其中，所述处理模块，具体用于：
    将输入第i层第一残差网络层的时域特征信息，通过所述第i层第一残差网络层的第一卷积层处理，得到第三时域特征信息，并通过所述第i层第一残差网络层的第二卷积层以及第三卷积层处理，得到第四时域特征信息，i∈{1，……，M}；
    基于所述第三时域特征信息和所述第四时域特征信息，得到所述第i层第一残差网络层输出的时域特征信息；
    其中，所述第一卷积层、所述第二卷积层以及所述第三卷积层均为一维卷积层；
    所述第一卷积层的输入通道数和输出通道数相同；
    所述第二卷积层的卷积核与所述第一卷积层的卷积核不同、且所述第三卷积层的卷积核与所述第一卷积层的卷积核相同。
  13. 根据权利要求11或12所述的装置，其中，所述编码模块还包括M层下采样层，所述M层下采样层与所述M个第一残差网络层交替排列；
    所述M层下采样层的输出通道数按照层级从高至低依次增大；
    所述处理模块，具体用于：
    将输入第x层第一残差网络层的时域特征信息，通过所述第x层第一残差网络层 处理，得到第五时域特征信息；
    将所述第五时域特征信息通过与所述第x层第一残差网络层连接的第x层下采样层输出，x∈{1，……，M}。
  14. 根据权利要求10所述的装置，其中，所述处理模块，具体用于：
    将所述第一时域特征信息经M层第二残差网络层依次处理，得到第六时域特征信息；
    将所述第六时域特征信息输入所述解码模块的后卷积层处理，得到所述目标语音信号。
  15. 根据权利要求14所述的装置，其中，所述处理模块，具体用于：
    将输入第j层第二残差网络层的时域特征信息，通过所述第j层第二残差网络层的第四卷积层处理，得到第七时域特征信息，并通过所述第j层第二残差网络层的第五卷积层以及第六卷积层进行处理，得到第八时域特征信息，j∈{1，……，M}；
    基于所述第七时域特征信息和所述第八时域特征信息，得到所述第j层第二残差网络输出的时域特征信息；
    其中，所述第四卷积层、所述第五卷积层以及所述第六卷积层均为一维卷积层；
    所述第四卷积层的输入通道数和输入通道数相同；
    所述第五卷积层的卷积核与所述第四卷积层的卷积核不同、且所述第六卷积层的卷积核与所述第四卷积层的卷积核相同。
  16. 根据权利要求14或15所述的装置，其中，所述解码模块还包括M层上采样层，所述M层上采样层与所述M个第二残差网络层交替排列；
    所述M层上采样层的输出通道数按照层级从高至低依次降低；
    所述处理模块，具体用于：
    将第y层上采样层输出的时域特征信息，通过与所述第y层上采样层连接的所述第y层第二残差网络层处理，得到所述第y层第二残差网络层输出的时域特征信息，y∈{1，……，M-1}。
  17. 根据权利要求10所述的装置，其中，所述处理模块，还用于在所述第二语音信号为多帧语音信号的情况下，在将所述第二语音信号经所述M层第一残差网络层依次处理得到第一时域特征信息之前，将所述多帧语音信号输入所述第一音频信号恢复模型之后，将所述多帧语音信号映射为一帧语音信号；
    所述处理模块，具体用于将所述一帧语音信号经所述M层第一残差网络层依次处理，得到所述第一时域特征信息。
  18. 根据权利要求10所述的装置，其中，所述装置还包括：获取模块；
    所述获取模块，用于在所述输入模块在第一语音信号丢失的情况下，将缓存的第二语音信号输入第一音频信号恢复模型之前，获取至少一组训练语音样本，每组训练语音样本包括标记为正确接收的语音包和标记为丢失的语音包；
    所述处理模块，还用于基于所述获取模块获取的所述至少一组训练语音样本，对第二音频信号恢复模型训练，得到所述第一音频信号恢复模型；
    其中，所述第二音频信号恢复模型中的损失函数包括以下至少两项：对抗损失参数、多分辨率时频域信号损失参数以及相对特征匹配损失参数。
  19. 一种电子设备，包括处理器和存储器，所述存储器存储可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如权利要求1至9任一项所述的音频信号恢复方法的步骤。
  20. 一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如权利要求1至9任一项所述的音频信号恢复方法的步骤。
  21. 一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如权利要求1至9任一项所述的音频信号恢复方法的步骤。
  22. 一种计算机程序产品，所述计算机程序产品被存储在非易失的存储介质中，所述计算机程序产品被至少一个处理器执行以实现如权利要求1至9任一项所述的音频信号恢复方法的步骤。
  23. 一种电子设备，所述电子设备被配置成用于执行如权利要求1至9任一项所述的音频信号恢复方法的步骤。
PCT/CN2024/103191 2023-07-05 2024-07-02 音频信号恢复方法、装置、电子设备及可读存储介质 Ceased WO2025007868A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310823707.1 2023-07-05
CN202310823707.1A CN116705040A (zh) 2023-07-05 2023-07-05 音频信号恢复方法、装置、电子设备及可读存储介质

Publications (1)

Publication Number Publication Date
WO2025007868A1 true WO2025007868A1 (zh) 2025-01-09

Family

ID=87845113

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/103191 Ceased WO2025007868A1 (zh) 2023-07-05 2024-07-02 音频信号恢复方法、装置、电子设备及可读存储介质

Country Status (2)

Country Link
CN (1) CN116705040A (zh)
WO (1) WO2025007868A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116705040A (zh) * 2023-07-05 2023-09-05 维沃移动通信有限公司 音频信号恢复方法、装置、电子设备及可读存储介质
CN118098251A (zh) * 2024-04-10 2024-05-28 北京字跳网络技术有限公司 一种音频处理方法、模型训练方法、装置、存储介质及电子设备
CN118314907A (zh) * 2024-04-15 2024-07-09 北京字跳网络技术有限公司 对丢包的音频的处理方法、装置、介质和设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101471073A (zh) * 2007-12-27 2009-07-01 华为技术有限公司 一种基于频域的丢包补偿方法、装置和系统
US20190051310A1 (en) * 2017-08-10 2019-02-14 Industry-University Cooperation Foundation Hanyang University Method and apparatus for packet loss concealment using generative adversarial network
CN113035205A (zh) * 2020-12-28 2021-06-25 阿里巴巴（中国）有限公司 音频丢包补偿处理方法、装置及电子设备
CN114866856A (zh) * 2022-05-06 2022-08-05 北京达佳互联信息技术有限公司 音频信号的处理方法、音频生成模型的训练方法及装置
CN115171707A (zh) * 2022-07-07 2022-10-11 百果园技术(新加坡)有限公司 语音流丢包补偿方法及其装置、设备、介质、产品
CN115171705A (zh) * 2022-06-01 2022-10-11 阿里巴巴云计算(北京)有限公司 一种语音丢包补偿的方法、语音通话的方法及装置
CN116705040A (zh) * 2023-07-05 2023-09-05 维沃移动通信有限公司 音频信号恢复方法、装置、电子设备及可读存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113612808B (zh) * 2021-10-09 2022-01-25 腾讯科技（深圳）有限公司 音频处理方法、相关设备、存储介质及程序产品

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101471073A (zh) * 2007-12-27 2009-07-01 华为技术有限公司 一种基于频域的丢包补偿方法、装置和系统
US20190051310A1 (en) * 2017-08-10 2019-02-14 Industry-University Cooperation Foundation Hanyang University Method and apparatus for packet loss concealment using generative adversarial network
CN113035205A (zh) * 2020-12-28 2021-06-25 阿里巴巴（中国）有限公司 音频丢包补偿处理方法、装置及电子设备
CN114866856A (zh) * 2022-05-06 2022-08-05 北京达佳互联信息技术有限公司 音频信号的处理方法、音频生成模型的训练方法及装置
CN115171705A (zh) * 2022-06-01 2022-10-11 阿里巴巴云计算(北京)有限公司 一种语音丢包补偿的方法、语音通话的方法及装置
CN115171707A (zh) * 2022-07-07 2022-10-11 百果园技术(新加坡)有限公司 语音流丢包补偿方法及其装置、设备、介质、产品
CN116705040A (zh) * 2023-07-05 2023-09-05 维沃移动通信有限公司 音频信号恢复方法、装置、电子设备及可读存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
GUAN YUNSHENG: "Research on the reconstruction method of speech time-frequency domain lost information", CHINESE MASTER'S THESES FULL-TEXT DATABASE, 1 May 2022 (2022-05-01), pages 1 - 96, XP093257537, DOI: 10.27040/d.cnki.ggzdu.2022.001094 *

Also Published As

Publication number Publication date
CN116705040A (zh) 2023-09-05

Similar Documents

Publication Publication Date Title
JP7490804B2 (ja) 非同期デコーダでエンド・ツー・エンド音声認識をストリーミングするためのシステムおよび方法
CN110648658B (zh) 一种语音识别模型的生成方法、装置及电子设备
WO2025007868A1 (zh) 音频信号恢复方法、装置、电子设备及可读存储介质
CN104966517B (zh) 一种音频信号增强方法和装置
CN112751820B (zh) 使用深度学习实现数字语音丢包隐藏
CN113450761B (zh) 一种基于变分自编码器的并行语音合成方法和装置
CN116368563B (zh) 使用深度生成网络的实时数据包丢失隐藏
CN111582020A (zh) 信号处理方法、装置、计算机设备及存储介质
Wang et al. A temporal-spectral generative adversarial network based end-to-end packet loss concealment for wideband speech transmission
WO2024055752A1 (zh) 语音合成模型的训练方法、语音合成方法和相关装置
CN114155834B (zh) 一种语音识别方法、装置、设备及存储介质
US20250364001A1 (en) Signal encoding using latent feature prediction
CN112289343B (zh) 音频修复方法、装置及电子设备和计算机可读存储介质
CN113763973B (zh) 音频信号增强方法、装置、计算机设备和存储介质
CN118800236A (zh) 音频转文本方法、装置、电子设备及可读存储介质
WO2023236961A1 (zh) 音频信号恢复方法、装置、电子设备及介质
CN110958417A (zh) 一种基于语音线索的视频通话类视频去除压缩噪声的方法
KR20250169167A (ko) 비자기회귀 디코딩을 사용한 오디오 생성
JP2023045835A (ja) 音声合成装置およびプログラム
CN116798409A (zh) 语音处理方法、存储介质及系统
CN116312494A (zh) 语音活动检测方法、装置、电子设备及可读存储介质
CN115457969A (zh) 基于人工智能的语音转换方法、装置、计算机设备及介质
CN115273803A (zh) 模型训练方法和装置、语音合成方法、设备和存储介质
CN121583261B (zh) 语音识别方法、系统和设备
Peng et al. Lmcodec2: Ultra-low bit rate codec with transformer for satellitevoice transmission

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24835349

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE