WO2025218210A1 - 对丢包的音频的处理方法、装置、介质和设备 - Google Patents

对丢包的音频的处理方法、装置、介质和设备

Info

Publication number
WO2025218210A1
WO2025218210A1 PCT/CN2024/139747 CN2024139747W WO2025218210A1 WO 2025218210 A1 WO2025218210 A1 WO 2025218210A1 CN 2024139747 W CN2024139747 W CN 2024139747W WO 2025218210 A1 WO2025218210 A1 WO 2025218210A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
audio
audio data
sample
predicted
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/139747
Other languages
English (en)
French (fr)
Inventor
徐林平
伍子谦
张德军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Publication of WO2025218210A1 publication Critical patent/WO2025218210A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/0017—Lossless audio signal coding; Perfect reconstruction of coded audio signal by transmission of coding error
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks

Definitions

  • the embodiments of the present disclosure relate to the field of audio processing technology, and more particularly to a method, apparatus, medium, and device for processing audio with packet loss.
  • audio streaming has become a crucial component of network communications.
  • audio packets can be lost during transmission due to various factors, such as network congestion, bandwidth limitations, and hardware failures. This can severely impact the quality of voice communication and degrade the user experience. Therefore, recovering from audio packet loss is an urgent problem that needs to be addressed.
  • the embodiments of the present disclosure describe a method and apparatus for processing audio with packet loss.
  • the method and apparatus can predict first predicted audio data corresponding to the first frame, and second predicted audio data corresponding to a target segment of the second frame, based on the historical audio frames of the first frame of the audio that was lost.
  • the target segment is temporally continuous with the first frame.
  • the audio data of the second frame is then reconstructed based on the second predicted audio data and the audio data of the second frame included in the audio. This ensures continuity and smoothness between the predicted first predicted audio data corresponding to the first frame and the preceding and following frames, resulting in better audio compensation.
  • a method for processing lost audio comprising: for a first frame of lost audio in audio, obtaining a pre-stored historical audio frame that is before and continuous with the first frame; based on the historical audio frame, predicting the first frame using a pre-trained neural network model to obtain first predicted audio data of the first frame, and second predicted audio data corresponding to a target segment of a second frame in the audio, where the target segment is temporally connected to the first frame; and reconstructing the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.
  • obtaining pre-stored historical audio frames that precede and are continuous with the first frame includes: if the current conditions meet preset conditions, determining to use the neural network model to perform packet loss compensation for the first frame of packet loss, wherein the preset conditions include at least one of the following: the frame preceding the first frame is a voiced frame, and the number of audio streams currently decoded by the audio decoder does not exceed a preset threshold; for the first frame of audio packet loss, obtaining pre-stored historical audio frames that precede and are continuous with the first frame of audio packet loss.
  • the neural network model can be used to perform packet loss compensation for the lost frame when the preset conditions are met.
  • reconstructing the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio includes: windowing the second predicted audio data and a target segment of the audio data of the second frame included in the audio, and then adding the resulting data to obtain audio data corresponding to the reconstructed second frame. This allows the reconstructed second frame to be more continuous and smooth with the first frame.
  • the neural network model is a time domain model, and the time domain model is trained in the following manner: the first sample history frame corresponding to the first sample frame in the first sample audio is input into the time domain model to be trained, and the time domain model to be trained predicts the first sample frame to obtain the first sample predicted audio data for the first sample frame and the first partial predicted audio data corresponding to the target segment of the next frame of the first sample frame; based on the first partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, the partial audio data corresponding to the target segment of the next frame of the first sample frame is reconstructed to obtain the first partial reconstructed audio data; the audio data of the first sample frame and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio are used as the first label, and the first sample predicted audio data and the first partial reconstructed audio data are used as the first predicted data; based on the difference loss between the first label and the first predicted data, the
  • the neural network model is a frequency domain model; and the frequency domain model is trained in the following manner: performing short-time Fourier transform (STFT) on the second sample history frame corresponding to the second sample frame in the second sample audio to obtain the historical STFT coefficient corresponding to the second sample history frame; inputting the historical STFT coefficient into the frequency domain model to be trained to obtain the first predicted STFT coefficient, wherein the first predicted STFT coefficient includes the second predicted STFT coefficient corresponding to the second sample frame and the partial predicted STFT coefficient corresponding to the partial audio data corresponding to the target segment of the next frame of the second sample frame; performing inverse short-time Fourier transform (iSTFT) based on the first predicted STFT coefficient to obtain the second predicted STFT coefficient of the second sample frame.
  • STFT short-time Fourier transform
  • the method is performed by an audio decoder, wherein, for a first frame of audio packet loss, obtaining a pre-stored historical audio frame that precedes and is continuous with the first frame includes: upon playing the first frame of audio packet loss, the audio decoder determines that the first frame is a packet loss frame and, for the first frame of audio packet loss, obtains a pre-stored historical audio frame that precedes and is continuous with the first frame; and the method further includes: the audio decoder plays the first frame based on the first predicted audio data and plays the second frame based on the reconstructed audio data of the second frame. This allows for smoother playback of the first and second frames.
  • the audio decoder includes a preset packet loss compensation algorithm; and the method further includes: if it is determined that the neural network model is not to be used, using the packet loss compensation algorithm included in the audio decoder to perform packet loss compensation on the first frame. In this way, packet loss compensation can be achieved without using the neural network model.
  • a device for processing lost audio comprising: an acquisition unit configured to, for a first frame of lost audio in the audio, acquire a pre-stored historical audio frame that is before and continuous with the first frame; a prediction unit configured to predict the first frame based on the historical audio frame using a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of the second frame in the audio, where the target segment is temporally connected to the first frame; and a reconstruction unit configured to reconstruct the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.
  • a computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above methods in the first aspect.
  • a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect.
  • an electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods in the first aspect is implemented.
  • FIG1 is a schematic diagram showing an application scenario in which an embodiment of the present disclosure can be applied.
  • FIG2 is a schematic flow chart showing a method for processing audio packet loss according to an embodiment
  • FIG3 is a schematic diagram showing an example of combining a neural network model with a packet loss compensation algorithm in an audio decoder
  • FIG4 is a schematic diagram showing an example of training a time domain model
  • FIG5 is a schematic diagram showing an example of a prediction result output by a frequency domain model to be trained
  • FIG6 is a schematic diagram showing an example of reconstructing the second frame
  • FIG7 shows a schematic block diagram of an apparatus for processing audio with packet loss according to an embodiment
  • FIG8 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application.
  • a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
  • the prompt information in response to receiving a user's active request, may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form.
  • the pop-up window may also contain a selection control for the user to select "agree” or “disagree” to provide personal information to the electronic device.
  • PLC Packet Loss Concealment
  • OPUS Silk Packet Loss Concealment
  • AI Artificial Intelligence
  • FIG. 1 illustrates a schematic diagram of an application scenario in which the embodiments of the present disclosure can be applied.
  • each frame can contain audio data for 160 samples.
  • packet loss occurs in frame x of audio S.
  • frame x previously stored historical audio frames preceding and following frame x can be retrieved from a buffer.
  • the historical audio frames corresponding to frame x include frames x-6, x-5, ..., and x-1, i.e., the six frames preceding frame x.
  • a pre-trained neural network model is used to predict frame x, obtaining first predicted audio data corresponding to frame x and second predicted audio data corresponding to a target segment of frame x+1 in the audio.
  • the target segment is temporally connected to frame x.
  • the first predicted audio data includes 160 sample points
  • the second predicted audio data includes 40 sample points.
  • x frames of audio can be played, and the data stored in the buffer can also be updated.
  • the buffered data is updated to x-5, x-4, and so on. Then, based on the second predicted audio data and the audio data of x+1 frame included in audio S, the audio data of x+1 frame is reconstructed. This ensures the continuity and smoothness of frames x and x+1.
  • a pre-stored historical audio frame that is before the first frame and continuous with the first frame is obtained.
  • the first frame is predicted by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to the target segment of the second frame in the audio.
  • the second frame is the next frame of the first frame.
  • the audio data of the second frame is reconstructed.
  • the embodiments of the present disclosure can ensure the continuity and smoothness between the first predicted audio data corresponding to the predicted first frame and the previous and next frames to obtain a better audio compensation effect.
  • FIG2 shows a flow chart of a method for processing packet loss audio according to one embodiment.
  • the method can be performed by an audio decoder.
  • the method for processing packet loss audio can include the following steps 201 to 203, specifically:
  • Step S1 If the current conditions meet the preset conditions, determine whether to use the neural network model to compensate for the first lost frame.
  • conditions can be pre-set based on actual needs to determine whether to use the neural network model to compensate for the first lost frame.
  • the preset conditions may include, but are not limited to: the previous frame of the first frame is a voiced frame, the number of audio streams currently decoded by the audio decoder does not exceed a preset threshold, etc.
  • the packet loss compensation algorithm included in the audio decoder can effectively recover the lost frame. Therefore, there is no need to call a neural network model.
  • a neural network model is required for voiced frames. Generally, the frame preceding a voiced frame is likely to be voiced as well. Therefore, when the frame preceding a lost frame is voiced, the neural network model can be used to compensate for the lost frame.
  • the audio decoder needs to decode multiple audio streams simultaneously, using a neural network model to compensate for lost frames will increase computational complexity. Therefore, the neural network model should only be used to compensate for lost frames when the number of audio streams currently being decoded by the audio decoder does not exceed a preset threshold.
  • Step S2 For the first frame of audio packet loss, obtain a pre-stored historical audio frame that is between the first frame and continuous with the first frame.
  • obtain a pre-stored historical audio frame that is before the first frame and continuous with the first frame when it is determined that the neural network model is used to compensate for packet loss in the first frame, obtain a pre-stored historical audio frame that is before the first frame and continuous with the first frame.
  • the historical audio frame corresponding to the first frame can be obtained so that the neural network model can be used to perform packet loss compensation.
  • the audio decoder may include a preset packet loss compensation algorithm
  • the above-mentioned method for processing lost audio also includes: in the case where it is determined that the neural network model is not to be used, using the packet loss compensation algorithm included in the audio decoder to compensate for packet loss on the first frame.
  • a schematic diagram illustrates an example of combining a neural network model with a packet loss compensation algorithm in an audio decoder.
  • historical audio frames used by the neural network model to predict audio data can be cached in a buffer.
  • the process shown in Figure 3 may include the following steps 301-306. Specifically: Step 301: Determine whether packet loss has occurred based on the currently received audio data. Step 302: If packet loss has not occurred, update the data required by the packet loss compensation algorithm in the audio decoder based on the received audio data. Update the data in the buffer based on the received audio data. Then, output the audio data.
  • Step 303 If packet loss has occurred, use the packet loss compensation algorithm in the audio decoder to perform packet loss compensation and obtain a first packet loss compensation result.
  • Step 304 Determine whether to use the neural network model.
  • Step 305 If not, update the data in the buffer based on the first packet loss compensation result obtained in step 303. Then, output the audio data containing the first packet loss compensation result.
  • step 306 if a neural network model is used, packet loss compensation is performed using the neural network model to obtain a second packet loss compensation result, which is then used to overwrite the first packet loss compensation result. Furthermore, corresponding data in the decoder may be updated based on the second packet loss compensation result.
  • this may include updating the decoder status, data required by the packet loss compensation algorithm, or data in the buffer. Subsequently, audio data containing the second packet loss compensation result is output.
  • This allows the neural network model to be combined with the packet loss compensation algorithm in the audio decoder to improve the efficiency and quality of packet loss compensation.
  • Step 202 Based on the historical audio frames, the first frame is predicted by a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to the target segment of the second frame in the audio.
  • a neural network model for predicting the partial audio data of the packet loss frame and its next frame can be pre-trained.
  • the neural network model can be a neural network model of various network structures. For example, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc.
  • CNN convolutional neural network
  • RNN recurrent neural network
  • LSTM long short-term memory network
  • the second frame can refer to the next frame of the first frame
  • the target segment of the second frame can refer to the previous segment of the second frame, so the target segment is connected to the first frame in time. That is to say, in addition to predicting the audio data corresponding to the packet loss frame, the neural network model can also predict the audio data of the previous segment of the next frame of the packet loss frame, which is used for smoothing with the next frame to avoid discontinuity between the predicted frame and the next frame.
  • the number of sampling points included in the target segment can be set according to actual needs. For example, if the audio sampling rate is 16kHz and the frame length is 10ms, each frame can include 160 sampling points. In this case, the target segment can be set to include 40 sampling points. If the audio sampling rate is 16kHz and the frame length is 20ms, each frame can include 320 sampling points. In this case, the target segment can be set to include 80 sampling points.
  • the neural network model may be a time-domain model, which can be trained by following steps 1 to 4: Specifically, in step 1, the first sample history frame corresponding to the first sample frame in the first sample audio is input into the time-domain model to be trained, and the time-domain model to be trained predicts the first sample frame to obtain first sample predicted audio data for the first sample frame and first partial predicted audio data corresponding to the target segment of the next frame of the first sample frame. In step 2, based on the first partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio, the partial audio data corresponding to the target segment of the next frame of the first sample frame is reconstructed to obtain first partial reconstructed audio data.
  • step 3 the audio data of the first sample frame and the partial audio data corresponding to the target segment of the next frame of the first sample frame in the first sample audio are used as the first label, and the first sample predicted audio data and the first partial reconstructed audio data are used as the first predicted data.
  • step 4 the model parameters of the time-domain model to be trained are adjusted based on the difference loss between the first label and the first predicted data. For example, with the goal of minimizing the difference loss between the first label and the first prediction data, the model parameters of the time domain model to be trained are adjusted to obtain a time domain model that meets the prediction requirements.
  • Figure 4 shows a schematic diagram of an example of training a time domain model.
  • the first sample audio can be any audio segment, and the first sample audio can include audio frames such as x-6, x-5, ..., x, x+1, ...x+5.
  • the x frame in the first sample audio is taken as the first sample frame, and the first sample history frame corresponding to the x frame can include the preset frame before the x frame.
  • the first sample history frame includes x-6, x-5, ...x-1 frames.
  • the time domain model 401 to be trained predicts the x frame to obtain the first sample predicted audio data for the x frame, and the first part of the predicted audio data corresponding to the previous segment (i.e., the previous preset number of sampling points) of the next frame (i.e., x+1 frame) of the x frame.
  • the audio data corresponding to the previous segment of the x+1 frame is reconstructed to obtain the first partial reconstructed audio data.
  • the audio data of the x frame in the first sample audio and the audio data of the previous segment of the x+1 frame can be used as the first label
  • the first sample predicted audio data for the x frame output by the time domain model to be trained 401 and the reconstructed first partial reconstructed audio data are used as the first predicted data.
  • a preset loss function can be used to calculate the difference loss between the first label and the first predicted data.
  • the model parameters of the time domain model to be trained 401 are adjusted to obtain a time domain model.
  • the difference loss can be calculated directly in the time domain.
  • the difference loss can also be calculated in the frequency domain.
  • STFT short-time Fourier transform
  • the first predicted data and the preset frames before and after the first label can be padded (for example, using the original values of the preset frames before and after for padding).
  • a short-time Fourier transform (STFT) is then performed to obtain the STFT coefficients corresponding to the first predicted data and the first label, respectively.
  • STFT short-time Fourier transform
  • the above-mentioned neural network model can also be a frequency domain model, which can be trained by the following steps one to six. Specifically: Step one, perform a short-time Fourier transform (STFT) on the second sample history frame corresponding to the second sample frame in the second sample audio to obtain the historical STFT coefficients corresponding to the second sample history frame. Step two, input the historical STFT coefficients into the frequency domain model to be trained to obtain the first predicted STFT coefficients.
  • the first predicted STFT coefficients may include the second predicted STFT coefficients corresponding to the second sample frame, and the partial predicted STFT coefficients corresponding to the partial audio data corresponding to the target segment of the next frame of the second sample frame.
  • Step three perform an inverse short-time Fourier transform (ISTFT) based on the first predicted STFT coefficients to obtain the second sample predicted audio data of the second sample frame, and the second partial predicted audio data corresponding to the target segment of the next frame of the second sample frame.
  • Step 4 Reconstruct the partial audio data corresponding to the target segment of the next frame of the second sample frame based on the second partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio, to obtain the second partial reconstructed audio data.
  • ISTFT inverse short-time Fourier transform
  • Step 5 Use the audio data of the second sample frame and the audio data corresponding to the target segment of the next frame of the second sample frame in the second sample audio as the second label, and use the second sample predicted audio data and the second partial reconstructed audio data as the second predicted data.
  • Step 6 Adjust the model parameters of the frequency domain model to be trained based on the difference loss between the second label and the second predicted data.
  • Figure 5 shows a schematic diagram of an example of the prediction result output by the frequency domain model to be trained.
  • the second sample audio can be any audio segment.
  • the second sample history frame corresponding to the second sample frame can include audio data of 1024 sampling points, and the frequency domain model to be trained can output the prediction result of the audio data of 384 sampling points corresponding to the packet loss position.
  • the 384 sampling points can include the predicted audio data of the second sample frame and the predicted audio data of the target segment of the next frame of the second sample frame.
  • the second sample history frame can first be short-time Fourier transformed to obtain the historical STFT coefficients corresponding to the second sample history frame.
  • STFT short-time Fourier transform
  • a padding operation can be performed.
  • the padding value can be 0 or some specific value.
  • the historical STFT coefficients s0-s8 can be obtained.
  • s0, s1, s7, and s8 are related to padding.
  • the historical STFT coefficients s0-s8 can be input into the frequency domain model to be trained, and the frequency domain model to be trained outputs the first predicted STFT coefficients s7-s11.
  • s7 and s8 in the first predicted STFT coefficients are used to replace s7 and s8 in the historical STFT coefficients.
  • the first predicted STFT coefficients are combined with the historical STFT coefficients to obtain s0-s11.
  • the combined result is subjected to an inverse short-time Fourier transform iSTFT to obtain the audio data of the 1024 sampling points included in the second sample historical frame, and the audio data of the 384 sampling points corresponding to the packet loss position.
  • the audio data of the 384 sampling points corresponding to the packet loss position can be used as the output prediction result.
  • Step 203 reconstructing the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.
  • the embodiment of the present disclosure since only the data of the historical audio frame corresponding to the first frame is used when predicting the audio data of the first frame, and the audio data of the next frame (i.e., the second frame) of the first frame is not used, the embodiment of the present disclosure does not need to wait until the audio data of the second frame is received before predicting the audio data of the first frame. Therefore, the embodiment of the present disclosure has no algorithmic delay. However, there may be a discontinuity between the predicted audio data of the first frame and the audio data of the next frame.
  • the embodiment of the present disclosure predicts the audio data corresponding to the target segment of the next frame of the first frame while predicting the audio data of the first frame. In this way, the audio data of the second frame can be reconstructed based on the second predicted audio data predicted by the neural network model and the audio data of the second frame included in the audio.
  • the method for processing audio with packet loss in the disclosed embodiments can treat all packet loss scenarios (e.g., single packet loss, continuous packet loss, etc.) as single packet loss tasks, and the states of the neural network model in continuous packet loss scenarios are independent of each other.
  • packet loss scenarios e.g., single packet loss, continuous packet loss, etc.
  • each single packet loss in the continuous packet loss can be treated as a separate packet loss task to call the neural network model for processing, and there is no dependency between multiple calls to the model.
  • the second predicted audio data corresponding to the target segment of the predicted second frame and the target segment of the audio data of the second frame included in the audio can be added after windowing operations, thereby obtaining the audio data corresponding to the reconstructed second frame.
  • the windowing operation can multiply the signal with a specific window function to reduce spectral leakage and correct the discontinuity of the signal.
  • window functions can be used in the windowing operation, for example, a Hanning window, a Hamming window, etc.
  • Figure 6 shows a schematic diagram of an example of reconstructing the second frame.
  • the audio sampling rate is 16kHz (kilohertz)
  • the frame length is 10ms
  • each frame can include 160 sampling points
  • the target segment includes 40 sampling points.
  • frame x is the first frame
  • frame x+1 is the second frame. That is, when the neural network model predicts, it predicts the audio data of the 160 sampling points of frame x, and also predicts the audio data of the first 40 sampling points of frame x+1 (that is, the next frame of frame x). In this example, the audio data of frame x+1 can be obtained.
  • the audio data of frame x+1 sent by the transmitting end is received, and for another example, the audio data of frame x+1 is predicted and obtained by the method of the disclosed embodiment.
  • the audio data of the first 40 sampling points of frame x+1 predicted by the previous neural network model and the audio data of the first 40 sampling points of frame x+1 obtained later can be windowed and added. That is, the overlapping parts of frame x and frame x+1 are windowed and added together to obtain the reconstructed audio data of frame x+1.
  • the aforementioned method for processing lost audio can be performed by an audio decoder.
  • step 201 can specifically include the following: when the audio decoder plays the first frame of audio, after determining that the first frame is a lost frame, for the first lost frame of audio, obtains a pre-stored historical audio frame that precedes and is continuous with the first frame.
  • step 203 can specifically include the following: the audio decoder plays the first frame based on the first predicted audio data, and plays the second frame based on the reconstructed audio data of the second frame. This allows for audio playback after packet loss compensation.
  • a pre-stored historical audio frame that is before the first frame and continuous with the first frame is obtained.
  • the first frame is predicted by a pre-trained neural network model to obtain the first predicted audio data of the first frame and the second predicted audio data corresponding to the target segment of the second frame in the audio.
  • the second frame is the next frame of the first frame.
  • the audio data of the second frame is reconstructed.
  • the embodiment of the present disclosure can ensure the continuity and smoothness between the first predicted audio data corresponding to the predicted first frame and the previous and next frames to obtain a better audio compensation effect.
  • a device for processing audio with packet loss may be deployed in an audio decoder.
  • FIG7 shows a schematic block diagram of a device for processing packet-lost audio according to an embodiment.
  • the device shown in FIG7 is used to execute the method shown in FIG2 .
  • the device 700 for processing packet-lost audio includes: an acquisition unit 701 configured to, for a first packet-lost frame in the audio, acquire a pre-stored historical audio frame that is before the first frame and continuous with the first frame.
  • a prediction unit 702 is configured to predict the first frame based on the historical audio frame using a pre-trained neural network model to obtain first predicted audio data of the first frame and second predicted audio data corresponding to a target segment of the second frame in the audio, where the target segment is temporally connected to the first frame.
  • a reconstruction unit 703 is configured to reconstruct the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.
  • the acquisition unit 701 is further configured to, when current conditions meet preset conditions, determine to use the neural network model to perform packet loss compensation for the first lost frame.
  • the preset conditions include at least one of the following: the frame preceding the first frame is a voiced frame, and the number of audio streams currently decoded by the audio decoder does not exceed a preset threshold.
  • the first lost frame of the audio a pre-stored historical audio frame preceding and continuous with the first frame is acquired.
  • the reconstruction unit 703 is further configured to perform a windowing operation on the second predicted audio data and the target segment of the audio data of the second frame included in the audio, and then add them together to obtain the audio data corresponding to the reconstructed second frame.
  • the neural network model is a time-domain model
  • the time-domain model is trained by: inputting a first sample history frame corresponding to a first sample frame in a first sample audio into the time-domain model to be trained, and having the time-domain model to be trained predict the first sample frame to obtain first sample predicted audio data for the first sample frame and first partial predicted audio data corresponding to a target segment of a next frame of the first sample frame. Based on the first partial predicted audio data and the partial audio data in the first sample audio corresponding to the target segment of the next frame of the first sample frame, the partial audio data corresponding to the target segment of the next frame of the first sample frame is reconstructed to obtain first partial reconstructed audio data.
  • the audio data in the first sample audio corresponding to the target segment of the next frame of the first sample frame is used as a first label, and the first sample predicted audio data and the first partial reconstructed audio data are used as first predicted data. Based on the difference loss between the first label and the first predicted data, the model parameters of the time-domain model to be trained are adjusted.
  • the above-mentioned neural network model is a frequency domain model; and the above-mentioned frequency domain model is trained in the following manner: performing short-time Fourier transform STFT on the second sample historical frame corresponding to the second sample frame in the second sample audio to obtain the historical STFT coefficient corresponding to the above-mentioned second sample historical frame; inputting the above-mentioned historical STFT coefficient into the frequency domain model to be trained to obtain the first predicted STFT coefficient.
  • the above-mentioned first predicted STFT coefficient includes the second predicted STFT coefficient corresponding to the above-mentioned second sample frame, and the partial predicted STFT coefficient corresponding to the partial audio data corresponding to the target segment of the next frame of the above-mentioned second sample frame; based on the above-mentioned first predicted STFT coefficient, an inverse short-time Fourier transform iSTFT is performed to obtain the second sample predicted audio data of the above-mentioned second sample frame, and the second partial predicted audio data corresponding to the target segment of the next frame of the second sample frame; based on the above-mentioned second partial predicted audio data and the partial audio data corresponding to the target segment of the next frame of the above-mentioned second sample frame in the above-mentioned second sample audio, the partial audio data corresponding to the target segment of the next frame of the above-mentioned second sample frame is reconstructed to obtain the second partial reconstructed audio data; the audio data of the above-mentioned second sample frame and the audio data
  • the device 700 is deployed in an audio decoder, and the acquisition unit 701 is further configured to, when the audio decoder plays the first frame of the audio, after determining that the first frame is a frame with packet loss, obtain a pre-stored historical audio frame that is before the first frame and continuous with the first frame for the first frame of the audio with packet loss; and the device 700 further includes: a playback unit (not shown in the figure), configured to, the audio decoder plays the first frame based on the first predicted audio data, and plays the second frame based on the reconstructed audio data of the second frame.
  • a playback unit not shown in the figure
  • the above-mentioned audio decoder includes a preset packet loss compensation algorithm; and the above-mentioned device 700 also includes: a packet loss compensation unit (not shown in the figure), configured to use the packet loss compensation algorithm included in the above-mentioned audio decoder to compensate for packet loss on the above-mentioned first frame when it is determined that the above-mentioned neural network model is not to be used.
  • a packet loss compensation unit not shown in the figure
  • the above-mentioned device embodiments correspond to the method embodiments.
  • the device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments.
  • a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed in a computer, the computer is caused to execute the method described in FIG. 2 .
  • an electronic device including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in FIG. 2 is implemented.
  • FIG8 shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiments of the present application.
  • the electronic device shown in FIG8 is only an example and should not limit the functions and scope of use of the embodiments of the present application.
  • the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803.
  • a processing device 801 e.g., a central processing unit, a graphics processing unit, etc.
  • RAM random access memory
  • Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803.
  • the processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804.
  • An input/output (I/O) interface 805 is also connected to the bus 804.
  • the following devices can be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809.
  • the communication device 809 can allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data.
  • Figure 8 shows an electronic device 800 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in Figure 8 may represent one device, or may represent multiple devices as needed.
  • an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart.
  • the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802.
  • the processing device 801 the above-mentioned functions defined in the method of the embodiment of the present application are performed.
  • An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon.
  • the computer program When the computer program is executed in a computer, the computer is caused to execute the method provided in the present disclosure.
  • the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above.
  • Computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
  • the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.
  • the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (Radio Frequency), or any suitable combination thereof.
  • the computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
  • the computer-readable medium carries one or more programs.
  • the electronic device : for a first frame of audio that has been lost, obtains a pre-stored historical audio frame that is before the first frame and continuous with the first frame; based on the historical audio frame, predicts the first frame using a pre-trained neural network model to obtain first predicted audio data for the first frame and second predicted audio data corresponding to a target segment of a second frame in the audio, where the target segment is temporally connected to the first frame; and reconstructs the audio data of the second frame based on the second predicted audio data and the audio data of the second frame included in the audio.
  • Computer program code for performing the operations of the embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages.
  • the program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or electronic device.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
  • LAN local area network
  • WAN wide area network
  • Internet service provider e.g., via the Internet using an Internet service provider

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本公开实施例提供了一种对丢包的音频的处理方法、装置、介质和设备。该方法的一具体实施方式包括：对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧；基于所述历史音频帧，通过预先训练的神经网络模型对所述第一帧进行预测，得到所述第一帧的第一预测音频数据，以及所述音频中的第二帧的目标段对应的第二预测音频数据，所述目标段在时间上与所述第一帧连接；基于所述第二预测音频数据和所述音频中包括的所述第二帧的音频数据，重构所述第二帧的音频数据。

Description

对丢包的音频的处理方法、装置、介质和设备
本申请要求2024年4月15日递交的、名称为“对丢包的音频的处理方法、装置、介质和设备”、申请号为202410452103.5的中国发明专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开实施例涉及音频处理技术领域，尤其涉及一种对丢包的音频的处理方法、装置、介质和设备。
背景技术
随着互联网技术和通讯技术的发展，音频流传输已经成为了网络通信的一个非常重要的部分。实践中，由于各种因素，比如网络拥堵、带宽限制、硬件故障等等，音频数据包在传输过程中可能会丢失，这会对语音通信的质量产生严重影响，造成用户体验下降。因此，对音频丢包进行修补恢复是亟需解决的问题。
发明内容
本公开的实施例描述了一种对丢包的音频的处理方法和装置，可以基于音频中丢包的第一帧的历史音频帧，预测第一帧对应的第一预测音频数据，以及预测第二帧的目标段对应的第二预测音频数据，其中，目标段在时间上与第一帧连续。之后，基于第二预测音频数据和音频中包括的第二帧的音频数据，重构第二帧的音频数据。由此，可以保障预测得到的第一帧对应的第一预测音频数据与前后帧之间的连续和平滑，以获得更好的音频补偿效果。
根据第一方面，提供了一种对丢包的音频的处理方法，包括：对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧；基于上述历史音频帧，通过预先训练的神经网络模型对上述第一帧进行预测，得到上述第一帧的第一预测音频数据，以及上述音频中的第二帧的目标段对应的第二预测音频数据，上述目标段在时间上与上述第一帧连接；基于上述第二预测音频数据和上述音频中包括的上述第二帧的音频数据，重构上述第二帧的音频数据。
在一个实施例中，上述对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧，包括：在当前条件满足预设条件的情况下，判断使用上述神经网络模型对丢包的第一帧进行丢包补偿，其中，上述预设条件包括以下中的至少一项：上述第一帧的前一帧为浊音帧、音频解码器当前解码的音频流数量不超过预设阈值；对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧。由此，可以在满足预设条件下，使用神经网络模型对丢包的帧进行丢包补偿。
在一个实施例中，上述基于上述第二预测音频数据和上述音频中包括的上述第二帧的音频数据，重构上述第二帧的音频数据，包括：将上述第二预测音频数据与上述音频中包括的上述第二帧的音频数据的目标段进行加窗操作之后相加，得到重构后的第二帧对应的音频数据。由此，可以使重构后的第二帧与第一帧之间更加连续、平滑。
在一个实施例中，上述神经网络模型为时域模型，上述时域模型通过以下方式训练得到：将第一样本音频中的第一样本帧对应的第一样本历史帧输入待训练时域模型，由上述待训练时域模型对上述第一样本帧进行预测，得到针对上述第一样本帧的第一样本预测音频数据，以及上述第一样本帧的下一帧的目标段对应的第一部分预测音频数据；基于上述第一部分预测音频数据和上述第一样本音频中的、上述第一样本帧的下一帧的目标段对应的部分音频数据，重构上述第一样本帧的下一帧的目标段对应的部分音频数据，得到第一部分重构音频数据；将上述第一样本音频中的、上述第一样本帧的音频数据和上述第一样本帧的下一帧的目标段对应的部分音频数据作为第一标签，将上述第一样本预测音频数据和上述第一部分重构音频数据作为第一预测数据；基于上述第一标签和上述第一预测数据的差异损失，调整上述待训练时域模型的模型参数。由此，可以训练得到时域模型，从而可以在时域进行丢包补偿。
在一个实施例中，上述神经网络模型为频域模型；以及，上述频域模型通过以下方式训练得到：将第二样本音频中的第二样本帧对应的第二样本历史帧进行短时傅里叶变换STFT，得到上述第二样本历史帧对应的历史STFT系数；将上述历史STFT系数输入待训练频域模型，得到第一预测STFT系数，其中，上述第一预测STFT系数包括上述第二样本帧对应的第二预测STFT系数、以及上述第二样本帧的下一帧的目标段对应的部分音频数据对应的部分预测STFT系数；基于上述第一预测STFT系数进行逆短时傅里叶变换iSTFT，得到上述第二样本帧的第二样本预测音频数据，以及第二样本帧的下一帧的目标段对应的第二部分预测音频数据；基于上述第二部分预测音频数据和上述第二样本音频中的、上述第二样本帧的下一帧的目标段对应的部分音频数据，重构上述第二样本帧的下一帧的目标段对应的部分音频数据，得到第二部分重构音频数据；将上述第二样本音频中的、上述第二样本帧的音频数据和上述第二样本帧的下一帧的目标段对应的音频数据作为第二标签，将上述第二样本预测音频数据和上述第二部分重构音频数据作为第二预测数据；基于上述第二标签和上述第二预测数据的差异损失，调整上述待训练频域模型的模型参数。由此，可以训练得到频域模型，从而可以在频域进行丢包补偿。
在一个实施例中，上述方法由音频解码器执行，上述对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧，包括：上述音频解码器在播放到上述音频的上述第一帧时，在确定上述第一帧为丢包的帧之后，对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧；以及上述方法还包括：上述音频解码器基于上述第一预测音频数据播放上述第一帧，基于上述第二帧的重构的音频数据播放上述第二帧。由此，可以使播放的第一帧和第二帧更加平滑。
在一个实施例中，上述音频解码器中包括预设的丢包补偿算法；以及，上述方法还包括：在判断不使用上述神经网络模型的情况中，使用上述音频解码器包括的丢包补偿算法对上述第一帧进行丢包补偿。由此，可以在不使用神经网络模型的情况下，实现丢包补偿。
根据第二方面，提供了一种对丢包的音频的处理装置，包括：获取单元，配置为，对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧；预测单元，配置为，基于上述历史音频帧，通过预先训练的神经网络模型对上述第一帧进行预测，得到上述第一帧的第一预测音频数据，以及上述音频中的第二帧的目标段对应的第二预测音频数据，上述目标段在时间上与上述第一帧连接；重构单元，配置为，基于上述第二预测音频数据和上述音频中包括的上述第二帧的音频数据，重构上述第二帧的音频数据。
根据第三方面，提供了一种计算机程序产品，包括计算机程序，上述计算机程序被处理器执行时实现如第一方面中任一项上述的方法。
根据第四方面，提供了一种计算机可读存储介质，其上存储有计算机程序，当上述计算机程序在计算机中执行时，令上述计算机执行第一方面中任一项上述的方法。
根据第五方面，提供了一种电子设备，包括存储器和处理器，上述存储器中存储有可执行代码，上述处理器执行上述可执行代码时，实现第一方面中任一项上述的方法。
附图说明
图1示出了本公开实施例可以应用于其中的一个应用场景的示意图；
图2示出了根据一个实施例的对丢包的音频的处理方法的流程示意图；
图3示出了将神经网络模型与音频解码器中的丢包补偿算法结合使用的一个例子的示意图；
图4示出了一个训练时域模型的例子的示意图；
图5示出了待训练频域模型输出预测结果的一个例子的示意图；
图6示出了重构第二帧的一个例子的示意图；
图7示出了根据一个实施例的对丢包的音频的处理装置的示意性框图；
图8示出了适于用来实现本申请实施例的电子设备的结构示意图。
具体实施方式
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当依据相关法律法规通过恰当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限定性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或者“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其它满足相关法律法规的方式也可应用于本公开的实现方式中。
下面结合附图和实施例，对本公开提供的技术方案做进一步的详细描述。可以理解的是，此处所描述的具体实施例仅仅用于解释相关发明，而非对该发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关发明相关的部分。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
如前所述，对音频丢包进行修补恢复是亟需解决的问题。现阶段，在实时通信场景下，接收端音频解码器中的PLC(Packet Loss Concealment，丢包隐藏)技术。例如，OPUS Silk中的PLC算法，通常可以在40ms(毫秒)短丢包场景下恢复较高的音频质量。但是在更长丢包场景(例如，60-100ms连续丢包)下恢复音频的能量衰减显著，质量下降显著。随着AI(Artificial Intelligence，人工智能)技术的发展，在有些方式中，可以采用AI模型实现PLC的相关研究。目前AIPLC模型多采用连续状态模型，在未丢包场景下模型状态也需要更新，以确保丢包时输出恢复音频的质量。但是，在实际应用中，音频解码器中PLC算法的调用概率偏低，连续状态模型会引入大量额外计算复杂度，且部分PLC模型会引入一定的算法延时(通常为20ms左右)，与当前的音频链路并不适配，难以落地应用。
为此，本公开的实施例提供了一种对丢包的音频的处理方法，可以基于丢包的音频帧的历史音频帧进行音频预测，可以保障预测得到的音频帧对应的预测音频数据与前后帧之间的连续和平滑，以获得更好的音频补偿效果。图1示出了本公开实施例可以应用于其中的一个应用场景的示意图。如图1所示，在图1所示的应用场景中，以音频的采样率为16kHz(千赫兹)，帧长为10ms为例，每帧中可以包括160个采样点的音频数据。假设音频S中的x帧发生了丢包，对于x帧，可以从buffer(缓冲器)中获取预先存储的x帧之前的、且与x帧连续的历史音频帧。本例中，x帧对应的历史音频帧包括x-6、x-5……x-1帧，即x帧之前的6帧。然后，基于历史音频帧，通过预先训练的神经网络模型对x帧进行预测，得到x帧对应的第一预测音频数据，以及音频中的x+1帧的目标段对应的第二预测音频数据。这里，目标段在时间上与x帧连接。本例中，第一预测音频数据包括160个采样点数据，第二预测音频数据包括40个采样点。基于第一预测音频数据，可以播放x帧的音频，还可以更新buffer中存储的数据。即，将buffer中缓存的数据更新为x-5、x-4……x帧。然后，基于第二预测音频数据和音频S中包括的x+1帧的音频数据，重构x+1帧的音频数据。由此，可以保障x帧和x+1帧的连续和平滑。
根据本公开实施例提供的对丢包的音频的处理方法和装置，对于音频中的丢包的第一帧，获取预存的、在该第一帧之前的且与第一帧连续的历史音频帧。之后，基于历史音频帧，通过预先训练的神经网络模型对第一帧进行预测，得到第一帧的第一预测音频数据，以及音频中的第二帧的目标段对应的第二预测音频数据。第二帧为第一帧的下一帧。最后，基于第二预测音频数据和音频中包括的第二帧的音频数据，重构第二帧的音频数据。由于，第一预测音频数据和第二预测音频数据是基于历史音频帧预测得到的，因此可以保障第一帧与前帧的连续和平滑，而基于第二预测音频数据重构第二帧的音频数据，可以保障第一帧与后帧的连续和平滑。因此，本公开的实施例可以保障预测得到的第一帧对应的第一预测音频数据与前后帧之间的连续和平滑，以获得更好的音频补偿效果。
继续参见图2，图2示出了根据一个实施例的对丢包的音频的处理方法的流程示意图。该方法可以通过音频解码器来执行。如图2所示，该对丢包的音频的处理方法，可以包括以下步骤201至步骤203，具体的：
步骤201，对于音频中的丢包的第一帧，获取预先存储的、在第一帧之前的且与第一帧连续的历史音频帧。在本实施例中，可以预先存储已有的(全部或者部分)音频帧。例如，可以预先在buffer(缓冲器)中缓存已有的部分音频帧。这样，对于音频中丢包的第一帧，可以从buffer中获取在第一帧之前的、且与第一帧连续的历史音频帧。这里，第一帧可以是指音频中丢包的帧。
在一些实现方式中，音频解码器中可以包括预设的丢包补偿算法。例如，以OPUS中的Silk解码器为例，Silk解码器中可以包括PLC算法。这里，可以预先训练用于对丢包帧及其下一帧的部分音频数据进行预测的神经网络模型，并将该神经网络模型与音频解码器中的丢包补偿算法结合使用。此时，上述步骤201可以包括以下步骤S1和步骤S2，具体的：
步骤S1，在当前条件满足预设条件的情况下，判断使用神经网络模型对丢包的第一帧进行丢包补偿。在本实现方式中，可以根据实际需要预先设定条件，以判断使用神经网络模型对丢包的第一帧进行丢包补偿。作为示例，预设条件可以包括但不限于：第一帧的前一帧为浊音帧、音频解码器当前解码的音频流数量不超过预设阈值等等。
实践中，当丢包帧为清音帧或者静音帧时，音频解码器中包括的丢包补偿算法可以很好的恢复丢包帧。因此，不需要调用神经网络模型，而浊音帧需要调用神经网络模型。通常，浊音帧的前一帧大概率也为浊音帧，所以丢包帧的前一帧为浊音帧时，可以使用神经网络模型对丢包帧进行丢包补偿。
此外，在音频解码器需要同时解码多路音频流的情况中，如果使用神经网络模型对丢包帧进行丢包补偿，计算复杂度会变高。因此，在音频解码器当前解码的音频流数量不超过预设阈值的情况下，才可以使用神经网络模型对丢包帧进行丢包补偿。
步骤S2，对于音频中的丢包的第一帧，获取预先存储的、在第一帧之间的且与第一帧连续的历史音频帧。在本实现方式中，在判断使用神经网络模型对丢包的第一帧进行丢包补偿的情况下，获取预存的第一帧之前的且与第一帧连续的历史音频帧。通过本实现方式，可以在当前条件满足预设条件的情况下，获取第一帧对应的历史音频帧，以便调用神经网络模型进行丢包补偿。
在另一些实现方式中，音频解码器中可以包括预设的丢包补偿算法，以及上述对丢包的音频的处理方法还包括：在判断不使用神经网络模型的情况中，使用音频解码器包括的丢包补偿算法对第一帧进行丢包补偿。
如图3所示，图3示出了将神经网络模型与音频解码器中的丢包补偿算法结合使用的一个例子的示意图。图3所示的例子中，可以在buffer(缓冲器)中缓存神经网络模型预测音频数据时所使用的历史音频帧。基于此，图3所示的过程可以包括以下步骤301-步骤306，具体的：步骤301，根据当前接收的音频数据判断音频是否发生丢包。步骤302，如果未发生丢包，则根据接收的音频数据更新音频解码器中的丢包补偿算法所需的数据。根据接收的音频数据更新buffer中的数据。之后，输出音频数据。步骤303，如果发生丢包，使用音频解码器中的丢包补偿算法进行丢包补偿，得到第一丢包补偿结果。步骤304，判断是否使用神经网络模型。步骤305，如果不使用神经网络模型，则基于步骤303得到的第一丢包补偿结果更新buffer中的数据。之后，输出包含有第一丢包补偿结果的音频数据。步骤306，如果使用神经网络模型，则使用神经网络模型进行丢包补偿，得到第二丢包补偿结果，并使用第二丢包补偿结果重写第一丢包补偿结果。此外，还可以根据第二丢包补偿结果更新解码器的对应的一些数据。例如，更新解码器的状态、更新丢包补偿算法所需的数据、更新buffer中的数据等等。之后，输出包含有第二丢包补偿结果的音频数据。由此，可以将神经网络模型与音频解码器中的丢包补偿算法结合使用，提高丢包补偿的效率和质量。
步骤202，基于历史音频帧，通过预先训练的神经网络模型对第一帧进行预测，得到第一帧的第一预测音频数据，以及音频中的第二帧的目标段对应的第二预测音频数据。在本实施例中，可以预先训练用于对丢包帧及其下一帧的部分音频数据进行预测的神经网络模型，该神经网络模型可以是各种网络结构的神经网络模型。例如，卷积神经网络(Convolutional Neural Network,CNN)、递归神经网络(Recurrent Neural Network,RNN)、长短期记忆网络(Long Short-Term Memory,LSTM)等等。神经网络模型可以基于历史音频帧对第一帧进行预测，得到第一帧的第一预测音频数据，以及第二帧的目标段对应的第二预测音频数据。这里，第二帧可以是指第一帧的下一帧，第二帧的目标段可以是指第二帧的前一段，所以，目标段在时间上与第一帧连接。也就是说，神经网络模型除了预测出丢包帧对应的音频数据之外，还可以预测出丢包帧的下一帧的前一段的音频数据，用于与下一帧进行平滑，以避免在预测帧与下一帧之间出现不连续。
这里，目标段包括的采样点的数量可以根据实际需要进行设定。举例来说，以音频的采样率为16kHz(千赫兹)，帧长为10ms为例，则每帧中可以包括160个采样点。此时，可以设定目标段包括40个采样点。以音频的采样率为16kHz(千赫兹)，帧长为20ms为例，则每帧中可以包括320个采样点，此时，可以设定目标段包括80个采样点。
在一些实现方式中，上述神经网络模型可以是时域模型，该时域模型可以通过以下步骤1至步骤4训练得到，具体的：步骤1，将第一样本音频中的第一样本帧对应的第一样本历史帧输入待训练时域模型，由待训练时域模型对第一样本帧进行预测，得到针对第一样本帧的第一样本预测音频数据，以及第一样本帧的下一帧的目标段对应的第一部分预测音频数据。步骤2，基于第一部分预测音频数据和第一样本音频中的、第一样本帧的下一帧的目标段对应的部分音频数据，重构第一样本帧的下一帧的目标段对应的部分音频数据，得到第一部分重构音频数据。步骤3，将第一样本音频中的、第一样本帧的音频数据和第一样本帧的下一帧的目标段对应的部分音频数据作为第一标签，将第一样本预测音频数据和第一部分重构音频数据作为第一预测数据。步骤4，基于第一标签和第一预测数据的差异损失，调整待训练时域模型的模型参数。例如，以第一标签和第一预测数据的差异损失最小化为目标，调整待训练时域模型的模型参数，从而得到满足预测需求的时域模型。
如图4所示，图4示出了一个训练时域模型的例子的示意图。在图4所示的例子中，第一样本音频可以是任意的一段音频，第一样本音频可以包括x-6、x-5、……、x、x+1、……x+5等音频帧。本例中，将第一样本音频中的x帧作为第一样本帧，x帧对应的第一样本历史帧可以包括x帧之前的预设帧。本例中，第一样本历史帧包括x-6、x-5、……x-1帧。将x帧对应的第一样本历史帧输入待训练时域模型401之后，由待训练时域模型401对x帧进行预测，得到针对x帧的第一样本预测音频数据，以及x帧的下一帧(即，x+1帧)的前一段(即，前预设数量个采样点)对应的第一部分预测音频数据。
之后，基于第一部分预测音频数据和第一样本音频中的x+1帧的前一段对应的部分音频数据，重构x+1帧的前一段对应的音频数据，得到第一部分重构音频数据。此时，可以将第一样本音频中的x帧的音频数据和x+1帧的前一段的音频数据作为第一标签，将待训练时域模型401输出的针对x帧的第一样本预测音频数据和重构得到的第一部分重构音频数据作为第一预测数据。最后，可以使用预设的损失函数，计算第一标签和第一预测数据之间的差异损失。以差异损失最小化为目标，调整待训练时域模型401的模型参数，得到时域模型。作为一个示例，可以在时域直接计算差异损失。作为另一个示例，还可以在频域计算差异损失。举例来说，为了考虑帧间连续性，在进行短时傅里叶变换(short-time Fourier transform，STFT)之前，可以将第一预测数据和第一标签的前后预设帧进行padding(填充)操作(例如，使用前后预设帧的原始值进行填充)。之后进行短时傅里叶变换，分别得到第一预测数据和第一标签对应的STFT系数，然后计算两个STFT系数之间的差异损失。经研究分析发现，在STFT域上，预测x帧与x-1帧之间不连续差异表现明显。而在训练中降低该差异，可以保证预测音频准确性的同时保证x帧与x-1帧之间的连续性。
在另一些实现方式中，上述神经网络模型还可以是频域模型，该频域模型可以通过以下步骤一至步骤六训练得到，具体的：步骤一，将第二样本音频中的第二样本帧对应的第二样本历史帧进行短时傅里叶变换(short-time Fourier transform，STFT)，得到第二样本历史帧对应的历史STFT系数。步骤二，将历史STFT系数输入待训练频域模型，得到第一预测STFT系数。这里，第一预测STFT系数可以包括第二样本帧对应的第二预测STFT系数、以及第二样本帧的下一帧的目标段对应的部分音频数据对应的部分预测STFT系数。步骤三，基于第一预测STFT系数进行逆短时傅里叶变换(Inverse Short-Time Fourier Transform，ISTFT)，得到第二样本帧的第二样本预测音频数据，以及第二样本帧的下一帧的目标段对应的第二部分预测音频数据。步骤四，基于第二部分预测音频数据和第二样本音频中的、第二样本帧的下一帧的目标段对应的部分音频数据，重构第二样本帧的下一帧的目标段对应的部分音频数据，得到第二部分重构音频数据。步骤五，将第二样本音频中的、第二样本帧的音频数据和第二样本帧的下一帧的目标段对应的音频数据作为第二标签，将第二样本预测音频数据和第二部分重构音频数据作为第二预测数据。步骤六，基于第二标签和第二预测数据的差异损失，调整待训练频域模型的模型参数。
如图5所示，图5示出了待训练频域模型输出预测结果的一个例子的示意图。在图5所示的例子中，第二样本音频可以是任意的一段音频。图5所示的例子中，第二样本帧对应的第二样本历史帧可以包括1024个采样点的音频数据，待训练频域模型可以输出丢包位置对应的384个采样点的音频数据的预测结果。可以理解，该384个采样点可以包括第二样本帧的预测音频数据和第二样本帧的下一帧的目标段的预测音频数据。
本例中，首先可以将第二样本历史帧进行短时傅里叶变换，得到第二样本历史帧对应的历史STFT系数。可以理解，在短时傅里叶变换STFT中，为了处理信号边界的问题，确保每个窗口能完整进行傅里叶变换，并减少由于窗口边界不连续导致的频谱泄露，可以进行padding(填充)操作。例如，填充的数值可以是0，也可以是某些特定值。本例中，短时傅里叶变换STFT的参数可以设置如下：fft(快速傅里叶变换)点数n_fft＝512，窗长window＝n_fft，窗移hop_size＝n_fft/4，padding＝n_fft/2。也就是说，本例中，可以对第二样本历史帧之前的256个采样点和之后的256个采样点进行填充。通过短时傅里叶变换STFT之后，可以得到历史STFT系数s0-s8。在短时傅里叶变换STFT时，s0、s1、s7、s8与padding相关。
之后，可以将历史STFT系数s0-s8输入待训练频域模型，由待训练频域模型输出第一预测STFT系数s7-s11。在进行逆短时傅里叶变换iSTFT时，使用第一预测STFT系数中的s7和s8替换历史STFT系数中的s7和s8。将第一预测STFT系数与历史STFT系数相结合，结合得到s0-s11。然后，对结合结果进行逆短时傅里叶变换iSTFT，得到第二样本历史帧包括的1024个采样点的音频数据，以及丢包位置对应的384个采样点的音频数据。丢包位置对应的384个采样点的音频数据，可以作为输出的预测结果。
可以理解，图5所示例子中的样本历史帧的采样点数量、预测结果的采样点数量等等仅仅是示意性的，而非限定。
步骤203，基于第二预测音频数据和音频中包括的第二帧的音频数据，重构第二帧的音频数据。在本实施例中，由于在预测第一帧的音频数据时，仅使用了第一帧对应的历史音频帧的数据，而未使用第一帧的下一帧(即，第二帧)的音频数据，所以本公开实施例不需要等到接收第二帧的音频数据之后再预测第一帧的音频数据。因此，本公开实施例无算法延时。但是，预测得到的第一帧的音频数据与下一帧的音频数据之间可能会存在不连续。为了缓解与下一帧之间的不连续，本公开实施例在预测第一帧的音频数据的同时，还预测了第一帧的下一帧的目标段对应的音频数据。这样，可以根据神经网络模型预测得到的第二预测音频数据和音频中包括的第二帧的音频数据，重构第二帧的音频数据。
可以理解，本公开实施例的对丢包的音频的处理方法可以将所有丢包情况(例如，单丢包、连续丢包等)视为单丢包任务，连续丢包场景神经网络模型的状态相互独立。举例来说，在连续丢包场景中，可以将连续丢包中的每个单丢包作为一个单独的丢包任务去调用神经网络模型进行处理，而模型的多次调用之间不存在依赖关系。
在一些实现方式中，上述步骤203可以具体如下进行：将第二预测音频数据与音频中包括的第二帧的音频数据的目标段进行加窗操作之后相加，得到重构后的第二帧对应的音频数据。
在本实现方式中，可以将预测得到的第二帧的目标段对应的第二预测音频数据与音频中包括的第二帧的音频数据的目标段，分别加窗操作之后相加，从而得到重构后的第二帧对应的音频数据。由此，可以缓解第一帧与第二帧之间的不连续。这里，加窗操作可以将信号与特定的窗函数相乘，以此减少频谱泄露并修正信号的不连续性。加窗操作中可以使用多种窗函数，例如，汉宁窗、汉明窗等等。
如图6所示，图6示出了重构第二帧的一个例子的示意图。在图6所示的例子中，音频的采样率为16kHz(千赫兹)，帧长为10ms，每帧中可以包括160个采样点，目标段包括40个采样点。图6中，x帧为第一帧，x+1帧为第二帧。即，神经网络模型在预测时，即预测了x帧的160个采样点的音频数据，还预测了x+1帧(即，x帧的下一帧)的前40个采样点的音频数据。本例中可以获取x+1帧的音频数据。例如，接收发送端发送的x+1帧的音频数据，又例如，通过公开实施例的方式预测获取x+1帧的音频数据。此时，可以将之前神经网络模型所预测的x+1帧的前40个采样点的音频数据，与之后获取的x+1帧的前40个采样点的音频数据，分别进行加窗操作之后相加。即，将x帧和x+1帧的重叠部分加窗后相加，得到重构后的x+1帧的音频数据。
在一些实现方式中，上述对丢包的音频的处理方法可以由音频解码器执行。基于此，上述步骤201可以具体包括以下内容：音频解码器在播放到音频的第一帧时，在确定第一帧为丢包的帧之后，对于音频中的丢包的第一帧，获取预先存储的、在第一帧之前的且与第一帧连续的历史音频帧。基于此，上述步骤203可以具体包括以下内容：音频解码器基于第一预测音频数据播放第一帧，基于第二帧的重构的音频数据播放第二帧。由此，可以实现丢包补偿后的音频播放。
回顾以上过程，在本公开的上述实施例中，对于音频中的丢包的第一帧，获取预存的、在该第一帧之前的且与第一帧连续的历史音频帧。之后，基于历史音频帧，通过预先训练的神经网络模型对第一帧进行预测，得到第一帧的第一预测音频数据，以及音频中的第二帧的目标段对应的第二预测音频数据。第二帧为第一帧的下一帧。最后，基于第二预测音频数据和音频中包括的第二帧的音频数据，重构第二帧的音频数据。由于，第一预测音频数据和第二预测音频数据是基于历史音频帧预测得到的，因此可以保障第一帧与前帧的连续和平滑。而基于第二预测音频数据重构第二帧的音频数据，可以保障第一帧与后帧的连续和平滑。因此，本公开的实施例可以保障预测得到的第一帧对应的第一预测音频数据与前后帧之间的连续和平滑，以获得更好的音频补偿效果。
根据另一方面的实施例，提供了一种对丢包的音频的处理装置。上述对丢包的音频的处理装置可以部署于音频解码器中。
图7示出了根据一个实施例的对丢包的音频的处理装置的示意性框图。图7所示装置用于执行图2所示方法。如图7所示，该对丢包的音频的处理装置700包括：获取单元701，配置为，对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧。预测单元702，配置为，基于上述历史音频帧，通过预先训练的神经网络模型对上述第一帧进行预测，得到上述第一帧的第一预测音频数据，以及上述音频中的第二帧的目标段对应的第二预测音频数据，上述目标段在时间上与上述第一帧连接。重构单元703，配置为，基于上述第二预测音频数据和上述音频中包括的上述第二帧的音频数据，重构上述第二帧的音频数据。
在本实施例的一些可选的实现方式中，获取单元701进一步配置为，在当前条件满足预设条件的情况下，判断使用上述神经网络模型对丢包的第一帧进行丢包补偿。上述预设条件包括以下中的至少一项：上述第一帧的前一帧为浊音帧、音频解码器当前解码的音频流数量不超过预设阈值。对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧。
在本实施例的一些可选的实现方式中，重构单元703进一步配置为，将上述第二预测音频数据与上述音频中包括的上述第二帧的音频数据的目标段进行加窗操作之后相加，得到重构后的第二帧对应的音频数据。
在本实施例的一些可选的实现方式中，上述神经网络模型为时域模型，上述时域模型通过以下方式训练得到：将第一样本音频中的第一样本帧对应的第一样本历史帧输入待训练时域模型，由上述待训练时域模型对上述第一样本帧进行预测，得到针对上述第一样本帧的第一样本预测音频数据，以及上述第一样本帧的下一帧的目标段对应的第一部分预测音频数据。基于上述第一部分预测音频数据和上述第一样本音频中的、上述第一样本帧的下一帧的目标段对应的部分音频数据，重构上述第一样本帧的下一帧的目标段对应的部分音频数据，得到第一部分重构音频数据。将上述第一样本音频中的、上述第一样本帧的音频数据和上述第一样本帧的下一帧的目标段对应的部分音频数据作为第一标签，将上述第一样本预测音频数据和上述第一部分重构音频数据作为第一预测数据。基于上述第一标签和上述第一预测数据的差异损失，调整上述待训练时域模型的模型参数。
在本实施例的一些可选的实现方式中，上述神经网络模型为频域模型；以及，上述频域模型通过以下方式训练得到：将第二样本音频中的第二样本帧对应的第二样本历史帧进行短时傅里叶变换STFT，得到上述第二样本历史帧对应的历史STFT系数；将上述历史STFT系数输入待训练频域模型，得到第一预测STFT系数。上述第一预测STFT系数包括上述第二样本帧对应的第二预测STFT系数、以及上述第二样本帧的下一帧的目标段对应的部分音频数据对应的部分预测STFT系数；基于上述第一预测STFT系数进行逆短时傅里叶变换iSTFT，得到上述第二样本帧的第二样本预测音频数据，以及第二样本帧的下一帧的目标段对应的第二部分预测音频数据；基于上述第二部分预测音频数据和上述第二样本音频中的、上述第二样本帧的下一帧的目标段对应的部分音频数据，重构上述第二样本帧的下一帧的目标段对应的部分音频数据，得到第二部分重构音频数据；将上述第二样本音频中的、上述第二样本帧的音频数据和上述第二样本帧的下一帧的目标段对应的音频数据作为第二标签，将上述第二样本预测音频数据和上述第二部分重构音频数据作为第二预测数据；基于上述第二标签和上述第二预测数据的差异损失，调整上述待训练频域模型的模型参数。
在本实施例的一些可选的实现方式中，装置700部署于音频解码器，获取单元701进一步配置为，上述音频解码器在播放到上述音频的上述第一帧时，在确定上述第一帧为丢包的帧之后，对于音频中的丢包的第一帧，获取预先存储的、在上述第一帧之前的且与上述第一帧连续的历史音频帧；以及，上述装置700还包括：播放单元(图中未示出)，配置为，上述音频解码器基于上述第一预测音频数据播放上述第一帧，基于上述第二帧的重构的音频数据播放上述第二帧。
在本实施例的一些可选的实现方式中，上述音频解码器中包括预设的丢包补偿算法；以及，上述装置700还包括：丢包补偿单元(图中未示出)，配置为，在判断不使用上述神经网络模型的情况中，使用上述音频解码器包括的丢包补偿算法对上述第一帧进行丢包补偿。
上述装置实施例与方法实施例相对应，具体说明可以参见方法实施例部分的描述，此处不再赘述。装置实施例是基于对应的方法实施例得到，与对应的方法实施例具有同样的技术效果，具体说明可参见对应的方法实施例。
根据另一方面的实施例，还提供一种计算机可读存储介质，其上存储有计算机程序，当上述计算机程序在计算机中执行时，令计算机执行图2所描述的方法。
根据再一方面的实施例，还提供一种电子设备，包括存储器和处理器，其特征在于，上述存储器中存储有可执行代码，上述处理器执行上述可执行代码时，实现图2所描述的方法。
上述内容对本公开的特定实施例进行了描述，其他实施例在所附权利要求书的范围内。在一些情况下，在权利要求书中记载的动作或步骤可以按照不同于实施例中的顺序来执行，并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要按照示出的特定顺序或者连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的，或者可能是有利的。
下面参考图8，其示出了适于用来实现本申请实施例的电子设备800的结构示意图。图8示出的电子设备仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图8所示，电子设备800可以包括处理装置(例如中央处理器、图形处理器等)801，其可以根据存储在只读存储器(ROM)802中的程序或者从存储装置808加载到随机访问存储器(RAM)803中的程序而执行各种适当的动作和处理。在RAM803中，还存储有电子设备800操作所需的各种程序和数据。处理装置801、ROM802以及RAM803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。
通常，以下装置可以连接至I/O接口805：包括例如触摸屏、触摸板、键盘、鼠标等的输入装置806；包括例如液晶显示器(LCD，Liquid Crystal Display)、扬声器、振动器等的输出装置807；包括例如磁带、硬盘等的存储装置808；以及通信装置809。通信装置809可以允许电子设备800与其他设备进行无线或有线通信以交换数据。虽然图8示出了具有各种装置的电子设备800，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。图8中示出的每个方框可以代表一个装置，也可以根据需要代表多个装置。
特别地，根据本申请的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本申请的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置809从网络上被下载和安装，或者从存储装置808被安装，或者从ROM802被安装。在该计算机程序被处理装置801执行时，执行本申请的实施例的方法中限定的上述功能。
本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令计算机执行本公开所提供的所述方法。
需要说明的是，本公开的实施例所述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD□ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开的实施例中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开的实施例中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(Radio Frequency，射频)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧；基于所述历史音频帧，通过预先训练的神经网络模型对所述第一帧进行预测，得到所述第一帧的第一预测音频数据，以及所述音频中的第二帧的目标段对应的第二预测音频数据，所述目标段在时间上与所述第一帧连接；基于所述第二预测音频数据和所述音频中包括的所述第二帧的音频数据，重构所述第二帧的音频数据。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开实施例的操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或电子设备上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
本公开中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于存储介质和计算设备实施例而言，由于其基本相似于方法实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本公开实施例所描述的功能可以用硬件、软件、固件或它们的任意组合来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。
以上所述的具体实施方式，对本公开实施例的目的、技术方案和有益效果进行了进一步的详细说明。所应理解的是，以上所述仅为本公开实施例的具体实施方式而已，并不用于限定本公开的保护范围，凡在本公开的技术方案的基础之上所做的任何修改、等同替换、改进等，均应包括在本公开的保护范围之内。

Claims (10)

  1. 一种对丢包的音频的处理方法，包括：
    对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧；
    基于所述历史音频帧，通过预先训练的神经网络模型对所述第一帧进行预测，得到所述第一帧的第一预测音频数据，以及所述音频中的第二帧的目标段对应的第二预测音频数据，所述目标段在时间上与所述第一帧连接；
    基于所述第二预测音频数据和所述音频中包括的所述第二帧的音频数据，重构所述第二帧的音频数据。
  2. 根据权利要求1所述的方法，其中所述对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧，包括：
    如果当前条件满足预设条件，使用所述神经网络模型对所述第一帧进行丢包补偿，其中，所述预设条件包括以下中的至少一项：所述第一帧的前一帧为浊音帧、音频解码器当前解码的音频流数量不超过预设阈值；以及
    获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧。
  3. 根据权利要求2所述的方法，其中所述音频解码器中包括预设的丢包补偿算法；并且所述方法还包括：
    如果所述当前条件未满足所述预设条件，使用所述音频解码器包括的预设的丢包补偿算法对所述第一帧进行丢包补偿。
  4. 根据权利要求1所述的方法，其中所述基于所述第二预测音频数据和所述音频中包括的所述第二帧的音频数据，重构所述第二帧的音频数据，包括：
    将所述第二预测音频数据与所述音频中包括的所述第二帧的音频数据的目标段进行加窗操作之后相加，得到重构后的第二帧对应的音频数据。
  5. 根据权利要求1所述的方法，其中所述神经网络模型为时域模型，所述时域模型通过以下方式训练得到：
    将第一样本音频中的第一样本帧对应的第一样本历史帧输入待训练时域模型，由所述待训练时域模型对所述第一样本帧进行预测，得到针对所述第一样本帧的第一样本预测音频数据，以及所述第一样本帧的下一帧的目标段对应的第一部分预测音频数据；
    基于所述第一部分预测音频数据和所述第一样本音频中的、所述第一样本帧的下一帧的目标段对应的部分音频数据，重构所述第一样本帧的下一帧的目标段对应的部分音频数据，得到第一部分重构音频数据；
    将所述第一样本音频中的、所述第一样本帧的音频数据和所述第一样本帧的下一帧的目标段对应的部分音频数据作为第一标签，将所述第一样本预测音频数据和所述第一部分重构音频数据作为第一预测数据；以及
    基于所述第一标签和所述第一预测数据的差异损失，调整所述待训练时域模型的模型参数。
  6. 根据权利要求1所述的方法，其中所述神经网络模型为频域模型；以及，所述频域模型通过以下方式训练得到：
    将第二样本音频中的第二样本帧对应的第二样本历史帧进行短时傅里叶变换STFT，得到所述第二样本历史帧对应的历史STFT系数；
    将所述历史STFT系数输入待训练频域模型，得到第一预测STFT系数，其中，所述第一预测STFT系数包括所述第二样本帧对应的第二预测STFT系数、以及所述第二样本帧的下一帧的目标段对应的部分音频数据对应的部分预测STFT系数；
    基于所述第一预测STFT系数进行逆短时傅里叶变换iSTFT，得到所述第二样本帧的第二样本预测音频数据，以及第二样本帧的下一帧的目标段对应的第二部分预测音频数据；
    基于所述第二部分预测音频数据和所述第二样本音频中的、所述第二样本帧的下一帧的目标段对应的部分音频数据，重构所述第二样本帧的下一帧的目标段对应的部分音频数据，得到第二部分重构音频数据；
    将所述第二样本音频中的、所述第二样本帧的音频数据和所述第二样本帧的下一帧的目标段对应的音频数据作为第二标签，将所述第二样本预测音频数据和所述第二部分重构音频数据作为第二预测数据；以及
    基于所述第二标签和所述第二预测数据的差异损失，调整所述待训练频域模型的模型参数。
  7. 根据权利要求1至6任一项所述的方法，所述方法由音频解码器执行，所述对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧，包括：
    所述音频解码器在播放到所述音频的所述第一帧时，在确定所述第一帧为丢包的帧之后，对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧；以及
    所述方法还包括：所述音频解码器基于所述第一预测音频数据播放所述第一帧，基于所述第二帧的重构的音频数据播放所述第二帧。
  8. 一种对丢包的音频的处理装置，包括：
    获取单元，配置为，对于音频中的丢包的第一帧，获取预先存储的、在所述第一帧之前的且与所述第一帧连续的历史音频帧；
    预测单元，配置为，基于所述历史音频帧，通过预先训练的神经网络模型对所述第一帧进行预测，得到所述第一帧的第一预测音频数据，以及所述音频中的第二帧的目标段对应的第二预测音频数据，所述目标段在时间上与所述第一帧连接；以及
    重构单元，配置为，基于所述第二预测音频数据和所述音频中包括的所述第二帧的音频数据，重构所述第二帧的音频数据。
  9. 一种计算机可读存储介质，其上存储有计算机程序，当所述计算机程序在计算机中执行时，令所述计算机执行权利要求1-7中任一项所述的方法。
  10. 一种电子设备，包括存储器和处理器，所述存储器中存储有可执行代码，所述处理器执行所述可执行代码时，实现权利要求1-7中任一项所述的方法。
PCT/CN2024/139747 2024-04-15 2024-12-16 对丢包的音频的处理方法、装置、介质和设备 Pending WO2025218210A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410452103.5A CN118314907A (zh) 2024-04-15 2024-04-15 对丢包的音频的处理方法、装置、介质和设备
CN202410452103.5 2024-04-15

Publications (1)

Publication Number Publication Date
WO2025218210A1 true WO2025218210A1 (zh) 2025-10-23

Family

ID=91732518

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/139747 Pending WO2025218210A1 (zh) 2024-04-15 2024-12-16 对丢包的音频的处理方法、装置、介质和设备

Country Status (2)

Country Link
CN (1) CN118314907A (zh)
WO (1) WO2025218210A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118314907A (zh) * 2024-04-15 2024-07-09 北京字跳网络技术有限公司 对丢包的音频的处理方法、装置、介质和设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140019142A1 (en) * 2012-07-10 2014-01-16 Motorola Mobility Llc Apparatus and method for audio frame loss recovery
CN111554322A (zh) * 2020-05-15 2020-08-18 腾讯科技（深圳）有限公司 一种语音处理方法、装置、设备及存储介质
CN112634912A (zh) * 2020-12-18 2021-04-09 北京猿力未来科技有限公司 丢包补偿方法及装置
CN115171705A (zh) * 2022-06-01 2022-10-11 阿里巴巴云计算(北京)有限公司 一种语音丢包补偿的方法、语音通话的方法及装置
CN116705040A (zh) * 2023-07-05 2023-09-05 维沃移动通信有限公司 音频信号恢复方法、装置、电子设备及可读存储介质
CN117793078A (zh) * 2024-02-27 2024-03-29 腾讯科技（深圳）有限公司 一种音频数据的处理方法、装置、电子设备和存储介质
CN118314907A (zh) * 2024-04-15 2024-07-09 北京字跳网络技术有限公司 对丢包的音频的处理方法、装置、介质和设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140019142A1 (en) * 2012-07-10 2014-01-16 Motorola Mobility Llc Apparatus and method for audio frame loss recovery
CN111554322A (zh) * 2020-05-15 2020-08-18 腾讯科技（深圳）有限公司 一种语音处理方法、装置、设备及存储介质
CN112634912A (zh) * 2020-12-18 2021-04-09 北京猿力未来科技有限公司 丢包补偿方法及装置
CN115171705A (zh) * 2022-06-01 2022-10-11 阿里巴巴云计算(北京)有限公司 一种语音丢包补偿的方法、语音通话的方法及装置
CN116705040A (zh) * 2023-07-05 2023-09-05 维沃移动通信有限公司 音频信号恢复方法、装置、电子设备及可读存储介质
CN117793078A (zh) * 2024-02-27 2024-03-29 腾讯科技（深圳）有限公司 一种音频数据的处理方法、装置、电子设备和存储介质
CN118314907A (zh) * 2024-04-15 2024-07-09 北京字跳网络技术有限公司 对丢包的音频的处理方法、装置、介质和设备

Also Published As

Publication number Publication date
CN118314907A (zh) 2024-07-09

Similar Documents

Publication Publication Date Title
JP7490894B2 (ja) 深層生成ネットワークを用いたリアルタイムパケット損失隠蔽
JP5479617B2 (ja) サブ帯域コード化復号器における損失パケットの隠蔽
JP5405659B2 (ja) 消去されたスピーチフレームを再構成するためのシステムおよび方法
US8200481B2 (en) Method and device for performing frame erasure concealment to higher-band signal
JP2025521290A (ja) 音声認識モデルのトレーニング方法、装置、記録媒体及び電子機器
US11837240B2 (en) Frame error concealment
WO2021227783A1 (zh) 一种语音处理方法、装置、设备及存储介质
JP7615510B2 (ja) 音声強調方法、音声強調装置、電子機器、及びコンピュータプログラム
BRPI0116844B1 (pt) Processo e dispositivo de redução de ruído
WO2025218210A1 (zh) 对丢包的音频的处理方法、装置、介质和设备
CN113763974B (zh) 一种丢包补偿方法、装置、电子设备及存储介质
WO2009152124A1 (en) Concealing audio artifacts
Andreev et al. Iterative autoregression: a novel trick to improve your low-latency speech enhancement model
CN109496333A (zh) 一种丢帧补偿方法及设备
Sacchetto et al. Implementation and optimization of Burg’s method for real-time packet loss concealment in networked music performance applications
WO2025213833A1 (zh) 基于神经网络的语音丢包补偿方法和装置
JP5604572B2 (ja) 複雑さ分散によるデジタル信号の転送誤り偽装
CN118800258A (zh) 一种在高级音频编码中适应窗切换的丢包补偿方法及装置
JP2023535666A (ja) 低域効果チャネルのためのフレーム損失隠蔽
US8812927B2 (en) Decoding device, decoding method, and program for generating a substitute signal when an error has occurred during decoding
HK40028876B (zh) 一种语音处理方法、装置、设备及存储介质
HK40028878B (zh) 一种语音处理方法、装置、设备及存储介质
US20150039979A1 (en) Method and apparatus for concealing error in communication system
HK40028873B (zh) 一种语音处理方法、装置、设备及存储介质
US20090106020A1 (en) Audio glitch reduction

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24935738

Country of ref document: EP

Kind code of ref document: A1