WO2026012017A1 - 一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品 - Google Patents

一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品

Info

Publication number
WO2026012017A1
WO2026012017A1 PCT/CN2025/098977 CN2025098977W WO2026012017A1 WO 2026012017 A1 WO2026012017 A1 WO 2026012017A1 CN 2025098977 W CN2025098977 W CN 2025098977W WO 2026012017 A1 WO2026012017 A1 WO 2026012017A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
video
time
timestamp
smoothing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/098977
Other languages
English (en)
French (fr)
Inventor
郭靖
肖欢
刘天成
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2026012017A1 publication Critical patent/WO2026012017A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/109—Selection of coding mode or of prediction mode among a plurality of temporal predictive coding modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content

Definitions

  • This application relates to the field of video processing technology, and in particular to a method, apparatus, device, computer-readable storage medium, and computer program product for processing video frames.
  • video frame data needs to be transmitted over a network.
  • video playback In environments with unstable network transmission, various problems may occur during video playback, such as video stuttering.
  • Real-time transmission processing methods for video frames in related technologies are associated with actual business scenarios.
  • parameters such as video encoding algorithm strategies, network transmission algorithm strategies, and flow control algorithm strategies need to be optimized due to differences between business scenarios.
  • Direct use will result in an estimation error in the video frame decoding waiting time, which will introduce a large delay and make it difficult to stably control the video frame decoding time, thus affecting the smoothness of video frame decoding.
  • This application provides a video frame processing method, applied to an electronic device, including:
  • a frame smoothing strategy is determined based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames;
  • the delay parameter is determined based on the frame smoothing strategy, and the first rendering timestamp is delayed based on the delay parameter to obtain the second rendering timestamp;
  • the decoding waiting time of the video frame is determined
  • the video frame In response to the video frame being stored in the cached frame queue for a period of time that reaches the decoding waiting time, the video frame is decoded.
  • This application provides a video frame processing method, applied to an electronic device, including:
  • a video frame to be encoded is obtained, and the video frame to be encoded is encoded to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.
  • This application provides a video frame processing apparatus, including:
  • the prediction processing module is configured to predict the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue.
  • a frame smoothing processing module is configured to determine a frame smoothing strategy based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames;
  • the frame smoothing processing module is further configured to determine the delay parameter based on the frame smoothing strategy, and delay the first rendering timestamp based on the delay parameter to obtain the second rendering timestamp;
  • the frame output module is configured to decode the video frame in response to the video frame being stored in the cached frame queue for a period of time equal to the decoding waiting time.
  • This application provides a video frame processing apparatus, including:
  • the processing module is configured to acquire a video frame to be encoded and encode the video frame to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.
  • This application provides an electronic device, the electronic device comprising:
  • Memory configured to store computer-executable instructions
  • a processor When a processor is configured to execute computer-executable instructions or computer programs stored in the memory, it implements the video frame processing method provided in the embodiments of this application.
  • This application provides a computer storage medium storing executable instructions. When these executable instructions are executed by a processor, they implement the video frame processing method provided in this application.
  • This application provides a computer program product, including a computer program or computer executable instructions.
  • the computer program or computer executable instructions When executed by a processor, they implement the video frame processing method provided in this application.
  • the embodiments of this application include at least the following beneficial effects:
  • the first rendering timestamp of the video frame is obtained, providing a reliable time reference benchmark for the calculation of subsequent decoding waiting time.
  • the corresponding frame smoothing strategy based on the video frames in the cache frame queue, an adaptive decision-making process for the frame smoothing strategy is realized.
  • the second rendering timestamp of the video frame is determined, realizing the selection of the corresponding frame smoothing strategy according to the specific situation of the video frames in the cache frame queue.
  • the decoding waiting time of the video frame is quantified, thereby improving the matching degree between the decoding waiting time of the video frame in the cache frame queue and the actual decoding requirements.
  • the accuracy of obtaining the decoding waiting time of the video frame is improved, achieving the beneficial effects of stably controlling the decoding time of the video frame and improving the smoothness of video frame decoding and rendering.
  • Figure 1 is a schematic diagram of the architecture of the video frame processing system provided in an embodiment of this application.
  • Figure 2A is a first structural schematic diagram of the electronic device provided in an embodiment of this application.
  • Figure 2B is a schematic diagram of the second structure of the electronic device provided in an embodiment of this application.
  • FIG. 3A is a first flowchart of the video frame processing method provided in an embodiment of this application.
  • Figure 3B is a second flowchart illustrating the video frame processing method provided in an embodiment of this application.
  • Figure 3C is a schematic diagram of the third process of the video frame processing method provided in the embodiments of this application.
  • FIG. 3D is a fourth flowchart of the video frame processing method provided in the embodiments of this application.
  • Figure 3E is a fifth flowchart illustrating the video frame processing method provided in an embodiment of this application.
  • Figure 3F is a sixth flowchart illustrating the video frame processing method provided in an embodiment of this application.
  • Figure 3G is a schematic diagram of the seventh process of the video frame processing method provided in the embodiments of this application.
  • Figure 3H is a schematic diagram of the eighth process of the video frame processing method provided in the embodiments of this application.
  • Figure 3I is a ninth flowchart illustrating the video frame processing method provided in an embodiment of this application.
  • Figure 3J is a schematic diagram of the tenth process of the video frame processing method provided in the embodiment of this application.
  • Figure 3K is a schematic diagram of the eleventh step of the video frame processing method provided in the embodiments of this application.
  • Figure 3L is a schematic diagram of the twelfth step of the video frame processing method provided in the embodiments of this application;
  • Figure 3M is a schematic diagram of the thirteenth step of the video frame processing method provided in the embodiments of this application.
  • Figure 4 is a schematic diagram of the time sliding window provided in an embodiment of this application.
  • Figure 5 is a flowchart illustrating the video frame processing method in a video call scenario provided in an embodiment of this application
  • Figure 6A is a schematic diagram illustrating the principle of video frame processing provided in the embodiments of this application.
  • Figure 6B is a schematic diagram illustrating the principle of storing video frames into a buffer frame queue according to an embodiment of this application;
  • first, second, third are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
  • module refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof.
  • a processor or multiple processors or memory
  • each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
  • the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.
  • the buffered frame queue also known as the frame smoothing queue, is a data structure used to store video frames to be decoded until they can be read from the buffered frame queue for the decoder to decode.
  • Acquisition timestamp refers to the time stamp corresponding to the moment a video frame is acquired by the acquisition device.
  • the acquisition timestamp is used to indicate the generation time of the video frame, so as to sort the video frames in chronological order and perform related temporal analysis.
  • the first rendering timestamp also known as the estimated rendering timestamp, refers to the predicted time point in the video rendering process when the rendering task of a video frame is completed.
  • the second rendering timestamp also known as the expected rendering timestamp, refers to the predicted time point for completing the rendering task of a video frame, obtained by delaying the first rendering timestamp during the video rendering process. It is used to plan the progress of the rendering task and ensure that the video frame can be rendered within the predetermined time.
  • Frame smoothing strategy This strategy uses time delay to smooth out video frames, thereby reducing or eliminating problems such as uneven video frame rendering and stuttering caused by sudden changes in video speed on the receiving device.
  • Smooth frame rendering is a technique that controls the decoding wait time of received video frames to make the rendering time of video frames as stable as possible, thereby making the video look smoother.
  • Receive timestamp refers to the time stamp recorded by the receiving device when the video frame data arrives at the receiving device.
  • the current timestamp refers to the number of seconds or milliseconds that have elapsed since a specific moment (e.g., the Unix epoch, which is midnight on January 1, 1970).
  • the current timestamp can be obtained using a time library in a programming language, such as Python's datetime library or JavaScript's Date object.
  • Decoding wait time refers to the time interval between when a video frame enters the buffer frame queue and when it is removed to call the decoder for decoding. In other words, the storage time of the video frame in the buffer frame queue. For example, when the storage time of the video frame in the buffer frame queue reaches the decoding wait time, the video frame is taken out of the buffer frame queue and transferred to the decoder for decoding.
  • Network jitter refers to the irregularity and instability of the arrival time of data packets of video frames during the transmission process from the sending end device to the receiving end device in a computer network due to network conditions, traffic congestion, etc., which manifests as changes in the time interval between adjacent data packets arriving at the receiving end device.
  • Packet loss retransmission is a network transmission mechanism used to recover data packets of video frames when they are lost in the network.
  • a sending device sends a data packet
  • the receiving device sends an acknowledgment packet (ACK) to indicate receipt. If the receiving device does not receive a data packet within the expected time (e.g., to determine whether a data packet has been received based on its sequence number), it sends a negative acknowledgment packet (NACK) to inform the sending device.
  • ACK acknowledgment packet
  • NACK negative acknowledgment packet
  • Video frames waiting to enter the buffer frame queue refer to video frames in the buffer (JitterBuffer). When a video frame satisfies the reference relationship, it will be transferred from the buffer to the buffer frame queue.
  • the reference relationship refers to the fact that when a video frame (such as a P-frame or B-frame) is decoded, it needs to rely on a reference frame (such as an I-frame or a decoded P-frame). If the reference frame of the current frame is not received, not decoded, or the decoding fails, the current frame does not meet the reference relationship and needs to be buffered to wait for entry into the buffer frame queue. If the reference frame of the current frame is ready (such as being received and decoded), the current frame meets the reference relationship and can be transferred from the buffer to the buffer frame queue.
  • a reference frame such as an I-frame or a decoded P-frame
  • the received video frame when the received video frame does not satisfy the reference relationship (that is, when the reference frame of the currently received video frame has not yet been received, for example, the currently received video frame is P2, and the previous reference frame corresponding to P2 can be represented as I0, and different video frames can correspond to the same reference frame), it will be buffered in the buffer of the receiving device.
  • the reference relationship that is, when the reference frame of the currently received video frame has not yet been received, for example, the currently received video frame is P2
  • the previous reference frame corresponding to P2 can be represented as I0, and different video frames can correspond to the same reference frame
  • Reference frames are frames in a video frame sequence that are used to help decode subsequent video frames; that is, the decoding of a video frame depends on the preceding reference frames. These reference frames are used by the encoder to generate prediction information, and the decoder uses this prediction information to reconstruct the video image.
  • Decoded frame refers to a video frame that has been successfully decoded. For example, after a video frame is removed from the buffer frame queue, the decoder is called to decode the removed video frame. After successful decoding, the removed video frame is recorded as a decoded frame.
  • the decoding frame interval parameter is the interval between decoding frames. For example, it is a statistical measure of the decoding interval between two adjacent decoding frames (i.e., two video frames decoded sequentially) within a historical time period of a preset duration from the current time, such as the average value or quantile.
  • the first time period also known as the jitter period, refers to the time range during which, due to changes in network latency or other reasons, the data packets of a video frame arrive at the receiving device in a situation where the time interval between adjacent data packets is greater than a preset time interval threshold (e.g., 100ms). For example, when the time window length is 2s, if there is a situation where the time interval between adjacent data packets received within 2s arrives at the receiving device is greater than the preset time interval threshold, then the 2s range corresponding to the time window is the first time period.
  • a preset time interval threshold e.g. 100ms
  • the second time period also known as the packet loss retransmission period, refers to the time range from the start of retransmission of data packets to the arrival of the retransmitted data packets at the receiving device when the time interval between the arrival of data packets at the receiving device is less than or equal to a preset time interval threshold (e.g., 100ms) and data packets are lost during transmission.
  • a preset time interval threshold e.g. 100ms
  • the sending device is sending a series of video frames, each frame being divided into multiple data packets for transmission.
  • the arrival time interval of the data packets during transmission is less than or equal to the preset time interval threshold (e.g., 100ms), but the data packets of one video frame are lost during transmission.
  • the sending device After receiving a retransmission request, the sending device will immediately begin retransmitting the lost data packets.
  • the time from when the sending device sends the retransmission data packets until the receiving device successfully receives these data packets and puts them into the video stream is the second time period.
  • the third time period refers to the time range during which data packets arrive at the receiving device within a preset time interval threshold (e.g., 100ms) during the transmission of video frame data packets, and no data packets are lost during transmission.
  • a preset time interval threshold e.g. 100ms
  • the sending device is sending a series of video frames, each frame is divided into multiple data packets for transmission, and within a certain period (e.g., 5s), the data packets are successfully transmitted to the receiving device.
  • a certain period e.g., 5s
  • the arrival time interval of data packets during transmission is less than or equal to the preset time interval threshold (e.g., 100ms).
  • This time period is the third time period.
  • Receive frame rate which represents the number of video frames entering the buffer frame queue per second.
  • Real-time transmission and processing methods for video frames in related technologies are related to actual business scenarios.
  • parameters such as video encoding algorithm strategies, network transmission algorithm strategies, and flow control algorithm strategies need to be optimized due to differences between business scenarios.
  • Direct use will result in jitter prediction deviations, which will introduce large delays and make it difficult to stably control the decoding time of video frames, thus affecting the smoothness of video frame decoding.
  • the frame is then sent to the decoder for decoding and rendering.
  • the reception time of video frames varies with factors such as network jitter, it is impossible to smoothly output frames using a single time base, affecting the smoothness of video frame decoding.
  • embodiments of this application provide a video frame processing method, apparatus, device, computer-readable storage medium, and computer program product, which can stably control the decoding time of video frames, thereby improving the smoothness of video frame decoding and rendering.
  • the electronic devices provided in this application can be implemented as various types of terminal devices, such as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smartphones, smart speakers, smart voice interaction devices, smart home appliances, smartwatches, smart TVs, in-vehicle terminals, smart headphones, and aircraft. Exemplary applications of the electronic devices implemented as terminal devices will be described below.
  • Figure 1 is a schematic diagram of the video frame processing system architecture provided in an embodiment of this application
  • Figure 1 involves server 100, terminal device 200-1, terminal device 200-2, and network 300.
  • Terminal device 200-1 and terminal device 200-2 are connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.
  • WAN wide area network
  • LAN local area network
  • the terminal device or server can implement the video frame processing method provided in this application by running various computer-executable instructions or computer programs.
  • computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions.
  • Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as instant messaging clients; or they can be applets embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run.
  • APPs native applications
  • the aforementioned computer-executable instructions can be any form of instruction
  • the aforementioned computer programs can be any form of application, module, or plugin.
  • server 100 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.
  • cloud services may be interactive processing services that can be invoked by terminals.
  • the video frame processing system provided in this application can be implemented collaboratively by a server and a terminal device.
  • terminal device 200-1 communicates with terminal device 200-2 via an instant messaging client.
  • the sensor device of terminal device 200-1 e.g., the built-in camera of terminal device 200-1 captures video frames, divides the video frames into multiple data packets, and transmits them to terminal device 200-2 via the server 100 in the background of the instant messaging client.
  • terminal device 200-2 In response to receiving the data packets of the video frames sent by terminal device 200-1, terminal device 200-2 combines the data packets into a video frame and stores it in a buffer frame queue.
  • terminal device 200-2 performs frame smoothing processing on the video frame using the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frame.
  • the video frame is removed from the buffer frame queue and sent to the decoder of terminal device 200-2.
  • the decoder is called to decode the video frame, and finally, the decoded video frame is displayed on the interface of terminal device 200-2 through video rendering.
  • the terminal device is used to capture video frames, encapsulate the video frames into data packets and send them to the server, and receive the peer video data packets forwarded by the server, combine them into video frames and store them in a buffered frame sequence.
  • the video frames are smoothed using the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames.
  • the video frame is removed from the buffered frame queue and the decoder is called to decode the video frame.
  • the server is used to relay video data packets between terminal devices to realize the establishment and maintenance of end-to-end real-time data transmission links.
  • the terminal device is used to collect local video streams and encode them into data packets, receive video data packets from other participants distributed by the server, combine them into video frames, and store them in a buffered frame sequence.
  • the video frames are then processed by the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames.
  • the video frame is removed from the buffered frame queue, and the decoder is called to decode the video frame.
  • the server is used to aggregate multiple video streams, perform signaling interaction management (such as participant join/leave notifications), and distribute data (such as broadcasting a video stream from one end to other terminals).
  • the terminal device (broadcaster) is used to capture the live stream and split it into data packets to upload to the server; the terminal device (viewer) is used to receive the live video data packets pushed by the server, combine them into video frames and store them in a cached frame sequence.
  • the video frames are then processed by the video frame processing method provided in this application embodiment to obtain the decoding waiting time of the video frames.
  • the server is used to receive the video stream from the broadcaster, perform transcoding (such as multi-resolution adaptation), and distribute it to the viewer through a content delivery network (CDN).
  • CDN content delivery network
  • the video frame processing system provided in this application can be implemented by a server alone.
  • server 100 acquires data packets, combines the data packets into video frames, and stores them in a buffer frame queue.
  • server 100 performs frame smoothing processing on the video frames using the video frame processing method provided in this application to obtain the decoding waiting time of the video frames.
  • the video frames are dequeued from the buffer frame queue and sent to the decoder of server 100.
  • the decoder is then called to decode the video frames to obtain the decoded video frames.
  • the video frame processing system provided in this application can be implemented by a terminal device.
  • terminal device 200-1 captures video frames, divides the video frames into multiple data packets, and transmits them to terminal device 200-2.
  • terminal device 200-2 receives the data packets of the video frames sent by terminal device 200-1
  • terminal device 200-2 combines the data packets into video frames and stores them in a buffer frame queue.
  • terminal device 200-2 performs frame smoothing processing on the video frames using the video frame processing method provided in this application to obtain the decoding waiting time of the video frames.
  • the video frames are dequeued from the buffer frame queue and sent to the decoder of terminal device 200-2.
  • the decoder is called to decode the video frames, and finally, the decoded video frames are displayed on the interface of terminal device 200-2 through video rendering.
  • FIG. 2A is a first structural schematic diagram of the electronic device provided in this application embodiment.
  • the electronic device can be implemented as terminal device 200-1, terminal device 200-2, or server 100 as described above.
  • the electronic device 400-1 shown in Figure 2A includes: at least one processor 410-1, a memory 450-1, at least one network interface 420-1, and a user interface 430-1.
  • the various components in the electronic device 400-1 are coupled together through a bus system 440-1. It can be understood that the bus system 440-1 is used to realize the connection and communication between these components.
  • the bus system 440-1 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440-1 in Figure 2.
  • the processor 410-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor can be a microprocessor or any conventional processor, etc.
  • User interface 430-1 includes one or more output devices 431-1 that enable the presentation of media content, including one or more speakers and/or one or more visual displays.
  • User interface 430-1 also includes one or more input devices 432-1, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
  • the memory 450-1 may be removable, non-removable, or a combination thereof.
  • Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc.
  • the memory 450-1 may optionally include one or more storage devices physically located away from the processor 401-1.
  • Memory 450-1 may include volatile memory or non-volatile memory, or both.
  • Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM).
  • ROM read-only memory
  • RAM random access memory
  • the memory 450-1 described in this application embodiment is intended to include any suitable type of memory.
  • the network communication module 452-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420-1, exemplary network interfaces 420-1 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.
  • the sending device acquires a video frame to be encoded (or the original video frame), performs video preprocessing, video encoding, and packetization on the video frame to be encoded, and obtains a data packet of the encoded video frame.
  • the acquisition timestamp and transmission time are sent to the receiving device along with the data packet.
  • the receiving device decodes the encoded video frame using the video frame processing method provided in this application embodiment.
  • Video coding is the process of converting video frames into a compressed format for storage and transmission. For example, the image of a video frame to be encoded is decomposed into smaller units (such as macroblocks), and then a series of mathematical transformations (such as discrete cosine transform) are applied to these units to eliminate redundant information in the data. Subsequently, the transform coefficients are adjusted through a quantization process to reduce the amount of data that needs to be transmitted. Video coding also includes dividing the image data of the video frame to be encoded into different layers, such as spatial layers, temporal layers, and redundancy layers, which can provide different levels of compression while maintaining video frame quality. Commonly used video coding standards include High Efficiency Video Coding (HEVC) and the Open Media Consortium (AOMedia Video 1, AV1).
  • HEVC High Efficiency Video Coding
  • AOMedia Video 1, AV1 Open Media Consortium
  • Packet segmentation refers to the process of dividing encoded video frames into smaller data packets. These data packets, including portions of the video frames, are used for transmission over the network to the end user's device (the receiving device).
  • the purpose of packet segmentation is to adapt to different network conditions, such as bandwidth limitations and latency requirements. Packet segmentation enables more efficient transmission control, such as adjusting the frequency, size, and order of data packet transmission to optimize network resource usage and ensure smooth video streaming in unstable network environments.
  • FIG3A is a first flowchart of the video frame processing method provided in this application embodiment, which can be executed by the aforementioned terminal device as the receiving end device, and will be described in conjunction with the steps shown in FIG3A.
  • step 101 the first rendering timestamp of the video frame is predicted based on the acquisition timestamp of the video frame in the cached frame queue.
  • the transmitting device of a video frame stores information such as the acquisition timestamp of each video frame and the transmission time of the transmitting device in the frame header information. After the transmitting device performs video preprocessing, video encoding and packetization on the video frame to be encoded, it obtains a data packet and sends the acquisition timestamp and transmission time information along with the data packet to the receiving device.
  • the receiving device receives data packets and combines them into video frames (here, video frames are encoded video frames).
  • video frames are encoded video frames.
  • the reference relationship means that the decoding of the video frame depends on the previous reference frame. If the reference frame of the current video frame has not yet been received by the receiving device, the video frame will wait in the buffer until the reference frame of the video frame is successfully received, that is, after the reference relationship is satisfied, the video frame will be sent to the buffer frame queue for subsequent processing such as determining the frame smoothing strategy.
  • step 101 shown in FIG3A can be implemented by steps 1011 to 1013, which are described in detail below.
  • the video frame receiving frame rate within the time period corresponding to the video frame is obtained, the video frame receiving frame rate is mapped to multiple preset frame rate levels, and the mapped frame rate level is used as the receiving frame rate level.
  • the frame rate level can be the video frame acquisition frame rate or encoding frame rate of the sending device, such as 12FPS, 15FPS, and 20FPS. Assuming that the number of video frames received per second is 18, and the closest frame rate on the flow control policy level is 20FPS, then the corresponding video frame receiving frame rate level is 20FPS.
  • This application embodiment maps the received frame rate of video frames to a preset frame rate level (such as 12FPS, 15FPS, etc.), which can filter out small fluctuations in the real-time frame rate and establish a stable frame rate benchmark, thereby improving the adaptability of the video frame processing method provided in this application embodiment to dynamic network environments.
  • a preset frame rate level such as 12FPS, 15FPS, etc.
  • step 1012 the receiving time difference of the video frames is obtained.
  • step 1012 shown in FIG3B can be implemented by steps 10121 to 10122, which are described in detail below.
  • step 10121 the difference between the received timestamp of multiple video frames and the sum of the video frame's acquisition timestamp, round-trip delay, and transmission time is obtained.
  • the difference between the received timestamps of multiple video frames and the sum of the video frame's acquisition timestamp, round-trip time, and transmission time can be calculated.
  • the transmitting device stores information such as the acquisition timestamp of each video frame and the transmission time of the transmitting device in the frame header information. After receiving the data packet and framing it, the receiving device obtains the acquisition timestamp and transmission time of the video frame from the frame header information.
  • the statistical range can be from the start of the video call to the current moment, or it can be one or more sampling periods (e.g., 2 seconds) during the video call process. The difference within one or more sampling periods can be counted. One or more sampling periods can be traced back from the current moment, so as to accurately reflect the current network status.
  • step 10122 the average value of the differences corresponding to multiple video frames is taken as the reception time difference.
  • the average of the differences corresponding to multiple video frames within the above statistical range is taken as the reception time difference.
  • This application embodiment calculates the difference between the sum of the received timestamp and the acquisition timestamp, round-trip delay, and transmission time of multiple video frames, and takes the average value as the received time difference. This can dynamically capture the overall trend of network transmission delay, filter out the interference of abnormal fluctuations in single frames, improve the accuracy of the received time difference in representing the real-time network status, and provide a more reliable delay compensation benchmark for the subsequent prediction of the first rendering timestamp.
  • the first rendering timestamp of the video frame is determined based on the received frame rate level, the received time difference, and the acquisition timestamp.
  • step 1013 shown in FIG3B can be implemented by steps 10131 to 10133, which are described in detail below.
  • step 10131 the reciprocal of the preset multiple of the received frame rate level is obtained.
  • the reciprocal of a preset multiple (e.g., 1000) of the receive frame rate level is obtained.
  • the receive frame rate level can be, for example, 12 FPS, 15 FPS, and 20 FPS.
  • the receive frame rate level can be flexibly set according to the requirements of the flow control strategy.
  • the reciprocal of the preset multiplier receiving frame rate level can be expressed as: 1000 ⁇ (1 / receiving frame rate level).
  • step 10132 the sum between the collection timestamp and the reciprocal is obtained.
  • the collection timestamp and the reciprocal are summed to obtain a sum.
  • step 10133 the summation of the summation and the receiving time difference is performed to obtain the first rendering timestamp.
  • First rendering timestamp video frame acquisition timestamp + 1000 / receiving frame rate level + receiving time difference, where the summation refers to the sum between the acquisition timestamp and the reciprocal, i.e., "video frame acquisition timestamp + 1000 / receiving frame rate level”.
  • a frame smoothing strategy is determined based on the video frames in the cached frame queue.
  • video frames are obtained by combining received data packets. For multiple data packets received within the current time window (configured according to a preset duration, such as 2s), if the time interval between all or more of the adjacent data packets is greater than the time interval threshold (such as 100ms), it is determined that the current moment is in the first time period (also known as the jitter time period). It is then determined that before the time window slides again according to a predetermined step size (such as 2s), a first frame smoothing strategy is applied to the buffer frame queue.
  • the starting point of the current time window is a historical moment that is a preset time interval away from the current moment, and the ending point of the current time window is the current moment.
  • the first frame smoothing strategy is used to delay the video frames in the buffer frame queue within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the buffer frame queue.
  • synchronous change refers to the dynamic relationship between the frame smoothing factor and the number of video frames in the cached frame queue. Specifically, when the number of cached frames is large, the frame smoothing factor is increased, thereby reducing the output frame rate and decreasing decoding pressure. Conversely, when the number of cached frames is small, the frame smoothing factor is decreased, increasing the output frame rate. This utilizes existing frame resources to fill gaps caused by network fluctuations, preventing the decoder from idling. Thus, the decoding strategy is dynamically adjusted based on network conditions and cache status, improving the smoothness and stability of video playback. For example, when network jitter causes longer packet reception intervals, the number of cached frames increases, the frame smoothing factor increases accordingly, and the output frame rate decreases, thereby mitigating the impact of network fluctuations on video playback.
  • the first frame smoothing strategy can be started for the video frames in the buffer frame queue until the jitter value (the time interval between the reception of adjacent data packets) in the 2-second time window is lower than the preset time interval threshold (e.g., 100ms), at which point the first frame smoothing strategy for the video frames in the buffer frame queue can be stopped.
  • the preset time interval threshold e.g. 100ms
  • jitter values can be measured using a sliding time window algorithm.
  • a time window with a length of 2 seconds is set. Within the current time window, whenever a data packet arrives, its arrival time is recorded; these arrival times are within the current time window. As time progresses, the time window slides along the time axis in steps of a certain size. As shown in Figure 4, time window 1 with a length of 2 seconds slides in steps of 1.5 seconds to obtain the next time window 2. Time window 1 and time window 2 overlap to ensure that the arrival times of data packets within the most recent 2 seconds are always included. Jitter can be measured by calculating the arrival time difference of all data packets within the time window.
  • the arrival time difference between it and the previous data packet in the window is calculated.
  • These time differences are combined into a jitter time series.
  • the arrival time difference between data packet 2 and data packet 1 the arrival time difference between data packet 3 and data packet 2, and the arrival time difference between data packet 4 and data packet 3 corresponding to time window 1 shown in Figure 4 are obtained.
  • the absolute value, mean, median, or preset quantile (e.g., 95th quantile) of these time differences can also be obtained as the jitter value corresponding to the current time window.
  • the 95th quantile of the arrival time difference between data packets corresponding to time window 1 in Figure 4 can be obtained as the jitter value corresponding to time window 1.
  • the time window will continue to slide, record the arrival time of new data packets, and update the jitter value.
  • the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not within the first time period)
  • the second time period also known as the packet loss retransmission time period
  • the starting point of the current time window is a historical time that is a preset duration (i.e., the length of the time window) away from the current time
  • the ending point of the time window is the current time (the step size of the window can also be preset
  • the ending point of the time window is the current time after each slide).
  • the second frame smoothing strategy is used to delay the video frames in the buffer frame queue based on the decoding frame interval parameter within the second time period.
  • the decoding frame interval parameter is the interval between the decoding times of the already decoded video frames, which can be a statistical number of the decoding intervals between the already decoded video frames within a historical time period that is a preset duration away from the current time, such as the average value, a preset quantile, etc.
  • the current time is determined to be in the second time interval. This can be achieved in the following way: If the receiving time interval of adjacent data packets is less than or equal to the time interval threshold, and at least one data packet is lost during transmission within the current time window, then the current time is determined to be in the second time interval.
  • this can accurately locate the packet loss retransmission scenario to avoid misjudging normal network fluctuations as packet loss retransmission, and can also be more targeted when the second frame smoothing strategy is triggered, reducing performance loss when there is no packet loss.
  • the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period)
  • the current time is in the third time period
  • a third frame smoothing strategy is applied to the buffer frame queue before the next slide of the time window.
  • the starting point of the time window is a historical time that is a preset duration away from the current time (i.e., the length of the time window)
  • the ending point of the time window is the current time.
  • the third frame smoothing strategy is used to delay a specific video frame in the buffer frame queue based on a fixed frame smoothing factor within the third time period.
  • the specific video frame refers to a video frame that meets the following condition: the time difference between the reception timestamp in the buffer frame queue and the decoding time of the previous decoded frame is less than a preset proportion of the decoding frame interval parameter.
  • the decoding time of a decoded frame can be the time when the decoding of the decoded frame is completed, or the time when the decoding of the decoded frame begins.
  • the current time is determined to be in the third time interval. This can be achieved in the following way: if the receiving time interval of adjacent data packets is less than or equal to the time interval threshold, and the data packets are not lost during transmission within the current time window, then the current time is determined to be in the third time interval.
  • the strategy based on the fixed frame smoothing factor is avoided in case of packet loss and retransmission, reducing unnecessary delay processing of video frames.
  • precise delay optimization is achieved for specific video frames in the buffer frame queue whose time difference between the received timestamp and the decoding time of the previous decoding frame is less than the preset proportion of the decoding frame interval parameter. While maintaining smooth video playback, the performance loss and playback delay caused by the mis-triggered strategy are effectively reduced.
  • the reception time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period), and a data packet is lost during transmission within the current time window, then the current time is determined to be in the second time period (also known as the packet loss retransmission period), and the second frame smoothing strategy is applied to the buffer frame queue before the next slide of the current time window; if the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and no data packet is lost during transmission within the current time window, then the current time is determined to be in the third time period, and the third frame smoothing strategy is applied to the buffer frame queue before the next slide of the current time window.
  • the time interval threshold i.e., the current time is not in the first time period
  • the second frame smoothing strategy is applied to the buffer frame queue before the next slide of the current time window
  • step 103 the first rendering timestamp is delayed based on the frame smoothing strategy to obtain the second rendering timestamp.
  • step 103 shown in FIG3A can be implemented by steps 1031A to 1033A, which are described in detail below.
  • a frame smoothing factor is determined based on the number of video frames in the buffer frame queue, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffer frame queue.
  • step 1031A can be implemented by steps 10311 to 10313, which are described in detail below.
  • step 10311 the initialized frame smoothing factor is determined based on the decoding frame interval parameter.
  • the initial frame smoothing factor can be determined based on the decoding frame interval parameter corresponding to the current video frame.
  • the decoding frame interval parameter is the interval between the decoding times of already decoded video frames. It can be a statistical count of the decoding intervals between already decoded video frames within a historical time period of a preset duration from the current time, such as the average value or a preset quantile (e.g., the 95th quantile).
  • the decoding frame interval (the interval between the timestamps of the two successful decoding times when two adjacent video frames are successfully decoded) is less than or equal to the minimum value between the jitter average and a preset threshold (e.g., 500).
  • the jitter average refers to the average of the reception time intervals of adjacent data packets in the current time window.
  • a maximum value constraint is applied to the decoding frame interval parameter, such that the decoded frame interval parameter after the maximum value constraint is equal to the first decoding frame interval parameter threshold; when the decoding frame interval parameter is less than the first decoding frame interval parameter threshold, a minimum value constraint is applied to the decoding frame interval parameter, such that the decoded frame interval parameter after the maximum value constraint is equal to the second decoding frame interval parameter threshold, wherein the first decoding frame interval parameter threshold is greater than the second decoding frame interval parameter threshold.
  • the first decoding frame interval parameter threshold is 125ms and the second decoding frame interval parameter threshold is 33ms
  • the value of the decoding frame interval parameter is taken as 125ms
  • the value of the decoding frame interval parameter is taken as 33ms.
  • the initial frame smoothing factor corresponding to the decoding frame parameter is determined according to the preset parameter range corresponding to the decoding frame interval parameter.
  • the initial frame smoothing factor can be set to 5/4, which means that when the receive frame rate is around 30FPS, the minimum output frame rate is 24FPS to smooth the output frame (removing it from the buffer frame queue for decoding).
  • a larger decoding frame interval parameter indicates that the decoder needs to wait longer to receive and process the next video frame, resulting in higher network latency.
  • the frame rate output is faster when the decoding frame interval parameter is larger. This improves the smoothness of the video by increasing the frame rate output and reduces the impact of video stuttering caused by network latency, thereby improving the user experience.
  • step 10312 the time interval between the entry of a video frame into the buffer frame queue and the previous video frame is obtained.
  • the receiving device receives data packets, combines the data packets into video frames, and stores the video frames in a buffer. Only when the video frames satisfy the dependency relationship, that is, when the reference frame corresponding to the video frame has been received, will the video frames waiting to enter the buffer frame queue be transferred from the buffer to the frame buffer queue. In other words, there is a time interval between the video frames entering the buffer frame queue.
  • step 10313 in response to a time interval less than a time interval threshold or a number of video frames greater than a first buffer frame number threshold, the initialized frame smoothing factor is adjusted according to the number of video frames, and the frame smoothing factor is determined.
  • the initialized frame smoothing factor is adjusted according to the number of video frames to determine the frame smoothing factor.
  • a time interval threshold e.g. 10ms
  • a first buffer frame number threshold e.g. 1, 1
  • the adjustment of the initialized frame smoothing factor according to the number of video frames in step 10313 shown in Figure 3F can be implemented through steps 10314 to 10315, which are described in detail below.
  • step 10314 in response to the number of video frames being greater than the second buffer frame number threshold, the initialized frame smoothing factor is configured according to a fixed value to obtain the frame smoothing factor, wherein the second buffer frame number threshold is greater than the first buffer frame number threshold.
  • the initialized frame smoothing factor is configured with a fixed value to obtain the frame smoothing factor, for example, the frame smoothing factor is set to a fixed value of 1/2.
  • step 10315 in response to the number of video frames being less than or equal to the second buffer frame number threshold, the initialized frame smoothing factor is inversely divided to obtain the frame smoothing factor.
  • the initial frame smoothing factor is inversely divided to obtain the frame smoothing factor, for example, the initial frame smoothing factor 5/4 is inversely divided to become 4/5.
  • step 1032A the delay parameter is obtained through the frame smoothing factor.
  • step 1032A can be implemented by steps 1032A1 to 1032A2, which will be described in detail below.
  • step 1032A the product of the frame smoothing factor and the decoding frame interval parameter is obtained.
  • the statistical value of the decoding frame interval within the time period corresponding to the current video frame can be obtained as the decoding frame interval parameter.
  • the time period corresponding to the current video frame can be a historical time period with a preset duration from the receiving timestamp of the current video frame, or a historical time period with a preset duration from the current moment.
  • the 95th percentile of the decoding frame interval within a time period with a duration of 2 seconds from the receiving timestamp of the current video frame (the time period corresponding to the video frame) can be obtained as the decoding interval parameter.
  • the frame smoothing factor can be obtained by referring to the frame smoothing factor determined based on the number of video frames in the buffer frame queue in step 1031A above.
  • step 1032A2 the minimum value of the product and the delay control parameter is determined and the sum of the current timestamp is calculated. The difference between the sum and the first rendering timestamp is then calculated to obtain the delay parameter.
  • the minimum value of the product and the delay control parameter (e.g., 125ms) is determined and summed with the current timestamp. The difference between the sum and the first rendering timestamp is then calculated to obtain the delay parameter.
  • Delay parameter current timestamp + min(decoding frame interval parameter ⁇ frame smoothing factor, 125) - first rendering timestamp, where "min” means to obtain the minimum value between "decoding frame interval parameter ⁇ frame smoothing factor” (product) and "125” (delay control parameter).
  • the delay parameter is less than or equal to the minimum value between the maximum jitter value and a preset threshold (e.g., 500ms), where the maximum jitter value refers to the maximum value of the reception time interval between adjacent data packets in the current first time period.
  • a preset threshold e.g. 500ms
  • the first rendering timestamp and the delay parameter are summed, and the sum is used as the second rendering timestamp.
  • a bidirectional constraint is imposed on the decoding interval parameter by setting a first decoding frame interval parameter threshold and a second decoding frame interval parameter threshold. This not only avoids the impact of extreme interval values caused by network fluctuations on frame rate stability, but also provides standardized input for the subsequent determination of the frame smoothing factor, ensuring the rationality of parameter calculation.
  • the frame smoothing factor is fixed to stabilize the frame output rhythm and prevent decoding overload.
  • the number of buffered frames is less than the second number of buffered frames, decoding is accelerated by taking the reciprocal of the frame smoothing factor, utilizing existing frame resources to fill the gaps in network fluctuations, avoiding decoder idleness, and improving the adaptability of the video frame decoding process to dynamic network jitter.
  • step 103 shown in FIG3A can be implemented by steps 1031B to 1033B, which are described in detail below.
  • the number of slow-smoothing frames is greater than a slow-smoothing frame number threshold (e.g., 0)
  • the number of slow-smoothing frames is decremented, and a frame smoothing factor is determined based on a decoding frame interval parameter.
  • the slow-smoothing frame number characterizes the number of video frames to be processed after the current video frame in the buffer frame queue. These video frames are those that need to be delayed using the second frame smoothing strategy.
  • the slow-smoothing frame number is decremented from an initial value, and the decrement amount is consistent with the number of video frames smoothed using the second frame smoothing strategy.
  • the slow-smoothing frame number is decremented once for each subsequent video frame until it equals the slow-smoothing frame number threshold.
  • the decoding frame interval parameter is the interval between the decoding times of the decoded frames. For example, it can be a statistical representation of the decoding interval between two adjacent decoded frames (i.e., two video frames decoded sequentially) within a historical time period preset from the current time, such as an average or quantile.
  • the number of slow smoothing frames is greater than 0, let's assume that the number of slow smoothing frames is 3 (the number of slow smoothing frames 3 here is obtained by subtracting from the initial value. For example, if the initial value is 5, it means that 2 video frames have been smoothed before processing the current video frame, so the number of slow smoothing frames corresponding to the currently processed video frame is obtained by subtracting the initial value of 5 to 3).
  • the next 3 video frames after the currently processed video frame also need to be processed by the second frame smoothing strategy.
  • the next 3 frames are delayed by the second frame smoothing strategy, the number of slow smoothing frames is reduced by 1 after each frame is processed, until the number of slow smoothing frames returns to 0.
  • a new initial value needs to be obtained based on the frame sorting buffer duration of the currently processed video frame.
  • the currently processed video frame is P3.
  • video frames P1 and P2 have already been smoothed.
  • the number of slowly smoothed frames corresponding to P3 is obtained by decreasing the initial value, which is determined by the ratio of the frame sorting buffer duration corresponding to P1 to the decoding frame interval parameter.
  • the initial value for P1 is 1. Since frame P2 was produced after P1, the initial value is decremented once.
  • the number of slowly smoothed frames decreases from the initial value of 1 to 0.
  • a new initial value is obtained based on the frame sorting buffer duration; in response to the new initial value being equal to the slow smoothing frame count threshold (i.e., the new initial value is still equal to the slow smoothing frame count threshold); the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained (see the description of step 1031C below); in response to the time difference being less than a preset proportion of the decoding frame interval parameter, a delay parameter is obtained through a fixed-value frame smoothing factor (see the description of step 1032C below); and the delay parameter is obtained through the first rendering timestamp and the delay parameter.
  • a slow smoothing frame count threshold e.g., 0
  • the delay parameter is obtained through the frame smoothing factor, where the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue.
  • step 1032B shown in FIG3I can be implemented by steps 1032B1 to 1032B2, which are described in detail below.
  • step 1032B1 the product of the frame smoothing factor and the decoding frame interval parameter is obtained, and the product is summed with the current timestamp to obtain the summation result.
  • the frame smoothing factor is multiplied by the decoding frame interval parameter and then added to the current timestamp to obtain the summation result.
  • summation result frame smoothing factor ⁇ decoding frame interval parameter + current timestamp.
  • step 1032B2 the summation result is subtracted from the first rendering timestamp to obtain the delay parameter.
  • the second rendering timestamp is determined using the first rendering timestamp and the delay parameter.
  • step 1033A the implementation method for determining the second rendering timestamp can be found in the description of step 1033A above.
  • steps 1031B to 1032B can be implemented as processing the frame smoothing strategy of the video frames buffered during the second time period, i.e. during packet loss retransmission.
  • the time difference between the retransmission recovery frame and the decoding time of the previous decoding frame can be obtained.
  • the process of delaying the first rendering timestamp through the third frame smoothing strategy to obtain the second rendering timestamp is initiated (see the description of steps 1032C to 1033C below). Otherwise, the frame is output in real time, i.e., the retransmission recovery frame is removed from the buffer frame queue and the retransmission recovery frame is decoded.
  • step 103 shown in FIG3A can be implemented by steps 1031C to 1033C, which are described in detail below.
  • step 1031C the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained.
  • the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame is obtained.
  • the decoding time of the previous decoded frame can be the time when the decoding of the decoded frame is completed or the time when the decoding of the decoded frame begins.
  • step 1032C in response to a decoding frame interval parameter where the time difference is less than a preset ratio, a delay parameter is obtained through a fixed-value frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the buffered frame queue.
  • the delay parameter is obtained by a fixed value frame smoothing factor, for example, the fixed value frame smoothing factor can be set to 1/2.
  • the video frame is output in real time.
  • step 1033C the second rendering timestamp is determined using the first rendering timestamp and the delay parameter.
  • step 1033A the implementation method for determining the second rendering timestamp can be found in the description of step 1033A above.
  • This application embodiment dynamically switches frame smoothing strategies for different network scenarios (the network scenarios corresponding to the first time period, the second time period, and the third time period, respectively), avoiding the limitations of a single frame smoothing strategy in complex network environments.
  • This improves the global adaptability of the decoding process to dynamic network states, achieving the beneficial effect of stably controlling the decoding time of video frames and improving the smoothness of video frame decoding and rendering.
  • step 104 the decoding waiting time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame.
  • the sum of the receiving timestamp and decoding time parameter of the video frame is obtained.
  • the decoding time parameter can be determined by statistically analyzing the decoding time of decoded video frames within a preset time period (e.g., the first 2 seconds of the video frame), such as the 95th percentile or average of the decoding time of decoded video frames within the preset time period.
  • the 95th percentile of the decoding time of a video frame is a statistical indicator used to describe the distribution of the time required to decode a video frame over a period of time.
  • the 95th percentile of the decoding time of a video frame means that within the preset time period, 95% of the video frames have a decoding time less than or equal to this value.
  • step 1042 the difference between the second rendering timestamp and the sum is used as the decoding wait time for the video frame.
  • Decoding wait time Second rendering timestamp - Receive timestamp - Decoding time parameter.
  • step 105 the video frame is decoded in response to the video frame being stored in the buffer frame queue for a certain period of time and the decoding waiting time being reached.
  • the video frame is taken out of the buffer frame queue and sent to the decoder, and the decoder is called to decode the video frame.
  • steps 106 to 107 may also be performed, which are described in detail below.
  • step 106 the time difference between the current timestamp and the decoding time of the previous decoding frame is obtained.
  • the decoding time of the previous decoding frame can be either the time when the decoding of the previous frame is completed or the time when the decoding of the previous frame begins.
  • the time interval between the received timestamp of the video frame and the decoding time of the previous decoded frame can also be obtained as the time difference.
  • step 107 in response to a time difference greater than or equal to a preset ratio of the decoding frame interval parameter, a video frame is removed from the buffer frame queue and the video frame is decoded.
  • a video frame is removed from the buffer frame queue (real-time frame output) and the video frame is decoded.
  • a preset ratio e.g., one-half
  • Steps 101 to 107 achieve the selection of a corresponding frame smoothing strategy based on the specific situation of video frames in the cached frame queue (such as the number of video frames, the time period of the current video frame, etc.). This results in different degrees of delay for the first rendering timestamp of different video frames, avoiding the introduction of excessive decoding waiting time in the future.
  • the frame smoothing strategy is decided based on the state of the video frame and is not affected by business scenario-related parameters (such as video encoding algorithm strategy, network transmission algorithm strategy, flow control algorithm strategy, and other business scenario-related parameters). This improves the accuracy of obtaining the decoding waiting time of video frames, achieving the beneficial effect of stably controlling the decoding time of video frames and improving the smoothness of video frame decoding and rendering.
  • the video call process is easily affected by the network transmission environment. For example, in an environment with unstable network transmission, various problems may occur in video playback, such as video stuttering, which seriously affects the video call quality.
  • FIG. 5 is a flowchart illustrating the video frame processing method in a video call scenario provided in this application embodiment, the following description focuses on the receiving device (such as terminal device 200-2) in the video call.
  • the frame smoothing strategy and frame smoothing factor of the video frame are determined by judging the current time period (corresponding to the first time period, second time period and third time period mentioned above), as well as the number of video frames in the current buffer frame queue and the decoding frame interval parameter, thereby determining the delay parameter to delay the first rendering timestamp to obtain the second rendering timestamp, and the decoding waiting time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame and the decoding time parameter.
  • step 201 data packets are received, data packets are combined into video frames, and in response to the video frames satisfying the reference relationship, the video frames are stored in the buffer frame queue.
  • FIG6B is a schematic diagram of the principle of storing video frames into a buffer frame queue provided in the embodiments of this application.
  • the receiving device After receiving data packets, the receiving device performs frame assembly, puts the video frames obtained by frame assembly into a buffer, and sorts them through a frame sorting module. When the video frames meet the reference relationship (as shown in FIG6A, the I0 frame reference frame is successfully received), the video frames are stored into the buffer frame queue for decoding processing of each subsequent video frame.
  • audio data and video frame data can be transmitted independently using protocols such as Real-time Transport Protocol (RTP) or Hypertext Transfer Protocol (HTTP).
  • Video frame data can be processed using the video frame processing method shown in Figure 5, providing a visual experience for the user on the receiving device.
  • Audio data can be captured by the microphone of the sending device, removing noise and unnecessary high- or low-frequency components, detecting silent or low-energy parts, and reducing or eliminating these parts.
  • the sending device's codec converts the analog or digital audio signal into a digital format and compresses it to reduce data size.
  • the encoded audio data is sent to the receiving end using a Real-time Transport Protocol (such as RTP).
  • the receiving end uses the same codec to decode the received encoded audio data back into the original audio signal, which is then played through a speaker, providing an auditory experience for the user on the receiving device.
  • audio data and video frame data can be encapsulated in the same container format for transmission, such as MP4, FLV, and MKV.
  • audio data and video frame data are encapsulated together at the receiving end and during transmission.
  • the audio data and video frame data can be decapsulated and processed separately, such as decoding and rendering video frames, decoding audio data, and performing post-processing such as echo cancellation and noise suppression.
  • audio data and video frame data can be encoded at the sending device and decoded at the receiving device to ensure reliable and efficient data transmission. Furthermore, to handle network jitter and latency, techniques such as buffering, packet loss recovery, and forward error correction can be employed.
  • Figure 6C is a flowchart of the frame smoothing strategy provided in this application embodiment. If the current time period is the first time period (i.e., the jitter time period). If the current time period is the first time period, the process proceeds to delay the first rendering timestamp using the first frame smoothing strategy to obtain the second rendering timestamp. Otherwise, it is determined whether the current time period is the second time period (i.e., the packet loss retransmission time period). If the current time period is the second time period, the process proceeds to delay the first rendering timestamp using the second frame smoothing strategy to obtain the second rendering timestamp.
  • the first time period i.e., the jitter time period
  • the process proceeds to delay the first rendering timestamp using the first frame smoothing strategy to obtain the second rendering timestamp.
  • the second time period i.e., the packet loss retransmission time period
  • the current time is determined to be in the first time period.
  • a preset duration e.g. 2s
  • the time interval threshold e.g. 100ms
  • the receiving time interval of adjacent data packets is less than or equal to the time interval threshold (i.e., the current time is not in the first time period)
  • it is determined whether there is packet loss e.g., whether packet loss occurred within the preset duration. If there is packet loss, the current time is in the second time period.
  • step 204 the first rendering timestamp is delayed based on the frame smoothing strategy to obtain the second rendering timestamp.
  • the implementation method for delaying the first rendering timestamp using the first frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031A to 1033A above; the implementation method for delaying the first rendering timestamp using the second frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031B to 1033B above; the implementation method for delaying the first rendering timestamp using the third frame smoothing strategy to obtain the second rendering timestamp can be found in the description of steps 1031C to 1033C above, and will not be repeated here.
  • step 205 the decoding wait time of the video frame is determined based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter.
  • the implementation of determining the decoding wait time of a video frame can be found in the description of step 104 above, and will not be repeated here.
  • step 206 the video frame is decoded in response to the video frame being stored in the buffer frame queue for a certain period of time and the decoding waiting time being reached.
  • the video frame is taken out of the buffer frame queue and sent to the decoder.
  • the decoder is then called to decode and render the video frame, so that the video frame is displayed on the receiving device.
  • Steps 201 to 206 achieve the selection of a corresponding frame smoothing strategy based on the specific situation of video frames in the cached frame queue (such as the number of video frames, the time period of the current video frame, etc.). This results in different degrees of delay for the first rendering timestamp of different video frames, minimizing the introduction of decoding waiting time and stably controlling the decoding time of video frames. This achieves the beneficial effects of reducing the stuttering rate of user video calls, improving the smoothness of video calls, and enhancing the user experience.
  • the video frame processing method provided in this application reduces the stuttering rate of online video calls from 5.2% to 3.2%, a reduction of 38%. This improves the smoothness of real-time video calls for users in weak network scenarios.
  • steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.
  • the video frame processing device 455 stored in the memory 450-1 may include:
  • the prediction processing module 4551 is configured to predict the first rendering timestamp of the video frame based on the acquisition timestamp of the video frame in the cached frame queue.
  • the frame smoothing processing module 4552 is configured to determine a frame smoothing strategy based on the video frames in the cached frame queue, wherein the frame smoothing strategy is used to predict the delay parameter of the first rendering timestamp by the receiving time interval of adjacent data packets, and the data packets are used to carry the video frames.
  • the frame smoothing processing module 4552 is further configured to determine the delay parameter based on the frame smoothing strategy, and delay the first rendering timestamp based on the delay parameter to obtain a second rendering timestamp.
  • the frame smoothing processing module 4552 is further configured to determine the decoding waiting time of the video frame based on the second rendering timestamp, the receiving timestamp of the video frame, and the decoding time parameter of the decoded video frame.
  • the frame output module 4553 is configured to decode the video frame in response to the video frame being stored in the cached frame queue for a duration equal to the decoding waiting time.
  • the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval between adjacent data packets is greater than a time interval threshold, determine that the current moment is within a first time period, and determine to apply a first frame smoothing strategy to the buffered frame queue before the next slide of the time window, wherein the starting point of the time window is a historical moment at a preset time distance from the current moment, the ending point of the time window is the current moment, and the first frame smoothing strategy is used to delay the video frames within the first time period based on a frame smoothing factor that changes synchronously with the number of video frames in the buffered frame queue.
  • the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, determine that the current time is in a second time period, and determine to apply a second frame smoothing strategy to the buffered frame queue before the next slide of the time window, wherein the starting point of the time window is a historical time that is a preset time interval away from the current time, the ending point of the time window is the current time, and the second frame smoothing strategy is used to delay the video frames in the buffered frame queue based on the decoding frame interval parameter within the second time period.
  • the frame smoothing processing module 4552 is further configured to determine that the current time is in a second time period if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, and at least one data packet is lost during transmission within the current time window.
  • the frame smoothing processing module 4552 is further configured to, for multiple data packets received within the current time window, if the reception time interval of adjacent data packets is less than or equal to a time interval threshold, determine that the current time is in a third time period, and determine to apply a third frame smoothing strategy to the buffer frame queue before the next slide of the time window, wherein the starting point of the time window is a historical time that is a preset time interval away from the current time, the ending point of the time window is the current time, and the third frame smoothing strategy is used to delay a specific video frame in the buffer frame queue based on a fixed frame smoothing factor within the third time period, wherein the time difference between the reception timestamp of the specific video frame and the decoding time of the previous decoded frame is less than a preset proportion of the decoding frame interval parameter.
  • the frame smoothing processing module 4552 is further configured to determine that the current time is in the third time period if the reception time interval of adjacent data packets is less than or equal to the time interval threshold, and the data packets are not lost during transmission within the current time window.
  • the frame smoothing processing module 4552 is further configured to determine the frame smoothing factor based on the number of video frames in the cached frame queue, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; obtain the delay parameter through the frame smoothing factor; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter.
  • the frame smoothing processing module 4552 is further configured to determine an initialized frame smoothing factor based on a decoded frame interval parameter; obtain the time interval between the video frame and the previous video frame entering the buffer frame queue; and adjust the initialized frame smoothing factor based on the number of video frames in response to the time interval being less than a time interval threshold or the number of video frames being greater than a first buffer frame number threshold, thereby determining the frame smoothing factor.
  • the frame smoothing processing module 4552 is further configured to, in response to the number of video frames being greater than a second cached frame number threshold, configure the initialized frame smoothing factor according to a fixed value to obtain the frame smoothing factor, wherein the second cached frame number threshold is greater than the first cached frame number threshold; and in response to the number of video frames being less than or equal to the second cached frame number threshold, perform reciprocal processing on the initialized frame smoothing factor to obtain the frame smoothing factor.
  • the frame smoothing processing module 4552 is further configured to obtain the product of the frame smoothing factor and the decoding frame interval parameter; determine the sum of the minimum value of the product and the delay control parameter with the current timestamp; and calculate the difference between the sum and the first rendering timestamp to obtain the delay parameter.
  • the frame smoothing processing module 4552 is further configured to, in response to the slow smoothing frame number being greater than the slow smoothing frame number threshold, decrease the slow smoothing frame number and determine a frame smoothing factor according to the decoding frame interval parameter, wherein the slow smoothing frame number is used to characterize the number of video frames to be processed after the video frame in the cached frame queue, the video frames to be processed are video frames that need to be delayed by the second frame smoothing strategy, the slow smoothing frame number is decreased from an initial value, and the decrease in the initial value is consistent with the number of video frames smoothed by the second frame smoothing strategy; obtain the delay parameter through the frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter
  • the frame smoothing processing module 4552 is further configured to: in response to the slow smoothing frame number decreasing to the slow smoothing frame number threshold, obtain a new initial value based on the frame sorting buffer duration, wherein the new initial value is the ratio of the frame sorting buffer duration to the decoding frame interval parameter; determine a new slow smoothing frame number based on the new initial value; return the steps of decreasing the slow smoothing frame number in response to the slow smoothing frame number being greater than the slow smoothing frame number threshold, and determining a frame smoothing factor according to the decoding frame interval parameter.
  • the frame smoothing processing module 4552 is further configured to obtain the product of the frame smoothing factor and the decoding frame interval parameter, sum the product with the current timestamp to obtain a summation result, and subtract the summation result from the first rendering timestamp to obtain the delay parameter.
  • the frame smoothing processing module 4552 is further configured to: obtain the time difference between the received timestamp of the video frame and the decoding time of the previous decoded frame; in response to the time difference being less than a preset proportion of the decoding frame interval parameter, obtain the delay parameter through the fixed value frame smoothing factor, wherein the frame smoothing factor is used to represent the ratio of the received frame rate to the output frame rate of the cached frame queue; and determine the second rendering timestamp through the first rendering timestamp and the delay parameter.
  • the frame output module 4553 is further configured to obtain the time difference between the current timestamp and the decoding time of the previous decoded frame; in response to the time difference being greater than or equal to a preset proportion of the decoding frame interval parameter, remove the video frame from the buffer frame queue and decode the video frame.
  • the prediction processing module 4551 is further configured to obtain the received frame rate level of the video frame; obtain the received time difference of the video frame; and determine the first rendering timestamp of the video frame based on the received frame rate level, the received time difference, and the acquisition timestamp.
  • the prediction processing module 4551 is further configured to obtain the video frame receiving frame rate within a period corresponding to the video frame, map the video frame receiving frame rate to a plurality of preset frame rate levels, and use the mapped frame rate level as the receiving frame rate level.
  • the prediction processing module 4551 is further configured to obtain the difference between the reception timestamp of a plurality of video frames and the sum of the acquisition timestamp, round-trip delay and transmission time of the video frames; and to use the average value of the differences corresponding to the plurality of video frames as the reception time difference.
  • the prediction processing module 4551 is further configured to obtain the reciprocal of the received frame rate level at a preset multiple; obtain the sum between the acquisition timestamp and the reciprocal; and perform summation processing on the sum and the received time difference to obtain the first rendering timestamp.
  • the frame smoothing processing module 4552 is further configured to obtain the sum of the receiving timestamp of the video frame and the decoding time parameter; and to use the difference between the second rendering timestamp and the sum as the decoding waiting time of the video frame.
  • the video frame processing device 456 stored in the memory 450-2 may include:
  • the processing module 4561 is configured to acquire a video frame to be encoded and encode the video frame to obtain an encoded video frame, wherein the encoded video frame is decoded by the video frame processing method described above in the embodiments of this application.
  • This application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium.
  • the processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video frame processing method described in this application embodiment.
  • This application provides a computer storage medium storing executable instructions.
  • the processor will execute the video frame processing method provided in this application, for example, the video frame processing method shown in any of the figures 3A to 3M.
  • the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.
  • computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
  • computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).
  • HTML Hyper Text Markup Language
  • computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.
  • the first rendering timestamp of the video frame is obtained, providing a reliable time reference benchmark for the subsequent calculation of decoding waiting time.
  • an adaptive decision-making process for the frame smoothing strategy is realized.
  • the second rendering timestamp of the video frame is determined, realizing the selection of the corresponding frame smoothing strategy according to the specific situation of the video frames in the cache frame queue.
  • the decoding waiting time of the video frame is quantified, thereby improving the matching degree between the decoding waiting time of the video frame in the cache frame queue and the actual decoding requirements.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本申请公开了一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品；方法包括：基于缓存帧队列中视频帧的采集时间戳，预测视频帧的第一渲染时间戳；根据缓存帧队列中的视频帧，确定帧平滑策略，其中，帧平滑策略用于通过相邻的数据包的接收时间间隔，预测第一渲染时间戳的延时参数，数据包用于承载视频帧；基于帧平滑策略确定延时参数，基于延时参数对第一渲染时间戳进行延时，得到第二渲染时间戳；基于第二渲染时间戳、视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定视频帧的解码等待时长；响应于视频帧在缓存帧队列中的存储时长达到解码等待时长，对视频帧进行解码。

Description

一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品
相关申请的交叉引用
本申请基于申请号为2024109239219、申请日为2024年7月10日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及视频处理技术领域，尤其涉及一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品。
背景技术
在如视频会议、屏幕共享或者直播等实时视频场景中，需要通过网络进行传输视频帧数据，在网络传输不稳定的环境中，视频播放会出现多种问题，例如视频卡顿等。
相关技术中的对于视频帧的实时传输处理方法(如网络实时通信方法(Web Real-Time Communications，WebRTC))与业务实际场景相关联，如对于视频编码算法策略、网络传输算法策略、流控算法策略等参数由于业务场景间的差异，需要进行参数的调优，直接使用会存在对于视频帧的解码等待时长的预估偏差，从而引入较大的延时，难以稳定控制视频帧的解码时间，影响了视频帧解码的流畅度。
发明内容
本申请实施例提供一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品，能够稳定控制视频帧的解码时间，从而提升视频帧解码及渲染的流畅度。
本申请实施例提供一种视频帧的处理方法，应用于电子设备，包括：
基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳；
根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，其中，所述帧平滑策略用于通过相邻的数据包的接收时间间隔，预测所述第一渲染时间戳的延时参数，所述数据包用于承载所述视频帧；
基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳；
基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长；
响应于所述视频帧在所述缓存帧队列中的存储时长达到所述解码等待时长，对所述视频帧进行解码。
本申请实施例提供一种视频帧的处理方法，应用于电子设备，包括：
获取待编码视频帧，并对所述待编码视频帧进行编码，得到编码后的视频帧，其中，所述编码后的视频帧通过本申请实施例上述的视频帧的处理方法进行解码。
本申请实施例提供一种处理音/视频码流的方法，应用于电子设备，其中，所述音/视频码流根据本申请实施例上述的视频帧的处理方法产生，或者基于本申请实施例上述的视频帧的处理方法进行解码。
本申请实施例提供一种视频帧的处理装置，包括：
预测处理模块，配置为基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳；
帧平滑处理模块，配置为根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，其中，所述帧平滑策略用于通过相邻的数据包的接收时间间隔，预测所述第一渲染时间戳的延时参数，所述数据包用于承载所述视频帧；
所述帧平滑处理模块，还配置为基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳；
所述帧平滑处理模块，还配置为基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长；
出帧模块，配置为响应于所述视频帧在所述缓存帧队列中的存储时长达到所述解码等待时长，对所述视频帧进行解码。
本申请实施例提供一种视频帧的处理装置，包括：
处理模块，配置为获取待编码视频帧，并对所述待编码视频帧进行编码，得到编码后的视频帧，其中，所述编码后的视频帧通过本申请实施例上述的视频帧的处理方法进行解码。
本申请实施例提供一种电子设备，所述电子设备包括：
存储器，配置为存储计算机可执行指令；
处理器，配置为执行所述存储器中存储的计算机可执行指令或者计算机程序时，实现本申请实施例提供的视频帧的处理方法。
本申请实施例提供一种计算机存储介质，所述计算机存储介质中存储有可执行指令，所述计算机可执行指令被处理器执行时，实现本申请实施例提供的视频帧的处理方法。
本申请实施例提供一种计算机程序产品，包括计算机程序或计算机可执行指令，所述计算机程序或计算机可执行指令被处理器执行时，实现本申请实施例提供的视频帧的处理方法。
本申请实施例至少包括以下有益效果：通过利用采集时间戳的时序稳定性作为基准，获取视频帧的第一渲染时间戳，为后续解码等待时长的计算提供了可靠的时间参考基准，通过根据缓存帧队列中的视频帧获取相应的帧平滑策略，实现了帧平滑策略的自适应决策过程，通过帧平滑策略和第一渲染时间戳，确定视频帧的第二渲染时间戳，实现了根据缓存帧队列中视频帧的具体情况选择对应的帧平滑策略，使得对不同的视频帧的第一渲染时间戳进行延时的程度不同，避免单一帧平滑策略导致的解码等待时长累积而引入过多的解码等待时长，最后，通过第二渲染时间戳、视频帧的接收时间戳和已解码视频的解码耗时参数，量化视频帧的解码等待时长，从而提升视频帧在缓存帧队列中的解码等待时长与实际解码需求的匹配度，综上所述，通过根据缓存帧队列中视频帧的状态进行平滑策略的决策，提升了获取视频帧的解码等待时长的准确性，达到了稳定控制视频帧的解码时间，提升视频帧解码及渲染的流畅度的有益效果。
附图说明
图1是本申请实施例提供的视频帧的处理系统架构的结构示意图；
图2A是本申请实施例提供的电子设备的第一结构示意图；
图2B是本申请实施例提供的电子设备的第二结构示意图；
图3A是本申请实施例提供的视频帧的处理方法的第一流程示意图；
图3B是本申请实施例提供的视频帧的处理方法的第二流程示意图；
图3C是本申请实施例提供的视频帧的处理方法的第三流程示意图；
图3D是本申请实施例提供的视频帧的处理方法的第四流程示意图；
图3E是本申请实施例提供的视频帧的处理方法的第五流程示意图；
图3F是本申请实施例提供的视频帧的处理方法的第六流程示意图；
图3G是本申请实施例提供的视频帧的处理方法的第七流程示意图；
图3H是本申请实施例提供的视频帧的处理方法的第八流程示意图；
图3I是本申请实施例提供的视频帧的处理方法的第九流程示意图；
图3J是本申请实施例提供的视频帧的处理方法的第十流程示意图；
图3K是本申请实施例提供的视频帧的处理方法的第十一流程示意图；
图3L是本申请实施例提供的视频帧的处理方法的第十二流程示意图；
图3M是本申请实施例提供的视频帧的处理方法的第十三流程示意图；
图4是本申请实施例提供的时间滑窗的示意图；
图5是本申请实施例提供的视频通话场景下的视频帧的处理方法的流程示意图；
图6A是本申请实施例提供的视频帧的处理的原理的示意图；
图6B是本申请实施例提供的视频帧存入缓存帧队列的原理的示意图；
图6C是本申请实施例提供的确定帧平滑策略的流程示意图。
需要指出，上述的“第一”、“第二”仅用于区分不同的方案，不代表用于区分方案的优劣程度或在实施过程中的优先级。
具体实施方式
为了使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请作进一步地详细描述，所描述的实施例不应视为对本申请的限制，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本申请保护的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
在以下的描述中，所涉及的术语“第一\第二\第三”仅仅是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本申请实施例能够以除了在这里图示或描述的以外的顺序实施。
本申请实施例中，术语“模块”或“单元”是指有预定功能的计算机程序或计算机程序的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
除非另有定义，本文所使用的所有的技术和科学术语与属于本申请实施例的技术领域的技术人员通常理解的含义相同。本文中所使用的术语只是为了描述本申请实施例的目的，不是旨在限制本申请实施例。
本申请实施例中相关数据收集处理在实例应用时应该严格根据相关国家法律法规的要求，获取个人信息主体的知情同意或单独同意，并在法律法规及个人信息主体的授权范围内，开展后续数据使用及处理。
对本申请实施例进行进一步详细说明之前，对本申请实施例中涉及的名词和术语进行说明，本申请实施例中涉及的名词和术语适用于如下的解释。
1)缓存帧队列，也称帧平滑队列，是一种数据结构，用于存储待解码的视频帧，直到它们可以被从缓存帧队列中读取，以供解码器进行解码。
2)采集时间戳，是指视频帧被采集设备采集的时刻所对应的时间标记。采集时间戳用于表示视频帧的生成时间，以便对视频帧进行时间顺序的排序和相关的时间性分析。
3)第一渲染时间戳，也称预估渲染时间戳，是指在视频渲染过程中，预测完成视频帧的渲染任务的时间点。
4)第二渲染时间戳，也称期望渲染时间戳，是指在视频渲染过程中，在第一渲染时间戳的基础上进行延时得到的预测完成视频帧的渲染任务的时间点，用于规划渲染任务的进度，确保视频帧能够在预定的时间内完成渲染。
5)帧平滑策略，用于通过对视频帧进行延时，从而实现平滑出帧的策略，以减少或消除视频接收端设备的视频帧渲染不均匀，视频画面忽快忽慢带来的卡顿等问题。
6)平滑出帧，是一种通过控制已接收视频帧的解码等待时长，使视频帧的渲染时间尽可能稳定的技术，从而使视频看起来更加流畅。
7)接收时间戳，指视频帧数据在到达接收端设备时，接收端设备记录下的时间标记。
8)当前时间戳，指从特定时刻(例如Unix纪元，即1970年1月1日午夜)开始，到当前时刻经过的秒数或者毫秒数。当前时间戳可以通过编程语言中的时间库来获取，如Python的datetime库或者JavaScript的Date对象。
9)解码等待时长，指视频帧从进入缓存帧队列到移出以调用解码器进行解码时的时间间隔，也就是视频帧在缓存帧队列中的存储时长，例如，当视频帧在缓存帧队列中的存储时长达到解码等待时长时，将视频帧从缓存帧队列取出转入解码器进行解码。
10)网络抖动，指在计算机网络中，视频帧的数据包从发送端设备到接收端设备的传输过程中，由于网络状况、流量拥塞等情况，导致数据包到达时间的不规则性和不稳定，表现为相邻的数据包到达接收端设备的时间间隔的变化。
11)丢包重传，是一种网络传输机制，用于在视频帧的数据包在网络中丢失时进行恢复。当发送端设备发送数据包后，接收端设备会发送确认包也即确认应答(Acknowledgement，ACK)表示已收到。如果接收端设备在期望时间内没有收到某个数据包(如根据数据包的序列号判断当前数据包是否接收到)，它会发送否定确认包也即否定应答(Negative Acknowledgement，NACK)告知发送端设备。
12)等待进入缓存帧队列的视频帧，指在缓冲区(JitterBuffer)中的视频帧，当视频帧满足参考关系时，将从缓冲区转移到缓存帧队列。
其中，参考关系是指视频帧(如P帧、B帧)在解码时，需依赖已接收并解码完成的参考帧(如I帧、已解码的P帧等)，若当前帧的参考帧未接收、未解码或解码失败，则当前帧不满足参考关系，需缓存至缓冲区，等待进入缓存帧队列；若当前帧的参考帧就绪(如接收且解码完成)，则当前帧满足参考关系，可从缓冲区转移至缓存帧队列。
例如，在丢包重传和网络抖动下，当接收到的视频帧不满足参考关系时(即当前接收的视频帧的参考帧还没有被接收到时，例如当前接收的视频帧为P2，P2对应的在前的参考帧可以表示为I0，不同的视频帧可以对应同一个参考帧)，会一直缓存在接收端设备的缓冲区中。
13)参考帧，是指在视频帧序列中用于帮助解码后续视频帧的帧，即视频帧的解码需要依赖在前的参考帧。这些参考帧被编码器用来生成预测信息，解码器则使用这些预测信息来重建视频图像。
14)解码帧，指已经解码成功的视频帧。例如，从缓存帧队列中移出视频帧后，调用解码器对移出的视频帧进行解码，解码成功后，移出的视频帧记为解码帧。
15)解码帧间隔参数，是解码帧的解码时间之间的间隔。例如，在距离当前时刻预设时长的历史时间段内，相邻的两个解码帧(也就是依次解码的两个视频帧)之间的解码间隔的统计数，例如平均值，分位数等。
16)第一时间段，也称抖动时间段，是指在视频帧的数据包在传输过程中，由于网络延迟的变化或其他原因，存在如下情况的时间范围：相邻数据包到达接收端设备的时间间隔大于预设的时间间隔阈值(如100ms)。例如，当时间滑窗长度为2s时，若在2s的时间内接收的相邻的数据包到达接收端设备的时间间隔中存在大于预设的时间间隔阈值的情况，则时间滑窗对应的2s范围为第一时间段。
17)第二时间段，也称丢包重传时间段，是指在视频帧的数据包传输过程中，数据包到达接收端设备的时间间隔小于或等于预设的时间间隔阈值(如100ms)，且有数据包在传输过程中丢失时，从重新传输数据包开始至重新传输的数据包到达接收端设备的时间范围。例如，在视频会议场景下，发送端设备正在发送一系列视频帧，每个帧都被分割成多个数据包进行传输，数据包在传输过程中的到达时间间隔小于或等于预设的时间间隔阈值(如100ms)，但其中一个视频帧的数据包在传输过程中丢失。发送端设备在接收到重传请求后，会立即开始重传丢失的数据包。从发送端设备发出重传数据包开始，直到接收端设备成功接收到这些数据包并将其放入视频流中的时刻为止，这段时间就是第二时间段。
18)第三时间段，是指在视频帧的数据包传输过程中，数据包到达接收端设备的时间间隔小于或等于预设的时间间隔阈值(如100ms)，并且无数据包在传输过程中丢失的时间范围。例如，假设发送端设备正在发送一系列视频帧，每个帧都被分割成多个数据包进行传输，在一段时间内(如5s)，数据包被成功地传输到接收端设备，在这个时间段内，没有数据包丢失，并且数据包在传输过程中的到达时间间隔小于或等于预设的时间间隔阈值(如100ms)，这个时间段为第三时间段。
19)接收帧率，表示每秒内进入缓存帧队列的视频帧的数量。
20)出帧帧率，表示每秒内从缓存帧队列移出以调用解码器进行解码的视频帧的数量。
相关技术中的对于视频帧的实时传输处理方法(如WebRTC等)与业务实际场景相关联，如对于视频编码算法策略、网络传输算法策略、流控算法策略等参数由于业务场景间的差异，需要进行参数的调优，直接使用会存在抖动的预估偏差，从而引入较大的延时，难以稳定控制视频帧的解码时间，影响了视频帧的解码流畅度。
相关技术的基于简单统计的视频帧的处理方法，统计相邻两个或多个视频帧的接收时间间隔，例如平均数、或者方差等统计变量，若当前收到的视频帧的统计变量在对应的阈值之外，则对当前视频帧增加一定量的等待延时后，再送入解码器进行解码和渲染，例如统计最近1s(15～30帧)的视频帧的接收间隔的平均方差，在网络抖动的情况下，当视频帧不满足参考关系时(如参考帧还未被接收)，视频帧将快速堆积在缓冲区，在满足参考关系(接收到对应的参考帧)后会快速出帧，此时方差快速的增加到某个阈值，经过视频帧平均间隔的1/2时间后，再送入解码器进行解码和渲染。但由于视频帧的接收时间会随着网络抖动等因素变化，无法以同一个时间基准来进行平滑出帧，影响了视频帧解码的流畅度。
基于此，本申请实施例提供一种视频帧的处理方法、装置、设备、计算机可读存储介质和计算机程序产品，能够稳定控制视频帧的解码时间，从而提升视频帧解码及渲染的流畅度。
本申请实施例提供的电子设备可以实施为笔记本电脑、平板电脑、台式计算机、机顶盒、移动设备(例如，移动电话，便携式音乐播放器，个人数字助理，专用消息设备，便携式游戏设备)、智能手机、智能音箱、智能语音交互设备、智能家电、智能手表、智能电视、车载终端、智能耳机、飞行器等各种类型的终端设备。下面，将说明电子设备实施为终端设备时的示例性应用。
参见图1，图1是本申请实施例提供的视频帧的处理系统架构的结构示意图，图1中涉及服务器100、终端设备200-1、终端设备200-2及网络300。终端设备200-1和终端设备200-2通过网络300连接服务器100，其中，网络300可以是广域网或者局域网，又或者是二者的组合。
在一些实施例中，终端设备或服务器可以通过运行各种计算机可执行指令或计算机程序来实现本申请实施例提供的视频帧的处理方法。举例来说，计算机可执行指令可以是微程序级的命令、机器指令或软件指令。计算机程序可以是操作系统中的原生程序或软件模块；可以是本地(Native)应用程序(APP，Application)，即需要在操作系统中安装才能运行的程序，如即时通信客户端；也可以是嵌入至任意APP中的小程序，即只需要下载到浏览器环境中就可以运行的程序。总而言之，上述的计算机可执行指令可以是任意形式的指令，上述计算机程序可以是任意形式的应用程序、模块或插件。
在一些实施例中，服务器100可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(Content Delivery Network，CDN)、以及大数据和人工智能平台等基础云计算服务的云服务器，其中，云服务可以是交互处理服务，供终端进行调用。
在一些实施例中，本申请实施例提供的视频帧的处理系统可以由服务器和终端设备协同实现。例如在视频通话过程中，参见图1，终端设备200-1通过即时通信客户端与终端设备200-2视频通信，终端设备200-1的传感器设备(例如终端200-1的内置的摄像头)采集视频帧，将视频帧分为多个数据包后经由即时通信客户端的后台的服务器100传输到终端设备200-2，响应于终端设备200-2接收到终端设备200-1发出的视频帧的数据包，将数据包组合为视频帧，存入缓存帧队列，接下来，终端设备200-2通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，送入终端设备200-2的解码器中，调用解码器对视频帧进行解码，最后通过视频渲染将解码后的视频帧显示在终端设备200-2的界面。
以视频通话场景为例，终端设备用于采集视频帧、将视频帧封装为数据包发送至服务器，并接收服务器转发的对端视频数据包、组合为视频帧后存入缓存帧序列，通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，调用解码器对视频帧进行解码；服务器用于中转终端设备间的视频数据包，实现端到端的实时数据传输链路建立与维护。
以多人视频会议场景为例，终端设备用于采集本地视频流并编码为数据包、接收服务器分发的其他参会者视频数据包，组合为视频帧后存入缓存帧序列，通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，调用解码器对视频帧进行解码；服务器用于汇聚多路视频流、执行信令交互管理(如参会者加入/退出通知)及数据分发(如将某一端视频流广播至其他终端)。
以实时直播场景为例，终端设备(主播端)用于采集直播画面并拆分为数据包上传至服务器；终端设备(观众端)用于接收服务器推送的直播视频数据包，组合为视频帧后存入缓存帧序列，通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，调用解码器对视频帧进行解码；服务器用于接收主播端视频流、执行转码(如多分辨率适配)、通过内容分发网络(Content Delivery Network，CDN)分发至观众端。
在一些实施例中，本申请实施例提供的视频帧的处理系统可以由服务器单独实现。例如服务器100获取数据包，并将数据包组合为视频帧，存入缓存帧队列，接下来，服务器100通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，送入服务器100的解码器中，调用解码器对视频帧进行解码，得到解码后的视频帧。
在一些实施例中，本申请实施例提供的视频帧的处理系统可以由终端设备实现。例如终端设备200-1的采集视频帧，将视频帧分为多个数据包后传输给终端设备200-2，响应于终端设备200-2接收到终端设备200-1发出的视频帧的数据包，终端设备200-2将数据包组合为视频帧，存入缓存帧队列，接下来，终端设备200-2通过本申请实施例提供的视频帧的处理方法对视频帧进行帧平滑处理，得到视频帧的解码等待时长，当视频帧在缓存帧队列中的存储时长达到等待时长时，将视频帧从缓存帧队列出帧，送入终端设备200-2的解码器中，调用解码器对视频帧进行解码，最后通过视频渲染将解码后的视频帧显示在终端设备200-2的界面。
以电子设备作为接收端设备，对编码后的视频帧进行处理为例，参见图2A，图2A是本申请实施例提供的电子设备的第一结构示意图，电子设备可以实施为上文的终端设备200-1、终端设备200-2或服务器100，图2A所示的电子设备400-1包括：至少一个处理器410-1、存储器450-1、至少一个网络接口420-1和用户接口430-1。电子设备400-1中的各个组件通过总线系统440-1耦合在一起。可理解，总线系统440-1用于实现这些组件之间的连接通信。总线系统440-1除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图2中将各种总线都标为总线系统440-1。
处理器410-1可以是一种集成电路芯片，具有信号的处理能力，例如通用处理器、数字信号处理器(Digital Signal Processor，DSP)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等，其中，通用处理器可以是微处理器或者任何常规的处理器等。
用户接口430-1包括使得能够呈现媒体内容的一个或多个输出装置431-1，包括一个或多个扬声器和/或一个或多个视觉显示屏。用户接口430-1还包括一个或多个输入装置432-1，包括有助于用户输入的用户接口部件，比如键盘、鼠标、麦克风、触屏显示屏、摄像头、其他输入按钮和控件。
存储器450-1可以是可移除的，不可移除的或其组合。示例性的硬件设备包括固、态存储器，硬盘驱动器，光盘驱动器等。存储器450-1可选地包括在物理位置上远离处理器401-1的一个或多个存储设备。
存储器450-1包括易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。非易失性存储器可以是只读存储器(ROM，Read Only Memory)，易失性存储器可以是随机存取存储器(Random Access Memory，RAM)。本申请实施例描述的存储器450-1旨在包括任意适合类型的存储器。
在一些实施例中，存储器450-1能够存储数据以支持各种操作，这些数据的示例包括程序、模块和数据结构或者其子集或超集，下面示例性说明。
操作系统451-1，包括用于处理各种基本系统服务和执行硬件相关任务的系统程序，例如框架层、核心库层、驱动层等，用于实现各种基础业务以及处理基于硬件的任务；
网络通信模块452-1，用于经由一个或多个(有线或无线)网络接口420-1到达其他电子设备，示例性的网络接口420-1包括：蓝牙、无线相容性认证(WiFi)、和通用串行总线(Universal Serial Bus，USB)等；
呈现模块453-1，用于经由一个或多个与用户接口430-1相关联的输出装置431-1(例如，显示屏、扬声器等)使得能够呈现信息(例如，用于操作外围设备和显示内容和信息的用户接口)；
输入处理模块454-1，用于对一个或多个来自一个或多个输入装置432-1之一的一个或多个用户输入或互动进行检测以及翻译所检测的输入或互动。
在一些实施例中，本申请实施例提供的装置可以采用软件方式实现，图2A示出了存储在存储器450-1中的视频帧的处理装置455，其可以是程序和插件等形式的软件，包括以下软件模块：预测处理模块4551，帧平滑处理模块4552和出帧模块4553，这些模块是逻辑上的，因此根据所实现的功能可以进行任意的组合或进一步拆分。将在下文中说明各个模块的功能。
以电子设备作为发送端设备，对待编码视频帧进行处理为例，参见图2B，图2B是本申请实施例提供的电子设备的第二结构示意图，电子设备可以实施为上文的终端设备200-1、终端设备200-2或服务器100，图2B所示的电子设备400-2包括：至少一个处理器410-2、存储器430-2和至少一个网络接口420-2。电子设备400-2中的各个组件通过总线系统440-2耦合在一起。可理解，总线系统440-2用于实现这些组件之间的连接通信。总线系统440-2除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图2B中将各种总线都标为总线系统440-2。对于处理器410-2和存储器430-2的具体说明参见上文，这里不再赘述。
在一些实施例中，本申请实施例提供的装置可以采用软件方式实现，图2B示出了存储在存储器430-2中的视频帧的处理装置456，其可以是程序和插件等形式的软件，包括以下软件模块：处理模块4561。将在下文中说明模块的功能。
在另一些实施例中，本申请实施例提供的装置可以采用硬件方式实现，作为示例，本申请实施例提供的装置可以是采用硬件译码处理器形式的处理器，其被编程以执行本申请实施例提供的视频帧的处理方法，例如，硬件译码处理器形式的处理器可以采用一个或多个应用专用集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、可编程逻辑器件(Programmable Logic Device，PLD)、复杂可编程逻辑器件(Complex Programmable Logic Device，CPLD)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或其他电子元件。
下面将结合本申请实施例提供的电子设备的示例性应用和实施，以电子设备(如终端设备)为执行主体，说明本申请实施例提供的视频帧的处理方法，其中，可以由上述的终端设备作为发送端设备，执行本申请实施例提供的视频帧的处理方法。
在一些实施例中，获取待编码视频帧，并对所述待编码视频帧进行编码，得到编码后的视频帧，其中，编码后的视频帧通过本申请实施例提供的视频帧的处理方法进行解码。
示例的，发送端设备采集待编码视频帧(或称原始视频帧)，对待编码视频帧进行视频前处理、视频编码和分包后，得到编码后的视频帧的数据包，将采集时间戳和发送耗时等信息随数据包发送给接收端设备，接收端设备通过本申请实施例提供的视频帧的处理方法，对编码后的视频帧进行解码。
其中，视频前处理是指在待编码视频帧被编码之前对其进行的一系列处理步骤。这些处理可以包括色彩校正、亮度/对比度调整、噪声抑制、滤波、图像大小调整等。目的是提高视频帧质量，使其更适合后续的编码处理，或者适应特定的显示需求。在前处理阶段，也可以包括帧率转换、场景检测等高级处理，以优化视频内容和减少编码过程中的复杂度；
视频编码是将待编码视频帧转换成压缩格式以便于存储和传输的过程。例如将待编码视频帧的图像分解成更小的单元(如宏块)，然后对这些单元应用一系列数学变换(如离散余弦变换)以消除数据中的冗余信息。随后，通过量化过程调整变换系数，以减少所需传输的数据量。视频编码还包括将待编码视频帧的图像数据分成不同的层，如空间层、时间层和冗余层，这样可以在保证视频帧质量的同时，提供不同程度的压缩。常用的视频编码标准包括高效视频编码标准(High Efficiency Video Coding，HEVC)和开放媒体联盟编码标准(AOMedia Video 1，AV1)等；
分包是指将编码后的视频帧分割成较小的数据包的过程。这些数据包包括视频帧的一部分，它们被用于通过网络传输到最终用户的设备(对应接收端设备)上。分包的目的是为了适应不同的网络条件，如带宽限制、延迟要求等。通过分包，可以实现更有效的传输控制，比如调整数据包发送的频率、大小和顺序，以优化网络资源的使用，并确保视频流能够在不稳定的网络环境中保持流畅。
下面将结合本申请实施例提供的电子设备的示例性应用和实施，以电子设备(实施为终端设备)为执行主体，说明本申请实施例提供的视频帧的处理方法。参见图3A，图3A是本申请实施例提供的视频帧的处理方法的第一流程示意图，可以由上述的终端设备作为接收端设备执行，将结合图3A示出的步骤进行说明。
在一些实施例中，参见图1，在视频通话过程中，终端设备200-1可以作为发送端设备，终端设备200-2可以作为接收端设备，这里，发送端设备和接收端设备是相对的概念，在视频通信过程中，一个终端设备同时作为发送端和接收端，例如，终端设备200-1的用户1在与终端设备200-2的用户2进行视频通话时，终端设备200-1的摄像头采集用户1的图像，编码为视频帧并发送给终端设备200-2，与此同时，终端设备200-1还接收终端设备200-2发送的视频帧，终端设备200-2发送的视频帧是对用户2进行图像采集得到的。这意味着同一设备上的硬件(如摄像头和屏幕)和软件(如编解码器和用户界面)需要协同工作，以实现数据的发送和接收。下文是从接收端设备的角度进行描述的。
在步骤101中，基于缓存帧队列中视频帧的采集时间戳，预测视频帧的第一渲染时间戳。
在一些实施例中，视频帧的发送端设备将每个视频帧的采集时间戳和发送端设备的发送耗时等信息保存在帧头信息中，在发送端设备对待编码视频帧进行视频前处理、视频编码和分包后，得到数据包，将采集时间戳和发送耗时等信息随数据包发送给接收端设备。
在一些实施例中，接收端设备接收数据包，将数据包组合为视频帧(这里的视频帧是编码后的视频帧)，当视频帧满足参考关系后，将视频帧存入缓存帧队列，这里，参考关系指的是视频帧的解码需要依赖前面的参考帧，如果当前视频帧的参考帧还没被接收端设备接收到，这个视频帧会一直在缓冲区中等待，直到视频帧的参考帧接收成功后，即满足参考关系后，才会将这个视频帧送入缓存帧队列中，从而进行后续的帧平滑策略的确定等处理。
在一些实施例中，参见图3B，图3A示出的步骤101，可以通过以下步骤1011至步骤1013实现，下面具体说明。
在步骤1011中，获取视频帧的接收帧率档位。
在一些实施例中，获取视频帧对应的时间段内的视频帧接收帧率，将视频帧接收帧率映射到多个预设的帧率档位中，将映射得到的帧率档位作为接收帧率档位。
示例的，可以统计每秒钟(对应上文视频帧对应的时间段)接收的视频帧数量(即视频帧接收帧率)，将视频帧接收帧率映射到流控策略档位上最接近的帧率，流控档位上的帧率档位可以有多个，例如帧率档位可以是发送端设备的视频帧采集帧率或编码帧率，如12FPS、15FPS和20FPS等，假设每秒钟接收的视频帧数量为18帧，流控策略档位上最接近的帧率为20FPS，此时视频帧对应的接收帧率档位为20FPS。
本申请实施例通过将视频帧的接收帧率映射至预设的帧率档位(如12FPS、15FPS等)，能够过滤实时帧率的小幅波动，建立稳定的帧率基准，从而提升本申请实施例提供的视频帧的处理方法对动态网络环境的适应性。
在步骤1012中，获取视频帧的接收时间差。
在一些实施例中，参见图3C，图3B示出的步骤1012，可以通过以下步骤10121至步骤10122实现，下面具体说明。
在步骤10121中，获取多个视频帧的接收时间戳与视频帧的采集时间戳、往返时延和发送耗时之和的差值。
在一些实施例中，可以统计多个视频帧的接收时间戳与视频帧的采集时间戳、往返时延和发送耗时之和的差值。视频帧的发送端设备将每帧视频的采集时间戳和发送端设备的发送耗时等信息保存在帧头信息中，接收端设备收到数据包并进行组帧后，从帧头信息中获取视频帧的采集时间戳和发送耗时。
作为示例，在视频通话场景中，统计范围可以是视频通话开始至当前时刻，也可以视频通话过程的一个或多个采样周期(例如2s)，统计一个或多个采样周期时间内的差值，一个多个采样周期，可以是从当前时刻往前追溯的，从而能够准确反映当前的网络状况。
示例的，每个视频帧对应的差值可以表示为：差值＝视频帧的接收时间戳—视频帧的采集时间戳—发送耗时—往返时延。
在步骤10122中，将多个视频帧对应的差值的平均值作为接收时间差。
承接步骤10121中的示例，将上述统计范围内的多个视频帧对应的差值的平均值作为接收时间差。
本申请实施例通过统计多个视频帧的接收时间戳与采集时间戳、往返时延及发送耗时之和的差值，并取平均值作为接收时间差，能够动态捕捉网络传输延迟的整体趋势，过滤单帧异常波动的干扰，提高了接收时间差对实时网络状态的表征精度，为后续第一渲染时间戳的预测提供更可靠的延迟补偿基准。
继续参见图3B，在步骤1013中，基于接收帧率档位、接收时间差和采集时间戳，确定视频帧的第一渲染时间戳。
在一些实施例中，参见图3D，图3B示出的步骤1013，可以通过以下步骤10131至步骤10133实现，下面具体说明。
在步骤10131中，获取预设倍数的接收帧率档位的倒数。
在一些实施例中，获取预设倍数(例如1000)的接收帧率档位的倒数，其中，接收帧率档位例如可以是12FPS、15FPS和20FPS等，接收帧率档位可以根据流控策略的需求进行灵活设置。
以预设倍数为1000为例，预设倍数的接收帧率档位的倒数可以表示为：1000×(1/接收帧率档位)。
在步骤10132中，获取采集时间戳与倒数之间的加和。
在一些实施例中，对采集时间戳和倒数进行加和处理，得到加和。
承接步骤10131中的示例，对采集时间戳和倒数进行加和处理，得到加和，可以表示为：视频帧的采集时间戳+1000×(1/接收帧率档位)＝视频帧的采集时间戳+1000/接收帧率档位。
在步骤10133中，对加和和接收时间差进行求和处理，得到第一渲染时间戳。
承接步骤10132中的示例，确定视频帧对应的第一渲染时间戳可以表示为：第一渲染时间戳＝视频帧的采集时间戳+1000/接收帧率档位+接收时间差，其中，加和是指采集时间戳与倒数之间的加和，即“视频帧的采集时间戳+1000/接收帧率档位”。
继续参见图3A，在步骤102中，根据缓存帧队列中的视频帧，确定帧平滑策略。
在一些实施例中，视频帧是根据接收的数据包进行组合得到的，针对在当前的时间滑窗(按照预设时长配置的，如2s)内接收的多个数据包，若全部或超过预设比例的相邻的数据包的接收时间间隔大于时间间隔阈值(如100ms)，则确定当前时刻处于第一时间段(也称抖动时间段)，并确定在时间滑窗按照预定的步长(如2s)下一次滑动之前，对缓存帧队列应用第一帧平滑策略，其中，当前的时间滑窗的起点是与当前时刻距离预设时长的历史时刻，当前的时间滑窗的终点是当前时刻，第一帧平滑策略用于在第一时间段内，基于与缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对缓存帧队列中的视频帧进行延时。
需要说明的是，“同步变化”指的是帧平滑因子与缓存帧队列中的视频帧数量之间的动态关联关系。具体而言，当缓存帧数量较多时，增大帧平滑因子，从而降低出帧帧率，减少解码压力；反之，当缓存帧数量较少时，减小帧平滑因子，提高出帧帧率，利用现有帧资源填补网络波动间隙，避免解码器空转，从而根据网络状况和缓存状态动态调整解码策略，提升视频播放的流畅度和稳定性。例如，当网络抖动导致数据包接收间隔变长时，缓存帧数量增加，帧平滑因子随之增大，出帧帧率降低，从而缓解网络波动对视频播放的影响。
示例的，在确定当前时刻处于抖动时间段后，即可以对缓存帧队列中的视频帧开始执行第一帧平滑策略，直至长度为2s的时间滑窗中的抖动值(相邻的数据包的接收时间间隔)已经低于预设的时间间隔阈值(例如100ms)，停止对缓存帧队列中的视频帧执行第一帧平滑策略。
示例的，参见图4，图4是本申请实施例提供的时间滑窗的示意图，可以通过滑动时间滑窗算法进行抖动值的测量，首先，设置一个时间滑窗，其长度为2s。在当前的时间滑窗内，每当一个数据包到达时，记录下它的到达时间，这些到达时间处于当前的时间滑窗中。随着时间的推移，时间滑窗会按照一定的步长沿着时间轴滑动，如图4中示出的长度为2s的时间滑窗1按步长1.5s滑动时，得到下一个时间滑窗2，并且时间滑窗1和时间滑窗2之间有重叠，以确保始终包括最近2s内的数据包到达时间。抖动可以通过计算时间滑窗内所有数据包的到达时间差来测量。例如，对于时间滑窗中的每个数据包，计算它与窗口中前一个数据包的到达时间差(相邻的数据包的接收时间间隔)，将这些时间差组合为抖动时间序列，例如，获取图4中示出的时间滑窗1对应的数据包2与数据包1之间的到达时间差，数据包3与数据包2之前的到达时间差以及数据包4与数据包3之前的到达时间差。还可以获取这些时间差的绝对值、均值、中位数或预设分位数(例如95分位数)等，作为当前时间滑窗对应的抖动值，例如获取图4中的时间滑窗1对应的数据包之间的到达时间差的95分位数，以作为时间滑窗1对应的抖动值，随着时间的推移，时间滑窗会持续滑动，记录新的数据包到达时间，并更新抖动值。
在一些实施例中，针对在当前的时间滑窗(按照预设时长配置的，如2s)内接收的多个数据包，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前时刻不处于第一时间段)，则确定当前时刻处于第二时间段(也称丢包重传时间段)，并确定在当前的时间滑窗下一次滑动之前对缓存帧队列应用第二帧平滑策略，其中，当前的时间滑窗的起点是与当前时刻距离预设时长(即时间滑窗的长度)的历史时刻，时间滑窗的终点是当前时刻(滑窗的步长也可以是预设的，每次滑动后，时间滑窗的终点都是当前时刻)，第二帧平滑策略用于在第二时间段内，基于解码帧间隔参数对缓存帧队列中的视频帧进行延迟，解码帧间隔参数是已经解码的视频帧的解码时间之间的间隔，可以是距离当前时刻预设时长的历史时间段内，已经解码的视频帧之间的解码间隔的统计数，例如平均值，预设分位数等。
其中，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前时刻不处于第一时间段)，则确定当前时刻处于第二时间段，可以通过以下方式实现：若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值，且在当前的时间滑窗内至少一个数据包在传输过程中丢失，则确定当前时刻处于第二时间段。
通过当相邻数据包接收时间间隔≤时间间隔阈值且当前滑窗内至少有一个数据包丢失时，确定当前时刻处于第二时间段(丢包重传时间段)，这既能精准定位丢包重传场景以避免将正常网络波动误判为丢包重传，又能在触发第二帧平滑策略时更具针对性，减少无丢包时的性能损耗。
在一些实施例中，针对在当前的时间滑窗(按照预设时长配置的，如2s)内接收的多个数据包，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前时刻不处于第一时间段)，则确定当前时刻处于第三时间段，并确定在时间滑窗下一次滑动之前对缓存帧队列应用第三帧平滑策略，其中，时间滑窗的起点是与当前时刻距离预设时长(即时间滑窗的长度)的历史时刻，时间滑窗的终点是当前时刻，第三帧平滑策略用于在第三时间段内，基于固定值的帧平滑因子对缓存帧队列中的特定视频帧进行延时，特定视频帧指的是满足如下条件的视频帧：在缓存帧队列中接收时间戳与上一个解码帧的解码时刻的时间差，小于预设比例的解码帧间隔参数。
这里，解码帧的解码时刻可以是解码帧的解码完成的时刻，还可以是解码帧的开始解码的时刻。
其中，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前时刻不处于第一时间段)，则确定当前时刻处于第三时间段，可以通过以下方式实现：若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，且在所述当前的时间滑窗内所述数据包在传输过程中未丢失，则确定当前时刻处于第三时间段。
通过当相邻数据包接收时间间隔≤时间间隔阈值且当前滑窗内没有数据包丢失时，确定当前时刻处于第三时间段，避免了在丢包重传等异常情况下错误应用基于固定帧平滑因子的策略，减少对视频帧不必要的延时处理，同时针对缓存帧队列中接收时间戳与上一个解码帧解码时刻时间差小于预设比例解码帧间隔参数的特定视频帧，实现精准的延时优化，在维持视频流畅播放的同时，有效降低因策略误触发导致的性能损耗和播放延迟。
作为上述的第二帧平滑策略和第三帧平滑策略的一个分情况使用的示例，针对在当前的时间滑窗(按照预设时长配置的，如2s)内接收的多个数据包，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前时刻不处于第一时间段)，且在当前的时间滑窗内有数据包在传输过程中丢失时，则确定当前时刻处于第二时间段(也称丢包重传时间段)，并确定在当前的时间滑窗下一次滑动之前对缓存帧队列应用第二帧平滑策略；若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值，且在当前的时间滑窗内无数据包在传输过程中丢失时，则确定当前时刻处于第三时间段，并确定在当前的时间滑窗下一次滑动之前对缓存帧队列应用第三帧平滑策略。
继续参见图3A，在步骤103中，基于帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳。
在一些实施例中，当帧平滑策略是用于在第一时间段(抖动时间段)内，基于与缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对视频帧进行延时的第一帧平滑策略时，参见图3E，图3A示出的步骤103，可以通过以下步骤1031A至步骤1033A实现，下面具体说明。
在步骤1031A中，根据缓存帧队列中的视频帧数量确定帧平滑因子，其中，帧平滑因子用于表示缓存帧队列的接收帧率与出帧帧率的比例。
在一些实施例中，参见图3F，图3E示出的步骤1031A，可以通过以下步骤10311至步骤10313实现，下面具体说明。
在步骤10311中，根据解码帧间隔参数确定初始化的帧平滑因子。
在一些实施例中，可以根据当前视频帧对应的解码帧间隔参数确定初始化的帧平滑因子。
示例的，解码帧间隔参数是已经解码的视频帧的解码时间之间的间隔，可以是距离当前时刻预设时长的历史时间段内，已经解码的视频帧之间的解码间隔的统计数，例如平均值，预设分位数(如95位分位数)等。这里，解码帧间隔(两个相邻的视频帧分别解码成功时，对应的两个解码成功时刻的时间戳之间的间隔)小于或等于抖动平均值与预设阈值(如500)之间的最小值，抖动平均值指当前的时间滑窗中的相邻的数据包的接收时间间隔的平均值。
在一些实施例中，当解码帧间隔参数大于第一解码帧间隔参数阈值时，对解码帧间隔参数进行最大值约束，使得最大值约束后的解码帧间隔参数等于第一解码帧间隔参数阈值；当解码帧间隔参数小于第一解码帧间隔参数阈值时，对解码帧间隔参数进行最小值约束，使得最大值约束后的解码帧间隔参数等于第二解码帧间隔参数阈值，其中，第一解码帧参数阈值大于第二解码帧间隔参数阈值。
例如，假设第一解码帧间隔参数阈值为125ms，第二解码帧间隔参数阈值为33ms，当解码帧间隔参数的值大于125ms时，则取解码帧间隔参数的值为125ms，当解码帧间隔参数的值小于33ms时，则取解码帧间隔参数的值为33ms。
在一些实施例中，根据解码帧间隔参数对应的预设参数范围，确定解码帧参数对应的初始化的帧平滑因子，其中，解码帧间隔参数对应的预设参数范围的取值越大对应的初始化的帧平滑因子的值越小(这里，帧平滑因子的值指接收帧率与出帧帧率的比值)，即解码帧间隔参数的值与帧平滑因子的值成反比，解码帧间隔参数的值与出帧帧率成正比。
示例的，在相同的接收帧率的情况下，解码帧间隔参数越大，对应的出帧帧率越大，当解码帧间隔参数大于90ms时，初始化的帧平滑因子可以设置为5/4，相当于接收帧率在30FPS左右时，最低以24FPS的出帧帧率去平滑出帧(从缓存帧队列移出以进行解码)；
当解码帧间隔参数在60～90ms之间时，初始化的帧平滑因子可以设置为4/3，相当于接收帧率在30FPS左右时，最低以22.5FPS的出帧帧率出帧；
当解码帧间隔参数小于60ms时，初始化的帧平滑因子可以设置为3/2，相当于接收帧率在30FPS左右时，最低以20FPS的出帧帧率出帧。
当解码帧间隔参数越大时表明解码器需要等待更长的时间来接收和处理下一个视频帧，网络延迟较高，通过根据解码帧间隔参数设置对应的帧平滑因子，使得解码帧间隔参数越大时出帧帧率越快，实现了通过提升出帧帧率，提升视频的流畅度，降低了网络延迟带来的视频卡顿的影响，从而达到提升用户体验的有益效果。
在步骤10312中，获取视频帧与前一帧视频帧进入缓存帧队列之间的时间间隔。
这里，同上文所述，接收端设备接收数据包，将数据包组合为视频帧后，将视频帧存入缓冲区，在当视频帧满足依赖关系时，即视频帧对应的参考帧已被接收时，才将等待进入缓存帧队列的视频帧从缓冲区转入帧缓存队列中，也就是说，在进入缓存帧队列的视频帧之间存在时间间隔。
在步骤10313中，响应于时间间隔小于时间间隔阈值，或视频帧数量大于第一缓存帧数量阈值，根据视频帧数量调整初始化的帧平滑因子，确定帧平滑因子。
在一些实施例中，当时间间隔小于时间间隔阈值(例如10ms)，或帧缓存队列中的视频帧数量大于第一缓存帧数量阈值(如1)时，根据视频帧数量调整初始化的帧平滑因子，确定帧平滑因子。参见图3G，图3F示出的步骤10313中的根据视频帧数量调整初始化的帧平滑因子，确定帧平滑因子，可以通过以下步骤10314至步骤10315实现，下面具体说明。
在步骤10314中，响应于视频帧数量大于第二缓存帧数量阈值，将初始化的帧平滑因子按照固定值进行配置，得到帧平滑因子，其中，第二缓存帧数量阈值大于第一缓存帧数量阈值。
在一些实施例中，当缓存帧队列中的视频帧数量大于第二缓存帧数量阈值(例如4)时，将初始化的帧平滑因子按照固定值进行配置，得到帧平滑因子，例如将帧平滑因子设置为固定值1/2。
在步骤10315中，响应于视频帧数量小于或等于第二缓存帧数量阈值，将初始化的帧平滑因子进行取倒数处理，得到帧平滑因子。
在一些实施例中，当缓存帧队列中的视频帧数量小于或等于第二缓存帧数量阈值(例如4)时，将初始化的帧平滑因子进行取倒数处理，得到帧平滑因子，例如将初始化的帧平滑因子5/4取倒数变为4/5。
继续参见图3E，在步骤1032A中，通过帧平滑因子获取延时参数。
在一些实施例中，参见图3H，图3E示出的步骤1032A，可以通过以下步骤1032A1至步骤1032A2实现，下面具体说明。
在步骤1032A1中，获取帧平滑因子和解码帧间隔参数的乘积。
在一些实施例中，可以获取当前视频帧对应时间段内的解码帧间隔的统计值(如95分位数、平均值等)作为解码帧间隔参数，这里，当前视频帧对应时间段，可以是距离当前视频帧的接收时间戳预设时长的历史时间段，也可以是距离当前时刻预设时长的历史时间段，例如可以获取距离当前视频帧的接收时间戳的时长为2s的时间段(视频帧对应时间段)内的解码帧间隔的95分位数作为解码间隔参数，获取帧平滑因子可以参见上文步骤1031A的根据缓存帧队列中的视频帧数量确定的帧平滑因子。
在步骤1032A2中，确定乘积与延时控制参数中的最小值与当前时间戳的加和，将加和与第一渲染时间戳进行求差，得到延时参数。
在一些实施例中，确定乘积与延时控制参数(例如125ms)中的最小值与当前时间戳的加和，将加和与第一渲染时间戳进行求差，得到延时参数。
示例的，延时参数的确定可以表示为：延时参数＝当前时间戳+min(解码帧间隔参数×帧平滑因子，125)—第一渲染时间戳，其中，“min”表示获取“解码帧间隔参数×帧平滑因子”(乘积)与“125”(延时控制参数)之间的最小值。
在一些实施例中，延时参数小于或等于抖动最大值与预设阈值(如500ms)之间的最小值，这里，抖动最大值指的是当前在第一时间段中相邻的数据包的接收时间间隔的最大值。
继续参见图3E，在步骤1033A中，通过第一渲染时间戳和延时参数，确定第二渲染时间戳。
在一些实施例中，将第一渲染时间戳和延时参数进行加和处理，将加和结果作为第二渲染时间戳。
示例的，通过第一渲染时间戳和延时参数，确定第二渲染时间戳，可以表示为：第二渲染时间戳＝延时参数+第一渲染时间戳。
本申请实施例在第一时间段(抖动时间段)通过设置第一解码帧间隔参数阈值和第二解码帧间隔参数阈值，对解码间隔参数进行双向约束，既能避免因网络波动导致的极端间隔值影响帧率稳定性，又为后续帧平滑因子的确定提供标准化输入，确保参数计算的合理性，通过建立解码帧间隔参数与初始化的帧平滑因子之间的映射关系，以及根据缓存帧队列中的视频帧数量动态调整初始化的帧平滑因子，当缓存帧数量大于或等于第二缓存帧数量(如4帧)时，固定帧平滑因子以稳定出帧节奏，防止解码过载；当缓存帧数量小于第二缓存帧数量时，通过帧平滑因子取倒数操作加速解码，利用现有帧资源填补网络波动间隙，避免解码器空转，提升了视频帧解码流程对动态网络抖动的适应能力。
在一些实施例中，参见图3I，当帧平滑策略是用于在第二时间段内，基于解码帧间隔参数对缓存帧队列中的视频帧进行延迟的第二帧平滑策略时，图3A示出的步骤103，可以通过以下步骤1031B至步骤1033B实现，下面具体说明。
在步骤1031B中，响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对缓慢平滑帧数进行递减，并根据解码帧间隔参数确定帧平滑因子，其中，缓慢平缓帧数用于表征缓存帧队列中在视频帧之后的待处理的视频帧的数量，待处理的视频帧是需要通过第二帧平滑策略进行延时的视频帧，缓慢平滑帧数是从初始值开始递减的，且初始值的递减量与通过第二帧平滑策略进行平滑处理的视频帧的数量一致。
在一些实施例中，当缓慢平滑帧数大于缓慢平滑帧数阈值(例如0)时，对缓慢平滑帧数进行递减，并根据解码帧间隔参数确定帧平滑因子，其中，缓慢平缓帧数用于表征缓存帧队列中在视频帧之后的待处理的视频帧的数量，待处理的视频帧是需要通过第二帧平滑策略进行延时的视频帧，缓慢平滑帧数是从初始值开始递减的，且初始值的递减量与通过第二帧平滑策略进行平滑处理的视频帧的数量一致，在第二时间段中，通过第二帧平滑策略对当前视频帧之后的视频帧进行延时之后，每出帧一个当前视频帧之后的视频帧，缓慢平滑帧数递减一次，直至缓慢平滑帧数等于缓慢平滑帧数阈值，解码帧间隔参数是解码帧的解码时间之间的间隔。例如，在距离当前时刻预设时长的历史时间段内，相邻的两个解码帧(也就是依次解码的两个视频帧)之间的解码间隔的统计数，例如平均值，分位数等。
示例的，当缓慢平滑帧数大于0时，假设此时的缓慢平滑帧数为3(这里的缓慢平滑帧数3是通过初始值递减得到的，例如初始值为5，也就是在处理当前视频帧之前已经平滑出帧了2个视频帧，所以当前处理的视频帧对应的缓慢平缓帧数是初始值5递减至3得到的)，说明除了当前正在通过第二帧平滑策略处理的视频帧之外，排在当前正在处理的视频帧之后的接下来的3帧视频也需要通过第二帧平滑策略进行处理，当对接下来的3帧分别通过第二帧平滑策略进行延时的过程中，每处理完一帧缓慢平滑帧数减去1，直至缓慢平滑帧数恢复为0。
在另一些实施例中，响应于缓慢平滑帧数递减至缓慢平滑帧数阈值(如0)，基于帧排序缓存时长获取新的初始值，其中，新的初始值是帧排序缓存时长与解码帧间隔参数的比值；基于新的初始值，确定新的缓慢平滑帧数，基于新的缓慢平滑帧数返回响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对缓慢平滑帧数进行递减，并根据解码帧间隔参数确定帧平滑因子的步骤。
示例的，当缓慢平滑帧数递减至0时，此时需要根据当前正在处理的视频帧的帧排序缓存时长获取新的初始值，假设当前处理的视频帧为P3，在P3之前已经平滑出帧了视频帧P1和P2，P3对应的缓慢平滑帧数是通过P1对应的帧排序缓存时长与解码帧间隔参数的比值确定的初始值递减得到的，如P1对应的初始值为1，由于在P1之后出帧了P2，初始值递减一次，处理P3时缓慢平滑帧数从初始值1递减为0，此时根据P3对应的帧排序缓存时长获取新的初始值，从而根据初始值知悉在P3之后还需要通过第二帧平滑策略进行处理的视频帧数量，假设视频帧P3的帧排序等待时长是200ms，解码帧间隔参数是50ms，此时计算出的新的初始值是4，说明除了当前的P3需要通过第二帧平滑策略进行处理之外，在接下来的4帧(如P4至P7)也需要通过第二帧平滑策略进行处理，每平滑出帧一帧，缓慢平滑帧数(初始值)递减一次，接下来4帧都平滑出去后，缓慢平滑帧数恢复为0，此时继续根据正在处理的视频帧的帧排序缓存时长和解码帧间隔参数来获取新的初始值。
在另一些实施例中，响应于缓慢平滑帧数递减至缓慢平滑帧数阈值(如0)，基于帧排序缓存时长获取新的初始值；响应于新的初始值等于缓慢平滑帧数阈值(即新的初始值仍等于缓慢平滑帧数阈值)；获取视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间差(可以参见下文步骤1031C的说明)；响应于时间差小于预设比例的解码帧间隔参数，通过固定值的帧平滑因子获取延时参数(可以参见下文步骤1032C的说明)；通过第一渲染时间戳和延时参数，确定第二渲染时间戳(可以参见下文步骤1033C的说明)；基于第二渲染时间戳、视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定视频帧的解码等待时长(可以参见下文步骤104的说明)；响应于视频帧在缓存帧队列中的存储时长达到解码等待时长，对视频帧进行解码(可以参见下文步骤105的说明)；响应于时间差大于或等于预设比例的解码帧间隔参数，从缓存帧队列中移出视频帧，并对视频帧进行解码(可以参见下文步骤107的说明)。
在一些实施例中，可以根据解码帧间隔参数确定帧平滑因子，这里，帧平滑因子的确定可以参见上文步骤10311中的说明，其中，步骤10311中的初始化的帧平滑因子可以作为此处确定的帧平滑因子。
继续参见图3I，在步骤1032B中，通过帧平滑因子获取延时参数，其中，帧平滑因子用于表示缓存帧队列的接收帧率与出帧帧率的比例。
在一些实施例中，参见图3J，图3I中示出的步骤1032B，可以通过以下步骤1032B1至步骤1032B2实现，下面具体说明。
在步骤1032B1中，获取帧平滑因子和解码帧间隔参数的乘积，并将乘积与当前时间戳进行求和处理，得到求和结果。
在一些实施例中，将帧平滑因子以解码帧间隔参数相乘后，再与当前时间戳相加，得到求和结果。
示例的，获取帧平滑因子和解码帧间隔参数的乘积，并将乘积与当前时间戳进行求和处理，得到求和结果，可以表示为：求和结果＝帧平滑因子×解码帧间隔参数+当前时间戳。
在步骤1032B2中，将求和结果与第一渲染时间戳进行求差处理，得到延时参数。
承接步骤1032B1中的示例，延时参数的获取可以表示为：延时参数＝帧平滑因子×解码帧间隔参数+当前时间戳—第一渲染时间戳。
继续参见图3I，在步骤1033B中，通过第一渲染时间戳和延时参数，确定第二渲染时间戳。
这里，确定第二渲染时间戳的实施方式可以参见上文步骤1033A的说明。
在另一些实施例中，步骤1031B至步骤1032B可以实施为对在第二时间段内即在丢包重传期间缓存的视频帧的帧平滑策略的处理，对于在第二时间段中已引入了较大延时的重传恢复帧(即数据包丢失重传后组帧得到的视频帧)，可以获取重传恢复帧距离上一个解码帧的解码时刻之间的时间差，当时间差小于二分之一的解码帧间隔参数时，转入通过第三帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的处理(可以参见下文步骤1032C至步骤1033C的说明)，否则，实时出帧，即从缓存帧队列中移出重传恢复帧，并对重传恢复帧进行解码。
本申请实施例在第二时间段(丢包重传时间段)通过第二帧平滑策略，基于解码帧间隔参数确定帧平滑因子并结合缓慢平滑帧数的递减机制，能够根据缓存帧队列中待处理视频帧的数量逐步释放视频帧，避免视频帧集中解码导致的硬件负载突增，同时在缓慢平滑帧数递减至阈值时，基于帧排序缓存时长动态获取新的初始值以持续控制后续帧的出帧节奏，并通过重传恢复帧与上一解码帧的时间差判断是否切换策略或实时出帧，从而有效减少丢包重传引入的过度延时或时序混乱，提升弱网环境下视频帧解码流程对缓存波动和重传数据的动态适应能力。
在一些实施例中，参见图3K，当帧平滑策略是用于在第三时间段内，基于固定值的帧平滑因子对缓存帧队列中的特定视频帧进行延时的第三帧平滑策略时，图3A示出的步骤103，可以通过以下步骤1031C至步骤1033C实现，下面具体说明。
在步骤1031C中，获取视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间差。
在一些实施例中，获取视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间差，这里，上一个解码帧的解码时刻可以是解码帧解码完成的时刻，也可以是解码帧开始解码的时刻。
在另一些实施例中，还可以获取当前时间戳与上一个解码帧的解码时刻之间的时间间隔，以作为时间差。
在步骤1032C中，响应于时间差小于预设比例的解码帧间隔参数，通过固定值的帧平滑因子获取延时参数，其中，帧平滑因子用于表示缓存帧队列的接收帧率与出帧帧率的比例。
在一些实施例中，当时间差小于预设比例(例如二分之一)的解码帧间隔参数时，通过固定值的帧平滑因子获取延时参数，例如固定值的帧平滑因子可以设置为1/2。
示例的，当时间差大于或等于预设比例的解码帧间隔参数时，对视频帧进行实时出帧。
在步骤1033C中，通过第一渲染时间戳和延时参数，确定第二渲染时间戳。
这里，确定第二渲染时间戳的实施方式可以参见上文步骤1033A的说明。
本申请实施例通过针对不同网络场景(第一时间段、第二时间段、第三时间段分别对应的网络场景)动态切换帧平滑策略，避免了单一帧平滑策略在复杂网络环境中的局限性，提升了解码流程对动态网络状态的全局适应性，达到稳定控制视频帧的解码时间，提升视频帧解码及渲染的流畅度的有益效果。
继续参见图3A，在步骤104中，基于第二渲染时间戳、视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定视频帧的解码等待时长。
在一些实施例中，参见图3L，图3A示出的步骤104，可以通过以下步骤1041至步骤1042实现，下面具体说明。
在步骤1041中，获取视频帧的接收时间戳和解码耗时参数的加和。
在一些实施例中，获取视频帧的接收时间戳和解码耗时参数的加和，解码耗时参数可以通过统计预设时间段(如视频帧前2s内)的已解码视频帧的解码耗时确定，如在预设时间段内的已解码视频帧的解码耗时的95分位数、平均值等，这里，视频帧的解码耗时的95分位数是一个统计学指标，用于描述在一段时间内，解码视频帧所需时间的分布情况，视频帧的解码耗时的95分位数指的是在预设时间段内，有95％的视频帧解码耗时小于或等于这个值。
在步骤1042中，将第二渲染时间戳与加和的差值作为视频帧的解码等待时长。
示例的，视频帧的解码等待时长的确定可以表示为：解码等待时长＝第二渲染时间戳—接收时间戳—解码耗时参数。
继续参见图3A，在步骤105中，响应于视频帧在缓存帧队列中的存储时长达到解码等待时长，对视频帧进行解码。
在一些实施例中，当视频帧在缓存帧队列中的存储时长达到解码等待时长时，将视频帧从缓存帧队列中取出送入解码器中，调用解码器对视频帧进行解码。
在一另些实施例中，参见图3M，在图3A示出的步骤105之后，还可以执以下步骤106至步骤107，下面具体说明。
在步骤106中，获取当前时间戳与上一个解码帧的解码时刻之间的时间差。
这里，上一个解码帧的解码时刻可以是解码帧解码完成的时刻，也可以是解码帧开始解码的时刻。
在另一些实施例中，还可以获取视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间间隔，以作为时间差。
在步骤107中，响应于时间差大于或等于预设比例的解码帧间隔参数，从缓存帧队列中移出视频帧，并对视频帧进行解码。
在一些实施例中，当时间差大于或等于预设比例(例如二分之一)的解码帧间隔参数时，从缓存帧队列中移出视频帧(实时出帧)，并对视频帧进行解码。
通过步骤101至步骤107，实现了根据缓存帧队列中视频帧的具体情况(如视频帧数量，当前视频帧处于的时间段等)选择对应的帧平滑策略，使得对不同的视频帧的第一渲染时间戳进行延时的程度不同，避免后续引入过多的解码等待时长，同时，帧平滑策略是根据视频帧的状态进行决策的，不受业务场景相关参数(如视频编码算法策略、网络传输算法策略、流控算法策略等与业务场景相关的参数)的影响，提升了获取视频帧的解码等待时长的准确性，达到了稳定控制视频帧的解码时间，提升视频帧解码及渲染的流畅度的有益效果。
下面，将说明本申请实施例在一个视频通话应用场景中的示例性应用。
在使用移动终端进行视频通话过程中，视频通话过程容易受网络传输环境的影响，例如在网络传输不稳定的环境中，视频播放会出现多种问题，如视频卡顿等，严重影响视频通话效果。
参见图5，图5是本申请实施例提供的视频通话场景下的视频帧的处理方法的流程示意图，下面以视频通话中的接收端设备(如终端设备200-2)为主体进行说明。
参见图6A，图6A是本申请实施例提供的视频帧的处理的原理的示意图，在基于视频帧的采集时间戳进行预测处理，得到视频帧的第一渲染时间戳的同时，通过对当前时间段的判断(对应上文的第一时间段、第二时间段和第三时间段)，以及当前的缓存帧队列中的视频帧数量和解码帧间隔参数，确定视频帧的帧平滑策略以及帧平滑因子，从而确定延时参数以对第一渲染时间戳进行延时，得到第二渲染时间戳，以及基于第二渲染时间戳、视频帧的接收时间戳以及解码耗时参数，确定视频帧的解码等待时长。
在步骤201中，接收数据包，将数据包组合为视频帧，响应于视频帧满足参考关系，将视频帧存入缓存帧队列。
在一些实施例中，参见图6B，图6B是本申请实施例提供的视频帧存入缓存帧队列的原理的示意图，接收端设备接收数据包后进行组帧，将组帧得到的视频帧放入缓冲区中，通过帧排序模块进行排序，当视频帧满足参考关系(如图6A中示出的I0帧参考帧接收成功)后，将视频帧存入缓存帧队列，以进行接下来的每个视频帧的解码处理。
示例的，在视频通话场景下，音频数据和视频帧数据可以分别通过实时传输协议(Real-time Transport Protocol，RTP)或超文本传输协议(Hypertext Transfer Protocol，HTTP)等协议独立传输。其中，视频帧数据可以通过图5示出的视频通话场景下的视频帧的处理方法进行处理，提供给接收端设备的用户在视觉上的体验；音频数据可以通过发送端设备的麦克风捕获原始音频信号，去除原始音频信号中的噪声和不必要的高频或低频分量，检测音频信号中的静默或低能量部分，并在这些部分减少或消除音频流等预处理操作后，通过发送端设备的编解码器将模拟或数字音频信号转换成数字格式，并进行压缩以减少数据大小，接下来，使用实时传输协议(如RTP)将编码后的音频数据发送到接收端，接收端使用相同的编解码器将接收到的编码音频数据解码回原始音频信号，解码后的音频信号通过扬声器播放，提供给接收端设备的用户在听觉上的体验。
在视频在线播放场景下，音频数据和视频帧数据可以被封装在同一容器格式中进行传输，如MP4、FLV、MKV等。在这种情况下，音频数据和视频帧数据在接收端以及传输过程中被封装在一起，但在接收端，音频数据和视频帧数据可以分别解封装并进行相应处理，如对视频帧进行解码渲染，对音频数据进行解码以及回声消除、噪声抑制等后处理。
本申请实施例不限制音频数据和视频帧数据的传输方式，无论采用哪种传输方式，音频数据和视频帧数据都可以在发送端设备进行编码，在接收端设备进行解码，以确保数据的可靠性和高效传输。此外，为了处理网络抖动和延迟，可以采用缓冲、丢包恢复和前向纠错等技术。
在步骤202中，基于视频帧的采集时间戳进行预测处理，得到视频帧的第一渲染时间戳。
在一些实施例中，获取视频帧的第一渲染时间戳(即预估渲染时间戳)的实施方式，可以参见上文步骤101的说明，这里不再赘述。
在步骤203中，根据缓存帧队列中的视频帧，确定帧平滑策略。
在一些实施例中，参见图6C，图6C是本申请实施例提供的确定帧平滑策略的流程示意图，首先判断当前时间段是否为第一时间段(即抖动时间段)，如果当前时间段为第一时间段，则转入通过第一帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的处理，否则判断当前时间段是否为第二时间段(即丢包重传时间段)，若当前时间段为第二时间段则转入通过第二帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的处理，若当前时间段既不是第一时间段也不是第二时间段(对应上文的第三时间段)，获取当前时间戳与上一个解码帧的解码时刻之间的时间差，当时间差小于二分之一的解码帧间隔参数时，转入通过第三帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的处理，否则，实时出帧，即从缓存帧队列中移出视频帧，并对视频帧进行解码。
其中，针对在当前的时间滑窗按照预设时长(如2s)内接收的多个数据包，若全部或超过预设比例的相邻的数据包的接收时间间隔大于时间间隔阈值(如100ms)，则确定当前时刻处于第一时间段；
针对在当前的时间滑窗按照预设时长内接收的多个数据包，若相邻的数据包的接收时间间隔均小于或等于时间间隔阈值(即当前不处于第一时间段)，则确定当前是否有丢包(如在预设时长内是否有丢包发生)，如果有丢包情况则当前时刻处于第二时间段。
在步骤204中，基于帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳。
这里，通过第一帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的实施方式，可以参见上文步骤1031A至步骤1033A的说明；通过第二帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的实施方式，可以参见上文步骤1031B至步骤1033B的说明；通过第三帧平滑策略对第一渲染时间戳进行延时，得到第二渲染时间戳的实施方式，可以参见上文步骤1031C至步骤1033C的说明，这里不再赘述。
在步骤205中，基于第二渲染时间戳、视频帧的接收时间戳以及解码耗时参数，确定视频帧的解码等待时长。
在一些实施例中，确定视频帧的解码等待时长的实施方式，可以参见上文步骤104的说明，这里不再赘述。
在步骤206中，响应于视频帧在缓存帧队列中的存储时长达到解码等待时长，对视频帧进行解码。
在一些实施例中，当视频帧在缓存帧队列中的存储时长达到解码等待时长，将视频帧从缓存帧队列中取出送入解码器中，调用解码器对视频帧进行解码及渲染，使得视频帧在接收端设备上显示。
通过步骤201至步骤206，实现了根据缓存帧队列中视频帧的具体情况(如视频帧数量，当前视频帧处于的时间段等)选择对应的帧平滑策略，使得对不同的视频帧的第一渲染时间戳进行延时的程度不同，尽可能减少了解码等待时长的引入并稳定控制了视频帧的解码时间，达到了降低用户视频通话的卡顿率，提升视频通话的流畅度，提升用户的体验的有益效果。
通过本申请实施例提供的视频帧的处理方法，线上的视频通话的卡顿率从5.2％降低到3.2％，降低了38％。提升了用户在弱网络场景下的实时视频通话的流畅度。
可以理解的是，虽然上述各个流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本实施例中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，上述流程图中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时间执行完成，而是可以在不同的时间执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
下面继续说明本申请实施例提供的视频帧的处理装置455实施为软件模块的示例性结构，在一些实施例中，如图2A所示，存储在存储器450-1的视频帧的处理装置455可以包括：
预测处理模块4551，配置为基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳。
帧平滑处理模块4552，配置为根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，其中，所述帧平滑策略用于通过相邻的数据包的接收时间间隔，预测所述第一渲染时间戳的延时参数，所述数据包用于承载所述视频帧。
在一些实施例中，所述帧平滑处理模块4552，还配置为基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳。
在一些实施例中，所述帧平滑处理模块4552，还配置为基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长。
出帧模块4553，配置为响应于所述视频帧在所述缓存帧队列中的存储时长达到所述解码等待时长，对所述视频帧进行解码。
在一些实施例中，所述帧平滑处理模块4552，还配置为针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔大于时间间隔阈值，则确定当前时刻处于第一时间段，并确定在所述时间滑窗下一次滑动之前，对所述缓存帧队列应用第一帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第一帧平滑策略用于在所述第一时间段内，基于与所述缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对所述视频帧进行延时。
在一些实施例中，所述帧平滑处理模块4552，还配置为针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第二时间段，并确定在所述时间滑窗下一次滑动之前对所述缓存帧队列应用第二帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第二帧平滑策略用于在所述第二时间段内，基于解码帧间隔参数对所述缓存帧队列中的视频帧进行延时。
在一些实施例中，所述帧平滑处理模块4552，还配置为若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，且在所述当前的时间滑窗内至少一个所述数据包在传输过程中丢失，则确定当前时刻处于第二时间段。
在一些实施例中，所述帧平滑处理模块4552，还配置为针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第三时间段，并确定在所述时间滑窗下一次滑动之前对所述缓存帧队列应用第三帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第三帧平滑策略用于在所述第三时间段内，基于固定值的帧平滑因子对所述缓存帧队列中的特定视频帧进行延时，所述特定视频帧的接收时间戳与上一个解码帧的解码时刻的时间差，小于预设比例的解码帧间隔参数。
在一些实施例中，所述帧平滑处理模块4552，还配置为若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，且在所述当前的时间滑窗内所述数据包在传输过程中未丢失，则确定当前时刻处于第三时间段。
在一些实施例中，当所述帧平滑策略是用于在所述第一时间段内，基于与所述缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对所述视频帧进行延时的第一帧平滑策略时，所述帧平滑处理模块4552，还配置为根据所述缓存帧队列中的视频帧数量确定所述帧平滑因子，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；通过所述帧平滑因子获取所述延时参数；通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
在一些实施例中，所述帧平滑处理模块4552，还配置为根据解码帧间隔参数确定初始化的帧平滑因子；获取所述视频帧与前一帧视频帧进入所述缓存帧队列之间的时间间隔；响应于所述时间间隔小于时间间隔阈值，或所述视频帧数量大于第一缓存帧数量阈值，根据所述视频帧数量调整所述初始化的帧平滑因子，确定所述帧平滑因子。
在一些实施例中，所述帧平滑处理模块4552，还配置为响应于所述视频帧数量大于第二缓存帧数量阈值，将所述初始化的帧平滑因子按照固定值进行配置，得到所述帧平滑因子，其中，所述第二缓存帧数量阈值大于所述第一缓存帧数量阈值；响应于所述视频帧数量小于或等于所述第二缓存帧数量阈值，将所述初始化的帧平滑因子进行取倒数处理，得到所述帧平滑因子。
在一些实施例中，所述帧平滑处理模块4552，还配置为获取所述帧平滑因子和解码帧间隔参数的乘积；确定所述乘积与延时控制参数中的最小值与当前时间戳的加和，将所述加和与所述第一渲染时间戳进行求差，得到所述延时参数。
在一些实施例中，当所述帧平滑策略是用于在所述第二时间段内，基于解码帧间隔参数对所述缓存帧队列中的视频帧进行延时的第二帧平滑策略时，所述帧平滑处理模块4552，还配置为响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对所述缓慢平滑帧数进行递减，并根据所述解码帧间隔参数确定帧平滑因子，其中，所述缓慢平缓帧数用于表征所述缓存帧队列中在所述视频帧之后的待处理的视频帧的数量，所述待处理的视频帧是需要通过所述第二帧平滑策略进行延时的视频帧，所述缓慢平滑帧数是从初始值开始递减的，且所述初始值的递减量与通过所述第二帧平滑策略进行平滑处理的视频帧的数量一致；通过所述帧平滑因子获取所述延时参数，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
在一些实施例中，所述帧平滑处理模块4552，还配置为响应于所述缓慢平滑帧数递减至所述缓慢平滑帧数阈值，基于帧排序缓存时长获取新的初始值，其中，所述新的初始值是所述帧排序缓存时长与解码帧间隔参数的比值；基于所述新的初始值，确定新的缓慢平滑帧数，基于所述新的缓慢平滑帧数返回所述响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对所述缓慢平滑帧数进行递减，并根据所述解码帧间隔参数确定帧平滑因子的步骤。
在一些实施例中，所述帧平滑处理模块4552，还配置为获取所述帧平滑因子和解码帧间隔参数的乘积，并将所述乘积与所述当前时间戳进行求和处理，得到求和结果；将所述求和结果与所述第一渲染时间戳进行求差处理，得到所述延时参数。
在一些实施例中，当所述帧平滑策略是用于在所述第三时间段内，基于固定值的帧平滑因子对所述缓存帧队列中的特定视频帧进行延时的第三帧平滑策略时，所述帧平滑处理模块4552，还配置为获取所述视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间差；响应于所述时间差小于预设比例的解码帧间隔参数，通过所述固定值的帧平滑因子获取所述延时参数，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
在一些实施例中，所述出帧模块4553，还配置为获取当前时间戳与上一个解码帧的解码时刻之间的时间差；响应于所述时间差大于或等于预设比例的解码帧间隔参数，从所述缓存帧队列中移出所述视频帧，并对所述视频帧进行解码。
在一些实施例中，所述预测处理模块4551，还配置为获取所述视频帧的接收帧率档位；获取所述视频帧的接收时间差；基于所述接收帧率档位、所述接收时间差和所述采集时间戳，确定所述视频帧的第一渲染时间戳。
在一些实施例中，所述预测处理模块4551，还配置为获取所述视频帧对应的周期时间内的视频帧接收帧率，将所述视频帧接收帧率映射到多个预设的帧率档位中，将映射得到的所述帧率档位作为所述接收帧率档位。
在一些实施例中，所述预测处理模块4551，还配置为获取多个所述视频帧的接收时间戳与所述视频帧的采集时间戳、往返时延和发送耗时之和的差值；将多个所述视频帧对应的所述差值的平均值作为所述接收时间差。
在一些实施例中，所述预测处理模块4551，还配置为获取预设倍数的所述接收帧率档位的倒数；获取所述采集时间戳与所述倒数之间的加和；对所述加和和所述接收时间差进行求和处理，得到所述第一渲染时间戳。
在一些实施例中，所述帧平滑处理模块4552，还配置为获取所述视频帧的接收时间戳和所述解码耗时参数的加和；将所述第二渲染时间戳与所述加和的差值作为所述视频帧的解码等待时长。
下面继续说明本申请实施例提供的视频帧的处理装置456实施为软件模块的示例性结构，在一些实施例中，如图2B所示，存储在存储器450-2的视频帧的处理装置456可以包括：
处理模块4561，配置为获取待编码视频帧，并对所述待编码视频帧进行编码，得到编码后的视频帧，其中，所述编码后的视频帧通过本申请实施例上述的视频帧的处理方法进行解码。
本申请实施例提供了一种计算机程序产品，计算机程序产品包括计算机可执行指令，计算机可执行指令存储在计算机可读存储介质中，电子设备的处理器从计算机可读存储介质读取计算机可执行指令，处理器执行计算机可执行指令，使得电子设备执行本申请实施例上述的视频帧的处理方法。
本申请实施例提供一种存储有可执行指令的计算机存储介质，其中存储有可执行指令，当可执行指令被处理器执行时，将引起处理器执行本申请实施例提供的视频帧的处理方法，例如，如图3A至图3M任一附图示出的视频帧的处理方法。
在一些实施例中，计算机可读存储介质可以是RAM、ROM、闪存、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
在一些实施例中，计算机可执行指令可以采用程序、软件、软件模块、脚本或代码的形式，按任意形式的编程语言(包括编译或解释语言，或者声明性或过程性语言)来编写，并且其可按任意形式部署，包括被部署为独立的程序或者被部署为模块、组件、子例程或者适合在计算环境中使用的其它单元。
作为示例，计算机可执行指令可以但不一定对应于文件系统中的文件，可以可被存储在保存其它程序或数据的文件的一部分，例如，存储在超文本标记语言(Hyper Text Markup Language，HTML)文档中的一个或多个脚本中，存储在专用于所讨论的程序的单个文件中，或者，存储在多个协同文件(例如，存储一个或多个模块、子程序或代码部分的文件)中。
作为示例，计算机可执行指令可被部署为在一个电子设备上执行，或者在位于一个地点的多个电子设备上执行，又或者，在分布在多个地点且通过通信网络互连的多个电子设备上执行。
综上所述，通过本申请实施例，通过利用采集时间戳的时序稳定性作为基准，获取视频帧的第一渲染时间戳，为后续解码等待时长的计算提供了可靠的时间参考基准，通过根据缓存帧队列中的视频帧获取相应的帧平滑策略，实现了帧平滑策略的自适应决策过程，通过帧平滑策略和第一渲染时间戳，确定视频帧的第二渲染时间戳，实现了根据缓存帧队列中视频帧的具体情况选择对应的帧平滑策略，使得对不同的视频帧的第一渲染时间戳进行延时的程度不同，避免单一帧平滑策略导致的解码等待时长累积而引入过多的解码等待时长，最后，通过第二渲染时间戳、视频帧的接收时间戳和已解码视频帧的解码耗时参数，量化视频帧的解码等待时长，从而提升视频帧在缓存帧队列中的解码等待时长与实际解码需求的匹配度，综上所述，通过根据缓存帧队列中视频帧的状态进行平滑策略的决策，提升了获取视频帧的解码等待时长的准确性，达到了稳定控制视频帧的解码时间，提升视频帧解码及渲染的流畅度的有益效果。
以上所述，仅为本申请的实施例而已，并非用于限定本申请的保护范围。凡在本申请的精神和范围之内所作的任何修改、等同替换和改进等，均包含在本申请的保护范围之内。

Claims (26)

  1. 一种视频帧的处理方法，应用于电子设备，包括：
    基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳；
    根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，其中，所述帧平滑策略用于通过相邻的数据包的接收时间间隔，预测所述第一渲染时间戳的延时参数，所述数据包用于承载所述视频帧；
    基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳；
    基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长；
    响应于所述视频帧在所述缓存帧队列中的存储时长达到所述解码等待时长，对所述视频帧进行解码。
  2. 根据权利要求1所述的方法，其中，所述视频帧是根据接收的数据包进行组合得到的，所述根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，包括：
    针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔大于时间间隔阈值，则确定当前时刻处于第一时间段，并确定在所述时间滑窗下一次滑动之前，对所述缓存帧队列应用第一帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第一帧平滑策略用于在所述第一时间段内，基于与所述缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对所述视频帧进行延时。
  3. 根据权利要求1或2所述的方法，其中，所述根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，包括：
    针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第二时间段，并确定在所述时间滑窗下一次滑动之前对所述缓存帧队列应用第二帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第二帧平滑策略用于在所述第二时间段内，基于解码帧间隔参数对所述缓存帧队列中的视频帧进行延时。
  4. 根据权利要求3所述的方法，其中，所述若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第二时间段，包括：
    若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，且在所述当前的时间滑窗内至少一个所述数据包在传输过程中丢失，则确定当前时刻处于第二时间段。
  5. 根据权利要求1、2或4所述的方法，其中，所述根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，包括：
    针对在当前的时间滑窗内接收的多个数据包，若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第三时间段，并确定在所述时间滑窗下一次滑动之前对所述缓存帧队列应用第三帧平滑策略，其中，所述时间滑窗的起点是与所述当前时刻距离预设时长的历史时刻，所述时间滑窗的终点是所述当前时刻，所述第三帧平滑策略用于在所述第三时间段内，基于固定值的帧平滑因子对所述缓存帧队列中的特定视频帧进行延时，所述特定视频帧的接收时间戳与上一个解码帧的解码时刻的时间差，小于预设比例的解码帧间隔参数。
  6. 根据权利要5所述的方法，其中，若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，则确定当前时刻处于第三时间段，包括：
    若相邻的所述数据包的接收时间间隔均小于或等于时间间隔阈值，且在所述当前的时间滑窗内所述数据包在传输过程中未丢失，则确定当前时刻处于第三时间段。
  7. 根据权利要求1至6任一项所述的方法，其中，当所述帧平滑策略是用于在所述第一时间段内，基于与所述缓存帧队列中的视频帧的数量同步变化的帧平滑因子，对所述视频帧进行延时的第一帧平滑策略时，所述基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳，包括：
    根据所述缓存帧队列中的视频帧数量确定所述帧平滑因子，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；
    通过所述帧平滑因子获取所述延时参数；
    通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
  8. 根据权利要求7所述的方法，其中，所述根据所述缓存帧队列中的视频帧数量确定所述帧平滑因子，包括：
    根据解码帧间隔参数确定初始化的帧平滑因子；
    获取所述视频帧与前一帧视频帧进入所述缓存帧队列之间的时间间隔；
    响应于所述时间间隔小于时间间隔阈值，或所述视频帧数量大于第一缓存帧数量阈值，根据所述视频帧数量调整所述初始化的帧平滑因子，确定所述帧平滑因子。
  9. 根据权利要求8所述的方法，其中，所述根据所述视频帧数量调整所述初始化的帧平滑因子，确定所述帧平滑因子，包括：
    响应于所述视频帧数量大于第二缓存帧数量阈值，将所述初始化的帧平滑因子按照固定值进行配置，得到所述帧平滑因子，其中，所述第二缓存帧数量阈值大于所述第一缓存帧数量阈值；
    响应于所述视频帧数量小于或等于所述第二缓存帧数量阈值，将所述初始化的帧平滑因子进行取倒数处理，得到所述帧平滑因子。
  10. 根据权利要求7至9任一项所述的方法，其中，所述通过所述帧平滑因子获取所述延时参数，包括：
    获取所述帧平滑因子和解码帧间隔参数的乘积；
    确定所述乘积与延时控制参数中的最小值与当前时间戳的加和，将所述加和与所述第一渲染时间戳进行求差，得到所述延时参数。
  11. 根据权利要求1至10任一项所述的方法，其中，当所述帧平滑策略是用于在所述第二时间段内，基于解码帧间隔参数对所述缓存帧队列中的视频帧进行延时的第二帧平滑策略时，所述基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳，包括：
    响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对所述缓慢平滑帧数进行递减，并根据所述解码帧间隔参数确定帧平滑因子，其中，所述缓慢平缓帧数用于表征所述缓存帧队列中在所述视频帧之后的待处理的视频帧的数量，所述待处理的视频帧是需要通过所述第二帧平滑策略进行延时的视频帧，所述缓慢平滑帧数是从初始值开始递减的，且所述初始值的递减量与通过所述第二帧平滑策略进行平滑处理的视频帧的数量一致；
    通过所述帧平滑因子获取所述延时参数，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；
    通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
  12. 根据权利要求11所述的方法，其中，所述方法还包括：
    响应于所述缓慢平滑帧数递减至所述缓慢平滑帧数阈值，基于帧排序缓存时长获取新的初始值，其中，所述新的初始值是所述帧排序缓存时长与解码帧间隔参数的比值；
    基于所述新的初始值，确定新的缓慢平滑帧数，基于所述新的缓慢平滑帧数返回所述响应于缓慢平滑帧数大于缓慢平滑帧数阈值，对所述缓慢平滑帧数进行递减，并根据所述解码帧间隔参数确定帧平滑因子的步骤。
  13. 根据权利要求11或12所述的方法，其中，所述通过所述帧平滑因子获取所述延时参数，包括：
    获取所述帧平滑因子和解码帧间隔参数的乘积，并将所述乘积与所述当前时间戳进行求和处理，得到求和结果；
    将所述求和结果与所述第一渲染时间戳进行求差处理，得到所述延时参数。
  14. 根据权利要求1至13任一项所述的方法，其中，当所述帧平滑策略是用于在所述第三时间段内，基于固定值的帧平滑因子对所述缓存帧队列中的特定视频帧进行延时的第三帧平滑策略时，所述基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳，包括：
    获取所述视频帧的接收时间戳与上一个解码帧的解码时刻之间的时间差；
    响应于所述时间差小于预设比例的解码帧间隔参数，通过所述固定值的帧平滑因子获取所述延时参数，其中，所述帧平滑因子用于表示所述缓存帧队列的接收帧率与出帧帧率的比例；
    通过所述第一渲染时间戳和所述延时参数，确定所述第二渲染时间戳。
  15. 根据权利要求1至14任一项所述的方法，其中，所述方法还包括：
    获取当前时间戳与上一个解码帧的解码时刻之间的时间差；
    响应于所述时间差大于或等于预设比例的解码帧间隔参数，从所述缓存帧队列中移出所述视频帧，并对所述视频帧进行解码。
  16. 根据权利要求1至15任一项所述的方法，其中，所述基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳，包括：
    获取所述视频帧的接收帧率档位；
    获取所述视频帧的接收时间差；
    基于所述接收帧率档位、所述接收时间差和所述采集时间戳，确定所述视频帧的第一渲染时间戳。
  17. 根据权利要求16所述的方法，其中，所述基于所述接收帧率档位、所述接收时间差和所述采集时间戳，确定所述视频帧的第一渲染时间戳，包括：
    获取预设倍数的所述接收帧率档位的倒数；
    获取所述采集时间戳与所述倒数之间的加和；
    对所述加和和所述接收时间差进行求和处理，得到所述第一渲染时间戳。
  18. 根据权利要求16或17所述的方法，其中，所述获取所述视频帧的接收帧率档位，包括：
    获取所述视频帧对应的周期时间内的视频帧接收帧率，将所述视频帧接收帧率映射到多个预设的帧率档位中，将映射得到的所述帧率档位作为所述接收帧率档位。
  19. 根据权利要求16或17所述的方法，其中，所述获取所述视频帧的接收时间差，包括：
    获取多个所述视频帧的接收时间戳与所述视频帧的采集时间戳、往返时延和发送耗时之和的差值；
    将多个所述视频帧对应的所述差值的平均值作为所述接收时间差。
  20. 根据权利要求1至19任一项所述的方法，其中，所述基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长，包括：
    获取所述视频帧的接收时间戳和所述解码耗时参数的加和；
    将所述第二渲染时间戳与所述加和的差值作为所述视频帧的解码等待时长。
  21. 一种视频帧的处理方法，应用于电子设备，包括：
    获取待编码视频帧，并对所述待编码视频帧进行编码，得到编码后的视频帧，其中，所述编码后的视频帧通过权利要求1至20任一项所述的方法进行解码。
  22. 一种处理音/视频码流的方法，应用于电子设备，其中，所述音/视频码流根据权利要求21所述的视频帧的处理方法产生，或者基于权利要求1至20任一项所述的视频帧的处理方法进行解码。
  23. 一种视频帧的处理装置，包括：
    预测处理模块，配置为基于缓存帧队列中视频帧的采集时间戳，预测所述视频帧的第一渲染时间戳；
    帧平滑处理模块，配置为根据所述缓存帧队列中的所述视频帧，确定帧平滑策略，其中，所述帧平滑策略用于通过相邻的数据包的接收时间间隔，预测所述第一渲染时间戳的延时参数，所述数据包用于承载所述视频帧；
    所述帧平滑处理模块，还配置为基于所述帧平滑策略确定所述延时参数，基于所述延时参数对所述第一渲染时间戳进行延时，得到第二渲染时间戳；
    所述帧平滑处理模块，还配置为基于所述第二渲染时间戳、所述视频帧的接收时间戳以及已解码视频帧的解码耗时参数，确定所述视频帧的解码等待时长；
    出帧模块，配置为响应于所述视频帧在所述缓存帧队列中的存储时长达到所述解码等待时长，对所述视频帧进行解码。
  24. 一种电子设备，包括：
    存储器，配置为存储计算机可执行指令；
    处理器，配置为执行所述存储器中存储的计算机可执行指令或者计算机程序时，实现权利要求1至20任一项所述的视频帧的处理方法，或实现权利要求21所述的视频帧的处理方法。
  25. 一种计算机存储介质，所述计算机存储介质中存储有可执行指令，所述计算机可执行指令被处理器执行时，实现权利要求1至20任一项所述的视频帧的处理方法，或实现权利要求21所述的视频帧的处理方法。
  26. 一种计算机程序产品，包括计算机程序或计算机可执行指令，所述计算机程序或计算机可执行指令被处理器执行时，实现权利要求1至20任一项所述的视频帧的处理方法，或实现权利要求21所述的视频帧的处理方法。
PCT/CN2025/098977 2024-07-10 2025-06-04 一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品 Pending WO2026012017A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410923921.9A CN121334376A (zh) 2024-07-10 2024-07-10 一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品
CN202410923921.9 2024-07-10

Publications (1)

Publication Number Publication Date
WO2026012017A1 true WO2026012017A1 (zh) 2026-01-15

Family

ID=98335358

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/098977 Pending WO2026012017A1 (zh) 2024-07-10 2025-06-04 一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品

Country Status (2)

Country Link
CN (1) CN121334376A (zh)
WO (1) WO2026012017A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121940568A (zh) * 2026-03-30 2026-04-28 中世云(北京)科技有限公司 一种加载直播画面的时延测试方法
CN122120535A (zh) * 2026-04-28 2026-05-29 湘江实验室 一种视频卡顿补偿方法及相关设备
CN122205134A (zh) * 2026-03-19 2026-06-12 北京强云创新科技有限公司 面向多路视频流的边缘服务器计算资源调度方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102413382A (zh) * 2011-12-27 2012-04-11 四川九洲电器集团有限责任公司 一种提高实时视频平滑性的方法
US20120281562A1 (en) * 2011-05-06 2012-11-08 Google Inc. Apparatus and method for rendering video with retransmission delay
WO2012154156A1 (en) * 2011-05-06 2012-11-15 Google Inc. Apparatus and method for rendering video using post-decoding buffer
CN112954402A (zh) * 2021-03-11 2021-06-11 北京字节跳动网络技术有限公司 视频显示方法、设备、存储介质及计算机程序产品
CN115515010A (zh) * 2021-06-07 2022-12-23 腾讯科技（深圳）有限公司 视频数据处理方法、装置和设备及计算机存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120281562A1 (en) * 2011-05-06 2012-11-08 Google Inc. Apparatus and method for rendering video with retransmission delay
WO2012154156A1 (en) * 2011-05-06 2012-11-15 Google Inc. Apparatus and method for rendering video using post-decoding buffer
CN102413382A (zh) * 2011-12-27 2012-04-11 四川九洲电器集团有限责任公司 一种提高实时视频平滑性的方法
CN112954402A (zh) * 2021-03-11 2021-06-11 北京字节跳动网络技术有限公司 视频显示方法、设备、存储介质及计算机程序产品
CN115515010A (zh) * 2021-06-07 2022-12-23 腾讯科技（深圳）有限公司 视频数据处理方法、装置和设备及计算机存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN122205134A (zh) * 2026-03-19 2026-06-12 北京强云创新科技有限公司 面向多路视频流的边缘服务器计算资源调度方法及系统
CN121940568A (zh) * 2026-03-30 2026-04-28 中世云(北京)科技有限公司 一种加载直播画面的时延测试方法
CN122120535A (zh) * 2026-04-28 2026-05-29 湘江实验室 一种视频卡顿补偿方法及相关设备

Also Published As

Publication number Publication date
CN121334376A (zh) 2026-01-13

Similar Documents

Publication Publication Date Title
US8711929B2 (en) Network-based dynamic encoding
CN106464601B (zh) 信道捆绑
US10756997B2 (en) Bandwidth adjustment for real-time video transmission
US8312164B2 (en) Media quality enhancement among connected media communication devices
US10862940B1 (en) Low latency live video on a communication session
US20140032735A1 (en) Adaptive rate of screen capture in screen sharing
CN106162199B (zh) 带反向信道消息管理的视频处理的方法和系统
CN109951254B (zh) 一种数据处理方法及装置、计算机可读存储介质
US20170094301A1 (en) Initial Bandwidth Estimation For Real-time Video Transmission
US11936698B2 (en) Systems and methods for adaptive video conferencing
US20170094294A1 (en) Video encoding and decoding with back channel message management
US20110261146A1 (en) Determining Buffer Size Based on Retransmission Latency
US9826260B2 (en) Video encoding device and video encoding method
US10516892B2 (en) Initial bandwidth estimation for real-time video transmission
CN110121114A (zh) 发送流数据的方法及数据发送设备
CN121334376A (zh) 一种视频帧的处理方法、装置、设备、计算机可读存储介质及计算机程序产品
US20170094296A1 (en) Bandwidth Adjustment For Real-time Video Transmission
CN107079132B (zh) 在视频电话中的端口重配置之后馈送经帧内译码的视频帧
EP3013012A1 (en) Networking device and method for adapting quality of video bitstreaming over a network
AU2019201095A1 (en) System and method for automatic encoder adjustment based on transport data
CN106101702B (zh) 带反向信道消息管理的视频编码及解码
JP2015525523A (ja) 高速デバイスおよび低速デバイスのバランスをとるためのｃｐｕの使用制御
CN113160342B (zh) 基于反馈的编码方法及装置、存储介质、电子设备
Cangzhou et al. An optimizing scheme for wireless video transmission on Android platform
CN119865635B (zh) 视频帧渲染频率调整方法、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25835920

Country of ref document: EP

Kind code of ref document: A1