WO2023151489A1

WO2023151489A1 - Video processing method, apparatus and device, and storage medium

Info

Publication number: WO2023151489A1
Application number: PCT/CN2023/074054
Authority: WO
Inventors: 陶燕波; 张凯明; 邓勇
Original assignee: 百果园技术(新加坡)有限公司; 陶燕波
Priority date: 2022-02-10
Filing date: 2023-01-31
Publication date: 2023-08-17
Also published as: CN114584835A; CN114584835B

Abstract

Disclosed in the embodiments of the present application are a video processing method, apparatus and device, and a storage medium. In the technical solution provided in the embodiments of the present application, a video frame to be decoded is acquired from a queue to be decoded, and is decoded, so as to obtain a decoded video frame; whether a frame insertion operation condition is met is determined according to a set playback frame rate influence parameter; when the frame insertion operation condition is met, a virtual video frame is generated according to a reference video frame which is recorded in frame insertion cache data, and according to the decoded video frame, and the reference video frame in the frame insertion cache data is updated according to the decoded video frame, so as to prepare for the next frame insertion operation; and the virtual video frame and the decoded video frame are added to a playback queue to be rendered. Video frames waiting to be rendered and displayed in a playback queue to be rendered are increased, thereby effectively improving a video playback frame rate, and optimizing the user experience.

Description

A video processing method, device, equipment and storage medium

This application claims the priority of the Chinese patent application with application number 202210126871.2 submitted to the China Patent Office on February 10, 2022, the entire content of which is incorporated in this application by reference.

technical field

The embodiments of the present application relate to the field of computer technology, and in particular, to a video processing method, device, device, and storage medium.

Background technique

In the process of video transmission, video frames are often transmitted in the form of video data packets. After the video receiving end receives the data packets and parses the video frames, it decodes the video frames and adds them to the queue to be rendered for rendering and playback. .

The user's viewing experience of video is very dependent on the frame rate of video playback. However, when the network is weak, the video sending end will sometimes reduce the sampling rate of the video to ensure the normal sending of video data packets, and when the network is weak, it will also There will be network congestion or serious packet loss, which will lead to a drop in the video playback frame rate and affect the user's video viewing experience.

Contents of the invention

The embodiment of the present application provides a video processing method, device, device, and storage medium to solve the technical problem in the related art that the video playback frame rate drops due to a weak network and affects the user's video viewing experience, so as to improve the video playback frame rate , to optimize the user video viewing experience.

In the first aspect, the embodiment of the present application provides a video processing method, including:

Obtain the video frame to be decoded from the queue to be decoded and decode to obtain the decoded video frame;

Determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters;

When the frame insertion operation condition is met, generate a virtual video frame according to the reference video frame recorded in the frame insertion buffer data and the decoded video frame, and update the frame insertion buffer data according to the decoded video frame reference video frame;

Adding the virtual video frame and the decoded video frame to a playback queue to be rendered.

In the second aspect, the embodiment of the present application provides a video processing device, including a video acquisition module, a frame insertion judgment module, a frame insertion operation module, and a rendering push module, wherein:

The video acquisition module is configured to obtain a video frame to be decoded from the queue to be decoded and decode it to obtain a decoded video frame;

The frame insertion judging module is configured to determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters;

The frame insertion operation module is configured to generate a virtual video frame according to the reference video frame recorded in the frame insertion buffer data and the decoded video frame when the frame insertion operation condition is met, and update the virtual video frame according to the decoded video frame The reference video frame in the interpolation frame buffer data;

The rendering push module is configured to add the virtual video frame and the decoded video frame to a playback queue to be rendered.

In a third aspect, an embodiment of the present application provides a video processing device, including: a memory and one or more processors;

the memory configured to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described in the first aspect.

In a fourth aspect, the embodiment of the present application provides a storage medium containing computer-executable instructions, and the computer-executable instructions are configured to execute the video processing method as described in the first aspect when executed by a computer processor.

In the fifth aspect, the embodiment of the present application provides a computer program product, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and at least one processor of the device reads the computer program from the computer-readable storage medium Acquiring and executing a computer program, so that the device executes the video processing method as described in the first aspect.

In the embodiment of the present application, the decoded video frame is obtained by obtaining the video frame to be decoded from the queue to be decoded, and judging whether the frame insertion operation condition is satisfied according to the setting playback frame rate influence parameter, and when the frame insertion operation condition is satisfied, according to the insertion The reference video frame recorded in the frame buffer data and the decoded video frame generate a virtual video frame, and update the reference video frame in the interpolation frame buffer data according to the decoded video frame to prepare for the next frame interpolation operation, and then the virtual video frame and the decoded video frame Add video frames to the playback queue to be rendered, increase the number of video frames waiting to be rendered and displayed in the playback queue to be rendered, effectively improve the frame rate of video playback, and optimize user experience.

Description of drawings

FIG. 1 is a flow chart of a video processing method provided in an embodiment of the present application;

FIG. 2 is a flow chart of another video processing method provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of a judgment flow chart of a frame insertion operation condition provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of a video processing device provided by an embodiment of the present application;

Fig. 5 is a schematic structural diagram of a video processing device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts relevant to the present application are shown in the drawings but not all content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe various operations (or steps) as sequential processing, many of the operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of operations can be rearranged. The above-described processing may be terminated when its operations are complete, but may also have additional steps not included in the figures. The above-described processes may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

Figure 1 shows a flow chart of a video processing method provided by the embodiment of the present application. The video processing method provided by the embodiment of the present application can be executed by a video processing device, and the video processing device can use hardware and/or software Realized and integrated in video processing equipment.

The video processing method performed by a video processing device is taken as an example for description below. With reference to Fig. 1, this video processing method comprises:

S101: Obtain a video frame to be decoded from a queue to be decoded, and perform decoding to obtain a decoded video frame.

The video frames to be decoded in the queue to be decoded provided by this embodiment are obtained from a set communication network (such as UDP network, TCP network, HTTP network), wherein the video frames to be decoded in the set communication network are provided by the video sending device , that is, the video sending device sends the video frame to be decoded to the video processing device through a set communication network. Exemplarily, the video sending device can select an appropriate encoding method according to the network state to encode the video data packet corresponding to the video frame to be decoded, for example, generate a video data packet based on forward error correction coding (the video data packet includes the video frame to be decoded and corresponding redundant package).

When the video processing device receives the video data packet and decodes to obtain the video frame to be decoded that satisfies the decoding condition (for example, the video frame to be decoded has been received completely, and the corresponding reference frame is also complete), the video frame to be decoded is added to the queue to be decoded waiting to be decoded.

In one embodiment, the video processing device acquires the video frames to be decoded from the queue to be decoded in real time, submits the video frames to be decoded to the decoding thread, and the decoding thread decodes the video frames to be decoded to obtain the decoded video frames.

S102: Determine whether the frame insertion operation condition is satisfied according to the set playing frame rate influencing parameter.

Wherein, the set play frame rate influencing parameter provided in this embodiment is used to reflect whether the play frame rate of the video frame rendered and played by the video processing device satisfies the set play frame rate requirement. It is understandable that when the parameters affecting the playback frame rate are set to meet the frame insertion operation conditions, the video frame playback frame rate fails to meet the set playback frame rate requirements. At this time, the video playback frame rate may be small, causing users to watch When video freezes are obvious and affect the user's video viewing experience, frame insertion processing is required to increase the video playback frame rate, reduce video playback freeze, and optimize the user's video viewing experience. However, when the parameter affecting the playback frame rate does not meet the frame insertion operation conditions, that is, the video frame playback frame rate has reached the set playback frame rate requirement at this time, the frame insertion processing may not be performed to reduce the occupation of processing resources.

Exemplarily, determine the set playback frame rate influencing parameter reflecting the video playback frame rate, and determine whether the currently decoded decoded video frame meets the frame insertion operation condition according to the set playback frame rate influence parameter, and if the frame insertion operation condition is satisfied, Then jump to step S103 to generate the virtual video frame. If the frame insertion operation condition is not satisfied, there is no need to generate a virtual video frame, and the decoded video frame can be directly added to the playback queue to be rendered, and replaced by the decoded video frame The reference video frame in the interpolation frame cache data is convenient for generating a virtual video frame when the frame interpolation operation is required next time.

S103: When the frame insertion operation condition is met, generate a virtual video frame according to the reference video frame and the decoded video frame recorded in the frame insertion buffer data, and update the reference video frame in the frame insertion buffer data according to the decoded video frame.

Exemplarily, when the currently decoded decoded video frame satisfies the frame interpolation operation condition, a virtual video frame is generated according to the reference video frame and the decoded video frame recorded in the frame interpolation buffer data. Further, after the virtual video frame is obtained, the reference video frame in the frame insertion buffer data is updated according to the decoded video frame. For example, replace the reference video frame in the interpolation frame buffer data with the decoded video frame currently decoded, at this time, the decoded video frame will be used as a new reference video frame, and the decoded video frame will be used as the next frame interpolation operation. The reference video frame and the decoded video frame obtained next time are used to generate a reference video frame.

In one embodiment, the generation of virtual video frames can be done through a preconfigured interpolation controller (INTER Controller), that is, the reference video frame is recorded in the frame interpolation buffer data of the frame interpolation controller (for example, the reference video frame is the decoded video frame of the previous frame), and the decoded video frame obtained by current decoding satisfies the frame interpolation requirements. When operating conditions, the decoded video frame is added to the frame interpolation controller, and the frame interpolation controller will generate a virtual video frame according to the newly received decoded video frame and the reference video frame recorded in the frame interpolation buffer record.

Wherein, the virtual video frame is generated according to the reference video frame and the decoded video frame, which can be obtained by weighting the image bits corresponding to two adjacent real videos (that is, the reference video frame and the decoded video frame) to obtain the virtual video frame, and the virtual video frame is generated The speed is faster and the CPU resource usage is reduced, and the Neon library can also be superimposed for assembly acceleration to further reduce the CPU usage. In other feasible embodiments, a virtual video frame may also be generated based on an optical flow prediction algorithm or a frame interpolation algorithm based on deep learning, so as to achieve a more realistic image effect.

S104: Add the virtual video frame and the decoded video frame to a playback queue to be rendered.

Exemplarily, after obtaining the virtual video frame, add the virtual video frame and the decoded video frame to the queue to be rendered, and wait for the rendering thread to obtain the video frame from the queue to be rendered for rendering and displaying. In one embodiment, since the rendering time of the virtual video frame is earlier than that of the currently decoded video frame, when adding the virtual video frame and the decoded video frame to the playback queue to be rendered, first insert the virtual video frame into the playback queue to be rendered. frame, and then insert the decoded video frame. When the rendering thread acquires video frames for rendering and playback, it first acquires the virtual video frame with the earlier playback time for rendering and playback.

It is understandable that users are highly sensitive to video freezes, and the criterion for judging freezes is the playback interval of video frames (that is, the reciprocal of the video frame rate), and the video frame rate is one of the most important indicators for measuring video quality , the human eye is also more sensitive to the subjective perception of the video frame rate. This solution can improve the overall user experience by enhancing the frame rate of video frames at low frame rates in weak network scenarios. When the frame rate of the video frame provided by the video sending device is low, or when the network environment is poor and the packet loss is serious, such as in RTC (Real-time Communications, real-time communication), weak network confrontation has been continuously studied Subject, although there are many anti-packet loss methods that can cover a certain degree of network fluctuations, when the network encounters severe congestion, packet loss or data disorder, the conventional anti-packet loss strategy will also fail, which can be provided by this solution The video processing method inserts virtual video frames between real decoded video frames, increases the frame rate of rendering and playing video frames, reduces video playback freeze rate, improves user experience in weak network scenarios, and effectively increases user usage time (for example, In the same weak network environment, a smooth video call experience can effectively extend the average call duration of users).

As mentioned above, the decoded video frame is obtained by obtaining the video frame to be decoded from the queue to be decoded, and judging whether the frame insertion operation condition is satisfied according to the set playback frame rate influence parameter, and the frame insertion operation is satisfied condition, generate a virtual video frame according to the reference video frame and decoded video frame recorded in the frame interpolation buffer data, and update the reference video frame in the interpolation frame buffer data according to the decoded video frame, to prepare for the next frame interpolation operation, and then Add virtual video frames and decoded video frames to the playback queue to be rendered, increase the video frames waiting to be rendered and displayed in the playback queue to be rendered, effectively improve the frame rate of video playback, and optimize user experience.

On the basis of the above-mentioned embodiments, FIG. 2 shows a flow chart of another video processing method provided by the embodiment of the present application, and the video processing method is an embodiment of the above-mentioned video processing method. With reference to Fig. 2, this video processing method comprises:

S201: Obtain a video frame to be decoded from a queue to be decoded, and perform decoding to obtain a decoded video frame.

In one embodiment, the reference video frame used for generating the virtual video frame is saved as frame interpolation buffer data, and the frame interpolation buffer data is recorded in a set memory space. Wherein, the memory space used for recording frame interpolation buffer data may be a fixedly opened space, or a corresponding space is opened when a setting application program (such as a video application program) is opened, or a setting function ( For example, a corresponding space is opened when a video application program starts a video call function, or a corresponding space is opened when the video call function is turned on and the first video frame is received.

For opening the corresponding space when the first video frame is received, the video processing method provided by this solution, after obtaining the video frame to be decoded from the queue to be decoded and decoding to obtain the decoded video frame, also includes: after the decoded video frame is the first video frame Frame time, open the memory space for recording frame interpolation buffer data.

Exemplarily, when the decoded video frame obtained by decoding is the first video frame (such as the first video frame of a video call), the frame insertion controller is initialized, and the memory space required for recording the frame insertion buffer data is opened, and the frame insertion Set the corresponding frame insertion attribute information in the controller.

Among them, the frame interpolation attribute information includes frame interpolation quantity information, frame interpolation resolution, frame interpolation algorithm, etc., and the frame interpolation quantity information is used to indicate the required generation and insertion between two real frames (reference video frame and decoded video frame). The number of virtual video frames, the frame interpolation resolution is used to indicate the resolution of the virtual video frame to be generated (for example, keep consistent with the reference video frame or decoded video frame), and the frame interpolation algorithm is used to indicate the virtual video frame used to generate Algorithm or interpolation frame generation function to call. In one embodiment, the frame insertion attribute information also includes the playing time information of the virtual frame, for example, when the number of inserted frames is 1, the playing time can be set in the middle of the corresponding playing time of two real video frames, and when the number of inserted frames is greater than 1 Frame time, according to the number of inserted frames, the respective playback times are evenly distributed between the corresponding playback times of the two real video frames.

In one embodiment, when the decoded video frame is the first video frame, after opening the memory space for recording the interpolation buffer data, the first video frame is added to the memory space for recording the interpolation buffer data In between, it is recorded as a reference video frame. It is understandable that when receiving the first video frame, the frame interpolation controller directly receives a video frame without generating a virtual frame or judging the change degree of subsequent video frames, and can directly convert the decoded video Frames are added to the playback queue to be rendered, waiting for the rendering display of the rendering thread.

S202: Determine the degree of video frame change between the decoded video frame and the previous decoded video frame.

Exemplarily, after the decoded video frame is obtained, it is necessary to determine the video frame change degree between the current decoded video frame and the last decoded video frame, and determine whether to execute the subsequent virtual video frame generation process according to the video frame change degree.

In one embodiment, the degree of change in the resolution of two decoded video frames can be used to indicate the degree of change in the video frame. Based on this, when determining the degree of change in the video frame between the decoded video frame and the previous decoded video frame, the specific is: determine the change degree of the video frame according to the resolution change value of the decoded video frame and the resolution of the last decoded video frame.

Exemplarily, the resolution of the currently decoded video frame and the resolution of the last decoded video frame are obtained, the resolution of the decoded video frame and the resolution change value of the last decoded video frame are determined, and the video frame is determined according to the resolution change value degree of change. For example, this solution divides the change degree of the video frame into two degrees of no change and change. or an abnormality in the video transmission process causes the resolution to change), the degree of change in the video frame is a change (for example, the corresponding flag bit is 1), and when the resolution of the two real video frames before and after the change does not change, the video frame change The degree is unchanged (for example, the corresponding flag bit is 0).

S203: Determine whether to update the reference video frame recorded in the frame insertion buffer data according to the comparison result of the change degree of the video frame and the set change threshold.

Exemplarily, the determined change degree of the video frame is compared with a set change threshold, and it is determined whether the change degree of the video frame reaches the set change threshold according to the comparison result. Further, when the degree of change of the video frame does not reach the set change threshold (that is, the resolutions of the decoded video frames of the two frames before and after are consistent), it can be considered that the change degree of the decoded video frames of the two frames before and after this time is relatively small, which is suitable for frame insertion operate. Based on this, when the degree of change of the video frame does not reach the set change threshold, jump to step S204 to further determine whether the currently decoded video frame satisfies the frame insertion operation condition.

When the change degree of the video frame reaches the set change threshold, it can be considered that the change degree of the decoded video frame of the two frames before and after this time is relatively large, and it is no longer suitable for the frame insertion operation, so as to avoid the situation that the generated virtual video frame has a poor effect. Based on this, when the change degree of the video frame reaches the set change threshold, according to the decoded video The frame updates the reference video frame recorded in the frame insertion buffer data, that is, the decoded video frame obtained by current decoding replaces the reference video frame recorded in the frame insertion buffer data. Further, after updating the reference video frame recorded in the frame interpolation buffer data, the current decoded video frame is directly added to the playback queue to be rendered.

S204: When there is no need to update the reference video frame recorded in the frame insertion buffer data, determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameter.

When the degree of change of the video frame does not reach the set change threshold (that is, the resolutions of the two decoded video frames before and after are consistent), it is further judged whether the decoded video frame satisfies the frame insertion operation condition, and if the decoded video frame satisfies the frame insertion operation condition, Then jump to step S205.

When the decoded video frame does not satisfy the frame insertion operation condition, the reference video frame in the frame insertion buffer data is updated according to the decoded video frame, that is, the currently decoded decoded video frame replaces the reference video frame recorded in the frame insertion buffer data. Further, after updating the reference video frame recorded in the frame interpolation buffer data, the current decoded video frame is directly added to the playback queue to be rendered, without generating a virtual video frame.

In one embodiment, setting the parameters affecting the playback frame rate includes the video receiving frame rate and/or the number of video frames to be played in the playback queue to be rendered. Wherein, the video receiving frame rate is a receiving frame rate at which the video processing device receives the video frame to be decoded. The number of video frames to be played is the number of video frames to be decoded that are added to the playback queue to be rendered and wait for the rendering thread to extract, render and display. Based on this, as shown in Fig. 3, as shown in a schematic diagram of a frame insertion operation condition judgment flow chart, this solution includes steps S2041-S2043 when determining whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters:

S2041: Determine whether the video receiving frame rate is within the set frame rate range, and/or whether the number of video frames to be played in the to-be-rendered playback queue is within the set number threshold.

Wherein, the video receiving frame rate may be the frame rate at which the video processing device receives the video frames to be decoded within a set time period (for example, 3 seconds). Exemplarily, it is determined that the video processing device receives the video receiving frame rate of the video frame to be decoded within the set time period, and compares the video receiving frame rate with the set frame rate range (for example, setting the set frame rate range to be less than or equal to 10fps) to determine whether the received video frame rate is within the set frame rate range. And/or determine the number of video frames to be played in the playback queue to be rendered, and compare the number of video frames to be played with a set number threshold (for example, set the set number threshold to 1 frame), and determine the number of video frames to be played in the playback queue to be rendered. Whether the number of video frames to be played is within the set number threshold.

S2042: In response to the received video frame rate being within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue being within the set number threshold, determine that the frame insertion operation condition is met.

S2043: In response to the received video frame rate not within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered play queue is not within the set number threshold, determine that the frame insertion operation condition is not met.

Exemplarily, when the received video frame rate is within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue is within the set number threshold, the currently decoded video frame is considered to meet the frame insertion operation condition. If the received video frame rate is not within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue is not within the set number threshold, it is considered that the currently decoded video frame does not meet the frame insertion operation conditions.

Among them, it is judged whether the video receiving frame rate is within the set frame rate range, the virtual video frame is only generated when the video receiving frame rate is within the set frame rate range, and the virtual video frame is generated when the video receiving frame rate is within the set frame rate range Otherwise, since the receiving frame rate of the current user is relatively high, there is no need to enable the frame insertion function, which effectively reduces the CPU usage. Judging whether the number of video frames to be played in the playback queue to be rendered is within the set number threshold, and generating virtual video frames only when the number of video frames to be played in the playback queue to be rendered is within the set number threshold, which can be effective Reduce the problem of buffer accumulation in the playback queue to be rendered, avoid the situation that the fluency of video playback is affected by too many video frames cached in the playback queue to be rendered, and reduce the need for existing fast playback and slow playback mechanisms (that is, the rendering thread according to the pending The number of cached video frames in the rendering and playing queue adaptively adjusts the effect of the rendering and playing speed of video frames) to ensure the video playback effect.

In one embodiment, the setting playback frame rate influencing parameter for judging whether the decoded video frame satisfies the frame insertion operation condition can be one of the video receiving frame rate and the number of video frames to be played (that is, the video receiving frame rate is at the set frame rate range, or when the number of video frames to be played in the playback queue to be rendered is within the set number threshold, it is determined that the frame insertion operation condition is met), it can also be a combination of the video receiving frame rate and the number of video frames to be played ( That is, when the video receiving frame rate is within the set frame rate range, and the number of video frames to be played in the to-be-rendered playback queue is within the set number threshold, it is determined that the frame insertion operation condition is satisfied), this embodiment uses the video receiving frame rate The combination with the number of video frames to be played is described as an example. For example, when the received video frame rate within 3 seconds is less than 10fps, and the number of video frames to be played in the playback queue to be rendered is less than or equal to 1 frame, it can be determined that the currently decoded video frame satisfies the frame insertion operation condition.

S205: When the frame insertion operation condition is met, generate a virtual video frame according to the reference video frame and the decoded video frame recorded in the frame insertion buffer data, and update the reference video frame in the frame insertion buffer data according to the decoded video frame.

Wherein, whenever a virtual video frame is generated, the playing time of the virtual video frame can be set. Based on this, in one embodiment, before adding the virtual video frame and the decoded video frame to the playback queue to be rendered, it also includes: determining the virtual video frame according to the playback time of the decoded video frame and the playback time of the last decoded video frame playback time.

Exemplarily, an intermediate value between the playing time of the current decoded video frame and the playing time of the last decoded video frame is used as the playing time of the virtual video frame. The rendering thread can be based on the sequence of playback time The virtual video frame and the corresponding decoded video frame are successively acquired for rendering and playback.

S206: Add the virtual video frame and the decoded video frame to a playback queue to be rendered.

As mentioned above, the decoded video frame is obtained by obtaining the video frame to be decoded from the queue to be decoded, and judges whether the frame insertion operation condition is satisfied according to the set playback frame rate influence parameter, and when the frame insertion operation condition is satisfied, according to the frame insertion buffer The reference video frame and decoded video frame recorded in the data generate a virtual video frame, and update the reference video frame in the interpolation frame buffer data according to the decoded video frame, in preparation for the next frame interpolation operation, and then convert the virtual video frame and the decoded video frame Add it to the playback queue to be rendered, increase the video frames waiting to be rendered and displayed in the playback queue to be rendered, effectively improve the video playback frame rate, and optimize the user experience. At the same time, according to the video receiving frame rate and/or the number of video frames to be played in the playback queue to be rendered, it is judged whether the current decoded video frame meets the frame insertion operation conditions, effectively reducing the generation of unnecessary virtual video frames and the buffering of the playback queue to be rendered In case of accumulation, improve the fluency of video playback. And the video processing method provided by this solution can be configured in the background of the system, and the frame insertion function will be automatically started when it is detected that the frame insertion operation conditions are met, without interactive confirmation with the user, optimizing the user experience. And when the resolution of the decoded video frame changes or when the first video frame is received, use the current decoded video frame to update the reference video frame recorded in the interpolation frame buffer data or initialize the frame interpolation controller to ensure that the next generation of correct virtual Video frame, to ensure the video playback effect after frame insertion.

As mentioned above, the decoded video frame is obtained by obtaining the video frame to be decoded from the queue to be decoded, and judges whether the frame insertion operation condition is satisfied according to the set playback frame rate influence parameter, and when the frame insertion operation condition is satisfied, according to the frame insertion buffer The reference video frame and decoded video frame recorded in the data generate a virtual video frame, and update the reference video frame in the interpolation frame buffer data according to the decoded video frame, in preparation for the next frame interpolation operation, and then convert the virtual video frame and the decoded video frame Add it to the playback queue to be rendered, increase the video frames waiting to be rendered and displayed in the playback queue to be rendered, effectively improve the video playback frame rate, and optimize the user experience.

Fig. 4 is a schematic structural diagram of a video processing device provided by an embodiment of the present application. Referring to FIG. 4 , the video processing device includes a video acquisition module 41 , a frame insertion judgment module 42 , a frame insertion operation module 43 and a rendering push module 44 .

Among them, the video acquisition module 41 is configured to obtain the video frame to be decoded from the queue to be decoded and decoded to obtain a decoded video frame; the frame insertion judgment module 42 is configured to determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters; The frame insertion operation module 43 is configured to generate a virtual video frame according to the reference video frame and the decoded video frame recorded in the frame insertion buffer data when the frame insertion operation condition is met, and update the reference video in the frame insertion buffer data according to the decoded video frame frame; the rendering push module 44 is configured to virtualize Video frames and decoded video frames are added to the playback queue to be rendered.

In one embodiment, setting the play frame rate influence parameter includes video reception frame rate and/or the number of video frames to be played in the play queue to be rendered, and the frame insertion judging module 42 determines whether the insertion frame rate is satisfied according to the set play frame rate influence parameter. When the frame operation condition is configured as:

Determine whether the video receiving frame rate is within the set frame rate range, and/or whether the number of video frames to be played in the playback queue to be rendered is within the set number threshold;

In response to the video receiving frame rate being within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue being within the set number threshold, it is determined that the frame insertion operation condition is met;

In response to the received video frame rate not within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue not within the set number threshold, it is determined that the frame insertion operation condition is not satisfied.

In one embodiment, the frame insertion operation module 43 is further configured to update the reference video frame in the frame insertion buffer data according to the decoded video frame when the frame insertion operation condition is not met.

In one embodiment, the frame insertion operation module 43 is further configured to:

Determine the degree of video frame change between the decoded video frame and the previous decoded video frame;

When the change degree of the video frame reaches the set change threshold, the reference video frame recorded in the frame interpolation buffer data is updated according to the decoded video frame.

In one embodiment, the frame insertion operation module 43 is configured to:

The video frame change degree is determined according to the resolution change value of the decoded video frame and the resolution of the last decoded video frame.

In one embodiment, the rendering push module 44 is further configured to determine the virtual video frame according to the playback time of the decoded video frame and the playback time of the last decoded video frame before adding the virtual video frame and the decoded video frame to the playback queue to be rendered. The playback time of the frame.

In one embodiment, the video acquisition module 41 is further configured to, after obtaining the video frame to be decoded from the queue to be decoded and decoded to obtain the decoded video frame, when the decoded video frame is the first video frame, enable the frame insertion buffer data for recording memory space.

The embodiment of the present application also provides a video processing device, and the video processing device can integrate the video processing device provided in the embodiment of the present application. Fig. 5 is a schematic structural diagram of a video processing device provided by an embodiment of the present application. With reference to Fig. 5, this video processing equipment comprises: input device 53, output device 54, memory A memory 52 and one or more processors 51; the memory 52 is configured to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 are implemented as The video processing method provided in the foregoing embodiments. The video processing apparatus, equipment, and computer provided above can be configured to execute the video processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

The embodiment of the present application also provides a storage medium containing computer-executable instructions, and the computer-executable instructions are configured to execute the video processing method provided in the foregoing embodiments when executed by a computer processor. Certainly, a storage medium containing computer-executable instructions provided in the embodiments of the present application, the computer-executable instructions are not limited to the video processing method provided above, and may also execute the video processing method provided in any embodiment of the present application. related operations. The video processing device, equipment, and storage medium provided in the above-mentioned embodiments can execute the video processing method provided in any embodiment of the present application. For technical details not described in detail in the above-mentioned embodiments, please refer to the Video processing method.

On the basis of the above-mentioned embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be implemented as a software product The computer program product is stored in a storage medium, and includes several instructions to make a computer device, a mobile terminal or a processor therein execute all or part of the video processing methods provided by various embodiments of the present application step.

The above are only preferred embodiments of the present application and the applied technical principles. The present application is not limited to the specific embodiments provided here, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present application, and the present application The scope is determined by the scope of the claims.

Claims

A video processing method, characterized in that, comprising:

Obtain the video frame to be decoded from the queue to be decoded and decode to obtain the decoded video frame;

Determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters;

When the frame insertion operation condition is met, generate a virtual video frame according to the reference video frame recorded in the frame insertion buffer data and the decoded video frame, and update the frame insertion buffer data according to the decoded video frame reference video frame;

Adding the virtual video frame and the decoded video frame to a playback queue to be rendered.
The video processing method according to claim 1, wherein the parameters affecting the set playback frame rate include video receiving frame rate and/or the number of video frames to be played in the playback queue to be rendered, and the playback frame according to the set Rate-affecting parameters determine whether frame insertion operation conditions are met, including:

Determine whether the video receiving frame rate is within the set frame rate range, and/or whether the number of video frames to be played in the playback queue to be rendered is within the set number threshold;

In response to the video receiving frame rate being within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered playback queue being within the set number threshold, determining that the frame insertion operation condition is satisfied;

In response to the received video frame rate not within the set frame rate range, and/or the number of video frames to be played in the to-be-rendered play queue not within the set number threshold, it is determined that the frame insertion operation condition is not met.
The video processing method according to claim 1, wherein the method further comprises:

When the frame insertion operation condition is not satisfied, the reference video frame in the frame insertion buffer data is updated according to the decoded video frame.
The video processing method according to claim 1, wherein after obtaining the video frame to be decoded from the queue to be decoded and decoding to obtain the decoded video frame, further comprising:

Determine the video frame change degree between the decoded video frame and the previous decoded video frame;

When the degree of change of the video frame reaches the set change threshold, the reference video frame recorded in the frame interpolation buffer data is updated according to the decoded video frame.
The video processing method according to claim 4, wherein the determination of the degree of change in the video frame between the decoded video frame and the last decoded video frame comprises:

The video frame change degree is determined according to the resolution change value of the decoded video frame and the resolution of the last decoded video frame.
The video processing method according to claim 1, wherein before adding the virtual video frame and the decoded video frame to the playback queue to be rendered, further comprising:

Determine the said The playback time of the virtual video frame.
The video processing method according to claim 1, wherein after obtaining the video frame to be decoded from the queue to be decoded and decoding to obtain the decoded video frame, further comprising:

When the decoded video frame is the first video frame, a memory space for recording frame insertion buffer data is opened.
A video processing device, characterized in that it includes a video acquisition module, a frame insertion judgment module, a frame insertion operation module, and a rendering push module, wherein:

The video acquisition module is configured to obtain a video frame to be decoded from the queue to be decoded and decode it to obtain a decoded video frame;

The frame insertion judging module is configured to determine whether the frame insertion operation condition is satisfied according to the set playback frame rate influencing parameters;

The frame insertion operation module is configured to generate a virtual video frame according to the reference video frame recorded in the frame insertion buffer data and the decoded video frame when the frame insertion operation condition is met, and update the virtual video frame according to the decoded video frame The reference video frame in the interpolation frame buffer data;

The rendering push module is configured to add the virtual video frame and the decoded video frame to a playback queue to be rendered.
A video processing device, characterized in that it includes: a memory and one or more processors;

the memory configured to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the video processing method according to any one of claims 1-7.
A storage medium containing computer-executable instructions, wherein the computer-executable instructions are configured to execute the video processing method according to any one of claims 1-7 when executed by a computer processor.
A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, the video processing method according to any one of claims 1-7 is implemented.