CN117291810B

CN117291810B - Video frame processing method, device, equipment and storage medium

Info

Publication number: CN117291810B
Application number: CN202311592412.4A
Authority: CN
Inventors: 陈思佳
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-11-27
Filing date: 2023-11-27
Publication date: 2024-03-12
Anticipated expiration: 2043-11-27
Also published as: CN117291810A

Abstract

The application discloses a video frame processing method, device, equipment and storage medium, and belongs to the field of image superdivision. The method comprises the following steps: obtaining target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to the original video frame, and the target resolution is larger than the original resolution of the original video frame; acquiring the superminute computing capability of the first equipment and the superminute computing capability of the second equipment associated with the first equipment, wherein the superminute computing capability of the first equipment comprises the superminute computing time of the first equipment aiming at different resolutions, and the superminute computing capability of the second equipment comprises the superminute computing time of the second equipment aiming at different resolutions; a manner of superdivision computation of the original video frame is determined based on at least one of the superdivision computation capability of the first device and the superdivision computation capability of the second device. In the method, the super-division computing mode is determined after the super-division computing capacity of the first equipment and the second equipment is comprehensively considered, so that the first equipment and the second equipment can be reasonably utilized, and the resource utilization rate is improved.

Description

Video frame processing method, device, equipment and storage medium

Technical Field

The present invention relates to the field of image super-resolution, and in particular, to a method, an apparatus, a device, and a storage medium for processing video frames.

Background

Super-resolution is simply referred to as super-resolution, and super-resolution techniques are capable of processing low-resolution images or videos into high-resolution images or videos. The pursuit of different users for resolution is also different, so that videos with different resolutions are often provided for the users, and the users can select the videos according to the equipment and network conditions.

When providing video playing service for users, the video playing client usually stores videos with different resolutions in a server, and provides videos with corresponding gears for the users according to the needs of the users. With the development of the super-division technology, the video playing client side also provides a super-division level for a user, and the user can select the super-division level to acquire the video with the resolution higher than the highest resolution stored in the server. However, in this scenario, the video is usually super-divided by the terminal on which the video playing client is installed, but the display capability of some terminals is not proportional to the super-division calculation capability, for example, the display capability of the super-definition television is very good, but the super-division calculation capability may not support super-division of the video, in this case, only the highest resolution video that can be provided by the server is played for the user.

Therefore, how to reasonably utilize the super-division computing power of the terminal and the server to provide the video processing method for the user is a problem to be solved.

Disclosure of Invention

The application provides a video frame processing method, a device, equipment and a storage medium, wherein the technical scheme is as follows:

according to an aspect of the present application, there is provided a method of processing video frames, the method being performed by a first device, the method comprising:

obtaining target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to an original video frame, and the target resolution is larger than the original resolution of the original video frame;

acquiring the super-resolution computing capability of the first device and the super-resolution computing capability of a second device associated with the first device, wherein the super-resolution computing capability of the first device comprises super-resolution computing time of the first device for different resolutions, and the super-resolution computing capability of the second device comprises super-resolution computing time of the second device for different resolutions;

a manner of superscore computation of the original video frame is determined based on at least one of the superscore computation capability of the first device and the superscore computation capability of the second device.

According to another aspect of the present application, there is provided a method for processing a video frame, the method further comprising:

monitoring real-time superdivision time for superdividing the original video frame into the target video frame;

stopping the super-division calculation under the condition that the real-time super-division time of the continuous i original video frames is larger than the inter-frame time interval corresponding to the target frame rate;

wherein the target frame rate is a play frame rate of the target video frame, and i is a positive integer.

and performing superdivision on the original video frame based on the superdivision calculation mode of the original video frame.

According to another aspect of the present application, there is provided a method for processing a video frame, where the performing, based on a manner of performing a super-division calculation on the original video frame, the super-division on the original video frame includes:

under the condition that the super-division calculation mode is that the first equipment performs super-division on the original video frame, the original video frame is super-divided into the target video frame; or alternatively, the first and second heat exchangers may be,

under the condition that the super-division calculation mode is that the first equipment performs down-shift super-division on the original video frame, super-dividing the original video frame into target video frames; or alternatively, the first and second heat exchangers may be,

And under the condition that the super-division calculation mode is that the first equipment super-divides the original video frames belonging to the first video frame sequence, the original video frames belonging to the first video frame sequence are super-divided into the target video frames.

According to another aspect of the present application, there is provided a method of processing video frames, the method being performed by a second device, the method comprising:

uploading the super-resolution computing capability of the second device to the first device;

acquiring a super-division calculation mode of an original video frame, wherein the super-division calculation mode of the original video frame is determined by at least one of super-division calculation capability of the first device and super-division calculation capability of the second device;

According to another aspect of the present application, there is provided a method of processing a video frame,

the second sequence of video frames includes the original video frames ordered as even; or alternatively, the first and second heat exchangers may be,

The second sequence of video frames includes the original video frames ordered as odd.

the second video frame sequence includes groups of images ordered as even; or alternatively, the first and second heat exchangers may be,

the second video frame sequence includes the group of images ordered odd;

wherein each of said image groups comprises a plurality of said original video frames.

the method comprises the steps of decoding test code streams with different resolutions, performing superdivision on the decoded test code streams, and rendering and playing the processed test code streams to realize detection of the superdivision computing capability of second equipment;

Recording single frame superdivision time and single frame rendering time in the detection process;

and calculating the super-division calculation capability of the second device, wherein the super-division calculation capability of the second device is the sum of the Shan Zhen super-division time and the single-frame rendering time.

According to another aspect of the present application, there is provided a processing apparatus for video frames, the apparatus comprising:

the first acquisition module is used for acquiring target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to the original video frame, and the target resolution is larger than the original resolution of the original video frame;

a second obtaining module, configured to obtain a superminute computing capability of the first device and a superminute computing capability of a second device associated with the first device, where the superminute computing capability of the first device includes superminute computing times of the first device for different resolutions, and the superminute computing capability of the second device includes superminute computing times of the second device for different resolutions;

and the determining module is used for determining the super-division calculation mode of the original video frame based on at least one of the super-division calculation capability of the first device and the super-division calculation capability of the second device.

the third acquisition module is used for acquiring target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to the original video frame, and the target resolution is larger than the original resolution of the original video frame;

the uploading module is used for uploading the super-division computing capacity of the second device to the first device;

a fourth obtaining module, configured to obtain a super-division calculation manner of an original video frame, where the super-division calculation manner of the original video frame is determined by at least one of a super-division calculation capability of the first device and a super-division calculation capability of the second device;

and the second super-division module is used for super-dividing the original video frame based on the super-division calculation mode of the original video frame.

According to another aspect of the present application, there is provided a computer device comprising: a processor and a memory, wherein at least one section of program is stored in the memory; the processor is configured to execute the at least one program in the memory to implement the method for processing a video frame.

According to another aspect of the present application, there is provided a computer readable storage medium having stored therein executable instructions loaded and executed by a processor to implement the above-described video frame processing method.

According to another aspect of the present application, there is provided a computer program product comprising computer instructions stored in a computer readable storage medium, from which a processor reads and executes the computer instructions to implement the above-described method of processing video frames.

The beneficial effects that this application provided technical scheme brought include at least:

the method determines the superminute calculation mode of the original video frame based on at least one of the superminute calculation capability of the first equipment and the superminute calculation capability of the second equipment, comprehensively considers the superminute calculation capability of the first equipment and the superminute calculation capability of the second equipment when the superminute calculation is carried out on the original video frame, has more reasonable judgment of the superminute calculation, and improves the utilization rate of resources on the premise of guaranteeing the superminute target.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 illustrates an architecture diagram of a computer system provided in one exemplary embodiment of the present application;

FIG. 2 illustrates a flow chart of a method of processing video frames provided by an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a method for processing video frames according to an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram illustrating a method for partitioning a video frame sequence according to an exemplary embodiment of the present application;

fig. 5 is a schematic diagram illustrating a method for partitioning a video frame sequence according to another exemplary embodiment of the present application;

FIG. 6 illustrates a schematic diagram of a first target code stream provided by an exemplary embodiment of the present application;

FIG. 7 is a flowchart illustrating a method for processing video frames according to another exemplary embodiment of the present application;

FIG. 8 illustrates an overall flowchart of a method for processing video frames provided by an exemplary embodiment of the present application;

fig. 9 is a block diagram illustrating a video frame processing apparatus according to an exemplary embodiment of the present application;

fig. 10 is a block diagram illustrating a video frame processing apparatus according to another exemplary embodiment of the present application;

fig. 11 shows a schematic structural diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be noted that, the user information (including but not limited to user equipment information, such as display capability of a terminal) and the data (including but not limited to super-resolution computing capability and the like) related to the application are information and data authorized by a user or fully authorized by each party, and the collection, use and processing of the related data need to comply with related laws and regulations and standards of related countries and regions. For example, information such as setting operations referred to in the present application is acquired with sufficient authorization.

It should be understood that although the terms "first," "second," etc. may be used in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

First, a description is made of related terms to which the present application relates.

Resolution ratio: representing the resolution of the system to the details. According to different application scenes, the display resolution, the screen resolution, the image resolution and the like can be subdivided. The display resolution refers to the physical resolution of the computer display itself, i.e. the number of pixels on the display. The display resolution is the highest resolution that can be displayed by a computer display. The screen resolution is also called display resolution, and refers to the resolution adopted by computer equipment when an image is actually displayed, and is a value which can be manually adjusted. For the same display, the screen resolution is no greater than the display resolution. The image resolution is used to indicate the number of pixels of an image. In general, the higher the resolution, the better the resolution of the detail and the clearer the displayed image. The resolution may be expressed in either a long and wide format or in digital format. For example, the image resolution is 1280×720, which may also be expressed as 720P; the image resolution is 1920×1080, which can be expressed as 1080P; the image resolution is 2560×1440, which can be expressed as 2K; the image resolution is 4096×2160, which can be expressed as 4K. Wherein P represents the number of pixels in the vertical direction, and a pixels in the vertical direction can be called aP; k represents the number of pixels in the horizontal direction, and the resolution of the pixels in the horizontal direction is more than 2000, which can be called 2K; the same applies to 4K, which is the resolution of 4000 or more for pixels in the horizontal direction. Sometimes a resolution close to 4000 pixels in the horizontal direction may also be referred to as 4K, for example 3656 x 2664 resolution.

Super-Resolution (SR): a method of restoring image detail based on known image information. The super resolution may be simply referred to as super resolution. Superresolution is simply the method by which an image is converted from low resolution to high resolution. For example, an image of 1280×720 is super-divided into images of 1920×1080. Because the video is actually composed of video frames from frame to frame, the video can use super-resolution to improve the resolution of the video, and thus the sharpness of the video.

Frame Rate (Frame Rate): the rate at which images appear on the display in frames. Where frame rate = frame number (Frames)/Time (Time), in Frames per second (f/s, frames Per Second, fps). It will be appreciated that at a frame rate of 60fps for a video, he will play 60 video frames in 1 second.

Video frame: base unit of video. Video is essentially a continuously playing still picture, which is a frame. When transmitting video, because the volume of the original video is often very large, in order to facilitate the transmission of the video, the encoding and decoding of the video are researched, the video is subjected to lossy encoding, the video volume is compressed, the video is convenient to transmit, and when the video is required to be played, only the decoding operation is needed. Because the static pictures forming the video are generally similar, a plurality of key frames with low similarity can be selected during video coding, all information of the key frames is reserved, and the difference information between the rest video frames and the key frames is reserved, so that the volume of the video can be greatly reduced. The IPB coding scheme is a great application of the compression concept. An I-frame (Intra-coded Picture frame) is a key frame, and is coded only according to the present frame information; a P frame (Predictive-coded Picture frame) is a predicted frame, and is encoded with respect to change information compared with a previous I frame or P frame; b frames (Bidirectionally Predicted Picture, bi-directionally predictive coded image frames) are coded for I frames, or P frames, or both, as compared to the previous and subsequent I frames.

Cloud gaming: cloud computing based online gaming. In cloud games, an application program of the game runs in a server, a terminal is equivalent to a display and an operation desk, and is responsible for displaying game pictures and receiving user operations, the operation of the user is transmitted to the server, the server performs game operation, rendering and the like, the server transmits the processed game pictures to the terminal in the form of videos, and the terminal displays the videos of the processed game pictures on a screen. The cloud game enables the terminal to be free from downloading, installing and storing game data, and enables the terminal to be capable of playing the cloud game only by having video decoding capability and network connection capability.

FIG. 1 illustrates a block diagram of a computer system 100 provided in an exemplary embodiment of the present application. The computer system 100 includes: a server 110 and a terminal 120.

The server 110 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server providing cloud computing services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), and cloud servers of basic cloud computing services such as big data and an artificial intelligent platform. The server 110 stores the original video, and the resolution of the original video is the original resolution; the server 110 renders an original video, wherein the resolution of the original video is the original resolution; the server 110 is configured to provide video to the terminal 120 in the form of video frames; the server 110 is configured to determine a superscore calculation mode based on the superscore calculation capability of the server 110 and the superscore calculation capability of the terminal 120; the server 110 is further configured to super-divide the original video frame into target video frames based on a super-division calculation manner of the original video frame, where a target resolution of the target video frame is higher than an original resolution of the original video frame.

The terminal 120 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (car), a wearable device, a PC (Personal Computer ), an unmanned predetermined terminal, a smart speaker, etc. The terminal 120 may be provided with a terminal running a target application, which may be an application supporting online video, for example, an online video playing application, an application capable of watching live broadcast, a cloud game application, etc. The form of the target application program is not limited in this embodiment, and includes, but is not limited to, a client, an application program, an applet, etc. installed in the terminal 120, and may also be in the form of a web page. The terminal 120 is configured to obtain a target resolution and a target frame rate selected by a user; the terminal 120 is configured to request the server to acquire a video frame, where the video frame is an original video frame or a target video frame; the terminal 120 is configured to super-divide the original video frame into target video frames based on the super-division calculation mode of the original video frame.

Communication between the server 110 and the terminal 120 may be through a network, such as a wired or wireless network.

Those skilled in the art will appreciate that the number of terminals 120 may be greater or lesser. Such as: the number of the terminals 120 may be only one, or tens or hundreds, or more. The number and device type of the terminals 120 are not limited in the embodiment of the present application.

Fig. 2 shows a flowchart of a method for processing a video frame according to an exemplary embodiment of the present application, where the method is performed by a first device, which is a computer device, and may be a terminal or a server shown in fig. 1. The method comprises the following steps.

Step 210: and obtaining target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to the original video frame, and the target resolution is larger than the original resolution of the original video frame.

In some embodiments, the target resolution is a preset resolution; or, the target resolution is a dynamically set resolution; or, the target resolution is a resolution selected by the user; or, the target resolution is a display resolution supported by the first device or the second device.

Illustratively, the target resolution is a preset resolution, the target resolution is 4K; or, the target resolution is a dynamically set resolution, for example, the target resolution is a dynamically set resolution based on the transmission speed of the first device, and when the transmission speed of the first device is slow, the target resolution is lower, which is 1080P; when the transmission speed of the first device is high, the target resolution is high, and is 4K; or the target resolution is the resolution selected by the user, and the user selects the target resolution to be 2K based on the self requirement; or, the target resolution is a display resolution supported by the first device or the second device, for example, the first device is a playing end of the video, the highest resolution supported by the first device is 2K, and when the user selects the desired target resolution to be 4K, the target resolution is set to 2K based on the highest display resolution of the first device being 2K.

Step 220: the method comprises the steps of obtaining the superscore computing capability of a first device and the superscore computing capability of a second device associated with the first device, wherein the superscore computing capability of the first device comprises the superscore computing time of the first device aiming at different resolutions, and the superscore computing capability of the second device comprises the superscore computing time of the second device aiming at different resolutions.

In some embodiments, different devices have different supercomputing capabilities, including supercomputing time of the device for different resolutions.

In some embodiments, where the first device is a server providing at least one of the original video frame and the target video frame, the super-resolution computing capability of the first device includes super-resolution computing time for different resolutions. Optionally, the super-resolution computing capability of the first device further includes encoding time to encode at least one of the original video frame and the target video frame into a bitstream.

In some embodiments, where the first device is a terminal that plays the target video frame, the super-resolution computing capability of the first device includes super-resolution computing times for different resolutions. Optionally, the super-resolution computing capability of the first device further includes a decoding time to decode the server-transmitted bitstream into at least one of the original video frame and the target video frame. Optionally, the super-resolution computing capability of the first device further comprises a rendering time for rendering the target video frame to a display screen of the terminal.

Step 230: a manner of superdivision computation of the original video frame is determined based on at least one of the superdivision computation capability of the first device and the superdivision computation capability of the second device.

In some embodiments, a superscore calculation mode of the original video frame is determined based on at least one of the superscore calculation capability of the first device and the superscore calculation capability of the second device, the superscore calculation mode being used to instruct the first device and the second device to superscore the original video frame.

In some embodiments, a manner of superdivision calculation of the original video frame is determined based on the first frame time. The first frame time is used to indicate the time from the user triggering the play operation to the rendering of the first video frame, i.e. the time the user perceives from the application. For example, the head frame time for a good user experience is 100ms. Illustratively, in order to ensure that the user has a good first frame experience, the first original video frame is not oversubscribed; or, the group of pictures (Group of Pictures, GOP) comprising the first original video frame is not super-divided; or, the top n original video frames are not subjected to super division; or, the top n image groups are not subjected to super division; or, the original video frame for the first n seconds is not super-divided. The group of pictures is the basic unit of video encoding and decoding.

The method includes the steps that in an application scene of short videos, the first video is not subjected to super-division, and in the playing process of the first video, the second video to the mth video are subjected to super-division; or, the first video is not subjected to super-division, and in the playing process of the first video, the first n original video frames from the second video to the mth video are subjected to super-division. Under the application scene of the long video, the first n original video frames are not subjected to super-division, and in the playing process of the first n original video frames, the original video frames after the n+1th video frames are subjected to super-division; or, the advertisement before and during the long video playing is not subjected to the super-division, and only the original video frames of the long video are subjected to the super-division. Under the application scene of online video, such as the scene that the generation and playing of video such as live broadcast, cloud game and the like are carried out simultaneously, the video delay requirement is higher, the frame extraction super-division is carried out on the original video frame, and the super-division is carried out on the original video frame serving as a key frame and part of predicted frames.

In some embodiments, a manner of superdivision calculation of the original video frame is determined based on the click time. The clip time is used to indicate the display time of a single video frame that causes the video to have a clip feel, i.e., how long one video frame is displayed will cause the user to feel the video clip. For example, the user perceives the video to be stuck for 200ms. Illustratively, the manner of superdivision computation of the original video frame is determined based on the superdivision computation power and the click time. The super-division computing capability is single-frame super-division time, and the super-division computing mode of the original video frame is determined based on the single-frame super-division time and the katen time.

In summary, the method provided by the embodiment of the present application determines the superminute calculation mode of the original video frame based on at least one of the superminute calculation capability of the first device and the superminute calculation capability of the second device, when the superminute calculation is performed on the video frame, the superminute calculation capability of the first device and the superminute calculation capability of the second device are comprehensively considered, the judgment of the superminute calculation is more reasonable, and the utilization rate of the resource is improved on the premise of guaranteeing the superminute target.

In some embodiments, the hyperspectral computing capability includes: single frame superminute time. Shan Zhen super-time is used to indicate the time to super-divide a single original video frame into target video frames.

In some embodiments, the super-resolution computing capability further includes a single frame encoding time to encode the video frame. Illustratively, a single frame encoding time is the time at which a single video frame is encoded; or, the single frame encoding time is an average encoding time of each frame when a plurality of video frames are encoded into an image group.

In some embodiments, the super-resolution computing capability further includes a single frame decoding time of the decoded video frame. Illustratively, a single frame decoding time is the time to decode a single video frame; or, the single frame encoding time is an average decoding time of each frame when decoding the group of images into a plurality of video frames.

In some embodiments, the super-resolution computing capability further includes a single frame rendering time to render the video frame onto the display screen.

In some embodiments, the first device is a server and the second device is a terminal; or the first equipment is a terminal, and the second equipment is a server; or the first equipment is a terminal, and the second equipment is a terminal; or, the first device is a server, and the second device is a server. Next, the present application will specifically describe taking the first device as a server and the second device as a terminal, but in actual implementation, the present application does not limit the selection of the first device and the second device.

The server 110 obtains the target resolution uploaded by the terminal 120, the target frame rate and the super resolution computing power 31 of the terminal. The server 110 determines the supercomputing manner 32 based on at least one of the supercomputing capabilities of the server and the supercomputing capabilities of the terminal. Four different types of superscore computation are shown, including but not limited to at least one of the following four types of superscore computation (the ordering does not represent the implementation merits or merits).

Mode one: the second device (terminal 120) performs superdivision.

Mode two: the first device (server 110) performs superdivision.

Mode three: the first device (server 110) performs a downshift superdivision.

Mode four: the first device (server 110) performs cooperative oversubscription with the second device (terminal 120).

It should be noted that the above four ways may be combined two by two, three by three, or four, so as to form new combination embodiments. For example, the first and second modes may be combined into an independent embodiment for implementation, and the step-by-step judgment is performed when determining the superscore while determining the superscore calculation mode. Firstly, judging whether the first mode can be executed, and if the first mode can be executed, executing the first mode; if the first mode cannot be executed, judging whether the second mode can be executed. If the second mode can be executed, executing the second mode; if the second mode cannot be executed, no oversubscription is performed.

Mode two and mode three may be combined into independent embodiments for implementation; mode one and mode four may be combined into independent embodiments to be implemented; the second and fourth modes may be combined into independent embodiments for implementation.

The first mode, the second mode and the third mode can be combined into independent embodiments for implementation, and the judgment is gradually carried out when the super-division calculation mode is determined. Firstly, judging whether the first mode can be executed, and if the first mode can be executed, executing the first mode; if the first mode cannot be executed, judging whether the second mode can be executed. If the second mode can be executed, executing the second mode; if the second mode cannot be executed, judging whether the third mode can be executed. If the third mode can be executed, executing the third mode; if the third mode cannot be executed, no oversubscription is performed.

The first mode, the second mode and the fourth mode can be combined into independent embodiments for implementation, and the judgment is gradually carried out when the super-division calculation mode is determined. Firstly, judging whether the first mode can be executed, and if the first mode can be executed, executing the first mode; if the first mode cannot be executed, judging whether the second mode can be executed. If the second mode can be executed, executing the second mode; if the second mode cannot be executed, then judging whether the fourth mode can be executed. If the fourth mode can be executed, executing the fourth mode; if the fourth mode cannot be executed, no oversubscription is performed.

The first mode, the second mode, the third mode and the fourth mode can be combined into independent embodiments for implementation, and when the super-division calculation mode is determined, the super-division calculation mode is judged step by step and then is judged according to the application scene. Firstly, judging whether the first mode can be executed, and if the first mode can be executed, executing the first mode; if the first mode cannot be executed, judging whether the second mode can be executed. If the second mode can be executed, executing the second mode; if the second mode cannot be executed, the third mode or the fourth mode is preferentially judged according to the application scene. For example, in a scene that is more focused on resolution, such as a scene that is played using a large-screen device, or a long video scene, when the second mode cannot be executed, it is first determined whether the fourth mode can be executed. If the fourth mode can be executed, executing the fourth mode; if the fourth mode cannot be executed, it is determined whether the third mode can be executed. If the third mode can be executed, executing the third mode; if the third mode cannot be executed, no oversubscription is performed. In a scene which is more focused on smoothness, such as a game scene and a scene played by using a small-screen device, when the second mode cannot be executed, whether the third mode can be executed is judged. If the third mode can be executed, executing the third mode; if the third mode cannot be executed, it is determined whether the fourth mode can be executed. If the fourth mode can be executed, executing the fourth mode; if the fourth mode cannot be executed, no oversubscription is performed.

The various combination embodiments described above are merely illustrative of various aspects of the present application and are not meant to be limiting of the embodiments of the present application. The order of the above-mentioned various ways of determining may also be changed in different embodiments, and will not be described herein.

The four types of super-resolution calculation will be described below.

Mode one (second equipment superminute)

In an alternative embodiment based on fig. 2, step 230 may alternatively be implemented as step 231.

Step 231: and under the condition that the single frame overdriving time of the second device is not greater than the inter-frame time interval corresponding to the target frame rate, the second device is determined to overdrive the original video frame.

Wherein the target frame rate is a play frame rate of the target video frame.

In some embodiments, the inter-frame time interval = time/frame number for the target frame rate. For example, the target frame rate is 30fps, that is, 30 video frames are played in 1 second, the inter-frame time interval corresponding to the target frame rate is 1s/30, which is equivalent to 1000ms/30, and the final result is 33ms after the decimal point is removed; or, the target frame rate is 60fps, namely, 60 video frames are played in 1 second, the inter-frame time interval corresponding to the target frame rate is 1s/60, the equivalent is 1000ms/60, and the final result is 16ms after decimal point removal.

In some embodiments, in a case where the single frame superminute time of the second device is not greater than the inter-frame time interval corresponding to the target frame rate, that is, in a case where the superminute calculation time of the second device for a single original video frame is faster than the playing time of the target video frame, the superminute of the original video frame and the playing of the target video frame can be implemented, so that the superminute calculation of the original video frame by the second device is determined.

In some embodiments, the determination is made to perform the super-division calculation on the original video frame by the second device in the event that the super-division calculation capability of the second device is not greater than the inter-frame time interval corresponding to the target frame rate. The super-division computing capability comprises single-frame super-division time and single-frame rendering time, the super-division computing capability is the sum of the single-frame super-division time and the single-frame rendering time, and the super-division computing capability of the second device is determined to perform super-division computation on the original video frame under the condition that the super-division computing capability of the second device is not more than the inter-frame time interval corresponding to the target frame rate; or the super-division computing capability comprises single-frame super-division time, single-frame rendering time and single-frame decoding time, the super-division computing capability is the sum of the single-frame super-division time, the single-frame rendering time and the single-frame decoding time, and under the condition that the super-division computing capability of the second equipment is not more than the inter-frame time interval corresponding to the target frame rate, the second equipment is determined to perform super-division computation on the original video frame.

In some embodiments, as shown in fig. 3, after determining the manner of superdivision calculation for superdividing the original video frame by the second device, the server 110 encodes the original video frame into an original code stream 33 and sends the original code stream 33 to the terminal 120, the terminal 120 decodes the original code stream to obtain the original video frame, superdividing the original video frame into the target video frame 34, and the terminal 120 renders the target video frame 35 to the display screen for playing. The resolution of the target video frame is the target resolution, and the playing frame rate of the target video frame is the target frame rate.

In summary, the method provided in the embodiment of the present application preferably considers whether the single-frame supertime of the second device satisfies the requirement of superdivision on the original video frame, and the judgment standard is that when the second device superdivides the original video frame, the playing frame rate of the target video frame can be satisfied, which does not cause the blocking and incontinence of the playing video, not only satisfies the requirement of the user on superdivision on the video frame, but also ensures that the video frame can be smoothly played while the video frame is superdivided.

Mode II (first equipment superminute)

In an alternative embodiment based on fig. 2, step 230 may alternatively be implemented as step 232.

Step 232: in the case where the single frame supertime of the second device is greater than the inter-frame time interval of the target frame rate and the single frame supertime of the first device is not greater than the inter-frame time interval of the target frame rate, the superdivision of the original video frames by the first device is determined.

Wherein the target frame rate is a play frame rate of the target video frame.

Under the condition that the single frame supertime of the second device is larger than the inter-frame time interval corresponding to the target frame rate, namely under the condition that the superminute calculation time of the second device on a single original video frame is slower than the playing time of the target video frame, if the second device performs superminute calculation on the original video frame, the video frame playing speed is faster than the superminute speed of the second device on the original video frame, so that the video frame playing is blocked, therefore, whether the first device can perform superminute calculation is judged, namely whether the single frame superminute time of the first device is not larger than the inter-frame time interval of the target frame rate is judged, and under the condition that the single frame superminute time of the first device is not larger than the inter-frame time interval of the target frame rate, the first device is determined to perform superminute calculation on the original video frame.

In some embodiments, the method further comprises determining, by the first device, to perform the super-division calculation on the original video frame in the event that the super-division calculation capability of the second device is greater than the inter-frame time interval of the target frame rate and the super-division calculation capability of the first device is not greater than the inter-frame time interval of the target frame rate. Illustratively, the super-resolution computing capability of the second device includes a single frame super-resolution time and a single frame rendering time, the super-resolution computing capability of the second device being a sum of the single frame super-resolution time and the single frame rendering time; the super-resolution computing power of the first device is a single frame super-resolution time. Or the super-division computing capability of the second device comprises single-frame super-division time, single-frame rendering time and single-frame decoding time, and the super-division computing capability of the second device is the sum of the single-frame super-division time, the single-frame rendering time and the single-frame decoding time; the super-division computing capability of the first device comprises single-frame super-division time and single-frame coding time, and the super-division computing capability of the first device is the sum of the single-frame super-division time and the single-frame coding time.

In some embodiments, as shown in fig. 3, after determining the manner of superdivision calculation for superdividing the original video frame by the first device, the server 110 superdividing the original video frame into the target video frames 36, encoding the target video frames into the target code stream 37, and sending the target code stream 37 to the terminal 120, where the terminal 120 decodes the target code stream to obtain the target video frames 35, and renders the target video frames 35 to the display screen for playing. The resolution of the target video frame is the target resolution, and the playing frame rate of the target video frame is the target frame rate.

In summary, in the method provided by the embodiment of the present application, when the single frame calculation time of the second device cannot meet the requirement that smooth playing of the target video frame is not affected while oversubscription is performed, whether the first device can meet the requirement is detected, and under the condition that the first device meets the requirement that smooth playing of the target video frame is not affected while oversubscription is performed, the first device oversubscription is performed on the original video frame, so that smooth playing of the overstepped video frame, that is, the target video frame, for the user can be ensured while overstepping.

Mode III (first device downshift super division)

In an alternative embodiment based on fig. 2, step 230 may alternatively be implemented as at least one of step 233 and step 234.

Step 233: under the condition that the single-frame superdivision time of the second device and the single-frame superdivision time of the first device are both larger than the inter-frame time interval of the target frame rate, determining that the first device performs down-shift superdivision on the original video frame, wherein the down-shift superdivision is used for indicating that the original video frame is superdivided into the reduced target resolution, or modifying the playing frame rate of the target video frame into the reduced target frame rate, or superdividing the original video frame into the reduced target resolution and modifying the playing frame rate of the target video frame into the reduced target frame rate.

Wherein the target frame rate is a play frame rate of the target video frame.

In some embodiments, in the event that the single frame superminute time of the second device and the single frame superminute time of the first device are both greater than an inter-frame time interval of the target frame rate, at least one of the target resolution and the target frame rate is reduced; determining, by the first device, to perform a downshift hyperscore on the original video frame, the downshift hyperscore being used to indicate that the original video frame is hyperscored into a reduced target resolution, or that a play frame rate of the target video frame is modified into a reduced target frame rate, or that the original video frame is hyperscored into a reduced target resolution and that a play frame rate of the target video frame is modified into a reduced target frame rate.

In some embodiments, as shown in fig. 3, after determining the manner of the super-division calculation for performing the down-shift super-division on the original video frame by the first device, the server 110 down-divides the original video frame into the target video frames 38, encodes the target video frames into the target code stream 39, and sends the target code stream 39 to the terminal 120, and the terminal 120 decodes the target code stream to obtain the target video frames 40, and renders the target video frames 40 to the display screen for playing. Wherein the resolution of the target video frame 40 is the reduced target resolution, or the play frame rate of the target video frame 40 is the reduced target frame rate, or the resolution of the target video frame 40 is the reduced target resolution and the play frame rate of the target video frame 40 is the reduced target frame rate.

In some embodiments, as shown in fig. 6, step 233 "determining that the original video frame is downshifted by the first device in the event that both the single frame superscalar time of the second device and the single frame superscalar time of the first device are greater than the inter-frame time interval of the target frame rate" may alternatively be implemented as at least one of steps 2331 through 2333.

Step 2331: and under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, reducing the target frame rate, so that the target resolution and the reduced target frame rate meet the single-frame superminute time of the first equipment.

Illustratively, the target resolution is 4K and the target frame rate is 120fps; under the condition that the single-frame super-division time of the first equipment and the single-frame super-division time of the second equipment are both larger than the inter-frame time interval of the target frame rate, the target frame rate is reduced, the reduced target frame rate is 60fps, whether the single-frame super-division time of the first equipment is not larger than the inter-frame time interval of the reduced target frame rate is judged, and if the single-frame super-division time of the first equipment is not larger than the inter-frame time interval of the reduced target frame rate, the 60fps is determined to be the reduced target frame rate; if the frame rate is larger than the preset threshold value, continuously reducing the target frame rate to 30fps, and continuously judging the relation between the super-division time of the single frame and the inter-frame time interval.

Step 2332: and under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, reducing the target resolution, so that the reduced target resolution and the target frame rate meet the single-frame superminute time of the first equipment.

Illustratively, the target resolution is 4K and the target frame rate is 120fps; under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, reducing the target resolution, wherein the reduced target resolution is 2K, judging whether the single-frame superminute time of the first equipment aiming at the reduced target resolution is not larger than the inter-frame time interval of the target frame rate, and if not, determining that 2K is the reduced target resolution; if the resolution is larger than the first threshold, continuing to reduce the target resolution, reducing the target resolution to 1080P, and continuing to judge.

Step 2333: and under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, reducing the target resolution and the target frame rate, so that the reduced target resolution and the reduced target frame rate meet the single-frame superminute time of the first equipment.

Illustratively, the target resolution is 4K and the target frame rate is 120fps; under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, the target resolution and the target frame rate are reduced, the reduced target resolution is 2K, the reduced target frame rate is 60fps, whether the single-frame superminute time of the first equipment aiming at the reduced target resolution is not larger than the inter-frame time interval of the reduced target frame rate is judged, if not larger than the inter-frame time interval of the reduced target frame rate, 2K is determined to be the reduced target resolution, and 60fps is determined to be the reduced target frame rate; if the target resolution is larger than the target frame rate, continuously reducing the target resolution and the target frame rate, reducing the target resolution to 1080P, reducing the target frame rate to 30fps, and continuously judging.

In some embodiments, step 2331, step 2332, and step 2333 may be performed interchangeably in sequence or simultaneously.

Optionally, in an application scenario where smoothness is more important, such as a game scenario, a scenario played using a small screen, step 2332 of lowering the target resolution is preferentially performed; in application scenarios where visual experience is more important, such as long video scenarios, scenarios where large screen play is used, step 2331 of reducing the target frame rate is preferably performed.

In some embodiments, a capability configuration table is obtained, the capability configuration table being used to indicate a target resolution and a target frame rate supported by the first device. Exemplary capability configuration tables are shown in table 1 below.

Table 1 capability configuration table

In some embodiments, the target resolution and the target frame rate are reduced based on the capability configuration table in the event that the single frame superminute time of the second device and the single frame superminute time of the first device are both greater than an inter-frame time interval of the target frame rate. The priority corresponding to the capability configuration table is gradually decreased from the tail to the head. Illustratively, the target resolution is 4K and the target frame rate is 30fps; based on the capability configuration table, inquiring the previous table information of 4K@30fps to be 2K@60fps, judging whether the single-frame super time of the first equipment aiming at the resolution is not more than the inter-frame time interval of the frame rate, and if not, performing super division according to 2K@60fps; if the information is larger than the preset value, continuously inquiring the information of the last table, namely 2K@30fps, and continuously judging.

Step 234: and under the condition that the single-frame superdivision time of the first device and the single-frame superdivision time of the second device are both larger than the inter-frame time interval of the target frame rate, determining that the original video frames are not superdivided.

In some embodiments, in the case where the single frame superdivision time of the first device and the single frame superdivision time of the second device are both greater than the inter-frame time interval of the target frame rate, that is, the first device and the second device cannot meet the requirement that the playing of the target video frame is not affected while the original video frame is superdivided, it is determined that the original video frame is not superdivided, and the original video frame is directly played for the user.

In some embodiments, step 233 is performed first to determine whether at least one of the single frame superminute time of the first device for the reduced target resolution and the single frame superminute time of the second device for the reduced target resolution is not greater than the inter-frame time interval of the reduced target frame rate after reducing the target resolution and the target frame rate, and if not, determining that the first device superminute the original video frame, the superminute target being the reduced target resolution and the target frame rate; if the video frame is larger than the original video frame, the original video frame is determined not to be subjected to super division.

In some embodiments, if the reduced target resolution and target frame rate are not higher than the original resolution and original frame rate, it is determined that the original video frame is not over-divided.

In summary, in the method provided in the embodiment of the present application, under the condition that both the first device and the second device do not meet the requirements, on one hand, the target resolution and the target frame rate are considered to be reduced, on the other hand, the super-resolution is considered to be abandoned, and the target resolution and the target frame rate can be reduced first, and when the original resolution and the original frame rate are reduced, it is directly determined that the super-resolution is not performed on the original video frame. The method can ensure that the video can be played for the user even if the original video frame is not subjected to super division, and the minimum requirement of the user is ensured.

Mode IV (first device and second device cooperate with each other to superminute)

In some embodiments, the original video frame belongs to the first video frame sequence or the second video frame sequence. The first video frame sequence and the second video frame sequence respectively comprise a plurality of original video frames, and the first video frame sequence and the second video frame sequence comprise different original video frames belonging to the same video or the same code stream.

In an alternative embodiment based on fig. 2, step 230 may alternatively be implemented as step 235.

Step 235: under the condition that the single-frame superdivision time of the first device and the single-frame superdivision time of the second device are both larger than the inter-frame time interval of the target frame rate, collaborative superdivision calculation is carried out on the original video frames, the fact that the first device superdivision is carried out on the original video frames belonging to the first video frame sequence is determined, the second device superdivision is carried out on the original video frames belonging to the second video frame sequence, and collaborative superdivision calculation is carried out on the first device and the second device to carry out superdivision calculation on different original video frames.

Wherein the target frame rate is a play frame rate of the target video frames, and the first video frame sequence and the second video frame sequence comprise different original video frames belonging to the same video or the same code stream.

In some embodiments, in the case that the single frame superdivision time of the first device and the single frame superdivision time of the second device are both greater than the inter-frame time interval of the target frame rate, that is, the first device and the second device cannot meet that the playing of the target video frame is not affected while the original video frame is superdivided, the first device and the second device perform cooperative superdivision calculation on the original video frame, the first device superdivide the original video frame belonging to the first video frame sequence, and the second device superdivide the original video frame belonging to the second video frame sequence.

In some embodiments, as shown in fig. 4, the first video frame sequence 50 and the second video frame sequence 60 comprise different original video frames belonging to the same video or the same bitstream. The first video frame sequence comprises original video frames ordered as odd numbers, and the second video frame sequence comprises original video frames ordered as even numbers; alternatively, the first video frame sequence comprises original video frames ordered as an even number and the second video frame sequence comprises original video frames ordered as an odd number.

In some embodiments, as shown in fig. 5, the first video frame sequence 50 and the second video frame sequence 60 comprise different original video frames belonging to the same video or the same bitstream. Each image group 70 includes a plurality of original video frames. The image groups 70 are divided by key frames 71, and the first original video frame in each image group 70 is the key frame 71. The first video frame sequence comprises image groups ordered as odd numbers, and the second video frame sequence comprises image groups ordered as even numbers; alternatively, the first video frame sequence comprises groups of pictures ordered even and the second video frame sequence comprises groups of pictures ordered odd.

In some embodiments, a ratio of the first video frame sequence to the original video frames included in the second video frame sequence is determined based on the single frame superminute time of the first device and the single frame superminute time of the second device. That is, the ratio of the first video frame sequence to the original video frames included in the second video frame sequence is the inverse of the ratio of the single-frame superminute time of the first device to the single-frame superminute time of the second device. Illustratively, the ratio of the single-frame superminute time of the first device to the single-frame superminute time of the second device is 3/5, and then the ratio of the number of original video frames in the first video frame sequence to the number of original video frames in the second video frame sequence is set to be 5/3.

In some embodiments, as shown in fig. 3, after determining the manner of performing the cooperative superdivision calculation on the original video frames by the first device and the second device, the server 110 superdivide the original video frames belonging to the first video frame sequence into target video frames 41, encodes the first video frame sequence and the second video frame sequence into a first target code stream 42, and sends the first target code stream 42 to the terminal 120, after decoding the target code stream, the terminal 120 superdivides the original video frames belonging to the second video frame sequence into target video frames 43, finally obtains target video frames 35, and renders the target video frames 35 into a display screen for playing. The resolution of the target video frame is the target resolution, and the playing frame rate of the target video frame is the target frame rate. The first target code stream is shown in fig. 6, where the original video frames belonging to the first video frame sequence have been super-divided into target video frames, and the original video frames belonging to the second video frame sequence are still original video frames.

In some embodiments, a ratio of the first video frame sequence to the original video frames included in the second video frame sequence is determined based on the superdivision computing capability of the first device and the superdivision computing capability of the second device. That is, the ratio of the first video frame sequence to the original video frames included in the second video frame sequence is the inverse of the ratio of the super-resolution computing capability of the first device to the super-resolution computing capability of the second device. Illustratively, the super-resolution computation of the first device can be a single frame super-resolution time, and the super-resolution computation capability of the second device can be a sum of the single frame super-resolution time and the single frame rendering time. And setting the ratio of the super-division computing capacity of the first device to the super-division computing capacity of the second device to be 5/6, and setting the ratio of the first video frame sequence to the original video frames included in the second video frame sequence to be 6/5.

In summary, in the method provided by the embodiment of the present application, under the condition that both the first device and the second device do not meet the requirements, the first device and the second device perform collaborative super-division calculation on the original video frame, divide the original video frame into two types, namely, a first video frame sequence and a second video frame sequence, and show three allocation manners of the first video frame sequence and the second video frame sequence, so that it can be ensured that the super-divided target video frame can still be provided for the user under the condition that the single-frame super-division time of the first device and the second device does not meet the requirements.

And when the first equipment performs the superscore calculation, monitoring the state of the first equipment in real time, and judging whether the superscore is continued or not.

In an alternative embodiment based on fig. 2, the method further comprises the following steps.

Step 240: real-time superminute time is monitored for superdividing an original video frame into target video frames.

In some embodiments, the first device monitors in real time a real time superminute time for superdividing the original video frame into the target video frames in the event of the first device superminute.

In some embodiments, the real-time superminute time is a single frame superminute time acquired in real-time; or the real-time superminute time is the sum of the single-frame superminute time obtained in real time and the single-frame coding time obtained in real time; or the real-time superminute time is the sum of the real-time acquired single frame superminute time and the real-time acquired single frame decoding time; or the real-time superminute time is the sum of the real-time acquired single frame superminute time, the real-time acquired single frame decoding time and the real-time acquired single frame rendering time.

Step 250: and stopping the super-division calculation under the condition that the real-time super-division time of the continuous i original video frames is larger than the inter-frame time interval corresponding to the target frame rate.

The target frame rate is the playing frame rate of the target video frame, and i is a positive integer.

In some embodiments, i is a preset value; or, i is a value dynamically determined according to the video type corresponding to the original video frame. Illustratively, i is a preset value of 5; or the video corresponding to the original video frame is a video type with low delay requirement such as a short video, and i is 10; or the video corresponding to the original video frame is a video type with high delay requirement such as a cloud game, and i is 3.

In some embodiments, after stopping the super-division computation, the original video frame is sent to the second device; or after stopping the super-division calculation, playing the video frames in the form of original video frames.

In some embodiments, when the super-division calculation mode of the original video frame is that the first device performs super-division on the first video frame sequence, and the second device performs cooperative super-division on the second video frame sequence, the first device stops the super-division calculation, and sends information for stopping the super-division to the second device, so as to instruct the second device to stop the super-division calculation; or, if the super-division calculation mode of the original video frame is that the first device performs super-division on the first video frame sequence and the second device performs cooperative super-division on the second video frame sequence, the first device stops the super-division calculation.

In summary, after the super-division calculation is started, the method provided by the embodiment of the present application monitors the real-time super-division time of the first device when the super-division calculation is performed in real time, determines whether the real-time super-division time is greater than the inter-frame time interval corresponding to the target frame rate, and considers that the first device is not suitable for continuing super-division at this time if the real-time super-division time of the continuous i frames is greater than the inter-frame time interval corresponding to the target frame rate, otherwise, causes user-unfriendly experiences such as stuck and dropped frames when playing the video. Therefore, the super-score calculation is stopped to ensure that the user can smoothly acquire the video frame.

After the superscore calculating mode is determined, the first device performs superscore based on the superscore calculating mode.

Step 260: and performing superdivision on the original video frame based on the superdivision calculation mode of the original video frame.

In some embodiments, the first device performs the super-division on the original video frame based on the determined super-division calculation mode of the original video frame.

In some embodiments, step 260 may alternatively be implemented as any of steps 261 through 263.

In some embodiments, steps 261 through 263 are in a side-by-side relationship, and only one of steps 261 through 263 can be performed.

Step 261: and under the condition that the super-division calculation mode is that the first equipment performs super-division on the original video frame, the original video frame is super-divided into target video frames.

In some embodiments, in the case that the super-division calculation mode is that the first device performs super-division on the original video frame, the first device super-divides the original video frame into target video frames, and encodes the target video frames to obtain a target code stream; or the first device super-divides the original video frame into target video frames, and encodes a plurality of target video frames in a mode of image groups to obtain a target code stream.

In some embodiments, the first device obtains an original code stream, decodes the original code stream to obtain an original video frame, and superdivides the original video frame into target video frames based on a superdivision calculation mode of superdivision of the original video frame by the first device.

In some embodiments, the resolution of the resulting target video frame is the target resolution and the playback frame rate of the target video frame is the target frame rate.

Step 262: and under the condition that the super-division calculation mode is that the first equipment performs down-shift super-division on the original video frame, super-dividing the original video frame into target video frames.

The downshifting super-division is used for indicating that the original video frames are super-divided into target video frames, the resolution of the target video frames is reduced target resolution, and the playing frame rate of the target video frames is reduced target frame rate.

In some embodiments, in the case that the super-division calculation mode is that the first device performs down-shift super-division on the original video frame, the first device super-divides the original video frame into target video frames, and encodes the target video frames to obtain a target code stream; or the first device super-divides the original video frame into target video frames, and encodes a plurality of target video frames in a mode of image groups to obtain a target code stream.

In some embodiments, the first device obtains an original code stream, decodes the original code stream to obtain an original video frame, and superdividing the original video frame into target video frames based on a superdivision calculation mode of performing a downshift superdivision on the original video frame by the first device.

In some embodiments, the resolution of the resulting target video frame is a reduced target resolution and the play frame rate of the target video frame is a reduced target frame rate.

Step 263: in case the super-division calculation is based on super-dividing the original video frames belonging to the first video frame sequence by the first device, super-dividing the original video frames belonging to the first video frame sequence into target video frames.

In some embodiments, the original video frames belong to a first video frame sequence or a second video frame sequence, and in the case that the super-division calculation mode is based on super-dividing the original video frames belonging to the first video frame sequence by the first device, the first device super-divides the original video frames belonging to the first video frame sequence into target video frames, and encodes the video frames which are super-divided into the target video frames in the first video frame sequence and the original video frames which are not super-divided in the second video frame sequence to obtain a target code stream; or the first device super-divides the original video frames in the image group belonging to the first video frame sequence into target video frames, and encodes the video frames super-divided into the target video frames in the first video frame sequence and the original video frames not super-divided in the second video frame sequence to obtain the target code stream.

In some embodiments, the first device obtains a target code stream, and the second video frame sequence in the target code stream has completed super-division, and the original video frames in the first video frame sequence are super-divided into target video frames based on a super-division calculation mode of super-dividing the first video frame sequence by the first device.

In summary, the method provided in the embodiment of the present application shows a manner that the first device performs superdivision on the original video frame to obtain the target video frame based on the determined superdivision calculation manner of the original video frame, and the superdivision calculation performed by the first device is different for different superdivision calculation manners, so that the integrity and rationality of the whole superdivision flow can be ensured.

The following is a method for processing a video frame executed by a second device, where the second device is only responsible for uploading the superscore computing capability of the second device to the first device, and is responsible for superscore computing when the superscore computing mode requires the second device to participate in the superscore computing, i.e. the second device does not directly participate in the determination of the superscore computing mode.

Fig. 7 is a flowchart illustrating a method for processing a video frame according to an exemplary embodiment of the present application, where the method is performed by a second device, which is a computer device, and may be the terminal or the server shown in fig. 1. The method comprises the following steps.

Step 310: and obtaining target resolution, wherein the target resolution is the resolution after the super-division algorithm is applied to the original video frame, and the target resolution is larger than the original resolution of the original video frame.

Illustratively, the target resolution is a preset resolution, the target resolution is 4K; or, the target resolution is a dynamically set resolution, for example, the target resolution is a dynamically set resolution based on the transmission speed of the second device, and when the transmission speed of the second device is slow, the target resolution is lower, which is 1080P; when the transmission speed of the second device is higher, the target resolution is higher, and is 4K; or the target resolution is the resolution selected by the user, and the user selects the target resolution to be 2K based on the self requirement; or, the target resolution is a display resolution supported by the first device or the second device, for example, the second device is a playing end of the video, the highest resolution supported by the second device is 2K, and when the user selects the desired target resolution to be 4K, the target resolution is set to 2K based on the highest display resolution of the second device being 2K.

Step 320: uploading the supercomputing capabilities of the second device to the first device.

In some embodiments, where the second device is a server providing video frames, the super-resolution computing capability of the second device includes super-resolution computing time for different resolutions. Optionally, the super-resolution computing capability of the second device further comprises an encoding time for encoding the video frames into a bitstream.

In some embodiments, where the second device is a terminal that plays video frames, the super-resolution computing capability of the second device includes super-resolution computing times for different resolutions. Optionally, the super-resolution computing capability of the second device further includes a decoding time to decode the server transmitted bitstream into video frames. Optionally, the super-resolution computing capability of the second device further comprises a rendering time for rendering the decoded video frames to a display screen of the terminal.

Step 330: the method comprises the steps of obtaining a super-division calculation mode of an original video frame, wherein the super-division calculation mode of the original video frame is determined by at least one of super-division calculation capability of a first device and super-division calculation capability of a second device.

In some embodiments, the super-division calculation mode is used to instruct the first device and the second device to perform the super-division on the original video frame.

In some embodiments, the manner of superdivision calculation is determined based on the first frame time. The first frame time is used to indicate the time from the user triggering the play operation to the rendering of the first video frame, i.e. the time the user perceives from the application. The head frame time for a good user experience is 100ms. Illustratively, in order to ensure that the user has a good first frame experience, the first original video frame is not oversubscribed; or, the group of pictures (Group of Pictures, GOP) comprising the first original video frame is not super-divided; or, the top n original video frames are not subjected to super division; or, the top n image groups are not super-divided. The group of pictures is the basic unit of video encoding and decoding. The method includes the steps that in an application scene of short videos, the first video is not subjected to super-division, and in the playing process of the first video, the second video to the mth video are subjected to super-division; or, the first video is not subjected to super-division, and in the playing process of the first video, the first n original video frames from the second video to the mth video are subjected to super-division. Under the application scene of the long video, the first n original video frames are not subjected to super-division, and in the playing process of the first n original video frames, the original video frames after the n+1th video frames are subjected to super-division; or, the advertisement before and during the long video playing is not subjected to the super-division, and only the original video frames of the long video are subjected to the super-division. Under the application scene of online video, such as the scene that the generation and playing of video such as live broadcast, cloud game and the like are carried out simultaneously, the video delay requirement is higher, the frame extraction super-division is carried out on the original video frame, and the super-division is carried out on the original video frame serving as a key frame and part of predicted frames.

In some embodiments, the superscore calculation is based on a katen time determination. The clip time is used to indicate the display time of a single video frame that causes the video to have a clip feel, i.e., how long one video frame is displayed will cause the user to feel the video clip. The user perceives the video to have a clip time of 200ms. Illustratively, the manner of superdivision computation of the original video frame is determined based on the superdivision computation power and the click time. The super-division computing capability is single-frame super-division time, and the super-division computing mode of the original video frame is determined based on the single-frame super-division time and the katen time.

Step 340: and performing superdivision on the original video frame based on the superdivision calculation mode of the original video frame.

In some embodiments, the second device superdivides the original video frame into target video frames based on the determined manner of superdivision calculation of the original video frame.

In summary, according to the method provided by the embodiment of the present application, the second device is used as the execution device and the indicated device of the processing method of the video frame, the superdivision calculation mode determined by the first device is obtained, and the superdivision is performed on the original video frame based on the superdivision calculation mode, so that the purpose of superdivision on the video frame can be achieved, and the method reasonably utilizes the resources of the second device, so that the second device also has a chance to participate in the superdivision calculation of the video frame.

And the second equipment performs superdivision based on the determined superdivision calculation mode.

In an alternative embodiment based on fig. 7, step 340 may alternatively be implemented as step 341 or step 342.

Step 341: and under the condition that the super-division calculation mode is that the second equipment performs super-division on the original video frame, the original video frame is super-divided into target video frames.

In some embodiments, in the case that the super-division calculation mode is that the second device performs super-division on the original video frame, the second device super-divides the original video frame into target video frames, and encodes the target video frames to obtain a target code stream; or the second device super-divides the original video frame into target video frames and encodes a plurality of target video frames in a mode of image groups to obtain a target code stream.

In some embodiments, the second device obtains an original code stream, decodes the original code stream to obtain an original video frame, and superdividing the original video frame into target video frames based on a superdivision calculation mode of superdivision of the original video frame by the second device.

Step 342: and under the condition that the super-division calculation mode is that the second equipment super-divides the original video frames belonging to the second video frame sequence, the original video frames belonging to the second video frame sequence are super-divided into target video frames.

In some embodiments, the original video frames belong to a first video frame sequence or a second video frame sequence, and in the case that the super-division calculation mode is that the second device performs super-division on the original video frames belonging to the second video frame sequence, the second device super-divides the original video frames belonging to the second video frame sequence into target video frames, and encodes the video frames which are super-divided into the target video frames in the second video frame sequence and the original video frames which are not super-divided in the first video frame sequence to obtain a target code stream; or the second device super-divides the original video frames in the image group belonging to the second video frame sequence into target video frames, and encodes the video frames super-divided into the target video frames in the second video frame sequence and the original video frames not super-divided in the first video frame sequence to obtain the target code stream.

In some embodiments, the second device obtains a target code stream, and the first video frame sequence in the target code stream has completed super-division, and the original video frames in the second video frame sequence are super-divided into target video frames based on a super-division calculation mode of super-dividing the second video frame sequence by the second device.

In some embodiments, the second sequence of video frames comprises original video frames ordered as even; or, the second video frame sequence comprises original video frames ordered as odd.

In some embodiments, the group of images includes a plurality of original video frames. The second video frame sequence includes groups of images ordered even in the middle; or, the second video frame sequence includes groups of images ordered as odd.

In some embodiments, a ratio of the first video frame sequence to the original video frames included in the second video frame sequence is determined based on the single frame superminute time of the first device and the single frame superminute time of the second device. Illustratively, the ratio of the single-frame superminute time of the first device to the single-frame superminute time of the second device is 3/5, and then the ratio of the number of original video frames in the first video frame sequence to the number of original video frames in the second video frame sequence is set to be 5/3.

In summary, the method provided in the embodiment of the present application shows a manner of obtaining the target video frame by superdividing the original video frame by the second device based on the determined superdividing calculation manner of the original video frame, and the superdividing calculation performed by the second device is different for different superdividing calculation manners, so that the integrity and rationality of the whole superdividing flow can be ensured.

And when the second equipment performs the superscore calculation, monitoring the equipment state of the second equipment in real time, and judging whether the superscore is continued or not.

In an alternative embodiment based on fig. 7, the method further comprises the following steps.

Step 350: real-time superminute time is monitored for superdividing an original video frame into target video frames.

In some embodiments, the second device monitors in real time the real time superminute time of superdividing the original video frame into the target video frame in the case of superminute by the second device.

Step 360: and stopping the super-division calculation under the condition that the real-time super-division time of the continuous i original video frames is larger than the inter-frame time interval corresponding to the target frame rate.

In some embodiments, after stopping the super-division computation, the original video frame is sent to the first device; or after stopping the super-division calculation, playing the video frames in the form of original video frames.

In some embodiments, when the super-division calculation mode of the original video frame is that the first device performs super-division on the first video frame sequence, and the second device performs cooperative super-division on the second video frame sequence, the second device stops the super-division calculation, and sends information of stopping the super-division to the first device, so as to instruct the first device to stop the super-division calculation; or, if the super-division calculation mode of the original video frame is that the first device performs super-division on the first video frame sequence and the second device performs cooperative super-division on the second video frame sequence, the second device stops the super-division calculation.

In summary, after the super-division calculation is started, the method provided by the embodiment of the application monitors the real-time super-division time of the second device in real time when the super-division calculation is performed, determines whether the real-time super-division time is greater than the inter-frame time interval corresponding to the target frame rate, and considers that the second device is not suitable for continuing super-division at this time if the real-time super-division time of the continuous i frames is greater than the inter-frame time interval corresponding to the target frame rate, otherwise, user-unfriendly experiences such as stuck and dropped frames occur when the video is played. Therefore, the super-score calculation is stopped to ensure that the user can smoothly acquire the video frame.

The second device detects the manner in which the second device's supercomputing capabilities.

Step 410: the test code streams with different resolutions are decoded, the decoded test code streams are subjected to superdivision, and the processed test code streams are rendered and played, so that the superdivision computing capability of the second equipment is detected.

In some embodiments, when detecting the super-division computing capability of the second device, the second device is introduced with test code streams with different resolutions, the second device decodes the test code streams with different resolutions, super-divides the decoded test code streams, and renders and plays the test code streams subjected to the super-division computation in the second device. When the test code stream is subjected to super-division, the test code stream is super-divided into different resolutions, so that the detection of the super-division computing capacity of the complete second equipment is realized; or when the test code stream is subjected to super division, the test code stream is super-divided into target resolution, so that the super-division computing capacity of the second equipment is rapidly detected.

Step 420: and recording the single frame superminute time and the single frame rendering time in the detection process.

In some embodiments, a single frame superminute time during the probing is recorded; or, recording single frame superminute time, single frame rendering time and single frame decoding time in the detection process.

In some embodiments, when the super-division computing capability is detected, the decoding of the test code stream and the rendering and playing of the super-division computed test code stream are performed simultaneously, so that the algorithm of the super-division computing capability corresponding to the second device is different according to the execution conditions of the three actions.

Step 430: and calculating the superdivision computing capability of the second device, wherein the superdivision computing capability of the second device is the sum of the single-frame superdivision time and the single-frame rendering time.

In some embodiments, the superminute computing power of the second device is computed, the superminute computing power of the second device being a single frame superminute time; or, the super-division computing capability of the second device is the sum of single-frame super-division time and single-frame rendering time; or, the super-division computing capability of the second device is a sum of a single frame super-division time, a single frame rendering time, and a single frame decoding time.

In summary, the method provided in the embodiment of the present application shows a method for detecting the superdivision computing capability of the second device by the second device, where the superdivision computing capability includes single frame superdivision time and single frame rendering time, the superdivision computing capability of the second device is obtained first before superdivision is performed, and then how to perform superdivision is determined based on the superdivision computing capability of the second device, instead of directly superdivision the second device or the first device, which can well consider the specific state of the second device, so that the whole superdivision process is more reasonable.

In order to better understand the present application, the following will exemplify the first device as a server and the second device as a terminal.

Fig. 8 is a flowchart illustrating a method for processing a video frame according to an exemplary embodiment of the present application.

Step 510: the terminal obtains the target resolution and the target frame rate selected by the user.

In some embodiments, the target resolution is a resolution after applying a super-resolution algorithm to the original video frame, the target resolution being greater than the original resolution of the original video frame.

Step 511: the display capability of the terminal is detected.

In some embodiments, the display capabilities include a maximum resolution and a maximum frame rate that can be displayed by a screen of the terminal. In an android device, the Display capability may be obtained through a Display interface provided by an android official, for example, the maximum resolution supported by the screen may be obtained through a getDefaultDisplay () method, and the maximum frame rate supported by the screen may be obtained through a getsupptedmodes () method.

Step 512: the display capabilities of the terminal match the target resolution and target frame rate selected by the user.

In some embodiments, determining whether the target resolution is greater than a maximum resolution, if so, then no match; and if not, matching. Judging whether the target frame rate is greater than the highest frame rate, if so, not matching; and if not, matching. If one of the target resolution and the target frame rate is not matched, the target resolution selected by the user is not matched with the target frame rate and the display capability of the terminal.

Step 514 is performed in case the display capability of the terminal matches the target resolution and the target frame rate selected by the user; in case the display capability of the terminal does not match the target resolution and the target frame rate selected by the user, step 513 is performed.

Step 513: the target resolution and the target frame rate are set to the maximum resolution and the highest frame rate supported by the terminal.

Step 514: whether the server natively supports the target resolution and the target frame rate.

In some embodiments, the server queries whether the server supports directly providing the terminal with a target code stream or target video, the target code stream or target video having a target resolution, and the target code stream or target video having a target frame rate. If the server originally supports the target resolution and the target frame rate, the method indicates that the target code stream or the target video can be directly provided for the user without super-division.

In the case that the server natively supports the target resolution and the target frame rate, step 515 is performed; in the event that the server is unable to natively support the target resolution and the target frame rate, step 516 is performed.

Step 515: the server transmits the code stream with the target resolution to the terminal for playing.

The server directly transmits a target code stream or a target video to the terminal, wherein the resolution of the target code stream or the target video is the target resolution, and the playing frame rate of the target code stream or the target video is the target frame rate.

Step 516: and acquiring the superscore computing capacity of the terminal and the superscore computing capacity of the server.

In some embodiments, the server's super-resolution computing capabilities include super-resolution computing time for different resolutions. Optionally, the super-resolution computing capability of the server further includes an encoding time to encode the video frames into a bitstream.

In some embodiments, the super-resolution computing capabilities of the terminal include super-resolution computing times for different resolutions. Optionally, the super-resolution computing capability of the terminal further includes a decoding time for decoding the code stream transmitted by the server into video frames. Optionally, the super-resolution computing capability of the terminal further comprises a rendering time for rendering the decoded video frames to a display screen of the terminal. The calculation manner of the super-resolution calculation capability of the terminal is shown in the exemplary embodiments corresponding to the steps 410 to 430, and will not be described herein.

Step 517: whether the super-resolution computing capability of the terminal meets the target resolution and the target frame rate.

In some embodiments, it is determined whether the super-resolution computing capability of the terminal for the target resolution is not greater than an inter-frame time interval corresponding to the target frame rate. The super-division computing capability of the terminal is the sum of single-frame super-division time and single-frame rendering time, and whether the sum of the single-frame super-division time of the terminal for super-dividing the original video frame into the target video frame and the single-frame rendering time for rendering the target video frame on the terminal screen is not larger than the inter-frame time interval corresponding to the target frame rate is judged.

If the super-resolution computing capability of the terminal meets the target resolution and the target frame rate, executing step 518; if the super-resolution computing capability of the terminal does not meet the target resolution and the target frame rate, step 519 is performed.

Step 518: it is determined that the original video frame is superdivided by the terminal.

In some embodiments, the server determines how to superdivide the original video frames by the terminal. The server transmits an original code stream or an original video to the terminal, and informs the terminal of the determined superminute calculation mode. After receiving the super-division of the original video frame by the terminal, the terminal decodes the original code stream or the original video to obtain the original video frame, super-divides the original video frame to obtain the target video frame, and finally obtains the target code stream or the target video.

In some embodiments, when the terminal performs the super division on the original video frame, the super division time of the single frame during the super division is monitored, and whether to continue the super division is determined according to the Shan Zhen super division time, as shown in the exemplary embodiments corresponding to the step 240 and the step 250 or the exemplary embodiments corresponding to the step 350 and the step 360, which are not described herein again.

Step 519: whether the super-resolution computing capability of the server meets the target resolution and the target frame rate.

In some embodiments, it is determined whether the super-resolution computing power of the server for the target resolution is not greater than an inter-frame time interval corresponding to the target frame rate. The super-division computing capability of the server is exemplified as single-frame super-division time, and whether the single-frame super-division time of the server for super-dividing the original video frame into the target video frame is not more than the inter-frame time interval corresponding to the target frame rate is judged.

If the super-resolution computing capability of the server meets the target resolution and the target frame rate, step 520 is executed; if the super resolution calculation capability of the server does not meet the target resolution and the target frame rate, any one of steps 521, 523, and 524 is executed.

Step 520: it is determined that the original video frame is superdivided by the server.

In some embodiments, the server determines how to super-divide the original video frames by the server. The server performs super-division on the acquired original video frames to obtain target video frames, encodes the target video frames into target code streams or target videos, then transmits the target video frames to the terminal, and informs the terminal that the target video frames are directly played without super-division.

In some embodiments, when the server performs the super division on the original video frame, the super division time of the single frame during the super division is monitored, and whether to continue the super division is determined according to the Shan Zhen super division time, as shown in the exemplary embodiments corresponding to the step 240 and the step 250 or the exemplary embodiments corresponding to the step 350 and the step 360, which are not described herein again.

Step 521: whether the super-resolution computing capability of the server satisfies at least one of the reduced target resolution and the reduced target frame rate.

In some embodiments, the server decreases at least one of the target resolution and the target frame rate, and determines whether the super-resolution computing capability of the server satisfies at least one of the decreased target resolution and the target frame rate.

In some embodiments, the server decreases at least one of the target resolution and the target frame rate according to the capability configuration table. The capacity configuration table is obtained by calling the hyper-division algorithm when the server is idle, and is used for indicating the target resolution and the target frame rate supported by the server.

The method for reducing at least one of the target resolution and the target frame rate by the server is shown in the exemplary embodiment corresponding to the third mode, and will not be described herein.

Step 522 is performed in the case where the super-resolution computing capability of the server satisfies at least one of the reduced target resolution and the reduced target frame rate; step 524 is performed in the event that the server's super-resolution computing power does not meet at least one of the reduced target resolution and the reduced target frame rate.

Step 522: it is determined that the original video frame is superdivided by the server.

In some embodiments, the server determines how to super-divide the original video frames by the server. The server performs super-division on the acquired original video frames to obtain target video frames, encodes the target video frames into target code streams or target videos, then transmits the target video frames to the terminal, and informs the terminal that the target video frames are directly played without super-division. The target resolution at this time is a reduced target resolution, or the target frame rate is a reduced target frame rate, or the target resolution is a reduced target resolution and the target frame rate is a reduced target frame rate.

Step 523: and determining that the terminal and the server perform cooperative superdivision on the original video frames.

Under the condition that the super-resolution computing capacity of the terminal or the super-resolution computing capacity of the server does not meet the target resolution or the target frame rate, judging whether the super-resolution computing capacity of the terminal and the super-resolution computing capacity of the server meet the target resolution and the target frame rate, determining a cooperative mode of cooperative super-resolution of the original video frames by the terminal and the server, super-dividing the original video frames belonging to the first video frame sequence by the terminal, and super-dividing the original video frames belonging to the second video frame sequence by the server.

In some embodiments, the first sequence of video frames comprises raw video frames ordered odd, and the second sequence of video frames comprises raw video frames ordered even; alternatively, the first video frame sequence comprises original video frames ordered as an even number and the second video frame sequence comprises original video frames ordered as an odd number.

In some embodiments, each group of images includes a plurality of original video frames. The first video frame sequence comprises image groups ordered as odd numbers, and the second video frame sequence comprises image groups ordered as even numbers; alternatively, the first video frame sequence comprises groups of pictures ordered even and the second video frame sequence comprises groups of pictures ordered odd.

In some embodiments, the ratio of the first video frame sequence to the original video frames included in the second video frame sequence is determined based on the single-frame superstration time of the terminal and the single-frame superstration time of the server. That is, the ratio of the first video frame sequence to the original video frame included in the second video frame sequence is the inverse of the ratio of the single-frame superminute time of the terminal to the single-frame superminute time of the server. For example, if the ratio of the single-frame superminute time of the terminal to the single-frame superminute time of the server is 3/5, the ratio of the number of original video frames in the first video frame sequence to the number of original video frames in the second video frame sequence is set to be 5/3.

In some embodiments, a ratio of the first video frame sequence to the original video frames included in the second video frame sequence is determined based on the super-division computing capability of the terminal and the super-division computing capability of the server. That is, the ratio of the original video frames included in the first video frame sequence to the second video frame sequence is the reciprocal of the ratio of the super-division computing power of the terminal to the super-division computing power of the server. Illustratively, the super-division calculation of the terminal can be single-frame super-division time, and the super-division calculation capability of the server is the sum of the single-frame super-division time and the single-frame rendering time. And setting the ratio of the super-division computing capacity of the terminal to the super-division computing capacity of the server to be 5/6, and setting the ratio of the original video frames included in the first video frame sequence to the second video frame sequence to be 6/5.

In some embodiments, it is determined that the terminal and the server perform cooperative superdivision, the server obtains an original video or an original code stream, superdivision is performed on original video frames belonging to a second video frame sequence, video frames in the second video frame sequence which have been superdivided into target video frames and video frames in the first video frame sequence are encoded and then sent to the terminal, the terminal superdivision is performed on the original video frames in the first video frame sequence to obtain target video frames, and the whole target code stream or the target video is played.

Step 524: it is determined that the original video frame is not super-divided.

In summary, the method provided by the embodiment of the present application shows the whole process of switching the resolution when the user requests, and directly plays the original video for the user when no super-resolution is required or neither the super-resolution computing capability is satisfied, so as to ensure the minimum experience of the user; when at least one of the terminal and the server supports the super division, the original video frame is super-divided into target video frames and then played, so that the user requirement is met.

Referring to fig. 9, a block diagram of a video frame processing apparatus according to an embodiment of the present application is shown. The device has the function of realizing the video frame processing method example, and the function can be realized by hardware or by executing corresponding software by hardware. The device may be the terminal or the server described above, or may be provided in the terminal or the server. As shown in fig. 9, the apparatus 600 may include: a first acquisition module 610, a second acquisition module 620, and a determination module 630.

A first obtaining module 610, configured to obtain a target resolution, where the target resolution is a resolution obtained by applying a super-resolution algorithm to an original video frame, and the target resolution is greater than the original resolution of the original video frame;

a second obtaining module 620, configured to obtain a super-resolution computing capability of the first device and a super-resolution computing capability of a second device associated with the first device, where the super-resolution computing capability of the first device includes super-resolution computing times of the first device for different resolutions, and the super-resolution computing capability of the second device includes super-resolution computing times of the second device for different resolutions;

a determining module 630, configured to determine a super-division calculation manner of the original video frame based on at least one of the super-division calculation capability of the first device and the super-division calculation capability of the second device.

In some embodiments, the determination module 630 includes a first determination sub-module.

And the first determining submodule is used for determining that the second equipment performs super-division on the original video frame under the condition that the single-frame super-division time of the second equipment is not larger than the inter-frame time interval corresponding to the target frame rate.

In some embodiments, the determination module 630 includes a second determination sub-module.

And the second determining submodule is used for determining that the first equipment performs super-division on the original video frame under the condition that the single-frame super-division time of the second equipment is larger than the inter-frame time interval of the target frame rate and the single-frame super-division time of the first equipment is not larger than the inter-frame time interval of the target frame rate.

In some embodiments, the determination module 630 includes at least one of a third determination sub-module and a fourth determination sub-module.

A third determining sub-module, configured to determine, when the single-frame superdivision time of the second device and the single-frame superdivision time of the first device are both greater than an inter-frame time interval of a target frame rate, that the first device performs a downshifting superdivision on the original video frame, where the downshifting superdivision is used to instruct to superdivide the original video frame into a reduced target resolution, or modify a play frame rate of the target video frame into a reduced target frame rate, or superdivide the original video frame into a reduced target resolution and modify a play frame rate of the target video frame into a reduced target frame rate;

and a fourth determining sub-module, configured to determine not to oversubscribe the original video frame in a case where the single-frame oversubscribe time of the first device and the single-frame oversubscribe time of the second device are both greater than the inter-frame time interval of the target frame rate.

In some embodiments, the third determination submodule includes at least one of a first lowering unit, a second lowering unit, and a third lowering unit.

A first reducing unit, configured to reduce the target frame rate, in a case where both the single-frame superminute time of the first device and the single-frame superminute time of the second device are greater than an inter-frame time interval of the target frame rate, so that the target resolution and the reduced target frame rate satisfy the single-frame superminute time of the first device;

a second reducing unit, configured to reduce the target resolution, in a case where both the single-frame super-resolution time of the first device and the single-frame super-resolution time of the second device are greater than an inter-frame time interval of the target frame rate, so that the reduced target resolution and the target frame rate satisfy the single-frame super-resolution time of the first device;

a third reducing unit, configured to reduce the target resolution and the target frame rate, when the single-frame super-resolution time of the first device and the single-frame super-resolution time of the second device are both greater than the inter-frame time interval of the target frame rate, so that the reduced target resolution and the reduced target frame rate satisfy the single-frame super-resolution time of the first device.

In some embodiments, the determination module 630 includes a fifth determination sub-module.

And a fifth determining submodule, configured to perform cooperative oversubscription calculation on the original video frames under the condition that the single-frame oversubscription time of the first device and the single-frame oversubscription time of the second device are both greater than an inter-frame time interval of a target frame rate, determine that the first device oversubscripts the original video frames belonging to the first video frame sequence, and the second device oversubscripts the original video frames belonging to the second video frame sequence, where the cooperative oversubscription calculation is that the first device and the second device oversubscripts different original video frames.

In some embodiments, the apparatus 600 further comprises a first monitoring module and a first stopping module.

The first monitoring module is used for monitoring real-time superdivision time for superdividing the original video frame into the target video frame;

and the first stopping module is used for stopping the super-division calculation under the condition that the real-time super-division time of the continuous i original video frames is larger than the inter-frame time interval corresponding to the target frame rate.

In some embodiments, the apparatus 600 further comprises a first superdivision module.

The first super-division module is used for super-dividing the original video frame based on the super-division calculation mode of the original video frame.

In some embodiments, the first supermolecular module comprises at least one of a first supermolecular module, a second supermolecular module, and a third supermolecular module.

The first supermolecular module is used for superdividing the original video frame into the target video frame under the condition that the superdivision calculation mode is that the first equipment superdivides the original video frame;

the second supermolecular module is used for superdividing the original video frame into target video frames under the condition that the superdivision calculation mode is that the first equipment performs downshifting superdivision on the original video frame;

and the third supermolecular module is used for superdividing the original video frames belonging to the first video frame sequence into the target video frames under the condition that the superdivision calculation mode is based on the superdivision of the original video frames belonging to the first video frame sequence by the first equipment.

In summary, the apparatus provided in the embodiment of the present application determines the superscore computing manner of the original video frame based on at least one of the superscore computing capability of the first device and the superscore computing capability of the second device, and comprehensively considers the superscore computing capability of the server and the terminal when the superscore computing is performed on the video frame, so that the judgment of the superscore computing is more reasonable, and the superscore experience of the user can be improved.

Referring to fig. 10, a block diagram of a video frame processing apparatus according to an embodiment of the present application is shown. The device has the function of realizing the video frame processing method example, and the function can be realized by hardware or by executing corresponding software by hardware. The device may be the terminal or the server described above, or may be provided in the terminal or the server. As shown in fig. 10, the apparatus 700 may include: a third acquisition module 710, an upload module 720, a fourth acquisition module 730, and a second superdivision module 740.

A third obtaining module 710, configured to obtain a target resolution, where the target resolution is a resolution obtained by applying a super-resolution algorithm to an original video frame, and the target resolution is greater than the original resolution of the original video frame;

an uploading module 720, configured to upload the supercomputing capability of the second device to the first device;

a fourth obtaining module 730, configured to obtain a super-division calculation manner of an original video frame, where the super-division calculation manner of the original video frame is determined by at least one of a super-division calculation capability of the first device and a super-division calculation capability of the second device;

the second superdivision module 740 is configured to superdivide the original video frame based on a superdivision calculation mode of the original video frame.

In some embodiments, the second supermolecular module 740 includes at least one of a fourth supermolecular module and a fifth supermolecular module.

A fourth supermolecular module, configured to superdivide the original video frame into target video frames when the superdivision calculation mode is based on superdivision of the original video frame by the second device;

and a fifth supermolecular module, configured to superdivide, when the superdivision calculation mode is based on superdivision of an original video frame belonging to a second video frame sequence by a second device, the original video frame belonging to the second video frame sequence into the target video frame.

In some embodiments, the apparatus 700 further includes a second monitoring module 750 and a second stopping module 760.

A second monitoring module 750, configured to monitor a real-time superdivision time for superdividing the original video frame into the target video frame;

the second stopping module 760 is configured to stop the super-resolution calculation when the real-time super-resolution time of consecutive i original video frames is greater than the inter-frame time interval corresponding to the target frame rate.

In some embodiments, the apparatus 700 further comprises a detection module, a recording module, and a calculation module.

The detection module is used for realizing the detection of the super-resolution computing capacity of the second equipment by decoding the test code stream code streams with different resolutions, applying a super-resolution algorithm to the decoded test code stream and rendering and playing the processed test code stream;

the recording module is used for recording the single-frame super-division algorithm process time and the single-frame rendering time in the detection process;

and the calculation module is used for calculating the super-division calculation capability of the second equipment, wherein the super-division calculation capability of the second equipment is the sum of the Shan Zhen super-division algorithm process time and the single-frame rendering time.

In summary, according to the apparatus provided by the embodiment of the present application, the second device is used as the executing device and the indicated device of the processing method of the video frame, the superdivision calculation mode determined by the first device is obtained, and the superdivision is performed on the original video frame based on the superdivision calculation mode, so that the purpose of superdivision on the video frame can be achieved, and the method reasonably utilizes the resources of the second device, so that the second device also has a chance to participate in the superdivision calculation of the video frame.

It should be noted that: in the device provided in the above embodiment, when implementing the functions thereof, only the division of the above functional modules is used as an example, in practical application, the above functional allocation may be implemented by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to implement all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein.

Fig. 11 shows a block diagram of a computer device according to an exemplary embodiment of the present application.

The computer apparatus 800 includes a central processing unit (Central Processing Unit, CPU) 801, a system Memory 804 including a random access Memory (Random Access Memory, RAM) 802 and a Read-Only Memory (ROM) 803, and a system bus 805 connecting the system Memory 804 and the central processing unit 801. The computer device 800 also includes a basic Input/Output system (I/O) 806 for facilitating the transfer of information between the various devices within the computer device, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

The basic input/output system 806 includes a display 808 for displaying information and an input device 809, such as a mouse, keyboard, or the like, for user input of information. Wherein the display 808 and the input device 809 are connected to the central processing unit 801 via an input output controller 810 connected to the system bus 805. The basic input/output system 806 can also include an input/output controller 810 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input output controller 810 also provides output to a display screen, a printer, or other type of output device.

The mass storage device 807 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable storage media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include a computer readable storage medium (not shown) such as a hard disk or a compact disk-Only (CD-ROM) drive.

The computer-readable storage medium may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable storage instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read-Only register (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-Only Memory (EEPROM), flash Memory or other solid state Memory technology, CD-ROM, digital versatile disks (Digital Versatile Disc, DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the one described above. The system memory 804 and mass storage device 807 described above may be collectively referred to as memory.

The memory stores one or more programs configured to be executed by the one or more central processing units 801, the one or more programs containing instructions for implementing the above-described method embodiments, the central processing unit 801 executing the one or more programs to implement the methods provided by the various method embodiments described above.

According to various embodiments of the present application, the computer device 800 may also operate through a network, such as the Internet, to remote computer devices on the network. I.e., the computer device 800 may be connected to a network 812 through a network interface unit 811 connected to the system bus 805, or other types of networks or remote computer device systems (not shown) may be connected using the network interface unit 811.

The memory further includes one or more programs stored in the memory, the one or more programs including steps for performing the methods provided by the embodiments of the present application, performed by the terminal device.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, on which a computer program is stored, which when executed by a processor, implements the above-described video frame processing method.

In an exemplary embodiment, a computer program product is also provided, which, when being executed by a processor, is adapted to carry out the above-mentioned method of processing video frames.

It should be understood that references herein to "a plurality" are to two or more. The character "/" generally indicates that the context-dependent object is an "or" relationship. In addition, the step numbers described herein are merely exemplary of one possible execution sequence among steps, and in some other embodiments, the steps may be executed out of the order of numbers, such as two differently numbered steps being executed simultaneously, or two differently numbered steps being executed in an order opposite to that shown, which is not limited by the embodiments of the present application.

The foregoing description of the preferred embodiments is merely exemplary in nature and is in no way intended to limit the invention, since it is intended that all modifications, equivalents, improvements, etc. that fall within the spirit and scope of the invention.

Claims

1. A method of processing video frames, the method performed by a first device, the method comprising:

determining a super-resolution calculation mode of the original video frame based on at least one of the super-resolution calculation capability of the first device and the super-resolution calculation capability of the second device;

the determining a super-division calculation mode of the original video frame based on at least one of the super-division calculation capability of the first device and the super-division calculation capability of the second device includes:

under the condition that the single-frame overdriving time of the second device and the single-frame overdriving time of the first device are both larger than the inter-frame time interval of the target frame rate, determining that the first device performs downshifting overdriving on the original video frame, or determining that the original video frame is subjected to cooperative overdriving calculation, or determining that the original video frame is not subjected to overdriving;

wherein the superdivision computing capability includes the Shan Zhen superdivision time, the Shan Zhen superdivision time being used to indicate a time to superdivide the original video frame into target video frames; the downshifting superdivision is used for indicating that the original video frame is superdivided into a reduced target resolution, or the playing frame rate of the target video frame is modified into a reduced target frame rate, or the original video frame is superdivided into the reduced target resolution and the playing frame rate of the target video frame is modified into the reduced target frame rate, and the target frame rate is the playing frame rate of the target video frame; the collaborative superdivision calculation is that the first device and the second device superdivide different original video frames, the collaborative superdivision calculation comprises the first device superdividing the original video frames belonging to a first video frame sequence, and the second device superdividing the original video frames belonging to a second video frame sequence; the first video frame sequence and the second video frame sequence comprise different original video frames belonging to the same video or the same code stream; the original video frame belongs to the first video frame sequence or the second video frame sequence.

2. The method of claim 1, wherein the determining the manner of superscore computation of the original video frame based on at least one of the superscore computing capability of the first device and the superscore computing capability of the second device further comprises:

and under the condition that the single frame superdivision time of the second equipment is not greater than the inter-frame time interval corresponding to the target frame rate, the second equipment is determined to superdivide the original video frame.

3. The method of claim 1, wherein the determining the manner of superscore computation of the original video frame based on at least one of the superscore computing capability of the first device and the superscore computing capability of the second device further comprises:

and under the condition that the single-frame super-division time of the second device is larger than the inter-frame time interval of the target frame rate and the single-frame super-division time of the first device is not larger than the inter-frame time interval of the target frame rate, determining that the first device super-divides the original video frame.

4. The method of claim 1, wherein determining, by the first device, to downshift the original video frame if the single-frame superscalar time of the first device and the single-frame superscalar time of the second device are both greater than an inter-frame time interval of a target frame rate, comprises:

Reducing the target frame rate under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, so that the target resolution and the reduced target frame rate meet the single-frame superminute time of the first equipment; or alternatively, the first and second heat exchangers may be,

reducing the target resolution under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, so that the reduced target resolution and the target frame rate meet the single-frame superminute time of the first equipment; or alternatively, the first and second heat exchangers may be,

and under the condition that the single-frame superminute time of the first equipment and the single-frame superminute time of the second equipment are both larger than the inter-frame time interval of the target frame rate, reducing the target resolution and the target frame rate, so that the reduced target resolution and the reduced target frame rate meet the single-frame superminute time of the first equipment.

5. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the first video frame sequence comprises the original video frames ordered odd, and the second video frame sequence comprises the original video frames ordered even; or alternatively, the first and second heat exchangers may be,

The first sequence of video frames includes the even ordered original video frames and the second sequence of video frames includes the odd ordered original video frames.

6. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the first video frame sequence comprises image groups ordered as odd numbers, and the second video frame sequence comprises image groups ordered as even numbers; or alternatively, the first and second heat exchangers may be,

the first video frame sequence comprises the even-ordered image groups, and the second video frame sequence comprises the odd-ordered image groups;

7. A method of processing video frames, the method performed by a second device, the method comprising:

Performing superdivision on the original video frame based on the superdivision calculation mode of the original video frame;

wherein the superdivision computing capability comprises a single frame superdivision time, the Shan Zhen superdivision time being used to indicate a time to superdivide the original video frame into target video frames; the super-division calculation mode comprises a first super-division calculation mode, and the first super-division calculation mode comprises the following steps: performing, by the first device, any one of downshifting the original video frame, performing a collaborative oversubscription calculation on the original video frame, and not oversubscription of the original video frame, the first oversubscription calculation being determined in the case of an inter-frame time interval in which both a single-frame oversubscription time of the second device and a single-frame oversubscription time of the first device are greater than a target frame rate; the downshifting superdivision is used for indicating that the original video frame is superdivided into a reduced target resolution, or the playing frame rate of the target video frame is modified into a reduced target frame rate, or the original video frame is superdivided into the reduced target resolution and the playing frame rate of the target video frame is modified into the reduced target frame rate, and the target frame rate is the playing frame rate of the target video frame; the collaborative superdivision calculation is that the first device and the second device superdivide different original video frames, the collaborative superdivision calculation comprises the first device superdividing the original video frames belonging to a first video frame sequence, and the second device superdividing the original video frames belonging to a second video frame sequence; the first video frame sequence and the second video frame sequence comprise different original video frames belonging to the same video or the same code stream; the original video frame belongs to the first video frame sequence or the second video frame sequence.

8. The method of claim 7, wherein the means for superdividing further comprises a second means for superdividing the original video frame by the second device, the second means for superdividing being determined if a single frame superdividing time of the second device is not greater than an inter-frame time interval corresponding to the target frame rate.

9. The method of claim 7, wherein the means for superdividing further comprises a third means for superdividing the original video frame by the first device, the third means for superdividing being determined if a single frame superdividing time of the second device is greater than an inter-frame time interval of the target frame rate and a single frame superdividing time of the first device is not greater than an inter-frame time interval of the target frame rate.

10. The method according to any one of claims 7 to 9, wherein the super-dividing the original video frame based on the super-division calculation method of the original video frame includes:

under the condition that the super-division calculation mode is that the second equipment performs super-division on the original video frame, the original video frame is super-divided into target video frames; or alternatively, the first and second heat exchangers may be,

And under the condition that the super-division calculation mode is that the second equipment performs super-division on the original video frames belonging to the second video frame sequence, the original video frames belonging to the second video frame sequence are super-divided into the target video frames.

11. A video frame processing apparatus, the apparatus comprising:

a second obtaining module, configured to obtain a super-resolution computing capability of a first device and a super-resolution computing capability of a second device associated with the first device, where the super-resolution computing capability of the first device includes super-resolution computing times of the first device for different resolutions, and the super-resolution computing capability of the second device includes super-resolution computing times of the second device for different resolutions;

a determining module, configured to determine a super-division calculation manner of the original video frame based on at least one of a super-division calculation capability of the first device and a super-division calculation capability of the second device;

The determining module includes:

a third determining sub-module, configured to determine, when the single-frame superdivision time of the second device and the single-frame superdivision time of the first device are both greater than an inter-frame time interval of a target frame rate, to perform downshifting superdivision on the original video frame by the first device; or, a fifth determining submodule, configured to perform collaborative super-division calculation on the original video frame under a condition that a single-frame super-division time of the first device and a single-frame super-division time of the second device are both greater than an inter-frame time interval of the target frame rate; or, a fourth determining submodule, configured to determine not to oversubscribe the original video frame in a case where both the single-frame oversubscribe time of the first device and the single-frame oversubscribe time of the second device are greater than an inter-frame time interval of the target frame rate;

12. A video frame processing apparatus, the apparatus comprising:

the uploading module is used for uploading the super-resolution computing capacity of the second device to the first device;

the second super-division module is used for super-dividing the original video frame based on the super-division calculation mode of the original video frame;

13. A computer device, the computer device comprising: a processor and a memory, wherein at least one section of program is stored in the memory; the processor is configured to execute the at least one program in the memory to implement the method for processing a video frame according to any one of claims 1 to 6, or the method for processing a video frame according to any one of claims 7 to 10.

14. A computer readable storage medium having stored therein executable instructions loaded and executed by a processor to implement the method of processing video frames according to any of the preceding claims 1 to 6 or the method of processing video frames according to any of the preceding claims 7 to 10.