WO2022068326A1

WO2022068326A1 - Image frame prediction method and electronic device

Info

Publication number: WO2022068326A1
Application number: PCT/CN2021/106928
Authority: WO
Inventors: 李煜; 赵铎; 胡笑鸣; 黄开兴; 蒋铭辉; 陈健; 周越海; 王亮; 高三山; 季美辰; 朱欢欢; 王军; 沈勇武
Original assignee: 华为技术有限公司
Priority date: 2020-09-30
Filing date: 2021-07-16
Publication date: 2022-04-07
Also published as: CN115398907A

Abstract

An image frame prediction method and an electronic device. According to the method, an electronic device can draw a first moving object in a first memory space and draw a first static object in a second memory space according to a drawing instruction for an N-th drawn frame, and draw a second moving object in a third memory space and draw a second static object in a fourth memory space according to a drawing instruction for an (N+2)-th drawn frame; the electronic device predicts a moving object in an (N+3)-th predicted frame according to the first moving object and the second moving object, and predicts a static object in the (N+3)-th predicted frame according to the first static object and the second static object; finally, the electronic device synthesizes a moving object and a static object of an N-th predicted frame into the (N+3)-th predicted frame. By implementing the technical solution provided by the present application, the electronic device can predict an image frame more accurately, and improve the frame rate at which an application plays back a video by using the predicted image frame, thereby improving the fluency of a video interface.

Description

A kind of image frame prediction method and electronic device

This application claims the priority of the Chinese patent application filed on September 30, 2020, with the application number of 202011069443.8 and the application title of "A Frame Prediction Method, Electronic Device and Computer-readable Storage Medium"; and filed in 2020 The priority of the Chinese patent application filed with the China Patent Office on September 30, 2020 with the application number 202011063375.4 and the application name is "Image Frame Generation Method and Electronic Device"; and filed with the China Patent Office on October 31, 2020, with the application number It is the priority of the Chinese patent application of 202011197968.X with the application title of "A method and electronic device for image frame prediction"; The priority of the Chinese patent application for a method and electronic device for image frame prediction; the priority of the Chinese patent application; and the priority of the Chinese patent application filed on December 16, 2020 with the application number 202011493948.7 and the application title "A method and electronic device for image frame prediction"; and The priority of the Chinese patent application filed on December 30, 2020 with the application number 202011629171.2 and the application title "A method and electronic device for image frame prediction"; the entire content of the above patent application is incorporated herein by reference Applying.

technical field

The present application relates to the field of electronic technology and the field of image processing, and in particular, to a method and electronic device for image frame prediction.

Background technique

The video interface displayed by the electronic device (video playing interface such as TV series, movie, game screen, etc.) is essentially a continuous picture. Taking a game screen as an example, the higher the frame rate of the game screen, the smoother the game screen displayed by the electronic device, and the better the user's visual experience. For games that need to be rendered in real time, the higher the frame rate, the more image frames (drawing frames for short) that the electronic device needs to draw and render by applications (video applications, game applications, etc.), and the greater the power consumption of the electronic device. Therefore, how to improve the smoothness of the video interface displayed by the electronic device under the condition of saving the power consumption of the electronic device is an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides an image frame prediction method and electronic device, which can improve the smoothness of the video interface displayed by the electronic device under the condition of saving the power consumption of the electronic device, which is an urgent problem to be solved.

In a first aspect, the present application provides an image frame prediction method, the method may include: when drawing the first drawing frame, the electronic device determines that the spatial information of the first drawing object changes according to the drawing instruction of the first drawing object, It is determined according to the drawing instruction of the second drawing object that the spatial information of the second drawing object has not changed, and the electronic device writes the color data of the first drawing object into the first color attachment, and writes the color data of the second drawing object into the second color Attachment; when drawing the second drawing frame, the electronic device determines that the spatial information of the first drawing object has changed according to the drawing instruction of the third drawing object, and determines that the spatial information of the fourth drawing object has not changed according to the drawing instruction of the fourth drawing object , the electronic device writes the color data of the third drawing object into the third color accessory, and writes the color data of the fourth drawing object into the fourth color accessory; the electronic device generates the first prediction frame according to the first color accessory and the third color accessory The fifth color accessory of the first predicted frame is generated according to the second color accessory and the fourth color accessory; the electronic device combines the fifth color accessory and the sixth color accessory into the first predicted frame.

With the method provided in the first aspect, the electronic device can determine whether the spatial information of the drawing object has changed according to the drawing instruction of the drawing object. That is to say, the electronic device can determine whether the drawing object is a moving object according to the drawing instruction of the drawing object. If the drawing instruction of the drawing object indicates that the spatial information changes, the drawing object is a moving object. If the drawing instruction of the drawing object does not indicate that the spatial information changes, the drawing object is a static object. If the drawing object is a moving object, the electronic device writes color data of the drawing object into the first color attachment. If the drawing object is a static object, the electronic device writes the color data of the drawing object into the second color attachment. In this way, the electronic device can separately store the color data of the moving object and the color data of the static object in the first drawing frame. Likewise, the electronic device may separately store the color data of the moving object and the color data of the static object in the second drawing frame. Then, the electronic device may predict the moving object in the first predicted frame according to the moving object in the first drawing frame and the moving object in the second drawing frame. The electronic device may predict the static object in the first predicted frame according to the static object in the first drawing frame and the static object in the second drawing frame. In this way, the electronic device can utilize the predicted frame predicted by the drawing frame, which can increase the frame rate, thereby improving the smoothness of the video interface displayed by the electronic device.

With reference to the first aspect, in a possible implementation manner, the electronic device generates a fifth color accessory of the first predicted frame according to the first color accessory and the third color accessory, including: the electronic device according to the first color accessory and the third color accessory, to determine the first motion vector of the third color accessory; the electronic device generates the fifth color accessory of the first predicted frame according to the second color accessory and the first motion vector.

In the above implementation manner, the electronic device only uses the color data of the moving object to calculate the motion vector of the moving object, so that the motion vector of the moving object can be calculated more accurately. Moreover, the electronic device only calculates the motion vector of the moving object according to the color data of the moving object. The electronic device will not miscalculate the motion vector of a moving object as the motion vector of a static object. Therefore, the moving object in the predicted frame predicted by the electronic device is more accurate.

With reference to the first aspect, in a possible implementation manner, the electronic device determines the first motion vector of the third color accessory according to the first color accessory and the third color accessory; specifically, the electronic device includes: The third color accessory is divided into Q pixel blocks, and the electronic device takes out the first pixel block from the Q pixel blocks of the third color accessory; the electronic device determines the second pixel block matching the first pixel block in the first color accessory ; the electronic device obtains the motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block; the electronic device determines the first motion vector of the third color accessory according to the motion vector of the first pixel block. According to the steps in this implementation manner, the electronic device can determine the motion vectors of all pixel blocks in the Q pixel blocks of the third color accessory. Each pixel block includes f*f (for example, 16*16) pixel points.

In the above implementation manner, the electronic device divides the color attachment of the moving object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the motion vector. This reduces the amount of computation, which in turn reduces the power consumption of electronic devices.

With reference to the first aspect, in a possible implementation manner, the electronic device determines, in the first color attachment, a second pixel block that matches the first pixel block, specifically including: the electronic device passes the first pixel block in the first pixel block The pixel points determine multiple candidate pixel blocks in the first color attachment, and the electronic device calculates the difference between the color values of the multiple candidate pixel blocks and the first pixel block; the electronic device calculates the first pixel block according to the multiple candidate pixel blocks. The difference between the color values of the pixel blocks determines a second pixel block matching the first pixel block; the second pixel block is a candidate pixel block with the smallest difference between the color values of the first pixel block and the plurality of candidate pixel blocks. In this way, the electronic device can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the first aspect, in a possible implementation manner, the electronic device generates the fifth color attachment of the first predicted frame according to the second color attachment and the first motion vector, which specifically includes: the electronic device determines the fifth color attachment according to the first motion vector. The motion vector of the color attachment, and the fifth color attachment is generated according to the motion vector of the second color attachment and the fifth color attachment. The motion vector of the fifth color attachment is K times the first motion vector, where K is greater than 0 and less than 1.

In conjunction with the first aspect, in one possible implementation, K is equal to 0.5. In this way, the objects in each image frame move at a constant speed, which is convenient for the electronic device to calculate, and can also make the user experience better when watching the video.

With reference to the first aspect, in a possible implementation manner, the electronic device generates a sixth color accessory of the first predicted frame according to the second color accessory and the fourth color accessory; specifically, the electronic device generates a sixth color accessory of the first predicted frame according to the second color accessory and the fourth color accessory; The color attachment determines the second motion vector of the fourth color attachment; the electronic device generates the sixth color attachment of the first predicted frame according to the fourth color attachment and the second motion vector.

In the above implementation manner, the electronic device only uses the color data of the static object to calculate the motion vector of the static object, so that the motion vector of the static object can be calculated more accurately. And, the electronic device only calculates the motion vector of the static object according to the color data of the static object. The electronic device will not miscalculate the motion vector of a static object as the motion vector of a moving object. Therefore, the static object in the predicted frame predicted by the electronic device is more accurate.

With reference to the first aspect, in a possible implementation manner, the electronic device determines the second motion vector of the fourth color accessory according to the second color accessory and the fourth color accessory; specifically, the electronic device divides the fourth color accessory into Q The electronic device takes out the third pixel block in the fourth color attachment; the electronic device calculates the first position of the third pixel block in the second color attachment; the electronic device calculates the first position of the third pixel block according to the The second position in the color attachment determines the motion vector of the third pixel block; the electronic device determines the second motion vector of the fourth color attachment according to the motion vector of the third pixel block. According to the steps in this implementation manner, the electronic device can determine the motion vector of each pixel block in the Q pixel blocks of the fourth color accessory.

In the above implementation manner, the electronic device divides the color attachment of the static object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the static vector. This reduces the amount of computation, which in turn reduces the power consumption of electronic devices.

With reference to the first aspect, in a possible implementation manner, the electronic device calculates the first position of the third pixel block in the second color attachment, which specifically includes: the electronic device obtains the drawing of the first drawing frame The first matrix in the instruction and the second matrix of the drawing instruction in the second drawing frame, the first matrix is used to record the corner information of the camera position of the first drawing frame, and the second matrix is used to record the camera position of the second drawing frame The electronic device calculates the first position of the third pixel block in the second color attachment according to the first matrix and the second matrix, and the depth value of the third pixel block.

With reference to the first aspect, in a possible implementation manner, the electronic device generates the sixth color attachment of the first predicted frame according to the fourth color attachment and the second motion vector, which specifically includes: the electronic device determines the sixth color attachment according to the second motion vector. The motion vector of the color attachment, and the sixth color attachment is generated according to the motion vector of the fourth color attachment and the sixth color attachment. The motion vector of the sixth color attachment is K times the second motion vector, where K is greater than 0 and less than 1.

With reference to the first aspect, in a possible implementation manner, the drawing instruction of the first drawing object includes an execution drawing instruction of the first drawing object and a drawing state device instruction of the first drawing object, wherein the execution drawing of the first drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the first drawing object, and generate a drawing result; the drawing state device instruction of the first drawing object is used to set the drawing state data on which the drawing instruction of the first drawing object depends. ; The drawing state data of the first drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the first drawing object.

With reference to the first aspect, in a possible implementation manner, the electronic device determining that the spatial information of the first drawing object has changed according to the drawing instruction of the first drawing object includes: the electronic device determining that there is a transition in the drawing instruction of the first drawing object The matrix parameter, the electronic device determines that the existing transition matrix parameter in the drawing instruction of the first drawing object is different from the transition matrix parameter of the corresponding first drawing object, and the transition matrix is used to describe the mapping of the local coordinate system of the drawing object to the world coordinate system relation.

With reference to the first aspect, in a possible implementation manner, the drawing instruction of the second drawing object includes an execution drawing instruction of the second drawing object and a drawing state device instruction of the second drawing object, wherein the execution drawing of the second drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the second drawing object, and generate a drawing result; the drawing state device instruction of the second drawing object is used to set the drawing state data on which the execution drawing instruction of the second drawing object depends ; The drawing state data of the second drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the second drawing object.

With reference to the first aspect, in a possible implementation manner, the electronic device determines that the spatial information of the second drawing object has not changed according to the drawing instruction of the second drawing object, including: the electronic device determining that the drawing instruction of the second drawing object does not contain any changes. There is a transition matrix parameter, and the electronic device determines that the existing transition matrix parameter in the drawing instruction of the second drawing object is the same as the transition matrix parameter of the corresponding second drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

With reference to the first aspect, in a possible implementation manner, the drawing instruction of the third drawing object includes an execution drawing instruction of the third drawing object and a drawing state device instruction of the third drawing object, wherein the execution drawing of the third drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the third drawing object, and generate a drawing result; the drawing state device instruction of the third drawing object is used to set the drawing state data on which the drawing instruction of the third drawing object depends. ; The drawing state data of the third drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the third drawing object.

With reference to the first aspect, in a possible implementation manner, the electronic device determining that the spatial information of the first drawing object has changed according to the drawing instruction of the third drawing object includes: the electronic device determining that the drawing instruction of the third drawing object is in the drawing instruction. Existing transition matrix parameters, the electronic device determines that the existing transition matrix parameters in the drawing instruction of the third drawing object are different from the transition matrix parameters of the corresponding third drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

With reference to the first aspect, in a possible implementation manner, the drawing instruction of the fourth drawing object includes an execution drawing instruction of the fourth drawing object and a drawing state device instruction of the fourth drawing object, wherein the execution drawing of the fourth drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the fourth drawing object, and generate a drawing result; the drawing state device instruction of the fourth drawing object is used to set the drawing state data on which the drawing instruction of the fourth drawing object depends. ; The drawing state data of the fourth drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the fourth drawing object.

With reference to the first aspect, in a possible implementation manner, the electronic device determines that the spatial information of the fourth drawing object has not changed according to the drawing instruction of the fourth drawing object, including: the electronic device determines that the drawing instruction of the fourth drawing object does not contain any changes. There is a transition matrix parameter, and the electronic device determines that the existing transition matrix parameter in the drawing instruction of the fourth drawing object is the same as the transition matrix parameter of the corresponding fourth drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

In combination with the first aspect, in a possible implementation manner, the electronic device determines that the spatial information of the first drawing object has changed according to the drawing instruction of the first drawing object, and determines the space of the second drawing object according to the drawing instruction of the second drawing object. The information has not changed, before the electronic device writes the color data of the first drawing object into the first color attachment, and before writing the color data of the second drawing object into the second color attachment, the method further includes: the electronic device creates the first color attachment. Memory space, second memory space, third memory space, fourth memory space, fifth memory space, sixth memory space, and seventh memory space; wherein, the first memory space is used to store the first color attachment, and the second memory space The space is used to store the second color attachment, the third memory space is used to store the third color attachment, the fourth memory space is used to store the fourth color attachment, the fifth memory space is used to store the fifth color attachment, and the sixth memory space is used to store the fifth color attachment. For storing the sixth color attachment, the seventh memory space is used for storing the first predicted frame.

In combination with the first aspect, in a possible implementation manner, when drawing the first drawing frame, the electronic device determines that the spatial information of the first drawing object changes according to the drawing instruction of the first drawing object, and according to the drawing of the second drawing object The instruction determines that the spatial information of the second drawing object has not changed, and the electronic device writes the color data of the first drawing object into the first color attachment, and after writing the color data of the second drawing object into the second color attachment, the method It also includes: the electronic device synthesizes the first color attachment and the second color attachment into the first drawing frame in the seventh memory space.

In combination with the first aspect, in a possible implementation manner, when drawing the second drawing frame, the electronic device determines that the spatial information of the first drawing object changes according to the drawing instruction of the third drawing object, and according to the drawing of the fourth drawing object The instruction determines that the spatial information of the fourth drawing object has not changed, the electronic device writes the color data of the third drawing object into the third color attachment, and after writing the color data of the fourth drawing object into the fourth color attachment, the method further includes : The electronic device synthesizes the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space.

With reference to the first aspect, in a possible implementation manner, the electronic device synthesizes the first color attachment and the second color attachment into the first drawing frame in the seventh memory space, which specifically includes: the electronic device according to the first depth attachment and The second depth attachment, the first color attachment and the second color attachment are combined into the first drawing frame in the seventh memory space, the first depth attachment is used for writing the depth data of the first drawing object, and the second depth attachment is used for Writes the second color attachment of the second drawable object.

With reference to the first aspect, in a possible implementation manner, the electronic device synthesizes the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space, which specifically includes: the electronic device according to the third depth attachment and The fourth depth attachment combines the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space; the third depth attachment is used for writing the depth data of the third drawing object, and the fourth depth attachment is used for Writes the depth data of the fourth draw object.

In a second aspect, an electronic device is provided, the electronic device may include: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used for storing computer program code, and the computer program code includes computer instructions , one or more processors invoke computer instructions to cause the electronic device to perform a method as in any possible manner of the first aspect.

In a third aspect, an electronic device is provided, comprising: one or more processors, a CPU, a graphics processor GPU, a memory, and a display screen; the memory is coupled to the one or more processors; the CPU is coupled to the GPU; wherein:

The memory can be used to store computer program codes, and the computer program codes include computer instructions; the CPU can be used to determine that the spatial information of the first drawing object changes according to the drawing instructions of the first drawing object when drawing the first drawing frame, and according to the second drawing frame. The drawing instruction of the drawing object determines that the spatial information of the second drawing object has not changed, and instructs the GPU to write the color data of the first drawing object into the first color attachment, and write the color data of the second drawing object into the second color attachment. The drawing instruction of a drawing object indicates that the spatial information of the first drawing object changes, and the drawing instruction of the second drawing object does not indicate that the spatial information of the second drawing object changes; when drawing the second drawing frame, according to the first drawing The drawing instruction of the third drawing object determines that the spatial information of the first drawing object has changed, and according to the drawing instruction of the fourth drawing object, it is determined that the spatial information of the fourth drawing object has not changed, and the GPU is instructed to write the color data of the third drawing object into the third drawing object. Three-color attachment, write the color data of the fourth drawing object into the fourth color attachment, the drawing instruction of the third drawing object indicates that the spatial information of the third drawing object changes, and the drawing instruction of the fourth drawing object does not indicate the fourth The spatial information of the drawn object changes.

The GPU can be used to write the color data of the first rendering object into the first color attachment, and write the color data of the second rendering object into the second color attachment; write the color data of the third rendering object into the third color attachment, and The color data of the fourth drawing object is written into the fourth color attachment; the fifth color attachment of the first predicted frame is generated according to the first color attachment and the third color attachment, and the first predicted frame is generated according to the second color attachment and the fourth color attachment The sixth color attachment of ; the fifth color attachment and the sixth color attachment are synthesized into the first prediction frame.

The display screen may be used to display the first drawing frame, the second drawing frame, and the first predicted frame.

With the electronic device provided by the third aspect, the electronic device can determine whether the spatial information of the drawing object has changed according to the drawing instruction of the drawing object. That is to say, the electronic device can determine whether the drawing object is a moving object according to the drawing instruction of the drawing object. If the drawing instruction of the drawing object indicates that the spatial information changes, the drawing object is a moving object. If the drawing instruction of the drawing object does not indicate that the spatial information changes, the drawing object is a static object. If the drawing object is a moving object, the electronic device writes color data of the drawing object into the first color attachment. If the drawing object is a static object, the electronic device writes the color data of the drawing object into the second color attachment. In this way, the electronic device can separately store the color data of the moving object and the color data of the static object in the first drawing frame. Likewise, the electronic device may separately store the color data of the moving object and the color data of the static object in the second drawing frame. Then, the electronic device may predict the moving object in the first predicted frame according to the moving object in the first drawing frame and the moving object in the second drawing frame. The electronic device may predict the static object in the first predicted frame according to the static object in the first drawing frame and the static object in the second drawing frame. In this way, the electronic device can utilize the predicted frame predicted by the drawing frame, which can increase the frame rate, thereby improving the smoothness of the video interface displayed by the electronic device.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: determine the first motion vector of the third color accessory according to the first color accessory and the third color accessory; The motion vector generates the fifth color attachment of the first predicted frame.

In the above implementation manner, the GPU in the electronic device only uses the color data of the moving object to calculate the motion vector of the moving object, so that the motion vector of the moving object can be calculated more accurately. Moreover, the GPU only calculates the motion vector of the moving object according to the color data of the moving object. The GPU will not miscalculate motion vectors for moving objects as motion vectors for static objects. Therefore, the moving object in the predicted frame predicted by the electronic device is more accurate.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: divide the third color accessory into Q pixel blocks, and take out the first pixel block from the Q pixel blocks of the third color accessory; A second pixel block matching the first pixel block is determined in a color attachment; the motion vector of the first pixel block is obtained according to the displacement of the second pixel block to the first pixel block; the first pixel block is determined according to the motion vector of the first pixel block. The first motion vector of the three-color attachment.

In the above implementation manner, the GPU divides the color attachment of the moving object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the motion vector. This reduces the amount of computation and thus reduces the power consumption of the GPU.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: determine multiple candidate pixel blocks in the first color attachment by using the first pixel point in the first pixel block, and calculate multiple candidate pixel blocks respectively. The difference between the color values of the candidate pixel block and the first pixel block; the second pixel block matching the first pixel block is determined according to the difference of the color values of the first pixel block of the multiple candidate pixel blocks; the second pixel block is A candidate pixel block with the smallest difference between the color values of the first pixel block and the multiple candidate pixel blocks. In this way, the GPU of the electronic device can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: determine the second motion vector of the fourth color accessory according to the second color accessory and the fourth color accessory; The motion vector generates the sixth color attachment of the first predicted frame.

In the above implementation manner, the GPU of the electronic device only uses the color data of the static object to calculate the motion vector of the static object, so that the motion vector of the static object can be calculated more accurately. And, the GPU only calculates the motion vector of the static object based on the color data of the static object. The GPU will not miscalculate motion vectors for static objects as motion vectors for moving objects. Therefore, the static objects in the predicted frame predicted by the GPU are more accurate.

In combination with the third aspect, in a possible implementation manner, the GPU may also be used to: divide the fourth color attachment into Q pixel blocks, and take out the third pixel block from the fourth color attachment; calculate that the third pixel block is in The first position in the second color attachment; the motion vector of the third pixel block is determined according to the first position and the second position in the fourth color attachment of the third pixel block; the fourth pixel block is determined according to the motion vector of the third pixel block The second motion vector for the color attachment.

In the above implementation manner, the GPU divides the color attachment of the static object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the static vector. This reduces the amount of computation and thus reduces the power consumption of the GPU.

With reference to the third aspect, in a possible implementation manner, the electronic device may also be used to: obtain the first matrix of the drawing instructions in the first drawing frame and the second matrix of the drawing instructions in the second drawing frame, and the third A matrix is used to record the corner information of the camera position of the first drawing frame, and the second matrix is used to record the corner information of the camera position of the second drawing frame; according to the first matrix and the second matrix, and the depth value of the third pixel block The first position of the third pixel block in the second color attachment is calculated.

In combination with the third aspect, in a possible implementation manner, the GPU may also be used to: create a first memory space, a second memory space, a third memory space, a fourth memory space, a fifth memory space, and a sixth memory space , the seventh memory space; wherein, the first memory space is used to store the first color accessory, the second memory space is used to store the second color accessory, the third memory space is used to store the third color accessory, and the fourth memory space is used for The fourth color attachment is stored, the fifth memory space is used to store the fifth color attachment, the sixth memory space is used to store the sixth color attachment, and the seventh memory space is used to store the first predicted frame.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: combine the first color attachment and the second color attachment into the first drawing frame in the seventh memory space.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: combine the first color attachment and the second color attachment in a seventh memory space into a first color attachment according to the first depth attachment and the second depth attachment Drawing frame; the third depth attachment is used to write the depth data of the first drawing object, and the fourth depth attachment is used to write the depth data of the second drawing object.

With reference to the third aspect, in a possible implementation manner, the GPU may also be used to: combine the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space.

In combination with the third aspect, in a possible implementation manner, the GPU may also be used to: combine the third color attachment and the fourth color attachment in the seventh memory space into the second depth attachment according to the third depth attachment and the fourth depth attachment Drawing frame; the third depth attachment is used to write the depth data of the third draw object, and the fourth depth attachment is used to write the depth data of the fourth draw object.

With reference to the third aspect, in a possible implementation manner, the drawing instruction of the first drawing object includes an execution drawing instruction of the first drawing object and a drawing state device instruction of the first drawing object, wherein the execution drawing of the first drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the first drawing object, and generate a drawing result; the drawing state device instruction of the first drawing object is used to set the drawing state data on which the drawing instruction of the first drawing object depends. ; The drawing state data of the first drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the first drawing object.

With reference to the third aspect, in a possible implementation manner, the CPU determines that the spatial information of the first drawing object has changed according to the drawing instruction of the first drawing object, and the CPU can also be specifically used to: determine the drawing instruction of the first drawing object. Existing transition matrix parameters, determining that the existing transition matrix parameters in the drawing instruction of the first drawing object are different from the transition matrix parameters of the corresponding first drawing object, and the transition matrix is used to describe the mapping of the local coordinate system of the drawing object to the world coordinate system relation. With reference to the third aspect, in a possible implementation manner, the drawing instruction of the second drawing object includes an execution drawing instruction of the second drawing object and a drawing state device instruction of the second drawing object, wherein the execution drawing of the second drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the second drawing object, and generate a drawing result; the drawing state device instruction of the second drawing object is used to set the drawing state data on which the execution drawing instruction of the second drawing object depends ; The drawing state data of the second drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the second drawing object.

With reference to the third aspect, in a possible implementation manner, the CPU determines that the spatial information of the second drawing object has not changed according to the drawing instruction of the second drawing object, and the CPU can also be specifically used to: determine the drawing instruction of the second drawing object There is no transition matrix parameter in , and the transition matrix parameter that exists in the drawing instruction that determines the second drawing object is the same as the transition matrix parameter of the corresponding second drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

With reference to the third aspect, in a possible implementation manner, the drawing instruction of the third drawing object includes an execution drawing instruction of the third drawing object and a drawing state device instruction of the third drawing object, wherein the execution drawing of the third drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the third drawing object, and generate a drawing result; the drawing state device instruction of the third drawing object is used to set the drawing state data on which the drawing instruction of the third drawing object depends. ; The drawing state data of the third drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the third drawing object.

In combination with the third aspect, in a possible implementation manner, the CPU determines that the spatial information of the third drawing object has not changed according to the drawing instruction of the third drawing object, and the CPU may also be specifically used to: determine the drawing instruction of the third drawing object There is no transition matrix parameter in , and the transition matrix parameter that exists in the drawing instruction that determines the third drawing object is the same as the transition matrix parameter of the corresponding third drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

With reference to the third aspect, in a possible implementation manner, the drawing instruction of the fourth drawing object includes an execution drawing instruction of the fourth drawing object and a drawing state device instruction of the fourth drawing object, wherein the execution drawing of the fourth drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the fourth drawing object, and generate a drawing result; the drawing state device instruction of the fourth drawing object is used to set the drawing state data on which the drawing instruction of the fourth drawing object depends. ; The drawing state data of the fourth drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the fourth drawing object.

With reference to the third aspect, in a possible implementation manner, the CPU determines that the spatial information of the fourth drawing object has not changed according to the drawing instruction of the fourth drawing object, and the CPU may also be specifically used to: determine the drawing instruction of the fourth drawing object There is no transition matrix parameter in , and the transition matrix parameter that exists in the drawing instruction that determines the fourth drawing object is the same as the transition matrix parameter of the corresponding fourth drawing object, and the transition matrix is used to describe the local coordinate system of the drawing object to the world coordinate system. mapping relationship.

In a fourth aspect, an image frame prediction apparatus is provided, the apparatus may include a drawing unit, a prediction unit, and a synthesis unit; wherein:

The drawing unit may be configured to, when drawing the first drawing frame, determine that the spatial information of the first drawing object has changed according to the drawing instruction of the first drawing object, and determine that the spatial information of the second drawing object has not occurred according to the drawing instruction of the second drawing object. Change, write the color data of the first drawing object into the first color attachment, and write the color data of the second drawing object into the second color attachment; when drawing the second drawing frame, determine the first color according to the drawing instruction of the third drawing object. When the spatial information of the drawing object changes, determine that the spatial information of the fourth drawing object has not changed according to the drawing instruction of the fourth drawing object, write the color data of the third drawing object into the third color attachment, and write the color data of the fourth drawing object into the third color attachment. Color data is written to the fourth color attachment.

The prediction unit may be configured to generate a fifth color attachment of the first predicted frame according to the first color attachment and the third color attachment, and generate a sixth color attachment of the first predicted frame according to the second color attachment and the fourth color attachment.

The synthesis unit may be used to synthesize the fifth color attachment and the sixth color attachment into the first predicted frame.

With the image frame prediction apparatus provided in the fourth aspect, the image frame prediction apparatus can determine whether the spatial information of the drawing object has changed according to the drawing instruction of the drawing object. That is to say, the image frame predicting apparatus can determine whether the drawing object is a moving object according to the drawing instruction of the drawing object. If the drawing instruction of the drawing object indicates that the spatial information changes, the drawing object is a moving object. If the drawing instruction of the drawing object does not indicate that the spatial information changes, the drawing object is a static object. If the rendering object is a moving object, the image frame prediction apparatus writes the color data of the rendering object into the first color attachment. If the rendering object is a static object, the image frame predicting apparatus writes the color data of the rendering object into the second color attachment. In this way, the image frame prediction apparatus can separately store the color data of the moving object and the color data of the static object in the first drawing frame. Likewise, the image frame prediction apparatus may separately store the color data of the moving object and the color data of the static object in the second drawing frame. Then, the image frame prediction apparatus may predict the moving object in the first predicted frame according to the moving object in the first drawing frame and the moving object in the second drawing frame. The image frame predicting apparatus can predict the static object in the first predicted frame according to the static object in the first drawing frame and the static object in the second drawing frame. In this way, the image frame predicting apparatus can utilize the predicted frame predicted by the drawing frame, which can increase the frame rate, thereby improving the smoothness of the video interface displayed by the image frame predicting apparatus.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: determine the first motion vector of the third color accessory according to the first color accessory and the third color accessory; A motion vector generates the fifth color attachment of the first predicted frame.

In the above implementation manner, the image frame prediction apparatus only uses the color data of the moving object to calculate the motion vector of the moving object, so that the motion vector of the moving object can be more accurately calculated. And, the image frame predicting apparatus calculates the motion vector of the moving object only based on the color data of the moving object. The image frame prediction apparatus does not miscalculate the motion vector of the moving object as the motion vector of the static object. Therefore, the moving object in the prediction frame predicted by the image frame prediction device is more accurate.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: divide the third color accessory into Q pixel blocks, and extract the first pixel block from the Q pixel blocks of the third color accessory; The second pixel block matching the first pixel block is determined in the first color attachment; the motion vector of the first pixel block is obtained according to the displacement of the second pixel block to the first pixel block; the motion vector of the first pixel block is determined according to the motion vector of the first pixel block. The first motion vector of the third color attachment. According to the steps in this implementation, the prediction unit may determine motion vectors of all pixel blocks in the Q pixel blocks of the third color attachment. Each pixel block includes f*f (for example, 16*16) pixel points.

In the above implementation manner, the prediction unit divides the color attachment of the moving object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the motion vector. In this way, the amount of computation can be reduced, thereby reducing the power consumption of the image frame prediction apparatus.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: determine multiple candidate pixel blocks in the first color attachment by using the first pixel point in the first pixel block, and calculate the multiple candidate pixel blocks respectively. The difference between the color values of the candidate pixel blocks and the first pixel block; the second pixel block matching the first pixel block is determined according to the difference of the color values of the first pixel block of the multiple candidate pixel blocks; the second pixel block is a candidate pixel block with the smallest difference between the color values of the first pixel block and the multiple candidate pixel blocks. In this way, the prediction unit can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: determine the motion vector of the fifth color accessory according to the first motion vector, and generate the first color accessory according to the motion vector of the second color accessory and the fifth color accessory. Five color accessories. The motion vector of the fifth color attachment is K times the first motion vector, where K is greater than 0 and less than 1.

In conjunction with the fourth aspect, in a possible implementation, K is equal to 0.5. In this way, the objects in each image frame move at a constant speed.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may be further configured to: determine the second motion vector of the fourth color accessory according to the second color accessory and the fourth color accessory; Two motion vectors generate the sixth color attachment of the first predicted frame.

In the above implementation manner, the prediction unit only uses the color data of the static object to calculate the motion vector of the static object, so that the motion vector of the static object can be calculated more accurately. And, the prediction unit calculates the motion vector of the static object only from the color data of the static object. The prediction unit does not miscalculate motion vectors of static objects as motion vectors of moving objects. Therefore, the static objects in the predicted frame predicted by the prediction unit are more accurate.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: divide the fourth color accessory into Q pixel blocks, and take out the third pixel block from the fourth color accessory; calculate the third pixel block at the first position in the second color attachment; determine the motion vector of the third pixel block according to the first position and the second position in the fourth color attachment of the third pixel block; determine the first pixel block according to the motion vector of the third pixel block Second motion vector for four color attachments. According to the steps in this implementation, the prediction unit may determine the motion vector of each pixel block in the Q pixel blocks of the fourth color attachment.

In the above implementation manner, the prediction unit divides the color attachment of the static object into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel of the static vector. In this way, the amount of computation can be reduced, thereby reducing the power consumption of the image frame prediction apparatus.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: obtain the first matrix in the drawing instructions of the first drawing frame and the second matrix of the drawing instructions in the second drawing frame, A matrix is used to record the corner information of the camera position of the first drawing frame, and the second matrix is used to record the corner information of the camera position of the second drawing frame; according to the first matrix and the second matrix, and the depth value of the third pixel block The first position of the third pixel block in the second color attachment is calculated.

With reference to the fourth aspect, in a possible implementation manner, the prediction unit may also be used to: determine the motion vector of the sixth color accessory according to the second motion vector, and generate the sixth color accessory according to the motion vector of the fourth color accessory and the sixth color accessory. Six color accessories. The motion vector of the sixth color attachment is K times the second motion vector, where K is greater than 0 and less than 1.

With reference to the fourth aspect, in a possible implementation manner, the drawing instruction of the first drawing object includes an execution drawing instruction of the first drawing object and a drawing state device instruction of the first drawing object, wherein the execution drawing of the first drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the first drawing object, and generate a drawing result; the drawing state device instruction of the first drawing object is used to set the drawing state data on which the drawing instruction of the first drawing object depends. ; The drawing state data of the first drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the first drawing object.

With reference to the fourth aspect, in a possible implementation manner, the drawing instruction of the second drawing object includes an execution drawing instruction of the second drawing object and a drawing state device instruction of the second drawing object, wherein the execution drawing of the second drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the second drawing object, and generate a drawing result; the drawing state device instruction of the second drawing object is used to set the drawing state data on which the execution drawing instruction of the second drawing object depends ; The drawing state data of the second drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the second drawing object.

With reference to the fourth aspect, in a possible implementation manner, the drawing instruction of the third drawing object includes an execution drawing instruction of the third drawing object and a drawing state device instruction of the third drawing object, wherein the execution drawing of the third drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the third drawing object, and generate a drawing result; the drawing state device instruction of the third drawing object is used to set the drawing state data on which the execution drawing instruction of the third drawing object depends ; The drawing state data of the third drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the third drawing object.

With reference to the fourth aspect, in a possible implementation manner, the drawing instruction of the fourth drawing object includes an execution drawing instruction of the fourth drawing object and a drawing state device instruction of the fourth drawing object, wherein the execution drawing of the fourth drawing object The instruction is used to trigger the electronic device to draw and render the drawing state data of the fourth drawing object, and generate a drawing result; the drawing state device instruction of the fourth drawing object is used to set the drawing state data on which the drawing instruction of the fourth drawing object depends. ; The drawing state data of the fourth drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the fourth drawing object.

With reference to the fourth aspect, in a possible implementation manner, the drawing unit may also be used to: determine the existence of transition matrix parameters in the drawing instruction of the first drawing object, and determine the existing transition matrix parameters in the drawing instruction of the first drawing object Different from the transition matrix parameter of the corresponding first drawing object, the transition matrix is used to describe the mapping relationship from the local coordinate system of the drawing object to the world coordinate system.

With reference to the fourth aspect, in a possible implementation manner, the drawing unit may also be used to: determine that the transfer matrix parameter does not exist in the drawing instruction of the second drawing object, and determine the existing transition matrix in the drawing instruction of the second drawing object The parameters are the same as the parameters of the transition matrix of the corresponding second drawing object, and the transition matrix is used to describe the mapping relationship between the local coordinate system of the drawing object and the world coordinate system. With reference to the fourth aspect, in a possible implementation manner, the drawing unit may also be used to: determine the existence of transition matrix parameters in the drawing instruction of the third drawing object, and determine the existence of transitions in the drawing instruction of the third drawing object The matrix parameter is different from the transition matrix parameter of the corresponding third drawing object, and the transition matrix is used to describe the mapping relationship from the local coordinate system of the drawing object to the world coordinate system.

With reference to the fourth aspect, in a possible implementation manner, the drawing unit may also be used to: determine that the transfer matrix parameter does not exist in the drawing instruction of the fourth drawing object, and determine the existing transition matrix in the drawing instruction of the fourth drawing object The parameters are the same as the parameters of the transition matrix of the corresponding fourth drawing object, and the transition matrix is used to describe the mapping relationship between the local coordinate system of the drawing object and the world coordinate system.

With reference to the fourth aspect, in a possible implementation manner, the drawing unit may also be used to: create a first memory space, a second memory space, a third memory space, a fourth memory space, a fifth memory space, and a sixth memory space space, the seventh memory space; wherein, the first memory space is used to store the first color accessory, the second memory space is used to store the second color accessory, the third memory space is used to store the third color accessory, and the fourth memory space is used to store the third color accessory. For storing the fourth color attachment, the fifth memory space is used for storing the fifth color attachment, the sixth memory space is used for storing the sixth color attachment, and the seventh memory space is used for storing the first prediction frame.

With reference to the fourth aspect, in a possible implementation manner, the synthesizing unit may be further configured to: combine the first color attachment and the second color attachment into the first drawing frame in the seventh memory space.

With reference to the fourth aspect, in a possible implementation manner, the synthesizing unit may further be configured to: combine the first color attachment and the second color attachment in the seventh memory space according to the first depth attachment and the second depth attachment A drawing frame; the third depth attachment is used for writing the depth data of the first drawing object, and the fourth depth attachment is used for writing the depth data of the second drawing object.

With reference to the fourth aspect, in a possible implementation manner, the synthesizing unit may be further configured to: combine the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space.

With reference to the fourth aspect, in a possible implementation manner, the synthesizing unit may be further configured to: combine the third color attachment and the fourth color attachment in the seventh memory space according to the third depth attachment and the fourth depth attachment Two drawing frames; the third depth attachment is used for writing the depth data of the third drawing object, and the fourth depth attachment is used for writing the depth data of the fourth drawing object.

In a fifth aspect, the present application provides a method for image frame prediction, the method may include: the electronic device determines the tenth moving object in the tenth drawing frame according to the tenth drawing instruction, and draws the eleventh moving object according to the second drawing instruction The eleventh moving object is determined in the frame; the electronic device determines that the tenth moving object and the eleventh moving object match according to the attributes of the tenth moving object and the eleventh moving object; the electronic device determines that the tenth moving object and the tenth moving object match For a moving object, the drawing result of the twelfth moving object in the tenth prediction frame is determined.

With the method provided by the embodiment of the present application, the electronic device can draw the frame according to the two frames of the application to obtain the predicted frame. This increases the frame rate of the application's video interface. Therefore, the smoothness of displaying the video interface of the application program on the electronic device can be improved.

With reference to the fifth aspect, in a possible implementation manner, the tenth drawing frame is displayed on the display screen of the electronic device before the eleventh drawing frame, and the tenth predicted frame is displayed on the display screen of the electronic device after the eleventh drawing frame. on the display. That is, the electronic device can draw frames according to the first two frames to predict the following predicted frames, thereby increasing the frame rate.

With reference to the fifth aspect, in a possible implementation manner, an image frame is separated between the tenth drawing frame and the eleventh drawing frame, and the tenth predicted frame is an image frame adjacent to the eleventh drawing frame. . For example, the electronic device predicts the fifth prediction frame using the second rendering frame and the fourth rendering frame. In this way, the electronic device can have enough time to calculate the motion vector and draw the first predicted frame, so as to avoid the video interface being stuck after the electronic device displays the fourth drawn frame and the fifth drawn frame is not finished.

With reference to the fifth aspect, in a possible implementation manner, the electronic device determines that the tenth moving object matches the eleventh moving object according to the first attribute of the tenth moving object and the second attribute of the eleventh moving object, specifically Including: the electronic device establishes a first index table, the first index table is used to save the moving objects in the tenth drawing frame and the properties of the moving objects, the first index table includes the tenth moving object and the properties of the tenth moving object; the electronic device A second index table is established, and the second index table is used to save the motion objects and the attributes of the motion objects in the eleventh drawing frame, and the second index table includes the attributes of the eleventh motion objects and the eleventh motion objects; The eleventh moving object is extracted from the second index table, and the tenth moving object matching the eleventh moving object is determined from the first index table. In this way, the electronic device can quickly find out the tenth moving object matching the eleventh moving object through the index table.

With reference to the fifth aspect, in a possible implementation manner, matching the tenth moving object with the eleventh moving object includes: the first attribute of the tenth moving object is the same as the second attribute of the eleventh moving object.

With reference to the fifth aspect, in a possible implementation manner, the electronic device determines the drawing result of the twelfth moving object in the tenth prediction frame according to the tenth moving object and the eleventh moving object, which specifically includes: the electronic device determining the first coordinate of the first point of the tenth moving object, and determining the second coordinate of the second point of the eleventh moving object; the electronic device determines the tenth moving object to the tenth moving object according to the displacement from the first coordinate to the second coordinate A tenth motion vector of a moving object; the electronic device determines a drawing result of the twelfth moving object in the tenth prediction frame according to the tenth motion vector and the eleventh moving object. In this way, the electronic device can use the motion vector of one point of the object to represent the motion vector of the entire object, and does not need to separately determine the motion vector of each block of the object. In this way, the calculation amount of the electronic device can be reduced, so that the power consumption of the electronic device can be saved.

With reference to the fifth aspect, in a possible implementation manner, the electronic device determines the first coordinates of the first point of the tenth moving object, which specifically includes: the electronic device determines the first coordinate according to the coordinates of all pixel points of the tenth moving object. The first coordinates of the point; the electronic device determining the second coordinates of the second point of the eleventh moving object specifically includes: the electronic device determining the second coordinates of the second point according to the coordinates of all pixel points of the eleventh moving object.

Further, the electronic device may obtain the coordinates of each pixel of the tenth moving object from the template image of the tenth moving object through the shader in the GPU. The template image of the tenth moving object is drawn in the GPU. The electronic device may obtain the coordinates of each pixel of the eleventh moving object from the template image of the eleventh moving object through the shader in the GPU. The template image of the eleventh moving object is drawn in the GPU.

With reference to the fifth aspect, in a possible implementation manner, the first point is the geometric center point of the tenth moving object, and the second point is the geometric center point of the eleventh moving object. Compared with other points of the moving object, the center point of the moving object is more convenient to calculate, so that the calculation amount of the electronic device can be reduced, and the power consumption of the electronic device can be reduced.

With reference to the fifth aspect, in a possible implementation manner, the electronic device determines the drawing result of the twelfth motion object in the tenth prediction frame according to the tenth motion vector and the eleventh motion object, which specifically includes: the electronic device According to the tenth motion vector and the first pixel point of the eleventh moving object, the second pixel point of the twelfth moving object in the tenth prediction frame is determined.

With reference to the fifth aspect, in a possible implementation manner, the electronic device determines the second pixel point of the twelfth moving object in the tenth prediction frame according to the tenth motion vector and the first pixel point of the eleventh moving object , specifically including: the electronic device determines the eleventh motion vector of the eleventh moving object to the twelfth moving object according to the tenth motion vector; the electronic device determines the eleventh motion vector according to the eleventh motion vector and the first pixel point of the eleventh moving object , determine the second pixel of the twelfth moving object in the tenth prediction frame, and the second pixel is the pixel that the first pixel moves from the eleventh drawing frame to the tenth prediction frame according to the eleventh motion vector point.

With reference to the fifth aspect, in a possible implementation manner, the eleventh motion vector is K times the tenth motion vector, and K is greater than 0 and less than 1.

In conjunction with the fifth aspect, in a possible implementation, K is equal to 0.5. In this way, the objects in each image frame move at a constant speed, which is convenient for the electronic device to calculate, and can also make the user experience better when watching the video.

In a sixth aspect, an electronic device is provided, comprising: one or more processors, a CPU, a graphics processor GPU, a memory, and a display screen; the memory is coupled to the one or more processors; the CPU is coupled to the GPU; wherein:

The memory is used to store computer program code, the computer program code includes computer instructions;

The CPU is configured to determine the tenth moving object in the tenth drawing frame according to the tenth drawing instruction, and determine the eleventh moving object in the eleventh drawing frame according to the second drawing instruction;

The GPU is configured to determine that the tenth moving object and the eleventh moving object match according to the attributes of the tenth moving object and the eleventh moving object; determine the tenth prediction frame according to the tenth moving object and the eleventh moving object The drawing result of the twelfth moving object in ;

The display screen is used for displaying the tenth drawing frame, the eleventh drawing frame and the tenth prediction frame.

The electronic device provided by the embodiment of the present application can draw a frame according to two frames of an application to obtain a predicted frame. This increases the frame rate of the application's video interface. Therefore, the smoothness of displaying the video interface of the application program on the electronic device can be improved.

With reference to the sixth aspect, in a possible implementation manner, the tenth drawing frame is displayed on the display screen before the eleventh drawing frame, and the tenth predicted frame is displayed on the display screen after the eleventh drawing frame. That is, the electronic device can draw frames according to the first two frames to predict the following predicted frames, thereby increasing the frame rate.

With reference to the sixth aspect, in a possible implementation manner, an image frame is separated between the tenth drawing frame and the eleventh drawing frame, and the tenth predicted frame is an image frame adjacent to the eleventh drawing frame. . For example, the electronic device predicts the fifth prediction frame using the second rendering frame and the fourth rendering frame. In this way, the electronic device can have enough time to calculate the motion vector and draw the first predicted frame, so as to avoid the video interface being stuck after the electronic device displays the fourth drawn frame and the fifth drawn frame is not finished.

With reference to the sixth aspect, in a possible implementation manner, the CPU is specifically used to: establish a first index table, the first index table is used to save the motion object and the attributes of the motion object in the tenth drawing frame, and the first index table Including the attributes of the tenth motion object and the tenth motion object; establishing a second index table, the second index table is used to save the motion object in the eleventh drawing frame and the attributes of the motion object, and the second index table includes the eleventh motion Properties of the object and the eleventh motion object. The GPU is specifically configured to: fetch the eleventh moving object from the second index table, and determine the tenth moving object matching the eleventh moving object from the first index table. In this way, the electronic device can quickly find the tenth moving object matching the eleventh moving object from the index table.

With reference to the sixth aspect, in a possible implementation manner, matching the tenth moving object with the eleventh moving object includes: the attributes of the tenth moving object are the same as the attributes of the eleventh moving object.

With reference to the sixth aspect, in a possible implementation manner, the GPU is specifically configured to: determine the first coordinate of the first point of the tenth moving object, and determine the second coordinate of the second point of the eleventh moving object; The displacement from the first coordinate to the second coordinate determines the tenth motion vector from the tenth moving object to the eleventh moving object; according to the tenth motion vector and the eleventh moving object, the twelfth motion in the tenth prediction frame is determined The result of drawing the object. In this way, the electronic device can use the motion vector of one point of the object to represent the motion vector of the entire object, and does not need to separately determine the motion vector of each block of the object. In this way, the calculation amount of the electronic device can be reduced, so that the power consumption of the electronic device can be saved.

With reference to the sixth aspect, in a possible implementation manner, the GPU is specifically used to: determine the first coordinates of the first point according to the coordinates of all the pixels of the tenth moving object; The coordinates of determine the second coordinates of the second point.

With reference to the sixth aspect, in a possible implementation manner, the first point is the geometric center point of the tenth moving object, and the second point is the geometric center point of the eleventh moving object.

With reference to the sixth aspect, in a possible implementation manner, the GPU is specifically configured to: determine the first pixel of the twelfth moving object in the tenth prediction frame according to the tenth motion vector and the first pixel of the eleventh moving object. two pixels.

With reference to the sixth aspect, in a possible implementation manner, the GPU is specifically configured to: determine the eleventh motion vector from the eleventh moving object to the twelfth moving object according to the tenth motion vector; according to the eleventh motion vector and the first pixel of the eleventh moving object, determine the second pixel of the twelfth moving object in the tenth prediction frame, and the second pixel is the first pixel drawn according to the eleventh motion vector from the eleventh The pixel in the frame moves to the tenth predicted frame.

With reference to the sixth aspect, in a possible implementation manner, the eleventh motion vector is K times the tenth motion vector, and K is greater than 0 and less than 1.

In combination with the sixth aspect, in a possible implementation, K is equal to 0.5.

In a seventh aspect, an embodiment of the present application provides a frame prediction method, wherein the method includes:

The electronic device determines the predicted motion vector of the first vertex from the target reference frame to the predicted frame according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame, wherein the first vertex is a vertex in the first block , the first block is a block in the target reference frame, and the target reference frame is a frame determined from the first reference frame or the second reference frame according to the position of the predicted frame relative to the first reference frame or the second reference frame, The first reference frame and the second reference frame are two adjacent frames in the video stream; the coordinates of the first vertex in the predicted frame are determined according to the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex ; According to the coordinates of the vertex of the first block in the predicted frame and the coordinates in the target reference frame, determine the pixel block of the first block in the predicted frame; display the predicted frame, which includes the pixel block.

With the frame prediction method provided in the seventh aspect of the embodiment of the present application, the electronic device first, for each vertex of the block, determines the position of the vertex in the predicted frame according to the predicted motion vector of the blocks around the vertex from the target reference frame to the predicted frame. Then, for each block, the position of each pixel of the block in the predicted frame is calculated according to the correspondence between the four vertices of the block in the reference frame and the predicted frame, and finally the entire predicted frame is obtained.

In the above-mentioned frame prediction method, the vertex is determined, and then the first block is stretched according to the predicted motion vectors of the blocks around the first block, wherein the first block is a block in the reference frame, so that the adjacent prediction frames are made. The vertices of the blocks are coincident, and the blocks are continuous without holes.

With reference to the seventh aspect, in a possible implementation manner, the electronic device obtains the coordinates of the pixels of the first block in the predicted frame and The corresponding relationship of the coordinates in the target reference frame, and further, according to the coordinates and the corresponding relationship of the pixels of the first block in the target reference frame, the coordinates of the pixels in the first block in the predicted frame are determined.

In combination with the seventh aspect, in a possible implementation manner, the electronic device inputs the coordinates of the four vertices of the first block in the predicted frame and the coordinates in the target reference frame into the homography transformation formula, respectively, to obtain the homography. System of equations, the system of homography equations consists of four equations. The electronic device obtains the homography transformation matrix corresponding to the first block by solving the homography equation system, and then obtains the homography transformation formula corresponding to the first block according to the homography transformation matrix corresponding to the first block. The homography transformation formula corresponding to the block is used to express the correspondence between the coordinates of the pixels of the first block in the predicted frame and the coordinates in the target reference frame.

With reference to the seventh aspect, in a possible implementation manner, the electronic device inputs the coordinates of the first pixel in the target reference frame into the homography transformation formula corresponding to the first block to obtain the coordinates of the first pixel in the predicted frame, The first pixel is one pixel of the first block.

With reference to the seventh aspect, in a possible implementation manner, the number of coordinates included in the region of the pixel block is greater than the number of coordinates of the first block in the predicted frame, wherein the region of the pixel block is determined by the vertices of the first block . In this case, the electronic device removes the coordinates of the pixels of the first block in the predicted frame from the coordinates in the area of the pixel block to obtain the first coordinates, and then inputs the first coordinates into the homography transformation formula corresponding to the first block , get the pixel of the first coordinate in the target reference frame.

In conjunction with the seventh aspect, in a possible implementation, the electronic device executes the predicted motion vector from the target reference frame to the predicted frame according to the blocks around the first vertex, and determines the prediction of the first vertex from the target reference frame to the predicted frame. Before the motion vector, the electronic device obtains the first reference frame and the second reference frame, and then determines the target reference frame from the first reference frame and the second reference frame according to the position of the frame to be predicted. The target reference frame is divided into blocks according to the square blocks of the first size, and then the motion vector of the block from the target reference frame to the predicted frame is calculated.

With reference to the seventh aspect, in a possible implementation manner, when the target reference frame is the first reference frame, the electronic device obtains the motion vector of the first block from the first reference frame to the second reference frame, and then converts the first block to the second reference frame. Half of the motion vector from the first reference frame to the second reference frame is determined as the predicted motion vector of the first block from the target reference frame to the predicted frame.

With reference to the seventh aspect, in a possible implementation manner, when the target reference frame is the first reference frame, the electronic device obtains the motion vector of the first block from the second reference frame to the first reference frame, and then converts the first block to the first reference frame. The negative half of the motion vector from the second reference frame to the first reference frame is determined as the predicted motion vector of the first block from the target reference frame to the predicted frame.

In conjunction with the seventh aspect, in a possible implementation manner, the electronic device determines the average value of the predicted motion vectors of the blocks around the first vertex from the target reference frame to the predicted frame as the average value of the predicted motion vectors of the first vertex from the target reference frame to the predicted frame. Predict the motion vector.

With reference to the seventh aspect, in a possible implementation manner, the electronic device adds the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex to obtain the coordinates of the first vertex in the predicted frame.

In an eighth aspect, an embodiment of the present application provides an electronic device, characterized in that the electronic device includes: one or more processors, a memory, and a display screen; the memory is coupled to the one or more processors, and the memory is used to store a computer Program code, the computer program code includes computer instructions, and the one or more processors are used for invoking the computer instructions to cause the electronic device to execute: according to the predicted motion vector of the block around the first vertex from the target reference frame to the predicted frame, determine the first vertex from the predicted frame. The predicted motion vector from the target reference frame to the predicted frame, wherein the first vertex is a vertex in the first block, the first block is a block in the target reference frame, and the target reference frame is relative to the first reference frame according to the predicted frame Or the position of the second reference frame, a frame determined from the first reference frame or the second reference frame, the first reference frame and the second reference frame are two adjacent frames in the video stream; The coordinates of the first vertex and the predicted motion vector of the first vertex determine the coordinates of the first vertex in the predicted frame; according to the coordinates of the vertex of the first block in the predicted frame and the coordinates in the target reference frame, determine the first block A block of pixels in a predicted frame; the predicted frame is displayed on the display screen, including the block of pixels.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: obtain the pixels of the first block in the predicted frame according to the coordinates of the vertices of the first block in the predicted frame and the coordinates in the target reference frame The corresponding relationship between the coordinates in the target reference frame and the coordinates in the target reference frame, according to the coordinates and the corresponding relationship of the pixels in the first block in the target reference frame, determine the coordinates of the pixels in the first block in the predicted frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: respectively input the coordinates of the four vertices of the first block in the predicted frame and the coordinates in the target reference frame into the homography transformation formula, Obtain the homography equation system, which includes four equations; solve the homography equation system to obtain the homography transformation matrix corresponding to the first block; according to the homography transformation matrix corresponding to the first block, obtain the first block. A homography transformation formula corresponding to one block, and the homography transformation formula corresponding to the first block is used to represent the correspondence between the coordinates of the pixels of the first block in the predicted frame and the coordinates in the target reference frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: input the coordinates of the first pixel in the target reference frame into the homography transformation formula corresponding to the first block to obtain the first pixel in the predicted frame The coordinates in , the first pixel is a pixel of the first block.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: remove the coordinates of the pixels of the first block in the predicted frame from the coordinates in the area of the pixel block to obtain the first coordinates; The coordinates are input into the homography transformation formula corresponding to the first block, and the pixels of the first coordinates in the target reference frame are obtained.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: acquire the first reference frame and the second reference frame; Determine the target reference frame; divide the target reference frame into blocks according to the square blocks of the first size; calculate the motion vector of the block from the target reference frame to the predicted frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: obtain a motion vector of the first block from the first reference frame to the second reference frame; Half of the motion vector of the reference frame is determined as the predicted motion vector of the first block from the target reference frame to the predicted frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: obtain a motion vector of the first block from the second reference frame to the first reference frame; The negative half of the motion vector of the reference frame is determined as the predicted motion vector of the first block from the target reference frame to the predicted frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: determine the average value of the predicted motion vectors of the blocks around the first vertex from the target reference frame to the predicted frame as the first vertex from the target reference frame. The predicted motion vector to the predicted frame.

With reference to the eighth aspect, in a possible implementation manner, the processor is specifically configured to: add the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex to obtain the first vertex in the predicted frame coordinate of.

In a ninth aspect, the present application provides an image frame generation method, the method may include: according to the depth values of the eleventh block and the twelfth block, the eleventh block and the twelfth block are in the image frame Position coordinates, determine the tenth position coordinates of the eleventh vertex of the predicted block in the predicted image frame; wherein, the eleventh block is a block in the first image frame; the twelfth block is the second image The block that matches the eleventh block determined according to the matching algorithm in the frame; the predicted block is generated according to the color data of the reference block and the tenth position coordinate, and the reference block is the eleventh block or the tenth position coordinate. generating the predicted image frame, the predicted image frame including the predicted block.

Among them, the color data may be the RGB value of each pixel included in the block.

It can be seen that in the embodiment of the present application, the depth value is combined to predict the block, so that the generated prediction block can be scaled according to the depth value, so that the picture displayed by the predicted image frame is more consistent with the scene drawn by the electronic device according to the actual data; When the device inserts the predicted image frame into the original image frame drawn according to the actual data, the transition between the original image frame and the predicted image frame is more natural and smooth, which improves the user experience.

With reference to the ninth aspect, in a possible implementation manner, the position coordinates in the image frame are the position coordinates in a first coordinate system, and the first coordinate system is a two-dimensional coordinate system, and the first coordinate system of the determined prediction block is The tenth position coordinates of the eleven vertices in the predicted image frame include: according to the first depth value of the eleventh block and the twelfth position coordinates of the eleventh block in the first coordinate system, Calculate the thirteenth position coordinate under the second coordinate system, and the second coordinate system is a three-dimensional coordinate system; according to the second depth value of the twelfth block and the first coordinate of the twelfth block The fourteenth position coordinates under the system, the fifteenth position coordinates under the second coordinate system are calculated; according to the thirteenth position coordinates and the fifteenth position coordinates, the second coordinates are calculated The sixteenth position coordinates under the system; and the tenth position coordinates under the first coordinate system are calculated according to the sixteenth position coordinates.

Wherein, the first coordinate system may be a screen coordinate system; the second coordinate system may be a camera coordinate system. Specifically, the position coordinates in the image frame in the embodiment of the present application are the coordinates in the screen coordinate system.

It can be seen that in the embodiment of the present application, the electronic device can convert the position coordinates in the two-dimensional coordinate system into the three-dimensional space in combination with the depth value, and then realize the prediction of the position coordinates in the three-dimensional space by combining the changes of the three dimensions, and then convert the three-dimensional coordinates into the three-dimensional space. The position coordinates predicted in the space are converted into two-dimensional space, and the position coordinates in the two-dimensional coordinate system are determined. In this example, the prediction of position coordinates is realized by combining the data of three dimensions, and then a certain proportion of enlargement and reduction can be performed based on the reference block when generating the predicted block, so that the finally generated predicted image frame and the electronic device are drawn according to the data of the application program. results are more similar.

With reference to the ninth aspect, in a possible implementation manner, before the generating the prediction block, the method further includes: according to the depth values of the thirteenth block and the fourteenth block, the thirteenth block and the The position coordinates of the fourteenth block in the image frame, determine the seventeenth position coordinates of the twelfth vertex of the predicted block in the predicted image frame; wherein, the thirteenth block is the same as the eleventh block. A block adjacent to a block, the fourteenth block is a block adjacent to the twelfth block, and the thirteenth block and the fourteenth block are mutually determined according to the matching algorithm. The matched block; according to the depth values of the fifteenth block and the sixteenth block, and the position coordinates of the fifteenth block and the sixteenth block in the image frame, it is determined that the thirteenth vertex of the prediction block is in the the eighteenth position coordinates in the predicted image frame; wherein, the fifteenth block is a block adjacent to the thirteenth block, and the sixteenth block is adjacent to the fourteenth block a block, the fifteenth block and the sixteenth block are mutually matched blocks determined according to the matching algorithm; block and the position coordinates of the eighteenth block in the image frame, determine the nineteenth position coordinates of the fourteenth vertex of the prediction block in the predicted image frame; wherein, the seventeenth block is the same as the The eleventh block and the fifteenth block are both adjacent blocks, the eighteenth block is a block adjacent to the twelfth block and the sixteenth block, and the eighteenth block is a block adjacent to both the twelfth block and the sixteenth block. The seventeenth block and the eighteenth block are mutually matched blocks determined according to the matching algorithm.

It can be seen that the electronic device can determine the position coordinates of the remaining vertices of the predicted block according to the adjacent blocks, so that there is no overlap or gap in the finally generated predicted image frame, which is beneficial to improve the picture display effect of the image frame.

With reference to the ninth aspect, in a possible implementation manner, the position coordinates in the image frame are the position coordinates in the first coordinate system, and the depth values of the thirteenth and fourteenth blocks, the The position coordinates of the thirteenth block and the fourteenth block in the image frame, and determining the seventeenth position coordinate of the twelfth vertex of the predicted block in the predicted image frame, including: according to the thirteenth block The third depth value and the twentieth position coordinates of the thirteenth block under the first coordinate system are calculated to obtain the twenty-first position coordinates under the second coordinate system, and the second coordinate system is a three-dimensional coordinate system; according to the second depth value of the fourteenth block and the twenty-second position coordinate of the fourteenth block in the first coordinate system, the first coordinate system under the second coordinate system is obtained by calculating Twenty-three position coordinates; according to the twenty-first position coordinates and the twenty-third position coordinates, the twenty-fourth position coordinates under the second coordinate system are calculated; according to the twenty-fourth position coordinates position coordinates, the seventeenth position coordinates under the first coordinate system are obtained by calculation.

With reference to the ninth aspect, in a possible implementation manner, the predicted block is generated according to the color data of the reference block and the tenth position coordinate, and the reference block is the eleventh block or the tenth block One of the two blocks includes: determining the positions of the tenth position coordinate, the seventeenth position coordinate, the eighteenth position coordinate and the nineteenth position coordinate respectively and the positions of the four vertices in the reference block The corresponding relationship of coordinates; the predicted block is generated according to the corresponding relationship and the color data of the reference block.

With reference to the ninth aspect, in a possible implementation manner, the color data includes color data of each pixel in the reference block, and the color data generated according to the corresponding relationship and the color data of the reference block The prediction block includes: generating a homography transformation formula according to the corresponding relationship; determining each pixel according to the homography transformation formula and the position coordinates of each pixel in the reference image frame The position coordinates of the pixels in the predicted image frame; the predicted block is generated according to the color data of each pixel and the position coordinates of each speed limit in the predicted image frame.

With reference to the ninth aspect, in a possible implementation manner, the reference block includes: a fifteenth vertex, a sixteenth vertex, a seventeenth vertex and an eighteenth vertex, and the determining of the tenth position coordinate, The corresponding relationship between the seventeenth position coordinates, the eighteenth position coordinates and the nineteenth position coordinates respectively and the position coordinates of the four vertices in the reference block, including: determining the tenth position coordinates and the The position coordinates of the fifteenth vertex are a set of corresponding position coordinates; it is determined that the seventeenth position coordinates and the position coordinates of the sixteenth vertex are a set of corresponding position vertices; the eighteenth position coordinates and The position coordinates of the seventeenth vertex are a set of corresponding position coordinates; it is determined that the nineteenth position coordinates and the position coordinates of the eighteenth vertex are a set of corresponding position coordinates; wherein, the reference block is quadrilateral, the position of the fifteenth vertex relative to the center point of the reference block is the same as the position of the eleventh block relative to the fifteenth block; the sixteenth vertex relative to the reference block The position of the center point is the same as the position of the thirteenth block relative to the seventeenth block; the position of the seventeenth vertex relative to the center point of the reference block is the same as that of the fifteenth block relative to the The position of the eleventh block is the same; the position of the eighteenth vertex relative to the center point of the reference block is the same as the position of the seventeenth block relative to the thirteenth block.

For example, the reference block is a square, the fifteenth vertex is the vertex in the upper left corner of the reference block, the sixteenth vertex is the vertex in the upper right corner of the reference block, and the seventeenth vertex is the vertex in the upper right corner of the reference block The vertex in the upper left corner of the reference block, the eighteenth vertex is the vertex in the upper right corner of the reference block; then the thirteenth block is a block adjacent to the right side of the eleventh block, and the tenth block The seventh block is a block immediately below the eleventh block, the fifteenth block is a block adjacent to the thirteenth block and the seventeenth block, and the fifteenth block The vertex of the upper left corner of the block coincides with the vertex of the lower right corner of the eleventh block.

With reference to the ninth aspect, in a possible implementation manner, the sixteenth position coordinates under the second coordinate system are calculated and obtained according to the thirteenth position coordinates and the fifteenth position coordinates, Including: calculating the displacement vector from the thirteenth position coordinate to the fifteenth position coordinate to obtain a first displacement vector; according to the thirteenth position coordinate or the fifteenth position coordinate, and the first proportional The sixteenth position coordinate is obtained by calculating the first displacement vector.

With reference to the ninth aspect, in a possible implementation manner, the first ratio is equal to a preset value.

It can be seen that in the case that the first ratio is equal to the preset value, the electronic device can quickly determine the sixteenth position coordinates.

With reference to the ninth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, the first image frame is located before the second image frame in the data stream, and the calculation obtains Before the sixteenth position coordinate, the method further includes: determining that the first ratio is equal to the ratio of the absolute value of the first time difference value to the absolute value of the second time difference value; the first time difference value is equal to the difference between the time point of the predicted image frame in the data stream and the time point of the reference frame in the data stream; the second time difference value is equal to the first image frame in the data stream The difference between the time point of and the time point of the second image frame in the data stream.

It can be seen that the electronic device can determine the first ratio according to the difference between the predicted image frame and the reference image frame, so that the scene displayed by the predicted image frame (relative to the reference image frame, the position of the block and the zoom degree) is different from the displayed predicted image. The moment of the frame is more consistent with the scene constructed by the electronic device based on the actual operating data.

With reference to the ninth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, the first image frame is located before the second image frame in the data stream, and the calculation obtains Before the sixteenth position coordinate, the method further includes: determining that the first ratio is equal to a ratio of a first value to a second value; the first value is equal to the first number plus one; the second value is equal to a second number plus one; the first number is equal to the number of image frames in the data stream that are spaced between the reference image frame and the predicted image frame; the second number is equal to the number of image frames in the data stream The number of image frames spaced between the reference image frame and the predicted image frame.

It can be seen that the electronic device can determine the first ratio according to the number of image frames separated between the predicted image frame and the reference image frame, so that in the process that the electronic device displays each image frame at a constant speed, the scene displayed by the predicted image frame (relative to The reference image frame, block position and zoom level) more closely matches the scene constructed by the electronic device based on the actual operating data at the moment when the predicted image frame is displayed.

With reference to the ninth aspect, in a possible implementation manner, the tenth position coordinate is calculated according to the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale is calculated to obtain the tenth position. Six position coordinates, including: if the reference image frame is the first image frame, calculating the sixteenth position according to the thirteenth position coordinates and the first displacement vector of the first scale coordinates; if the reference image frame is the second image frame, the sixteenth position coordinate is obtained by calculating the fifteenth position coordinate and the first displacement vector of the first scale.

With reference to the ninth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, the first image frame is located before the second image frame in the data stream, and the determining prediction The eleventh vertex of the block is before the tenth position coordinate in the predicted image frame, and the method further includes: determining one image frame in the first image frame and the second image frame as the reference image frame ; Determine that another image frame other than the reference image frame in the first image frame and the second image frame is a matching image frame; Carry out block division for the reference image frame to obtain the reference block; determine according to the matching algorithm that the block that matches the reference block in the matched image frame is a matching block; wherein, if the reference block is the eleventh block, the matching block is the first block twelve blocks; if the reference block is the twelfth block, the matching is the eleventh block.

The electronic device may designate the first image frame or one of the second image frames as the reference image frame, for example, the electronic device may designate the first image frame tested in the data stream as the reference image frame. Optionally, the electronic device may determine, based on the position of the predicted image frame relative to the first image frame and the second image frame, that one image frame in the first image frame and the second image frame is the reference image frame; specifically, if If the predicted image frame is located between the first image frame and the second image frame in the data stream, the first image frame is determined as the reference image frame; if the predicted image frame is located after the first image frame and the second image frame in the data stream , the second image frame is determined to be the reference image frame.

It can be seen that the electronic device can divide the reference image frame into blocks, and search for blocks in the matching image frame in units of blocks in the reference image frame, and then determine the blocks that match each other in the first image frame and the second image frame.

With reference to the ninth aspect, in a possible implementation manner, the determining according to the matching algorithm that the block that matches the reference block in the matching image frame is a matching block includes: determining the first block of the matching image frame. The block with the highest similarity with the reference block in a region is the matching block; the first region is the region of the matching image frame with the reference position coordinates as the center and the preset shape of the first area, the The reference position coordinates are the position coordinates of the reference block in the reference image frame.

It can be seen that the electronic device can determine the first region in the matching image frame according to the position coordinates of the reference block, which may narrow the search range to a certain extent, thereby helping to improve the matching efficiency and matching accuracy of the block.

With reference to the ninth aspect, in a possible implementation manner, before the block with the highest similarity to the reference block in the first region of the matched image frame is determined as the matched block, the method further includes: The first area is obtained by calculation according to the reference depth value of the reference block, wherein the larger the reference depth value is, the smaller the first area obtained by calculation is.

It can be seen that the electronic device can adjust the size of the first area according to the depth value, which is beneficial to improve the efficiency of determining the matching block.

With reference to the ninth aspect, in a possible implementation manner, before the block with the highest similarity to the reference block in the first region of the matched image frame is determined as the matched block, the method further includes: If the reference depth value of the reference block is greater than or equal to the first threshold, it is determined that the first area is equal to the first preset area; if the reference depth value is greater than the second threshold and less than the first threshold, according to the reference The depth value is calculated to obtain the first area, wherein, the larger the reference depth value is, the smaller the first area is; if the reference depth value is less than or equal to the second threshold value, it is determined that the first area is equal to the first area. Two preset areas; wherein the first threshold is greater than the second threshold, and the second preset area is greater than the first preset area.

For example, the value range of the depth value may be greater than or equal to 0 and less than or equal to 1, the first threshold may be equal to 0.8, the second threshold may be equal to 0.2, the first preset area may be twice the area of the reference block; the second preset area may be equal to The area of the reference image frame.

It can be seen that when the depth value is large, the electronic device searches for the matching block of the reference block in a small range; when the depth value is small, the electronic device can search for the matching block of the reference block in the entire image frame. It is beneficial to improve the efficiency of determining matching blocks.

In a tenth aspect, the present application provides an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program codes, The computer program code includes computer instructions invoked by the one or more processors to cause the electronic device to execute: the eleventh block according to the depth values of the eleventh and twelfth blocks and the position coordinates of the twelfth block in the image frame, determine the tenth position coordinate of the eleventh vertex of the predicted block in the predicted image frame; wherein, the eleventh block is a block in the first image frame The twelfth block is the block that matches the eleventh block determined according to the matching algorithm in the second image frame; According to the color data of the reference block and the tenth position coordinate, a prediction block is generated, and the The reference block is one of the eleventh block or the twelfth block; the predicted image frame is generated, and the predicted image frame includes the predicted block.

With reference to the tenth aspect, in a possible implementation manner, the position coordinates in the image frame are the position coordinates in a first coordinate system, and the first coordinate system is a two-dimensional coordinate system. In terms of predicting the tenth position coordinate of the eleventh vertex in the image frame, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: according to the first position of the eleventh block A depth value and the twelfth position coordinate of the eleventh block under the first coordinate system are calculated to obtain the thirteenth position coordinate under the second coordinate system, and the second coordinate system is a three-dimensional coordinate system; According to the second depth value of the twelfth block and the fourteenth position coordinate of the twelfth block under the first coordinate system, the fifteenth position coordinate under the second coordinate system is obtained by calculating; According to the thirteenth position coordinates and the fifteenth position coordinates, the sixteenth position coordinates under the second coordinate system are calculated; according to the sixteenth position coordinates, the first position coordinates are calculated and obtained the tenth position coordinate in the coordinate system.

With reference to the tenth aspect, in a possible implementation manner, before generating the prediction block, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: according to the tenth aspect The depth values of the third block and the fourteenth block, and the position coordinates of the thirteenth block and the fourteenth block in the image frame, determine the twelfth vertex of the prediction block in the prediction image frame. seventeen position coordinates; wherein, the thirteenth block is a block adjacent to the eleventh block, the fourteenth block is a block adjacent to the twelfth block, and the thirteenth block is a block adjacent to the twelfth block. The thirteenth block and the fourteenth block are mutually matched blocks determined according to the matching algorithm; according to the depth values of the fifteenth block and the sixteenth block, the fifteenth block and the sixteenth block the position coordinates of the block in the image frame, and determine the eighteenth position coordinates of the thirteenth vertex of the predicted block in the predicted image frame; wherein, the fifteenth block is adjacent to the thirteenth block The sixteenth block is a block adjacent to the fourteenth block, and the fifteenth block and the sixteenth block are mutually matched blocks determined according to the matching algorithm ; According to the depth values of the seventeenth block and the eighteenth block, the position coordinates of the seventeenth block and the eighteenth block in the image frame, determine that the fourteenth vertex of the predicted block is in the predicted image The nineteenth position coordinate in the frame; wherein, the seventeenth block is a block adjacent to both the eleventh block and the fifteenth block, and the eighteenth block is a block adjacent to the eighteenth block. The twelve blocks and the sixteenth block are both adjacent blocks, and the seventeenth block and the eighteenth block are mutually matched blocks determined according to the matching algorithm.

With reference to the tenth aspect, in a possible implementation manner, the position coordinates in the image frame are the position coordinates in the first coordinate system, and in the depth value according to the thirteenth block and the fourteenth block, the The location coordinates of the thirteenth block and the fourteenth block in the image frame, determining the seventeenth location coordinates of the twelfth vertex of the predicted block in the predicted image frame, the one or more processing a controller, specifically for invoking the computer instructions to cause the electronic device to execute: according to the third depth value of the thirteenth block and the twentieth position of the thirteenth block in the first coordinate system coordinates, the twenty-first position coordinates under the second coordinate system are calculated, and the second coordinate system is a three-dimensional coordinate system; according to the second depth value of the fourteenth block and the fourteenth block in the The twenty-second position coordinate under the first coordinate system is calculated to obtain the twenty-third position coordinate under the second coordinate system; according to the twenty-first position coordinate and the twenty-third position coordinate, The twenty-fourth position coordinate under the second coordinate system is obtained by calculation; according to the twenty-fourth position coordinate, the seventeenth position coordinate under the first coordinate system is calculated and obtained.

With reference to the tenth aspect, in a possible implementation manner, in the aspect of generating the prediction block according to the color data of the reference block and the tenth position coordinate, the one or more processors are specifically configured to call the the computer instructions to cause the electronic device to execute: determine the tenth position coordinate, the seventeenth position coordinate, the eighteenth position coordinate and the nineteenth position coordinate respectively and the four vertices in the reference block The corresponding relationship of the position coordinates of the reference block; the predicted block is generated according to the corresponding relationship and the color data of the reference block.

With reference to the tenth aspect, in a possible implementation manner, the color data includes color data of each pixel in the reference block, and the color data according to the corresponding relationship and the reference block is included in the color data. In terms of generating the prediction block, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to perform: generating a homography transformation formula according to the correspondence; generating a homography transformation formula according to the homography; The transformation formula and the position coordinates of each pixel in the reference image frame determine the position coordinates of each pixel in the predicted image frame; according to the color data of each pixel and the The predicted block is generated from the position coordinates of each rate limit in the predicted image frame.

With reference to the tenth aspect, in a possible implementation manner, the reference block includes: a fifteenth vertex, a sixteenth vertex, a seventeenth vertex and an eighteenth vertex, and in the determining of the tenth position coordinate , the corresponding relationship between the seventeenth position coordinates, the eighteenth position coordinates and the nineteenth position coordinates respectively and the position coordinates of the four vertices in the reference block, the one or more processors, specifically For invoking the computer instruction to cause the electronic device to execute: determine that the tenth position coordinate and the position coordinate of the fifteenth vertex are a set of corresponding position coordinates; determine that the seventeenth position coordinate and all The position coordinates of the sixteenth vertex are a set of corresponding position vertices; it is determined that the eighteenth position coordinates and the position coordinates of the seventeenth vertex are a set of corresponding position coordinates; the nineteenth position coordinates are determined The position coordinates corresponding to the eighteenth vertex are a set of position coordinates; wherein the reference block is a quadrilateral, and the position direction of the fifteenth vertex relative to the center point of the reference block is the same as that of the tenth vertex. The position direction of a block relative to the fifteenth block is the same; the position direction of the sixteenth vertex relative to the center point of the reference block is the same as the position of the thirteenth block relative to the seventeenth block The position direction of the seventeenth vertex relative to the center point of the reference block is the same as the position direction of the fifteenth block relative to the eleventh block; the eighteenth vertex relative to the The position direction of the center point of the reference block is the same as the position direction of the seventeenth block relative to the thirteenth block.

With reference to the tenth aspect, in a possible implementation manner, in the calculation according to the thirteenth position coordinates and the fifteenth position coordinates, the sixteenth position coordinates under the second coordinate system are obtained by calculation In an aspect, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: calculate a displacement vector from the thirteenth position coordinate to the fifteenth position coordinate, and obtain the first Displacement vector; the sixteenth position coordinate is obtained by calculating according to the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale.

With reference to the tenth aspect, in a possible implementation manner, the first ratio is equal to a preset value.

With reference to the tenth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream. Before obtaining the sixteenth position coordinates, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: determining that the first ratio is equal to the absolute value of the first time difference The ratio to the absolute value of the second time difference; the first time difference is equal to the difference between the time point of the predicted image frame in the data stream and the time point of the reference frame in the data stream; The second time difference value is equal to the difference between the time point of the first image frame in the data stream and the time point of the second image frame in the data stream.

With reference to the tenth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream. Before obtaining the sixteenth position coordinates, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to perform: determining that the first ratio is equal to the difference between the first value and the second value. the first value is equal to the first number plus one; the second value is equal to the second number plus one; the first number is equal to the difference between the reference image frame and the predicted image frame in the data stream the number of image frames spaced between; the second number is equal to the number of image frames spaced between the reference image frame and the predicted image frame in the data stream.

With reference to the tenth aspect, in a possible implementation manner, in the calculation according to the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale, the In terms of position coordinates, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: if the reference image frame is the first image frame, according to the first image frame The thirteenth position coordinates and the first displacement vector of the first scale are calculated to obtain the sixteenth position coordinates; if the reference image frame is the second image frame, the fifteenth position coordinates are calculated according to the The sixteenth position coordinate is obtained by calculating with the first displacement vector of the first scale.

With reference to the tenth aspect, in a possible implementation manner, the image frame where the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream. The eleventh vertex of the predicted block is before the tenth position coordinate in the predicted image frame, and the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: determine the first image frame and one image frame in the second image frame is the reference image frame; determine another image frame other than the reference image frame in the first image frame and the second image frame In order to match the image frame; the reference image frame is divided into blocks to obtain the reference block; according to the matching algorithm, the block that matches the reference block in the matched image frame is determined as a matching block; wherein, if the If the reference block is the eleventh block, the matching block is the twelfth block; if the reference block is the twelfth block, the matching block is the eleventh block.

With reference to the tenth aspect, in a possible implementation manner, in the aspect of determining, according to the matching algorithm, that the block that matches the reference block in the matched image frame is a matching block, the one or more processors , which is specifically used to invoke the computer instruction to cause the electronic device to execute: determine the block with the highest similarity to the reference block in the first region of the matching image frame as the matching block; the first region is The matching image frame is a region of a preset shape with a first area centered on a reference position coordinate, where the reference position coordinate is the position coordinate of the reference block in the reference image frame.

With reference to the tenth aspect, in a possible implementation manner, before the determining that the block with the highest similarity to the reference block in the first region of the matching image frame is the matching block, the one or more a processor is further configured to invoke the computer instructions to cause the electronic device to perform: calculating and obtaining the first area according to the reference depth value of the reference block, wherein the larger the reference depth value, the greater the calculated value. The first area is smaller.

With reference to the tenth aspect, in a possible implementation manner, before the determining that the block with the highest similarity to the reference block in the first region of the matching image frame is the matching block, the one or more a processor, further configured to invoke the computer instructions to cause the electronic device to execute: if the reference depth value of the reference block is greater than or equal to a first threshold, determine that the first area is equal to a first preset area; The reference depth value is greater than the second threshold value and smaller than the first threshold value, and the first area is calculated according to the reference depth value, wherein the larger the reference depth value is, the smaller the first area is; If the reference depth value is less than or equal to the second threshold, it is determined that the first area is equal to a second preset area; wherein, the first threshold is greater than the second threshold, and the second preset area is greater than the first a preset area.

An eleventh aspect provides a method for generating an image frame, the method may include: in a first drawing cycle, according to a drawing instruction of a target application, when the drawing instruction satisfies a first condition, storing a drawing result in a The seventh storage space, as the first drawing result, when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space as the second drawing result; according to the first drawing result and the The second drawing result generates a seventh image frame; in the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space, as a third drawing result, when the drawing instruction satisfies the second condition, no drawing is performed; an eighth image frame is generated according to the third drawing result and the second drawing result. Wherein, the rendering result generated when the rendering instruction satisfies the first condition may present a dynamic interface on the display screen. The effect that the drawing result generated when the drawing instruction satisfies the second condition is presented on the display screen may be a control. In this way, by implementing the method of the first aspect, the electronic device can store the drawing results obtained according to the drawing instructions of different conditions in different storage spaces according to the drawing instructions of the target application, so that one copy can be shared in the two drawing cycles. Drawing result; reducing the drawing time of the electronic device when the drawing instruction satisfies the second condition, and reducing the power consumption of the electronic device.

With reference to the eleventh aspect, in a possible implementation manner, the method further includes: in the third drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, storing the drawing result in the into the seventh storage space, as the fourth drawing result, when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space, as the fifth drawing result, the third The drawing period is after the first drawing period; the ninth image frame is generated according to the fourth drawing result and the fifth drawing result; in the fourth drawing period, the first drawing result and the fourth drawing result are generated according to the fourth drawing period The drawing result generates the sixth drawing result; the tenth image frame is generated according to the sixth drawing result and the seventh drawing result; the seventh drawing result is that in the fifth drawing cycle, the drawing instruction of the target application satisfies all requirements. The drawing result obtained under the second condition, the fifth drawing cycle is before the fourth drawing cycle. Among them, this example can be used in interpolated frames. The electronic device may generate a predicted sixth rendering result according to the rendering results when the two rendering instructions satisfy the first condition; and generate the predicted insertion in combination with the seventh rendering result obtained when the rendering instructions satisfy the second condition in the previous rendering cycle image frame. In the process of generating the predicted inserted image frame, the electronic device can reuse the seventh rendering result, which reduces the power consumption when generating the inserted image frame; and in the process of generating the inserted image frame, the seventh rendering result is There is no need to perform prediction calculation for the effect presented in the . The electronic device only calculates the prediction calculation based on the rendering result when the two rendering instructions satisfy the first condition. The power consumption of the electronic device is further reduced.

With reference to the eleventh aspect, in a possible implementation manner, in that the electronic device generates a seventh image frame according to the first drawing result and the second drawing result, the method includes: The drawing result and the second drawing result are stored in a ninth storage space; and the seventh image frame is generated in the ninth storage space according to the first drawing result and the second drawing result.

With reference to the eleventh aspect, in a possible implementation manner, in the aspect of generating a ninth image frame according to the fourth rendering result and the fifth rendering result, the method includes: converting the fourth rendering result and the fifth rendering result is stored in the ninth storage space; and the ninth image frame is generated in the ninth storage space according to the fourth rendering result and the fifth rendering result.

With reference to the eleventh aspect, in a possible implementation manner, in the aspect of generating the tenth image frame according to the sixth rendering result and the seventh rendering result, the method includes: combining the sixth rendering result with the all The seventh rendering result is stored in the ninth storage space; and the tenth image frame is generated in the ninth storage space according to the sixth rendering result and the seventh rendering result.

With reference to the eleventh aspect, in a possible implementation manner, the seventh storage space consists of a tenth storage space and an eleventh storage space, and the first drawing result is stored in the tenth storage space, The fourth drawing result is stored in the eleventh storage space, and generating a sixth drawing result according to the first drawing result and the fourth drawing result includes: storing the first drawing result storing in the eleventh storage space; and generating the sixth drawing result in the eleventh storage space according to the first drawing result and the fourth drawing result.

With reference to the eleventh aspect, in a possible implementation manner, the seventh storage space consists of at least three storage spaces, and the at least three storage spaces include a tenth storage space, an eleventh storage space and a tenth storage space Two storage spaces; the first drawing result is stored in the tenth storage space, the fourth drawing result is stored in the eleventh storage space, and the first drawing result and the fourth drawing result are stored in the eleventh storage space. Aspects of generating a sixth drawing result include: storing the first drawing result and the fourth drawing result in a twelfth storage space; storing the first drawing result and the fourth drawing result in the twelfth storage space according to the The fourth rendering result generates the sixth rendering result.

With reference to the eleventh aspect, in a possible implementation manner, the third drawing period is adjacent to the fourth drawing period; the fifth drawing period and the third drawing period are the same drawing period, so The fifth drawing result and the seventh drawing result are the same drawing result. When generating the inserted image frame, the electronic device can reuse the drawing result used to generate the image of the previous frame, which reduces the power consumption of the electronic device.

With reference to the eleventh aspect, in a possible implementation manner, in the aspect of generating a seventh image frame according to the first drawing result and the second drawing result, the method includes: storing the second drawing result in a In the seventh storage space; the seventh image frame is generated in the seventh storage space according to the first drawing result and the second drawing result.

In conjunction with the eleventh aspect, in a possible implementation manner, the electronic device applying the method includes a counting unit; the initialization value of the counting unit is a first value, and the value of the counting unit is updated at the first value every time. and the second value is switched once; the counting unit is updated at the moment when the drawing cycle starts; in a drawing cycle, if the updated value of the counting unit is the second value, then at the moment when the drawing cycle starts Empty the eighth storage space; and when the drawing instruction satisfies the second condition, store the drawing result in the eighth storage space; if the updated value of the counting unit is the first value, then draw When the cycle starts, the second storage unit is not emptied; and when the drawing instruction satisfies the second condition, the drawing is not performed. It can be seen that by setting the counting unit, in the process of drawing according to the drawing instruction of the target application program, the electronic device can share a drawing result obtained when the drawing instruction satisfies the second condition for every two adjacent drawing cycles, reducing the drawing result of the electronic device. The amount of data processing in the process reduces the power consumption of electronic equipment.

With reference to the eleventh aspect, in a possible implementation manner, the method further includes: during a third drawing cycle, generating a fourth drawing result according to the first drawing result and the third drawing result; The second rendering result and the fourth rendering result generate a ninth image frame.

With reference to the eleventh aspect, in a possible implementation manner, the electronic device applying the method includes a counting unit, the initialization value of the counting unit is a first value, and the counting unit is based on the first value, the second value, The sequence of the third numerical value is repeatedly updated and switched between the three numerical values, and the counting unit is updated at the moment when the drawing cycle starts. In one drawing cycle, if the updated value of the counting unit is the second numerical value, Then clear the eighth storage space at the moment when the drawing cycle starts; and when the drawing instruction satisfies the second condition, store the drawing result into the eighth storage space; if the updated value of the counting unit is If the first value or the third value is used, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not performed. It can be seen that by setting the counting unit, in the process of drawing according to the drawing instruction of the target application, the electronic device can share a drawing result obtained when the drawing instruction satisfies the second condition every three drawing cycles, which reduces the number of times in the drawing process of the electronic device. The amount of data processing, reduces the power consumption of electronic equipment.

With reference to the eleventh aspect, in a possible implementation manner, the ninth storage space is composed of at least two storage spaces, and the at least two storage spaces are based on the first storage space of the target application when the target application is running. An instruction alternates between a first state and a second state; at the same time point, only one of the at least two storage spaces is in the first state, and the rest of the storage spaces are in the first state Two states; when the at least two storage spaces are in the first state, the image frames in the at least two storage spaces can be transmitted to a display device for display; when the at least two storage spaces are in the second state , the electronic device can perform drawing in the at least two storage spaces; two image frames generated by two adjacent drawing cycles are respectively stored in two different storage spaces in the ninth storage space. It can be seen that the electronic device can rotate the storage space in which the image frames are stored, and draw the next image frame in the process of displaying the previous image frame, which improves the efficiency of generating and displaying the image frame by the electronic device.

With reference to the eleventh aspect, in a possible implementation manner, the first condition is: the drawing instruction includes an instruction to enable depth testing; the second condition is: the drawing instruction includes an instruction to disable depth testing instruction. It can be seen that by turning on or off the depth test command, the division of the drawing results is achieved, and then the layers used for the display space are reused. The amount of data processing in the process of drawing image frames is small, which reduces the power consumption of electronic devices.

With reference to the eleventh aspect, in a possible implementation manner, the time nodes of any two adjacent drawing cycles are the time points at which the target application invokes the second instruction. Wherein, the first instruction and the second instruction may be the same instruction. That is, each time the electronic device rotates the storage space in which the image frame is stored, it can be regarded as the end of the previous drawing period and the start of the next drawing period in the two adjacent drawing periods.

A twelfth aspect provides an electronic device, the electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used for storing computer program codes, the memory is The computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the electronic device to execute: during the first drawing cycle, according to the drawing instructions of the target application, when the drawing instructions satisfy Under the first condition, the drawing result is stored in the seventh storage space as the first drawing result, and when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space as the second drawing result; A seventh image frame is generated according to the first drawing result and the second drawing result; in the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, The drawing result is stored in the seventh storage space, and as the third drawing result, when the drawing instruction satisfies the second condition, no drawing is performed; generating according to the third drawing result and the second drawing result Eighth image frame.

With reference to the twelfth aspect, in a possible implementation manner, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: during the third drawing cycle, according to the target application The drawing instruction of the program, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space, as the fourth drawing result, when the drawing instruction satisfies the second condition, the drawing The result is stored in the eighth storage space, and as the fifth drawing result, the third drawing cycle is located after the first drawing cycle; the ninth image frame is generated according to the fourth drawing result and the fifth drawing result ; During the fourth drawing cycle, generate a sixth drawing result according to the first drawing result and the fourth drawing result; generate a tenth image frame according to the sixth drawing result and the seventh drawing result; the seventh drawing result The drawing result is the drawing result obtained when the drawing instruction of the target application satisfies the second condition in the fifth drawing cycle, and the fifth drawing cycle is before the fourth drawing cycle.

With reference to the twelfth aspect, in a possible implementation manner, in the aspect of generating the seventh image frame according to the first rendering result and the second rendering result, the one or more processors specifically use invoking the computer instructions to cause the electronic device to execute: store the first drawing result and the second drawing result in a ninth storage space; and store the first drawing result in the ninth storage space according to the first drawing The result and the second rendering result generate the seventh image frame.

With reference to the twelfth aspect, in a possible implementation manner, in the aspect of generating the ninth image frame according to the fourth rendering result and the fifth rendering result, the one or more processors specifically use invoking the computer instructions to cause the electronic device to execute: storing the fourth drawing result and the fifth drawing result into the ninth storage space; in the ninth storage space according to the fourth The drawing result and the fifth drawing result generate the ninth image frame.

With reference to the twelfth aspect, in a possible implementation manner, in the aspect of generating the tenth image frame according to the sixth rendering result and the seventh rendering result, the one or more processors are specifically configured to call The computer instructions cause the electronic device to execute: store the sixth drawing result and the seventh drawing result in the ninth storage space; and store the sixth drawing result in the ninth storage space according to the sixth drawing result and the seventh rendering result to generate the tenth image frame.

With reference to the twelfth aspect, in a possible implementation manner, the seventh storage space is composed of a tenth storage space and an eleventh storage space, and the first drawing result is stored in the tenth storage space, The fourth drawing result is stored in the eleventh storage space, and in the aspect of generating the sixth drawing result according to the first drawing result and the fourth drawing result, the one or more processors, It is specifically used to invoke the computer instruction to cause the electronic device to execute: store the first drawing result in the eleventh storage space; and store the first drawing result in the eleventh storage space according to the first drawing result and the fourth rendering result to generate the sixth rendering result.

With reference to the twelfth aspect, in a possible implementation manner, the seventh storage space consists of at least three storage spaces, and the at least three storage spaces include a tenth storage space, an eleventh storage space and a tenth storage space Two storage spaces; the first drawing result is stored in the tenth storage space, the fourth drawing result is stored in the eleventh storage space, and the first drawing result and the fourth drawing result are stored in the eleventh storage space. In the aspect of generating the sixth drawing result, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: storing the first drawing result and the fourth drawing result in the tenth drawing result Two storage spaces; the sixth drawing result is generated in the twelfth storage space according to the first drawing result and the fourth drawing result.

With reference to the twelfth aspect, in a possible implementation manner, the third drawing period is adjacent to the fourth drawing period; the fifth drawing period and the third drawing period are the same drawing period, so The fifth drawing result and the seventh drawing result are the same drawing result.

With reference to the twelfth aspect, in a possible implementation manner, in the aspect of generating the seventh image frame according to the first rendering result and the second rendering result, the one or more processors specifically use invoking the computer instructions to cause the electronic device to execute: store the second drawing result in the seventh storage space; and store the second drawing result in the seventh storage space according to the first drawing result and the The second rendering result generates the seventh image frame.

With reference to the twelfth aspect, in a possible implementation manner, the electronic device includes a counting unit; the initialization value of the counting unit is a first value, and the value of the counting unit is updated between the first value and the first value every time. Switch between two values once; the counting unit is updated at the moment when the drawing cycle starts; in a drawing cycle, if the updated value of the counting unit is the second value, then the counting unit is cleared at the moment when the drawing cycle starts the eighth storage space; and when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space; if the updated value of the counting unit is the first value, the drawing cycle starts The second storage unit is not emptied at the time of ; and when the drawing instruction satisfies the second condition, the drawing is not performed.

With reference to the twelfth aspect, in a possible implementation manner, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: during the third drawing cycle, according to the The first rendering result and the third rendering result generate a fourth rendering result; and a ninth image frame is generated according to the second rendering result and the fourth rendering result.

With reference to the twelfth aspect, in a possible implementation manner, the electronic device includes a counting unit, the initialization value of the counting unit is a first value, and the counting unit is based on the first value, the second value, and the third value. The sequence of updating and switching is repeated between three values, and the counting unit is updated at the moment when the drawing cycle starts. In a drawing cycle, if the updated value of the counting unit is the second value, then the drawing The eighth storage space is emptied at the moment when the cycle starts; and when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space; if the updated value of the counting unit is the first value or the third value, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not performed.

With reference to the twelfth aspect, in a possible implementation manner, the ninth storage space is composed of at least two storage spaces, and the at least two storage spaces are based on the first storage space of the target application when the target application is running. An instruction alternates between a first state and a second state; at the same time point, only one of the at least two storage spaces is in the first state, and the rest of the storage spaces are in the first state Two states; when the at least two storage spaces are in the first state, the image frames in the at least two storage spaces can be transmitted to a display device for display; when the at least two storage spaces are in the second state , the electronic device can perform drawing in the at least two storage spaces; two image frames generated by two adjacent drawing cycles are respectively stored in two different storage spaces in the ninth storage space.

With reference to the twelfth aspect, in a possible implementation manner, the first condition is: the drawing instruction includes an instruction to enable depth testing; and the second condition is: the drawing instruction includes an instruction to disable depth testing.

With reference to the twelfth aspect, in a possible implementation manner, the time nodes of any two adjacent drawing cycles are the time points at which the target application invokes the second instruction.

In a thirteenth aspect, an embodiment of the present application provides an image frame prediction method. The method may include: when drawing the twenty-first drawing frame of the first application, the electronic device converts the drawing instruction of the twenty-first drawing frame to the drawing instruction of the twenty-first drawing frame. Drawing according to the first drawing range, the twenty-first drawing result is obtained, and the size of the first drawing range is larger than the size of the twenty-first drawing frame of the first application; when drawing the twenty-second drawing frame of the first application, the electronic The device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range, and obtains the twenty-second drawing result. The size of the twenty-second memory space is larger than the size of the twenty-second drawing frame, wherein the second The size of the eleventh drawing frame is the same as the size of the twenty-second drawing frame; the electronic device predicts and generates the twenty-third predicted frame of the first application according to the twenty-first drawing result and the twenty-second drawing result, wherein , the size of the twenty-third prediction frame is the same as the size of the twenty-first rendering frame.

In this way, the electronic device can obtain the predicted frame. The frame rate of the electronic device can be increased without increasing the drawing frame. In this way, while saving the power consumption of the electronic device, the smoothness of the video interface displayed by the electronic device can be improved. Further, in the predicted frame predicted by the electronic device, there may be rendering content that is not in the twenty-first rendering frame and the twenty-second rendering frame displayed by the electronic device. Therefore, the drawing content in the predicted frame predicted by the electronic device is closer to the shooting content in the shooting field of view of the camera. Therefore, the image frame predicted by the electronic device can be more accurate.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device draws the drawing instruction of the twenty-first drawing frame according to the first drawing range to obtain the twenty-first drawing result, which specifically includes: The first parameter in the first drawing instruction of the twenty-first drawing frame issued by the application is modified to the first drawing range; the first parameter is used to set the size of the drawing range of the twenty-first drawing frame; The drawing instruction of the twenty-first drawing frame is drawn according to the first drawing range, and the twenty-first drawing result is obtained.

With reference to the thirteenth aspect, in a possible implementation manner, the size of the first drawing range is larger than the size of the twenty-first drawing frame of the first application, specifically including: the width of the first drawing range is the twenty-first drawing range The width of the frame is K3 times, the height of the first drawing range is K4 times the height of the twenty-first drawing frame, and K3 and K4 are greater than 1.

With reference to the thirteenth aspect, in a possible implementation manner, K3 and K4 are determined by the fixed values of the system configuration of the electronic device, or by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device draws the drawing instruction of the modified twenty-first drawing frame according to the first drawing range to obtain the twenty-first drawing result, which specifically includes: the electronic device draws the drawing according to the first drawing range. K3 and K4 generate a first conversion matrix, and the electronic device adjusts the size of the drawing content in the drawing instruction of the modified twenty-first drawing frame into the first drawing range according to the first conversion matrix, and obtains the twenty-first Plot the result.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range to obtain the twenty-first drawing result, which specifically includes: The second parameter in the second drawing instruction of the twenty-second drawing frame issued by the application is modified to the second drawing range; the second parameter is used to set the size of the drawing range of the twenty-second drawing frame; The drawing instructions of the twenty-two drawing frames are drawn according to the second drawing range, and the twenty-second drawing result is obtained.

With reference to the thirteenth aspect, in a possible implementation manner, the size of the second drawing range is larger than the size of the twenty-second drawing frame of the first application, which specifically includes: the width of the second drawing range is the twenty-second drawing range. The width of the frame is K5 times, the height of the second drawing range is K6 times the height of the twenty-second drawing frame, and K5 and K6 are greater than 1.

With reference to the thirteenth aspect, in a possible implementation manner, K5 and K6 are determined by the fixed values of the system configuration of the electronic device, or by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device draws the modified drawing instruction of the twenty-second drawing frame according to the second drawing range to obtain the twenty-second drawing result, which specifically includes: the electronic device The second conversion matrix is generated according to K5 and K6, and the electronic device adjusts the size of the drawing content in the drawing instruction of the 22nd drawing frame after modification according to the second conversion matrix and draws it into the second drawing range, to obtain the twentieth drawing range. a plot result.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device predicts and generates the twenty-third predicted frame of the first application according to the twenty-first rendering result and the twenty-second rendering result, specifically including: the current device According to the twenty-first rendering result and the twenty-second rendering result, the twenty-third rendering result of the twenty-third predicted frame is predicted and generated; the electronic device cuts the twenty-third rendering result into the twenty-third predicted frame.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device predicts and generates the twenty-third drawing result of the twenty-third predicted frame according to the twenty-first drawing result and the twenty-second drawing result, Specifically, it includes: the electronic device determines the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result; the electronic device predicts and generates the first motion vector according to the twenty-second rendering result and the first motion vector The twenty-third rendering result of the twenty-third predicted frame.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device determines the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result, which specifically includes: the electronic device Divide the twenty-second rendering result into Q pixel blocks, and the electronic device takes out the first pixel block from the Q pixel blocks in the twenty-second rendering result; the electronic device determines in the twenty-first rendering result that it is the same as the first pixel. The second pixel block matched by the block; the electronic device obtains the motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block; the electronic device determines the first pixel block of the twenty-second rendering result according to the motion vector of the first pixel block. a motion vector.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device determines, in the twenty-first drawing result, a second pixel block that matches the first pixel block, which specifically includes: In the twenty-first drawing result, a plurality of candidate pixel blocks are determined for the first pixel point of the The difference between the color values of the first pixel block determines a second pixel block that matches the first pixel block, and the second pixel block is the candidate with the smallest difference in color value from the first pixel block among the multiple candidate pixel blocks. pixel block.

With reference to the thirteenth aspect, in a possible implementation manner, when drawing the twenty-first drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-first drawing frame according to the first drawing range, and obtains the first drawing range. The twenty-first drawing result specifically includes: when drawing the twenty-first drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-first drawing frame in the twenty-first memory space according to the first drawing range, The twenty-first drawing result is obtained, and the size of the twenty-first memory space is greater than or equal to the size of the first drawing range.

With reference to the thirteenth aspect, in a possible implementation manner, when drawing the twenty-second drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range, and obtains the The twenty-two drawing results specifically include: when drawing the twenty-second drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-second drawing frame in the twenty-second memory space according to the second drawing range, The twenty-second drawing result is obtained, and the size of the twenty-second memory space is greater than or equal to the size of the second drawing range.

With reference to the thirteenth aspect, in a possible implementation manner, the electronic device predicts and generates the twenty-third drawing result of the twenty-third predicted frame according to the twenty-second drawing result and the first motion vector, specifically including: an electronic device The device predicts and generates a twenty-third drawing result according to the third drawing range according to the twenty-second drawing result and the first motion vector, wherein the size of the third drawing range is larger than the size of the twenty-third predicted frame.

With reference to the thirteenth aspect, in a possible implementation manner, when drawing the twenty-first drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-first drawing frame according to the first drawing range, and obtains the first drawing range. After the twenty-first rendering result, the method further includes: the electronic device cuts the twenty-first rendering result into a twenty-first rendering frame.

With reference to the thirteenth aspect, in a possible implementation manner, when drawing the twenty-second drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range, and obtains the After the twenty-second rendering result, the method further includes: the electronic device cuts the twenty-second rendering result into a twenty-second rendering frame.

In a fourteenth aspect, the present application provides a method for predicting an image frame, the method may include: when drawing the twenty-first drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-first drawing frame to the drawing content of the drawing instruction of the twenty-first drawing frame. In the twenty-one memory space, the twenty-first drawing result is obtained, and the size of the twenty-first memory space is larger than the size of the default memory space, and the default memory space is the memory space provided by the electronic device system for storing image frames for display; When drawing the twenty-second drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-second drawing frame to the twenty-second memory space, and obtains the twenty-second drawing result, the size of the twenty-second memory space larger than the size of the default memory space; the electronic device generates the twenty-third drawing result in the twenty-third memory space according to the twenty-first drawing result and the twenty-second drawing result, and the size of the twenty-third memory space is larger than the size of the default memory space; the electronic device cuts the twenty-third drawing result to be the same size as the default memory space to obtain the twenty-third predicted frame.

With reference to the fourteenth aspect, in a possible implementation manner, the size of the twenty-first memory space is larger than the size of the default memory space, specifically including: the first size of the twenty-first memory space is the third size of the default memory space K1 times the size, the second size of the twenty-first memory space is K2 times the fourth size of the default memory space, and K1 and K2 are greater than 1.

The size of the twenty-second memory space is larger than the size of the default memory space, specifically including: the fifth size of the twenty-second memory space is K1 times the third size of the default memory space, and the fifth size of the twenty-second memory space is K2 times the fourth dimension of the default memory space.

The size of the twenty-third memory space is larger than the size of the default memory space, specifically including: the seventh size of the twenty-third memory space is K1 times the third size of the default memory space, and the eighth size of the twenty-third memory space is K2 times the fourth dimension of the default memory space.

Here, the first size of the twenty-first memory space may be the width of the twenty-first memory space, and the second size of the twenty-first memory space may be the height of the twenty-first memory space. The third size of the default memory space may be the width of the default memory space, and the fourth size of the default memory space may be the height of the default memory space. The fifth size of the twenty-second memory space may be the width of the twenty-second memory space, and the sixth size of the twenty-second memory space may be the height of the twenty-second memory space. The seventh size of the twenty-third memory space may be the width of the twenty-third memory space, and the eighth size of the twenty-third memory space may be the height of the twenty-third memory space. In this way, the electronic device can enlarge the width and height of the twenty-first memory space according to different sizes. The electronic device may enlarge the width and height of the twenty-second memory space according to different sizes. The electronic device may enlarge the width and height of the twenty-third memory space according to different sizes.

With reference to the fourteenth aspect, in a possible implementation manner, when drawing the twenty-first drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-first drawing frame into the twenty-first memory space, Obtaining the twenty-first drawing result specifically includes: when drawing the twenty-first drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-first drawing frame into the first drawing range of the twenty-first memory space , to obtain the twenty-first drawing result; the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is greater than the size of the default memory space.

With reference to the fourteenth aspect, in a possible implementation manner, the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is greater than the size of the default memory space, specifically including : The ninth size of the first drawing range is K3 times the third size of the default memory space, the tenth size of the first drawing range is K4 times the fourth size of the default memory space, K3 is greater than 1, and less than or equal to K1 , K4 is greater than 1, and less than or equal to K2.

The ninth size of the first drawing range may be the width of the first drawing range, and the tenth size of the first drawing range may be the height of the first drawing range.

With reference to the fourteenth aspect, in a possible implementation manner, K3 is equal to K1, K4 is equal to K1, and K1, K2, K3, and K4 are fixed values of the system configuration of the electronic device. The electronic equipment can configure K1, K2, K3, K4 according to the experience value. The electronic device directly configures the fixed value to reduce the amount of calculation.

With reference to the fourteenth aspect, in a possible implementation manner, K3 and K4 are determined by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, K3 and K4 set by the electronic device can be determined according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the electronic device is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the fourteenth aspect, in a possible implementation manner, when drawing the twenty-second drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-second drawing frame to the twenty-second memory space, and obtains: The twenty-second drawing result specifically includes: when drawing the twenty-second drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-second drawing frame into the second drawing range of the twenty-second memory space, The twenty-second drawing result is obtained; the size of the second drawing range is less than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space.

With reference to the fourteenth aspect, in a possible implementation manner, the size of the second drawing range is smaller than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space, specifically including: The eleventh size of the second drawing range is K5 times the third size of the default memory space, the twelfth size of the second drawing range is K6 times the fourth size of the default memory space, and K5 is greater than 1 and less than or equal to K1 , K6 is greater than 1, and less than or equal to K2.

The eleventh size of the second drawing range may be the width of the second drawing range, and the twelfth size of the second drawing range may be the height of the second drawing range.

With reference to the fourteenth aspect, in a possible implementation manner, K5 and K6 are fixed values of the system configuration of the electronic device. The electronic device directly configures the fixed value to reduce the amount of calculation.

With reference to the fourteenth aspect, in a possible implementation manner, K5 and K6 are determined by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, K5 and K6 set by the electronic device can be determined according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the electronic device is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the fourteenth aspect, in a possible implementation manner, the electronic device generates the twenty-third drawing result in the twenty-third memory space according to the twenty-first drawing result and the twenty-second drawing result, and the second The size of the thirteenth memory space is larger than the size of the default memory space, which specifically includes: the electronic device determines the first motion vector of the twenty-second drawing result according to the twenty-first drawing result and the twenty-second drawing result; The twenty-two rendering results and the first motion vector generate the twenty-third rendering result in the twenty-third memory space. In this way, the electronic device can predict the twenty-third rendering result of the twenty-third predicted frame according to the twenty-first rendering frame and the twenty-second rendering frame.

With reference to the fourteenth aspect, in a possible implementation manner, the electronic device determines the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result, which specifically includes: the electronic device Divide the twenty-second rendering result into Q pixel blocks, and the electronic device takes out the first pixel block from the Q pixel blocks in the twenty-second rendering result; the electronic device determines in the twenty-first rendering result that it is the same as the first pixel. The second pixel block matched by the block; the electronic device obtains the motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block; the electronic device determines the first pixel block of the twenty-second rendering result according to the motion vector of the first pixel block. a motion vector. According to the steps in this implementation manner, the electronic device can determine the motion vectors of all pixel blocks in the Q pixel blocks of the twenty-second rendering result. Each pixel block includes f*f (for example, 16*16) pixel points.

In the above implementation manner, the electronic device divides the twenty-second rendering result into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel in the twenty-second rendering result. This reduces the amount of computation, which in turn reduces the power consumption of electronic devices.

With reference to the fourteenth aspect, in a possible implementation manner, the electronic device determines, in the twenty-first drawing result, a second pixel block that matches the first pixel block, which specifically includes: In the twenty-first drawing result, a plurality of candidate pixel blocks are determined for the first pixel point of the The difference between the color values of the first pixel block determines a second pixel block that matches the first pixel block, and the second pixel block is the candidate with the smallest difference in color value from the first pixel block among the multiple candidate pixel blocks. pixel block.

In this way, the electronic device can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the fourteenth aspect, in a possible implementation manner, the electronic device generates the twenty-third drawing result in the twenty-third memory space according to the twenty-second drawing result and the first motion vector, which specifically includes: electronic The device determines the motion vector of the twenty-third rendering result according to the first motion vector, and generates the twenty-third rendering result according to the twenty-second rendering result and the motion vector of the twenty-third rendering result. The motion vector of the twenty-third rendering result is G times the first motion vector, and G is greater than 0 and less than 1.

With reference to the fourteenth aspect, in a possible implementation, G is equal to 0.5. In this way, the objects in each image frame move at a constant speed, which is convenient for the electronic device to calculate, and can also make the user experience better when watching the video.

With reference to the fourteenth aspect, in a possible implementation manner, the electronic device generates the twenty-third drawing result in the twenty-third memory space according to the twenty-second drawing result and the first motion vector, which specifically includes: electronic The device generates the twenty-third drawing result within the third drawing range of the twenty-third memory space according to the twenty-second drawing result and the first motion vector; the size of the third drawing range is less than or equal to the twenty-third memory space The size of the third drawing range is larger than the size of the default memory space.

With reference to the fourteenth aspect, in a possible implementation manner, the size of the third drawing range is smaller than or equal to the size of the twenty-third memory space, and the size of the third drawing range is greater than the size of the twenty-third memory space, specifically Including: the thirteenth size of the third drawing range is K7 times the third size of the default memory space, the fourteenth size of the third drawing range is K8 times the fourth size of the default memory space, and K7 is greater than 1 and less than or equal to K1, K8 is greater than 1, and less than or equal to K2.

The thirteenth size of the third drawing range may be the width of the third drawing range, and the fourteenth size of the third drawing range may be the height of the third drawing range.

With reference to the fourteenth aspect, in a possible implementation manner, when drawing the twenty-first drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-first drawing frame into the twenty-first memory space, Before obtaining the twenty-first drawing result, the method may further include: the electronic device creates a twenty-first memory space, a twenty-second memory space, and a twenty-third memory space, and the twenty-first memory space can be used to store the first memory space. The twenty-first drawing result of the twenty-first drawing frame, the twenty-second memory space can be used to store the twenty-second drawing result of the twenty-second drawing frame, and the twenty-third memory space can be used to store the twenty-second drawing result. Twenty-third rendering result of the three predicted frames.

With reference to the fourteenth aspect, in a possible implementation manner, when drawing the twenty-first drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-first drawing frame into the twenty-first memory space, After the twenty-first drawing result is obtained, the method further includes: the electronic device cuts the twenty-first drawing result into the same size as the default memory space to obtain the twenty-first drawing frame.

With reference to the fourteenth aspect, in a possible implementation manner, when drawing the twenty-second drawing frame, the electronic device draws the drawing content of the drawing instruction of the twenty-second drawing frame to the twenty-second memory space, and obtains: After the twenty-second rendering result, the method may further include: cutting the twenty-second rendering result into the same size as the default memory space by the electronic device to obtain the twenty-second rendering frame.

A fifteenth aspect provides an electronic device, comprising: one or more processors, a CPU, a graphics processor GPU, a memory, and a display screen; the memory is coupled to the one or more processors; the CPU is coupled to the GPU; wherein:

The memory can be used to store computer program codes, and the computer program codes include computer instructions; the CPU can be used to instruct the GPU to perform the drawing when drawing the twenty-first drawing frame, and to instruct the GPU to perform the drawing when drawing the second drawing;

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may be used to draw the drawing content of the drawing instruction of the twenty-first drawing frame to the twenty-first memory space when drawing the twenty-first drawing frame , obtain the twenty-first drawing result, the size of the twenty-first memory space is larger than the size of the default memory space, and the default memory space is the memory space provided by the electronic device system to store the image frames for display; When the second frame is drawn, the drawing content of the drawing instruction of the twenty-second drawing frame is drawn to the twenty-second memory space, and the twenty-second drawing result is obtained, and the size of the twenty-second memory space is larger than the size of the default memory space; According to the twenty-first drawing result and the twenty-second drawing result, the twenty-third drawing result is generated in the twenty-third memory space, and the size of the twenty-third memory space is larger than the size of the default memory space; The twenty-third rendering result is cut to the same size as the default memory space, and the twenty-third predicted frame is obtained.

With reference to the fifteenth aspect, in a possible implementation manner, the size of the twenty-first memory space is larger than the size of the default memory space, specifically including: the first size of the twenty-first memory space is the third size of the default memory space K1 times the size, the second size of the twenty-first memory space is K2 times the fourth size of the default memory space, and K1 and K2 are greater than 1.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may also be used to: when drawing the twenty-first drawing frame, draw the drawing content of the drawing instruction of the twenty-first drawing frame to the twenty-first drawing frame In the first drawing range of the memory space, the twenty-first drawing result is obtained; the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is larger than the size of the default memory space.

With reference to the fifteenth aspect, in a possible implementation manner, the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is greater than the size of the default memory space, specifically including : The ninth size of the first drawing range is K3 times the third size of the default memory space, the tenth size of the first drawing range is K4 times the fourth size of the default memory space, K3 is greater than 1, and less than or equal to K1 , K4 is greater than 1, and less than or equal to K2.

With reference to the fifteenth aspect, in a possible implementation manner, K3 is equal to K1, K4 is equal to K1, and K1, K2, K3, and K4 are fixed values of the system configuration of the electronic device. The electronic equipment can configure K1, K2, K3, K4 according to the experience value. The electronic device directly configures the fixed value to reduce the amount of calculation.

With reference to the fifteenth aspect, in a possible implementation manner, K3 and K4 are determined by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, K3 and K4 set by the electronic device can be determined according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the electronic device is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may also be used to: when drawing the twenty-second drawing frame, draw the drawing content of the drawing instruction of the twenty-second drawing frame to the twenty-second drawing frame In the second drawing range of the memory space, the twenty-second drawing result is obtained; the size of the second drawing range is smaller than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space.

With reference to the fifteenth aspect, in a possible implementation manner, the size of the second drawing range is less than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space, specifically including: The eleventh size of the second drawing range is K5 times the third size of the default memory space, the twelfth size of the second drawing range is K6 times the fourth size of the default memory space, and K5 is greater than 1 and less than or equal to K1 , K6 is greater than 1, and less than or equal to K2.

With reference to the fifteenth aspect, in a possible implementation manner, K5 and K6 are fixed values of the system configuration of the electronic device. The electronic device directly configures the fixed value to reduce the amount of calculation.

With reference to the fifteenth aspect, in a possible implementation manner, K5 and K6 are determined by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, K5 and K6 set by the electronic device can be determined according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the electronic device is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may be used to: determine the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result; The twenty-two rendering results and the first motion vector generate the twenty-third rendering result in the twenty-third memory space. In this way, the GPU can predict the twenty-third rendering result of the twenty-third predicted frame according to the twenty-first rendering frame and the twenty-second rendering frame.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may be used to: the electronic device divides the twenty-second rendering result into Q pixel blocks, and extracts the 22nd rendering result from the Q pixel blocks of the twenty-second rendering result. A pixel block determines a second pixel block matching the first pixel block in the twenty-first rendering result; obtains the motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block; The motion vector of the block determines the first motion vector of the twenty-second rendering result. According to the steps in this implementation manner, the GPU may determine motion vectors of all pixel blocks in the Q pixel blocks of the twenty-second rendering result. Each pixel block includes f*f (for example, 16*16) pixel points.

In the above implementation manner, the GPU divides the twenty-second rendering result into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel in the twenty-second rendering result. This reduces the amount of computation and thus reduces the power consumption of GPUs in electronic devices.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may be used to: determine a plurality of candidate pixel blocks in the twenty-first drawing result by using the first pixel point in the first pixel block; The difference between the color values of the multiple candidate pixel blocks and the first pixel block; the second pixel block matching the first pixel block is determined according to the difference between the color values of the first pixel block of the multiple candidate pixel blocks, and the second pixel block The block is a candidate pixel block with the smallest difference between the color values of the first pixel block and the plurality of candidate pixel blocks.

In this way, the GPU in the electronic device can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may be used to: determine the motion vector of the twenty-third rendering result according to the first motion vector, and determine the motion vector of the twenty-third rendering result according to the twenty-second rendering result and the twenty-third rendering result The motion vector of generates the twenty-third rendering result. The motion vector of the twenty-third rendering result is G times the first motion vector, and G is greater than 0 and less than 1.

In conjunction with the fifteenth aspect, in a possible implementation, G is equal to 0.5. In this way, the objects in each image frame move at a constant speed, which is convenient for the calculation of the GPU in the electronic device, and can also make the user experience better when watching the video.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may also be used to: generate the second drawing within the third drawing range of the twenty-third memory space according to the twenty-second drawing result and the first motion vector Thirteen drawing results; the size of the third drawing range is less than or equal to the size of the twenty-third memory space, and the size of the third drawing range is greater than the size of the default memory space.

With reference to the fifteenth aspect, in a possible implementation manner, the size of the third drawing range is less than or equal to the size of the twenty-third memory space, and the size of the third drawing range is greater than the size of the twenty-third memory space, specifically Including: the thirteenth size of the third drawing range is K7 times the third size of the default memory space, the fourteenth size of the third drawing range is K8 times the fourth size of the default memory space, and K7 is greater than 1 and less than or equal to K1, K8 is greater than 1, and less than or equal to K2.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU can be used to: create the twenty-first memory space, the twenty-second memory space, the twenty-third memory space, and the twenty-first memory space can be used For storing the twenty-first drawing result of the twenty-first drawing frame, the twenty-second memory space can be used to store the twenty-second drawing result of the twenty-second drawing frame, and the twenty-third memory space can be used to store the twenty-second drawing result of the twenty-second drawing frame. The twenty-third rendering result of the twenty-third predicted frame.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may also be used for: the electronic device cuts the twenty-first drawing result to the same size as the default memory space to obtain the twenty-first drawing frame.

With reference to the fifteenth aspect, in a possible implementation manner, the GPU may also be used for: the electronic device cuts the twenty-second drawing result into the same size as the default memory space to obtain the twenty-second drawing frame.

A sixteenth aspect provides an image frame prediction apparatus, the apparatus may include a first drawing unit, a second drawing unit, and a generating unit; wherein:

The first drawing unit may be configured to draw the drawing instruction of the twenty-first drawing frame according to the first drawing range when drawing the twenty-first drawing frame of the first application, to obtain the twenty-first drawing result, the first The size of the drawing range is greater than the size of the twenty-first drawing frame of the first application;

The second drawing unit may be configured to draw the drawing instruction of the twenty-second drawing frame according to the second drawing range when drawing the twenty-second drawing frame of the first application, to obtain the twenty-second drawing result, the twenty-second drawing result. The size of the memory space is greater than the size of the twenty-second drawing frame, wherein the size of the twenty-first drawing frame is the same as the size of the twenty-second drawing frame;

The generating unit may be configured to predict and generate the twenty-third predicted frame of the first application according to the twenty-first rendering result and the twenty-second rendering result, wherein the size of the twenty-third predicted frame is the same as that of the twenty-first rendering frame. are the same size.

With reference to the sixteenth aspect, in a possible implementation manner, the first drawing unit may also be used to draw the drawing content of the drawing instruction of the twenty-first drawing frame to the second drawing when drawing the twenty-first drawing frame. In the eleventh memory space, the twenty-first drawing result is obtained, and the size of the twenty-first memory space is larger than the size of the default memory space, which is the memory space provided by the electronic device system for storing image frames for display.

With reference to the sixteenth aspect, in a possible implementation manner, the second drawing unit may also be used to draw the drawing content of the drawing instruction of the twenty-second drawing frame to the electronic device when drawing the twenty-second drawing frame. For the twenty-second memory space, the twenty-second drawing result is obtained, and the size of the twenty-second memory space is larger than the size of the default memory space.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may be further configured to generate the twentieth drawing in the twenty-third memory space according to the twenty-first drawing result and the twenty-second drawing result. Three drawing results, the size of the twenty-third memory space is larger than the size of the default memory space.

With reference to the sixteenth aspect, in a possible implementation manner, the image frame prediction apparatus may further include a clipping unit, and the clipping unit may be configured to clip the twenty-third rendering result to the size of the default memory space In the same way, the twenty-third predicted frame is obtained.

In this way, the image frame prediction apparatus can obtain the predicted frame. The frame rate of the image frame prediction apparatus can be increased without increasing the number of rendering frames. In this way, while saving the power consumption of the image frame prediction apparatus, the smoothness of the video interface displayed by the image frame prediction apparatus can be improved. Further, in the prediction frame predicted by the image frame prediction device, there may be rendering content that is not in the twenty-first rendering frame and the twenty-second rendering frame displayed by the image frame prediction device. Therefore, the drawing content in the predicted frame predicted by the image frame prediction device is closer to the shooting content in the shooting field of view of the camera. Therefore, the image frame predicted by the image frame prediction device can be more accurate.

With reference to the sixteenth aspect, in a possible implementation manner, the size of the twenty-first memory space is larger than the size of the default memory space, specifically including: the first size of the twenty-first memory space is the third size of the default memory space K1 times the size, the second size of the twenty-first memory space is K2 times the fourth size of the default memory space, and K1 and K2 are greater than 1.

Here, the first size of the twenty-first memory space may be the width of the twenty-first memory space, and the second size of the twenty-first memory space may be the height of the twenty-first memory space. The third size of the default memory space may be the width of the default memory space, and the fourth size of the default memory space may be the height of the default memory space. The fifth size of the twenty-second memory space may be the width of the twenty-second memory space, and the sixth size of the twenty-second memory space may be the height of the twenty-second memory space. The seventh size of the twenty-third memory space may be the width of the twenty-third memory space, and the eighth size of the twenty-third memory space may be the height of the twenty-third memory space. In this way, the image frame prediction apparatus can enlarge the width and height of the twenty-first memory space according to different sizes. The image frame prediction apparatus may enlarge the width and height of the twenty-second memory space according to different sizes. The image frame prediction apparatus may enlarge the width and height of the twenty-third memory space according to different sizes.

With reference to the sixteenth aspect, in a possible implementation manner, the first drawing unit may be further configured to: when drawing the twenty-first drawing frame, draw the drawing content of the drawing instruction of the twenty-first drawing frame to the drawing content of the drawing instruction of the twenty-first drawing frame. In the first drawing range of the twenty-first memory space, the twenty-first drawing result is obtained; the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is greater than the size of the default memory space .

With reference to the sixteenth aspect, in a possible implementation manner, the size of the first drawing range is less than or equal to the size of the twenty-first memory space, and the size of the first drawing range is greater than the size of the default memory space, specifically including : The ninth size of the first drawing range is K3 times the third size of the default memory space, the tenth size of the first drawing range is K4 times the fourth size of the default memory space, K3 is greater than 1, and less than or equal to K1 , K4 is greater than 1, and less than or equal to K2.

With reference to the sixteenth aspect, in a possible implementation manner, K3 is equal to K1, K4 is equal to K1, and K1, K2, K3, and K4 are fixed values of the system configuration of the image frame prediction apparatus. The image frame predicting apparatus may configure K1, K2, K3, and K4 according to empirical values. Directly configuring the fixed value in the image frame prediction apparatus can reduce the amount of calculation.

With reference to the sixteenth aspect, in a possible implementation manner, K3 and K4 are determined by the electronic device according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, K3 and K4 set by the image frame prediction apparatus can be determined according to the drawing parameters included in the drawing instruction of the twenty-first drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the image frame prediction apparatus is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the sixteenth aspect, in a possible implementation manner, the second drawing unit may be further configured to: when drawing the twenty-second drawing frame, draw the drawing content of the drawing instruction of the twenty-second drawing frame to the drawing content of the drawing instruction of the twenty-second drawing frame. In the second drawing range of the twenty-two memory space, the twenty-second drawing result is obtained; the size of the second drawing range is less than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space .

With reference to the sixteenth aspect, in a possible implementation manner, the size of the second drawing range is less than or equal to the size of the twenty-second memory space, and the size of the second drawing range is greater than the size of the default memory space, specifically including: The eleventh size of the second drawing range is K5 times the third size of the default memory space, the twelfth size of the second drawing range is K6 times the fourth size of the default memory space, and K5 is greater than 1 and less than or equal to K1 , K6 is greater than 1, and less than or equal to K2.

With reference to the sixteenth aspect, in a possible implementation manner, K5 and K6 are fixed values of the system configuration of the image frame prediction apparatus. Directly configuring the fixed value in the image frame prediction apparatus can reduce the amount of calculation.

With reference to the sixteenth aspect, in a possible implementation manner, K5 and K6 are determined by the image frame prediction apparatus according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, K5 and K6 set by the image frame prediction apparatus can be determined according to the drawing parameters included in the drawing instruction of the twenty-second drawing frame. In this way, the magnification of the drawing range of different drawing frames can be different. In this way, the magnification of the drawing range by the image frame prediction apparatus is more consistent with the drawing content in the drawing instruction of the drawing frame.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may also be configured to: determine the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result; A twenty-third drawing result is generated in the twenty-third memory space according to the twenty-second drawing result and the first motion vector. In this way, the generating unit in the image frame prediction apparatus can predict the twenty-third rendering result of the twenty-third predicted frame according to the twenty-first rendering frame and the twenty-second rendering frame.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may also be used to: divide the twenty-second rendering result into Q pixel blocks, and extract the first pixel from the Q pixel blocks of the twenty-second rendering result. a pixel block; determine the second pixel block matching the first pixel block in the twenty-first rendering result; obtain the motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block; according to the first pixel block The motion vector of the pixel block determines the first motion vector of the twenty-second rendering result. According to the steps in this implementation manner, the image frame prediction apparatus may determine motion vectors of all pixel blocks in the Q pixel blocks of the twenty-second rendering result. Each pixel block includes f*f (for example, 16*16) pixel points.

In the above implementation manner, the image frame prediction apparatus divides the twenty-second rendering result into blocks to calculate the motion vector, and does not need to calculate the motion vector of each pixel in the twenty-second rendering result. This reduces the amount of computation, which in turn reduces the power consumption of electronic devices.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may be further configured to: determine multiple candidate pixel blocks in the twenty-first drawing result by using the first pixel point in the first pixel block; respectively; Calculate the difference between the color values of the multiple candidate pixel blocks and the first pixel block; determine the second pixel block that matches the first pixel block according to the difference between the color values of the first pixel block of the multiple candidate pixel blocks, and the first pixel block The two-pixel block is a candidate pixel block with the smallest difference between the color values of the first pixel block and the plurality of candidate pixel blocks.

In this way, the image frame prediction apparatus can more accurately find the matching pixel block of each pixel block, so that the motion vector of each pixel block can be calculated more accurately.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may be further configured to: determine the motion vector of the twenty-third rendering result according to the first motion vector, and determine the motion vector of the twenty-third rendering result according to the first motion vector, and The motion vector of the rendering result generates the twenty-third rendering result. The motion vector of the twenty-third rendering result is G times the first motion vector, and G is greater than 0 and less than 1.

In conjunction with the sixteenth aspect, in a possible implementation, G is equal to 0.5. In this way, the objects in each image frame move at a constant speed, which is convenient for the image frame prediction device to calculate, and can also make the user experience better when watching the video.

With reference to the sixteenth aspect, in a possible implementation manner, the generating unit may be further configured to: generate a third drawing range within the third drawing range of the twenty-third memory space according to the twenty-second drawing result and the first motion vector. Twenty-three drawing results; the size of the third drawing range is less than or equal to the size of the twenty-third memory space, and the size of the third drawing range is greater than the size of the twenty-third memory space.

With reference to the sixteenth aspect, in a possible implementation manner, the size of the third drawing range is less than or equal to the size of the twenty-third memory space, and the size of the third drawing range is greater than the size of the twenty-third memory space, specifically Including: the thirteenth size of the third drawing range is K7 times the third size of the default memory space, the fourteenth size of the third drawing range is K8 times the fourth size of the default memory space, and K7 is greater than 1 and less than or equal to K1, K8 is greater than 1, and less than or equal to K2.

With reference to the sixteenth aspect, in a possible implementation manner, the image frame prediction apparatus may further include a creation unit, and the creation unit may be configured to: create a twenty-first memory space, a twenty-second memory space, and a second memory space. Thirteen memory spaces, the twenty-first memory space can be used to store the twenty-first drawing result of the twenty-first drawing frame, and the twenty-second memory space can be used to store the twenty-second drawing frame of the twenty-second drawing frame. For the drawing result, the twenty-third memory space can be used to store the twenty-third drawing result of the twenty-third predicted frame.

With reference to the sixteenth aspect, in a possible implementation manner, the clipping unit may also be used to: clip the twenty-first drawing result to be the same size as the default memory space to obtain the twenty-first drawing frame.

With reference to the sixteenth aspect, in a possible implementation manner, the clipping unit may also be used to: clip the twenty-second drawing result to be the same size as the default memory space to obtain the twenty-second drawing frame.

A seventeenth aspect provides an electronic device, comprising: one or more processors; one or more memories; the one or more memories stores one or more computer programs, and the one or more computer programs include instructions, when The instructions, when executed by one or more processors, cause the electronic device to perform any one of the first, fifth, seventh, ninth, eleventh, thirteenth, and fourteenth aspects possible implementations.

In an eighteenth aspect, an embodiment of the present application provides a chip, the chip is applied to an electronic device, the chip includes one or more processors, and the processor is configured to invoke computer instructions to cause the electronic device to execute the first aspect, Any possible implementation manner of the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect, the thirteenth aspect and the fourteenth aspect.

A nineteenth aspect provides a computer product that, when the computer program product runs on a computer, causes the computer to perform the first aspect, the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect, and the first aspect. Any possible implementation manner of the thirteenth aspect and the fourteenth aspect.

In a twentieth aspect, a computer-readable storage medium is provided, comprising instructions, characterized in that, when the above-mentioned instructions are executed on an electronic device, the electronic device executes the first aspect, the fifth aspect, the seventh aspect, the first aspect, and the third aspect. Any possible implementation manner of the ninth aspect, the eleventh aspect, the thirteenth aspect and the fourteenth aspect.

It can be understood that the electronic device provided in the seventeenth aspect, the chip provided in the eighteenth aspect, the computer program product provided in the nineteenth aspect, and the computer storage medium provided in the twentieth aspect are all used to execute the embodiments of the present application. provided method.

Description of drawings

FIG. 1 is a schematic diagram of a user interface 100 of a tablet computer 10 provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a static object of the Nth drawing frame in the user interface 100 provided by the embodiment of the present application;

3 is a schematic diagram of a dynamic object of the Nth drawing frame in the user interface 100 provided by the embodiment of the present application;

4A is a schematic diagram of a drawing frame 300A provided by an embodiment of the present application;

4B is a schematic diagram of a prediction frame 300B provided by an embodiment of the present application;

FIG. 5A is a schematic diagram of a drawing frame 500 provided by an embodiment of the present application;

FIG. 5B is a schematic diagram of a partial pixel block of the drawing frame 500 provided by an embodiment of the present application;

5C is a schematic diagram of a pixel block in a prediction frame predicted according to the pixel block in FIG. 5B provided by an embodiment of the present application;

6A is a logical block diagram of a method for image frame prediction provided by an embodiment of the present application;

6B is a flowchart of a method for predicting an image frame provided by an embodiment of the present application;

7A is a schematic diagram of an Nth drawing frame provided by an embodiment of the present application;

7B is a schematic diagram of depth attachment and color attachment of a dynamic object in the Nth drawing frame provided by an embodiment of the present application;

7C is a schematic diagram of depth attachment and color attachment of a static object in the Nth drawing frame provided by an embodiment of the present application;

8A is a schematic diagram of an N+2th drawing frame provided by an embodiment of the present application;

8B is a schematic diagram of a depth attachment and a color attachment of a dynamic object in the N+2th drawing frame provided by an embodiment of the present application;

8C is a schematic diagram of depth attachment and color attachment of a static object in the N+2th drawing frame provided by an embodiment of the present application;

9A-9C are schematic diagrams of the process of calculating the motion vector of the pixel block 902 in the N+2th frame through a diamond search provided by an embodiment of the present application;

10A is a schematic diagram of the color attachment of a dynamic object in the predicted N+3th predicted frame provided by an embodiment of the present application;

10B is a schematic diagram of the color attachment of the static object in the predicted N+3th predicted frame provided by an embodiment of the present application;

FIG. 10C is a schematic diagram of the predicted N+3 th prediction frame provided by an embodiment of the present application;

11 is a logical block diagram of 90fps frame insertion provided by an embodiment of the present application;

12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

13 is a schematic diagram of a system framework of an electronic device provided by an embodiment of the present application;

FIG. 14 is a schematic diagram of a user interface of a tablet computer 10 provided by an embodiment of the present application;

15 is a schematic diagram of the Wth drawing frame provided by an embodiment of the present application;

16 is a schematic diagram of the W+2th drawing frame provided by an embodiment of the present application;

17 is a schematic flowchart of a method for predicting an image frame provided by an embodiment of the present application;

18 is a schematic diagram of an electronic device acquiring object attributes provided by an embodiment of the present application;

19A is a schematic diagram of the conversion of a moving object in different coordinate systems provided by an embodiment of the present application;

19B is a schematic diagram of a static object provided in an embodiment of the present application in different coordinate systems;

20A-20C are schematic diagrams of a template image of a GPU drawing the Wth drawing frame object provided by an embodiment of the present application;

21A-21C are schematic diagrams of GPU rendering a template image of the W+2th rendering frame object provided by an embodiment of the present application;

22A-22B are schematic diagrams of a process of calculating a motion vector of a moving object according to an embodiment of the present application;

FIG. 23 is a schematic diagram of related modules for image frame prediction provided by an embodiment of the present application;

24 is a flowchart of a frame prediction method disclosed in an embodiment of the present application;

FIG. 25 is a schematic diagram of obtaining a reference frame disclosed by an embodiment of the present application;

FIG. 26 is another schematic diagram of acquiring a reference frame disclosed by an embodiment of the present application;

FIG. 27 is a schematic diagram of determining a target reference frame disclosed by an embodiment of the present application;

FIG. 28 is a schematic diagram of dividing a target reference frame disclosed by an embodiment of the present application;

29 is a schematic flowchart of calculating a predicted motion vector of a block disclosed in an embodiment of the present application;

30 is a schematic diagram of an acquisition block from a target reference frame to a matching frame disclosed by an embodiment of the present application;

31A is a schematic diagram of determining a predicted motion vector of a block according to an embodiment of the present application;

31B is a schematic diagram of another method of determining a predicted motion vector of a block disclosed in an embodiment of the present application;

32A is a schematic diagram of determining blocks around a vertex according to an embodiment of the present application;

32B is a schematic diagram of determining a predicted motion vector of a vertex disclosed in an embodiment of the present application;

33 is a schematic diagram of determining vertex coordinates disclosed in an embodiment of the present application;

34 is a flowchart of determining coordinates of pixels in a block in a predicted frame disclosed in an embodiment of the present application;

35A is a schematic diagram of a homography transformation formula corresponding to an acquisition block disclosed by an embodiment of the present application;

35B is a schematic diagram of determining coordinates of pixels in a predicted frame disclosed by an embodiment of the present application;

35C is a schematic diagram of a prediction frame disclosed in an embodiment of the present application;

FIG. 36 is a schematic diagram of another prediction frame disclosed in an embodiment of the present application;

37 is a schematic flowchart of a method for generating an image frame provided by an embodiment of the present application;

38 is a schematic diagram of a positional relationship provided by an embodiment of the present application;

39A is a schematic diagram of an image frame provided by an embodiment of the present application;

39B is a schematic diagram of dividing an image frame according to an embodiment of the present application;

40A to 40C are schematic diagrams of a group of determining matching blocks provided by an embodiment of the present application;

41A is a schematic flowchart of a method for determining position coordinates in a predicted image frame provided by an embodiment of the present application;

41B to 41C are schematic diagrams of a group of matching blocks provided by an embodiment of the present application;

41D to 41F are schematic diagrams of predicted position coordinates in a set of camera coordinate systems provided by an embodiment of the present application;

42 is a schematic diagram of determining a vertex of a prediction block provided by an embodiment of the present application;

43 is a schematic diagram of generating a prediction block provided by an embodiment of the present application;

44A to 44B are a set of schematic diagrams about prediction blocks provided by an embodiment of the present application;

45A to 45B are a set of schematic diagrams about prediction blocks provided by an embodiment of the present application;

46A to 46I are schematic diagrams of a group of generated image frames provided by an embodiment of the present application;

FIG. 47 is a software structural block diagram of a group of electronic devices 100 provided by an embodiment of the present application;

FIG. 48 is a schematic structural diagram of another electronic device 100 provided by an embodiment of the present application;

49A is a schematic diagram of creating a frame buffer object provided by an embodiment of the present application;

49B is a schematic diagram of drawing and displaying an original image frame provided by an embodiment of the present application;

49C is a schematic diagram of drawing and displaying a predicted image frame provided by an embodiment of the present application;

50A is a schematic diagram of a custom dynamic layer provided by an embodiment of the present application;

50B is a schematic diagram of a custom UI layer provided by an embodiment of the present application;

50C is a schematic diagram of an image frame provided by an embodiment of the present application;

50D is a schematic diagram of a drawing process provided by an embodiment of the present application;

FIG. 50E is a schematic diagram of generating an image frame provided by an embodiment of the present application;

51A is a schematic flowchart of a method for generating an image frame provided by an embodiment of the present application;

FIG. 51B and FIG. 51C are respectively schematic diagrams of a composite image frame provided by an embodiment of the present application;

FIG. 51D and FIG. 51E are effect diagrams that a drawing result may present according to an embodiment of the present application;

FIG. 51F is a schematic diagram of generating an image frame provided by an embodiment of the present application;

52 is a schematic diagram of a modular interaction provided by an embodiment of the present application;

53A is a schematic flowchart of a method for generating an image frame provided by an embodiment of the present application;

53B-53D are schematic diagrams of a group of prediction processes provided by an embodiment of the present application;

53E-53I are schematic diagrams of a group of drawing processes provided by an embodiment of the present application;

54A-54C are schematic diagrams of a group of user interfaces of the tablet computer 10 provided by the embodiments of the present application;

55A is a schematic diagram of a drawing frame A, a drawing frame B, and a predicted frame obtained according to the drawing frame A and the drawing frame B provided by an embodiment of the present application;

FIG. 55B is a schematic diagram of a camera shooting field of view provided by an embodiment of the present application;

56 is a flowchart of a method for predicting an image frame provided by an embodiment of the present application;

57 is a schematic diagram of a default memory space provided by an embodiment of the present application;

58 is a schematic diagram of a twenty-first memory space provided by an embodiment of the present application;

59 is a schematic diagram of a twenty-second memory space provided by an embodiment of the present application;

60 is a schematic diagram of a twenty-third memory space provided by an embodiment of the present application;

61 is a schematic diagram of the first drawing range, the twenty-first drawing result, and the U-th drawing frame in the twenty-first memory space provided by an embodiment of the present application;

62 is a schematic diagram of the second drawing range, the twenty-second drawing result, and the U+2th drawing frame in the twenty-second memory space provided by an embodiment of the present application;

63A-63C are schematic diagrams of the process of calculating the motion vector of the pixel block 6205 in the U+2th drawing frame by diamond search provided by an embodiment of the present application;

FIG. 64 is a schematic diagram of the third drawing range, the twenty-third drawing result, and the U+3-th predicted frame in the twenty-third memory space provided by the embodiment of the present application.

Detailed ways

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations of the present application. As used in the specification of this application and the appended claims, the singular expressions "a," "an," "the," "above," "the," and "the" are intended to also Plural expressions are included unless the context clearly dictates otherwise. It will also be understood that, as used in this application, the term "and/or" refers to and includes any and all possible combinations of one or more of the listed items.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as implying or implying relative importance or implying the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

Since the embodiments of the present application relate to the application of an image frame prediction method, for ease of understanding, related terms and concepts involved in the embodiments of the present application are first introduced below.

(1) Image frame

In this embodiment of the present application, each frame of image that the electronic device uses to display on the display screen is referred to as an image frame. In this embodiment of the present application, an image frame may be a frame of an image of an application, a drawing result drawn by an electronic device according to a drawing instruction of the application, or a prediction result predicted based on an existing drawing result. As shown in FIG. 1 , an electronic device (ie, tablet 10 ) displays a user interface 100 . At time T0, the Nth image frame is displayed in the user interface 10. The Nth image frame is a drawing frame. The timing diagram 101 in FIG. 1 shows the image frames that the electronic device can display from time T0 to time Tn.

(2) Drawing frame

In the embodiment of the present application, the image frame drawn by the electronic device according to the drawing instruction and drawing parameters of the application program when the application program is running is called a drawing frame. The drawing instructions and drawing parameters of the application may be automatically generated by the application graphics framework or the application engine, or may be written by the application developer. The drawing parameters corresponding to the drawing frame can contain one or more objects. The electronic device may draw one or more objects as corresponding elements in the drawing frame. For example, the user interface 100 shown in FIG. 1 may be a frame of drawing. Element 102 (shown in FIG. 2 ) and element 103 (shown in FIG. 3 ) in the user interface 100 are obtained after drawing and rendering the objects in the drawing parameters.

It is understandable that the drawing parameters of the drawing frame contain properties of multiple objects. The properties of the object can include one or more of the color value (eg, RGB value) of each pixel in the object, the depth value of each pixel in the object, the stencil buffer, the object's transition matrix, etc. . The CPU may send a drawing instruction to the GPU for instructing the GPU to perform drawing according to the drawing parameter. The GPU can draw an object according to a drawing instruction. The object drawn by the GPU according to the drawing instruction carrying the transition matrix can be called a moving object. Objects drawn by the GPU according to drawing instructions that do not carry a transition matrix can be called static objects. Optionally, the drawing instruction corresponding to the static object may carry a transition matrix, and the transition matrix is an all-zero matrix, that is, the elements of each row and each column in the matrix are 0.

In the embodiment of the present application, taking a game application as an example, a moving object can move in a game scene. The position of the first element in the user interface drawn and rendered by the moving object changes in two adjacent image frames. The user can see in the user interface that the position of the first element is moved. Static objects are static in the game scene, but the position of the static objects in different image frames will be different due to the change of the camera's shooting angle. The position of the second element in the user interface drawn and rendered by the static object can be changed in two adjacent image frames, and the magnitude of the position change is related to the shooting position and shooting angle of the camera. For example, the first element may be element 103 in FIG. 3 , which is a moving cart. The second element may be the element 102 in FIG. 2, and the element 102 is a static background. The position of the static background in different image frames can change due to the change of the camera's shooting angle.

(3) Predicted frame

In this embodiment of the present application, an image frame newly generated by the electronic device according to the existing drawing frame data is called a prediction frame. The drawing parameters of the predicted frame are obtained from the drawing parameters of the two drawing frames. For example, the electronic device may generate the first predicted frame from the first rendering frame and the second rendering frame. The first predicted frame is an image frame following the second drawing frame. That is, after the electronic device displays the second drawing frame, it displays the first predicted frame. The first drawing frame is an image frame before the second drawing frame (an image frame may exist between the first drawing frame and the second drawing frame). That is, the first drawing frame is displayed on the display screen of the electronic device before the second drawing frame. It can be understood that, if the Nth image frame is a drawing frame, it may be referred to as the Nth drawing frame in this embodiment of the present application. If the N th image frame is a predicted frame, it may be referred to as the N th predicted frame in this embodiment of the present application.

It can be understood that the objects contained in the drawing parameters of the predicted frame are the same as the objects contained in the drawing parameters of the drawing frame displayed one frame before the predicted frame. Here, the objects included in the drawing parameters of the image frame may be simply referred to as the objects included in the image frame. For example, the objects included in the drawing parameters of the predicted frame may be simply referred to as the objects included in the predicted frame.

Here, for the specific process of generating the predicted frame by the electronic device by drawing two frames, reference may be made to the following, which will not be repeated here.

(4) Image frame prediction

In this embodiment of the present application, the process of generating the first predicted frame by the electronic device through the first drawing frame and the second drawing frame is called image frame prediction.

(5) Color attachment

In this embodiment of the present application, a color attachment is a memory space used to store color data (eg, RGB values of pixels) of each pixel in the drawing result when the electronic device draws according to the drawing instruction. Color attachments are available as part of the FBO.

(6) Depth attachment

In this embodiment of the present application, a depth attachment is a memory space used to store the depth data of each pixel in the drawing result when the electronic device draws according to the drawing instruction. Color attachments are available as part of the FBO. Understandably, the smaller the depth value of the pixel in the depth attachment, the closer it is to the camera. When synthesizing an image frame, for two pixels with equal coordinate values in two color attachments, a pixel with a smaller depth value can cover another pixel with a larger depth value. That is, the color displayed by the final display pixel point is the color of the pixel point with the smaller depth value in the two color attachments.

In order to increase the frame rate and improve the smoothness of the video, the electronic device can insert the predicted frame between the drawn frames of the application. The electronic device may perform image frame prediction according to the drawn frame of the application to obtain the predicted frame. The moving speed and moving direction of the moving object and the static object in the image frame may be different, that is, the motion vector is different. In the process of image frame prediction, if a motion vector is calculated together with a moving object and a static object in the image frame, the calculated motion vector will be inaccurate. This can lead to distortions or holes in the predicted image frames.

FIG. 4A exemplarily shows a schematic diagram of drawing a frame. FIG. 4B exemplarily shows a schematic diagram of a predicted frame obtained according to the frame drawn in FIG. 4A . As shown in FIG. 4A , a drawing frame 300A may include a static object (static background) 301 and a moving object (moving car) 302 . The electronic device can predict the predicted frame 300B according to the rendering frame 300A and the motion vector of the rendering frame. A predicted frame 300B is shown in Figure 4B. The objects contained in the predicted frame 300B are the same as those in the rendering frame 300A. That is, a static object 301 and a moving object 302 may be included in the predicted frame 300B. Relative to the moving object 302 in the drawing frame 300A, the moving object 301 in the predicted frame 300B has moved forward, and the moving part is the part 303 shown in FIG. 4B . When the moving object moves in the moving direction shown in the figure, the region 303 where the moving object 302 moves in the predicted frame 300B may lack pixel information. Since this part of the pixels in the drawing frame 300A is covered by the moving object 301, if the electronic device predicts the predicted frame by taking the drawing frame as a whole, during the prediction process, after the moving object moves, the electronic device may not be able to obtain the pixel information of this part . Therefore, the region 303 in the predicted frame 300B may lack pixel information.

FIG. 5A exemplarily shows a drawing frame 500 . The drawing frame 500 may include a static background and a moving cart 521 . The electronic device may divide the drawing frame 500 into Q pixel blocks. The rendering frame 400 in FIG. 5A may be divided into Q pixel blocks. The size of Q is related to the resolution of the display and the tiles in the GPU. Tile size is f*f. Here, the description is made by taking N equal to 20 as an example. The drawing frame 500 can be divided into a total of 20 pixel blocks from a pixel block 501 to a pixel block 520 . The pixel block 513 includes both the moving car 512 and the static background. Pixel block 501-pixel block 512, pixel block 514-pixel block 520 only include static background. The static background in the drawing frame 500 is different from the moving car of the moving car. However, when the motion vector is calculated by taking the pixel block as a whole, the calculated motion vector of the static background in the pixel block is the same as the motion vector of the moving car. In this way, the static background in the predicted predicted frame may not be a complete one, and there may be empty areas (ie, no pixel areas) in the static background. In the following, the pixel block 507-pixel block 509 and the pixel block 512-pixel block 514 in the drawing frame 500 are taken as examples for description.

FIG. 5A exemplarily shows pixel block 507-pixel block 509, pixel block 512-pixel block 514 in the rendering frame 500 in FIG. 5A. Pixel block 507-pixel block 509, and pixel block 512 and pixel block 514 only include a static background. The motion vectors of pixel blocks such as pixel block 507-pixel block 509, pixel block 512 and pixel block 514 calculated by the electronic device may be the same, that is, the motion vector of the static background. The pixel block 513 includes a moving cart 521 and a static background. The motion vector of the pixel block 513 calculated by the electronic device may be the motion vector of the motion cart 521 . In this way, the motion vector of the pixel block 513 calculated by the electronic device is different from the motion vector of the pixel block 507 - the pixel block 509 and the pixel block 512 and the pixel block 514 . The electronic device may predict the pixel block 507-pixel 509 and the pixel block 512-pixel block 514 in the predicted frame according to the pixel block 507-pixel 509 and the pixel block 512-pixel block 514.

FIG. 5C shows pixel blocks 507-509 and pixel blocks 512-pixel blocks 514 in the predicted frame predicted by the electronic device according to the pixel blocks in FIG. 5B. Since the motion vector of the pixel block in FIG. 5B is different from the motion vector of other pixel blocks, the predicted displacement of the pixel block 513 in FIG. 5C is different from that of other pixel blocks, resulting in that the pixel block 513 and the pixel block 512 are not connected, A hole region 522 appears between the pixel block 512 and the pixel block 513 . The displacement of the pixel block 513 is greater than the displacement of the pixel block 514 , causing the pixel block 513 to cover the pixel block 514 .

In order to save the power consumption of the electronic device while improving the fluency of the application video interface displayed by the electronic device, an embodiment of the present application provides a method for image frame prediction. The method may include: first, when drawing the first drawing frame, the electronic device writes the color data of the first drawing object into the first color attachment, writes the color data of the second drawing object into the second color attachment, and the first drawing The drawing instruction of the object indicates that the spatial information of the first drawing object changes, and the drawing instruction of the second drawing object does not indicate that the spatial information of the second drawing object changes. Then, when drawing the second drawing frame, the electronic device writes the color data of the third drawing object into the third color attachment, writes the color data of the fourth drawing object into the fourth color attachment, and writes the color data of the fourth drawing object into the drawing instruction of the third drawing object. Indicates that the spatial information of the third drawing object changes, and the drawing instruction of the fourth drawing object does not indicate that the spatial information of the fourth drawing object changes. Then, the electronic device predicts the fifth color accessory of the first predicted frame according to the first color accessory and the third color accessory, and predicts the sixth color accessory of the first predicted frame according to the second color accessory and the fourth color accessory. Finally, the electronic device synthesizes the fifth color attachment and the sixth color attachment into the first predicted frame.

The drawing instruction of the first drawing object indicates that the spatial information of the first drawing object changes, that is, the first drawing object is a moving object. The drawing instruction of the second drawing object does not indicate that the spatial information of the second drawing object changes, that is, the second drawing object is a static object. The electronic device writes the color data of the moving object in the first drawing frame into the first color attachment, and writes the color data of the static object in the first drawing frame into the second color attachment. In this way, the electronic device stores the color data of the dynamic object and the color data of the static object in the first drawing frame in different color attachments respectively. Similarly, when the second drawing frame is drawn, the drawing instruction of the third drawing object in the second drawing frame indicates that the spatial information of the third drawing object changes, that is, the third drawing object is a moving object. The drawing instruction of the fourth drawing object does not indicate that the spatial information of the fourth drawing object changes, that is, the fourth drawing object is a static object. In this way, the electronic device stores the color data of the dynamic object and the color data of the static object in the second drawing frame in different color attachments respectively. Then, the moving object in the predicted frame is predicted according to the moving object in the drawing frame, the static object in the predicted frame is predicted according to the static object in the drawing frame, and the color attachment of the moving object and the color attachment of the static object are synthesized into an image frame. In this way, the predicted frame can be predicted more accurately.

A method for predicting an image frame provided by an embodiment of the present application will be described in detail below with reference to the accompanying drawings. First, FIG. 6A exemplarily shows the process of drawing N drawing frames, N+2 drawing frames, and obtaining the N+3 th predicted frame by an electronic device in an image frame prediction method provided by an embodiment of the present application.

Figure (a) in FIG. 6A exemplarily shows a process in which the electronic device draws the Nth drawing frame. As shown in (a) of FIG. 6A , drawing the Nth drawing frame by the electronic device may include the following steps:

601. The electronic device acquires a drawing instruction of the Nth drawing frame, and determines whether the drawing instruction carries a transition matrix. If yes, go to 602a, if not, go to 602b.

There can be multiple drawing instructions in the Nth drawing frame. The electronic device can draw an object according to a drawing instruction. It can be understood that, after the electronic device draws one drawing instruction, it draws another drawing instruction of the Nth drawing frame until all drawing instructions in the Nth drawing are drawn. A drawing instruction can carry information such as vertex coordinates, vertex ID, depth information, color information, etc. of the object drawn according to the draw call instruction. If the drawing instruction carries a transition matrix, the object drawn by the electronic device according to the drawing instruction is a moving object. If the drawing instruction does not carry the transition matrix, the object drawn by the electronic device according to the drawing instruction is a static object.

Optionally, all draw instructions in the Nth draw frame may carry transition matrices. If the transition matrix carried in the drawing instruction is an all-zero matrix, the object drawn by the electronic device according to the drawing instruction is a static object. If the values of the elements in the transition matrix carried by the drawing instruction are not all 0, the object drawn by the electronic device according to the drawing instruction is a moving object. If the drawing content of the drawing instruction is a moving object, the electronic device draws the drawing content of the drawing instruction in the first memory space. The first memory space may be referred to as D1FBO (dynamic framebuffer object, dynamic frame buffer object). If the drawing content of the drawcall instruction is a static object, the electronic device draws the drawing content of the drawcall instruction in the second memory space. The second memory space may be referred to as S1FBO (static framebuffer object, static frame buffer object).

Here, the frame buffer object FBO is a memory space that can be used to store color data, depth data, etc. of the drawing object.

In this embodiment of the present application, the electronic device may draw moving objects and static objects in the drawing frame according to the information carried in the drawing instruction. In this embodiment of the present application, the drawing instruction may be referred to as a draw call instruction. In this embodiment of the present application, the drawing instructions can be distinguished according to whether the transition matrix is carried, or the drawing instructions can be distinguished by whether the elements in the carried transition matrix are all 0s.

If the drawing commands are distinguished according to whether the drawing commands carry a transition matrix, the electronic device can classify the drawing commands into two types, one is a drawing command with a transition matrix, and the other is a drawing command without a transition matrix. The drawing content of the drawing instruction carrying the transition matrix is the moving object. The drawing content of a drawing instruction that does not carry a transition matrix is a static object. In this embodiment of the present application, the electronic device may draw the two types of drawing instructions in different memory spaces.

If it is distinguished according to whether the transfer matrix carried in the drawing instruction is an all-zero matrix, the electronic device can divide the drawing instructions into two categories, one is the drawing instruction whose elements in the carried transfer matrix are not all 0s, and the other is the drawing instruction that carries A drawing instruction whose elements in the transition matrix are all 0 (that is, a matrix of all 0s). The drawing content of the drawing instruction whose elements in the carried transition matrix are not all 0 is a moving object. The drawing content of the drawing instruction whose transfer matrix is an all-zero matrix is a static object. In this embodiment of the present application, the electronic device may draw these two types of drawing instructions in different memory spaces.

602a. The electronic device writes the drawing content of the drawing instruction into the color attachment A and the depth attachment A in the first memory space.

The electronic device writes the drawing content of the drawing instruction into the color attachment A and the depth attachment A in the first memory space. In the Nth drawing frame, there may be multiple drawing instructions whose drawing contents are moving objects, and the electronic device may draw the drawing contents of the multiple drawing instructions into the color attachment A and the depth attachment A in sequence. In an achievable manner, if there are L drawing instructions in the Nth drawing frame that carry a transition matrix, the drawing content of the drawing instruction is a moving object. The electronic device can sequentially draw the drawing contents with color information of the L drawing instructions in the Nth drawing frame on the canvas 1, and the finally obtained drawing result with color information of all moving objects can be called color attachment A. Specifically, the electronic device may draw the drawing content with color information of the first drawing instruction in the L drawing instructions in the Nth drawing frame on the canvas 1 of the electronic device. Then, the electronic device draws the drawing content with color information of the second drawing instruction in the L drawing instructions in the Nth drawing frame on the canvas 1 of the electronic device. Until the electronic device draws the drawing content with color information of the L-th drawing instruction in the N-th drawing frame on the canvas 1 of the electronic device, the finally obtained drawing result with color information of all moving objects can be called This is the color attachment A in the examples of this application.

The electronic device can sequentially draw the drawing contents with depth information and without color information of the L drawing instructions in the Nth drawing frame on the canvas 2, and finally obtain a drawing result with the depth information of all moving objects, which can be called a depth attachment. A. Specifically, the electronic device may draw, on the canvas 2 , the drawing content of the first drawing instruction that has depth information but does not include color information among the L drawing instructions. Then, the electronic device may draw the drawing content of the second drawing instruction among the L drawing instructions that has depth information but does not include color information on the canvas 2 . Until the electronic device draws the drawing content of the Lth drawing instruction in the L drawing instructions that has depth information but does not contain color information on the canvas 2, the finally obtained drawing result with the depth information of all moving objects can be referred to as the present application. Depth Attachment A in Examples.

It can be understood that both the canvas 1 and the canvas 2 in the above are in the first memory space.

For the depth attachment A and the color attachment A, reference may be made to the description in step S105 of FIG. 6B below, which will not be repeated here.

602b. The electronic device writes the drawing content of the drawing instruction into the color attachment B and the depth attachment B in the second memory space.

The description is given by taking as an example that the Nth drawing frame has M drawing instructions whose drawing contents are static objects. The electronic device may sequentially draw the drawing contents with color information of the M drawing instructions in the Nth drawing frame on the canvas 3 . Specifically, the electronic device may draw the drawing content with color information of the first drawing instruction among the M drawing instructions on the canvas 3, and then draw the drawing content with color information of the second drawing instruction among the M drawing instructions The content is drawn in canvas 3. Until, the electronic device draws the drawing content with color information of the Mth drawing instruction among the M drawing instructions on the canvas 3, and the finally obtained drawing result with color information of all static objects can be called color attachment B.

The electronic device may sequentially draw the M drawing instructions in the Nth drawing frame with depth information and without color information to the canvas 4, and finally obtain the depth attachment B. Specifically, the electronic device may draw the first drawing instruction in the M drawing instructions with depth information and without color information on the canvas 4, and then draw the second drawing instruction in the M drawing instructions. Drawing content with depth information and no color information is drawn in canvas 4. Until, the electronic device draws the drawing content of the M-th drawing instruction with depth information but not including color information in the canvas 4, and the finally obtained drawing result with the depth information of all static objects can be called depth Annex B.

It can be understood that canvas 3 and canvas 4 are in the second memory space.

603. The electronic device combines the color attachment A and the color attachment B into an image frame of the Nth drawing frame according to the depth attachment A and the depth attachment B.

The electronic device may combine the color attachment A and the color attachment B into the image frame of the Nth drawing frame in the seventh memory space. It can be understood that the color attachment A may include multiple moving objects in the Nth drawing frame, and the color attachment B may include multiple static objects in the Nth drawing frame. The electronic device can obtain the depth information of each moving object and each pixel in the color annex A from the depth annex A. The electronic device can obtain each static object in the color annex B and the depth information of each pixel point from the depth annex B. The electronic device may combine the color annex A and the color annex B into an image frame according to the depth information of each pixel in the color annex A and the depth information of each pixel in the color annex B. Image frames can contain moving objects in color attachment A and static objects in color attachment B. For example, in the image frame, at the first pixel, if the depth value of the first pixel in color attachment A is smaller than the depth value of the first pixel in color attachment B, then in the image frame, the color attachment A The first pixel of color will cover the first pixel of color attachment B. At the first pixel, if the depth value of the first pixel in color annex A is greater than the depth value of the first pixel in color annex B, then in the image frame, the first pixel in color annex B will cover the color The first pixel of Annex A.

604. The electronic device displays the Nth drawing frame.

The electronic device may send the image frame of the Nth drawing frame to the display screen for display, and finally, the display screen in the electronic device may display the Nth drawing frame.

Figure (b) in FIG. 6A exemplarily shows a process in which the electronic device draws the N+2 th drawing frame. As shown in (b) of FIG. 6A , drawing the N+2 th drawing frame by the electronic device may include the following steps:

605. The electronic device acquires the drawing instruction of the N+2th drawing frame, and determines whether the drawing instruction carries a transition matrix. If yes, go to 606a, if not, go to 606b.

The electronic device may acquire multiple drawing instructions of the N+2th drawing frame. In the drawing instruction of the N+2th drawing frame, the transition matrix may or may not be carried. If the drawing instruction carries the transition matrix, go to step 606a, and if the drawing instruction does not carry the transition matrix, go to step 606b.

For step 605, reference may be made to the description in step 601, and details are not repeated here.

606a. The electronic device writes the drawing content of the drawing instruction into the color attachment C and the depth attachment C in the third memory space.

If the drawing content of the drawing instruction is a moving object, the electronic device writes the drawing content of the drawing instruction into the color attachment C and the depth attachment C in the third memory space. If there are K drawing instructions carrying a transition matrix in the N+2th drawing frame, the drawing content of the drawing instruction is a moving object. The electronic device can sequentially draw the drawing contents with color information of the K drawing instructions in the N+2th drawing frame on the canvas 5, and the finally obtained drawing result with color information of all moving objects can be called color attachment C. Specifically, the electronic device may draw the drawing content with color information of the first drawing instruction in the K drawing instructions in the N+2 th drawing frame on the canvas 5 of the electronic device. Then, the electronic device draws the drawing content with color information of the second drawing instruction in the K drawing instructions in the N+2 th drawing frame on the canvas 5 of the electronic device. Until the electronic device draws the drawing content with color information of the K-th drawing instruction in the K-th drawing instruction in the N+2-th drawing frame to the canvas 5 of the electronic device, and finally obtains the drawing result with the color information of all moving objects It may be referred to as color attachment C in the embodiment of the present application.

The electronic device can draw the K drawing instructions with depth information and without color information in the N+2th drawing frame on the canvas 6 in turn, and finally obtain the drawing result with the depth information of all moving objects, which can be called. For depth attachment C. Specifically, the electronic device may draw, on the canvas 6 , the drawing content of the first drawing instruction among the K drawing instructions that has depth information but does not include color information. Then, the electronic device may draw the drawing content of the second drawing instruction among the K drawing instructions with depth information but not including color information on the canvas 6 . Until the electronic device draws the Kth drawing instruction in the K drawing instructions with depth information and does not contain the drawing content of the color information on the canvas 6, and finally the image of the drawing result with the depth information of all moving objects can be called the present application. Depth Attachment C in Examples.

It can be understood that both canvas 5 and canvas 6 are in the third memory space.

For details about the depth attachment C and the color attachment C, reference may be made to the description in step S109 of FIG. 6B below, which will not be repeated here.

606b. The electronic device writes the drawing content of the drawing instruction into the color attachment D and the color attachment D in the fourth memory space.

If the drawing content of the drawing instruction of the N+2th drawing frame is a static object, the electronic device writes the drawing content of the drawing instruction into the color attachment D and the depth attachment D in the fourth memory space. If there are J drawing instructions in the N+2th drawing frame that do not carry a transition matrix, the drawing content of the drawing instruction is a static object. The electronic device can sequentially draw the drawing contents with color information of the J drawing instructions in the N+2 th drawing frame on the canvas 7, and the finally obtained drawing result with color information of all static objects can be called color attachment D. Specifically, the electronic device may draw the drawing content with color information of the first drawing instruction in the J drawing instructions in the N+2 th drawing frame on the canvas 7 of the electronic device. Then, the electronic device draws the drawing content with color information of the second drawing instruction in the J drawing instructions in the N+2th drawing frame on the canvas 7 of the electronic device. Until the electronic device draws the drawing content with color information of the J-th drawing instruction in the J-th drawing instruction in the N+2-th drawing frame to the canvas 7 of the electronic device, and finally obtains the drawing result with the color information of all static objects It can be called color accessory D in the embodiment of the present application.

The electronic device can sequentially draw the drawing contents with depth information and without color information of the J drawing instructions in the N+2th drawing frame on the canvas 8, and finally obtain the drawing result with the depth information of all static objects, which can be called For depth annex D. Specifically, the electronic device may draw, on the canvas 8, the drawing content of the first drawing instruction among the J drawing instructions that has depth information but does not include color information. Then, the electronic device may draw the drawing content of the second drawing instruction among the J drawing instructions with depth information but not including color information on the canvas 8 . Until the electronic device draws the drawing content of the Jth drawing instruction with depth information and not including color information on the canvas 8 among the J drawing instructions, the finally obtained drawing result with the depth information of all static objects can be referred to as the present application. Depth Attachment D in Examples.

It can be understood that both canvas 7 and canvas 8 are in the third memory space.

For details about the depth attachment D and the color attachment D here, reference may be made to the description in step S109 of FIG. 6B below, which will not be repeated here.

607. The electronic device combines the color attachment C and the color attachment D into an image frame of the Nth drawing frame according to the depth attachment C and the depth attachment D.

The electronic device may combine the color attachment C and the color attachment D into the image frame of the N+2 th drawing frame in the seventh memory space. It can be understood that the color attachment C may include multiple moving objects in the N+2 th drawing frame, and the color attachment D may include multiple static objects in the N+2 th drawing frame. The electronic device can obtain the depth information of each moving object and each pixel in the color attachment C from the depth attachment C. The electronic device can obtain the depth information of each static object in the color attachment D and the depth information of each pixel point from the depth attachment D. The electronic device may combine the color attachment C and the color attachment D into an image frame according to the depth information of each pixel point in the color attachment C and the depth information of each pixel point in the color attachment D. Image frames can contain moving objects in color attachment C and static objects in color attachment D. For example, in the image frame, at the first pixel, if the depth value of the first pixel in color attachment C is less than the depth value of the first pixel in color attachment D, then in the image frame, the color attachment C The first pixel of color will cover the first pixel of color attachment D. In the image frame, at the first pixel, if the depth value of the first pixel in color attachment C is greater than the depth value of the first pixel in color attachment D, then in the image frame, the first pixel in color attachment D The dot will cover the first pixel of color attachment C.

608. The electronic device displays the N+2th drawing frame.

The electronic device can send the image frame of the N+2 th drawing frame to the display screen for display, and finally, the display screen in the electronic device can display the N+2 th drawing frame.

Figure (c) in FIG. 6A exemplarily shows the process of how the electronic device predicts the N+3 th predicted frame. As shown in (c) in Figure 6A, the process is as follows:

1. The electronic device calculates the motion vector A according to the color attachment A and the color attachment C, and calculates the motion vector A according to the depth attachment B and the color attachment.

Annex D calculates the motion vector B.

As shown in (c) of FIG. 6A , the electronic device can calculate the motion vector A according to the color attachment A and the color attachment C. The electronic device calculates the motion vector B according to the depth attachment B and the color attachment D. For the specific calculation process, reference may be made to the description in step S112 in FIG. 6B , which will not be repeated here.

2. The electronic device obtains color attachment E according to color attachment C and motion vector A, and obtains color attachment F according to color attachment D and motion vector B.

The electronic device may obtain the color attachment E of the N+3 th prediction frame according to the color attachment of the N+2 th drawing frame and the motion vector A. That is to say, the electronic device can predict the moving object in the N+3 th frame according to the moving object in the N+2 th drawing frame and the motion vector of the moving object.

The electronic device may obtain the color attachment F of the N+3 th frame according to the color attachment D and the motion vector B of the N+2 th frame. That is to say, the electronic device can predict the static object in the N+3 th frame according to the static object of the N+2 th drawing frame and the moving object of the static object.

Here, specific reference may be made to the description in step S114 in FIG. 6B , which will not be repeated here.

3. The electronic device combines the color attachment E and the color attachment F into the image frame of the N+3 th prediction frame in the seventh memory space.

In this embodiment of the present application, the depth information of the pixel at the first coordinate in the color annex E of the N+3 prediction frame may be the same as the depth information of the pixel at the first coordinate in the color annex C in the N+2 drawing frame. Information is the same. The depth information of the pixel at the second coordinate in the color attachment F of the N+3th prediction frame may be the same as the depth information of the pixel at the second coordinate in the color attachment D of the N+2th drawing frame. That is to say, the electronic device can use the depth value of each pixel in the depth attachment C as the depth value of each pixel in the color attachment E. The electronic device may use the depth value of each pixel in the depth attachment D as the depth value of each pixel in the color attachment F. The electronic device may combine the color attachment E and the color attachment F into the image frame of the N+3th predicted frame according to the depth values of the pixels extracted from the depth attachment C and the depth attachment D.

4. The electronic device displays the N+3 predicted frame.

The electronic device can send the image frame of the N+3 th predicted frame to the display screen for display, and finally, the display screen in the electronic device can display the N+3 th predicted frame.

FIG. 6A briefly introduces the basic steps of the image frame prediction method provided by the embodiment of the present application, and FIG. 6B introduces the flowchart of the image frame prediction method provided by the embodiment of the present application in detail. As shown in FIG. 6B , the embodiment of the present application provides A method of image frame prediction may include:

S101-S102, the electronic device starts to perform the method for image frame prediction.

S101. When the target application starts to draw, the CPU of the electronic device sends an instruction for instructing the GPU to create a memory space to the GPU.

The target application is an application with animation effects in the user interface, such as a game application. The embodiments of the present application are described below by taking the target application as a game application as an example. When the game application installed in the electronic device runs, the CPU of the electronic device sends an instruction for instructing the GPU to create a memory space to the GPU.

S102. The GPU of the electronic device creates a first memory space, a second memory space, a third memory space, a fourth memory space, a fifth memory space, a sixth memory space, and a seventh memory space in the memory.

In response to the instruction sent by the CPU for instructing the GPU to create a memory space, the GPU may create a first memory space, a second memory space, a third memory space, a fourth memory space, a fifth memory space, and a sixth memory space in the memory space and the seventh memory space. The first memory space may be used to store the drawing result of the drawing instruction carrying the transition matrix in the Nth drawing frame, such as depth attachment and color attachment. The second memory space may be used to store the drawing result of the drawing instruction that does not carry the transition matrix in the Nth drawing frame. The third memory space may be used to store the drawing result of the drawing instruction carrying the transition matrix in the N+2th drawing frame. The fourth memory space may be used to store the drawing result of the drawing instruction that does not carry the transition matrix in the N+2th drawing frame. The GPU may predict the predicted rendering result A of the N+3th predicted frame according to the rendering result in the first memory space and the rendering result in the third memory space in the fifth memory space. The GPU may predict the predicted rendering result B of the N+3th predicted frame in the sixth memory space according to the rendering result in the second memory space and the rendering result in the fourth memory space. The GPU can synthesize image frames in the seventh memory space. For example, the GPU synthesizes the Nth drawing frame in the seventh memory space, as well as the N+2th drawing frame and the N+3th prediction frame.

S103-S106, the electronic device draws the Nth drawing frame.

S103, the CPU of the electronic device acquires the drawing parameters of the Nth drawing frame.

When the target application in the electronic device draws, the target application can call a drawing instruction to draw. The CPU of the electronic device 100 may acquire the drawing parameters of the Nth drawing frame of the application program through the interface in the three-dimensional image processing library. The drawing parameters of the Nth drawing frame are used to draw and render the Nth drawing frame. The drawing parameters of the Nth drawing frame may include information carried in the drawing instructions (such as the draw call instruction) of the Nth drawing frame, such as the coordinates, pixel values, depth values and other information of each vertex in the drawing content of the draw call instruction .

S104. The CPU of the electronic device sends a drawing instruction for instructing the GPU to draw the Nth drawing frame to the GPU according to the drawing parameters of the Nth drawing frame.

The CPU of the electronic device may send a drawing instruction for instructing the GPU to draw the Nth drawing frame to the GPU according to the drawing parameters of the Nth drawing frame. It can be understood that, the drawing parameters of the Nth drawing frame acquired by the CPU may include information of multiple drawing instructions. In this way, the CPU may sequentially send a plurality of drawing instructions to the GPU for instructing the GPU to draw the Nth drawing frame. In this embodiment of the present application, the drawing instruction includes an execution drawing (draw call) instruction and a drawing state setting instruction.

The execution of the drawing instruction may be used to trigger the GPU to perform drawing and rendering with respect to the current drawing state data, and generate a drawing result, such as the glDrawElements instruction in OpenGL. OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics.

The drawing state setting instruction can be used to set the current drawing state data on which the drawing instruction is executed. For example, setting the state data includes the instruction of the vertex information buffer index that the drawing depends on, such as glBindBuffer in OpenGL, where the vertex information buffer index is used for Indicates vertex information data of a drawing object. The vertex information data is a set of coordinate position, color and other data used to describe the vertices of a two-dimensional or three-dimensional vector model for drawing in the drawing and rendering process.

The drawing state setting instruction may further include instructions such as setting the vertex index, texture information, and spatial position of the drawing object, for example, the glActiveTexture and glBindBufferRange instructions in OpenGL. A drawing object may be an object that can be drawn by an electronic device according to all vertices and vertex information contained in a drawing instruction.

For a more vivid description, a possible OpenGL drawing instruction according to the execution order can be as follows:

glBindBufferRange(target=GL_UNIFORM_BUFFER,index=1,buffer=738,offset=0,size=352)//Instruct the GPU to modify the partial drawing global information, such as the position of the car;

glBindBuffer(target=GL_ARRAY_BUFFER,buffer=buffer0)//Instructs the GPU to store the buffer0 index information of the vertex information of the car (such as vertex position, color, etc.) into GL_ARRAY_BUFFER;

glBindBuffer(target=GL_ELEMENT_ARRAY_BUFFER, buffer=buffer1)//Instructs the GPU to save the index of buffer1 that saves the vertex index information of the car (such as the drawing order of the vertices) into GL_ELEMENT_ARRAY_BUFFER;

glActiveTexture(texture=GL_TEXTURE0)

glBindTexture(target=GL_TEXTURE_2D, texture=texture1)//Instruct the GPU to save the index of texture1 that stores the texture information of the car into GL_TEXTURE0;

…

glDrawElements(GLenummode,GLsizeicount,GLenumtype,const void*indices)//Instructs the GPU to perform cart drawing.

In a possible implementation manner, the CPU can set the parameters or data for the spatial information of the drawing object in the drawing state setting instruction to determine that the spatial position of the drawing object in the world coordinate system relative to the previous frame occurs. to determine whether to modify the memory space (FBO, and explain below) where the drawing result of the drawing instruction based on the current drawing state data is stored, where the parameter or data of the spatial information of the drawing object can be the XXX instruction (for example, in OpenGL The transfer matrix parameter in the glBindBufferRange), the transfer matrix is used to describe the mapping relationship between the local coordinate system (Local/Object Space) of the model to the world coordinate system. For example, if a vertex in the drawing object is in the local coordinate system The coordinates below are U(x1, y1, z1, 1), and the transition matrix is S, then the position of this vertex in the world coordinate system W(x2, y2, z2, 1) and the coordinates in the local coordinate system U(x1 , y1, z1, 1) The relationship is: W=U*T.

A possible embodiment is that if there is a transition matrix parameter in the drawing state setting instruction, that is, the transition matrix parameter of the corresponding drawing object is refreshed, the electronic device determines that the drawing object is in world coordinates relative to the drawing object in the previous frame. If the spatial position in the system changes, the electronic device modifies the memory space where the drawing result of the drawing instruction based on the current drawing state data is stored to the memory space (eg, the first memory space) for storing the moving object. If there is no transition matrix parameter in the drawing state setting instruction, that is, the transition matrix parameter of the corresponding drawing object is not refreshed, the electronic device determines that the spatial position of the drawing object in the world coordinate system relative to the previous frame is not change. Then, the electronic device modifies the memory space where the drawing result of the drawing instruction based on the current drawing state data is stored to the memory space (eg, the second memory space) for storing the static object.

A possible embodiment is that if there is a transition matrix parameter in the drawing state setting instruction that is different from the transition matrix parameter of the corresponding drawing object, the electronic device determines that the drawing object is in the world coordinate system relative to the drawing object in the previous frame. If the spatial position of the object changes, the electronic device modifies the memory space where the drawing result of the drawing instruction based on the current drawing state data is stored to the memory space (eg, the first memory space) for storing the moving object. If the transition matrix parameter in the drawing state setting instruction is the same as the transition matrix parameter of the corresponding drawing object, the electronic device determines that the spatial position of the drawing object in the world coordinate system relative to the previous frame has not changed. Then, the electronic device modifies the memory space where the drawing result of the drawing instruction based on the current drawing state data is stored to the memory space (eg, the second memory space) for storing the static object.

For example, in glBindBufferRange(target=GL_UNIFORM_BUFFER,index=1,buffer=738,offset=0,size=352), the parameter target indicates the type of the binding buffer, the parameter index indicates the index of the binding point, and the parameter buffer indicates the buffer that participates in the binding operation. The index of the buffer, the parameter offset indicates the offset of the position to be bound relative to the starting position of the buffer, and the parameter size indicates the size of the part of the buffer that participates in the binding. If the parameter size of the glBindBufferRange function is 352, and the buffer ID (738) has been saved, it means that the spatial location information corresponding to the buffer is the location information modified by the glBufferSubData instruction. Then, the electronic device determines that the spatial position of the drawing object in the world coordinate system has changed with respect to the previous frame. Otherwise, the electronic device determines that the spatial position of the drawing object in the world coordinate system with respect to the previous frame has not changed. The electronic device can obtain the transition matrix parameters in the drawing frame of the program through the hook function. For example, the electronic device can obtain the transfer matrix parameters through the glBindBufferRange function.

S105. If the drawing instruction of the Nth drawing frame carries the transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment A and the depth attachment A in the first memory space; Carrying the transfer instruction, the GPU writes the drawing content of the drawing instruction into the color attachment B and the depth attachment B in the second memory space.

The drawing instructions of the Nth drawing frame may include a drawing instruction that carries a transition matrix and a drawing instruction that does not carry a transition matrix. As shown in FIG. 7A , FIG. 7A exemplarily shows the Nth drawing frame, and the Nth drawing frame may include a static background 102 and a moving car 103 . The drawing instructions of the Nth drawing frame may include a drawing instruction for drawing the static background 102 and a drawing instruction for drawing the moving cart 103 .

If a drawing instruction of the Nth drawing frame carries a transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment A and the depth attachment A in the first memory space. For example, a drawing instruction to draw a moving car can carry a transition matrix. The GPU writes the drawing content of the drawing instruction for drawing the moving cart 103 into the color attachment A and the depth attachment A in the first memory space. As shown in FIG. 7B , FIG. 7B exemplarily shows the color attachment A and the depth attachment A of the first memory space. The drawing content of the drawing instruction of the moving cart 103 is written in the color attachment A and the depth attachment A. Understandably, Color Attachment A is actually a block of memory space in memory. FIG. 7B schematically shows the color data in the memory space in the form of an image scale for visual understanding. Similarly, depth attachment A is actually a memory space in memory. FIG. 7B schematically shows the depth data in the memory space in the form of an image scale for the purpose of visual understanding.

In this embodiment of the present application, the color attachment A may be referred to as the first color attachment, and the depth attachment A may be referred to as the first depth attachment.

If a drawing instruction of the Nth drawing frame does not carry a transition matrix, the GPU writes the drawing content of the one drawing instruction into the color attachment B and the depth attachment B in the second memory space. For example, for the static background 102 shown in FIG. 7A, the drawing instruction for drawing the static background 102 may not carry a transition matrix. The GPU writes the drawing content of the drawing instruction for drawing the static background 102 into the color attachment B and the depth attachment B in the second memory space. As shown in FIG. 7C , FIG. 7C exemplarily shows color attachment B and depth attachment B of the second memory space. In the color attachment B and the depth attachment B, the rendering content of the rendering instruction of the static background 102 is written. Understandably, color attachment B is actually a block of memory space in memory. FIG. 7C schematically shows the color data in the memory space in the form of an image scale for visual understanding. Likewise, depth attachment B is actually a memory space in memory. In FIG. 7C, for visual understanding, the depth data in the memory space is shown schematically in the form of an image scale.

In this embodiment of the present application, the color attachment B may be referred to as the second color attachment, and the depth attachment B may be referred to as the second depth attachment.

It can be understood that, in the Nth drawing frame, there may be multiple drawing instructions that carry the transition matrix and multiple drawing instructions that do not carry the transition matrix. The GPU draws each drawing instruction of the Nth drawing frame in turn. Each time the GPU draws a drawing instruction, it can determine whether the drawing instruction carries a transition matrix. If the drawing instruction carries the transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment A and the depth attachment A in the first memory space. If the transfer matrix is not carried in the drawing instruction, the GPU writes the drawing content of the drawing instruction into the color attachment B and the depth attachment B in the second memory space. Finally, the color attachment A of the embodiment of the present application sequentially writes the drawing contents with color information of all the drawing instructions carrying the transition matrix in the Nth drawing frame. The depth attachment A sequentially writes the drawing contents with depth information of all the drawing instructions carrying the transition matrix in the Nth drawing frame. Similarly, in the color annex B of the embodiment of the present application, the drawing contents with color information of all the drawing instructions that do not carry the transition matrix in the Nth drawing frame are written in sequence. The depth attachment B sequentially writes the drawing contents with depth information of all the drawing instructions that do not carry the transition matrix in the Nth drawing frame.

It can be understood that the color attachment A and the depth attachment A may be two separate memory spaces in the first memory space. Optionally, the color attachment A and the depth attachment A may also be a piece of memory space in the first memory space, that is, color data and depth data can be written in this piece of memory space. Likewise, color attachment B and depth attachment B may be two separate memory spaces in the second memory space. Optionally, the color attachment B and the depth attachment B may also be a piece of memory space in the second memory space, that is, color data and depth data can be written in this piece of memory space.

For step S105, reference may be made to

steps

602a and 602b above.

S106. In the seventh memory space, the GPU of the electronic device combines the color attachment A and the color attachment B of the Nth rendering frame into the Nth rendering frame according to the depth attachment A and the depth attachment B.

In the seventh memory space, the GPU of the electronic device may combine the color attachment A and the color attachment B of the Nth drawing frame into the Nth drawing frame according to the depth attachment A and the depth attachment B. For synthesizing color attachment A and color attachment B according to the depth attachment, reference may be made to the description of synthesizing two color attachments based on depth information in the depth test in the prior art, which will not be repeated here.

S107-S110, the electronic device draws the N+2th drawing frame.

S107, the CPU of the electronic device acquires the drawing parameters of the N+2th drawing frame.

The CPU of the electronic device may acquire the drawing parameters of the N+2th drawing frame. Specifically, the CPU of the electronic device 100 can acquire the drawing parameters of the N+2 th drawing frame of the application program through the interface in the three-dimensional image processing library. The drawing parameters of the N+2th drawing frame are used to draw and render the N+2th drawing frame. The drawing parameters of the N+2 th drawing frame may include the information carried in the drawing instruction (eg, the draw call instruction) of the N+2 th drawing frame, such as the coordinates, pixel value, depth of each vertex in the drawing content of the draw call instruction value, etc.

It can be understood that, before the electronic device draws the N+2 th drawing frame, the electronic device displays the N+1 th frame. If the N+1 th frame is a drawing frame, the electronic device may draw the N+1 th frame according to the steps of drawing the N+2 th frame in steps S107 to S110 . If the N+1 th frame is a predicted frame, the electronic device may predict the N+1 th predicted frame according to steps S111-S115.

S108 , the CPU of the electronic device sends a drawing instruction for instructing the GPU to draw the N+2 th drawing to the GPU according to the drawing parameters of the N+2 th drawing frame.

The CPU of the electronic device may send a drawing instruction for instructing the GPU to draw the N+2 th drawing frame to the GPU according to the drawing parameters of the N+2 th drawing frame. It can be understood that the drawing parameters of the N+2th drawing frame acquired by the CPU may include information of multiple drawing instructions. In this way, the CPU may sequentially send multiple drawing instructions to the GPU for instructing the GPU to draw the N+2 th drawing frame. Here, specific reference may be made to the description in step S104, which is not repeated here.

S109, if the transfer matrix is carried in the drawing instruction of the N+2th drawing frame, the GPU writes the drawing content of the drawing instruction of the N+2th frame into the color attachment C and the depth attachment C in the third memory space; if The drawing instruction of the N+2th drawing frame does not carry the transition matrix, and the GPU writes the drawing content of the drawing instruction of the N+2th frame into the color attachment D and the depth attachment D in the fourth memory space.

The drawing instructions of the N+2th drawing frame may include a drawing instruction that carries a transition matrix and a drawing instruction that does not carry a transition matrix. As shown in FIG. 8A , FIG. 8A exemplarily shows the N+2 th drawing frame, and the N+2 th drawing frame may include a static background 102 and a moving cart 103 . The drawing instructions of the N+2th drawing frame may include a drawing instruction for drawing the static background 102 and a drawing instruction for drawing the moving cart 103 .

If the drawing instruction carries the transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment C and the depth attachment C in the third memory space. For example, a drawing instruction to draw a moving car can carry a transition matrix. The GPU writes the drawing content of the drawing instruction for drawing the moving cart 103 into the color attachment C and the depth attachment C in the third memory space. As shown in FIG. 8B , FIG. 8B exemplarily shows the color attachment C and the depth attachment C of the third memory space. The drawing content of the drawing instruction of the moving cart 103 is written in the color attachment C and the depth attachment C. Understandably, Color Attachment C is actually a block of memory space in memory. FIG. 8B schematically shows the color data in the memory space in the form of an image scale for visual understanding. Similarly, depth attachment C is actually a memory space in memory. FIG. 8B schematically shows the depth data in the memory space in the form of an image scale for visual understanding.

If the drawing instruction does not carry the transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment D and the depth attachment D in the fourth memory space. For example, for the static background 102 shown in FIG. 8A, the drawing instruction for drawing the static background 102 may not carry a transition matrix. The GPU writes the drawing content of the drawing instruction for drawing the static background 102 into the color attachment D and the depth attachment D in the fourth memory space. As shown in FIG. 8C , FIG. 8C exemplarily shows the color attachment D and the depth attachment D of the fourth memory space. In the color attachment D and the depth attachment D, the drawing content of the drawing instruction of the static background 102 is written. Understandably, the color attachment D is actually a block of memory space in memory. In FIG. 8C , for visual understanding, the color data in the memory space is schematically displayed in the form of an image scale. Likewise, depth attachment D is actually a memory space in memory. In FIG. 8C , for visual understanding, the depth data in the memory space is schematically displayed in the form of an image scale.

In this embodiment of the present application, the color accessory C may be referred to as the third color accessory, and the color accessory D may be referred to as the fourth color accessory. Depth attachment C may be referred to as the third depth attachment, and depth attachment D may be referred to as the fourth depth attachment.

It can be understood that, in the N+2th drawing frame, there may be multiple drawing instructions that carry the transition matrix and multiple drawing instructions that do not carry the transition matrix. The GPU draws each drawing instruction of the N+2th drawing frame in turn. Each time the GPU draws a drawing instruction, it can determine whether the drawing instruction carries a transition matrix. If the drawing instruction carries the transition matrix, the GPU writes the drawing content of the drawing instruction into the color attachment C and the depth attachment C in the third memory space. If the transfer matrix is not carried in the drawing instruction, the GPU writes the drawing content of the drawing instruction into the color attachment D and the depth attachment D in the fourth memory space. Finally, in the color attachment C of the embodiment of the present application, the drawing contents with color information of all the drawing instructions carrying the transition matrix in the N+2 th drawing frame are written in sequence. In the depth attachment C, the drawing contents with depth information of all the drawing instructions carrying the transition matrix in the Nth drawing frame are written in sequence. Similarly, in the color attachment D of the embodiment of the present application, the drawing contents with color information of all the drawing instructions that do not carry the transition matrix in the N+2 th drawing frame are written in sequence. In the depth attachment D, the drawing contents with depth information of all drawing instructions that do not carry the transition matrix in the N+2th drawing frame are sequentially written.

It can be understood that the color attachment A and the depth attachment A may be two separate memory spaces in the first memory space. Optionally, the color attachment A and the depth attachment A may also be a piece of memory space in the first memory space, that is, color data and depth data can be written in this piece of memory space. Likewise, color attachment B and depth attachment B may be two separate memory spaces in the second memory space. Optionally, the color attachment B and the depth attachment B may also be a piece of memory space in the second memory space, that is, color data and depth data can be written in this piece of memory space. For step S109, reference may be made to

steps

606a and 606b above.

S110. In the seventh memory space, the GPU combines the color attachment C and the color attachment D into the N+2th drawing frame according to the depth attachment C and the depth attachment D.

In the seventh memory space, the GPU of the electronic device may combine the color attachment C and the color attachment D of the N+2 th drawing frame into the N+2 th drawing frame according to the depth attachment C and the depth attachment D. For synthesizing the color attachment C and the color attachment D according to the depth attachment, reference may be made to the description of the synthesis of the two color attachments based on the depth information in the depth test in the prior art, which will not be repeated here.

It can be understood that, after the GPU sends the image frame of the Nth frame in the seventh memory space for display, it can clear the image frame of the Nth frame stored in the seventh memory space. Then, the GPU can synthesize the image frame of the N+2th frame in the seventh memory space. Similarly, after the GPU sends the image frame of the N+2th frame in the seventh memory space for display, the image frame of the N+2th frame stored in the seventh memory space can be cleared.

S111-S115, the electronic device predicts the N+3 th prediction frame.

S111. The CPU of the electronic device sends an instruction for instructing the GPU to calculate the motion vector to the GPU.

The CPU of the electronic device may send an instruction to the GPU for instructing the GPU to calculate the motion vector. This instruction is used to instruct the shader in the GPU to calculate the motion vector. The directive can be a dispatch directive. The embodiment of the present application does not limit the specific form of the instruction for calculating the motion vector.

S112, the GPU of the electronic device calculates the motion vector A of the color appendix C of the N+2th drawing frame by using the color appendix A, and uses the depth appendix B and the depth appendix D to calculate the motion vector of the color appendix D of the N+2th drawing frame B.

The GPU of the electronic device can separately calculate the motion vector of the moving object and the motion vector of the static object in the N+2th frame. The GPU of the electronic device can calculate the motion vector of the moving object (color attachment C) in the N+2th frame, that is, the motion vector A, by using the color attachment A and the color attachment C. The GPU of the electronic device may calculate, according to the depth attachment B and the depth attachment D, the motion vector of the static object (color attachment D) in the N+2th frame, that is, the motion vector B.

In this embodiment of the present application, the motion vector V may be referred to as the first motion vector, and the motion vector B may be referred to as the second motion vector.

In a possible implementation manner, the GPU of the electronic device uses the color attachment A to calculate the motion vector A of the color attachment C of the N+2th drawing frame, which may specifically include the following steps:

1. The GPU may divide the color attachment C of the N+2th frame into Q pixel blocks. Each pixel block may contain f*f (for example, 16*16) pixels. As shown in (b) of FIG. 9A .

2. The GPU takes the first pixel block in color attachment C (for example, pixel block 902 in (b) in FIG. 9A ), and searches for a matching pixel block matching the first pixel block in color attachment A of the Nth drawing frame , such as the pixel block 901 in (a) of FIG. 9A .

In the embodiment of the present application, among all the candidate blocks in the Nth frame, the candidate block with the smallest absolute difference with the RGB value of the first pixel block is referred to as a matching pixel block matching the pixel block. The electronic device needs to find a matching pixel block 901 that matches the pixel block in the Nth drawing frame. Optionally, the GPU of the electronic device may find a matching pixel block that matches the first pixel block in the Nth drawing frame by using a diamond search algorithm. As shown in FIG. 9B , the GPU can find a matching pixel block 902 in the Nth frame of the (b) figure by a diamond search algorithm that matches the pixel block 902 in the N+2th frame of the (a) figure. The electronic device can use the pixel point 9011 in the upper left corner of the pixel block 901 to perform a diamond search. The electronic device may find the pixel block 901 with the pixel point 1104 as the upper left corner in the Nth drawing frame to match the pixel block 902 . For details of the diamond search algorithm, reference may be made to the description in the prior art, which will not be repeated here.

As shown in FIG. 9C , FIG. 9C exemplarily shows a schematic diagram of the electronic device finding a pixel block 901 that matches the pixel block 901 in the Nth drawing frame. As shown in (a) of FIG. 9C , the electronic device performs a diamond search for the pixel point 9011 in the upper left corner of the pixel block 902 . In an achievable manner, as shown in (b) of FIG. 9C , the electronic device first performs a diamond search with the pixel point 9012 in the Nth drawing frame as the center point. The coordinates of the pixel point 9011 are the same as the coordinates of the pixel point 9012 . The electronic device can calculate that the pixel block 902 is respectively in the pixel block with the upper left pixel point as the pixel point 9012, the upper left pixel point is the pixel block with the pixel point 1001, the upper left pixel point is the pixel block with the pixel point 1002, the upper left pixel point is the pixel block with the pixel point 1002, and the upper left pixel point is the pixel block with the pixel point 1002. The point is the pixel block of pixel 1003, the upper left pixel is the pixel block of pixel 1004, the upper left pixel is the pixel block of pixel 1005, the upper left pixel is the pixel block of pixel 1006, and the upper left pixel is the pixel block of pixel 1006. For the pixel block of pixel point 1007, the upper left pixel point is the pixel value difference of the pixel block of pixel point 1008. The electronic device can then perform a small diamond search with the upper left pixel point in the pixel block with the smallest pixel value difference as the center. For example, in the above pixel block, the pixel value difference between the pixel block whose upper left corner is the pixel point 1003 and the pixel block 902 is the smallest. Then the electronic device can perform a small diamond search with the pixel point 1003 as the center. That is, the electronic device can calculate that the pixel block 902 is respectively in the pixel block with the upper left pixel point as the pixel point 1101, the upper left pixel point is the pixel block of the pixel point 1102, the upper left pixel point is the pixel block of the pixel point 1103, and the upper left corner pixel point is the pixel block of the pixel point 1103. The pixel point is the pixel block of the pixel point 1104 . Finally, the electronic device can determine that the pixel value difference between the pixel block whose upper left pixel point is the pixel point 1104 (ie, the pixel block 901 shown in FIG. 9B ) and the pixel block 902 is the smallest. Then the electronic device can determine that the pixel block 901 in the Nth frame is a matching pixel block of the pixel block 902 .

3. The GPU calculates the first displacement from the matched pixel block to the first pixel block, and determines the motion vector A1 of the first pixel block according to the first displacement.

For example, as shown in FIG. 9B , the matching block with the pixel block 902 in FIG. 9B is the pixel block 901 . Then the motion vector of the pixel block 902 is the motion vector V1 illustrated in (b) of FIG. 9B .

4. The GPU can calculate the motion vector of each pixel block in the Q pixel blocks in the color annex C according to the above steps 1-3, namely V1, V2, . . . , VQ. The motion vector of color attachment C is V=(V1, V2, . . . , VQ).

In a possible implementation manner, the GPU of the electronic device uses the depth attachment B and the depth attachment D to calculate the motion vector B of the color attachment D of the N+2th drawing frame, which may specifically include the following steps:

1. The GPU may divide the color attachment D of the N+2th frame into Q pixel blocks. Each small block may contain f*f (eg 16*16) pixels.

2. When the GPU draws the Nth frame, it can obtain the corner information of the camera position of the Nth frame through the hook glBufferSubData interface (saved in a 4x4 matrix), and record it as M1.

3. When the GPU draws the N+2 frame, it can obtain the corner information of the camera position of the N+2 frame through the hook glBufferSubData interface (saved in a 4x4 matrix), and record it as M2. The GPU and computes the transformation matrix T between M1 and M2. The matrix T represents the conversion relationship between the two frames of Camera.

here,

The matrix R1 represents the rotation property of the camera in the Nth drawing frame, and the matrix T1 represents the translation property of the camera in the Nth drawing frame. The matrix R2 represents the rotation property of the camera in the N+2th drawing frame, and the matrix T2 represents the translation property of the camera in the N+2th drawing frame. The matrix P is the projection matrix, and the GPU can directly obtain it through the hook glBufferSubData interface.

4. The GPU obtains the second pixel block and the position 2 of the second pixel block in the N+2th frame in the color attachment D of the N+2th frame, and obtains the depth value D2 of the second pixel block in the depth attachment D .

5. The GPU uses the position position2 of the second pixel block, the depth value D2, and the matrix T to calculate the position position1 of the second pixel block in the Nth drawing frame. Then the motion vector of the second pixel block B1=position2-position1. The GPU may calculate the motion vectors B1, B2, . . . , BQ of each pixel block in the color attachment D according to the step of calculating the motion vector of the second pixel block. Motion vector B=(B1, B2, . . . , BQ) for color attachment D.

Here, vector G2=(position2, D2, 1), vector G1=G2*T,

position1=(G1.x/G1.w, G1.y/G1.w)

Here G1 is a 1*4 vector, G1.x is the first element of G1, G1.y is the second element of G1, and G1.w is the fourth element of G1.

S113. The CPU of the electronic device sends an instruction for instructing the GPU to draw the predicted frame to the GPU.

The CPU of the electronic device 100 may send an instruction for drawing the N+3 th predicted frame to the GPU after the GPU calculates the motion vector A of the color attachment C and the motion vector B of the color attachment D.

S114. The GPU predicts the color attachment E of the N+3 th prediction frame according to the color attachment C and the motion vector A, and predicts the color attachment F of the N+3 th drawing frame according to the color attachment D and the motion vector B.

In response to the instruction instructing the GPU to draw the N+3 th predicted frame, the GPU may draw the color attachment E written in the moving object and the color attachment F written in the static object in the N+3 th predicted frame. Specifically, the GPU may predict the color attachment E of the N+3 th prediction frame according to the color attachment C and the motion vector A, and predict the color attachment F of the N+3 th drawing frame according to the color attachment D and the motion vector B. Illustratively, color attachment E may be as shown in FIG. 10A , and color attachment F may be as shown in FIG. 10B .

In this embodiment of the present application, the color attachment E may be referred to as the fifth color attachment, and the color attachment F may be referred to as the sixth color attachment.

It can be understood that the object in the Nth drawing frame is the same as the object in the N+2th drawing frame. For the same object A, the position of the object A in the Nth drawing frame may be different from the position of the object A in the N+2th drawing frame. Likewise, the objects in the N+2 th rendering frame and the N+3 th prediction frame are the same, but the positions of the objects in the N+2 th rendering frame and the N+3 th prediction frame are different. The GPU can predict the N+3 prediction frame by using the object of the N+2 drawing frame and the motion vector of the object. In this embodiment of the present application, the Nth drawing frame may be referred to as the first drawing frame. The N+2th drawing frame may be referred to as the second drawing frame. The N+3 th prediction frame may be referred to as the first prediction frame.

S115. In the seventh memory space, the GPU combines the color attachment E and the color attachment F according to the depth attachment C and the depth attachment D into the image frame of the N+3 th prediction frame.

In the seventh memory space, the GPU of the electronic device can combine the color attachment E and the color attachment F of the Nth drawing frame into the image frame of the N+3th drawing frame according to the depth attachment C and the depth attachment D. Exemplarily, the N+3 th predicted frame may be as shown in FIG. 10C . For synthesizing the color attachment E and the color attachment F according to the depth attachment, reference may be made to the description of synthesizing two images based on the depth attachment in the depth test in the prior art, which will not be repeated here. It can be understood that, since the interval between the N+2 th rendering frame and the N+3 th prediction frame is very short, the N+3 th prediction frame can be synthesized by using the depth information in the N+2 th rendering frame. Here, the GPU can also predict the depth attachments of moving objects and the depth attachments of static objects in the N+3th frame. Then, the color attachment E and the color attachment F are synthesized using the depth attachment of the moving object and the depth attachment of the static object in the N+3 frame. This application does not limit this.

The main process for the electronic device to draw the drawing frame is that the GPU executes several drawcall instructions. The GPU draws the contents of each drawcall instruction to the FBO one by one. Then each drawcall instruction needs to execute the rendering pipeline process once on the GPU. The process of the rendering pipeline is mainly divided into vertex shader, tessellation (not required), geometry shader (not required), primitive assembly (not required), rasterization, fragment shader, test mixing stage (not required) must). In this way, it is expensive for the GPU to draw a frame to draw a frame. In addition, the vertex information and coordinate information required by the GPU to execute each drawcall instruction need to be prepared by the GPU, and these preparations are also very computationally intensive for the CPU.

The process of predicting an image frame has almost no computational consumption for the CPU, and only needs to send some instructions to the GPU. For the GPU, only the motion vector of moving objects and the motion vector of static objects need to be calculated. These calculations are all parallel and only need to be calculated once. Each computing unit performs a small number of basic operations, which can reduce GPU calculations and improve performance.

It can be understood that the embodiment of the present application is not limited to the electronic device predicting the N+3 th predicted frame by using the N th drawing frame and the N+2 th drawing frame. Optionally, the electronic device may further predict the N+2 th frame by using the N th frame and the N+1 th frame. Optionally, the electronic device may also predict the N+4th frame by using the Nth frame and the N+3th frame. This embodiment of the present application does not limit this.

It can be understood that, when the electronic device displays the video interface of the target application, the electronic device can predict the predicted frame according to different strategies. That is, the strategy of the electronic device predicting the N+3th frame according to the Nth frame and the N+2th frame in the first time period, and the strategy of predicting the N+2th frame according to the Nth frame and the N+1th frame in the second time period. . For example, when the GPU performs many tasks, the electronic device may first predict the N+3 th frame according to the N th frame and the N+2 th frame. When the GPU performs few tasks, the electronic device may predict the N+2th frame according to the Nth frame and the N+1th frame. This embodiment of the present application does not limit this.

As shown in Figure 11, a frame insertion logic block diagram of 90 frames per second (90 frames per second, 90 fps) is transmitted. In FIG. 11, frames 0 to 2 are drawing frames. Starting from frame 3, odd-numbered frames are predicted frames, and even-numbered frames are drawn frames. For example, odd-numbered frames such as the third frame, the fifth frame, and the seventh frame are predicted frames. The 4th, 6th, 8th, etc. even-numbered frames are drawing frames. Frame 3 is generated by the electronic device based on frame 0 and frame 2. Frame 5 is generated by the electronic device based on

frames

2 and 4. The 7th frame is generated by the electronic device based on the 4th and 6th frames. The 89th frame is generated by the electronic device based on the 86th and 88th frames. Here, the 0th frame may be referred to as the 0th drawing frame. The third frame may be referred to as a third predicted frame.

The following first introduces the exemplary electronic device 100 provided by the embodiments of the present application.

FIG. 12 is a schematic structural diagram of an electronic device 100 provided by an embodiment of the present application.

The embodiment will be described in detail below by taking the electronic device 100 as an example. It should be understood that electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2. Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194 and Subscriber identification module (subscriber identification module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate with each other through the I2C bus interface, so as to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled with the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through a CSI interface, so as to realize the photographing function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The SIM interface can be used to communicate with the SIM card interface 195 to realize the function of transferring data to the SIM card or reading data in the SIM card.

The USB interface 130 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger.

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives input from the battery 142 and/or the charging management module 140 and supplies power to the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and then turn it into an electromagnetic wave for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110, and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellites Wireless communication solutions such as global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared technology (IR). The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou navigation satellite system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information, and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronization Dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM, such as the fifth generation DDR SDRAM is generally called DDR5 SDRAM) and so on.

Non-volatile memory may include magnetic disk storage devices, flash memory.

Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, and can include single-level memory cells (single-level cells, SLC), multi-level memory cells (multi-level memory cells) according to the level of storage cell potential cell, MLC), triple-level cell (TLC), fourth-level storage unit (quad-level cell, QLC), etc., according to the storage specification can include universal flash storage (English: universal flash storage, UFS) , embedded multimedia memory card (embedded multi media Card, eMMC) and so on.

The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (eg, machine instructions) of an operating system or other running programs, and can also be used to store data of users and application programs.

The non-volatile memory can also store executable programs and store data of user and application programs, etc., and can be loaded into the random access memory in advance for the processor 110 to directly read and write.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information to analog audio signal output, and also to convert analog audio input to digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be answered by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through a human mouth, and input the sound signal into the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D can be the USB interface 130, or can be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation whose intensity is less than the first pressure threshold acts on the short message application icon, the instruction for viewing the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenarios.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of the flip holster using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Further, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, characteristics such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

Distance sensor 180F for measuring distance. The electronic device 100 can measure the distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light to the outside through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used in holster mode, pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, accessing application locks, taking pictures with fingerprints, answering incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be disposed on the display screen 194 , and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the location where the display screen 194 is located.

The keys 190 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

Motor 191 can generate vibrating cues. The motor 191 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as call and data communication.

FIG. 13 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present application.

The system framework 1300 for implementing image frame prediction provided by the embodiments of the present application includes a software architecture and hardware devices. Among them, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, a system library, and a kernel layer.

The application layer can include a series of application packages.

As shown in Figure 13, the application layer may target application 1301. The application layer may also include camera (not shown in FIG. 13 ), gallery (not shown in FIG. 13 ), calendar (not shown in FIG. 13 ), calls (not shown in FIG. 13 ), maps (not shown in FIG. 13 ) (not shown in FIG. 13 ), navigation (not shown in FIG. 13 ), etc. applications (which may also be referred to as applications). The target application 1301 may be a game application.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment of the present application, the application framework layer may include an application engine 1310 . The application engine 1310 may include a rendering system (Rendering System) 1311. When the electronic device 100 is running the target application 1301 , the rendering system 1311 in the application engine 1310 corresponding to the target application 1301 can acquire the drawing parameters of the target application 1301 . The rendering system 1311 can also call the interface in the three-dimensional graphics processing library 1330 according to the drawing parameters, so as to realize the rendering of the image frame of the target application 1301 . The application engine 1310 may be a game engine corresponding to a game application. The 3D graphics processing library 1330 may be Vulkan, OpenGL, OpenGL ES.

A system library can include multiple functional modules. For example: surface manager (not shown in Figure 13), Media Libraries (not shown in Figure 13), platform interface 1320, 3D graphics processing library 1330 (eg: OpenGL ES), two dimensional graphics engine (eg: SGL) (not shown in FIG. 13 ), etc.

The Surface Manager is used to manage the display subsystem and provides a fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The platform interface 1320 may be used to receive the API for configuring the cache transmitted by the 3D graphics processing library 1330 . In response to the API for configuring the cache, the platform interface 1320 can drive the motion random access memory through a driver in the driver layer. In turn, the platform interface 1320 may configure storage space in the Motion RAM for use by the target application. In this embodiment of the present application, the platform interface 1320 may be EGL. EGL is the interface between the Khronos rendering API (such as OpenGL, OpenGL ES or OpenVG) and the underlying native platform windowing system. EGL handles graphics context management, surface/buffer binding, rendering synchronization, and enables "high-performance, accelerated, mixed-mode 2D and 3D rendering using other Khronos APIs".

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The three-dimensional graphics processing library 1330 may be OpenGL ES. OpenGL ES is an application programming interface/library that is a subset of the OpenGL 3D graphics API. OpenGL ES includes a variety of functional functions/application programming interfaces, such as glBindFrameBuffer interface 1333', glDrawArrays interface (not shown in the figure). The electronic device 100 may call OpenGL ES to implement the drawing of the image frame.

The hook module (HOOK System) 1331 can acquire and call the parameters of interfaces such as glBindFrameBuffer interface 1333' and glDrawArrays interface in the three-dimensional image processing library 1330 by hooking some interfaces in the three-dimensional image processing library 1330. For example, the hook module (HOOK System) 1331 hooks the glBindFrameBuffer interface 1333' in the 3D image processing library 1330 through the glBindFrameBuffer interface 1333, and can obtain the parameters for calling the glBindFrameBuffer interface 1333' in the 3D image processing library 1330.

In this embodiment of the present application, when the target application 1301 draws, the rendering system 1311 in the application engine 1310 may call the eglSwapBuffers interface 1332 and the glBindFrameBuffer interface 1333 and other interfaces in the hook module 1331 . Then the hook module (HOOK System) 1331 can obtain and call the parameters of interfaces such as glBindFrameBuffer interface 1333' and glDrawArrays interface in the three-dimensional image processing library 1330 by hooking some interfaces in the three-dimensional image processing library 1330 to realize the insertion into the target application 1301. The predicted frame is realized, and the motion vector is calculated according to the drawing frame of the target application 1301 and the predicted frame is obtained.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer may include drivers 1340 . The driver 1340 may include a variety of drivers for implementing the driver for the hardware device. For example, the driver 1340 may include a graphics memory driver 1341, a GPU driver 1342, and the like.

The hardware devices may include: a display device Display1350 , a graphics processor GPU1351 , a cache 1352 and an application processor 1353 . Display device 1350 may be display screen 194 shown in FIG. 12 . The graphics processor GPU 1351 and the application processor 1353 may be integrated in the processor 110 shown in FIG. 12 . The cache 1352 may be the internal memory 121 shown in FIG. 12 . Here the display device 1350 may refer to the description of the display screen 194 above. The graphics processor 1351 may refer to the above description of the GPU. The application processor 1353 may refer to the description of FIG. 12 above. The cache 1352 may refer to the description of the internal memory 121 above. It will not be repeated here.

In the following, the workflow of the software and hardware of the electronic device 100 is exemplarily described in conjunction with the capturing and photographing scene.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and then starts the camera driver by calling the kernel layer. The camera 193 captures still images or video.

In the embodiment of the present application, the color attachment C is the drawing result of the motion drawing object in the drawing instruction of the N+2th frame. For the calculation method of the motion vector of the color attachment C, reference may be made to the method described in the above-mentioned FIGS. 9A-9C . Optionally, the motion vector calculation method of color attachment C may also refer to FIG. 17-FIG. 22B for the calculation process of calculating the motion vector of the moving object.

In a possible implementation manner, the image frame prediction performed by the electronic device may specifically include: first, the electronic device divides the first drawing frame and the second drawing frame into R*S blocks respectively. Then, the electronic device determines, in the second rendering frame, a first matching block that matches the first candidate block in the first rendering frame. Next, the electronic device calculates the motion vector E from the candidate block to the first matching block. The electronic device determines, according to the motion vector E, a motion vector F of the first matching block from the second drawing frame to the first prediction frame. The electronic device draws the first matching block in the first prediction frame according to the motion vector F and the position of the first matching block in the second rendering frame. According to the above steps, the electronic device can determine a matching block to which each candidate block in the first rendering frame matches in the second rendering frame. The electronic device may finish rendering the first predicted frame. Here, the value of R*S is determined by the tile size and display screen resolution in the GPU of electronic devices (such as mobile electronic devices such as mobile phones and tablet computers). The GPU can divide the display into several tiles for rendering. The size of a tile is generally 16*16 or 32*32. If the display resolution of the electronic device is U*V (eg, 1580*720), then R*S is equal to (U/16)*(V/16) or (U/32)*(V/32). For example, the Wth drawing frame is divided into (1580/16)*(720/16) tiles. Further, the process of determining the first matching block matched with the first candidate block in the first drawing frame by the electronic device in the second drawing frame can refer to the description in the Wth frame and the W+2th frame above in Figures 9A-9C. , and will not be repeated here.

As shown in FIG. 14 , the electronic device (ie, tablet 10 ) displays a user interface 1400 . At time T0, the Wth image frame is displayed in the user interface 10. The W-th image frame is a drawing frame. The timing diagram 1403 in FIG. 14 shows the image frames that the electronic device can display from time T0 to time Tn.

FIG. 15 exemplarily shows a schematic diagram in the Wth frame divided into 16*16 blocks. In the embodiment of the present application, R*S is equal to 16*16 as an example. The Wth frame in Figure 15 is divided into 16*16 small blocks. Object 1501 and Object 1502 are each composed of a plurality of small blocks. The electronic device needs to calculate the motion vector of each block in the object 1501 from the Wth frame to the W+2th frame. Take candidate block 1511 in object 1501 as an example. The electronic device needs to determine the matching block that matches the candidate block 1511 in the W+2th frame. FIG. 16 exemplarily shows a diagram in the W+2th frame divided into 16*16 blocks. In the embodiment of the present application, R*S is equal to 16*16 as an example. The W+2th frame in Figure 16 is divided into 16*16 small blocks. The object 301 and the object 302 are respectively composed of a plurality of small blocks. The block 1611 in the object 1601 shown in FIGS. 15 and 16 matches the candidate block 1511 . However, the electronic device needs to perform a search to find a block 1611 that matches the candidate block 1511. The electronic device can use the diamond search algorithm to find the block 1611 that matches the candidate block 1511. Here, the specific process of the diamond search can refer to the description in FIG. 9B-FIG. 9C, which will not be repeated here.

In order to save the power consumption of the electronic device while improving the fluency of the application program video interface displayed by the electronic device, an embodiment of the present application provides a method for predicting an image frame. The method may include: first, the electronic device according to the first The drawing instruction determines the first moving object in the first drawing frame, and the second moving object is determined in the second drawing frame according to the second drawing instruction. Then, the electronic device determines that the first moving object and the second moving object match according to the attribute of the first moving object and the attribute of the second moving object. Next, the electronic device calculates the first coordinates of the center point of the first moving object and the second coordinates of the second moving object. Then, the electronic device determines a motion vector E from the first moving object to the second moving object according to the displacement from the first coordinate to the second coordinate. Finally, the electronic device determines the first pixel of the third moving object in the first predicted frame according to the motion vector E and the first pixel of the second moving object. In this way, the electronic device can insert a predicted image frame between every two drawing frames of the application program, which can improve the frame rate at which the electronic device displays the video interface of the application program. Therefore, the smoothness of displaying the video interface of the application program on the electronic device can be improved. Also, the electronic device can use the motion vector of the geometric center point of the object to represent the motion vector of the entire object, and it is not necessary to separately determine the motion vector of each block of the object. In this way, the calculation amount of the electronic device can be reduced, so that the power consumption of the electronic device can be saved.

In this embodiment of the present application, the attribute of the first motion object may be referred to as the first attribute, and the attribute of the second motion object may be referred to as the second attribute.

A method for predicting an image frame provided by an embodiment of the present application will be described in detail below with reference to the software and hardware structure of the above-mentioned exemplary electronic device 100 . The embodiments of the present application are described by taking the electronic device predicting the W+3 th predicted frame according to the W th rendering frame and the W+2 th rendering frame as an example. As shown in FIG. 17 , an image frame prediction method provided by an embodiment of the present application may include:

S701-S709: The electronic device draws the Wth drawing frame.

S701. When the target application is drawing, the CPU of the electronic device 100 acquires the drawing parameters of the Wth drawing frame.

The target application is an application with animation effects in the user interface, such as a game application. The embodiments of the present application are described below by taking the target application as a game application as an example. When the game application installed in the electronic device runs, the game application can call the drawing instruction to draw. The CPU of the electronic device 100 can obtain the Wth drawing frame of the application program through the interface in the three-dimensional image processing library. The drawing parameters of the Wth drawing frame are used to draw and render the Wth drawing frame. The drawing parameters of the Wth drawing frame may include attributes of each object in the Wth drawing frame, such as the vertex buffer ID, vertex number, texture ID, shader ID, transformation matrix, stencil buffer, etc. of each object.

Specifically, the CPU can obtain the drawing parameters of the Wth drawing frame of the application through the GL HOOK interface. As shown in Figure 18, the GL HOOK interface can obtain the drawing parameters of each image frame of the application from the game application.

Further, in some embodiments, the CPU can obtain the texture ID from OpenGL through the HOOKG1BindTexture interface and store it in the object structure. The CPU can obtain the vertex Buffer ID from OpenGL through HOOKGlBindBuffer and store it in the object structure. The CPU can obtain the data for calculating the number of vertices from OpenGL through the HOOKGlBufferData interface, and store the calculated number of vertices in the object structure. The CPU can obtain the transformation matrix of the object in the world coordinate system from OpenGL through the HOOKGlBufferSubData interface and store it in the global matrix variable.

Further, the object structure for storing object information can be defined according to the following code:

It can be understood that an object structure can be used to store the vertexBufferID, vertexNum, textureID of the object, the minimum value of the bounding box X-axis coordinate, the maximum value of the bounding box X-axis coordinate, the minimum value of the bounding box Y-axis coordinate, and the bounding box Y-axis. The maximum coordinate value, the X-axis coordinate of the center of the bounding box, and the Y-axis coordinate of the center of the bounding box. Here, the minimum value of the X-axis coordinate of the bounding box of the object is the minimum value of the X-axis coordinates of all the vertices of the object. The maximum value of the X-axis coordinate of the bounding box of the object is the maximum value of the X-axis coordinates of all the vertices of the object. The minimum value of the Y-axis coordinate of the bounding box of the object is the minimum value of the Y-axis coordinates of all the vertices of the object. The maximum value of the object's bounding box Y-axis coordinate is the maximum value of the Y-axis coordinates of all vertices of the object. The CPU can determine the X-axis coordinate of the center of the bounding box and the Y-axis coordinate of the center of the bounding box according to the minimum value of the X-axis coordinate of the bounding box, the maximum value of the X-axis coordinate of the bounding box, the minimum value of the Y-axis coordinate of the bounding box, and the maximum value of the Y-axis coordinate of the bounding box. It can be understood that the object structure in the embodiment of the present application is not limited to the object structure shown above, which is not limited in the embodiment of the present application.

It can be understood that, in this embodiment of the present application, the attribute of the object may include a transformation matrix, or may not include a transformation matrix. If the attribute of the object includes a transformation matrix, then the object is a moving object in this embodiment of the present application. As shown in FIG. 19A , the object (ie, the sphere) corresponding to the element 1401 in FIG. 14 is taken as an example. The sphere in (a) and (b) of FIG. 19A contains the transformation matrix, and the sphere is a moving object. If the attribute of the object does not include a transformation matrix, the object is a static object in this embodiment of the application. As shown in FIG. 19B , the object (ie, the rectangular parallelepiped) corresponding to the element 1402 in FIG. 14 is taken as an example. The cuboid in (a) and (b) of FIG. 19B does not contain a transformation matrix, and the cuboid is a static object. When the CPU determines that an object is a moving object, the moving object may be set to a first identifier. When the CPU determines that an object is a static object, the static object may be set with a second identifier. For example, the CPU may set the first value in the pixel bits of the moving object's stencil buffer. The CPU can set a second value in the pixel bits of the static object's stencil buffer. In some possible examples, the first value may be 1 and the second value may be 0.

In this embodiment of the present application, the electronic device 100 may be a smart phone, a tablet device, a smart watch, a computer, etc., which is not limited here.

S702 , the CPU of the electronic device 100 sends an instruction A for applying for a memory space to the GPU.

When or before the CPU acquires the drawing parameters of the image frame, the CPU of the electronic device 100 may send the instruction A for applying for memory space to the GPU. That is, when the target application starts to call the drawing instruction to draw, the CPU applies to the GPU for memory space A, and the memory space A is used to store the stencil buffer of the Wth drawing frame, that is, stencil buffer1. Stencil buffer (stencil buffer) or stencil buffer is another kind of data buffer other than color buffer, pixel buffer and depth buffer, which is common in computer graphics hardware such as OpenGL three-dimensional drawing. The stencil buffer is a pixel-based, integer-valued buffer, usually assigning a byte-length value to each pixel.

Here, the instruction A may be the instruction glGenTextures(1,&texture) and the instruction glBindTexture(target,texture). The CPU can send the instruction glGenTextures(1,&texture) and the instruction glBindTexture(target,texture) to the GPU to apply for memory space A from the GPU. It can be understood that the specific form of the instruction A is not limited in the embodiments of the present application.

S703. The GPU of the electronic device 100 allocates the memory space A for the template buffer of the Wth drawing frame.

In response to the instruction A sent by the CPU, the GPU allocates a memory space A for the stencil buffer of the Wth drawing frame. For example, as shown in FIG. 14 , in the time period from time T0 to time Tn shown in FIG. 14 , the drawing frames of the application are the Wth frame, W+2th frame, W+4th frame, W+6th frame, ..., W+nth frame. Memory space A stores the stencil buffer for the Wth drawing frame. After the GPU sends the Wth drawing frame to the display screen for display, the memory space A is used to store the stencil buffer of the W+4th drawing frame. After the GPU sends the W+2 drawing frame to the display screen for display, the memory space A is used to store the stencil buffer of the W+6 drawing frame. After the GPU sends the W+4 drawing frame to the display screen for display, the memory space A is used to store the stencil buffer of the W+8 drawing frame. After the GPU sends the W+6th drawing frame to the display screen for display, the memory space A is used to store the stencil buffer of the W+10th drawing frame. After all the drawing frames of the application are drawn, rendered and displayed on the display, the GPU can release the memory space A.

It can be understood that after the GPU sends the Wth drawing frame to the display screen for display, the memory space A is used to store the stencil buffer of the W+4th drawing frame, which specifically includes: when the GPU sends the Wth drawing frame to the display screen. After displaying, the GPU clears the stencil buffer of the Wth drawing frame in the memory space A, and stores the stencil buffer of the W+4th drawing frame.

Further, the memory space A may include a first template cache and a second template cache. The first template buffer is used for the GPU to draw the template image corresponding to the object of the Wth drawing frame. That is, the GPU can draw objects in the first stencil buffer. The second stencil buffer is used to back up the objects drawn in the first stencil buffer. The stencil image corresponding to the object may refer to an image drawn by the GPU according to the stencil buffer of the object, and the image is not colored by the shader corresponding to the object coloring ID. If the pixel bit in the stencil buffer of the object is the first value (for example, 1), that is, when the object is a moving object, the GPU can draw the template image of the moving object as a template image with the first color. If the pixel bit in the object's stencil buffer is a second value (eg, 0), the GPU can draw the stencil image of the still image as a stencil image with the second color. The first color and the second color are different. Optionally, the first color is black and the second color is white.

S704, the CPU of the electronic device 100 determines, according to the drawing parameters, the moving object contained in the Wth drawing frame and the attributes of the moving object.

The CPU of the electronic device 100 determines, according to the drawing parameters that can draw the Wth frame, the multiple objects included in the Wth drawing frame, and determines the attributes of the multiple objects. For example, if the drawing parameters of the Wth drawing frame include parameters of two objects (object A and object B), the CPU can determine the object A and the object B in the Wth drawing frame.

S705 , the CPU of the electronic device 100 establishes a first index table according to the motion objects included in the Wth drawing frame and the attributes of the motion objects.

The CPU may establish the first index table according to the multiple objects included in the Wth drawing frame and the attributes of the multiple objects. The first index table only stores the index information of the moving object in the Wth drawing frame (for example, the ID of the moving object and the attribute corresponding to the moving object). If there are only two moving objects in the Wth drawing frame, moving object 1 and moving object 2. Then, the first index table of the Wth drawing frame may be as shown in Table 1. The moving object 1 corresponds to the object ID 01 in Table 1, that is, it indicates that the ID of the moving object 1 is 01. Object ID01 may include attributes such as vertex buffer ID01, vertex count, texture ID01, shader ID01, transformation matrix M01, stencil buffer, and so on. The object ID01 is a moving object, so the attributes of the object ID01 include the transformation matrix M01. The moving object 2 corresponds to the object ID 11 in Table 1, that is, it indicates that the ID of the object 2 is 11. Object ID11 may include attributes such as vertex buffer ID11, vertex count, texture ID11, shader ID11, transformation matrix M11, stencil buffer, and so on. The object ID11 is a moving object, so the attributes of the object ID11 include the transformation matrix M11.

Table 1

S706, the CPU of the electronic device 100 sends the instruction B for memory mapping to the GPU.

The CPU of the electronic device 100 may send the instruction B for mapping the first index table to the GPU. Here, instruction B may be glMapBufferRange(). The CPU can let the GPU map the first index table by sending the instruction glMapBufferRange() to the GPU. The specific form of the instruction B is not limited in this embodiment of the present application.

S707. In response to the instruction B, the GPU of the electronic device 100 maps the first index table in the CPU.

In response to the instruction B, the GPU of the electronic device 100 may map the first index table in the CPU. Here, the CPU can store the first index table on the hardware, and the GPU maps the first index table in the CPU, which means that the GPU can access and read the first index table on the hardware.

S708: The CPU of the electronic device 100 sends a first drawing instruction for drawing the Wth drawing frame to the GPU.

The CPU may send a drawing instruction for drawing the Wth drawing frame to the GPU. The drawing instruction for the Wth drawing frame may be glDrawArrays(). That is, the CPU may send the instruction glDrawArrays( ) to the GPU, and in response to the instruction, the GPU starts to draw the Wth drawing frame. It can be understood that, the drawing instruction of the Wth drawing frame may include a plurality of instructions for drawing objects in the Wth drawing frame. The GPU may draw multiple objects in the Wth drawing frame according to multiple instructions for drawing objects.

S709, the GPU of the electronic device 100 draws the object in the wth drawing frame and draws the template image of the moving object in the memory space A.

The GPU can draw all objects in the Wth drawing frame and render them for display. For the process of drawing a complete drawing frame by the GPU, reference may be made to the process of GPU drawing in the prior art, and details are not described herein again.

The GPU can also draw a template image of the moving object in the memory space A, and the template image is used for the motion vector calculation in the subsequent steps. The specific process of the motion vector calculation is described in S719-S721, which will not be repeated here.

For the process of drawing the template image of the moving object by the GPU, reference may be made to the process of drawing the template image shown in FIGS. 20A-20C . As shown in (b) of FIG. 20A , the GPU may draw the template image of the object 1 of the Wth drawing frame in the first template buffer. Object 1 is a moving object, and the pixel bit of the template image of Object 1 drawn by the GPU is 1 (which can be indicated by black). As shown in (a) of FIG. 20A , the initial state of the second template cache is empty. That is, in the initial state, nothing is stored in the second template cache. After drawing an object in the first stencil buffer, the GPU saves the object drawn in the first stencil buffer into the second stencil buffer. As shown in (c) of FIG. 20B , the GPU saves the object drawn in the first stencil buffer to the second stencil buffer. As shown in (d) of FIG. 20B , the GPU may draw the template image of the object 2 of the Wth drawing frame in the first template buffer. Object 2 is a static object, so the pixel bit of the template image of Object 2 drawn by the GPU is 0 (which can be indicated by white). As shown in (e) of FIG. 20C , the GPU may again save the stencil image of the object 2 drawn in the first stencil buffer to the second stencil buffer. Here, the W-th drawing frame may be the W-th drawing frame shown in FIG. 15 above. Object 1 may be a sphere object corresponding to element 1501 in FIG. 15 . Object 2 may be a cuboid object corresponding to element 1502 .

Optionally, the GPU may only draw the template image of the moving object of the Wth drawing frame in the first template buffer.

S710-S718: The electronic device draws the W+2th drawing frame.

S710. The CPU of the electronic device 100 acquires the drawing parameters of the W+2 th drawing frame.

The CPU can obtain the drawing parameters of the W+2 drawing frame of the application through the GL HOOK interface. The drawing parameters of the W+2 th drawing frame are used to draw and render the W+2 th drawing frame. The drawing parameters of the W+2th drawing frame may include object attributes of each object in the W+2th drawing frame, such as the vertex buffer ID, vertex number, texture ID, shader ID, transformation matrix, stencil buffer, etc. of each object and other properties. Here, for details, reference may be made to the description in step S701, which is not repeated here.

S711 , the CPU of the electronic device 100 sends an instruction C for applying for a memory space to the GPU.

The CPU of the electronic device 100 may send the instruction C to apply for memory space to the GPU. Instruction C can be used to request memory space B. Memory space B is used to store the stencil buffer of the W+2 drawing frame, that is, stencil buffer2. Here, for the description of the instruction C, reference may be made to the description of the instruction A, which will not be repeated here. For step S711, reference may be made to the description of step S702, which will not be repeated here.

S712. The GPU of the electronic device 100 allocates a memory space B for the stencil buffer of the W+2 th drawing frame.

In response to the instruction C sent by the CPU, the GPU allocates a memory space B for the stencil buffer of the Wth drawing frame. For example, as shown in FIG. 14 , in the time period from time T0 to time Tn shown in FIG. 14 , the drawing frames of the application are the Wth frame, W+2th frame, N+4th frame, Frame N+6, ..., frame N+n. Memory space B stores the stencil buffer of the W+2 drawing frame. After the GPU sends the W+2 drawing frame to the display screen for display, the memory space B is used to store the stencil buffer of the N+6 drawing frame.

After the GPU sends the W+2 drawing frame to the display screen for display, the memory space B is used to store the stencil buffer of the N+6 drawing frame, which specifically includes: after the GPU sends the Wth drawing frame to the display screen for display , the GPU clears the stencil buffer of the W+2th drawing frame in the memory space B, and stores the stencil buffer of the N+6th drawing frame.

Further, the memory space B may include a third template cache and a fourth template cache. The third stencil buffer is used for the GPU to draw the stencil image corresponding to the object of the W+2 th drawing frame. That is, the GPU can draw objects in the third stencil buffer. The fourth stencil buffer is used to back up the objects drawn in the first stencil buffer.

Here, step S712 refers to the description in step S703, and details are not repeated here.

Optionally, the GPU may only draw the template image of the moving object of the W+2 th drawing frame in the third template buffer.

S713 , the CPU of the electronic device 100 determines, according to the drawing parameters, the moving objects included in the W+2 th drawing frame and the attributes of the moving objects.

The CPU of the electronic device may determine, according to the drawing parameters of the W+2 th drawing frame, multiple objects included in the W+2 th drawing frame, and determine the attributes of the multiple objects. The CPU can determine the moving object in the W+2th drawing frame and the attributes of the moving object. In step 713, reference may be made to the description in step S704, which is not repeated here.

S714 , the CPU of the electronic device 100 establishes a second index table according to the moving objects included in the W+2 th drawing frame and the attributes of the moving objects.

The CPU may establish the second index table according to the multiple objects included in the W+2 th drawing frame and the attributes of the multiple objects. For example, if the W+2-th drawing frame only includes two moving objects, such as moving object 1 and moving object 2 . Then, the second index table of the Wth drawing frame may be as shown in Table 2. The moving object 1 corresponds to the object ID 02 in Table 2, that is, it indicates that the ID of the moving object 1 is 02. Object ID02 may include attributes such as vertex buffer ID02, vertex count, texture ID02, shader ID02, transformation matrix M02, stencil buffer, and so on. The object ID02 is a moving object, so the attributes of the object ID02 include the transformation matrix M02. The moving object 2 corresponds to the object ID 12 in Table 1, that is, it indicates that the ID of the object 2 is 12. Object ID12 may include attributes such as vertex buffer ID12, vertex count, texture ID12, shader ID12, transformation matrix M12, stencil buffer, and so on. The object ID12 is a moving object, so the attributes of the object ID12 include the transformation matrix M12.

Table 2

S715 , the CPU of the electronic device 100 sends the instruction D for memory mapping to the GPU.

The CPU can also send instruction D to the GPU. For the instruction D, reference may be made to the description of the instruction B above, which will not be repeated here. Instruction D is used to instruct the GPU to map the second index table in the CPU. Here, step S715 may be step S706, which will not be repeated here.

S716. In response to the instruction D, the GPU of the electronic device 100 maps the second index table in the CPU.

In response to instruction D, the GPU may map the second index table in the CPU. That is, the GPU can access and read the second index table stored by the CPU in some hardware. The second index table may be the object index table shown in Table 2 above. Step S716 may refer to step S707, which will not be repeated here.

S717: The CPU of the electronic device 100 sends a drawing instruction for drawing the W+2 th drawing frame to the GPU.

The CPU may send a drawing instruction to the GPU for drawing the W+2 th drawing frame. For the drawing instruction of the W+2 th drawing frame, reference may be made to the description of the drawing instruction of the W th drawing frame. In response to the drawing instruction of the W+2 th drawing frame, the GPU starts to draw the W+2 th drawing frame. It can be understood that the drawing instruction of the W+2 th drawing frame may include a plurality of instructions for drawing objects in the W+2 th drawing frame. That is, when there are multiple objects in the W+2 th drawing frame, the GPU may draw the multiple objects in the W+2 th drawing frame according to multiple instructions for drawing objects.

S718 , the GPU of the electronic device 100 draws the object in the W+2 th drawing frame and the template image of the moving object in the memory space B.

The GPU can draw all objects in the W+2 drawing frame and render them for display. For the process of drawing a complete drawing frame by the GPU, reference may be made to the process of GPU drawing in the prior art, and details are not described herein again.

The GPU can also draw a template image of the moving object in the memory space B, and the template image is used for the motion vector calculation in the subsequent steps. It can be understood that the moving object in step S718 refers to the moving object in the W+2 drawing frame. The specific process of the motion vector calculation is described in S719-S721, which will not be repeated here.

For the process of drawing a template image of a moving object by the GPU, reference may be made to the process of drawing a template image shown in FIGS. 21A-21C . As shown in (b) of FIG. 21A , the GPU may draw the template image of the object 1 of the W+2-th drawing frame in the third template buffer. Object 1 is a moving object, so the pixel bit of the template image of Object 1 drawn by the GPU is 1 (it can be indicated by black). As shown in (a) of FIG. 21A , the initial state of the fourth template cache is empty. That is, in the initial state, nothing is stored in the second template cache. After drawing an object in the third stencil buffer, the GPU saves the object drawn in the third stencil buffer into the fourth stencil buffer. As shown in (c) of FIG. 21B , the GPU saves the object drawn in the third stencil buffer into the fourth stencil buffer. As shown in (d) of FIG. 21B , the GPU may draw the template image of the object 2 of the W+2 th drawing frame in the first template buffer. Object 2 is a static object, so the pixel bit of the template image of Object 2 drawn by the GPU is 0 (which can be indicated by white). As shown in (e) of FIG. 21C, the GPU may again save the stencil image of the object 2 drawn in the third stencil buffer to the fourth stencil buffer. Here, the W+2 th drawing frame may be the N+3 th drawing frame shown in FIG. 16 above. Object 1 may be a sphere object corresponding to element 1601 in FIG. 16 . Object 2 may be the cuboid object corresponding to element 1602 .

S719-S724: Predict the W+3 th prediction frame.

S719. The CPU in the electronic device 100 sends an instruction for instructing the GPU to calculate the motion vector to the GPU.

The CPU in the electronic device 100 may send an instruction to the GPU for instructing the GPU to calculate the motion vector. This instruction is used to instruct the shader in the GPU to calculate the motion vector. The directive can be a dispatch directive. The embodiment of the present application does not limit the specific form of the instruction for calculating the motion vector.

It can be understood that, during the time period when the electronic device displays the video interface of the application, the CPU may cyclically send an instruction for calculating the motion vector to the GPU. After the GPU draws a frame, the CPU can send an instruction to the GPU for instructing the GPU to calculate the motion vector once. That is, after step S709, the CPU can also execute step S719.

S720-S722: Calculate the motion vector.

S720. The GPU in the electronic device 100 determines the first moving object in the first index table, and the first moving object matches the second moving object in the second index table.

The GPU can now retrieve the second moving object from the second index table, and then go to the first index table to find the first moving object that matches the second moving object.

If there is only one moving object in the second index table corresponding to the W+2 th drawing frame, the moving object is the second moving object. The GPU may determine the moving object and the static object according to the identifier carried by each object in the W+2th drawing frame. That is, the object carrying the first identification is a moving object, and the object carrying the second identification is a static object. Only the moving objects in the N+2 drawing frames are stored in the second index table. The second moving object may be any moving object in the second index table. The W+2 th drawing frame may contain multiple objects. If there is only one moving object among the multiple objects, the moving object is the second moving object. In a possible implementation manner, the GPU may retrieve the second moving object from the second index table in the W+2 th drawing frame. Then the GPU finds the first moving object matching the second moving object in the first index table of the Wth drawing frame.

Here, the matching of the first moving object and the second moving object means that the M attributes of the first moving object are the same as the M attributes of the second moving object. Here, M is a positive integer, for example, the value of M can be 1, 2, 3 or other integers. The value of M is not limited. For example, the number of vertices, texture ID, shader ID, transformation matrix, stencil buffer, etc. of the first moving object, and the number of vertices, texture ID, shader ID, transformation matrix, stencil buffer, etc. attributes of the second moving object same. It can be understood that, in the image frame sequence of the target application, the first moving object and the second moving object may be the same object. However, the coordinates of the pixels of the first moving object in the Wth drawing frame are different from the coordinates of the pixels of the second moving object in the W+2th drawing frame. In a feasible implementation manner, the GPU may determine that the first moving object matches the second moving object according to the following code.

/ / Determine whether the index of the two objects is the same, whether the representation is the same object

S721, the GPU in the electronic device 100 calculates the first coordinate of the geometric center point of the first moving object in the Wth drawing frame according to the template image of the first moving object, and calculates the second moving object according to the template image of the second moving object The second coordinate of the geometric center point in the W+2th drawing frame.

The GPU of the electronic device 100 can obtain the coordinates of all the pixels of the first moving object by using the shader according to the template image of the first moving object. First, the GPU can determine the bounding box of the first moving object according to the minimum value X1min of the X axis, the maximum value X1max of the X axis, the minimum value Y1min of the Y axis, and the maximum value Y1max of the Y axis in the coordinates of all the pixel points of the first moving object. . The coordinates of the four vertices of the bounding box of the first moving object may be respectively (X1min, Y1min), (X1max, Y1min), (X1min, Y1max), and (X1max, Y1max). The GPU may determine the first coordinate of the first moving object according to the bounding box of the first moving object. The first coordinate is ((X1min+X1max)/2, (Y1min+Y1max)/2).

The GPU of the electronic device 100 can use the shader to obtain the coordinates of all the pixels of the first moving object according to the template image of the second moving object. First, the GPU can determine the bounding box of the second moving object according to the minimum value X2min of the X axis, the maximum value X2max of the X axis, the minimum value Y2min of the Y axis, and the maximum value Y2max of the Y axis in the coordinates of all the pixel points of the second moving object. . The coordinates of the four vertices of the bounding box of the second moving object may be respectively (X2min, Y2min), (X2max, Y2min), (X2min, Y2max), and (X2max, Y2max). The GPU may determine the second coordinates of the second moving object according to the bounding box of the second moving object. The second coordinate is ((X2min+X2max)/2, (Y2min+Y2max)/2).

S722, the GPU in the electronic device 100 determines the motion vector E of the first moving object from the Wth drawing frame to the W+2th drawing frame according to the displacement from the first coordinate to the second coordinate, and determines the second motion according to the motion vector E The motion vector F of the object from the W+2-th drawn frame to the N+3-th predicted frame.

The GPU may determine the motion vector E for moving the first moving object to the second moving object according to the displacement from the first coordinate to the second coordinate. Then, the GPU determines the motion vector F of the second moving object from the W+2 th drawing frame to the N+3 th prediction frame according to the motion vector E. The motion vector F is equal to K times the motion vector E. K is greater than 0 and less than 1. Optionally, K=0.5. That is, in the image frame sequence of the application, the movement of the same moving object is uniform. That is to say, the speed of the first moving object moving from the first frame to the second frame is the same as the speed of moving from the second frame to the third frame.

As shown in FIG. 22A , the first moving object may be the moving object 2201 illustrated in (a) of FIG. 22A . The second moving object may be the moving object 2202 illustrated in (b) of FIG. 22A . As shown in (c) of FIG. 22B , the motion vector E of the first moving object moving to the second moving object is the motion vector of moving the moving object 2201 to the moving object 2202 in (c) of FIG. 22B . As shown in (d) of FIG. 22B , the geometric center point of the moving object 501 is the point 2203 . The geometric center point of the moving object 2202 is point 2204 . The motion vector from the moving object 2201 to the moving object 2202 is the motion vector 2205 from the point 2203 to the point 2204 . Motion vector E may be motion vector 2205 in Figure 22B. The motion vector F may be K times the motion vector 2205.

S723. The CPU in the electronic device 100 sends an instruction E for drawing the N+3 th predicted frame to the GPU.

After the GPU calculates the motion vector F, the CPU of the electronic device 100 may send an instruction E for drawing the N+3 th predicted frame to the GPU. After the GPU receives the instruction E, in response to the instruction E, the GPU draws the N+3 predicted frame. It can be understood that the instruction E may include multiple instructions for drawing objects, that is, the GPU may draw multiple objects in the N+3 predicted frame according to the multiple instructions for drawing objects.

S724 , the GPU in the electronic device 100 draws the third moving object in the N+3 th prediction frame according to the motion vector F and the coordinates of each pixel point of the second moving object in the W+2 th drawing frame.

In response to instruction D, the GPU draws the N+3 predicted frame. Specifically, the GPU draws the third moving object in the N+3 prediction frame according to the motion vector F and the coordinates of each pixel of the second moving object in the W+2 drawing frame, wherein the second moving object and the second moving object Motion object matching. That is, each pixel in the second moving object is shifted by the motion vector F, that is, the pixel constituting the third motion vector. The third motion vector and the motion vector F have the same number of vertices, the same shader ID, the same texture ID, and the same attributes such as transformation matrices.

It can be understood that after the electronic device draws the Wth drawing frame, that is, after the electronic device completes step S709, the electronic device will also draw the W+1th frame. The W+1th frame may be a drawing frame or a prediction frame. As shown in Figure 11, if the W-th drawing frame is the 0th frame, then the W+1-th frame is the first frame, then the W+1-th frame is the W+1-th drawing frame, then the electronic device can follow the steps The drawing process shown in S701-S709 draws the W+1 th drawing frame. If the W-th drawing frame is the second frame shown in FIG. 11 , then the W+1-th frame is the third frame in FIG. 11 , and at this time, the W+1-th frame is the W+1-th predicted frame. The electronic device may draw the W+1 th predicted frame according to the steps shown in steps S719 to S724.

It can be understood that, the electronic device can repeatedly perform steps S719 to 722 until the motion vector of each moving object is predicted.

It can be understood that the embodiment of the present application is not limited to the electronic device predicting the W+3 th predicted frame by using the W th rendering frame and the W+2 th rendering frame. Optionally, the electronic device may also predict the W+2 th frame by using the W th frame and the W+1 th frame. Optionally, the electronic device may also predict the W+4th frame by using the Wth frame and the W+3th frame. This embodiment of the present application does not limit this.

It can be understood that, with respect to the electronic device predicting the W+3 prediction frame through the Wth drawing frame and the W+2 drawing frame, the electronic device predicts the W+2th frame through the Wth frame and the W+1th frame. The computing power of the GPU is required to be higher, that is, the GPU needs to calculate the motion vector and draw the W+2th frame before the W+1th frame is displayed. If the computing power of the GPU does not meet the requirements, the video interface of the electronic device will freeze after the electronic device displays the N+1th frame.

It can be understood that, when the electronic device displays the video interface of the target application, the electronic device can predict the predicted frame according to different strategies. That is, the strategy of the electronic device predicting the N+3th frame according to the Wth frame and the W+2th frame in the first time period, and the strategy of predicting the W+2th frame according to the Wth frame and the W+1th frame in the second time period. . For example, when the GPU performs many tasks, the electronic device may first predict the W+3 th frame according to the W th frame and the W+2 th frame. When the GPU performs few tasks, the electronic device may predict the W+2 th frame according to the W th frame and the W+1 th frame. This embodiment of the present application does not limit this.

By implementing the image frame prediction method provided by the embodiment of the present application, the electronic device can determine the first moving object in the first drawing frame according to the first drawing instruction, and determine the second moving object in the second drawing frame according to the second drawing instruction moving objects. Then, the electronic device determines that the first moving object and the second moving object match according to the first attribute of the first moving object and the second attribute of the second moving object. Next, the electronic device calculates the first coordinates of the center point of the first moving object, and calculates the second coordinates of the second moving object. Then, the electronic device determines a motion vector E from the first moving object to the second moving object according to the displacement from the first coordinate to the second coordinate. Finally, the electronic device determines the first pixel of the third moving object in the first predicted frame according to the motion vector E and the first pixel of the second moving object. In this way, the electronic device can insert a predicted image frame between every two drawing frames of the application program, which can improve the frame rate at which the electronic device displays the video interface of the application program. Therefore, the smoothness of displaying the video interface of the application program on the electronic device can be improved. Also, the electronic device can use the motion vector of the geometric center point of the object to represent the motion vector of the entire object, and it is not necessary to separately determine the motion vector of each block of the object. In this way, the calculation amount of the electronic device can be reduced, so that the power consumption of the electronic device can be saved.

The following describes the relevant modules of the image frame prediction apparatus for implementing an image frame prediction method provided by the present application with reference to FIG. 23 . As shown in Figure 23, the image frame prediction apparatus may include the following modules: HOOK system module 2300, Frame buffer Object Composition module 2310, UI Separating module 2320, Motion vector computation Module 2330 , Frame Prediction module 2340 , Debug system module 2350 , Frame Rate Control module 2360 . in:

The hook module HOOK system2300 is used to call the relevant interface of the 3D graphics processing library to obtain the drawing parameters of the target application. Here, reference may be made to the description of the hook module 1331 in FIG. 13 , which will not be repeated here.

The frame synthesis module 2310 is used for synthesizing the separated UI controls and the interface content of the target application into one frame of data.

The UI control separation module 2320 is used to separate the UI controls from the drawing parameters obtained from the target application. Here UI controls refer to some buttons, input boxes, etc. in the user interface of the target application that can be used to receive user operations. The UI control separation module 2320 may include a UI control detection (UI Detection) module 2321 and a UI frame buffer dump (UI Frame buffer Dump) module 2322. Among them, the UI control detection module 2321 is used to detect UI controls. The UI frame buffer dump module 2322 is used to store the UI controls detected by the UI detection module 2321 .

The motion vector calculation module 2330 is used to calculate the motion vector of the image frame. The motion vector calculation module 2330 may include: a diamond search algorithm (Diamond Search Algorithm) module 2331, a reprojection algorithm (Reprojection Algorithm) module 2332, and an object based algorithm (Object Based Algorithm) module 2333. The diamond search algorithm module 2331 is used to calculate the motion vector of each small block in the image frame through the diamond search algorithm. The reprojection algorithm module 2332 is used to calculate motion vectors of static objects in the image frame. The object-based algorithm module 2333 is used to calculate the motion vector of the moving object in the image frame.

The frame prediction module 2340 can be used to predict one image frame according to the two-frame rendering frame of the target application. The predicted image frame is the predicted frame in this embodiment of the present application.

The system debugging module 2350 is used to output test data, dump the predicted frame, and the drawing and debugging system module 2350 may include a Debug Power Key module 2351, a Motion Vector Showing module 2352, and a UI Information (UI Information) module. ) module 2353, frame dump (FrameDump) module 2354. The power button debugging module 2351 can be used to trigger the switching of the diamond search algorithm module 2331 , the reprojection algorithm module 2332 , the object-based algorithm module 2333 and other modules in the motion vector calculation module 2330 . The motion vector display module 2352 can be used to draw motion vector arrows on the frame data to visualize the motion vectors. The UI information module 2353 can be used to display information such as frame rate. The frame dump module 2354 can be used to dump the predicted frame data into an image file for debugging.

The frame rate control module 2360 is used to control the frame rate of the image frame of the target application program displayed by the electronic device.

With the rapid development of multimedia technology, people have higher requirements for video resolution. However, the current frame rate of many videos is only 30 frames per second. When people watch such videos, problems such as visual perception will appear, which makes it difficult to meet users' requirements for videos. Generally, the higher the frame rate of the video, the better the performance of its fluency and smoothness. Therefore, the existing technology can increase the frame rate of the video by inserting the predicted frame into the original frame of the video stream, and improve the smoothness of the video. , to improve the user experience.

In a possible implementation manner, the electronic device generally uses a block matching algorithm to generate the predicted frame. Specifically, after the electronic device divides the video frame into blocks, the motion vector of the block is directly applied to all pixels of the block to obtain the predicted frame. However, when the motion vectors of adjacent blocks are far apart, this method will cause holes and blurred images in the predicted frame.

Embodiments of the present application provide a frame prediction method, an electronic device, and a computer-readable storage medium. In the method, the electronic device first, for each vertex of a block, according to the block around the vertex from the target reference frame to the prediction frame. The predicted motion vector determines the position of the vertex in the predicted frame, and further, for each block, the corresponding relationship between the four vertices of the block in the reference frame and the predicted frame is used to calculate the position of each pixel of the block in the predicted frame. position, and finally get the entire predicted frame.

The following introduces some related concepts in the embodiments of the present application.

1. Video post-processing technology

In the embodiments of the present application, the video post-processing technology refers to a video processing technology that processes the collected video through a video post-processing algorithm to obtain a video effect that is improved in a certain aspect, including video stabilization technology, Video high dynamic range (HDR) processing technology and frame rate conversion technology, etc. Among them, the video post-processing algorithm is used to process the collected multiple pictures or videos.

In this embodiment, the video post-processing algorithm may be invoked by the image processing module provided in this embodiment of the present application to process the collected pictures or videos.

2. Frame rate

The frame rate is the number of frames per second in the video, that is, the number of times the video is updated per second. Due to the persistence of vision mechanism, when images are played continuously at 16 frames per second and above, the image sequence will achieve a visual effect that looks smooth and coherent. In general, normal resolution video up to 30 frames per second is considered acceptable, and video with frame rates up to 60 frames per second is considered high definition video. Generally, the higher the frame rate of the video, the better the smoothness and smoothness of the video.

With the rapid development of intelligent terminals and multimedia technologies, people have higher requirements for video resolution. At present, the frame rate of many videos is only 30 frames per second. When people watch this kind of video, there will be problems such as stuttering in visual perception, and the user experience will be poor. Therefore, the electronic device can use the video frame rate conversion technology to interpolate the low frame rate video into the high frame rate video. For example, the electronic device can increase the video frame rate from 30 frames per second to 60 frames per second, making the video more smooth and continuous, so as to enhance the realism and interaction of people watching videos.

3. Frame rate conversion technology

Frame rate conversion technology is mainly divided into two categories, one is a non-motion compensation algorithm, and the other is an algorithm based on motion estimation and motion compensation.

Among them, non-motion compensation algorithms include simple frame repetition and frame averaging algorithms. Specifically, the frame repetition algorithm completes the frame interpolation by directly copying the frames in the input video sequence. This method is simple and efficient, but the quality of the interpolation frames for moving objects in the video is poor, and there will be very obvious blurring and blurring. Overlapping problem; the frame averaging algorithm synthesizes intermediate frames by weighting and averaging between adjacent frames in the input video sequence according to certain rules. This method has low time and space complexity, but for scenes with motion, interpolation Obvious aliasing problems with moving objects in frame images.

When processing a video with a motion scene, in order to synthesize high-quality interpolated frames, the electronic device may use an algorithm based on motion estimation and motion compensation to generate interpolated frames. The main idea of the algorithm based on motion estimation and motion compensation is to capture the motion information of objects in the video frame sequence, and then fuse and synthesize the inserted frames according to the motion information and the input adjacent frames.

4. Block matching algorithm

Algorithms based on motion estimation and motion compensation include Block Matching Algorithm (BMA). The main steps of the algorithm are: firstly, the original video frame is divided into several pixel blocks, and it is assumed that all pixels in the pixel block have the same displacement, and then, according to a certain matching search principle, find the matching current pixel block in the reference frame. block, and calculate the relative displacement between the two matching blocks as the motion vector of each block between the two frames.

In the prior art, the main idea of the frame prediction method based on the block matching algorithm is that after the electronic device obtains the predicted motion vector of each block in the reference frame, for each block, the predicted motion vector of the block is directly applied to the block. On all pixels of , the corresponding block in the predicted frame is obtained, and finally the entire predicted frame is obtained, wherein the predicted motion vector is the motion vector of the block from the reference frame to the predicted frame. When the motion vector gap between adjacent blocks is large, this method will cause local discontinuity in the predicted frame, resulting in a hole effect.

The frame prediction method provided by the embodiment of the present application is described in detail below with reference to FIG. 12 .

This method can be implemented by the electronic device 100 shown in FIG. 12 . Please refer to FIG. 24 . FIG. 24 is a flowchart of a frame prediction method disclosed in an embodiment of the present application. As shown in FIG. 24 , the frame prediction method includes: Some or all of the following steps:

S101. Acquire a first reference frame and a second reference frame.

Specifically, the electronic device may acquire two adjacent frames of images from a video stream, and determine the acquired two adjacent frames of images as a first reference frame and a second reference frame, wherein the video stream may be video, or It can be a game or other multimedia resources, which is not limited here.

Specifically, please refer to FIG. 25 , which is a schematic diagram of obtaining a reference frame disclosed in an embodiment of the present application, wherein the horizontal axis represents the time of the video stream, and the arrows on the horizontal axis represent the video frames in the video stream. As shown in the figure, the video stream is an image sequence composed of multiple video frames arranged in time sequence. Specifically, the video stream includes multiple frames of images such as the first frame, the second frame, the third frame, and the Nth frame. . For example, the electronic device may acquire the second frame and the third frame in the video stream, determine the second frame in the video stream as the first reference frame, and determine the third frame in the video stream as the second reference frame.

Please refer to FIG. 26. FIG. 26 is another schematic diagram of obtaining a reference frame disclosed by an embodiment of the present application. Among them, (A) in Figure 26 represents a video of a ball moving, specifically, there is a ball in the video screen, the ball moves in the direction of the virtual image, the dashed arrow represents the direction of the ball movement, and the video background is a still image with a radial pattern , the line at the top of the ball is a reference line indicating the position of the ball; the images of (B) and (C) in Figure 26 are two reference frames obtained by the electronic device from the video respectively, wherein (B) in Figure 26 ) is the first reference frame, (C) in FIG. 26 is the second reference frame, as shown in the figure, taking the line at the top of the ball as a reference, the ball in the first reference frame is at the eleventh position, and the second reference frame The ball in the twelfth position is in the twelfth position. It can be known from the relative position of the ball in the first reference frame and the second reference frame. The ball in the first reference frame and the second reference frame is moving relative to the background. There is a displacement from the reference frame to the second reference frame.

S102. Determine a target reference frame from the first reference frame and the second reference frame according to the position of the frame to be predicted.

Specifically, the electronic device may determine a frame of image from the first reference frame and the second reference frame as the target reference frame according to the position of the frame to be predicted. It should be noted that the position of the frame to be predicted can be determined according to the application scenario and user requirements. For example, in order to improve the frame rate of the game, the electronic device can generate the next frame of the first two frames of images according to the first two frames of images. Too long game delay can be avoided. For another example, when processing video without real-time requirements, the electronic device can generate an intermediate image according to two adjacent video frames, and the predicted frame generated by this method is more accurate.

Please refer to FIG. 27. FIG. 27 is a schematic diagram of determining a target reference frame disclosed by an embodiment of the present application. Wherein, as shown in (A) in FIG. 27 , the electronic device may determine the second frame and the third frame in the video stream as the first reference frame and the second reference frame, and the electronic device determines the position of the frame to be predicted according to the position of the frame to be predicted. The target reference frame includes two cases as shown in (B) and (C) in FIG. 27 . Wherein, as shown in (B) in FIG. 27 , when the position of the frame to be predicted is between the first reference frame and the second reference frame, the electronic device can determine that the first reference frame is the target reference frame, and accordingly, The second reference frame is a matching frame; as shown in (C) in FIG. 27 , when the position of the frame to be predicted is located between the second reference frame and the fourth frame, the electronic device can determine that the second reference frame is the target Reference frame, correspondingly, the first reference frame is a matching frame. It should be noted that the position of the predicted frame may be located at a middle position between the two frames of images, or may be located at other positions between the two frames of images, which is not limited here.

S103. Divide the target reference frame into blocks according to the square blocks of the first size.

Specifically, the electronic device may firstly divide the target reference frame, wherein the size of the division may be 4×4, 8×8, and 16×16. Please refer to FIG. 28. FIG. 28 is a schematic diagram of dividing a target reference frame according to an embodiment of the present application. As shown in the figure, (A) in FIG. 28 is the target reference frame. After the electronic device divides the target reference frame into blocks, the divided target reference frame as shown in (B) in FIG. 28 can be obtained. In order to better illustrate the situation of the reference frame after the block, the area of the edge of the ball is taken to illustrate the scheme. After the area of the edge of the ball is enlarged, it can be shown in (C) in FIG. Irregular linear regions are stationary background regions. It should be noted that (C) in Figure 28 is an enlarged ball, and the fringe interval of the ball in (C) in Figure 28 is wider than that in (A) and (B) in Figure 28, because The reasons for the drawing are not clearly reflected in Figure 28, and are hereby explained.

S104: Calculate the motion vector of the first block from the target reference frame to the predicted frame, where the first block is a block in the target reference frame.

Specifically, the electronic device may first obtain the motion vector of the block from the target reference frame to the matching frame, and then calculate the predicted motion vector of the block from the target reference frame to the predicted frame according to the motion vector of the block from the target reference frame to the matching frame. For example, when the target reference frame is the first reference frame, the electronic device may first obtain the motion vector of the block from the first reference frame to the second reference frame, and then calculate the motion vector of the block from the first reference frame to the second reference frame according to the motion vector of the block from the first reference frame to the second reference frame. The predicted motion vector of the block from the first reference frame to the predicted frame.

Please refer to FIG. 29. FIG. 29 is a schematic flowchart of calculating a predicted motion vector of a block from a target reference frame to a predicted frame disclosed in an embodiment of the present application. Step S104 may include some or all of the following steps:

S1041. Obtain the motion vector of the block from the target reference frame to the matching frame.

Specifically, the electronic device can determine the matching block of the block in the matching frame according to the block in the target reference frame, wherein the area corresponding to the block in the target reference frame in the matching frame is the matching block, and then according to the target reference frame The displacement of the block in to the matching block of the block in the matching frame, determines the motion vector of the block from the target reference frame to the matching frame. For example, when the target reference frame is the first reference frame, the electronic device may acquire the motion vector of each block in the first reference frame from the first reference frame to the second reference frame. For another example, the target reference frame is the second reference frame When , the electronic device may acquire a motion vector of each block in the second reference frame from the second reference frame to the first reference frame.

Please refer to FIG. 30. FIG. 30 is a schematic diagram of an acquisition block from a target reference frame to a matching frame disclosed in an embodiment of the present application. As shown in the figure, (A) in FIG. 30 is the target reference frame, and according to the blocks in the target reference frame, the matching block determined in the matching frame may be as shown in (B) in FIG. 30 . Further, according to (A) and (B) in FIG. 30 , a motion vector as shown in (C) in FIG. 30 can be obtained, wherein the arrow indicates the displacement of the block from the target reference frame to the prediction frame. As shown in (C) of FIG. 30 , there are three blocks in the figure that have displacements from the target reference frame to the matching frame, and the electronic device can determine the displacement of the block in the target reference frame to the matching block of the block in the matching frame as The motion vector of the block from the target reference frame to the matching frame, the motion vector of other blocks without displacement in the schematic diagram is 0.

The method for the electronic device to find the matching block in the matching frame according to the block in the target reference frame includes a full search method (Full Search Method, FS), a three-step method, a diamond search method, and the like. Specifically, the full search method calculates the absolute error sum for all reference pixel positions within the search range, and the motion vector is the offset corresponding to the absolute error and the minimum point. Since most of the best matching points appear near the center of the search area, the FS algorithm mainly adopts a spiral search sequence centered on the area, with the center point as the starting point, and spirals from the inside to the outside in the current frame. Search until all target pixels in the macroblock are traversed. It should be noted that, the blocks determined by the electronic device in the matching frame may overlap, and accordingly, there may be some regions in the matching frame that are not determined as matching blocks.

In some other embodiments, the electronic device may also obtain the motion vector of the block from the target reference frame to the matching frame according to the hardware device, and there may be other methods to obtain the motion vector of the block from the target reference frame to the matching frame, which is not limited here.

S1042. Determine the predicted motion vector of the block from the target reference frame to the predicted frame according to the motion vector of the block from the target reference frame to the matching frame.

In one implementation, when the target reference frame is the first reference frame, the electronic device may determine the predicted motion vector of the block from the first reference frame to the predicted frame according to the motion vector of the block from the first reference frame to the second reference frame.

In some embodiments, the motion between blocks may be considered as uniform motion, and it is understood that, in this case, the electronic device may determine half of the motion vector of the block from the first reference frame to the second reference frame as The predicted motion vector for this block. Specifically, please refer to FIG. 31A , which is a schematic diagram of determining a predicted motion vector of a block according to an embodiment of the present application. As shown in the figure, a block in the target reference frame is schematically drawn, wherein Mv is the motion vector of the block from the first reference frame to the second reference frame, represented by a solid arrow; Mv' is the block The predicted motion vector from the first reference frame to the predicted frame, indicated by dashed arrows. As shown in the figure, the motion vector of the block from the first reference frame to the second reference frame is (3, 5), and the electronic device may determine half of the predicted motion vector of the block from the target reference frame to the predicted frame as the prediction of the block The motion vector, that is, according to Mv'=1/2Mv, the predicted motion vector of the block is obtained as (1.5, 2.5).

In another implementation, when the target reference frame is the second reference frame, the electronic device may determine the predicted motion vector of the block from the second reference frame to the predicted frame according to the motion vector of the block from the second reference frame to the first reference frame .

In some embodiments, the motion between blocks is considered to be a uniform motion, and it is understood that in this case, the electronic device may convert the block from the second reference frame to the first reference frame by half the negative value of the motion vector Determined as the predicted motion vector for this block. Specifically, please refer to FIG. 31B , which is a schematic diagram of another method of determining a predicted motion vector of a block disclosed by an embodiment of the present application. As shown in the figure, a block in the target reference frame is schematically drawn, wherein Mv is the motion vector of the block from the second reference frame to the first reference frame, represented by a solid arrow; Mv' is the block The predicted motion vector from the second reference frame to the predicted frame, indicated by dashed arrows. As shown in the figure, the motion vector of the block from the first reference frame to the second reference frame is (-3, -5), and the electronic device can determine the negative half of the predicted motion vector of the block from the target reference frame to the predicted frame. is the predicted motion vector of the block, that is, according to Mv'=-1/2Mv, the predicted motion vector of the block is obtained as (1.5, 2.5).

In other embodiments, the electronic device can also obtain the predicted motion vector of the block from the target reference frame to the predicted frame through a network model that senses video motion acceleration. Specifically, the network model uses the quadratic optical flow prediction method , among which, the main idea of the quadratic optical flow prediction method is that, assuming that the motion of the object in the interval is uniformly accelerated, the electronic device can calculate the object in the interval by obtaining the initial velocity of the object and the acceleration of the object in the interval. displacement at any time. With this method, the motion vector between adjacent frames can be estimated more accurately. It should be noted that, the electronic device may also obtain the predicted motion vector of the block from the target reference frame to the predicted frame by other methods, which is not limited here.

S105. Determine the predicted motion vector of the first vertex from the target reference frame to the predicted frame according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame, where the first vertex is a vertex in the first block.

Specifically, the electronic device can first determine the block around the vertex in the target reference frame, and then determine the predicted motion vector of the vertex according to the predicted motion vector of the block around the vertex, wherein the predicted motion vector of the block is the block from the target reference frame. The motion vector from frame to predicted frame, and the predicted motion vector of the vertex is the motion vector of the vertex from the target reference frame to the predicted frame.

In some embodiments, the electronic device may, for each vertex of each block in the target reference frame, determine the blocks around the vertex in the target reference frame, and then determine each vertex according to the predicted motion vector of the blocks around each vertex. The predicted motion vector for the vertex. In some embodiments, the electronic device may determine the block closest to the vertex as the block surrounding the vertex. Specifically, please refer to FIG. 32A , which is a schematic diagram of determining blocks around a vertex according to an embodiment of the present application. As shown in FIG. 32A, taking a block in the target reference frame as an example, the four blocks closest to the vertex a of block A are block A, block B, block C, and block D, respectively. The electronic device can determine these four blocks. are the four blocks around vertex a.

Further, the electronic device may determine the average value of the predicted motion vectors of the blocks around the vertex as the predicted motion vector of the vertex. Specifically, please refer to FIG. 32B , which is a schematic diagram of determining a predicted motion vector of a vertex according to an embodiment of the present application. As shown in FIG. 32B, the motion vector of block A is Mv'A, where Mv'A=0, the motion vector of block B is Mv'B, the motion vector of block C is Mv'C, and the motion vector of block D is Mv'D, the electronic device determines the average value of the predicted motion vectors of the blocks around the vertex as the predicted motion vector of the vertex, and obtains the coordinates of the vertex a. It should be noted that there are no four blocks around the vertex of the block on the edge of the target reference frame. Optionally, the electronic device may determine the predicted motion vector of the vertex on the edge of the target reference frame to be zero.

S106. Determine the coordinates of the first vertex in the predicted frame according to the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex.

Specifically, the electronic device may add the coordinates of the vertex in the target reference frame and the predicted motion vector of the vertex to obtain the coordinates of the vertex in the predicted frame. Please refer to FIG. 33 , which is a schematic diagram of determining vertex coordinates according to an embodiment of the present application. As shown in the figure, the coordinates of vertex a in the target reference frame are (x _a1 , y _a1 ), the predicted motion vector of vertex a is (1, 2), and the electronic device compares the coordinates of vertex a in the target reference frame with this The predicted motion vectors of vertex a are added to obtain the coordinates of vertex a in the predicted frame as (x _a1 +1, y _a1 +2).

In some embodiments, the electronic device may calculate each vertex in the target reference frame, and finally obtain the coordinates of all the vertices in the target reference frame in the predicted frame. It should be noted that, in the case where the predicted motion vectors of the vertices on the edge of the target reference frame are determined to be zero, the coordinates of the vertices in the target reference frame are equal to the coordinates of the vertices in the predicted frame.

S107: Determine the pixel block of the first block in the predicted frame according to the coordinates of the vertex of the first block in the predicted frame and the coordinates in the target reference frame.

Specifically, the electronic device may first obtain the correspondence between the coordinates of the vertices of the first block in the target reference frame and the coordinates in the predicted frame according to the coordinates of the vertices of the first block in the target reference frame and the coordinates in the predicted frame , and further determine the correspondence between the coordinates of the vertex of the first block in the target reference frame and the coordinates in the predicted frame as the correspondence between the coordinates of the pixels of the first block in the target reference frame and the predicted frame, and finally, according to the The correspondence between the coordinates of the pixels in the block in the target reference frame and the coordinates in the predicted frame determines the coordinates of the pixels in the block in the predicted frame, wherein the first block is a block in the target reference frame.

Please refer to FIG. 34. FIG. 34 is a flowchart of determining coordinates of pixels in a block in a predicted frame disclosed in an embodiment of the present application. Step S107 may include the following steps:

S1071 , according to the coordinates of the vertices of the block in the target reference frame and the coordinates in the predicted frame, obtain the correspondence between the coordinates of the pixels of the block in the target reference frame and the coordinates in the predicted frame.

Specifically, the electronic device can obtain the homography transformation formula corresponding to the block according to the four pairs of coordinates of the block, wherein the coordinates of a vertex in the target reference frame and the coordinates in the predicted frame are a pair of coordinates, and the block corresponding to The homography transformation formula is used to express the correspondence between the coordinates of the pixels of the block in the target reference frame and the coordinates in the predicted frame. It can be understood that the electronic device calculates each block in the target reference frame, and can obtain a homography transformation formula corresponding to each block.

Please refer to FIG. 35A. FIG. 35A is a schematic diagram of a homography transformation formula corresponding to an acquisition block disclosed by an embodiment of the present application. As shown in the figure, the four vertices of block A are vertex a, vertex b, vertex c and Vertex d, the coordinates of a vertex in the target reference frame and the predicted frame are a pair of coordinates, specifically, a pair of coordinates of vertex a are (x _a , y _a ) and (x _a ', y _a '), vertex b A pair of coordinates of (x _b , y _b ) and (x _b ', y _b '), a pair of coordinates of vertex c are (x _c , y _c ) and (x _c ', y _c '), vertex d A pair of coordinates is (x _d , y _d ) and (x _d ', y _d ').

As shown in the figure, the electronic device can input the four pairs of coordinates of the block A into the homography transformation formula respectively to obtain the equation group corresponding to the block A, wherein the equation group includes four equations. Among them, the homography transformation formula is as follows:

Among them, (x ₁ , y ₁ ) is the coordinates of the vertex in the target reference frame, (x ₁ ', y ₁ ') is the vertex in the target reference frame whose coordinates are (x ₁ , y ₁ ) in the predicted frame The coordinates of , H is the unknown homography transformation matrix. Among them, the homography transformation matrix is as follows:

Further, the electronic device can obtain the homography matrix HA corresponding to the block A by solving the equation system corresponding to the block _A , and then substitute the homography matrix HA corresponding to the block _A into the homography transformation formula to obtain the block A. The homography transformation formula corresponding to A, and the homography transformation formula corresponding to block A are as follows:

The homography transformation formula corresponding to the block A is used to represent the correspondence between the coordinates of the pixels of the block in the target reference frame and the coordinates in the predicted frame.

S1072. Determine the coordinates of the pixels of the block in the predicted frame according to the correspondence between the coordinates of the pixels of the block in the target reference frame and the coordinates in the predicted frame.

Specifically, the electronic device may determine the pixel block of the first block in the predicted frame according to the homography transformation formula corresponding to the first block.

The electronic device may input the coordinates of the pixels in the block in the target reference frame into the homography transformation formula corresponding to the block to obtain the coordinates of the pixels in the predicted frame.

Please refer to FIG. 35B . FIG. 35B is a schematic diagram of determining the coordinates of pixels in a predicted frame disclosed by an embodiment of the present application. As shown in the figure, taking two pixels in block A as an example, one pixel in block A The coordinates of the point in the target reference frame are (x ₁ , y ₁ ), and by inputting (x ₁ , y ₁ ) into the homography transformation formula corresponding to block A, the coordinates of the pixel in the predicted frame (x ₁ ',y ₁ '), the coordinate of another pixel in block A in the target reference frame is (x ₂ , y ₂ ), input (x ₂ , y ₂ ) into the homography transformation formula corresponding to block A, The coordinates (x ₂ ', y ₂ ') of the pixel in the predicted frame can be obtained.

In some embodiments, the electronic device may first determine the area of the pixel block of the first block in the predicted frame according to the four vertices of the first block, for example, determine the quadrilateral obtained by connecting the four vertices of the first block as The first block is the region of the pixel block in the predicted frame.

In one implementation, when the area of the pixel block of the first block in the predicted frame is larger than the area of the first block in the target reference frame, the electronic device may input the coordinates of each pixel of the first block in the target reference frame The homography transformation formula corresponding to the block is used to obtain the coordinates of each pixel in the first block in the predicted frame. For example, the number of obtained coordinates of the first block in the predicted frame can be N, where N is greater than A positive integer of 0.

Further, the electronic device may acquire coordinates in the area of the pixel block. For example, there are M coordinates in the area of the pixel block. In this case, it can be understood that M>N, and M is a positive integer greater than 0. Further, the electronic device may remove the coordinates of the pixels in the N first blocks in the predicted frame from the M coordinates included in the pixel block area of the predicted frame to obtain the first coordinates, where the first coordinates include at least two coordinates. , input the first coordinate into the homography transformation formula corresponding to the first block, and obtain the pixel of the first coordinate in the target reference frame.

In another implementation, when the area of the first block in the pixel block of the predicted frame is larger than the area of the first block in the target reference frame, the electronic device may obtain the coordinates in the area of the pixel block, and put the area of the pixel block in the area of the pixel block. Enter the homography transformation formula corresponding to the first block for each coordinate of , and obtain the coordinates of each coordinate in the area of the pixel block in the target reference frame, and then obtain the pixel block corresponding to the first block.

Please refer to FIG. 35C . FIG. 35C is a schematic diagram of a prediction frame disclosed in an embodiment of the present application. Taking block A as an example, the electronic device obtains a pixel block corresponding to block A according to the homography transformation formula corresponding to block A, which can be as follows shown in Figure 35C.

S108. Display the predicted frame, which includes the pixel blocks of the first block in the predicted frame.

The electronic device can process each block in the predicted frame through steps S101 to S107, and can obtain a pixel block of each block in the predicted frame, wherein each coordinate in the pixel block corresponds to a pixel in the target reference frame , the electronic device can display each pixel block through the display screen, and finally, the displayed predicted frame is continuous. Please refer to FIG. 36 . FIG. 36 is a schematic diagram of a prediction frame disclosed by an embodiment of the present application. Taking part of the prediction frame as an example in the figure, the blocks in the prediction frame finally obtained by the electronic device are in a stretched state. The blocks are continuous and there is no void area.

During the process of predicting the image frame by the electronic device, the electronic device can not only predict the color information of the predicted frame, but also can predict the depth information of the predicted frame. In this way, the electronic device can also synthesize the predicted frame according to the predicted depth information.

The present application provides an image frame generation method, which may include: determining, according to the depth values of the eleventh block and the twelfth block, and the position coordinates of the eleventh block and the twelfth block in the image frame, determining The tenth position coordinate of the eleventh vertex of the predicted block in the predicted image frame; wherein, the eleventh block is a block in the first image frame; the twelfth block is in the second image frame according to matching The block determined by the algorithm that matches the eleventh block; the predicted block is generated according to the color data of the reference block and the tenth position coordinate, and the reference block is the eleventh block or the twelfth block one of the blocks; generating the predicted image frame, the predicted image frame including the predicted block.

It can be seen that in the embodiment of the present application, the block is predicted in combination with the depth value, so that the generated predicted block can be scaled according to the depth value, so that the picture displayed by the predicted image frame is more consistent with the scene drawn by the electronic device according to the actual data; When the device inserts the predicted image frame into the original image frame drawn according to the actual data, the transition between the original image frame and the predicted image frame is more natural and smooth, which improves the user experience.

In one embodiment, when the electronic device runs the application program, the electronic device can generate the image frame f ₃ according to the image frame f ₁ and the image frame f ₂ ; the image frame f ₁ and the image frame f ₂ are original image frames, Image frame _f3 is a predicted image frame. Inserting the predicted image frame into the original image frame can achieve the purpose of increasing the frame rate. For example, the electronic device can determine blocks that match each other between the image frame f ₁ and the image frame f ₂ according to a block matching algorithm; each set of matched blocks includes a block B ₁ and a block B ₂ , and the block B ₁ is located in the image frame f ₁ , block B ₂ is located in image frame f ₂ ; for each group of matched blocks do the following: Determine the group from the coordinates of block B ₁ in image frame f ₁ and the coordinates of block B ₂ in image frame f ₂ The displacement vector corresponding to the matched block; according to the coordinates of the block B ₁ or block B ₂ in the image frame, and the displacement vector, determine the coordinates of the block B ₁ or block B ₂ in the image frame f ₃ (the coordinates in the image frame f ₃ A block consists of a block B ₁ or a block B ₂ , for example, the image frame f ₃ is obtained by recombining (by performing position transformation) a plurality of blocks B ₁ ). It can be seen that by performing the above operations for each group of matched blocks, the position of one block in each group of matched blocks in the image frame _f3 can be determined, and then the generation of a block according to each group of matched blocks and the position of the block in the image frame _f3 can be generated. Image frame _f3 . This embodiment generates a predicted frame, which can achieve the purpose of increasing the frame rate. In the above-mentioned embodiment, when the display content of the image frame presents the scene in the three-dimensional space, it has the characteristics of "closer and farther small"; specifically, the display content of the image frame can be regarded as the scene in the three-dimensional space projected onto the projection plane for presentation. The projection process can be regarded as the scene in the three-dimensional space projected onto the projection plane by the camera point; according to the principle of the camera shooting the object, the camera point can be regarded as the focal point. Specifically, as shown in Figure 46E, the camera point can be the image The camera point in 46E. For the same object in three-dimensional space, if the distance between the object and the camera point is short, the display area of the object in the image frame is larger; if the distance between the object and the camera point is far, the object's Display area is small. In this embodiment, the electronic device only determines the coordinates of the block in the image frame _f3 according to the displacement vector. Since the displacement vector refers to the displacement vector in the plane, it only includes two-dimensional variables in the three-dimensional space, so the other One-dimensional data, so that the final generated image frame f ₃ cannot show the effect of "near big and far small"; therefore, there is a certain deviation between the display content of the image frame f ₃ and the three-dimensional scene constructed by the running data of the application, which makes the electronic After the device inserts the predicted frame (f ₃ in this example) in the original image frame (f ₁ , f ₂ in this example), the electronic device does not have a smooth transition during the display of the image frame.

In some embodiments, a method of generating an image frame is provided. The electronic device may generate the predicted image frame according to the three-dimensional data of the original image frame, so that the display area of the object in the predicted frame has the effect of "closer and far smaller". Implementing the method of this embodiment, while inserting the predicted image frame to improve the frame rate of the electronic device, the scene restored by the predicted frame is closer to the three-dimensional scene constructed according to the data of the application program, thereby making the difference between the original image frame and the predicted image frame. transitions are more natural and smooth.

The following embodiments introduce an image frame generation method provided by the embodiments of the present application. As shown in Figure 37, the method may include:

S3701, the electronic device performs block division on the reference image frame to obtain a reference block.

_The electronic device _{first determines one} _of the image frames f1 and _f2 as the reference image frame according to the position of the image frame _f3 to be predicted relative to the image frame f1 and the image frame f2.

The image frame f ₁ and the image frame f ₂ are image frames drawn according to the data of the application program; the image frame f ₃ is the predicted image frame predicted according to the image frame f ₁ and the image frame f ₂ .

Wherein, the image frame f ₁ is located before the image frame f ₂ in the data stream, and the electronic device can preset the image frame f ₁ and one image frame in the image frame f ₂ as a reference image frame; for example, the electronic device can set the data stream. The image frame f ₁ at the front of the middle is the reference image frame. It should be noted that the earlier the position of the image frame in the data stream is, the earlier the time point of displaying the image frame is when the electronic device displays the image frame according to the data stream.

Optionally, the reference image frame is determined according to the position _of the image frame _f3 relative to the image frame f1 and the image frame f2 in the data stream. For example, it can include the following two situations: (1) Position relationship 1 as shown in FIG. 38: the image frame f ₁ is located before the image frame f ₂ in the data stream; if the image frame f ₃ is located in the image frame f ₁ and Between the image frames f ₂ , the reference image frame is the image frame f ₁ and the image frame f ₁ in front of the image frame f ₂ . (2) Positional relationship 2 shown in Fig. 38: the image frame f ₁ is located before the image frame f ₂ in the data stream; if the image frame f ₃ is located after the image frame f ₁ and the image frame f ₂ , the reference image frame is Image frame f ₁ and image frame f ₂ following the image frame f ₂ . It should be noted that the position of the image frame _f3 in the data stream can be determined according to the application scenario and user requirements. For example, in order to improve the frame rate of the game, the electronic device can predict and generate the next frame of the image according to the first two frames of images. It can avoid the game delay being too long, that is, the position relationship 2 in Figure 38; for example, when processing a video with no or low requirements for delay, the electronic device can generate an intermediate image according to two adjacent image frames. frame, the predicted frame generated by this method is more accurate, that is, the position relationship 1 in Figure 38.

The electronic device divides the reference image frame according to blocks of a first size, and the first size may be a preset size. For example, the electronic device may set the first size to be 1/50 of the area of the reference image frame; for another example, the electronic device may set the first size to be a size formed by nine pixels in the reference image frame.

For example, the image frame f ₁ and the image frame f ₂ are shown in FIG. 39A . If the image frame f ₁ is a reference image frame, and the image frame f ₁ is divided into blocks, the effect shown in FIG. 39B can be obtained.

_S3702 , the electronic device determines _a matching block in the matching image frame that matches the reference block, and obtains that the block B1 in the image frame _f1 and the block B2 in the image frame _f2 are mutually matched blocks.

The matching image frame is an image frame other than the reference image frame in the image frame f ₁ and the image frame f ₂ .

Specifically, when the image frame f ₁ is a reference image frame, the image frame f ₂ is a matching image frame; when the image frame f ₂ is a reference image frame, the image frame f ₁ is a matching image frame.

Specifically, please refer to FIG. 40A , step S3703 specifically includes the following steps:

S1021, the electronic device determines the position coordinates (x _c , y _c ) and the depth value d _c of the reference block in the reference image frame.

The position coordinates in the image frame specifically refer to coordinates in the screen coordinate system. The screen coordinate system is a two-dimensional coordinate system. The maximum value of the horizontal and vertical coordinates of the screen coordinate system is determined according to the number of pixels contained in the display area of the electronic device (image frame/display area of the screen);

The electronic device stores the corresponding position coordinates of each pixel in the image frame in the image frame, and the position of the block in the image frame may be obtained by averaging the position coordinates of all pixels in the block in the image frame. Optionally, you can specify the coordinates of a point in the block as the coordinates of the block in the image frame; for example, specify the coordinates in the image frame of the pixel closest to the vertex in the upper left corner of the block as the coordinates of the block; or, specify the coordinates of the block in the image frame. The position of the pixel closest to the center point in the image frame is the coordinate of the block in the image frame.

Among them, the electronic device stores the depth value corresponding to each pixel in the image frame, and the depth value can represent the coordinates of the pixel in the Z dimension in the camera coordinate system, and the Z dimension is the dimension perpendicular to the projection plane in the camera coordinate system; When the device draws an image frame, it can be considered that the electronic device constructs a three-dimensional scene in the camera coordinate system according to the data of the application program, and then projects the constructed three-dimensional scene on the projection screen, and then obtains the image frame on the projection plane. For example, when the electronic device runs the Open Graphics Library (OpenGL), the electronic device stores the drawing result of the drawn image frame in the frame buffer object (Frame Buffer Object, FBO), and the frame buffer object is set with a color Attachment (ColorAttachment) and depth attachment (Depth Attachment), the color value information of the pixel and the position coordinates of the pixel in the image frame are stored in the color attachment; the depth value corresponding to the pixel is stored in the depth attachment.

Specifically, the depth value of the block can be calculated according to the depth value of the pixels included in the reference block. Specifically, the depth value of the block may be obtained by averaging all pixels included in the block. Optionally, the depth value of the block may be the depth value corresponding to the specified pixel in the block, for example, setting the depth value of the pixel at the center point of the block (the pixel closest to the center point) as the depth value of the block.

_S1022 , the electronic device determines an area of a second size according to the depth value dc.

_Wherein , the smaller the dc, the larger the second size; the larger the _dc , the smaller the second size. For example, the second size is represented by S _A. If the size of the reference block is S _B , the size of the entire image frame is S _C ; Q=S _C ÷ S _B , the following formula can be used:

S _A = Q ^(1-d _c ⁾ *S _B

In this example, the value range of the depth value can be from 0 to 1; it can be seen that through the above formula, when _dc =0, S _A =S _C ; when _dc =1, S _A =S _B ; when 0 When <d _c <1, _SA gradually increases as d ₁ increases gradually.

Optionally, the electronic device may preset the mapping relationship between _dc and the second size _SA . For example, when w ₁ <d _c ≤w ₂ , the area of the second size SA is S ₁ _; when w ₃ <d _c ≤w ₄ , the area of the second size SA is _S ₂ ......w _{(H-1 )} <d _c ≤ w _H , the area of the second size S _A is _SH ; when d _c >w _H , the second size S _A is equal to the size S _B of the block; wherein, w ₁ is a positive integer; w ₁ <w ₂ <w ₃ <w ₄ <...<w _(H-1) <w _H ; S ₁ >S ₂ >... > S _H .

The shape of the region may be preset, for example, the shape of the region may be a square, a circle, a triangle, and the like.

S1023, the electronic device determines, in the area of the second size, the block with the highest similarity with the reference block as a matching block, and obtains that block B ₁ and block B ₂ are a set of mutually matching blocks.

It should be noted that the electronic device can calculate the distance between the Z dimension and the camera point (camera) of the coordinates represented by the depth value in the camera coordinate system according to the depth value. The display content of the image frame can be regarded as the effect of projecting the scene in the camera coordinate system onto the projection plane; the intersection of the line connecting the position point in the 3D scene with the camera point (camera) and the projection plane is the projection onto the projection screen. location point.

For example, assuming that the reference image frame is image frame f ₁ , the matching image frame is image frame f ₂ ; the calculation is performed according to the depth value d ₁ , and z _{1 is calculated by design, and z 1} _represents the conversion of the depth value d ₁ to the camera coordinate system The coordinates indicated below. Referring to FIG. 40B , when z ₁ =z ₄ , the block I ₁ ′ is obtained by the projection of the quadrilateral I ₁ on the Z=z ₄ plane. At this time, the search range is an area consisting of 25 blocks centered on (x ₁ , y ₁ ). Please refer to Fig. 40C, when z ₁ =z ₅ , z ₅ >z ₄ ; the quadrilateral I ₁ is placed on the Z=z ₅ plane, then the block I ₁ ″ obtained by projecting the quadrilateral I ₁ onto the projection plane is smaller than the block I ₁ '; therefore, it can be understood that the farther away from the camera point camera, the smaller the projected area of the same object on the projection plane; and the farther away from the camera point camera, the same object is displaced by the same distance in the X and Y dimensions. , the displacement projection on the projection plane is also smaller. Therefore, the electronic device can set the search range under the condition of z ₁ =z ₅ to be smaller than the search range under the condition of z ₁ =z ₄ ; as shown in Figure 40C, z ₁ =z When the value is ₅ , the search range is an area consisting of 9 blocks centered at (x ₁ , y ₁ ). It can be seen that since d ₁ can indirectly represent the position coordinates and camera points corresponding to the block in the camera coordinate system Therefore, in this embodiment of the present application, the area of the second size can be directly determined according to d ₁ .

It should be noted that, during the process that the electronic device searches for a matching block within the search range, the electronic device may move one pixel at a time to perform matching search. For example, within the search range, the electronic device first determines whether the block B _S1 _whose upper left corner vertex coincides with the upper left corner vertex of the search range matches the reference block; The device judges whether it matches; if it does not match, ... until the vertex in the upper right corner of block B _S1 coincides with the vertex in the upper right corner of the search range; if it does not match, the electronic device moves the block down one line (one pixel), and repeats the above operations, Until the block traversal completes the entire search range.

_S3703 , the electronic device determines the position coordinates of the vertex _Yp ₁ _of the predicted block in the predicted image frame ( _x ₃₁ , y ₃₁ ).

Among them, block B ₁ and block B ₂ are blocks that match each other. Block B ₁ represents a block in image frame f ₁ and block B ₂ represents a block in image frame f ₂ .

In this example, the position coordinates of a vertex in the block in the image frame are used to represent the position coordinates of the block; the electronic device can set the vertex p ₁ in the block to represent the coordinates of the block, and the position of the vertex p ₁ relative to the block is preset . Specifically, the shape of the block is a square, and the vertex located at the upper left corner of the block in the screen coordinate system is set as p ₁ by the electronic device. In this example, the position coordinates of the vertex B ₁ p ₁ in the upper left corner of the block B ₁ in the image frame f ₁ in the screen coordinate system represent the position coordinates of the block B ₁ in the image frame f ₁ ; in the screen coordinate system, the block B ₂ The position coordinates of the upper left corner vertex B ₂ p ₁ in the image frame f ₂ represent the position coordinates of the block B ₂ in the image frame f ₂ ; in the screen coordinate system, the upper left corner vertex Yp ₁ of the predicted block is in the reference image frame. The position represents the position coordinates of the prediction block in the reference image frame.

Please refer to FIG. 41A, step S3703 specifically includes the following steps:

S1031, the electronic device calculates the position coordinates (x _e1 , y _e1 , z in the camera coordinate system) according to the depth value d ₁ of the block B ₁ and the position coordinates (x ₁ , y ₁ ) of the block B ₁ in the image frame f ₁ _e1 ).

In this example, the position coordinates of a block are represented by the position coordinates of a vertex in the block in the image frame. Specifically, the position coordinates of the vertex Bp ₁ of the block B ₁ in the image frame f ₁ may be set to represent the position coordinates of the block B ₁ . For example, the shape of the block B ₁ is a square, and when the block B ₁ is placed in the image frame f ₁ , the vertex Bp ₁ is the vertex located at the upper left corner of the block B ₁ in the screen coordinate system.

The position coordinates in the image frame specifically refer to the coordinates in the screen coordinate system; the electronic device calculates the specific position coordinates (x _e1 , y _e1 , z _e1 ) according to the depth value d ₁ and the position coordinates (x ₁ , y ₁ ). The process includes:

If the resolution of the screen in the screen coordinate system is (Sw, Sh), that is, the abscissa includes Sw pixels, the ordinate includes Sh pixels, and the position coordinates of the pixel in the lower left corner of the screen coordinate system are (0, 0), and the upper right The position coordinates of the corner pixels are (Sw-1, Sh-1); the electronic device first normalizes the position coordinates (x ₁ , y ₁ ) in the screen coordinate system to obtain (x _n1 , y _n1 ); Wherein, x _n1 =x ₁ /(Sw-1)*2-1; y _n1 =y ₁ /(Sh-1)*2-1. The value range of the depth value is (0, 1), and z _n1 can be obtained by normalizing the depth value d ₁ ; specifically, z _n1 =d ₁ *2-1. The normalized values of x _n1 , y _n1 and z _n1 are all between (-1, 1). Therefore, normalized coordinates (x _n1 , y _n1 , z _n1 , _wn1 ) can be obtained according to the position coordinates (x ₁ , y ₁ ) and d ₁ ; wherein, _wn1 =-1.

After that, convert the normalized (x _n1 , y _n1 , z _n1 , w _n1 ) to the clipping coordinate system to obtain (x _c1 , y _c1 , z _c1 , w _c1 ); _c1 , y _c1 , z _c1 , w _c1 ) are converted into the camera coordinate system to obtain (x _e1 , y _e1 , z _e1 , 1); wherein, w _c1 =-z _e1 . And (x _c1 , y _c1 , z _c1 , w _c1 ) and (x _e1 , y _e1 , z _e1 , 1) have the following relationship:

where w _c1 =-z _e1 . The matrix P is known, and the matrix P is as follows:

in,

r, n, f and t are all known constants.

Under the clipping coordinate system w _c1 =-z _e1 ; therefore, under the condition of known w _n1 =-1, w _c1 =-z _e1 and (x _n1 , y _n1 , z _n1 , w _n1 ), we can get the clipping coordinate system coordinates; under the known clipping coordinate system (x _c1 , y _c1 , z _c1 , w _c1 ), w _c1 =-z _e1 ; and matrix P, the electronic device can calculate ( x _e1 , y _e1 , z _e1 , 1).

It can be seen that the electronic device can obtain the corresponding position coordinates (x _e1 , y _e1 , z _e1 , 1) of the block B ₁ in the camera coordinate system according to d ₁ and (x ₁ , y ₁ ) of the block B ₁ .

S1032, the electronic device calculates the position coordinates (x _e2 , y _e2 , z _e2 ) in the camera coordinate system according to the depth value d ₂ of the block B ₂ and the position coordinates (x ₂ , y ₂ ) of the block B ₂ in the reference image frame ).

Specifically, please refer to the calculation process of S1041. The electronic device can obtain the position coordinates (x _e2 , y _e2 , z _e2 ) of the block B ₂ in the camera coordinate system according to d ₂ and (x ₂ , y ₂ ) of the block B ₂ ,1).

S1033, the electronic device is based on the position coordinates (x _e1 , y _e1 , z _e1 , 1) corresponding to the block B ₁ in the camera coordinate system and the position coordinates (x _e2 , y _e2 , z ) corresponding to the block B ₂ in the camera coordinate system _e2 , 1), the position coordinates (x _e3 , y _e3 , z _e3 , 1) in the camera coordinate system are obtained by calculation.

Among them, (x _e3 , y _e3 , z _e3 , 1) are predicted; (x _e1 , y _e1 , z _e1 , 1) represent the position coordinates of block B ₁ in the camera coordinate system; (x _e2 , y _e2 ) , z _e2 , 1) represent the position coordinates of the block B ₂ in the camera coordinate system; in the embodiment of the present application, the position of a vertex of the predicted block in the predicted image frame is predicted according to the block B ₁ and the block B ₂ ; In this example, the electronic device first predicts the position coordinates (x _e3 , y _e3 , z _e3 , 1) under the camera coordinate system according to the corresponding position coordinates of the block B ₁ and the block B ₂ under the camera coordinates; (x _e3 , y _e3 , z _e3 , 1) in the coordinate system is converted to the screen coordinate system, and the position coordinates (x ₃₁ , y ₃₁ ) of one vertex of the prediction block in the screen coordinate system are obtained.

Wherein, the last components in (x _e1 , y _e1 , z _e1 , 1) and (x _e2 , y _e2 , z _e2 , 1) are all 1, which is only convenient for calculation. The camera coordinate system is a three-dimensional coordinate system, and the electronic device can calculate (x _e3 , y _e3 , z _e3 ) according to (x _e1 , y _e1 , z _e1 ) and (x _e2 , y _e2 , z _e2 ) in the camera coordinate system , and then get (x _e3 , y _e3 , z _e3 , 1).

Image frame _f1 _precedes image frame f2 in the data stream. The position coordinates of the block B ₁ in the screen coordinate system are A ₁ (x _e1 , y _e1 , z _e1 ), and the block B ₂ corresponds to the position point A ₂ (x _e2 , y _e2 , z _e2 ) in the camera coordinate system. The electronic device can calculate the displacement vector β ₁ of A ₁ to A ₂ . After that, it is determined that A ₃ (x _e3 , y _e3 , z _e3 ) is equal to the position coordinates of the reference block in the camera coordinate system (if the reference block is block B ₁ , the corresponding position coordinates of the reference block in the camera coordinate system are point A ₁ ; if the reference block is block B ₂ , the position coordinate corresponding to the reference block in the camera coordinate system is point A ₂ ) plus the displacement vector β ₁ of the first scale; the first scale is equal to: the predicted image frame is in the data stream The ratio of the time point of the reference image frame in the data stream minus the time point of the reference image frame in the data stream, and the time point of the image frame f ₂ in the data stream minus the time point of the image frame f ₁ in the data stream.

For example, please refer to FIG. 41B . FIG. 41B illustrates a set of mutually matched blocks, block B ₁ and block B ₂ , in the image frame f ₁ and the image frame f ₂ in the embodiment of the present application. Referring to FIG. 41C , the block B ₁ and block B ₂ in FIG. 41B are placed in the screen coordinate system to obtain the schematic diagram shown in FIG. 41C . Please refer to FIG. 41D. FIG. 41D is a schematic diagram of a camera coordinate system provided by an embodiment of the present application. Figure 41D includes the position point A ₁ (x _e1 , y _e1 , z _e1 ) corresponding to the block B ₁ in the camera coordinate system, and the position point A ₂ (x _e2 , y _e2 , y e2 ) corresponding to the block B ₂ in the camera coordinate system, z _e2 ). The electronic device can calculate displacement vectors β ₁ (x _e2 -x _e1 , y _e2 -y _e1 , z _e2 -z _e1 ) of A ₁ to A ₂ . Denote the predicted image frame as f ₃ . (1) Please refer to Fig. 41E, if the image frame f ₁ , the image frame f ₂ and the image frame f ₃ have the positional relationship 1 shown in Fig. 41E: the time point corresponding to the image frame f ₁ in the data stream is t ₁ , the image The time point corresponding to the frame f ₂ in the data stream is t ₃ , and the time point corresponding to the image frame f ₃ in the data stream is t ₂ , then the image frame f ₁ can be set as the reference image frame, that is, the block B ₁ is the reference block ; the first ratio is equal to the ratio of (t ₂ -t ₁ ) to (t ₃ -t ₁ ), then it can be determined: displacement vector β ₂ =(t ₂ -t ₁ )/(t ₃ -t ₁ )β ₁ , then A ₃ (x _e3 , y _e3 , z _e3 )=(x _e1 , y _e1 , z _e1 )+(t ₂ -t ₁ )/(t ₃ -t ₁ )β ₁ . (2) Please refer to Fig. 41F, if the image frame f ₁ , the image frame f ₂ and the image frame f ₃ have the positional relationship 2 shown in Fig. 41F: the time point corresponding to the image frame f ₁ in the data stream is t ₁ , and the image The time point corresponding to the frame f ₂ in the data stream is t ₂ , and the time point corresponding to the image frame f ₃ in the data stream is t ₃ ; then the image frame f ₂ can be set as the reference image frame, that is, the block B ₂ is the reference block ; the first ratio is equal to the ratio of (t ₃ -t ₂ ) to (t ₂ -t ₁ ), then it can be determined: displacement vector β ₂ =(t ₃ -t ₂ )/(t ₂ -t ₁ )β ₁ , then A ₃ (x _e3 , y _e3 , z _e3 )=(x _e2 , y _e2 , z _e2 )+(t ₃ -t ₂ )/(t ₂ -t ₁ )β ₁ .

S1034 , the electronic device converts (x _e3 , y _e3 , z _e3 , 1) in the camera coordinate system to the screen coordinate system to obtain (x ₃₁ , y ₃₁ ).

Specifically, the transformation from the camera coordinate system to the screen coordinate system is the inverse transformation of the entire process in step S1041, which can be completed by the following steps:

where w _c1 =-z _e1 . According to w _c1 =-z _e1 and [x _c3 y _c3 z _c3 w _c3 ], the normalized coordinates (x _n3 , y _n3 , z _n3 , w _n3 ) are calculated; according to the normalized coordinates (x _n3 , y _n3 , z _n3 , w _n3 ) and the screen resolution (Sw, Sh) will be scattered to obtain (x ₃₁ , y ₃₁ ) in the screen coordinate system.

Among them, the normalized coordinates are regarded as coordinates in the normalized coordinate system.

It should be noted that, for the explanation of the screen coordinate system, normalized coordinate system, clipping coordinate system, and camera coordinate system in the embodiments of the present application, please refer to the book: Chapter 4 of the Seventh Edition "OpenGL Super Collection", 3D In the Graphics Computing chapter, an explanation of the coordinate space in OpenGL. (English name: OpenGLSuperBible, Chapter 4, mathfor 3D Graphics, Coordinate spaces inOpenGL). Specifically, the screen coordinate system corresponds to (Windowspace) in the English original; the normalized coordinate system corresponds to (Normalizeddevicecoordinate(NDC)space) in the English original, and the clip coordinate system corresponds to (Clip Space) in the English original ), the camera coordinate system corresponds to (View Space) in the English original.

S3704, the electronic device determines the corresponding position coordinates of the vertex Yp ₂ of the predicted block in the predicted image frame according to the depth values of the blocks B ₃ and B ₄ and the position coordinates of the block B ₃ and the block B ₄ in the image frame (x ₃₂ , y ₃₂ ).

It should be noted that, the execution process of step S3705 is the same as that of step S3704. Replace the depth value of block B ₁ in S3704 and the position coordinates in the image frame with the depth value of block B ₃ and the position coordinates in the image frame respectively; replace the depth value of block B ₂ in S3704 with the position coordinates in the image frame. The position coordinates are respectively replaced with the depth value of the block B ₄ and the position coordinates in the image frame; then the output result is the position coordinates (x ₃₂ , y ₃₂ ) corresponding to the vertex Yp ₂ in the predicted image frame.

Among them, block B ₃ is a block adjacent to block B ₁ in image frame f ₁ , block B ₄ is a block adjacent to block B ₂ in image frame f ₂ , and block B ₃ and block B ₄ are mutually Matching blocks, please refer to step S3703 for the explanation of matching between blocks;

The position of the vertex Yp ₂ relative to the prediction block is different from the position of the vertex Yp ₁ relative to the prediction block.

Since block B ₁ and block B ₂ are mutually matched blocks, block B ₃ and block B ₄ are mutually matched blocks, block B ₁ and block B ₃ are adjacent, and block B ₂ and block B ₄ are adjacent; therefore, the block _The position of _B3 relative to block B1 is the same as that of block _B4 relative to block B2 _. In the embodiment of the present application, the vertex Yp ₂ of the prediction block has a preset association relationship with the block B3 and the block B4; for example, the vertex Yp ₂ is the vertex in the upper right corner of the prediction block, then the block B ₃ is the right side of the block B ₁ block. (It should be noted that, in this embodiment of the present application, the vertex Yp ₁ of the prediction block is determined by block B ₁ and block B ₂ , and the color data of the prediction block is based on the color of block B ₁ and/or block B ₂ Therefore, the vertex Yp ₁ in this example represents a vertex in the prediction block; the vertex Yp ₂ of the prediction block in this example is determined according to the blocks adjacent to the block B ₁ and the block B ₂ , and the block is In the case of a square, Yp ₂ can represent any one of the other three vertices in the prediction block other than the vertex Yp ₁ ).

In some embodiments, the block is a square, and the position coordinates of the upper left corner of the block in the screen coordinate system are regarded as the position coordinates of the block in the screen system. Please refer to FIG. 42. FIG. 42 is a schematic diagram of determining each vertex of a prediction block provided in an embodiment of the present application. The vertex Yp ₁ of the prediction block represents the vertex of the upper left corner of the prediction block. The corresponding position coordinates of the vertex Yp ₁ in the predicted image frame f ₃ are calculated according to the vertex B ₁ p ₁ in the upper left corner of the block B ₁ and the vertex B ₂ p ₂ in the upper left corner of the block B ₂ . The vertex Yp ₂ of the prediction block represents the vertex of the upper right corner of the prediction block; the corresponding position coordinates of the vertex Yp ₂ in the predicted image frame f ₃ are based on the vertex B ₃ p ₁ of the upper left corner of the block B ₃ and the vertex of the upper left corner of the block B ₄ B ₄ p ₁ is calculated; block B ₃ is a block adjacent to the right of block B ₁ , and block B ₄ is a block adjacent to the right of block B ₂ . The vertex Yp ₃ of the prediction block represents the vertex of the lower right corner of the prediction block; the corresponding position coordinates of the vertex Yp ₃ in the predicted image frame f ₃ are based on the vertex B ₅ p ₁ of the upper left corner of the block B ₅ and the vertex of the upper left corner of the block B ₆ Calculated by B ₆ p ₁ ; the top left vertex of block B ₅ coincides with the bottom right vertex of block B ₁ , and the top left vertex of block B ₆ coincides with the bottom right vertex of block B ₂ . The vertex Yp ₄ of the prediction block represents the vertex of the lower left corner of the prediction block; the position coordinates of the vertex Yp ₄ in the predicted image frame f ₃ are based on the vertex B ₇ p ₁ of the upper left corner of the block B ₇ and the vertex B of the upper left corner of the block B ₈ ₈ calculated by p ₁ ; the vertex of the upper left corner of block B ₇ coincides with the vertex of the lower left corner of block B ₁ , and the vertex of the upper left corner of block B ₈ is coincident with the vertex of the lower left corner of block B ₂ .

S3705, the electronic device generates the predicted image frame according to the color data of the reference block and the position coordinates of the vertex Yp ₁ and the vertex Yp ₂ of the predicted block in the predicted image frame.

Specifically, the electronic device first generates a prediction block according to the color data of the reference block and the vertices Yp ₁ and Yp ₂ of the prediction block; since the prediction image frame is composed of multiple prediction blocks, the prediction image can be generated according to the multiple prediction blocks frame.

The color data may specifically refer to the color value of the pixel, for example, the RGB value of the pixel. Color data can be stored in the FBO's color attachment.

Specifically, the vertices of the reference block and the predicted block correspond to each other, and the electronic device generates the predicted block according to the corresponding relationship of the vertices, the positional relationship of the pixels in the block relative to the vertices, and the color data of the reference block. The vertex in the upper left corner of the reference block corresponds to the vertex in the upper left corner of the prediction block, the vertex in the upper right corner of the reference block corresponds to the vertex in the upper right corner of the prediction block, the vertex in the lower right corner of the reference block corresponds to the vertex in the lower right corner of the prediction block, and the vertex in the lower left corner of the reference block corresponds to the vertex in the upper right corner of the prediction block. The vertices correspond to the vertices in the lower left corner of the prediction block; then the display content in the reference block is placed in the area formed by the vertices of the prediction block according to the above-mentioned corresponding relationship, and the prediction block is generated; Do one or more of the following: stretch, shrink. For example, in the case where the reference block is block B1, referring to FIG. 43, the electronic device maps the color data ₍ display content ₎ _of block B1 to the In image frame _f3 , a prediction block in image frame _f3 is generated. It should be noted that the above process is called "mapping" in some embodiments; that is, the process of mapping the texture of block B ₁ to the four vertices of the predicted block; specifically, the electronic device performs the mapping process in triangles Therefore, the block B ₁ can be split into two equilateral right triangles, and the prediction block can be split into two triangles; then the contents of the two triangles in block B ₁ can be split according to the correspondence between the vertices Map to the two triangles of the prediction block respectively.

Optionally, the electronic device can obtain the homography transformation formula corresponding to the two blocks according to the four pairs of coordinates formed by the four vertices of the reference block and the predicted block, wherein the coordinates in the reference image frame and the corresponding one in the predicted image frame are obtained. The coordinates of the pair of vertices are a pair of coordinates, and the homography transformation formula corresponding to the block is used to represent the correspondence between the coordinates of the pixels of the block in the reference image frame and the coordinates in the predicted frame. It can be understood that the electronic device calculates each block in the reference image frame, and can obtain the homography transformation formula corresponding to each block.

Specifically, if the reference block is block B ₁ ; the four vertices of block B ₁ are respectively vertex B ₁ p ₁ , vertex B ₁ p ₂ , vertex B ₁ p ₃ and vertex B ₁ p ₄ ; the four vertices of the prediction block are They are: vertex Yp ₁ , vertex Yp ₂ , vertex Yp ₃ , vertex Yp ₄ ; vertex B ₁ p ₁ corresponds to vertex Yp ₁ , vertex B ₁ p ₂ corresponds to vertex Yp ₂ , vertex B ₁ p ₃ corresponds to vertex Yp ₃ Correspondingly, vertex B ₁ p ₄ corresponds to vertex Yp ₄ ; the position coordinate of a vertex in the reference image frame and the position coordinate of the corresponding vertex in the predicted image frame are a pair of position coordinates, specifically, a pair of position coordinates of vertex B ₁ p ₁ The pair of position coordinates are (x _a , y _a ) and (x _a ', y _a '), and the pair of position coordinates of vertex B ₁ p ₂ are (x _b , y _b ) and (x _b ', y _b ') , a pair of position coordinates of vertex B ₁ p ₃ are (x _c , y _c ) and (x _c ', y _c '), and a pair of position coordinates of vertex B ₁ p ₄ are (x _d , y _d ) and ( x _d ',y _d ').

The electronic device may input the four pairs of position coordinates of the block B ₁ into the homography transformation formula respectively to obtain an equation group corresponding to the block B ₁ , where the equation group includes four equations. Among them, the homography transformation formula is as follows:

Among them, (x ₁ , y ₁ ) is the position coordinate of the vertex in the reference block, and the position coordinate of the vertex corresponding to the vertex (x ₁ , y ₁ ) in the reference block in the predicted image frame is (x ₁ ', y ₁ '), H is the unknown homography transformation matrix. Among them, the homography transformation matrix is as follows:

Further, the electronic device can obtain the homography matrix H _A corresponding to the block B ₁ by solving the equation system corresponding to the block B ₁ , and then substitute the homography matrix H _A corresponding to the block B ₁ into the homography transformation formula, The homography transformation formula corresponding to block B ₁ can be obtained, and the homography transformation formula corresponding to block B ₁ is as follows:

The homography transformation formula corresponding to the block B ₁ is used to represent the correspondence between the position coordinates of the pixels of the block B ₁ in the reference image frame and the position coordinates in the predicted image frame.

After that, the electronic device determines the position coordinates of the pixels of the reference block in the predicted image frame according to the correspondence between the position coordinates of the pixels of the block in the reference image frame and the position coordinates in the predicted image frame.

The electronic device may input the position coordinates of the pixels in the reference block in the reference image frame into the homography transformation formula corresponding to the block to obtain the position coordinates of the pixels in the predicted frame.

After that, the electronic device maps the color data to the position coordinates in the predicted image frame according to the color data of each pixel in the reference block and the position coordinates of each pixel in the predicted image frame to generate a predicted block.

Specifically, in the case of the positional relationship between the image frame f ₁ , the image frame f ₂ and the image frame f ₃ as shown in the positional relationship 1 in FIG. 41E , the image frame generated according to the image frame f ₁ and the image frame f ₂ f ₃ is shown in Fig. 44A; in order to facilitate the observation of image frame f ₁ , image frame f ₂ and image frame f ₃ , the position change of the car and the zoom degree of the car, image frame f ₁ , image frame f ₂ and image frame f The car in ₃ is placed in the same coordinate system, resulting in Figure 44B.

Specifically, in the case of the positional relationship between the image frame f ₁ , the image frame f ₂ and the image frame f ₃ as shown in the second positional relationship in FIG. 41F , the image frame generated according to the image frame f ₁ and the image frame f ₂ f ₃ is shown in Fig. 45A; in order to facilitate the observation of image frame f ₁ , image frame f ₂ and image frame f ₃ , the position change of the car and the zoom degree of the car, image frame f ₁ , image frame f ₂ and image frame f The car in ₃ is placed in the same coordinate system, resulting in Figure 45B.

The following embodiments introduce an image frame generation method provided by the embodiments of the present application. As shown in Figure 46A, the method may include:

S201, the electronic device performs block division on a reference image frame to obtain a reference block.

The divided image frame f ₁ includes block B ₁ , the image frame f ₂ includes block B ₂ , the reference block is one of block B ₁ and block B ₂ , and block B ₃ is obtained by scaling the reference block, Block _B3 refers to the predicted block in the predicted image frame.

For specific explanations, please refer to S3701.

S202, the electronic device determines a predicted displacement vector from the reference block to block _B3 , where the predicted displacement vector refers to a displacement vector in the plane where the image frame is located.

The display content in block _B3 is obtained by scaling according to the content of the reference block; the position coordinates of block _B3 in the image frame _f3 are obtained by shifting the coordinates of the reference block in the reference image frame. Image frame _f3 is a predicted image frame, ie block _B3 is predicted.

The electronic device performs a block matching algorithm on the image frame f ₁ and the image frame f ₂ shown in FIG. 39A , and determines the block B ₂ matched by the block B ₁ , the block B ₁ is a block in the image frame f ₁ , and the block B ₂ is a block in image frame _f2 .

The reference block may be the block B ₁ in the image frame f ₁ or the block B ₂ in the image frame f ₂ . Specifically, the reference block may be preset. For example, the image frame f ₁ is located before the image frame f ₂ in the data stream; the reference block can be set as the block in the image frame f ₁ , that is, the reference block is block B ₁ ; the reference block can also be set as the image frame f ₂ The block in , that is, the reference block is block B ₂ .

Optionally, the reference block may be determined according to the position of the image frame _f3 relative to the image frame f1 and the image frame f2 in the data stream, which may include the following _two cases: ( ₁ ) If the image frame _f3 is located in the image frame f3 Between the frame f1 and the image frame f2, the reference block is the block in the image frame f1 _earlier in the data stream, as shown in Figure 46C, positional relationship ₁ _: the image frame _f1 is located in the image frame f1 in the data stream. Before frame _f3 , image frame _f3 is located before image frame f2, and the reference block is block B1 in image frame _f1 _; ₍ ₂ ) if image frame _f3 is located after image frame f1 and image frame _f2 , then The reference block is the block in the image frame f ₂ that is later in the data stream, as shown in Figure 46D, the positional relationship 2: in the data stream, the image frame f ₁ is located before the image frame f ₂ , and the image frame f ₂ is located in the image frame Before _f3 , the reference block is block _B2 in image frame _f2 . It should be noted that the position of the image frame _f3 in the data stream can be determined according to the application scenario and user requirements. For example, in order to improve the frame rate of the game, the electronic device can predict and generate the next frame of the image according to the first two frames of images. It can avoid the game delay being too long, that is, the position relationship 2 in Figure 46D; for example, when processing a video with no or low requirements for delay, the electronic device can generate an intermediate image according to two adjacent image frames. frame, the predicted frame generated by this method is more accurate, that is, the position relationship 1 in FIG. 46C.

The electronic device may determine the predicted displacement vector of the reference block according to the coordinates of the block B ₁ and the block B ₂ in the plane where the image frame is located. It should be noted that when the electronic device generates or displays an image frame, each pixel in the image frame has corresponding coordinates. The electronic device may determine the coordinates of the block according to the coordinates of the pixels in the block; and determine the predicted displacement vector of the reference block according to the coordinates of the block B ₁ and the coordinates of the block B ₂ . Specifically, the coordinates of the block can be determined according to any one of the following three ways: (1) the coordinates obtained by averaging the coordinates of all pixels included in the block are taken as the coordinates of the block; (2) the center of the block The coordinates of the point (pixel) are used as the coordinates of the block; (3) the coordinates corresponding to the specified vertices of the block are used as the coordinates of the block, for example, the coordinates of the vertex of the upper left corner of the block are used as the coordinates of the block.

Specifically, the image frame f ₁ includes a block B ₁ , and the coordinates of the block B ₁ in the plane where the image frame is located are (x ₁ , y ₁ ); the image frame f ₂ includes a block B ₂ , and the block B ₂ is in the image frame The coordinates in the plane where it is located are (x ₂ , y ₂ ); for the convenience of viewing the relationship between block B ₁ and block B ₂ , the image frame f ₁ includes the car image of block B ₁ and the image frame f ₂ includes _The car image of block B2, placed in the Cartesian coordinate system shown in Figure 46E, results in Figure 46B. As shown in FIG. 46B , the displacement vector of block B ₁ to block B ₂ is β ₁ =(x ₂ -x ₁ , y ₂ -y ₁ ).

As shown in FIG. 46C , when the relative positions of the image frames f ₁ , f ₂ , and f ₃ in the data stream are position relationship 1, the time point corresponding to the image frame f ₁ in the data stream is t ₁ , and the image frame f ₂ The corresponding time point in the data stream is t ₃ , the corresponding time point of the image frame f ₃ in the data stream is t ₂ ; the reference block is the block B _{1 in the image frame f 1} _. Calculate the difference between the time point t ₂ corresponding to the image frame f ₃ and the time point t ₁ corresponding to the image frame f ₁ to obtain the difference value d ₁ =(t ₂ -t ₁ ); calculate the time point corresponding to the image frame f ₂ The difference between t ₃ and the time point t ₁ corresponding to the image frame f ₁ is obtained as the difference d ₂ =(t ₃ -t ₁ ); the ratio d ₁ /d ₂ of d ₁ and d ₂ is obtained. Assuming that the object in the image frame is moving at a uniform speed in the time period from t ₁ to t ₃ , the predicted displacement vector of the reference block can be determined as β ₂ =d ₁ /d ₂ *β ₁ =(t ₂ -t ₁ )/(t ₃ -t ₁ )*(x ₂ -x ₁ , y ₂ -y ₁ )=((t ₂ -t ₁ )/(t ₃ -t ₁ )*(x ₂ -x ₁ ), (t ₂ -t ₁ )/(t ₃ -t ₁ )*(y ₂ -y ₁ )).

As shown in FIG. 46D , when the relative positions of the image frames f ₁ , f ₂ , and f ₃ in the data stream are position relationship two, the time point corresponding to the image frame f ₁ in the data stream is t ₁ , and the image frame The time point corresponding to f ₂ in the data stream is t ₂ , and the time point corresponding to the image frame f ₃ in the data stream is t ₃ ; the reference block is block B _{2 in the image frame f 2} _. Calculate the difference between the time point t ₃ corresponding to the image frame f ₃ and the time point t ₁ corresponding to the image frame f ₂ to obtain the difference value d ₃ =(t ₃ -t ₂ ); calculate the time point corresponding to the image frame f ₂ The difference between t ₂ and the time point t ₁ corresponding to the image frame f ₁ obtains the difference d ₄ =(t ₂ −t ₁ ); obtains the ratio d ₃ /d ₄ of d ₃ and d ₄ . Assuming that the object in the image frame is moving at a constant speed in the time period from t ₁ to t ₃ , the predicted displacement vector of the reference block can be determined as β ₂ =d ₃ /d ₄ *β ₁ =(t ₃ -t ₂ ) /t ₂ -t ₁ )*(x ₂ -x ₁ , y ₂ -y ₁ )=((t ₃ -t ₂ )/t ₂ -t ₁ )*(x ₂ -x ₁ ),(t ₃ - t ₂ )/t ₂ -t ₁ )*(y ₂ -y ₁ )).

It should be noted that the time point corresponding to the image frame in the data stream may represent the time point at which the image frame is displayed according to the data stream; for example, the current data stream is video data with a duration of 30 seconds, if the image frame f ₁ corresponds to the data stream The time point is the 15th second, and the corresponding time point of the image frame f ₂ in the data stream is 17 seconds; then when the electronic device plays the video, the image frame f 1 is displayed when the video is played to the 15th second, and the image frame f ₁ is displayed when the video is played to the 15th second. Image frame f2 is displayed at ₁₇ seconds.

Optionally, if the time data of the image frame is not included in the data stream, the electronic device can calculate the time point corresponding to displaying each image frame according to the frame rate preset by the system, and use the time point of displaying the image frame instead. The corresponding time point of the image frame in the data stream; for example, the preset frame rate of the electronic device is 30 frames per second, if the data stream includes 300 image frames, the corresponding playback time of the data stream is 10 seconds, and the electronic device The device can calculate the time point when each image frame in the data stream is displayed; and then execute the methods shown in FIG. 46C and FIG. 46D .

In FIG. 46C and FIG. 46D , the predicted displacement vector of the reference block is added to the coordinates of the plane where the image frame is located to obtain (x ₃ , y ₃ ), and (x ₃ , y ₃ ) is the block B ₃ in Coordinates in the plane where the image frame is located; block B ₃ is located in image frame f ₃ ; block B ₃ is scaled according to the reference block.

S203, the electronic device determines the scaling ratio of the block reference block to the block _B3 according to the coordinates _of the block B1 and the block _B2 in the Z dimension.

The scaling ratio of the reference block to the block B ₃ refers to the scaling ratio of the area of the block B ₃ relative to the area of the reference block.

In order to facilitate a better understanding of the embodiments of the present application, a principle applied when an electronic device generates an image frame is first introduced. When the camera captures the image frame, it will produce the feature of "closer and far smaller", that is, the closer the same object is to the camera, the larger the object will be in the generated image. Similarly, in the embodiment of the present application, the data of the image frame includes three-dimensional data; when the electronic device generates the image frame, it can be considered that the electronic device firstly constructs the three-dimensional scene in the three-dimensional space according to the data of the image frame, and then converts the three-dimensional scene into the three-dimensional scene. Projecting onto the projection plane in the 3D space to generate an image frame, the projection plane is located on one side of the 3D scene. It can be understood that if the object in the 3D scene is closer to the projection plane, the display area of the object in the image frame will be larger; if the object in the 3D scene is farther from the projection plane, the object in the image frame will be displayed. The smaller the display area.

For example, please refer to FIG. 46E. FIG. 46E is a schematic diagram of a principle applied when an electronic device generates an image frame. A three-dimensional Cartesian coordinate system is established in FIG. 46E, and a car is included in the three-dimensional scene constructed from the data of the image frame. The projection plane can be a plane constructed by the X dimension and the Y dimension; the electronic device can set the camera point camera in the three-dimensional space, the three-dimensional scene is projected onto the projection plane N through the camera point camera, and the straight line formed by any point in the three-dimensional scene and the camera point camera The intersection point with the projection plane is the position point where the arbitrary point is projected on the projection plane. The coordinate of the camera point camera in the Z dimension is z _c , and the coordinate of the projection plane N in the Z dimension is z _n . It should be noted that, in the embodiment of the present application, the projection plane N and the camera point camera are on the same side of the three-dimensional scene. Figure 46E includes scene 1 and scene 2; the coordinates of any point on the car in scene 1 and scene 2 are the same in the X dimension and the Y dimension, and only the coordinates are different in the Z dimension. The coordinate of the projection plane N in the Z dimension is z _n ; the car includes point p ₁ and point p ₂ ; in scene 1, the coordinates of point p ₁ and point p ₂ in the Z dimension are z ₅ , point p ₁ and point p 2 The projection of p ₂ onto the projection plane corresponds to point p ₁ ' and point p ₂ 'respectively; in scene 2, the coordinates of point p ₁ and point p ₂ in the Z dimension are z ₆ , and the point p ₁ and point p ₂ are projected to the projection The plane corresponds to point p ₁ ” and point p ₂ ” respectively, z ₅ <z ₆ ; since z ₅ <z ₆ , z ₅ -z _c <z ₆ -z _c , so in scene one, point p ₁ and point p ₂ It is closer to the camera point camera; therefore, according to geometric knowledge, it can be determined that the distance between point p ₁ ' and point p ₂ ' is greater than the distance between p ₁ " and point p ₂ "; specifically, |p ₂ "- p ₁ ”|/|p ₂ '-p ₁ '|=|z ₅ -z _c |/|z ₆ -z _c |. It can be seen that in three-dimensional space, when the distance between two points is determined and the line formed by the two points is parallel to the projection plane, the closer the distance between the two points to the projection plane, the closer the distance between the two points projected on the projection plane. In the same way, it can be concluded that the closer the object in the three-dimensional space is to the projection plane, the larger the area occupied by the projection onto the projection plane.

In this embodiment, the image frame f ₃ is predicted according to the image frame f ₁ and the image frame f _2. Therefore, the coordinates of the block B _{1 in the image frame f 1} _and the block B ₂ in the image frame f ₂ in the Z dimension can be obtained according to the The prediction yields the coordinates of block B ₃ in the Z dimension. Specifically, please refer to FIG. 46C and FIG. 46D. FIG. 46C and FIG. 46D respectively illustrate schematic diagrams of two kinds of positional relationships; positional relationship one and positional relationship two.

For positional relationship 1: in conjunction with FIG. 46C and FIG. 46E, block B ₁ is located in image frame f ₁ , and the coordinate of block B ₁ in the Z dimension is z ₁ ; block B ₂ is located in image frame f ₂ , and block B ₂ is located in Z The coordinate in the dimension is z ₂ ; the block B ₃ is located in the image frame _f ₃ , assuming that the coordinate _of the block B ₃ in the Z dimension is z ₃ ; the reference block is B ₁ _; See Figure 46F for the relationship. The corresponding time point of the image frame f ₁ in the data stream is t ₁ , the corresponding time point of the image frame f ₂ in the data stream is t ₃ , and the corresponding time point of the image frame f ₃ in the data stream is t ₂ ; then calculate The difference between the time point t ₂ corresponding to the image frame f ₃ and the time point t ₁ corresponding to the image frame f ₁ , obtain the time difference value d ₁ =(t ₂ -t ₁ ); calculate the time corresponding to the image frame f ₂ The difference between the point t ₃ and the time point t ₁ corresponding to the image frame f ₁ is obtained as the time difference d ₂ =(t ₃ -t ₁ ); the ratio d ₁ /d ₂ of d ₁ and d ₂ is obtained. Calculate the difference between block B ₂ and block B ₁ in the Z dimension to obtain z ₂ -z ₁ ; then z ₃ can be calculated from z ₁ , z ₂ -z ₁ and d ₁ /d ₂ , specifically, z ₃ =z ₁ +(z ₂ −z ₁ )d ₁ /d ₂ . 46E, if the position of the camera point camera in the Z dimension is z _c ; the distance between the coordinates of the block B ₁ in the Z dimension and the camera point camera Δ ₁ =|z ₁ -z _c |; the Z dimension corresponding to the block B ₃ The distance between the coordinates and the camera point camera Δ ₂ =|z ₃ -z _c |; according to the conclusion that can be drawn from Figure 46E, the farther the distance of the same object in the three-dimensional space from the camera point, the smaller the area projected onto the projection plane; In this embodiment of the present application, block B ₁ and block B ₂ are matched blocks, that is, block B ₁ and block B ₂ display the same object in three-dimensional space; since block B ₃ is predicted based on block B ₁ and block B ₂ , therefore, block B ₃ , block B ₁ and block B ₂ all show the same object in three-dimensional space; in conjunction with FIG. 46E , the ratio of the side length L _B3 of block B ₃ to the side length L _B1 of block B ₁ is equal to Δ The ratio of ₁ to _Δ2 , ie L _B3 /L _B1 =Δ ₁ /Δ ₂ . The ratio of the area S _B3 of block B ₃ to the area S _B1 of block B ₁ is equal to the square of the ratio of Δ ₁ to Δ ₂ , S _B3 /S _B1 =(Δ ₁ /Δ ₂ ) ² . Then, in the case of position relationship one, the scaling ratio of the reference block B ₁ is (Δ ₁ /Δ ₂ ) ² . Expand the above formula: (Δ ₁ /Δ ₂ ) ² =(z ₁ -z _c ) ² /(z ₃ -z _c ) ² =(z ₁ -z _c ) ² /(z ₁ -z _c +(z ₂ -z ₁ )d ₁ /d ₂ ) ² =(z ₁ -z _c ) ² /(z ₁ -z _c +(z ₂ -z ₁ )(t ₂ -t ₁ )/(t ₃ -t ₁ )) ² .

For positional relationship two: in conjunction with FIG. 46D and FIG. 46E, block B ₁ is located in image frame f ₁ , and the coordinate of block B ₁ in the Z dimension is z ₁ ; block B ₂ is located in image frame f ₂ , and block B ₂ is located in Z The coordinate in the dimension is z ₂ ; the block B ₃ is located in the image frame _f ₃ , assuming that the coordinate _of the block B ₃ in the Z dimension is z ₃ ; the reference block is B ₂ _; See Figure 46H for the relationship. The corresponding time point of the image frame f ₁ in the data stream is t ₁ , the corresponding time point of the image frame f ₂ in the data stream is t ₂ , and the corresponding time point of the image frame f ₃ in the data stream is t ₃ ; then calculate The difference between the time point t ₃ corresponding to the image frame f ₃ and the time point t ₂ corresponding to the image frame f ₂ , obtain the time difference value d ₃ =(t ₃ -t ₂ ); calculate the time corresponding to the image frame f ₂ The difference between the point t ₂ and the time point t ₁ corresponding to the image frame f ₁ is obtained as the time difference d ₄ =(t ₂ -t ₁ ); the ratio d ₃ /d ₄ of d ₃ and d ₄ is obtained. Calculate the difference between block B ₂ and block B ₁ in the Z dimension, and obtain z ₂ -z ₁ ; z ₃ can be calculated from z ₂ , z ₂ -z ₁ and d ₃ /d ₄ , optional, z ₃ =z ₂ +(z ₂ -z ₁ )d ₃ /d ₄ . 46E, if the position of the camera point camera in the Z dimension is z _c ; the distance between the coordinates of the block B ₂ in the Z dimension and the camera point camera Δ ₃ =|z ₂ -z _c |; the Z dimension corresponding to the block B ₃ The distance between the coordinates and the camera point camera Δ ₄ =|z ₃ -z _c |; according to the conclusion that can be drawn from Figure 46E, the farther the distance of the same object in the three-dimensional space from the camera point, the smaller the area projected onto the projection plane; In this embodiment of the present application, block B ₁ and block B ₂ are matched blocks, that is, block B ₁ and block B ₂ display the same object in a three-dimensional space; since block B ₃ is predicted based on block B ₁ and block B ₂ , therefore, block B ₃ , block B ₁ and block B ₂ all show the same object in three-dimensional space; in conjunction with FIG. 46E , the ratio of the side length L _B3 of block B ₃ to the side length L _B2 of block B ₂ is equal to Δ The ratio of ₃ to Δ4, ie L _B3 /L _B2 =Δ ₃ /Δ ₄ _. The ratio of the area S _B3 of block B ₃ to the area S _B2 of block B ₂ is equal to the square of the ratio of Δ ₃ to Δ ₄ , S _B3 /S _B1 =(Δ ₃ /Δ ₄ ) ² . Then, in the case of the second positional relationship, the scaling ratio of the reference block B ₂ is (Δ ₃ /Δ ₄ ) ² . Expand the above formula: (Δ ₃ /Δ ₄ ) ² =(z ₂ -z _c ) ² /(z ₃ -z _c ) ² =(z ₂ -z _c ) ² /(z ₂ -z _c +(z ₂ -z ₁ )d ₃ /d ₄ ) ² =(z ₂ -z _c ) ² /(z ₂ -z _c +(z ₂ -z ₁ )(t ₃ -t ₂ )/(t ₂ -t ₁ )) ² .

S204, the electronic device generates a block B ₃ according to the reference block and the reference block scaling ratio.

Specifically, as described in S3703, for position relationship 1, block B ₃ can be obtained by scaling the reference block B ₁ by (Δ ₁ /Δ ₂ ) ² ; for position relationship 2, scaling the reference block B ₂ (Δ ₃ /Δ ) ₄ ) ² to get block B ₃ .

S205 , the electronic device determines, according to the coordinates of the reference block on the plane where the image frame is located, the predicted displacement vector of the reference block and the scaling ratio of the reference block, the coordinates of the block _B3 on the plane where the image frame is located.

According to the explanation of step S3703, in conjunction with FIG. 46C, in the case of position relationship 1: if the coordinates of the center point of block B ₁ are (x ₁ , y ₁ ), then any point on block B ₁ (x _b1 , y _b1 ) ) scaled by the center point is (x _b1 *Δ ₁ /Δ ₂ +x ₁ *(Δ ₂ -Δ ₁ )/Δ ₂ , y _b1 *Δ ₁ /Δ ₂ +y ₁ *(Δ ₂ - Δ ₁ )/Δ ₂ ); the scaled coordinates are added to the predicted displacement vector, and with reference to FIG. 46C , the coordinates of the block B ₃ on the plane where the image frame is located can be obtained, that is, B ₃ (x _b1 *Δ ₁ /Δ _{2 )} +x ₁ *(Δ ₂ -Δ ₁ )/Δ ₂ +(t ₂ -t ₁ )/(t ₃ -t ₁ )*(x ₂ -x ₁ ), y _b1 *Δ ₁ /Δ ₂ +y ₁ *(Δ ₂ -Δ ₁ )/Δ ₂ +(t ₂ -t ₁ )/(t ₃ -t ₁ )*(y ₂ -y ₁ )).

46D, in the case of position relationship 2: if the coordinates of the center point of block B ₂ are (x ₂ , y ₂ ), then any point (x _b2 , y _b2 ) on block B ₂ is scaled with the center point The coordinates of (x _b2 *Δ ₃ /Δ ₄ +x ₂ *(Δ ₄ -Δ ₃ )/Δ ₄ , y _b2 *Δ ₃ /Δ ₄ +y ₃ *(Δ ₄ -Δ ₃ )/Δ ₄ ) ; The scaled coordinates are added to the predicted displacement vector, and combined with Figure 46D, the coordinates of the block B ₃ in the plane where the image frame is located can be obtained, that is, B ₃ (x _b2 *Δ ₃ /Δ ₄ +x ₂ *(Δ ₄ -Δ ₃ )/Δ ₄ +(t ₃ -t ₂ )/t ₂ -t ₁ )*(x ₂ -x ₁ ), y _b2 *Δ ₃ /Δ ₄ +y ₃ *(Δ ₄ -Δ ₃ ) /Δ ₄ +(t ₃ -t ₂ )/t ₂ -t ₁ )*(y ₂ -y ₁ )).

S206, the electronic device generates an image frame _f3 according to the coordinates of the block _B3 and the block _B3 on the plane where the image frame is located.

46B and 46C; for S204-S206, please refer to FIG. 46G, the electronic device first scales the block B ₁ according to the scaling ratio, and the scaled block B ₁ is denoted as block B ₄ ; then, according to the predicted displacement vector β ₂ The block B ₄ is translated to obtain the block B _{3 in the image frame f 3} _; finally, a plurality of blocks B ₃ constitute the image frame f ₃ .

46B and 46D; for S204-S206, please refer to FIG. 46I, the electronic device first scales the block B ₂ according to the scaling ratio, and the scaled block B ₂ is denoted as block B ₅ ; then, according to the predicted displacement vector β ₂ The block B ₅ is translated to obtain the block B _{3 in the image frame f 3} _; finally, a plurality of blocks B ₃ constitute the image frame f ₃ .

It should be noted that FIG. 46G and FIG. 46I are for better understanding of this solution; specifically, the electronic device can generate the data of the image frame f ₃ according to the data of the reference block, the scaling ratio and the predicted displacement vector β ₂ , and according to The data of image frame _f3 shows image frame _f3 .

It can be seen that, in the embodiment of the present application, when the electronic device generates the predicted image frame (image frame f ₃ ) according to the original image frame (image frame f ₁ and image frame f ₂ ), it can combine the Z dimension perpendicular to the plane where the image frame is located. The coordinates scale the blocks in the image frame, so that after inserting the predicted image frame, the transition between the original image frame and the predicted image frame is more natural; it makes the predicted image frame observed by the user and the electronic device constructed according to the data of the application. The scene fits better, improving the visual experience when inserting predicted image frames.

Referring to FIG. 47 , in this embodiment of the present application, the electronic device may create a storage space 401 in the cache 1352 , and the storage space 401 includes accessories. Specifically, the type and quantity of the accessories may be determined according to the actual situation. Combining S3701 to 106, S201 to S205; the electronic device can set A-type attachments and B-type attachments; the electronic device can store the corresponding drawing results in the A-type attachments and B-type attachments; specifically, the storage space 401 may specifically refer to OpenGL The frame buffer object (Frame Buffer Object, FBO) in the; A type attachment can be a color attachment, in OpenGL, it can be ColorAttachment, which can be used to store the color data in the image frame, such as the RGB value of the pixel in the image frame, etc., and at the same time Type A attachments may also include the coordinates of the pixels in the image frame. For example, in S3702, the electronic device may determine the coordinates of the block according to the coordinates of the pixels, and determine the predicted displacement vector according to the coordinates of the block. The B-type attachment can store the depth attachment of the depth information of the image frame. In OpenGL, it can specifically be: Depth Attachment, and the depth information can specifically be the data of the Z dimension described in S3701 to S3705, and S201 to S205.

The storage space in the embodiment of the present application is a space defined in the cache of the electronic device. The storage space includes the following functions: (1) Store the drawing results generated according to the data of the application; specifically, the storage space can include at least one attachment, and each attachment includes at least one buffer type, and the buffer type can include a color type , depth type and template type; the electronic device can store the drawing results on different attachments according to the drawing instructions, and finally form an image frame in the storage space. (2) Storing and synthesizing image frames in the storage space: The electronic device can draw the drawing results into a storage space, and then generate image frames. At least one storage space for storing image frames can be set; the electronic device can store different image frames in different storage spaces, for example, the first image frame is stored in the first storage space, and the second image frame is stored in the second storage space. image frame. In this embodiment of the present application, the electronic device may generate the predicted image frame in the third storage space according to the first image frame in the first storage space and the second image frame in the second storage space.

In this embodiment of the present application, the location of the storage space on the hardware device may be specifically set in the random access memory of the graphics processor 1351, or may be set in the random access memory communicatively connected with the GPU.

Some common names for the storage space in the embodiments of the present application may be: frame buffer memory, frame buffer for short, or frame buffer; it should be noted that the storage space may be called by other different names in different programming languages.

A frame buffer includes at least one attachment; an attachment includes at least one buffer type; the buffer type may include: color type, depth type and template type. For example, the frame buffer may include three attachments, which may specifically be: a color type attachment, a depth type attachment and a template type attachment.

For example, the storage space in the embodiments of the present application may be called vkframebuffer in the Vulkan programming language; it may be called a frame buffer object (Frame Buffer Object, FBO) in the Open Graphics Library (Open Graphics Library, OpenGL) programming language.

Please refer to FIG. 48. FIG. 48 is another schematic diagram of an electronic device 100 provided by an embodiment of the present application; the software block diagram of the electronic device 100 shown in FIG. 43 and FIG. 44A and FIG. 12 above is combined. The electronic device 100 may include an application processor 1353, a graphics processor 1351 and a display device 1350; in this embodiment of the present application, the application processor 1353 may run the target application 1301, and according to the data and data in the running process of the target application 1301 It is required to call the API in the 3D graphics processing framework to realize the processing of application data; data communication can be performed between the application processor 1353 and the graphics processor 1351, and the three-dimensional graphics processing framework in the application processor 1353 can communicate with the graphics processor 1351. The 3D graphics processing server in the 3D graphics processing server sends the data of the target application 1301 and the instruction to execute the API; in response to the application data and the instruction to execute the API sent by the 3D graphics processing framework, the 3D graphics processing server can call the 3D graphics processing library 1330 to perform the rendering operation , and finally store the rendered image frame in the cache 1352 ; then the graphics processor 1351 transmits the image frame in the cache to the display device 1350 for display. In this embodiment of the present application, the cache 1352 may specifically refer to a dynamic random access memory in the graphics processor 1351 . For example, in this example, the three-dimensional graphics processing library 1330 may be OpenGL ES; the three-dimensional graphics processing framework may be an OpenGL ES framework; and the three-dimensional graphics processing server may be an OpenGL ES server.

Before the electronic device performs the steps of S3701 to 106, the electronic device first starts the application program, and allocates and/or creates a storage space according to the requirements of the application program; then performs the steps of S3701 to S106 in the storage space. For example, the electronic device runs OpenGL, and the storage space is FBO; before the steps of S3701 to S106, the electronic device includes: when the electronic device starts the application, the electronic device assigns the FBO 0 corresponding to the application, and then the electronic device according to The data call creation instruction of the application program creates at least three FBOs. In this embodiment of the present application, when the steps from S3701 to S106 are executed, the electronic device may only create three FBOs: FBO 1, FBO 2 and FBO 3; FBO 1 is used for storage The drawing result corresponding to the image frame f ₁ , the FBO 2 is used to store the drawing result corresponding to the image frame f ₂ , and the FBO 3 is used to store the drawing result corresponding to the image frame f ₃ . The specific process of allocating FBO 0 and creating FBO 1, FBO 2, and FBO 3 above can refer to FIG. 49A, which includes S301 and S302:

S301, the application processor 1353 starts the application program, and the application processor 1353 instructs the graphics processor 1351 to configure FBO 0 for the application program.

S302, the application processor 1353 calls the creation instruction for creating the storage space, and sends the creation instruction to the graphics processor 1351, and the graphics processor 1351 creates FBO 1, FBO 2 and FBO 3.

The creation instruction can be the glGenFramebuffers API in OpenGL.

Wherein, FBO 1 , FBO 2 and FBO 3 may be used to store the drawing results corresponding to the image frame f ₁ , the image frame f ₂ and the image frame f ₃ respectively.

Wherein, after the electronic device performs S301, please refer to FIG. _49B for the process _of drawing and displaying the image f1 or the image frame f2 by the electronic device. FIG. 49B uses the electronic device to draw and display the image frame f1 as _an example, including S401-S406.

S401, the application processor 1353 invokes a binding instruction for binding the application program to the storage space, and sends the binding instruction to the graphics processor 1351, and the graphics processor 1351 binds the application program to the FBO 1.

Specifically, the binding instruction may be the glBindFrameBuffer API in OpenGL. When the application is bound to the FBO, the drawing result obtained by the electronic device drawing according to the data of the application is stored in the bound FBO.

S402, the application processor 1353 invokes the attachment adding instruction for adding an attachment in the storage space, and sends the attachment adding instruction to the graphics processor 1351, and the graphics processor 1351 adds the A-type attachment and the B-type attachment in the FBO 1.

Among them, the attachment adding instruction is used to add attachments in the FBO. The attachment adding instruction can be the glFramebufferTexture2D API in OpenGL. Type A attachments can be color attachments, which in OpenGL can be: ColorAttachment; Type B attachments can be depth attachments, which can be Depth Attachment in OpenGL. The information of the color value of the pixel and the coordinate information of the pixel in the image frame are stored in the A-type attachment, which specifically includes the coordinates in the plane where the image frame is located as described in S3702. Depth information is stored in the B-type attachment, which specifically includes a depth value. The depth value can represent the data of the Z dimension perpendicular to the projection plane in the camera coordinate system.

S403, the application processor 1353 calls the drawing instruction for drawing, and sends the drawing instruction to the graphics processor 1351, the graphics processor 1351 performs the drawing operation, and stores the drawing result in the A-type attachment and B-type attachment of the FBO 1 , get the image frame f ₁ .

Wherein, the drawing instructions may include a variety of different drawing instructions, so as to realize different functions when performing the drawing operation. The drawing instruction can be the glDraw API in OpenGL; the glDraw API can include the glDrawArrays API and the glDrawElements API.

S404, the application processor 1353 calls the attachment binding instruction, and sends the attachment binding instruction to the graphics processor 1351, and the graphics processor 1351 binds the A-type attachment and B-type attachment in FBO 1 to FBO 0;

It should be noted that the functions of FBO 0 and FBO K in the embodiments of the present application are different, and K is a positive integer.

FBO K is used to store the drawing results/data of the application; the function of FBO 0 is: the electronic device can send the image frame in FBO 0 to the display screen. Specifically, when the electronic device runs the application, at least one FBO K can be created for the application according to the data storage requirements of the application. FBO 0 is assigned to the application when the electronic device application starts. When an electronic device displays an image frame, it needs to draw (copy) the image frame in FBO K to FBO 0 first, so that the electronic device can send and display the image frame in FBO 0 when the vertical synchronization signal arrives, and then display the image frame in FBO 0. The image frame is displayed on the screen. The electronic device sets at least one FBO 0.

In the case where the number of FBO 0s set by the electronic device is greater than two: the electronic device can send the image frame in one of the FBO 0 to the display device, and at the same time, draw the image frame in the remaining FBO 0, thereby reducing the " The time of the whole process of "draw to display" improves the efficiency of "generate to display" image frames of electronic devices. For example, the application program of the electronic device is set with FBO 1, FBO 2 and FBO 3, and the FBO 0 of the application program includes FBO 01 and FBO 02. After the electronic device draws in FBO 1 and obtains the drawing result corresponding to the image frame f ₁ , The drawing result corresponding to the image frame f ₁ is drawn to the FBO 01. After that, the electronic device calls the rotation instruction. The rotation instruction can be the eglSwapBuffers API in OpenGL, and the drawing result corresponding to the image frame f ₁ in the FBO 01 is sent to the display device for display. Display; in the process of sending the drawing result corresponding to the image frame f ₁ in the FBO 01 to the display device for display: the electronic device can draw in the FBO 2 to obtain the drawing result corresponding to the image frame f ₂ , and after the completion of the drawing result for the image frame f 2 After the image frame f ₂ is drawn, the drawing result corresponding to the image frame f ₂ in the FBO 2 is drawn into the FBO 02 . After that, the electronic device calls the rotation instruction to send the drawing result corresponding to the image frame f ₂ in the FBO 02 to the display device for display; at the same time, the electronic device can according to the drawing result corresponding to the image frame f ₁ in the FBO 1 and the image in the FBO 2 The drawing result corresponding to frame f ₂ is drawn in FBO 3 to obtain the drawing result corresponding to image frame f ₃ , and after the drawing of image frame f ₃ is completed, the drawing result corresponding to image frame f ₃ in FBO 3 is drawn to FBO 01 middle.

Among them, the electronic device binds the A-type accessories and B-type accessories in FBO 1 to FBO 0, which can be regarded as the electronic device taking the A-type accessories and B-type accessories in FBO 1 as the input of FBO 0, so as to facilitate the follow-up according to FBO 0 The input is processed, and the final image frame that can be displayed on the display is obtained at FBO 0.

The attachment binding instruction may specifically be the glBindTexture API in OpenGL.

S405, the application processor 1353 calls the drawing instruction, and sends the drawing instruction to the graphics processor 1351, and the graphics processor 1351 performs the drawing operation on the drawing results in the A-type attachment and B-type attachment, generates an image frame f ₁ , and converts the image frame f ₁ is stored in FBO 0.

Specifically, the drawing instruction may be the glDraw API in OpenGL. The drawing operation performed in S306 is different from the drawing operation performed in S304.

S406, the application processor 1353 calls the rotation instruction for sending the image frame for display, and sends the rotation instruction to the graphics processor 1351, and the graphics processor 1351 transmits the image frame in the FBO 0 to the display screen 340, and the display screen 340 displays the image frame.

Specifically, the rotation instruction may be the eglSwapBuffers API in OpenGL. When FBO 0 contains two FBOs, the electronic device can call the rotation instruction to alternately display the FBOs in FBO 0. Specifically, FBO 0 includes FBO 01 and FBO 02. If the electronic device currently sends and displays the image frame in FBO 01, and the electronic device calls the rotation instruction eglSwapBuffers API, the electronic device sends the image frame in FBO 02 for display. When the rotation command eglSwapBuffers API is called again, the electronic device will send and display the image frame in FBO 01. Repeating the above operation can realize the process of alternately sending and displaying the image frames in FBO 01 and FBO 02.

Wherein, after the electronic device completes S301, please refer to FIG. 49C for the process of drawing and displaying the image _f3 by the electronic device, and FIG. 49C includes S501-S507. It can be determined from S3701 to S106 that when the electronic device generates the image frame _f3 , the electronic device has completed the drawing _of the image frame f1 and the image frame f2 _; in this example, the drawing result corresponding to the image frame f1 is stored in the _FBO1 , The drawing result corresponding to the image frame f ₂ is stored in the FBO 2, the A-type attachment corresponding to the image frame f ₁ is denoted as A _f1 , the B-type attachment corresponding to the image frame f ₂ is denoted as B _f1 ; the A-type attachment corresponding to the image frame f ₂ The attachment is denoted as A _f2 , and the B-type attachment corresponding to the image frame f ₂ is denoted as B _f2 .

S501, the application processor 1353 invokes the binding instruction for binding the application program to the storage space, and sends the binding instruction to the graphics processor 1351, and the graphics processor 1351 binds the application program to the FBO 3.

Specifically, the binding instruction may be the glBindFrameBuffer API in OpenGL.

S502, the application processor 1353 calls the attachment binding instruction, and sends the attachment binding instruction to the graphics processor 1351, and the graphics processor 1351 associates A _f1 and B _f1 in FBO 1 and A _f2 and B _f2 in FBO 2 with the FBO 3 to bind.

Among them, A _f1 , B _f1 , A _f2 and B _f2 are bound to FBO 3; the specific function is: when the graphics processor 1351 performs subsequent drawing operations, A _f1 , B _f1 , A _f2 and B _f2 are used as inputs, The drawing results obtained from A _f1 , B _f1 , A _f2 and B _f2 are stored in FBO 3 .

S503 , the application processor 1353 invokes the attachment adding instruction, and sends the attachment adding instruction to the graphics processor 1351 , and the graphics processor 1351 adds an A-type attachment in the FBO 3 , denoted as A _f3 .

Among them, the A-type attachment is specifically a color attachment, and in OpenGL, it can specifically be: ColorAttachment.

Among them, the attachment adding instruction can be specifically in OpenGL: glFramebufferTexture2D API.

S504, the application processor 1353 calls the drawing instruction, and sends the drawing instruction to the graphics processor 1351, and the graphics processor 1351 performs the drawing operation according to A _f1 , B _f1 , A _f2 and B _f2 , and stores the obtained drawing result in A _f3 .

Specifically, the drawing instruction may be the glDraw API in OpenGL.

Wherein, the drawing operation is performed according to A _f1 , B _f1 , A _f2 and B _f2 in S405 , and the specific algorithms to be performed may be the algorithms described in S3701 to 106 and S201 to 205 .

S505, the application processor 1353 calls the attachment binding instruction, and sends the attachment binding instruction to the graphics processor 1351, and the graphics processor 1351 binds A _f3 in FBO 3 to FBO 0;

Among them, please refer to S307 for the explanation of FBO 0.

Among them, the electronic device binds A _f3 in FBO 3 to FBO 0, which can be regarded as the electronic device using A _f3 in FBO 3 as the input of FBO 0, so as to facilitate subsequent drawing operations according to the input of FBO 0 to obtain image frames .

S506, the application processor 1353 calls the drawing instruction, and sends the drawing instruction to the graphics processor 1351, and the graphics processor 1351 performs the drawing operation according to the drawing result in A _f3 , generates the image frame f ₃ , and stores the image frame f ₃ in the FBO 0 middle.

Specifically, the drawing instruction may be the glDraw API in OpenGL. The drawing operation in S407 is different from that in S405.

S507, the application processor 1353 calls the rotation instruction for sending the image frame for display, and sends the rotation instruction to the graphics processor 1351, and the graphics processor 1351 transmits the image frame _f3 in the FBO 0 to the display screen 340, and the display screen 340 Image frame _f3 is displayed.

Specifically, the rotation instruction may be the eglSwapBuffers API in OpenGL.

The game interface displayed by the electronic device may include game content and UI controls. Generally, the position and size of UI controls in the game interface do not change. Therefore, in the embodiment of the present application, when the electronic device draws the drawing frame issued by the application program, the electronic device can draw the UI controls in the drawing frame of the first frame, and the subsequent drawing frames can directly use the drawing frame of the first frame to draw the frame. The drawing result of the UI controls in . In this way, repeated drawing can be avoided and the power consumption of the electronic device can be saved.

In the embodiment of the present application, the image frame generated according to the data of the target application program is divided into a dynamic layer and a user interface layer (User Interface, UI) layer; it should be noted that the above division is only for the purpose of more A good understanding of the technical solutions in the embodiments of the present application; in practice, the electronic device performs a drawing operation according to the data of the target application, and finally generates an image frame, and there is no concept of a dynamic layer and a UI layer.

For example, at present, playing video games on electronic devices has become one of the important ways for many users to entertain and relax. During the process of running the game application program, the electronic device constructs a smooth game screen by displaying frame-by-frame image frames on the display screen. Please refer to FIG. 50A . FIG. 50A is a schematic diagram of a dynamic layer in a game scene. The dynamic layer may be a game screen in the game scene, and the game screen may change continuously with the actual operation of the game application.

The UI layer defined in the embodiment of the present application mainly includes controls in the display interface. Usually, the positions of the controls in the display interface are relatively fixed, and the controls exist on the display interface during the entire process of running the target application; for example, , navigation bar, window frame, text box, button, drop-down menu, etc.; specifically, in the game scene, the controls may refer to the control buttons, setting buttons, etc. in the game application program, for example, please refer to Figure 50B, 50B is a schematic diagram of a UI layer in a game scene. As shown in the figure, the controls may include direction indicators, game duration, game delay, game setting buttons, and game control buttons; it can be seen that the above controls always exist on the display screen in the game scene. Only the displayed data changes. For example, the game lag control will update the lag value according to changes in network conditions, but the game lag control will always be displayed on the display.

The image frame 1C finally displayed by the electronic device may be composed of the dynamic layer as shown in FIG. 50A and the UI layer as shown in FIG. 50B defined in the embodiments of the present application.

The storage space in the embodiment of the present application is a space defined by the electronic device in the cache, and the storage space can be regarded as a mount point, and an association between a texture image and a rendering object is established through the mount point. The storage space includes the following functions: (1) Store the drawing results generated according to the data of the target application; specifically, the storage space can include at least one attachment, and each attachment includes at least one buffer type, and the buffer type can include color type, depth type and template type; the electronic device can store the drawing results on different attachments according to the drawing instructions, and finally form an image frame in the storage space. (2) Storing and synthesizing image frames in the storage space: The electronic device can store the drawing results stored in multiple storage spaces in the same storage space, and synthesize image frames in the storage space. (3) When the vertical synchronization signal VSYNC arrives, the stored image frames are transmitted to the display screen for display; at least one storage space for storing image frames can be set; when the number of storage spaces for storing image frames is greater than or equal to 2 , the target application can call the first instruction to rotate the storage space in which the image frame is stored; for example, if the ninth storage space for storing the image frame consists of the thirteenth storage space and the fourteenth storage space, Then when the target application program calls the first instruction, the electronic device can firstly rotate the thirteenth storage space storing the image frame to the first state, and when the thirteenth storage space is in the first state, when the vertical synchronization signal VSYNC arrives. The image frame in the thirteenth storage space can be transmitted to the display device, and then the display device displays the image frame; when the thirteenth storage space is in the first state, the electronic device can store the image frame in the fourteenth storage space according to the drawing instruction of the target application. Drawing in the space to generate the eighth image frame; when the target application calls the first instruction again, the electronic device rotates the fourteenth storage space to the first state, and draws in the thirteenth storage space to generate the ninth image frame; By repeating the above operations, the display of the screen can be realized. When the storage space is in the second state, the electronic device may draw or synthesize image frames in the storage space according to the instruction of the target application. In this embodiment of the present application, when the ninth storage space is composed of at least two storage spaces, an off-screen rendering function can be implemented. When the target application in the electronic device is started, the electronic device can configure at least one storage space for the target application; when the electronic device runs the target application, the electronic device can select a storage space for binding, and then the electronic device can The drawing results and rendering results obtained by drawing can be stored in the bound storage space according to the drawing instructions of the target application; specifically, the electronic device can store the obtained drawing results in different storage spaces according to the drawing instructions. It should be noted that, in the embodiment of the present application, in the case where one storage space is composed of multiple storage spaces, it means that the multiple storage spaces have the same at least one characteristic in the embodiment of the present application; for example, the In the claims, the seventh storage space is composed of the tenth storage space and the eleventh storage space, indicating that the tenth storage space and the eleventh storage space have the same characteristics, that is, the tenth storage space and the eleventh storage space are both used. for storing the drawing result when the drawing condition satisfies the first condition.

In the embodiment of the present application, the location of the storage space on the hardware device may be specifically set in a random access memory of a graphics processing unit (graphics processing unit, GPU), or may be set in a random access memory communicatively connected with the GPU.

Some common names of the storage space in the embodiments of the present application may be: frame buffer memory, frame buffer for short, or frame buffer; it should be noted that there may be other different names in different programming languages.

A frame buffer includes at least one attachment; an attachment includes at least one buffer type; the buffer type may include: color type, depth type and template type. For example, the frame buffer may include three attachments, which may specifically be: a color type attachment, a depth type attachment, and a template type attachment.

For example, the storage space in the embodiments of the present application may be called vkframebuffer in the Vulkan programming language; it may be called a frame buffer object (Frame Buffer Object, FBO) in the Open Graphics Library (Open Graphics Library, OpenGL) programming language. The following is a detailed description of FBO in OpenGL: In OpenGL, to create an FBO, at least the following conditions must be met: FBO contains at least one attachment, which can be a color attachment, a depth attachment, or a stencil buffer attachment; at least one of the attachments is a color attachment ; and the above attachment is already in the cache. After the target application is started, it can first call the glGenFramebuffers application programming interface (application programming interface, API) in OpenGL to generate at least one FBO, and the glGenFramebuffers API is specifically used to create and generate FBO; then, according to the drawing instructions of the target application, call glBindFrameBuffer The API binds the target application to one of at least one FBO. The glBindFrameBuffer API is specifically used to bind the target application to the FBO; when the target application is bound to the FBO, the target application can call The glDrawElements API performs the drawing operation, and the glDrawElements API is the drawing instruction, which is specifically used to perform the drawing operation in the FBO; after that, the drawing result is stored in the bound FBO; after that, the existing drawing result in at least one FBO is stored in the rotation In FBO, and synthesize image frames in FBO; when receiving eglSwapBuffers API, rotate for rotating FBO, so that in the first state, the image frame in rotating FBO can be sent to the display device when VSYNC arrives; eglSwapBuffers The API is the specific representation of the first instruction in OpenGL.

In some embodiments, the electronic device may configure at least one storage space for the target application; when the target application is bound to different storage spaces, the electronic device may store different drawing results in different storage spaces according to the drawing instruction middle.

For example, in the embodiment of the present application, the electronic device stores the drawing result in different storage spaces according to the drawing instruction of the target application, and when the drawing instruction satisfies the first condition, stores the drawing result in the seventh storage space; drawing When the instruction satisfies the second condition, the drawing result is stored in the eighth storage space; in the embodiment of the present application, the first condition is that the drawing instruction includes an instruction for enabling depth testing, and the drawing result can present a three-dimensional effect. The drawing result below can be regarded as being used to present the dynamic layer shown in FIG. 50A ; the second condition is that the drawing instruction includes the case where the depth test is disabled, and the drawing result can only present a two-dimensional effect. Therefore, the drawing result under the second condition can be regarded as being used to render the UI layer shown in FIG. 50B . Finally, the electronic device mixes the drawing result in the seventh storage space and the drawing result in the eighth storage space in the ninth storage space, so as to synthesize the image frame shown in FIG. 50C in the ninth storage space.

In the embodiment of the present application, when the drawing instruction satisfies the first condition, the drawing operation is performed according to the drawing instruction to obtain the first drawing result; if the first drawing result is displayed on the display screen, the presented effect may be as shown in FIG. 50A The three-dimensional layer/image shown; when the drawing instruction satisfies the second condition, the drawing operation is performed according to the drawing instruction to obtain the second drawing result. If the second drawing result is finally presented on the display screen, the presented effect can be as follows: The two-dimensional layer/image shown in FIG. 50B ; the electronic device finally mixes the corresponding drawing results when the drawing instructions meet different conditions to obtain a final image frame, as shown in FIG. 50C . Specifically, in the embodiment of the present application, the drawing process of the electronic device executing the target application is shown in FIG. 50D . It can be seen that during the drawing process of the target application, the drawing instruction of the target application satisfies the first condition and the Continuously switch between satisfying the second condition; every two image frames in the image frame of the electronic device share a drawing result when the drawing instruction satisfies the second condition, so that the electronic device can reduce the drawing instruction by half when generating the image frame to satisfy the second condition. Conditional drawing operations reduce the power consumption of electronic devices. It should be noted that, in the game scene, the dynamic layer as shown in Figure 50A is the fundamental reason that affects the game experience. When the frame rate of the dynamic layer is high, the user will feel that the game screen is smooth; The game screen may feel stuck, and the game application may drop frames; the UI layer shown in Figure 50B changes little in the game scene, which is not the root cause of affecting the game experience. Therefore, in the embodiment of the present application, every two frames of images share a drawing instruction corresponding to the drawing result when the second condition is satisfied, which reduces the power consumption of the electronic device with little or no impact on the game experience. Specifically, in the embodiment of the present application, the first condition refers to that the drawing instruction includes an instruction to enable depth testing; the second condition refers to that the drawing instruction includes an instruction to disable depth testing.

In this embodiment of the present application, one drawing period represents the time taken for the entire drawing process of one image frame from the start of drawing to the completion of drawing the image frame; specifically, when the electronic device runs the target application, the electronic device can The drawing instruction determines the time points when the image frame starts and ends, and then determines the drawing period of the image frame. Specifically, the time point when the target application program calls the first instruction may be the time point when two adjacent drawing stages end and start; please refer to the explanation of the storage space for the specific function of the first instruction.

An embodiment of the present application provides a method for generating an image frame, so as to reduce power consumption when an electronic device runs a game application.

Exemplarily, please refer to FIG. 50E. FIG. 50E is a possible schematic diagram of generating an image frame in this embodiment of the present application. As shown in FIG. 50E,

Step (1): when the electronic device detects that the target application is started, the electronic device first creates a seventh storage space 5001, an eighth storage space 5002 and a ninth storage space 5003 associated with the target application;

Step (2): when the drawing instruction of the target application satisfies the first condition, the drawing result is stored in the seventh storage space 5001 as the first drawing result;

Step (3): when the drawing instruction of the target application satisfies the second condition, the drawing result is stored in the seventh storage space 5002 as the second drawing result;

Step (4): store the first drawing result in the seventh storage space 5001 and the second drawing result in the eighth storage space 5002 in the ninth storage space 5003, and in the ninth storage space 5003 according to the first drawing result and synthesizing the seventh image frame with the second drawing result;

Step (5): when the drawing instruction of the target application satisfies the first condition, the drawing result is overwritten and stored in the seventh storage space 5001 as the third drawing result;

Step (6): store the third drawing result in the seventh storage space 5001 and the second drawing result in the eighth storage space 5002 into the ninth storage space 5003, and in the ninth storage space 5003 according to the stored three The drawing result and the second drawing result are combined into an eighth image frame.

Wherein, step (2), step (3) and step (4) may constitute a first drawing cycle; step (5) and step (6) may constitute a second drawing cycle.

It should be noted that, in this example, the ninth storage space may include at least two storage spaces.

After the electronic device synthesizes the image frames in the target storage space, it needs to transmit the image frames to the display screen for display, and then delete the image frames to facilitate the synthesis and storage of the next image frame. For example, in FIG. 50E , the electronic device first synthesizes the seventh image frame, then transmits the seventh image frame in the ninth storage space 5003 to the display screen for display, and then clears the ninth storage space 5003 so as to store the seventh image frame in the ninth storage space 5003 for display. The eighth image frame is synthesized in space 5003 . It should be noted that the electronic device can also be set with multiple storage spaces, and the multiple storage spaces constitute the ninth storage space 5003; the generated image frames are stored in different storage spaces, and then the rotation of the storage spaces is realized according to the first instruction, Then, the image frames in the storage space are continuously transmitted to the display screen for display. For example, a thirteenth storage space and an eighth storage space can be set in the ninth storage space; when the thirteenth storage space for storing the seventh image frame is in the first state, that is, when the seventh image frame is displayed on the display screen of the electronic device, The electronic device may synthesize the eighth image frame in the fourteenth storage space.

It can be seen that in the embodiment of the present application, the drawing results obtained when the drawing instructions meet different conditions are stored separately, which realizes the multiplexing of some drawing results, reduces the data processing amount of the electronic device when running the target application program, and reduces the electronic device. Power consumption while running the target application.

As shown in Figure 51A, the method may include:

S5101, in the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, store the drawing result in the seventh storage space as the first drawing result, when the drawing instruction satisfies the second condition , and the drawing result is stored in the eighth storage space as the second drawing result.

The time nodes at the start and end of two adjacent drawing cycles may be the time points when the target application calls the first instruction, and the first instruction is used to rotate the storage space; for example, when the drawing instruction is an instruction in OpenGL , the first drawing finger can be eglSwapBuffers API;

In this example, when the electronic device intercepts the first instruction for the first time, the electronic device can create a seventh storage space and an eighth storage space; the seventh storage space is used to store the drawing result when the first condition is satisfied in the drawing instruction; the eighth storage space The storage space is used to store the drawing result when the second condition is satisfied in the drawing instruction;

Among them, the electronic device is provided with an interception module; the interception module can intercept the instructions of the target application, and perform operations according to the intercepted instructions; when the electronic device intercepts the drawing instructions of the target application, if the drawing execution includes the enabling depth If the test instruction is used, it is determined that the drawing instruction satisfies the first condition; if the drawing instruction includes an instruction to close the depth test, it is determined that the drawing instruction satisfies the second condition. When the drawing instruction satisfies the first condition, the electronic device is performing the drawing process according to the target application, and the depth test is enabled; when the drawing instruction satisfies the second condition, the electronic device is performing the drawing process according to the target application, and the depth test is is closed.

In the embodiment of the present application, the electronic device is provided with an interception module, and the interception module can intercept the drawing instruction of the target application, realize the identification of the drawing instruction, and then judge that the drawing instruction satisfies the first condition or the second condition.

The electronic device is provided with a counting unit, the initialization value of the counting unit is the first value, and the value of the counting unit is switched between the first value and the second value every time the value of the counting unit is updated; the moment when the drawing cycle starts is the moment when the counting unit is updated In a drawing cycle, if the numerical value of the updated counting unit is the second numerical value, then the eighth storage space is emptied; and when the drawing instruction meets the second condition, the drawing result is stored in the eighth storage space; if after the update If the value of the count unit is the first value, the second storage unit is not emptied; and when the drawing instruction satisfies the second condition, the drawing is not performed. Since the time nodes at the start and end of two adjacent drawing cycles may be the time points at which the target application calls the first instruction; therefore, when the intercepting module intercepts the first instruction once, the counting unit is updated once. Specifically, the first value may be 0, and the second value may be 1.

Wherein, if the first drawing result is presented on the display device, the final presented display effect may be as shown in FIG. 50A .

S5102, generating a seventh image frame according to the first drawing result and the second drawing result;

Wherein, the electronic device may synthesize the seventh image frame in the seventh storage space 5001 ; and may also synthesize the image frame in the ninth storage space 5003 . If the electronic device synthesizes the seventh image frame in the seventh storage space 5001, the seventh storage space 5001 may be composed of at least two storage spaces. If the electrons synthesize the seventh image frame in the ninth storage space 5003, the ninth storage space may be composed of at least two storage spaces. The electronic device can rotate the storage space for storing the image frames, so that the image frames in the storage space can be transmitted to the display device, and then displayed by the display device.

Please refer to FIG. 51B , which is a schematic diagram of synthesizing a seventh image frame in the ninth storage space 5003 according to the first drawing result in the seventh storage space 5001 and the second drawing result in the eighth storage space 5002 .

Please refer to FIG. 51C , which is a schematic diagram of synthesizing a seventh image frame in the seventh storage space 5001 according to the first drawing result in the seventh storage space 5001 and the second drawing result in the eighth storage space 5002 .

The effect that the seventh image frame is finally presented on the display device may be as shown in FIG. 50C .

S5103, in the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, store the drawing result in the seventh storage space as the third drawing result, when the drawing instruction satisfies the second condition , do not draw;

Wherein, if the third drawing result is presented on the display device, the final presented display effect may be as shown in Figure 51D;

S5104: Generate an eighth image frame according to the third rendering result and the second rendering result.

The effect that the eighth image frame is finally presented on the display device may be as shown in FIG. 51E .

Wherein, please refer to the explanation of step S5102. Similarly, the eighth image frame may be synthesized in the seventh storage space 5001 or may be synthesized in the ninth storage space 5003.

It can be seen that in the above example, the electronic device stores the first drawing result obtained when the drawing instruction satisfies the first condition and the second drawing instruction obtained when the drawing instruction satisfies the second condition separately in the first drawing cycle, and according to the first drawing result and the second drawing result to generate the seventh image frame; in the second drawing cycle, the electronic device only needs to draw when the drawing instruction meets the first condition, obtain the third drawing result, reuse the second drawing result, and draw according to the second drawing The result and the third drawing result generate an eighth image frame; the data processing amount of the electronic device when the drawing instruction satisfies the first condition is reduced, and the power consumption of the electronic device is reduced.

The flow shown in FIG. 51A is explained below with reference to a possible example.

Please refer to FIG. 51F . FIG. 51F includes a seventh storage space 5001 , an eighth storage space 5002 , a ninth storage space 5003 and a counting unit 5104 . The electronic device draws one image frame in each drawing cycle; as shown in the figure, the initial value of the counting unit is the first value, and the first value in the figure is specifically represented by 0; when the target application is started, the interception module intercepts for the first time When the target application program calls the first instruction; the numerical value of the counting unit 5104 is updated to the second numerical value, and the second numerical value is specifically represented by 1 in the accompanying drawing; Enter the first drawing cycle, in the first drawing cycle, in the drawing instruction When the first condition is met, the first drawing result is stored in the seventh storage space 5001; when the drawing instruction meets the second condition, since the value of the counting unit is the second value, the second drawing result is stored in the second storage space 5002; and then synthesize the seventh image frame in the ninth storage space 5003 according to the first rendering result and the second rendering result. When the interception module intercepts the invocation of the first instruction by the target application program again; the value of the counting unit 5104 is updated to the first value; the second drawing cycle is entered, and in the second drawing cycle, when the drawing instruction satisfies the first condition, The third drawing result is stored in the seventh storage space 5001; when the drawing instruction satisfies the second condition, since the value of the counting unit is the first value, the drawing operation is not performed; after that, according to the third drawing result and the second drawing result The eighth image frame is synthesized in the ninth storage space 5003 . Similarly, the ninth image frame and the tenth image frame can be obtained. It can be seen that, in this example, the value of the counting unit can be used to determine whether to perform a drawing operation in the current drawing cycle when the drawing instruction satisfies the second condition. Specifically, the first value may be 0, and the second value may be 1. It should be noted that the value of the counting unit can be adjusted according to specific conditions. For example, the technology unit can be switched in the order of 0, 1...N, where N is a positive integer greater than or equal to 2; the initial value of the counting unit is 0; in one drawing cycle, in one drawing cycle, if the update If the value of the counting unit after the update is 1, the eighth storage space is emptied at the moment when the drawing cycle begins; and when the drawing instruction satisfies the second condition, the drawing result is stored in the eighth storage space; if the value of the updated counting unit is If it is a value other than 1, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not performed.

Please refer to FIG. 52, which is a schematic diagram of a modular interaction provided by an embodiment of the present application; FIG. 52 includes a target application 1301, an interception module 5201, a storage space configuration module 5202, a cache 1352 and a display device 1350; specifically , you can refer to this example in conjunction with Figure 13.

The interception module 5201 may be located in the three-dimensional graphics processing library 1330 shown in FIG. 13 . The interception module 5201 is used to intercept the drawing instruction of the target application 1301, and in response to the interception of the drawing instruction, call the API in the three-dimensional graphics processing library 1330 to complete the processing of the data of the target application; it should be noted that this application The embodiment does not limit the position of the interception module in the software framework. The interception module may also be located between the application framework layer and the system library, or the interception module may be located in the application framework layer.

The storage space configuration module 5202 can be located in the 3D graphics processing library 1330 shown in FIG. 13 , and the cache configuration module can configure the storage space by calling the function/instruction/API in the 3D graphics processing library 1330 .

Display device 1350 is used to display the composite image frame.

The following explains the functions of some functional functions/instructions/APIs: Binding instructions: The binding instructions are used to bind the target application to the storage space, and store the drawing results in the bound storage space; drawing instructions: It is used to perform a drawing operation according to the target application 1301; the first instruction is used to rotate the storage space where the image frame is stored to a state that can be read by the display device 1350. The first instruction may serve as a time node where two adjacent cycles end and start. Specifically, when the three-dimensional graphics processing library 1330 is OpenGL, the binding instruction may be the glBindFrameBuffer API; the drawing instruction may be: glDrawElements API; and the first instruction may be eglSwapBuffers API.

The image frame generation method in this embodiment of the present application may include the steps shown in FIG. 52 :

Step (1) The electronic device creates a seventh storage space 5001 and an eighth storage space 5002 when the target application 1301 is started;

The seventh storage space 5001 is used to store the drawing result obtained when the drawing instruction satisfies the first condition;

The eighth storage space 5002 is used to store the drawing result obtained when the drawing instruction satisfies the second condition.

Step (2) The electronic device intercepts the binding instruction of the target application 1301, and firstly binds the target application 1301 with the seventh storage space 5001.

Step (3) The electronic device intercepts the drawing instruction, and according to the drawing instruction, when the drawing instruction satisfies the first condition, the drawing result obtained in the drawing process is stored in the seventh storage space 5001, and when the drawing instruction satisfies the second condition, the The drawing result obtained during the drawing process is stored in the eighth storage space 5002 . The electronic device may store the drawing result in the seventh storage space 5001 and the drawing result in the eighth storage space 5002 into the ninth storage space 5003 ; and synthesize the seventh image frame in the ninth storage space 5003 .

Specifically, when the drawing instruction includes an instruction for enabling depth testing, it is determined that the drawing instruction satisfies the first condition; the electronic device calls the binding instruction to bind the target application with the seventh storage space 5001, so that according to the data of the target application The generated drawing result is stored in the seventh storage space 5001; when the drawing instruction includes an instruction to close the depth test, it is determined that the drawing instruction satisfies the second condition; the electronic device calls the binding instruction to carry out the target application with the eighth storage space 5002. Binding, so that the drawing result generated according to the data of the target application is stored in the eighth storage space 5002 .

Step (4) The electronic device intercepts the first instruction, and rotates the ninth storage space 5003 to a state where the image frame can be transmitted to the display device 1350; when the vertical synchronization signal arrives, the seventh image frame is transmitted to the display device 1350 for display.

It should be noted that, the time point when the electronic device intercepts the first instruction may represent the time node when two adjacent drawing cycles end and start.

Step (5) The electronic device intercepts the drawing instruction, and according to the drawing instruction, when the drawing instruction satisfies the first condition, the drawing result obtained in the drawing process is overwritten and stored in the seventh storage space 5001, and when the drawing instruction satisfies the second condition, Drawing is not performed. The electronic device may store the drawing result in the seventh storage space 5001 and the drawing result in the eighth storage space 5002 into the ninth storage space 5003 ; and synthesize the eighth image frame in the ninth storage space 5003 .

Step (6) The electronic device intercepts the first instruction, and rotates the ninth storage space 5003 to a state where the image frame can be transmitted to the display device 1350; when the vertical synchronization signal arrives, the eighth image frame is transmitted to the display device 1350 for display.

It should be noted that the ninth storage space may be composed of multiple storage spaces, that is, the storage space for storing the seventh image frame and the storage space for storing the eighth image frame may be different; the electronic device stores the ninth storage space in step (4). The storage space with the seventh image frame is rotated to the state where the image frame can be transmitted to the display device 1350; in step (6), the storage space stored with the eighth image frame in the ninth storage space is rotated to be able to transmit the image frame to the display. The state of the device 1350; the storage space for storing the seventh image frame and the storage space for storing the eighth image frame is not the same storage space.

At present, when an electronic device runs a target application, if it wants to increase the frame rate of the display screen corresponding to the target application, there are usually two common methods; one method is to generate more data per unit time according to the data of the target application The actual drawn image frames, and then generate a display screen based on more image frames per unit time to achieve the purpose of improving the frame rate; another method is that the electronic device can generate a predicted image frame based on at least two actually drawn image frames. Image frame: The electronic device generates a display screen according to the actual drawn image frame and the predicted image frame on the display screen. Since the predicted image frame is added, the purpose of improving the frame rate is achieved.

The following embodiment introduces another method for generating an image frame provided by the embodiment of the present application, so as to reduce the power consumption of an electronic device when a prediction frame is inserted. As shown in Figure 53A, the method may include:

S5301, in the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the electronic device stores the drawing result in the seventh storage space as the first drawing result, when the drawing instruction satisfies the second condition When conditions are met, the drawing result is stored in the eighth storage space as the second drawing result; the seventh image frame is generated according to the first drawing result and the second drawing result.

S5302, in the third drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the electronic device stores the drawing result in the seventh storage space, as the fourth drawing result, when the drawing instruction satisfies the first condition In the second condition, the drawing result is stored in the eighth storage space as the fifth drawing result, and the third drawing period is located after the first drawing period; the ninth image frame is generated according to the fourth drawing result and the fifth drawing result.

S5303, during the fourth drawing cycle, the electronic device generates a sixth drawing result according to the first drawing result and the fourth drawing result; generates a tenth image frame according to the sixth drawing result and the seventh drawing result; In five drawing cycles, the drawing result obtained when the drawing instruction of the target application satisfies the second condition, the fifth drawing cycle is before the fourth drawing cycle.

Wherein, the specific process of generating the sixth drawing result according to the first drawing result and the fourth drawing result may include the following steps:

It should be noted that, in order to facilitate the description of the process of generating the sixth drawing result according to the first drawing result and the fourth drawing result, in this example, the possible display effects of the first drawing result, the fourth drawing result and the sixth drawing result are used. Figure 53B-Figure 53D are shown schematically; Figure 53B shows the effect diagram that the first drawing result may present; Figure 53C shows the effect diagram that the fourth drawing result may present; Figure 53D indicates that the sixth drawing result may present The resulting effect diagram.

s11: the electronic device performs a block matching algorithm on the first drawing result and the fourth drawing result, and determines the blocks that match each other in the first drawing result and the fourth drawing result;

Please refer to FIG. 53B ; wherein, the position coordinates of the first target block are (x ₁ , y ₁ ), and the first target block specifically displays the first area on the car. Please refer to FIG. 53C; wherein, the position coordinates of the second target block are (x ₂ , y ₂ ), and the second target block and the first target block display the first area on the car as well; since the first target block and the second target block The blocks represent the same area on the car, so the first target block and the second target block are blocks that match each other.

s12: The electronic device calculates the motion vector corresponding to each pair of matched blocks according to the blocks matched with each other, and obtains the motion vector corresponding to each block in the fourth drawing result;

Please refer to FIG. 53C, as shown in FIG. 53C, the motion vector of the second target block relative to the first target block is (x ₂ -x ₁ , y ₂ -y ₁ ), then the motion vector of the second target is (x ₂ -x ₁ , y ₂ -y ₁ );

s13: The electronic device generates the position coordinates corresponding to each block in the sixth drawing result according to the motion vector corresponding to each block in the fourth drawing result and the position coordinates corresponding to each block in the fourth drawing result, and obtains the sixth drawing result;

Referring to FIG. 53D, the third target block is also used to represent the first area on the car; the coordinates of the third target block are (x ₃ , y ₃ ), and the coordinates (x ₃ , y ₃ ) of the third target block are determined by the The coordinates (x ₂ , y ₂ ) of the two target blocks and the motion vector (x ₂ -x ₁ , y ₂ -y ₁ ) are added together, that is, x ₃ =x ₂ +x ₂ -x ₁ ; y ₃ =y ₂ +y ₂ -y ₁ .

It should be noted that FIGS. 53B to 53D are only illustrative of the target block representing the first area of the car. In the process of actually generating the sixth drawing result, the electronic device may traverse the first drawing result and the fourth drawing result. For each block in , the above-mentioned processing flow of obtaining the third target block according to the first target block and the second target block is performed for each block, and finally the coordinates of each block in the sixth drawing result are obtained, and the sixth drawing result is generated.

It can be seen that in this example, the electronic device multiplexes the drawing result when the drawing instruction meets the second condition every other image frame, which reduces the power consumption of the electronic device; at the same time, when the electronic device is generating the tenth image frame to be inserted At the time of writing, it is not necessary to perform calculation according to all the data of the first image frame and the ninth image frame, which reduces the data processing amount of the electronic device during the drawing process, and reduces the power consumption of the electronic device when generating the image frame.

The electronic device reduces the number of times of drawing the UI layer defined in the embodiments of the present application, and reduces the power consumption of the electronic device; at the same time, the electronic device can predict the dynamic layer, and synthesize image frames according to the predicted dynamic layer and the UI layer , to achieve the effect of frame insertion, and improve the frame rate and fluency of the electronic device when playing the display screen according to the image frame; Use the data corresponding to the UI layer to reduce the power consumption of the electronic device when running the game application; at the same time, the electronic device can predict and generate the next dynamic layer according to the two dynamic layers drawn according to the actual running data of the game, and then The image frames of the game application are synthesized according to the predicted dynamic layer, the frame insertion processing in the game scene is realized, and the refresh rate of the display screen when the electronic device runs the game application is improved.

The flow shown in FIG. 51A and FIG. 53A is explained below with reference to a possible example.

Please refer to FIG. 53E, which is a schematic diagram of generating ten image frames; the seventh image frame, the eighth image frame, etc. in FIG. 53E are represented by numbers, 1, 2, 3, 4.

Figure 53E shows a schematic diagram of the process of generating ten image frames;

Among them, the seventh image frame and the eighth image frame; the ninth image frame and the tenth image frame; can be regarded as adopting the flow method shown in FIG. 51A . In FIG. 53E, the first drawing result, the third drawing result, the fourth drawing result, the sixth drawing result, the seventh drawing result, the tenth drawing result, and the thirteenth drawing result are all obtained when the drawing instruction satisfies the first condition Plot the result. The second drawing result, the fifth drawing result, the eighth drawing result, the eleventh drawing result and the fourteenth drawing result are all drawing results obtained when the drawing instruction satisfies the second condition. The fourth drawing result and the seventh drawing result generate the ninth drawing result; the seventh drawing result and the tenth drawing result generate the twelfth drawing result; and the tenth drawing result and the thirteenth drawing result generate the fifteenth drawing result.

The flow shown in FIG. 53A will be explained below in conjunction with another possible example.

Please refer to FIG. 53F, which shows a schematic diagram of generating six image frames; in this example, the first drawing result, the third drawing result, the sixth drawing result and the eighth drawing result are obtained when the drawing instruction satisfies the first condition The drawing result; the second drawing result, the fourth drawing result, the seventh drawing result and the ninth drawing result are the drawing results obtained when the drawing instruction satisfies the second condition. Wherein, the first drawing result and the third drawing result generate the fifth drawing result; the sixth drawing result and the eighth drawing result generate the tenth drawing result.

In the example shown in Figure 53F above, the counting unit can be updated and switched between the first value, the second value, and the third value, and the sequence of updating and switching is from the first value to the second value to the third value. Switch in sequence; specifically, the initial value of the counting unit is the first value, when entering the first drawing cycle, the updated value is the second value, entering the next drawing cycle and updating to the third value, and then entering the next cycle and updating to the first value A value, ... When the value of the counting unit is the first value, the drawing instruction does not execute the drawing operation when the second condition is met; when the value of the counting unit is the second value or the third value, the drawing instruction meets the second condition. The drawing result is stored in the eighth storage space. Specifically, the first value may be 0; the second value may be 1; and the third value may be 2.

Please refer to FIG. 53G, which illustrates a possible schematic diagram during the drawing process. Figure 53G includes a seventh storage space 5001, an eighth storage space 5002, a ninth storage space 5003, a tenth storage space 5004, and an eleventh storage space 5005. In the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the first drawing result is stored in the seventh storage space 5001; according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition In the second condition, the second drawing result is stored in the eighth storage space 5002; the seventh image frame is generated in the ninth storage space 5003 according to the first drawing result and the second drawing result. In the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the third drawing result is stored in the tenth storage space 5004; according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition In the second condition, no drawing is performed; an eighth image frame is generated in the ninth storage space 5003 according to the third drawing result and the second drawing result. In the third drawing cycle, the first drawing result in the seventh storage space 5001 and the third drawing result in the tenth storage space are stored in the eleventh storage space 5005; The drawing result and the third drawing result generate a fourth drawing result; and a ninth image frame is generated according to the fourth drawing result and the second drawing result.

Please refer to FIG. 53H, which shows a possible schematic diagram of a drawing process. FIG. 53H includes a seventh storage space 5001 , an eighth storage space 5002 , a ninth storage space 5003 , a tenth storage space 5004 , and an eleventh storage space 5005 . In the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the first drawing result is stored in the seventh storage space 5001; according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition In the second condition, the second drawing result is stored in the eighth storage space 5002; the seventh image frame is generated in the ninth storage space 5003 according to the first drawing result and the second drawing result. In the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the first drawing result is stored in the tenth storage space 5004; according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition In the second condition, the fourth drawing result is stored in the eighth storage space 5002; the eighth image frame is generated in the ninth storage space 5003 according to the third drawing result and the fourth drawing result. In the third drawing cycle, the first drawing result in the seventh storage space 5001 and the third drawing result in the tenth storage space 5004 are stored in the eleventh storage space 5005; The first drawing result and the third drawing result generate the fifth drawing result; the ninth drawing result is generated in the ninth storage space 5003 according to the fourth drawing result in the eighth storage space 5002 and the fifth drawing result in the eleventh storage space 5005 image frame.

Please refer to FIG. 53I, which is another possible example of the process shown in FIG. 53A. Fig. 53I shows a schematic diagram of generating eight image frames; in Fig. 53I, the seventh image frame, the eighth image frame, ... etc. are represented by numbers, 1, 2, 3, 4, ... 8.

In FIG. 53I, the first drawing result, the third drawing result, the fourth drawing result, the seventh drawing result, and the tenth drawing result are all drawing results obtained when the drawing instruction satisfies the first condition. The second drawing result, the fifth drawing result, the eighth drawing result and the eleventh drawing result are all drawing results obtained when the drawing instruction satisfies the second condition. Wherein, the first drawing result and the fourth drawing result generate the sixth drawing result; the fourth drawing result and the seventh drawing result generate the ninth drawing result; and the seventh drawing result and the tenth drawing result generate the twelfth drawing result.

In this embodiment of the present application, each frame of image that the electronic device uses to display on the display screen is referred to as an image frame. In this embodiment of the present application, an image frame may be a frame of an image of an application, a drawing result drawn by an electronic device according to a drawing instruction of the application, or a prediction result predicted based on an existing drawing result. As shown in Figure 54A, the electronic device (ie, tablet 10) displays a user interface 5400. At time T0, the Nth image frame is displayed in the user interface 10. The Nth image frame is a drawing frame. The timing diagram 5401 in FIG. 54A shows the image frames that the electronic device can display from time T0 to time Tn.

It can be understood that the size of the image frame, the size of the drawing frame, and the size of the predicted frame are consistent with the display size of the application to which they belong. For example, as shown in FIG. 54A , the display size of the application to which the Nth image frame shown in FIG. 54A belongs on the tablet computer 10 is: the width is L and the height is H. Then the size of the Nth image frame may be: the width is L and the height is H. Also as shown in FIG. 54B , the display size of the application to which the image frame 5402 shown in FIG. 54B belongs on the tablet computer 10 is: the width is L0 and the height is H. Then the size of the image frame 5402 can be: the width is L0 and the height is H. Again, as shown in FIG. 54C, an image frame 5403 is shown, and a control bar 5404 is shown in FIG. 54C. Among them, the control bar 540404 may include the control 5405. The control bar 5404 may be drawn by the system of the tablet computer 10 . The display size of the application to which the image frame 5403 belongs on the tablet computer 10 is: the width is L1, and the height is H0. Then the size of the image frame 5403 can be: the width is L1 and the height is H0.

In order to increase the frame rate and improve the smoothness of the video, the electronic device can insert the predicted frame between the drawn frames of the application. The electronic device may perform image frame prediction according to the drawn frame of the application to obtain the predicted frame. As shown in FIG. 54A, the electronic device may insert a predicted image frame between every two drawing frames. In this way, the frame rate of the image frame displayed by the electronic device can be increased.

It can be understood that the drawing result drawn by the electronic device according to the drawing instruction of the application program may be an image frame drawn and rendered by the electronic device according to a frame of image captured by the camera. Objects in image frames can move due to camera movement, rotation, and other positional transformations. For example, FIG. 55A exemplarily shows that the camera 200 photographs the cylinder 5502 , the cylinder 5503 and the rectangular parallelepiped 5504 at different positions, and a frame of image photographed in the photographing field of view of the camera 5500 may be an image frame. FIG. 55B exemplarily shows the camera field of view 5501 of the camera 200 . As shown in FIG. 55A , at position 1, a frame of image captured in the camera field of view 5505 of the camera 5500 may be the image frame 5508 in FIG. 55A , that is, the drawing frame A. At position 2, a frame of image captured in the camera field of view 5506 of the camera 5500 may be the image frame 5509 in FIG. 55A, ie, the drawing frame B. The electronic device can predict the image frame 5510 of the camera 5500 at the position 3 according to the drawing frame A and the drawing frame B, that is, the predicted frame 5510 . The predicted frame is predicted according to the drawing frame B and the drawing frame A, and the objects contained in the predicted frame are the same as the objects contained in the drawing frame B. The drawing frame A includes the cylinder 5502 and the cuboid 204, and the drawing frame B only includes the cuboid 5504. Therefore, only the cuboid 5504 is included in the predicted frame. If the drawing is normal, at position 3, the drawing content drawn by the electronic device is the object that can be photographed by the photographing field of view 5507 of the camera 5500 . At position 3, the photographic field of view 5507 of the camera 200 includes a partial area of the rectangular parallelepiped 5504 and a partial area of the cylinder 5503. The cylinder 5503 that does not exist in the drawing frame B cannot be predicted in the predicted frame. In this way, the predicted prediction frame is inaccurate.

In order to make the predicted image frame more accurate, an embodiment of the present application provides a method for image frame prediction, the method may include: when drawing the twenty-first drawing frame of the first application, the electronic device The drawing instruction of a drawing frame is drawn according to the first drawing range, and the twenty-first drawing result is obtained, and the size of the first drawing range is larger than the size of the first drawing frame of the first application; frame, the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range, and obtains the twenty-second drawing result, and the size of the twenty-second memory space is larger than the size of the second drawing frame, wherein, The size of the twenty-first drawing frame is the same as the size of the twenty-second drawing frame; the electronic device predicts and generates the twenty-third predicted frame of the first application according to the twenty-first drawing result and the twenty-second drawing result , wherein the size of the third prediction frame is the same as the size of the twenty-first rendering frame. In this way, when the electronic device draws the twenty-first drawing frame and the twenty-second drawing frame according to the larger drawing range, it can draw more than the twenty-first drawing frame and the twenty-second drawing frame displayed by the electronic device. some drawing content. In this way, the electronic device can obtain the predicted frame. The frame rate of the electronic device can be increased without increasing the drawing frame. In this way, while saving the power consumption of the electronic device, the smoothness of the video interface displayed by the electronic device can be improved. Further, in the predicted frame predicted by the electronic device, there may be rendering content that is not in the first rendering frame and the second rendering frame displayed by the electronic device. Therefore, the drawing content in the predicted frame predicted by the electronic device is closer to the shooting content in the shooting field of view of the camera. Therefore, the image frame predicted by the electronic device can be more accurate.

It can be understood that the drawing instruction of the U-th drawing frame may include drawing content. When the drawing range is relatively small, the U-th drawing frame drawn and rendered by the GPU may only contain a part of the drawing content included in the drawing instruction of the U-th drawing frame. When the drawing range is relatively large, the GPU can draw and render more drawing contents included in the drawing instruction of the U-th drawing frame. For example, if the drawing range is from 50 to 100 on the abscissa and from 50 to 100 on the ordinate, one of the drawing contents in the drawing instruction of the U-th drawing frame is from 10 to 60 on the abscissa and from 10 to 60 on the ordinate. of an object. Then the drawing content only has the horizontal coordinate from 50 to 60, and the vertical coordinate from 50 to 60 is drawn and rendered into the drawing range. If the drawing range is expanded to an area with the horizontal coordinate from 0 to 150 and the vertical coordinate from 0 to 150, then all the drawing content can be drawn and rendered into the drawing range.

In this embodiment of the present application, the U-th rendering frame may be referred to as the twenty-first rendering frame, the U+2-th rendering frame may be referred to as the twenty-second rendering frame, and the U+3-th predicted frame may be referred to as the twenty-third predicted frame.

A method for predicting an image frame provided by an embodiment of the present application will be described in detail below with reference to the accompanying drawings. FIG. 56 shows a flowchart of an image frame prediction method provided by an embodiment of the present application. As shown in FIG. 56, an image frame prediction method provided by an embodiment of the present application may include the following steps:

S5601-S5602, the electronic device starts to execute the image frame prediction method.

S5601. When the target application starts to draw, the CPU of the electronic device sends an instruction for instructing the GPU to create a memory space to the GPU.

The target application is an application with animation effects in the user interface, such as a game application. The embodiments of the present application are described below by taking the target application as a game application as an example. When the game application installed in the electronic device runs, the CPU of the electronic device sends an instruction for instructing the GPU to create a memory space to the GPU. Specifically, the instruction can carry information on the number and size of the created memory space.

S5602, the GPU of the electronic device creates a twenty-first memory space (FBO1), a twenty-second memory space (FBO2) in the memory with a size larger than the default memory space based on the size of the default memory space (FBO0) in the electronic device, and The twenty-third memory space (FBO3).

In response to the instruction sent by the CPU, the GPU creates a twenty-first memory space (FBO1), a twenty-second memory space (FBO2), and a second memory space with a size larger than the default memory space in the memory based on the size of the default memory space of the electronic device. Thirteen memory spaces (FBO3). In this embodiment of the present application, the default memory space is a memory space provided by a system of an electronic device (for example, a rendering system) for storing image frames for display. The default memory space can store image frames for display of the target application, or image frames for display in other applications. As shown in FIG. 57 , the default memory space may include multiple attachments with consecutive logical addresses. For example, the default memory space shown in FIG. 57 may include n memories such as attachment 5701, attachment 5702, . . . , attachment 570n. Typically, n is less than or equal to three. Each of the n attachments in the default memory space can have a width of L and a height of H. In this embodiment of the present application, the width of the default memory space may be referred to as the third size, and the height of the default memory space may be referred to as the fourth size. It can be understood that the width of the default memory space refers to the width of each attachment in the default memory space, and the height of the default memory space may refer to the height of each attachment in the default memory space.

Optionally, the n attachments in the default memory space may include a color attachment (color attachment) and a depth attachment (depth attachment). For example, attachment 5701 may be a color attachment. In this embodiment of the present application, the color accessory is a piece of memory used to store color data (eg, RGB values of pixels) of each pixel in the drawing result when the electronic device draws according to the drawing instruction. Color attachments can be used as part of an FBO (frame buffer object). In the embodiment of the present application, the depth accessory is a piece of memory used to store the depth data of each pixel in the drawing result when the electronic device draws according to the drawing instruction. Color attachments are available as part of the FBO. Understandably, the smaller the depth value of the pixel in the depth attachment, the closer it is to the camera. When synthesizing an image frame, for two pixels with equal coordinate values in two color attachments, a pixel with a smaller depth value can cover another pixel with a larger depth value. That is, the color displayed by the final display pixel point is the color of the pixel point with the smaller depth value in the two color attachments.

The size of the twenty-first memory space is larger than the size of the default memory space. In a possible implementation manner, the width of the twenty-first memory space is K1 times the width of the default memory space, and the height of the twenty-first memory space is K2 times the height of the default memory space. As shown in the schematic diagram of the twenty-first memory space shown in FIG. 58 , the logical addresses of the twenty-first memory space may be arranged consecutively. The twenty-first memory space may include a plurality of accessories with consecutive logical addresses. For example, the attachment 5801, the attachment 5802, ..., the attachment 580n, etc. shown in FIG. 58 have n memories. In general, n may be less than or equal to three. The width of each of the n attachments in the twenty-first memory space may be L·K1, and the height may be H·K2. Both K1 and K2 are greater than 1. K1 may be equal to K2. K1 and K2 may be configured by a system of electronic devices. Optionally, the width of each of the n accessories in the twenty-first memory space may be L+Q1, and the height may be H+Q2. Both Q1 and Q2 are greater than 0. Q1 and Q2 may be equal. Q1 and Q2 may be configured by the system of the electronic device. Optionally, both K1 and K2 of the system configuration of the electronic device are fixed values. That is, the values of K1 and K2 remain unchanged each time the electronic device executes the image frame prediction method provided by the embodiment of the present application.

In this embodiment of the present application, the width of the twenty-first memory space may be referred to as the first size, and the height of the twenty-first memory space may be referred to as the second size.

In one possible implementation, K1 and K2 may be floating point numbers, and Q1 and Q2 may be integers.

The size of the twenty-second memory space is larger than the size of the default memory space. In a possible implementation manner, the width of the twenty-second memory space is K1 times the width of the default memory space, and the height of the twenty-second memory space is K2 times the height of the default memory space. As shown in the schematic diagram of the twenty-second memory space in FIG. 59, the logical addresses of the twenty-second memory space may be arranged consecutively. The twenty-second memory space may include multiple attachments with consecutive logical addresses. For example, n attachments such as attachment 5901, attachment 5902, . . . , attachment 590n, etc. shown in FIG. 59 . In general, n may be less than or equal to three. The width of each memory in the n attachments of the twenty-second memory space may be L·K1, and the height may be H·K2. Both K1 and K2 are greater than 1. K1 may be equal to K2. K1 and K2 may be configured by a system of electronic devices. Optionally, the width of each of the n accessories in the twenty-second memory space may be L+Q1, and the height may be H+Q2. Both Q1 and Q2 are greater than 0. Q1 and Q2 may be equal. Q1 and Q2 may be configured by the system of the electronic device.

In this embodiment of the present application, the width of the twenty-second memory space may be referred to as the fifth size, and the height of the twenty-second memory space may be referred to as the sixth size.

The size of the twenty-third memory space is larger than the size of the default memory space. In a possible implementation manner, the width of the twenty-third memory space is K1 times the width of the default memory space, and the height of the twenty-third memory space is K2 times the height of the default memory space. As shown in the schematic diagram of the twenty-third memory space shown in FIG. 60, the logical addresses of the twenty-third memory space may be arranged consecutively. The twenty-third memory space may include a plurality of accessories with consecutive logical addresses. For example, there are n accessories such as accessories 6001, accessories 6002, . . . , accessories 600n, etc. shown in FIG. 60 . The width of each of the n attachments in the twenty-third memory space may be L·K1, and the height may be H·K2. Both K1 and K2 are greater than 1. K1 may be equal to K2. K1 and K2 can be configured by the electronic equipment system. Optionally, the width of each of the n accessories in the twenty-third memory space may be L+Q1, and the height may be H+Q2. Both Q1 and Q2 are greater than 0. Q1 and Q2 may be equal. Q1 and Q2 may be configured by the system of the electronic device. Optionally, both Q1 and Q2 of the system configuration of the electronic device are fixed values. That is, the values of Q1 and Q2 remain unchanged each time the electronic device executes the image frame prediction method provided by the embodiment of the present application.

In this embodiment of the present application, the width of the twenty-third memory space may be referred to as the seventh size, and the height of the twenty-third memory space may be referred to as the eighth size.

In a possible implementation manner, the size of the twenty-first memory space is the same as the size of the twenty-second memory space and the size of the twenty-third memory space. That is, if the width of the twenty-first memory space is L·K1, and the height is H·K2. Then the width of the twenty-second memory space is L·K1, and the height is H·K2. The width of the twenty-third memory space is L·K1, and the height is H·K2.

S5603-S5606, the electronic device draws the U-th drawing frame.

S5603, the electronic device acquires the drawing parameters of the U-th drawing frame.

When the target application in the electronic device draws, the target application can call a drawing instruction to draw. The CPU of the electronic device 100 may acquire the drawing parameters of the U-th drawing frame of the application program through the interface in the three-dimensional image processing library. The drawing parameters of the U-th drawing frame are used to draw and render the U-th drawing frame. The drawing parameters of the U-th drawing frame may include information carried in the drawing instructions (such as the draw call instruction) of the U-th drawing frame, such as the coordinates, color values, depth values and other information of each vertex in the drawing content of the draw call instruction .

S5604. The CPU in the electronic device sends a drawing instruction to the GPU for instructing the GPU to draw the U-th drawing frame.

The CPU of the electronic device may send a drawing instruction for instructing the GPU to draw the U-th drawing frame to the GPU according to the drawing parameters of the U-th drawing frame. It can be understood that, the drawing parameters of the U-th drawing frame acquired by the CPU may include information of multiple drawing instructions. In this way, the CPU may sequentially send to the GPU multiple drawing instructions for instructing the GPU to draw the U-th drawing frame. In this embodiment of the present application, the drawing instruction includes an execution drawing (draw call) instruction and a drawing state setting instruction.

The drawing state setting instruction may further include instructions such as setting the vertex index, texture information, and spatial position of the drawing object, for example, glActiveTexture and glBindBufferRange instructions in OpenGL. A drawing object may be an object that can be drawn by an electronic device according to all vertices and vertex information contained in a drawing instruction.

For a more vivid description, taking the electronic device to draw the cuboid 5504 in FIG. 55A as an example, a possible OpenGL drawing instruction according to the execution sequence may be as follows:

glBindBufferRange(target=GL_UNIFORM_BUFFER,index=1,buffer=738,offset=0,size=352)//Instruct the GPU to modify the part to draw global information, such as the position of the cuboid 5504 in Figure 55A;

glBindBuffer(target=GL_ARRAY_BUFFER, buffer=buffer0)//Instructs the GPU to store the buffer0 index information of the vertex information (such as vertex position, color, etc.) of the cuboid 5504 into GL_ARRAY_BUFFER;

glBindBuffer(target=GL_ELEMENT_ARRAY_BUFFER, buffer=buffer1)//Instructs the GPU to save the index of buffer1 that stores the vertex index information of the cuboid 5504 (for example, the drawing order of the vertices) into GL_ELEMENT_ARRAY_BUFFER;

glActiveTexture(texture=GL_TEXTURE0)

glBindTexture(target=GL_TEXTURE_2D, texture=texture1)//Instructs the GPU to save the index of texture1 that stores the texture information of the cuboid 5504 into GL_TEXTURE0;

…

glDrawElements(GLenummode,GLsizeicount,GLenumtype,const void*indices)//Instructs the GPU to perform cuboid 5504 drawing.

S5605. In the twenty-first memory space, the GPU of the electronic device draws the drawing content in the drawing instruction of the U-th drawing frame into the first drawing range, and obtains the twenty-first drawing result, where the size of the first drawing range is less than or equal to the size of the twenty-first memory space, which is greater than the size of the default memory space.

The GPU of the electronic device may draw the drawing content of the drawing instruction of the U-th drawing frame into the twenty-first memory space to obtain the twenty-first drawing result. Optionally, the GPU of the electronic device may draw the drawing content of the drawing instruction of the U-th drawing frame into the first drawing range of the twenty-first memory space to obtain the twenty-first drawing result. The size of the first drawing range is smaller than or equal to the size of the twenty-first memory space, and is larger than the size of the default memory space. It can be understood that, in this embodiment of the present application, the drawing range in any attachment in the twenty-first memory space may be referred to as the first drawing range.

In a possible implementation manner, the size of the first drawing range is determined by the electronic device according to the viewport parameter in the target application. The viewport parameter of the target application is used to specify the width and height of the drawing range in which the image frame of the target application is drawn in the electronic device. Generally, the width of the drawing range specified by the viewport parameter of the target application is the same as the width of the default memory space in the electronic device, and the height of the drawing range specified by the viewport parameter is the same as the height of the default memory space in the electronic device. The electronic device can modify the viewport parameter of the target application through the hook technology to specify the size of the first drawing range. For example, if the width of the drawing range of the U-th drawing frame specified by the viewport of the target application is L and the height is H. Then, the electronic device can modify the viewport parameter of the target application through the hook technology, and designate the width of the first drawing range as L·K3 and the height as H·K4. Both K3 and K4 are greater than 1. Optionally, K3 and K4 are floating point numbers greater than 1. K3 is less than or equal to K1, and K4 is less than or equal to K2. Here, the viewport parameter in the U-th drawing frame may be the first parameter in this embodiment of the present application.

It can be understood that the viewport parameter in the drawing instruction issued by the application is used to specify the size of the canvas on which the drawing content in the drawing instruction of the application is drawn. Generally, the electronic device draws the drawing content in the drawing instruction of each frame of the application program according to the size of the canvas, and then renders the drawn canvas to the display screen. The canvas size specified by the viewport parameter in the application can be the same as the size of the display screen of the electronic device, or it can be smaller than the size of the display screen of the electronic device, which is not limited here. As shown in (a) of FIG. 61 , (a) of FIG. 61 shows an attachment 6102 and a first drawing range 6101 in the twenty-first memory space. The memory 802 has a width of L·K1 and a height of H·K2. The first drawing range 6101 has a width of L·K3 and a height of H·K4. The following description will be given by taking the electronic device drawing the drawing content of the drawing instruction in the U-th drawing frame to the first drawing range 6101 as an example.

In one possible implementation, K3 is equal to K1, which is a fixed value for the system configuration of the electronic device. K4 is equal to K2, a fixed value for the system configuration of the electronic device. It can be understood that the size of the first drawing range is equal to the size of the twenty-first memory space.

In a possible implementation manner, the values of K3 and K4 may be determined by including the camera's corner parameter in the drawing parameters of the previous drawing frame in the U-th drawing frame. If the image frame displayed by the electronic device is as shown in FIG. 54A, the electronic device can insert a prediction frame into every two rendering frames, then the rendering frame preceding the U-th rendering frame is the U-2-th rendering frame.

Further, the specific calculation process for determining K3 by the electronic device may be as follows:

1. Calculate the coordinate transformation matrix T between the N-2th drawing frame and the Uth drawing frame.

T=(P1V1) ^-1 *(P2V2) (Formula 1)

where (P1V1) ^-1 is the inverse of (P1V1). V1 is a view matrix (view matrix) included in the drawing parameters of the U-2th drawing frame, and P1 is a projection matrix (projection matrix) included in the drawing parameters of the U-2th drawing frame. V2 is the observation matrix included in the drawing parameters of the U-th drawing frame, and P2 is the projection matrix included in the drawing parameters of the U-th drawing frame.

In this embodiment of the present application, the observation matrix is a transformation matrix between the world space (world space) and the observation space (camera space). For example, the coordinates of the vertex 1 in the drawing instruction of the U-th drawing frame can be converted from the coordinates in the world space to the coordinates in the observation space through the observation matrix. The projection matrix is the transformation matrix between the viewing space and the clip space. For example, the coordinates of vertex 1 can be transformed from coordinates in view space to coordinates in clip space through a projection matrix. World space is the corresponding space in world coordinates. The observation space is the space corresponding to the camera coordinate system (the coordinate system constructed with the camera as the coordinate origin). The position of the object described in the observation space is the position in the camera coordinates. The clip space defines the coordinate range of the object that can be displayed on the display screen of the electronic device.

2. Calculate the maximum value Z(Ai)'max of the X-axis component of the position offset of all the pixels in the rightmost column of the Nth frame between the Nth frame and the N-2th frame.

If the rightmost column in the Nth frame includes r pixel points such as pixel points A1, A2, A3, ..., Ai, ..., Ar, etc. The position offset Z(Ai) of the pixel point Ai between the U-th frame and the U-2-th frame is:

Z(Ai)=Ai(xai,yai)-Aprev (Formula 2)

where Ai(xai, yai) is the coordinate of the pixel point Ai in the pixel space of the Nth frame; Aprev is the coordinate of the pixel point Ai in the pixel space of the U-2th drawing frame. The electronic device establishes a coordinate system with the lower left corner of the U-th drawing frame as the origin, the X axis to the right, and the upper Y axis. The space corresponding to the coordinate system may be referred to as the pixel space in the Nth frame. Similarly, the electronic device establishes a coordinate system with the lower left corner of the U-2th drawing frame as the origin, the X axis to the right, and the upper Y axis. The space corresponding to this coordinate system may be referred to as the pixel space in the U-2th frame.

The electronic device can obtain the coordinates Ai(xai, yai) of the pixel space of the pixel point Ai in the Nth frame, and the width of the drawing range obtained by the electronic device in the Uth frame is L0 and the height is H0. The depth value of the pixel point Ai is Dai, then the electronic device can calculate the coordinate Ai_clip of the pixel point Ai in the Nth frame projection space as:

Ai_clip＝(xai/L0,yai/H0,2*Dai-1,1) (Formula 3)

The coordinate Ai_clip_prev of the pixel point Ai in the N-2th frame clipping space is:

Ai_clip_prev=T*Ai_clip (Formula 4)

The coordinate Aprev of the pixel point Ai in the pixel space of the N-2th frame is:

Aiprev=(Ai_clip_prev.x/Ai_clip_prev.w*L0,Ai_clip_prev.y/Ai_clip_prev.w*H0) (Formula 5)

Ai_clip_prev is a 1-by-4 vector, where Ai_clip_prev.x is the first element of Ai_clip_prev, A_clip_prev.y is the second element of Ai_clip_prev, and Ai_clip_prev.w is the fourth element of Ai_clip_prev.

If Z(Ai).x>=0, then modify the value of Z(Ai) to (0,0), if Z(Ai).x<0, then take the absolute value of each value in Z(Ai), to get Z(Ai)'.

The electronic device calculates the position offset of r pixel points such as pixel points A1, A2, A3, . . . , Ai, . The electronic device determines the maximum value Z( Ai)'max.

3. Calculate the maximum value Z(Bi)'max of the X-axis component of the position offset of all the pixels in the leftmost column of the Nth frame between the Uth frame and the U-2th frame.

If the rightmost column in the Nth frame includes pixel points B1, B2, B3, ..., Bi, ..., Br, etc. r pixels. The position offset Z(Bi) of the pixel point Bi between the U-th frame and the U-2-th frame is:

Z(Bi)=Bi(xbi,ybi)-Biprev (Equation 6)

Where Bi(xbi, ybi) is the coordinate of the pixel point Bi in the Nth frame in the pixel space; Biprev is the coordinate of the pixel point Bi in the pixel space of the U-2th drawing frame.

The electronic device can obtain the coordinates Bi(xbi, ybi) in the pixel space of the pixel point Bi in the U-th frame, and the width of the drawing range obtained by the electronic device in the U-th frame is L0 and the height is H0. The depth value of the pixel point Bi is Dbi, then the electronic device can calculate the coordinate Bi_clip of the pixel point Bi in the U-th frame clipping space as:

Bi_clip=(xai/L0,yai/H0,2*Dbi-1,1) (Formula 7)

The coordinate Bi_clip_prev of the pixel Bi in the U-2th frame clipping space is:

Bi_clip_prev=T*Bi_clip (Equation 8)

The coordinate Biprev of the pixel point Bi in the pixel space of the U-2th frame is:

Biprev=(Bi_clip_prev.x/Bi_clip_prev.w*L0,Bi_clip_prev.y/Bi_clip_prev.w*H0) (Equation 9)

Among them, Bi_clip_prev is a 1-by-4 vector, where Bi_clip_prev.x is the first element of Bi_clip_prev, B_clip_prev.y is the second element of Bi_clip_prev, and Bi_clip_prev.w is the fourth element of Bi_clip_prev.

If Z(Bi).x>=0, then modify the value of Z(Bi) to (0,0), if Z(Bi).x<0, then take the absolute value of Z(Bi) to get the final Z(Bi)'.

The electronic device calculates the position offset of r pixel points such as pixel points B1, B2, B3, . . . , Bi, . The electronic device determines the maximum value Z( Bi)'max.

4. Calculate K3 according to the position offset Z(Ai)'max and Z(Bi)'max.

K3=((Z(Ai)’max+Z(Bi)’max)/2+L)/L (Equation 10)

Optionally, if the width of the first drawing range is L+Q3, then Q3 can be:

Q3=(Z(Ai)’max+Z(Bi)’max)/2 (Equation 11)

Further, the specific calculation process for determining K4 by the electronic device may be as follows:

5. Calculate the coordinate transformation matrix T between the U-2th drawing frame and the Uth drawing frame.

Here, you can refer to the above step 1, which will not be repeated here.

6. Calculate the maximum value Z(Ci)'max of the X-axis component of the position offset of all the pixels in the uppermost column in the U-th frame between the U-th frame and the U-2-th frame.

Here, the calculation process of Z(Ci)'max can refer to the process of calculating Z(Ai)'max in the above formula 2-formula 5, which will not be repeated here.

7. Calculate the maximum value Z(Di)'max of the position offset X-axis component of all the pixels in the lowermost column of the U-th frame between the U-th frame and the U-2-th frame.

Here, the calculation process of Z(Di)'max can refer to the process of calculating Z(Ai)'max in the above formula 2-formula 5, which will not be repeated here.

8. Calculate K4 according to the position offset Z(Ci)'max and Z(Di)'max.

K4=((Z(Ci)’max+Z(Di)’max)/2+H)/H (Equation 12)

Optionally, if the height of the first drawing range is H+Q4, then Q4 can be:

Q4=(Z(Ci)’max+Z(Di)’max)/2 (Equation 13)

The embodiments of the present application are described below by taking an example that the size of the first drawing range is smaller than the size of the twenty-first memory space. For example, as shown in (a) of FIG. 61 , the width of one attachment 6102 in the twenty-first memory space is L·K1 and the height is H·K2. The width of the first drawing range is L·K3 and the height is H·K4. Accessory 6102 may be accessory 5701 or accessory 5702 shown in FIG. 58 , or accessory 570n.

The electronic device may draw the drawing content of the drawing instruction of the U-th drawing frame within the first drawing range of the twenty-first memory space. The size of the first drawing range is larger than the size of the drawing range specified by the viewport parameter of the target application. Since the first drawing range is enlarged, the electronic device can draw more drawing content into the first drawing range. However, if the drawing content of the U-th drawing frame also follows the enlargement ratio of the drawing range, the drawing content in the first drawing range will not increase. The electronic device needs to perform a bitwise transformation with the camera coordinate origin as the center point of the drawing content in the U-th drawing frame, so that the drawing content will shrink to the middle and become smaller in the camera coordinate. Therefore, the electronic device can draw more drawing contents within the first drawing range.

The electronic device may modify the projection matrix in the drawing parameters of the U-th drawing frame, so that the drawing content of the U-th drawing frame is drawn more within the first drawing range. The electronic device can generate a matrix T1 according to the determined values of K3 and K4, and the projection matrix P can be modified to P*T1. The matrix T1 can be:

In this embodiment of the present application, the matrix T1 may be referred to as a first transformation matrix.

In one possible implementation, the electronic device can modify the projection matrix through the hookglBufferSubData function. The glBufferSubData function is: glBufferSubData(GLenum target, GLintptr offset, GLsizeiptr size, const void*data).

The glBufferSubData function is used to write the data of size size pointed to in data into the buffer memory corresponding to the target at the offset position. After the electronic device hooks this function, it is determined that the data contains the information of the projection matrix P according to the value of the target as GL_UNIFORM_BUFFER and the size of the size as 2848. The electronic device uses the memory information to take out the P in the data, and write P*T1 into the specified location of the data. The electronic device draws the drawing content of the U-th drawing frame into the first drawing range according to the modified projection matrix (ie, P*T1), and can obtain the twenty-first drawing result. In this way, the twenty-first drawing result may include more drawing contents. For example, there is a square with a size of 100*100 in the drawing content of the drawing instruction in the U-th drawing frame. With the modified projection matrix (ie P*T1), the size of the square will become (100*1/K3)*(100*1/K4). Since the electronic device expands the width of the first drawing range by K3 times and the height of the first drawing range by K4 times, the size of the square in the first drawing range is (100*1/K3*K3)*( 100*1/K4*K4)=(100*100). In this way, the drawing range becomes larger, but the size of the drawn content does not increase. Therefore, more drawing content can be drawn in the enlarged drawing range.

The twenty-first rendering result of the U-th rendering frame may be shown as the twenty-first rendering result 6103 in (b) of FIG. 61 . The width in the twenty-first drawing result 6103 is L·K3 and the height is H·K4. That is, the size of the twenty-first drawing result may be the same as the size of the first drawing range. Optionally, the size of the twenty-first drawing result may be the same as the size of the twenty-first memory space. The following description takes an example that the size of the twenty-first drawing result is the same as the size of the first drawing range.

S5606. In the default memory space, the GPU cuts the size of the twenty-first drawing result to be the same as the size of the default memory space to obtain the U-th drawing frame.

Generally, the size of the display screen of the electronic device is the same as the size of the default memory space. Therefore, before the image frame of the target application is displayed, the electronic device needs to cut the size of the image frame to be the same as the size of the default memory space. In the default memory space, the GPU can cut the size of the twenty-first drawing result to be the same as the size of the default memory space to obtain the U-th drawing frame. The U-th rendering frame is an image frame for display by the electronic device. The U-th rendering frame may be shown as the U-th rendering frame 6104 illustrated in (c) of FIG. 61 . The U-th drawing frame 6104 may move to the left.

In a possible implementation manner, the electronic device may clip the twenty-first drawing result through the function glBlitFramebuffer to obtain the U-th drawing frame.

The glBlitFramebuffer function can be:

void glBlitFramebuffer

GLint srcX0,//The abscissa of the first pixel in the twenty-first drawing result, srcX0=(K3–1)*L/2

GLint srcY0,//The ordinate of the first pixel in the twenty-first drawing result, srcY0=(K4–1)*L/2

GLint srcX1,//The abscissa of the second pixel in the twenty-first drawing result, srcX1=srcX0+L

GLint srcY1,//The ordinate of the second pixel in the twenty-first drawing result, srcY1=srcY0+H

GLint dstX0,//The abscissa of the lower left corner vertex of the default memory space

GLint dstY0,//The ordinate of the lower left corner vertex of the default memory space

GLint dstX1,//The abscissa of the upper right corner of the default memory space

GLint dstY1,//The ordinate of the right corner vertex of the default memory space

GLbitfield mask,

GLenum filter).

The glBlitFramebuffer function is used to clip the twenty-first drawing result to the default memory space. Specifically, the electronic device cuts the size of the twenty-first drawing result to be the same as the size of the default memory space. Then the electronic device writes the cut twenty-first drawing result into the default memory space. The first pixel point is the pixel point at the vertex position of the lower left corner of the twenty-first clipped drawing result. The second pixel is the pixel at the vertex position of the upper right corner of the twenty-first cut drawing result.

S5607-S5610, the electronic device draws the U+2th drawing frame.

S5607. The CPU of the electronic device acquires the drawing parameters of the U+2th drawing frame.

The CPU of the electronic device may acquire the drawing parameters of the U+2th drawing frame. Specifically, the CPU of the electronic device 100 may acquire the drawing parameters of the U+2th drawing frame of the application program through the interface in the three-dimensional image processing library. The drawing parameters of the U+2th drawing frame are used to draw and render the U+2th drawing frame. The drawing parameters of the U+2th drawing frame may include the information carried in the drawing instructions (such as the draw call instruction) of the U+2th drawing frame, such as the coordinates, color value, depth of each vertex in the drawing content of the draw call instruction value, etc.

It can be understood that, before the electronic device draws the U+2 th drawing frame, the electronic device displays the U+1 th frame. If the U+1th frame is a drawing frame, the electronic device may draw the U+1th frame according to the steps of drawing the U+2th drawing frame in steps S5607 to S5610. If the U+1th frame is a predicted frame, the electronic device may predict the U+1th predicted frame according to steps S5611-S5615.

S5608. The CPU in the electronic device sends a drawing instruction to the GPU for instructing the GPU to draw the U+2th drawing frame.

The CPU of the electronic device may send a drawing instruction for instructing the GPU to draw the U+2 th drawing frame to the GPU according to the drawing parameters of the U+2 th drawing frame. It can be understood that the drawing parameters of the U+2th drawing frame acquired by the CPU may include information of multiple drawing instructions. In this way, the CPU may sequentially send to the GPU multiple drawing instructions for instructing the GPU to draw the U+2 th drawing frame. Here, specific reference may be made to the description in step S5604, which will not be repeated here.

S5609. In the twenty-second memory space, the GPU of the electronic device draws the drawing content in the drawing instruction of the U+2th drawing frame into the second drawing range, and obtains the twenty-second drawing result. The size is less than or equal to the size of the twenty-second memory space and greater than the size of the default memory space.

The GPU of the electronic device may draw the drawing content of the drawing instruction of the U+2th drawing frame into the twenty-second memory space to obtain the twenty-second drawing result. Optionally, the GPU of the electronic device may draw the drawing content of the drawing instruction of the U+2th drawing frame into the second drawing range of the twenty-second memory space to obtain the twenty-second drawing result. The size of the second drawing range is smaller than or equal to the size of the twenty-second memory space, and is larger than the size of the default memory space. It can be understood that, in this embodiment of the present application, the drawing range in any attachment in the twenty-second memory space may be referred to as the second drawing range.

In a possible implementation manner, the size of the second drawing range is determined by the electronic device according to the viewport parameter in the target application. The viewport parameter of the target application is used to specify the width and height of the drawing range in which the image frame of the target application is drawn in the electronic device. Generally, the width of the drawing range specified by the viewport parameter of the target application is the same as the width of the default memory space in the electronic device, and the height of the drawing range specified by the viewport parameter is the same as the height of the default memory space in the electronic device. The electronic device may modify the viewport parameter of the target application through the hook technology to specify the size of the second drawing range. For example, if the width of the drawing range of the U-th drawing frame specified by the viewport of the target application is L and the height is H. Then, the electronic device can modify the viewport parameter of the target application through the hook technology, and designate the width of the second drawing range as L·K5 and the height as H·K6. Both K5 and K6 are greater than 1. Optionally, K5 and K6 are floating point numbers greater than 1. K5 is less than or equal to K1, and K6 is less than or equal to K2. The viewport parameter of the U+2 th drawing frame may be the second parameter in this embodiment of the present application.

As shown in (a) of FIG. 62 , (a) of FIG. 62 shows an attachment 6202 and a second drawing range 6201 in the twenty-second memory space. Attachment 6202 has a width of L·K1 and a height of H·K2. The second drawing range 6201 has a width of L·K5 and a height of H·K6. The following description will be given by taking the electronic device drawing the drawing content of the drawing instruction in the U+2 th drawing frame to the second drawing range 6201 as an example. The accessory 6202 may be the accessory 5901 or the accessory 5902, or the accessory 590n in the twenty-second memory space shown in FIG. 59 .

In one possible implementation, K5 is equal to K1, which is a fixed value for the system configuration of the electronic device. K6 is equal to K2, a fixed value for the system configuration of the electronic device. It can be understood that the size of the second drawing range is equal to the size of the twenty-second memory space.

In a possible implementation manner, the values of K5 and K6 may be determined by including the camera's corner parameter in the drawing parameters of the previous drawing frame in the U+2th drawing frame. If the image frame displayed by the electronic device is as shown in FIG. 54A, the electronic device can insert a prediction frame into every two rendering frames, then the rendering frame preceding the U+2-th rendering frame is the U-th rendering frame. For the calculation process of K5 and K6, reference may be made to the calculation process of K3 described above, which will not be repeated here.

The electronic device may draw the drawing content of the drawing instruction of the U+2 drawing frame in the second drawing range of the twenty-second memory space. The size of the second drawing range is larger than the size of the drawing range specified by the viewport parameter of the target application. Since the second drawing range is enlarged, the electronic device can draw more drawing content into the second drawing range. However, if the drawing content of the U+2th drawing frame also follows the enlargement ratio of the drawing range, the drawing content in the second drawing range will not increase. The electronic device needs to make the drawing content in the U+2 drawing frame under the camera coordinates, and perform a bitwise transformation with the origin of the camera coordinates as the center point, so that the drawing content will shrink in the middle and become smaller under the camera coordinates. Therefore, the electronic device can draw more drawing contents within the first drawing range.

The electronic device may modify the projection matrix in the drawing parameters of the U-th drawing frame, so that the drawing content of the U+2-th drawing frame is drawn more within the first drawing range. The electronic device can generate a matrix T2 according to the determined values of K5 and K6, and the projection matrix P can be modified to P*T2. The matrix T2 can be:

In this embodiment of the present application, the matrix T2 may be referred to as a second transformation matrix.

The glBufferSubData function is used to write the data of size size pointed to in data into the buffer memory corresponding to the target at the offset position. After the electronic device hooks this function, it is determined that the data contains the information of the projection matrix P according to the value of the target as GL_UNIFORM_BUFFER and the size of the size as 2848. The electronic device uses the memory information to take out the P in the data, and write P*T2 into the specified location of the data. The electronic device draws the drawing content of the U+2th drawing frame into the second drawing range according to the modified projection matrix (ie, P*T2), and can obtain the twenty-second drawing result. The twenty-second drawing result may include more drawing contents in the drawing instruction of the U+2th drawing frame. Here, reference may be made to the description of the glBufferSubData function in step S305, which will not be repeated here.

The twenty-second rendering result of the U+2-th rendering frame may be shown as the twenty-second rendering result 6203 in (b) of FIG. 62 . The width in the twenty-second drawing result 6203 is L·K5, and the height is designated as H·K6. That is, the size of the twenty-second drawing result may be the same as the size of the second drawing range. Optionally, the size of the twenty-second drawing result may be the same as the size of the twenty-second memory space. The following description takes an example that the size of the twenty-second drawing result is the same as the size of the second drawing range.

S5610. In the default memory space, the GPU of the electronic device cuts the size of the twenty-second drawing result to the same size as the default memory space, and obtains the U+2th drawing frame.

Generally, the size of the display screen of the electronic device is the same as the size of the default memory space. Therefore, before the image frame of the target application is displayed, the electronic device needs to cut the size of the image frame to be the same as the size of the default memory space. In the default memory space, the GPU can cut the size of the twenty-second drawing result to be the same as the size of the default memory space to obtain the U+2th drawing frame. The U+2th rendering frame is an image frame for display by the electronic device. The U+2 th drawing frame may be shown as the U+2 th drawing frame 6204 illustrated in (c) of FIG. 62 . The U+2-th rendering frame 6204 may be moved to the left.

In a possible implementation manner, the electronic device may clip the twenty-second drawing result through the function glBlitFramebuffer to obtain the U+2th drawing frame.

The glBlitFramebuffer function can be:

void glBlitFramebuffer(

GLint srcX2,//The abscissa of the third pixel in the twenty-second drawing result, srcX2=(K5–1)*L/2

GLint srcY2,//The ordinate of the third pixel in the twenty-second drawing result, srcY2=(K6–1)*L/2

GLint srcX3,//The abscissa of the fourth pixel in the twenty-second drawing result, srcX3=srcX2+L

GLint srcY3,//The ordinate of the fourth pixel in the twenty-second drawing result, srcY3=srcY2+H

GLint dstY1,//The ordinate of the upper right corner vertex of the default memory space

GLbitfield mask,

GLenum filter).

The glBlitFramebuffer function can be used to clip the twenty-second drawing result to the default memory space. Specifically, the electronic device cuts the size of the twenty-second drawing result to be the same as the size of the default memory space. Then the electronic device writes the cut twenty-second drawing result into the default memory space. The third pixel point is the pixel point at the lower left corner vertex position of the twenty-second clipped drawing result. The fourth pixel is the pixel at the vertex position of the upper right corner of the twenty-second clipped drawing result.

S5611-S5615, the electronic device predicts the U+3th prediction frame.

S5611. The electronic device sends an instruction for instructing the GPU to calculate the motion vector to the GPU.

S5612 , the GPU of the electronic device uses the twenty-first rendering result and the twenty-second rendering result to calculate the motion vector S of the twenty-second rendering result.

The GPU of the electronic device may use the twenty-first rendering result and the twenty-second rendering result to calculate the motion vector of the twenty-second rendering result.

In a possible implementation manner, the GPU of the electronic device uses the twenty-first rendering result and the twenty-second rendering result to calculate the motion vector of the twenty-second rendering result, which may specifically include the following steps:

1. The GPU may divide the twenty-second rendering result 6203 into Q pixel blocks. Each pixel block may contain f*f (eg 16*16) pixel points. The twenty-second rendering result 6203 is the rendering result of the U+2th rendering frame. As shown in (b) of Fig. 63A.

2. The GPU takes the first pixel block in the twenty-second rendering result 6203 (for example, the pixel block 6205 in (b) in FIG. 63A ), and searches the twenty-first rendering result 6103 for a pixel block that matches the first pixel block. Matching pixel blocks, such as pixel block 6105 in (a) of Figure 63A. The twenty-first rendering result 6103 is the rendering result of the U-th rendering frame.

In the embodiment of the present application, among all the candidate blocks in the twenty-first rendering result, the candidate block with the smallest absolute difference with the RGB value of the first pixel block is called a matching pixel block matching the th pixel block. The electronic device needs to find a matching pixel block 6105 that matches the pixel block in the twenty-first drawing result. Optionally, the GPU of the electronic device may find a matching pixel block that matches the first pixel block in the U-th drawing frame by using a diamond search algorithm. As shown in FIG. 63B , the GPU can use the diamond search algorithm to find the drawing result of the U+2th drawing frame of (a) in the drawing result of the U-th frame in (b) (ie, the twenty-first drawing result 6103 ) (ie, the pixel block 6205 in the twenty-second rendering result 6203) matches the matching pixel block. The electronic device can use the pixel point 6211 in the upper left corner of the pixel block 6205 to perform a diamond search. The electronic device may find that the pixel block 6105 with the pixel point 6106 as the upper left corner matches the pixel block 6205 in the drawing result of the U-th drawing frame. For details of the diamond search algorithm, reference may be made to the description in the prior art, which will not be repeated here.

As shown in FIG. 63C , (a) and (b) of FIG. 63C exemplarily show a schematic diagram of the electronic device finding a matching pixel block 6105 with the pixel block 6205 in the U-th rendering frame. As shown in (a) of FIG. 63C , the electronic device performs a diamond search on the upper left pixel point 6211 of the pixel block 6205 . As shown in (b) of FIG. 63C , in an achievable manner, the electronic device first performs a diamond search with the pixel point 9012 in the U-th drawing frame as the center point. The coordinates of the pixel point 6211 are the same as the coordinates of the pixel point 6212 . The electronic device can calculate that the pixel block 6205 is respectively in the pixel block with the upper left pixel point as the pixel point 6212, the upper left pixel point is the pixel block of the pixel point 6301, the upper left pixel point is the pixel block of the pixel point 6302, the upper left pixel point is the pixel block of the pixel point 6302, and the upper left pixel point is the pixel block of the pixel point 6302. The point is the pixel block of pixel point 6303, the upper left pixel point is the pixel block of pixel point 6304, the upper left pixel point is the pixel block of pixel point 6305, the upper left pixel point is the pixel block of pixel point 6306, and the upper left pixel point is the pixel block of pixel point 6306. For the pixel block of pixel point 6307, the upper left pixel point is the color value difference of the pixel block of pixel point 6308. The electronic device can then perform a small diamond search with the upper left pixel point in the pixel block with the smallest color value difference as the center. For example, in the above-mentioned pixel block, the pixel block whose upper left pixel point is the pixel point 6307 and the pixel block 6205 have the smallest color value difference. Then the electronic device can perform a small diamond search with the pixel point 6307 as the center. That is, the electronic device can calculate that the pixel block 6205 is respectively in the pixel block with the upper left pixel point as the pixel point 6311, the upper left pixel point is the pixel block with the pixel point 6312, the upper left pixel point is the pixel block with the pixel point 6313, and the upper left corner pixel point is the pixel block with the pixel point 6313. The pixel point is a pixel block of pixel point 6314. Finally, the electronic device can determine that the pixel block whose upper left pixel point is the pixel point 6314 (ie, the pixel block 6105 shown in FIG. 63B ) has the smallest difference in color value with the pixel block 6105 . Then the electronic device can determine that the pixel block 6105 in the Nth frame is a matching pixel block of the pixel block 6205 .

For example, as shown in FIG. 63B , the matching block with the pixel block 6205 in FIG. 63B is the pixel block 6105 . Then the motion vector of the pixel block 6205 is the motion vector A1 illustrated in (b) of FIG. 63B .

4. The GPU can calculate the motion vector of each pixel block in the Q pixel blocks in the twenty-second rendering result 6203 according to the above steps 1-3, namely A1, A2, . . . , AQ. The motion vector of the twenty-second rendering result is S=(A1, A2, . . . , AQ).

S5613: The electronic device sends an instruction to the GPU for instructing the GPU to draw the predicted frame.

The CPU of the electronic device may send an instruction for drawing the U+3 th predicted frame to the GPU after the GPU calculates the motion vector S of the twenty-second drawing result.

S5614. In the twenty-third memory space, the GPU of the electronic device predicts and obtains the twenty-third drawing result according to the motion vector S and the twenty-second drawing result, and the size of the twenty-third drawing result is the same as the twenty-second drawing result. are the same size.

The objects in the U+2th rendering frame and the U+3th prediction frame are the same, but the positions of the objects in the U+2th rendering frame and the U+3th prediction frame are different. The GPU may generate the twenty-third rendering result using the twenty-second rendering result of the U+2-th rendering frame and the motion vector of the twenty-second rendering result. Specifically, the GPU may predict the motion vector V of the twenty-third rendering result according to the motion vector S of the twenty-second rendering result. The motion vector V may be equal to G times the motion vector S. G is less than 1, and G may be equal to 0.5. The GPU may generate the twenty-third rendering result according to the color value of each pixel of the twenty-second rendering result and the displacement of each pixel under the motion vector V.

In a possible implementation manner, the GPU of the electronic device generates the twenty-third drawing result in the twenty-third memory space. Further, the GPU may generate the twenty-third drawing result within the third drawing range in the twenty-third memory space. The size of the third drawing range is smaller than or equal to the size of the twenty-third memory space, and is larger than the size of the default memory space. It can be understood that, in this embodiment of the present application, the drawing range in any attachment in the twenty-third memory space may be referred to as the third drawing range.

In a possible implementation manner, the size of the third drawing range is determined by the electronic device according to the viewport parameter in the target application. The viewport parameter of the target application is used to specify the width and height of the drawing range in which the image frame of the target application is drawn in the electronic device. Generally, the width of the drawing range specified by the viewport parameter of the target application is the same as the width of the default memory space in the electronic device, and the height of the drawing range specified by the viewport parameter is the same as the height of the default memory space in the electronic device. The electronic device may modify the viewport parameter of the target application through the hook technology to specify the size of the third drawing range. For example, if the width of the drawing range of the U-th drawing frame specified by the viewport of the target application is L and the height is H. Then, the electronic device can modify the viewport parameter of the target application through the hook technology, and designate the width of the third drawing range as L·K7 and the height as H·K8. Both K7 and K8 are greater than 1. Optionally, K7 and K8 are floating point numbers greater than 1. K7 is less than or equal to K1, and K8 is less than or equal to K2.

As shown in (a) of FIG. 64 , (a) of FIG. 64 shows an attachment 6402 and a third drawing range 6401 in the twenty-third memory space. Attachment 6402 has a width of L·K1 and a height of H·K2. The third drawing range 6401 has a width of L·K7 and a height of H·K8. The following description will be given by taking the electronic device generating the twenty-third drawing result in the third drawing range 6401 as an example. The accessory 6402 may be the accessory 6001 or the accessory 6002 or the accessory 600n in the twenty-third memory space shown in FIG. 60 .

In one possible implementation, K7 is equal to K1, a fixed value for the system configuration of the electronic device. K8 is equal to K2, a fixed value for the system configuration of the electronic device. It can be understood that the size of the third drawing range is equal to the size of the twenty-third memory space.

In a possible implementation manner, the values of K7 and K8 may be determined by including the camera's corner parameter in the drawing parameters in the U-th drawing frame and the U+2-th drawing frame. For the calculation process of K7 and K8, reference may be made to the description of the calculation process of K3 above, which will not be repeated here.

The twenty-third rendering result of the U+3-th prediction frame may be shown as the twenty-third rendering result 6403 in (b) of FIG. 64 . The width in the twenty-third drawing result 6403 is L·K5 and the height is H·K6.

It can be understood that the enlargement multiples (eg K7 and K8) of the drawing range in the predicted frame may be determined according to the drawing parameters of the two drawing frames in which the predicted frame is generated. The enlargement multiples of the drawing range of the drawing frame (for example, K3 and K4) can be determined by drawing parameters of the drawing frame and drawing parameters of the drawing frame preceding the drawing frame. If the drawing frame is the first frame in which the image frame prediction method starts to be executed, the system of the electronic device may configure a fixed value for the expansion multiple of the drawing range of the drawing frame.

S5615. In the default memory space, the GPU of the electronic device cuts the size of the twenty-third drawing result to be the same as the size of the original memory space to obtain the U+3th predicted frame.

Generally, the size of the display screen of the electronic device is the same as the size of the default memory space. Therefore, before the image frame of the target application is displayed, the electronic device needs to cut the size of the image frame to be the same as the size of the default memory space. In the default memory space, the GPU can cut the size of the twenty-second drawing result to be the same as the size of the default memory space to obtain the U+3th predicted frame. The U+3-th predicted frame is an image frame for display by the electronic device. The U+3-th predicted frame may be shown as the U+3-th predicted frame 6404 illustrated in (c) in FIG. 64 . For a specific implementation manner of clipping the twenty-third rendering result by the electronic device, reference may be made to the above description of clipping the twenty-first rendering result and the twenty-second rendering result by the electronic device, which will not be repeated here.

In this embodiment of the present application, the U-th drawing frame may be referred to as the first drawing frame. The U+2th drawing frame may be referred to as the second drawing frame. The U+3-th predicted frame may be referred to as the first predicted frame.

The process of predicting an image frame has almost no computational consumption for the CPU, and only needs to send some instructions to the GPU. For the GPU, only the motion vector of the image frame needs to be calculated. These calculations are all parallel and only need to be calculated once. Each computing unit performs a small number of basic operations, which can reduce GPU calculations and improve performance.

It can be understood that the embodiment of the present application is not limited to the electronic device predicting the U+3 th predicted frame by using the U th rendering frame and the U+2 th rendering frame. Optionally, the electronic device may also predict the U+2 th drawing frame by using the N th frame and the N+1 th frame. Optionally, the electronic device may also predict the N+4th frame by using the Nth frame and the N+3th frame. This embodiment of the present application does not limit this.

It can be understood that, when the electronic device displays the video interface of the target application, the electronic device can predict the predicted frame according to different strategies. That is, the electronic device predicts the N+3th frame according to the Nth frame and the U+2th drawing frame in the first time period, and predicts the U+2th drawing frame according to the Nth frame and the N+1th frame in the second time period. strategy. For example, when the GPU performs many tasks, the electronic device may first predict the N+3 th frame according to the N th frame and the U+2 th drawing frame. When the GPU performs few tasks, the electronic device may predict the U+2th drawing frame according to the Nth frame and the N+1th frame. This embodiment of the present application does not limit this.

In a possible implementation manner, the twenty-first drawing result and the twenty-second drawing result in this embodiment of the present application are used to draw the interface scene content (for example, game content) of the target application, and the UI in the target application is not drawn. controls.

It can be understood that the twenty-first drawing result in this embodiment of the present application may be the first drawing range in which the drawing content of the drawing instruction of the U-th drawing frame is drawn. Twenty-one drawing results 6103. The twenty-first drawing result may be an attachment to the twenty-first memory space in which the drawing content of the drawing instruction of the U-th drawing frame is drawn. At this time, the size of the twenty-first drawing result is equal to the size of the attachment in the twenty-first memory space. That is, the width of the twenty-first drawing result is the same as the width of the twenty-first memory space, and the height of the twenty-first drawing result is the same as the height of the twenty-first memory space. Likewise, the twenty-second drawing result in this embodiment of the present application may be the second drawing range in which the drawing content of the drawing instruction of the U+2th drawing frame is drawn. Twenty-two drawing results 6203. The twenty-second drawing result may be an attachment to the twenty-second memory space in which the drawing content of the drawing instruction of the U+2th drawing frame is drawn. At this time, the size of the twenty-second drawing result is equal to the size of the attachment in the twenty-second memory space. That is, the width of the twenty-second drawing result is the same as the width of the twenty-second memory space, and the height of the twenty-second drawing result is the same as the height of the twenty-second memory space.

With the image frame prediction method provided by the embodiment of the present application, when drawing the first drawing frame, the electronic device draws the drawing content of the drawing instruction of the first drawing frame into the twenty-first memory space, and obtains the twenty-first As a result of drawing, the size of the twenty-first memory space is larger than the size of the default memory space, which is the memory space provided by the electronic device system for storing image frames for display; when drawing the second drawing frame, the electronic device will The drawing content of the drawing instruction of the second drawing frame is drawn into the twenty-second memory space, and the twenty-second drawing result is obtained. The size of the twenty-second memory space is larger than the size of the default memory space; the electronic device draws according to the twenty-first The result and the twenty-second drawing result, the twenty-third drawing result is generated in the twenty-third memory space, and the size of the twenty-third memory space is larger than the size of the default memory space; the electronic device cuts the twenty-third drawing result. Cut to the same size as the default memory space to get the third predicted frame. In this way, the electronic device draws the first drawing frame and the second drawing frame in the enlarged memory space, and can draw more drawing content than the first drawing frame and the second drawing frame displayed by the electronic device. In this way, in the predicted frame predicted by the electronic device, there may be rendering content that is not in the first rendering frame and the second rendering frame displayed by the electronic device. Therefore, the drawing content in the predicted frame predicted by the electronic device is closer to the shooting content in the shooting field of view of the camera. Therefore, the image frame predicted by the electronic device can be more accurate.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

As used in the above embodiments, the term "when" may be interpreted to mean "if" or "after" or "in response to determining..." or "in response to detecting..." depending on the context. Similarly, depending on the context, the phrases "in determining..." or "if detecting (the stated condition or event)" can be interpreted to mean "if determining..." or "in response to determining..." or "on detecting (the stated condition or event)" or "in response to the detection of (the stated condition or event)".

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state drives), and the like.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented. The process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed , which may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store program codes.

Claims

A method for image frame prediction, comprising:

When drawing the first drawing frame, the electronic device determines that the spatial information of the first drawing object has changed according to the drawing instruction of the first drawing object, and determines that the spatial information of the second drawing object has not changed according to the drawing instruction of the second drawing object. When a change occurs, the electronic device writes the color data of the first drawing object into the first color attachment, and writes the color data of the second drawing object into the second color attachment;

When drawing the second drawing frame, the electronic device determines that the spatial information of the first drawing object has changed according to the drawing instruction of the third drawing object, and determines the space of the fourth drawing object according to the drawing instruction of the fourth drawing object The information does not change, and the electronic device writes the color data of the third drawing object into the third color attachment, and writes the color data of the fourth drawing object into the fourth color attachment;

The electronic device generates a fifth color attachment of the first predicted frame according to the first color attachment and the third color attachment, and generates the first predicted frame according to the second color attachment and the fourth color attachment The sixth color attachment;

The electronic device synthesizes the fifth color attachment and the sixth color attachment into the first predicted frame.
The method according to claim 1, wherein the electronic device generates a fifth color accessory of the first predicted frame according to the first color accessory and the third color accessory, comprising:

The electronic device determines a first motion vector of the third color accessory according to the first color accessory and the third color accessory;

The electronic device generates a fifth color accessory of the first predicted frame based on the third color accessory and the first motion vector.
The method according to claim 2, wherein the electronic device determines the first motion vector of the third color accessory according to the first color accessory and the third color accessory; specifically comprising:

The electronic device divides the third color accessory into Q pixel blocks, and the electronic device takes out a first pixel block from the Q pixel blocks of the third color accessory;

The electronic device determines, in the first color attachment, a second pixel block that matches the first pixel block;

obtaining, by the electronic device, a motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block;

The electronic device determines a first motion vector of the third color accessory according to the motion vector of the first pixel block.
The method according to claim 3, wherein the electronic device determines, in the first color attachment, a second pixel block that matches the first pixel block, specifically comprising:

The electronic device determines a plurality of candidate pixel blocks in the first color attachment by using the first pixel points in the first pixel block;

The electronic device respectively calculates the difference between the color values of the plurality of candidate pixel blocks and the first pixel block;

The electronic device determines the second pixel block matching the first pixel block according to the difference between the color values of the first pixel block of the plurality of candidate pixel blocks, and the second pixel block is a multi-pixel block. The candidate pixel block with the smallest difference in color value from the first pixel block among the candidate pixel blocks.
The method according to claim 1, wherein the electronic device generates a sixth color accessory of the first predicted frame according to the second color accessory and the fourth color accessory, which specifically includes:

The electronic device determines a second motion vector of the fourth color accessory according to the second color accessory and the fourth color accessory;

The electronic device generates a sixth color accessory of the first predicted frame based on the fourth color accessory and the second motion vector.
The method according to claim 5, wherein the electronic device determines the second motion vector of the fourth color accessory according to the second color accessory and the fourth color accessory, and specifically includes:

The electronic device divides the fourth color accessory into Q pixel blocks, and the electronic device takes out a third pixel block from the fourth color accessory;

The electronic device calculates the first position of the third pixel block in the second color attachment;

The electronic device determines the motion vector of the third pixel block according to the first position and the second position of the third pixel block in the fourth color attachment;

The electronic device determines a second motion vector of the fourth color accessory according to the motion vector of the third pixel block.
The method according to claim 6, wherein the electronic device calculates the first position of the third pixel block in the second color attachment, which specifically includes:

The electronic device obtains a first matrix in the drawing instructions of the first drawing frame and a second matrix of the drawing instructions in the second drawing frame, where the first matrix is used to record the first drawing frame The corner information of the camera position of the second matrix is used to record the corner information of the camera position of the second drawing frame;

The electronic device calculates a first position of the third pixel block in the second color attachment according to the first matrix and the second matrix, and the depth value of the third pixel block.
The method according to any one of claims 1-7, wherein the drawing instruction of the first drawing object includes an execution drawing instruction of the first drawing object and a drawing state device instruction of the first drawing object , wherein the execution drawing instruction of the first drawing object is used to trigger the electronic device to draw and render the drawing state data of the first drawing object, and generate a drawing result, and the drawing state device of the first drawing object The instruction is used to set the drawing state data on which the execution drawing instruction of the first drawing object depends, and the drawing state data of the first drawing object includes vertex information data, vertex index, texture information, vertex information of the first drawing object Information cache index;

The drawing instruction of the second drawing object includes an execution drawing instruction of the second drawing object and a drawing state device instruction of the second drawing object, wherein the executing drawing instruction of the second drawing object is used to trigger the The electronic device draws and renders the drawing state data of the second drawing object, and generates a drawing result. The drawing state device instruction of the second drawing object is used to set the drawing on which the drawing instruction of the second drawing object depends. State data, the drawing state data of the second drawing object includes vertex information data, vertex index, texture information, and vertex information cache index of the second drawing object;

The drawing instruction of the third drawing object includes an execution drawing instruction of the third drawing object and a drawing state device instruction of the third drawing object, wherein the executing drawing instruction of the third drawing object is used to trigger the The electronic device draws and renders the drawing state data of the third drawing object, and generates a drawing result. The drawing state device instruction of the third drawing object is used to set the drawing on which the drawing instruction of the third drawing object depends. State data, the drawing state data of the third drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the third drawing object;

The drawing instruction of the fourth drawing object includes an executing drawing instruction of the fourth drawing object and a drawing state device instruction of the fourth drawing object, wherein the executing drawing instruction of the fourth drawing object is used to trigger the The electronic device draws and renders the drawing state data of the fourth drawing object, and generates a drawing result. The drawing state device instruction of the fourth drawing object is used to set the drawing on which the drawing instruction of the fourth drawing object depends. State data, the drawing state data of the fourth drawing object includes vertex information data, vertex index, texture information, and vertex information buffer index of the fourth drawing object.
The method according to any one of claims 1-8, wherein determining, by the electronic device, according to a drawing instruction of the first drawing object, that the spatial information of the first drawing object has changed, comprising: determining, by the electronic device, that the spatial information of the first drawing object has changed There is a transition matrix parameter in the drawing instruction of the first drawing object, and the electronic device determines that the existing transition matrix parameter in the drawing instruction of the first drawing object is different from the corresponding transition matrix parameter of the first drawing object , the transition matrix is used to describe the mapping relationship from the local coordinate system of the drawing object to the world coordinate system;

Determining, by the electronic device, that the spatial information of the second drawing object has not changed according to the drawing instruction of the second drawing object, includes: determining, by the electronic device, that there is no transition matrix parameter in the drawing instruction of the second drawing object, and The electronic device determines that the existing transition matrix parameter in the drawing instruction of the second drawing object is the same as the corresponding transition matrix parameter of the second drawing object;

Determining, by the electronic device, that the spatial information of the first drawing object has changed according to the drawing instruction of the third drawing object, includes: determining, by the electronic device, that a transfer matrix parameter exists in the drawing instruction of the third drawing object, the electronic The device determines that the existing transition matrix parameter in the drawing instruction of the third drawing object is different from the corresponding transition matrix parameter of the third drawing object;

Determining, by the electronic device, that the spatial information of the fourth drawing object has not changed according to the drawing instruction of the fourth drawing object, includes: determining, by the electronic device, that there is no transition matrix parameter in the drawing instruction of the second drawing object, and The electronic device determines that the existing transition matrix parameter in the drawing instruction of the second drawing object is the same as the transition matrix parameter of the corresponding second drawing object.
The method according to any one of claims 1-8, wherein when the first drawing frame is drawn, the electronic device determines that the spatial information of the first drawing object changes according to a drawing instruction of the first drawing object , according to the drawing instruction of the second drawing object, it is determined that the spatial information of the second drawing object has not changed, and the electronic device writes the color data of the first drawing object into the first color attachment, and writes the second drawing object Before the color data of the object is written into the second color attachment, the method further includes:

The electronic device creates a first memory space, a second memory space, a third memory space, a fourth memory space, a fifth memory space, a sixth memory space, and a seventh memory space; wherein the first memory space is used for The first color accessory is stored, the second memory space is used to store the second color accessory, the third memory space is used to store the third color accessory, and the fourth memory space is used to store the second color accessory. Four color attachments, the fifth memory space is used to store the fifth color attachment, the sixth memory space is used to store the sixth color attachment, and the seventh memory space is used to store the first predicted frame.
The method according to claim 10, wherein when the first drawing frame is drawn, the electronic device determines that the spatial information of the first drawing object changes according to the drawing instruction of the first drawing object, and according to the second drawing The drawing instruction of the object determines that the spatial information of the second drawing object has not changed, and the electronic device writes the color data of the first drawing object into the first color attachment, and writes the color data of the second drawing object. After adding the second color accessory, the method further includes:

The electronic device combines the first color attachment and the second color attachment into the first drawing frame in the seventh memory space.
The method according to claim 10, wherein when the second drawing frame is drawn, the electronic device determines that the spatial information of the first drawing object changes according to the drawing instruction of the third drawing object, and according to the third drawing object The drawing instruction of the fourth drawing object determines that the spatial information of the fourth drawing object has not changed, and the electronic device writes the color data of the third drawing object into the third color attachment, and writes the color data of the fourth drawing object into the third color attachment. After the four-color attachment, the method further includes:

The electronic device combines the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space.
The method according to claim 11, wherein the electronic device synthesizes the first color attachment and the second color attachment into the first drawing frame in the seventh memory space, specifically comprising: :

The electronic device combines the first color attachment and the second color attachment into the first drawing frame in the seventh memory space according to the first depth attachment and the second depth attachment, and the first depth attachment The depth data of the first drawing object is written, and the second depth attachment is used for writing the second color attachment of the second drawing object.
The method according to claim 12, wherein the electronic device synthesizes the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space, specifically comprising: :

The electronic device combines the third color attachment and the fourth color attachment into the second drawing frame in the seventh memory space according to the third depth attachment and the fourth depth attachment; the third depth attachment The fourth depth attachment is used for writing the depth data of the third drawing object, and the fourth depth attachment is used for writing the depth data of the fourth drawing object.
A method for image frame prediction, comprising:

The electronic device determines the tenth moving object in the tenth drawing frame according to the tenth drawing instruction, and determines the eleventh moving object in the eleventh drawing frame according to the eleventh drawing instruction;

The electronic device determines that the tenth moving object matches the eleventh moving object according to the attribute of the tenth moving object and the attribute of the eleventh moving object;

The electronic device determines, according to the tenth moving object and the eleventh moving object, a drawing result of the twelfth moving object in the tenth prediction frame.
The method according to claim 15, wherein the tenth drawing frame is displayed on the display screen of the electronic device before the eleventh drawing frame, and the tenth predicted frame is displayed before the tenth drawing frame. A drawn frame is then displayed on the display screen of the electronic device.
The method according to any one of claims 15 or 16, wherein an image frame is spaced between the tenth drawing frame and the eleventh drawing frame, and the tenth predicted frame is the first Eleven draw the frame adjacent to the next image frame.
The method according to any one of claims 15-17, wherein the electronic device determines the tenth moving object according to a first attribute of the tenth moving object and a second attribute of the eleventh moving object The moving object is matched with the eleventh moving object, and specifically includes:

The electronic device establishes a first index table, and the first index table is used to save the moving object and the attributes of the moving object in the tenth drawing frame, and the first index table includes the tenth moving object and all the moving objects. Describe the properties of the tenth moving object;

The electronic device establishes a second index table, the second index table is used to save the motion object and the attributes of the motion object in the eleventh drawing frame, and the second index table includes the eleventh motion object and the properties of the eleventh moving object;

The electronic device retrieves the eleventh moving object from the second index table, and determines the tenth moving object matching the eleventh moving object from the first index table.
The method according to any one of claims 15-18, wherein the matching of the tenth moving object with the eleventh moving object comprises: the attribute of the tenth moving object is the same as that of the eleventh moving object. Motion objects have the same properties.
The method according to claim 15, wherein the electronic device determines, according to the tenth moving object and the eleventh moving object, a drawing result of the twelfth moving object in the tenth prediction frame, which specifically includes: :

The electronic device determines a first coordinate of a first point of the tenth moving object, and determines a second coordinate of a second point of the eleventh moving object;

The electronic device determines a tenth motion vector from the tenth moving object to the eleventh moving object according to the displacement from the first coordinate to the second coordinate;

The electronic device determines a drawing result of the twelfth moving object in the tenth prediction frame according to the tenth motion vector and the eleventh moving object.
The method according to claim 20, wherein determining, by the electronic device, the first coordinate of the first point of the tenth moving object specifically includes:

The electronic device determines the first coordinates of the first point according to the coordinates of all pixel points of the tenth moving object;

The electronic device determines the second coordinate of the second point of the eleventh moving object, specifically including:

The electronic device determines the second coordinates of the second point according to the coordinates of all pixel points of the eleventh moving object.
The method according to any one of claims 20 or 21, wherein the first point is a geometric center point of the tenth moving object, and the second point is a geometric center point of the eleventh moving object center point.
The method according to claim 20, wherein the electronic device determines the drawing result of the twelfth moving object in the tenth predicted frame according to the tenth motion vector and the eleventh moving object, and specifically includes:

The electronic device determines, according to the tenth motion vector and the first pixel point of the eleventh moving object, the second pixel point of the twelfth moving object in the tenth predicted frame.
The method according to claim 23, wherein the electronic device determines, according to the tenth motion vector and the first pixel point of the eleventh moving object, the twelfth in the tenth predicted frame The second pixel of the moving object, specifically including:

The electronic device determines, according to the tenth motion vector, an eleventh motion vector from the eleventh moving object to the twelfth moving object;

The electronic device determines, according to the eleventh motion vector and the first pixel of the eleventh moving object, the second pixel of the twelfth moving object in the tenth predicted frame, and the second The pixel point is a pixel point that the first pixel point moves from the eleventh drawing frame to the tenth prediction frame according to the eleventh motion vector.
The method according to claim 24, wherein the eleventh motion vector is K times the tenth motion vector, and the K is greater than 0 and less than 1.
The method of claim 25, wherein the K is equal to 0.5.
An electronic device, characterized in that the electronic device comprises: one or more processors (CPU), a graphics processing unit (GPU), a memory, and a display screen; the memory is coupled to the one or more processors; the CPU coupled to the GPU; wherein:

the memory for storing computer program code, the computer program code comprising computer instructions;

The CPU is configured to determine the tenth moving object in the tenth drawing frame according to the tenth drawing instruction, and determine the eleventh moving object in the eleventh drawing frame according to the second drawing instruction;

The GPU is configured to determine that the tenth moving object and the eleventh moving object are matched according to the attributes of the tenth moving object and the eleventh moving object; The eleventh moving object, determining the drawing result of the twelfth moving object in the tenth prediction frame;

The display screen is configured to display the tenth drawing frame, the eleventh drawing frame, and the tenth prediction frame.
The electronic device according to claim 27, wherein the tenth drawing frame is displayed on the display screen before the eleventh drawing frame, and the tenth predicted frame is displayed before the eleventh drawing frame The frame is then displayed in the display.
The electronic device according to any one of claims 27 or 28, wherein an image frame is spaced between the tenth drawing frame and the eleventh drawing frame, and the tenth predicted frame is the The eleventh drawing frame is adjacent to the next image frame.
The electronic device according to any one of claims 27-29, wherein the CPU is specifically used for:

establishing a first index table, the first index table is used to save the motion object and the attributes of the motion object in the tenth drawing frame, and the first index table includes the tenth motion object and the tenth motion properties of objects;

A second index table is established, the second index table is used to save the motion object and the attributes of the motion object in the eleventh drawing frame, and the second index table includes the eleventh motion object and the first Eleven attributes of moving objects;

The GPU is specifically used for:

The eleventh moving object is retrieved from the second index table, and the tenth moving object matching the eleventh moving object is determined from the first index table.
The electronic device according to any one of claims 27-30, wherein the matching of the tenth moving object with the eleventh moving object comprises: an attribute of the tenth moving object is the same as that of the tenth moving object. A moving object has the same properties.
The electronic device according to claim 31, wherein the GPU is specifically used for:

determining the first coordinate of the first point of the tenth moving object, and determining the second coordinate of the second point of the eleventh moving object;

determining a tenth motion vector from the tenth moving object to the eleventh moving object according to the displacement from the first coordinate to the second coordinate;

According to the tenth motion vector and the eleventh moving object, a drawing result of the twelfth moving object in the tenth prediction frame is determined.
The electronic device according to claim 32, wherein the GPU is specifically used for:

Determine the first coordinates of the first point according to the coordinates of all pixel points of the tenth moving object;

The second coordinate of the second point is determined according to the coordinates of all pixel points of the eleventh moving object.
The electronic device according to claim 32 or 33, wherein the first point is a geometric center point of the tenth moving object, and the second point is a geometric center point of the eleventh moving object .
The electronic device according to claim 32, wherein the GPU is specifically used for:

According to the tenth motion vector and the first pixel point of the eleventh moving object, the second pixel point of the twelfth moving object in the tenth prediction frame is determined.
The electronic device according to claim 35, wherein the GPU is specifically used for:

determining an eleventh motion vector from the eleventh moving object to the twelfth moving object according to the tenth motion vector;

According to the eleventh motion vector and the first pixel of the eleventh moving object, the second pixel of the twelfth moving object in the tenth prediction frame is determined, and the second pixel is the The first pixel is moved from the eleventh drawing frame to the pixel in the tenth prediction frame according to the eleventh motion vector.
The electronic device according to claim 36, wherein the eleventh motion vector is K times the tenth motion vector, and the K is greater than 0 and less than 1.
The electronic device of claim 37, wherein the K is equal to 0.5.
A frame prediction method, characterized in that the method comprises:

Determine the predicted motion vector of the first vertex from the target reference frame to the predicted frame according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame, wherein the first vertex is the first vertex A vertex in a block, the first block is a block in the target reference frame, and the target reference frame is based on the position of the predicted frame relative to the first reference frame or the second reference frame, from the A frame determined in the first reference frame or the second reference frame, the first reference frame and the second reference frame are two adjacent frames in the video stream;

Determine the coordinates of the first vertex in the predicted frame according to the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex;

According to the coordinates of the vertex of the first block in the predicted frame and the coordinates in the target reference frame, determine the pixel block of the first block in the predicted frame;

The predicted frame is displayed including the pixel block.
The method according to claim 39, wherein the determining the pixel block of the first block in the predicted frame according to the coordinates of the vertex of the first block in the predicted frame comprises:

According to the coordinates of the vertices of the first block in the predicted frame and the coordinates in the target reference frame, obtain the coordinates of the pixels of the first block in the predicted frame and the coordinates in the target reference frame The correspondence between the coordinates in ;

According to the coordinates of the pixels of the first block in the target reference frame and the corresponding relationship, the coordinates of the pixels in the first block in the prediction frame are determined.
The method according to claim 40, wherein the pixel of the first block is obtained according to the coordinates of the vertex of the first block in the prediction frame and the coordinates in the target reference frame The correspondence between the coordinates in the predicted frame and the coordinates in the target reference frame includes:

The coordinates of the four vertices of the first block in the prediction frame and the coordinates in the target reference frame are respectively input into the homography transformation formula to obtain a homography equation system, the homography equation system including four equations;

Solving the homography equation system to obtain the homography transformation matrix corresponding to the first block;

According to the homography transformation matrix corresponding to the first block, the homography transformation formula corresponding to the first block is obtained, and the homography transformation formula corresponding to the first block is used to represent the pixels of the first block The correspondence between the coordinates in the predicted frame and the coordinates in the target reference frame.
The method according to claim 41, wherein, according to the coordinates of the pixels of the first block in the target reference frame and the corresponding relationship, it is determined that the pixels in the first block are in the Coordinates in the predicted frame, including:

Input the coordinates of the first pixel in the target reference frame into the homography transformation formula corresponding to the first block to obtain the coordinates of the first pixel in the predicted frame, where the first pixel is one pixel of the first block.
The method according to claim 42, wherein the number of coordinates included in the area of the pixel block is greater than the number of coordinates of the first block in the predicted frame, and the pixel block area is composed of the first block. determining the vertices of the block, and determining the coordinates of the pixels in the first block in the predicted frame according to the coordinates of the pixels in the first block in the target reference frame and the corresponding relationship, further comprising: :

Eliminate the coordinates of the pixels of the first block in the predicted frame from the coordinates in the area of the pixel block to obtain the first coordinates;

The first coordinate is input into the homography transformation formula corresponding to the first block to obtain the pixel of the first coordinate in the target reference frame.
The method according to claim 43, wherein the prediction of the first vertex from the target reference frame to the predicted frame is determined according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame Before the motion vector, the method further includes:

obtaining the first reference frame and the second reference frame;

Determine the target reference frame from the first reference frame and the second reference frame according to the position of the frame to be predicted;

dividing the target reference frame into blocks according to square blocks of the first size;

A motion vector for the block from the target reference frame to the predicted frame is calculated.
The method according to claim 44, wherein when the target reference frame is the first reference frame, the calculating the motion vector of the block from the target reference frame to the prediction frame comprises: :

obtaining the motion vector of the first block from the first reference frame to the second reference frame;

A half of the motion vector of the first block from the first reference frame to the second reference frame is determined as the predicted motion vector of the first block from the target reference frame to the prediction frame.
The method according to claim 45, wherein when the target reference frame is the second reference frame, the calculating a motion vector of the first block from the target reference frame to the predicted frame ,include:

obtaining the motion vector of the first block from the second reference frame to the first reference frame;

determining the negative half of the motion vector of the first block from the second reference frame to the first reference frame as the predicted motion vector of the first block from the target reference frame to the predicted frame .
The method according to claim 46, wherein, according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame, determining the first vertex from the target reference frame to the predicted frame Predicted motion vectors, including:

An average value of the predicted motion vectors of the blocks around the first vertex from the target reference frame to the predicted frame is determined as the predicted motion vector of the first vertex from the target reference frame to the predicted frame.
The method according to claim 47, wherein, according to the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex, determining that the first vertex is in the predicted frame coordinates in , including:

The coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex are added to obtain the coordinates of the first vertex in the predicted frame.
An electronic device, characterized in that the electronic device comprises: one or more processors, a memory and a display screen;

The memory is coupled to the one or more processors for storing computer program code comprising computer instructions for invoking the computer instructions by the one or more processors to cause all The electronic equipment described above performs:

Determine the predicted motion vector of the first vertex from the target reference frame to the predicted frame according to the predicted motion vector of the blocks around the first vertex from the target reference frame to the predicted frame, wherein the first vertex is the first vertex A vertex in a block, the first block is a block in the target reference frame, and the target reference frame is based on the position of the predicted frame relative to the first reference frame or the second reference frame, from the A frame determined in the first reference frame or the second reference frame, the first reference frame and the second reference frame are two adjacent frames in the video stream;

Determine the coordinates of the first vertex in the predicted frame according to the coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex;

According to the coordinates of the vertex of the first block in the predicted frame and the coordinates in the target reference frame, determine the pixel block of the first block in the predicted frame;

The predicted frame is displayed including the pixel block.
The electronic device of claim 49, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

According to the coordinates of the vertices of the first block in the predicted frame and the coordinates in the target reference frame, obtain the coordinates of the pixels of the first block in the predicted frame and the coordinates in the target reference frame The correspondence between the coordinates in ;

According to the coordinates of the pixels of the first block in the target reference frame and the corresponding relationship, the coordinates of the pixels in the first block in the prediction frame are determined.
The electronic device according to claim 50, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

The coordinates of the four vertices of the first block in the prediction frame and the coordinates in the target reference frame are respectively input into the homography transformation formula to obtain a homography equation system, the homography equation system including four equations;

Solving the homography equation system to obtain the homography transformation matrix corresponding to the first block;

According to the homography transformation matrix corresponding to the first block, the homography transformation formula corresponding to the first block is obtained, and the homography transformation formula corresponding to the first block is used to represent the pixels of the first block The correspondence between the coordinates in the predicted frame and the coordinates in the target reference frame.
The electronic device according to claim 51, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

Input the coordinates of the first pixel in the target reference frame into the homography transformation formula corresponding to the first block to obtain the coordinates of the first pixel in the predicted frame, where the first pixel is one pixel of the first block.
The electronic device of claim 52, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

Eliminate the coordinates of the pixels of the first block in the predicted frame from the coordinates in the area of the pixel block to obtain the first coordinates;

The first coordinate is input into the homography transformation formula corresponding to the first block to obtain the pixel of the first coordinate in the target reference frame.
The electronic device of claim 49, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

obtaining the first reference frame and the second reference frame;

Determine the target reference frame from the first reference frame and the second reference frame according to the position of the frame to be predicted;

dividing the target reference frame into blocks according to square blocks of the first size;

A motion vector for the block from the target reference frame to the predicted frame is calculated.
The electronic device of claim 54, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

obtaining the motion vector of the first block from the first reference frame to the second reference frame;

A half of the motion vector of the first block from the first reference frame to the second reference frame is determined as the predicted motion vector of the first block from the target reference frame to the prediction frame.
The electronic device of claim 54, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

obtaining the motion vector of the first block from the second reference frame to the first reference frame;

determining the negative half of the motion vector of the first block from the second reference frame to the first reference frame as the predicted motion vector of the first block from the target reference frame to the predicted frame .
The electronic device of claim 49, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

An average value of the predicted motion vectors of the blocks around the first vertex from the target reference frame to the predicted frame is determined as the predicted motion vector of the first vertex from the target reference frame to the predicted frame.
The electronic device of claim 49, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

The coordinates of the first vertex in the target reference frame and the predicted motion vector of the first vertex are added to obtain the coordinates of the first vertex in the predicted frame.
An image frame generation method, characterized in that the method comprises:

Determine the tenth position of the eleventh vertex of the predicted block in the predicted image frame according to the depth values of the eleventh block and the twelfth block and the position coordinates of the eleventh block and the twelfth block in the image frame Position coordinates;

Wherein, the eleventh block is a block in the first image frame; the twelfth block is a block in the second image frame determined according to a matching algorithm that matches the eleventh block;

generating a prediction block according to the color data of the reference block and the tenth position coordinate, where the reference block is one of the eleventh block or the twelfth block;

The predicted image frame is generated, the predicted image frame including the predicted block.
The method according to claim 59, wherein the position coordinates in the image frame are the position coordinates in a first coordinate system, and the first coordinate system is a two-dimensional coordinate system, and the determining the tenth position of the prediction block The tenth position coordinate of a vertex in the predicted image frame, including:

According to the first depth value of the eleventh block and the twelfth position coordinate of the eleventh block in the first coordinate system, the thirteenth position coordinate in the second coordinate system is calculated and obtained, and the The second coordinate system is a three-dimensional coordinate system;

According to the second depth value of the twelfth block and the fourteenth position coordinate of the twelfth block under the first coordinate system, the fifteenth position coordinate under the second coordinate system is obtained by calculating;

According to the thirteenth position coordinates and the fifteenth position coordinates, the sixteenth position coordinates under the second coordinate system are obtained by calculation;

According to the sixteenth position coordinate, the tenth position coordinate in the first coordinate system is obtained by calculation.
The method according to claim 59 or 60, wherein before the generating the prediction block, the method further comprises:

According to the depth values of the thirteenth block and the fourteenth block, and the position coordinates of the thirteenth block and the fourteenth block in the image frame, it is determined that the twelfth vertex of the predicted block is in the predicted image frame The seventeenth position coordinate in ;

Wherein, the thirteenth block is a block adjacent to the eleventh block, the fourteenth block is a block adjacent to the twelfth block, and the thirteenth block and all The fourteenth block is a mutually matched block determined according to the matching algorithm;

According to the depth values of the fifteenth block and the sixteenth block, and the position coordinates of the fifteenth block and the sixteenth block in the image frame, it is determined that the thirteenth vertex of the predicted block is in the predicted image frame The eighteenth position coordinate in ;

Wherein, the fifteenth block is a block adjacent to the thirteenth block, the sixteenth block is a block adjacent to the fourteenth block, and the fifteenth block and all The sixteenth block is a mutually matched block determined according to the matching algorithm;

According to the depth values of the seventeenth block and the eighteenth block, and the position coordinates of the seventeenth block and the eighteenth block in the image frame, it is determined that the fourteenth vertex of the predicted block is in the predicted image frame The nineteenth position coordinates in ;

The seventeenth block is a block adjacent to the eleventh block and the fifteenth block, and the eighteenth block is adjacent to the twelfth block and the sixteenth block. The blocks are all adjacent blocks, and the seventeenth block and the eighteenth block are mutually matched blocks determined according to the matching algorithm.
The method according to claim 61, wherein the position coordinates in the image frame are the position coordinates in the first coordinate system, the depth values of the thirteenth block and the fourteenth block, the tenth The position coordinates of the three blocks and the fourteenth block in the image frame, and determining the seventeenth position coordinates of the twelfth vertex of the predicted block in the predicted image frame, including:

According to the third depth value of the thirteenth block and the twentieth position coordinate of the thirteenth block under the first coordinate system, the twenty-first position coordinate under the second coordinate system is calculated, so The second coordinate system is a three-dimensional coordinate system;

According to the second depth value of the fourteenth block and the twenty-second position coordinate of the fourteenth block in the first coordinate system, the twenty-third position in the second coordinate system is obtained by calculation Position coordinates;

According to the twenty-first position coordinate and the twenty-third position coordinate, calculate the twenty-fourth position coordinate under the second coordinate system;

According to the twenty-fourth position coordinate, the seventeenth position coordinate in the first coordinate system is obtained by calculation.
The method according to claim 61 or 62, wherein the generating the prediction block according to the color data of the reference block and the tenth position coordinate comprises:

determining the correspondence between the tenth position coordinate, the seventeenth position coordinate, the eighteenth position coordinate and the nineteenth position coordinate and the position coordinates of the four vertices in the reference block respectively;

The prediction block is generated according to the correspondence and the color data of the reference block.
The method according to claim 63, wherein the color data includes color data of each pixel in the reference block, and the generated color data is generated according to the corresponding relationship and the color data of the reference block. The prediction block described above includes:

generating a homography transformation formula according to the corresponding relationship;

According to the homography transformation formula and the position coordinates of each pixel in the reference image frame, determine the position coordinates of each pixel in the predicted image frame;

The prediction block is generated based on the color data of each pixel and the position coordinates of each speed limit in the predicted image frame.
The method according to claim 63 or 64, wherein the reference block comprises: a fifteenth vertex, a sixteenth vertex, a seventeenth vertex and an eighteenth vertex, and the determining the tenth position coordinate , the correspondence between the seventeenth position coordinates, the eighteenth position coordinates and the nineteenth position coordinates respectively and the position coordinates of the four vertices in the reference block, including:

determining that the tenth position coordinate and the position coordinate of the fifteenth vertex are a set of corresponding position coordinates;

determining that the seventeenth position coordinates and the position coordinates of the sixteenth vertex are a set of corresponding position vertices;

determining that the eighteenth position coordinates and the position coordinates of the seventeenth vertex are a set of corresponding position coordinates;

determining that the nineteenth position coordinates and the position coordinates of the eighteenth vertex are a set of corresponding position coordinates;

The reference block is a quadrilateral, and the position direction of the fifteenth vertex relative to the center point of the reference block is the same as the position direction of the eleventh block relative to the fifteenth block; The position direction of the sixteen vertices relative to the center point of the reference block is the same as the position direction of the thirteenth block relative to the seventeenth block; the seventeenth vertex relative to the center point of the reference block The position direction is the same as the position direction of the fifteenth block relative to the eleventh block; the position direction of the eighteenth vertex relative to the center point of the reference block is the same as that of the seventeenth block relative to The positions and directions of the thirteenth blocks are the same.
The method according to claim 60, wherein, calculating the sixteenth position coordinates in the second coordinate system according to the thirteenth position coordinates and the fifteenth position coordinates, comprising: :

Calculate the displacement vector from the thirteenth position coordinate to the fifteenth position coordinate to obtain the first displacement vector;

The sixteenth position coordinate is calculated according to the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale.
The method of claim 66, wherein the first ratio is equal to a preset value.
The method according to claim 66, wherein the image frame where the reference block is located is a reference image frame, the first image frame is located before the second image frame in the data stream, and the calculated Before the sixteenth position coordinates, the method further includes:

determining that the first ratio is equal to the ratio of the absolute value of the first time difference to the absolute value of the second time difference;

The first time difference value is equal to the difference between the time point of the predicted image frame in the data stream and the time point of the reference frame in the data stream;

The second time difference value is equal to the difference between the time point of the first image frame in the data stream and the time point of the second image frame in the data stream.
The method according to claim 66, wherein the image frame where the reference block is located is a reference image frame, the first image frame is located before the second image frame in the data stream, and the calculated Before the sixteenth position coordinates, the method further includes:

determining that the first ratio is equal to the ratio of the first numerical value to the second numerical value;

The first value is equal to the first amount plus one; the second value is equal to the second amount plus one;

the first number is equal to the number of image frames spaced between the reference image frame and the predicted image frame in the data stream; the second number is equal to the reference image frame and the predicted image frame in the data stream The number of image frames spaced between the predicted image frames.
The method according to claim 68 or 69, wherein the first displacement vector is calculated according to the thirteenth position coordinate or the fifteenth position coordinate and the first displacement vector of a first scale to obtain the first Sixteen location coordinates, including:

If the reference image frame is the first image frame, calculating the sixteenth position coordinate according to the thirteenth position coordinate and the first displacement vector of the first scale;

If the reference image frame is the second image frame, the sixteenth position coordinate is calculated according to the fifteenth position coordinate and the first displacement vector of the first scale.
The method according to any one of claims 59-70, wherein the image frame where the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream, The determining that the eleventh vertex of the predicted block is before the tenth position coordinate in the predicted image frame, the method further includes:

determining that one of the first image frame and the second image frame is the reference image frame;

determining that another image frame other than the reference image frame in the first image frame and the second image frame is a matching image frame;

Perform block division for the reference image frame to obtain the reference block;

It is determined according to the matching algorithm that the block matched with the reference block in the matched image frame is the matching block; wherein, if the reference block is the eleventh block, the matching block is the twelfth block block; if the reference block is the twelfth block, the matching block is the eleventh block.
The method according to claim 71, wherein the determining according to the matching algorithm that the block matched with the reference block in the matching image frame is a matching block, comprising:

Determine the block with the highest similarity with the reference block in the first area of the matching image frame as the matching block; the first area is the first area in the matching image frame with the reference position coordinate as the center, the first area An area with a preset shape, and the reference position coordinates are the position coordinates of the reference block in the reference image frame.
The method according to claim 72, wherein, before the determining that the block with the highest similarity to the reference block in the first region of the matching image frame is the matching block, the method further comprises:

The first area is obtained by calculation according to the reference depth value of the reference block, wherein the larger the reference depth value is, the smaller the first area obtained by calculation is.
The method according to claim 72, wherein, before the determining that the block with the highest similarity to the reference block in the first region of the matching image frame is the matching block, the method further comprises:

If the reference depth value of the reference block is greater than or equal to a first threshold, determining that the first area is equal to a first preset area;

If the reference depth value is greater than the second threshold value and smaller than the first threshold value, the first area is calculated according to the reference depth value, wherein the larger the reference depth value is, the smaller the first area is;

If the reference depth value is less than or equal to the second threshold, determining that the first area is equal to a second preset area;

The first threshold is greater than the second threshold, and the second preset area is greater than the first preset area.
An electronic device, characterized in that the electronic device comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used for storing computer program codes, the The computer program code includes computer instructions invoked by the one or more processors to cause the electronic device to perform:

Determine the tenth position of the eleventh vertex of the predicted block in the predicted image frame according to the depth values of the eleventh block and the twelfth block and the position coordinates of the eleventh block and the twelfth block in the image frame Position coordinates;

Wherein, the eleventh block is a block in the first image frame; the twelfth block is a block in the second image frame determined according to a matching algorithm that matches the eleventh block;

generating a prediction block according to the color data of the reference block and the tenth position coordinate, where the reference block is one of the eleventh block or the twelfth block;

The predicted image frame is generated, the predicted image frame including the predicted block.
The electronic device according to claim 75, wherein the position coordinates in the image frame are the position coordinates in a first coordinate system, and the first coordinate system is a two-dimensional coordinate system, and the position coordinates in the determined prediction block are In terms of predicting the tenth position coordinate of the eleventh vertex in the image frame, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute:

According to the first depth value of the eleventh block and the twelfth position coordinate of the eleventh block in the first coordinate system, the thirteenth position coordinate in the second coordinate system is calculated and obtained, and the The second coordinate system is a three-dimensional coordinate system;

According to the second depth value of the twelfth block and the fourteenth position coordinate of the twelfth block under the first coordinate system, the fifteenth position coordinate under the second coordinate system is obtained by calculating;

According to the thirteenth position coordinates and the fifteenth position coordinates, the sixteenth position coordinates under the second coordinate system are obtained by calculation;

According to the sixteenth position coordinate, the tenth position coordinate in the first coordinate system is obtained by calculation.
The electronic device according to claim 75 or 76, wherein before the generating the prediction block, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

According to the depth values of the thirteenth block and the fourteenth block, and the position coordinates of the thirteenth block and the fourteenth block in the image frame, it is determined that the twelfth vertex of the predicted block is in the predicted image frame The seventeenth position coordinate in ;

Wherein, the thirteenth block is a block adjacent to the eleventh block, the fourteenth block is a block adjacent to the twelfth block, and the thirteenth block and all The fourteenth block is a mutually matched block determined according to the matching algorithm;

According to the depth values of the fifteenth block and the sixteenth block, and the position coordinates of the fifteenth block and the sixteenth block in the image frame, it is determined that the thirteenth vertex of the predicted block is in the predicted image frame The eighteenth position coordinate in ;

Wherein, the fifteenth block is a block adjacent to the thirteenth block, the sixteenth block is a block adjacent to the fourteenth block, and the fifteenth block and all The sixteenth block is a mutually matched block determined according to the matching algorithm;

According to the depth values of the seventeenth block and the eighteenth block, and the position coordinates of the seventeenth block and the eighteenth block in the image frame, it is determined that the fourteenth vertex of the predicted block is in the predicted image frame The nineteenth position coordinates in ;

The seventeenth block is a block adjacent to the eleventh block and the fifteenth block, and the eighteenth block is adjacent to the twelfth block and the sixteenth block. The blocks are all adjacent blocks, and the seventeenth block and the eighteenth block are mutually matched blocks determined according to the matching algorithm.
The electronic device according to claim 77, wherein the position coordinates in the image frame are the position coordinates in the first coordinate system, and in the depth value according to the thirteenth block and the fourteenth block, the The location coordinates of the thirteenth block and the fourteenth block in the image frame, determining the seventeenth location coordinates of the twelfth vertex of the predicted block in the predicted image frame, the one or more processing a device, specifically for invoking the computer instructions to cause the electronic device to execute:

According to the third depth value of the thirteenth block and the twentieth position coordinate of the thirteenth block under the first coordinate system, the twenty-first position coordinate under the second coordinate system is calculated, so The second coordinate system is a three-dimensional coordinate system;

According to the second depth value of the fourteenth block and the twenty-second position coordinate of the fourteenth block in the first coordinate system, the twenty-third position in the second coordinate system is obtained by calculation Position coordinates;

According to the twenty-first position coordinate and the twenty-third position coordinate, calculate the twenty-fourth position coordinate under the second coordinate system;

According to the twenty-fourth position coordinate, the seventeenth position coordinate in the first coordinate system is obtained by calculation.
The electronic device according to claim 77 or 78, wherein, in the aspect of generating the prediction block according to the color data of the reference block and the tenth position coordinate, the one or more processors are specifically configured to The computer instructions are invoked to cause the electronic device to perform:

determining the correspondence between the tenth position coordinate, the seventeenth position coordinate, the eighteenth position coordinate and the nineteenth position coordinate and the position coordinates of the four vertices in the reference block respectively;

The prediction block is generated according to the correspondence and the color data of the reference block.
The electronic device according to claim 79, wherein the color data includes color data of each pixel in the reference block, the color data according to the corresponding relationship and the color data of the reference block In terms of generating the prediction block, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute:

generating a homography transformation formula according to the corresponding relationship;

According to the homography transformation formula and the position coordinates of each pixel in the reference image frame, determine the position coordinates of each pixel in the predicted image frame;

The prediction block is generated according to the color data of each pixel and the position coordinates of each speed limit in the predicted image frame.
The electronic device according to claim 79 or 80, wherein the reference block comprises: a fifteenth vertex, a sixteenth vertex, a seventeenth vertex and an eighteenth vertex, and in the determining of the tenth vertex In terms of the correspondence between the position coordinates, the seventeenth position coordinates, the eighteenth position coordinates, and the nineteenth position coordinates, respectively, and the position coordinates of the four vertices in the reference block, the one or more processors , specifically for invoking the computer instructions to cause the electronic device to execute:

determining that the tenth position coordinate and the position coordinate of the fifteenth vertex are a set of corresponding position coordinates;

determining that the seventeenth position coordinates and the position coordinates of the sixteenth vertex are a set of corresponding position vertices;

determining that the eighteenth position coordinates and the position coordinates of the seventeenth vertex are a set of corresponding position coordinates;

determining that the nineteenth position coordinates and the position coordinates of the eighteenth vertex are a set of corresponding position coordinates;

The reference block is a quadrilateral, and the position direction of the fifteenth vertex relative to the center point of the reference block is the same as the position direction of the eleventh block relative to the fifteenth block; The position direction of the sixteen vertices relative to the center point of the reference block is the same as the position direction of the thirteenth block relative to the seventeenth block; the seventeenth vertex relative to the center point of the reference block The position direction is the same as the position direction of the fifteenth block relative to the eleventh block; the position direction of the eighteenth vertex relative to the center point of the reference block is the same as that of the seventeenth block relative to The positions and directions of the thirteenth blocks are the same.
The electronic device according to claim 76, wherein, in the calculation according to the thirteenth position coordinates and the fifteenth position coordinates, the sixteenth position coordinates in the second coordinate system are obtained by calculation In one aspect, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute:

Calculate the displacement vector from the thirteenth position coordinate to the fifteenth position coordinate to obtain the first displacement vector;

The sixteenth position coordinate is calculated according to the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale.
The electronic device according to claim 82, wherein the first ratio is equal to a preset value.
The electronic device according to claim 82, wherein the image frame in which the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream, and before the calculation Before obtaining the sixteenth position coordinates, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

determining that the first ratio is equal to the ratio of the absolute value of the first time difference to the absolute value of the second time difference;

The first time difference value is equal to the difference between the time point of the predicted image frame in the data stream and the time point of the reference frame in the data stream;

The second time difference value is equal to the difference between the time point of the first image frame in the data stream and the time point of the second image frame in the data stream.
The electronic device according to claim 82, wherein the image frame in which the reference block is located is a reference image frame, and the first image frame is located before the second image frame in the data stream, and before the calculation Before obtaining the sixteenth position coordinates, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

determining that the first ratio is equal to the ratio of the first numerical value to the second numerical value;

The first value is equal to the first amount plus one; the second value is equal to the second amount plus one;

the first number is equal to the number of image frames spaced between the reference image frame and the predicted image frame in the data stream; the second number is equal to the reference image frame and the predicted image frame in the data stream The number of image frames spaced between the predicted image frames.
The electronic device according to claim 81 or 82, characterized in that, in the calculation based on the thirteenth position coordinate or the fifteenth position coordinate, and the first displacement vector of the first scale, the obtained In the sixteenth position coordinate aspect, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute:

If the reference image frame is the first image frame, calculating the sixteenth position coordinate according to the thirteenth position coordinate and the first displacement vector of the first scale;

If the reference image frame is the second image frame, the sixteenth position coordinate is calculated according to the fifteenth position coordinate and the first displacement vector of the first scale.
A method for generating an image frame, characterized in that, applied to an electronic device, the method comprising:

During the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the first drawing result, and when the drawing instruction satisfies the second condition When conditions are met, the drawing result is stored in the eighth storage space as the second drawing result;

generating a seventh image frame according to the first rendering result and the second rendering result;

During the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the third drawing result. When the drawing instruction satisfies the second condition, no drawing is performed;

An eighth image frame is generated according to the third rendering result and the second rendering result.
The method of claim 87, wherein the method further comprises:

In the third drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the fourth drawing result, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space. When the second condition is met, the drawing result is stored in the eighth storage space as a fifth drawing result, and the third drawing cycle is located after the first drawing cycle;

generating a ninth image frame according to the fourth rendering result and the fifth rendering result;

During the fourth drawing cycle, generating a sixth drawing result according to the first drawing result and the fourth drawing result;

The tenth image frame is generated according to the sixth drawing result and the seventh drawing result; the seventh drawing result is the drawing obtained when the drawing instruction of the target application satisfies the second condition in the fifth drawing cycle As a result, the fifth drawing cycle precedes the fourth drawing cycle.
The method according to claim 88, wherein the generating a seventh image frame according to the first rendering result and the second rendering result comprises:

The first drawing result and the second drawing result are stored in the ninth storage space;

The seventh image frame is generated in the ninth storage space according to the first rendering result and the second rendering result.
The method according to claim 89, wherein the generating a ninth image frame according to the fourth rendering result and the fifth rendering result comprises:

storing the fourth drawing result and the fifth drawing result into the ninth storage space;

The ninth image frame is generated in the ninth storage space according to the fourth rendering result and the fifth rendering result.
The method according to claim 88 or 89, wherein the generating the tenth image frame according to the sixth drawing result and the seventh drawing result comprises:

storing the sixth drawing result and the seventh drawing result into the ninth storage space;

The tenth image frame is generated in the ninth storage space according to the sixth rendering result and the seventh rendering result.
The method according to any one of claims 88 to 91, wherein the seventh storage space is composed of a tenth storage space and an eleventh storage space, and the first drawing result is stored in the tenth storage space space, the fourth drawing result is stored in the eleventh storage space, and the generating a sixth drawing result according to the first drawing result and the fourth drawing result includes:

storing the first drawing result in the eleventh storage space;

The sixth drawing result is generated in the eleventh storage space according to the first drawing result and the fourth drawing result.
The method according to any one of claims 88 to 91, wherein the seventh storage space is composed of at least three storage spaces, and the at least three storage spaces include a tenth storage space and an eleventh storage space and the twelfth storage space; the first drawing result is stored in the tenth storage space, the fourth drawing result is stored in the eleventh storage space, and the first drawing result and the fourth drawing result are stored in the eleventh storage space. The drawing result generates a sixth drawing result, including:

storing the first drawing result and the fourth drawing result in the twelfth storage space;

The sixth drawing result is generated in the twelfth storage space according to the first drawing result and the fourth drawing result.
The method according to any one of claims 88 to 93, wherein the third drawing period is adjacent to the fourth drawing period; the fifth drawing period and the third drawing period are the same drawing period period, the fifth drawing result and the seventh drawing result are the same drawing result.
The method according to claim 87, wherein the generating a seventh image frame according to the first rendering result and the second rendering result comprises:

storing the second drawing result in the seventh storage space;

The seventh image frame is generated in the seventh storage space according to the first rendering result and the second rendering result.
The method according to any one of claims 88-95, wherein the electronic device comprises a counting unit; the initialization value of the counting unit is a first value, and the value of the counting unit is updated at the first value every time Switch between the numerical value and the second numerical value once; the counting unit is updated at the moment when the drawing cycle starts;

During a drawing cycle,

If the updated value of the counting unit is the second value, the eighth storage space is emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing result is stored in the first Eight storage spaces;

If the updated value of the counting unit is the first value, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not performed.
The method of claim 87, wherein the method further comprises:

During the third drawing cycle, generating a fourth drawing result according to the first drawing result and the third drawing result;

A ninth image frame is generated according to the second rendering result and the fourth rendering result.
The method according to claim 97, wherein the electronic device comprises a counting unit, the initialization value of the counting unit is a first value, and the counting unit is based on the first value, the second value, and the third value. The sequence performs repeated update switching among the three values, and the counting unit is updated at the moment when the drawing cycle starts,

During a drawing cycle,

If the updated value of the counting unit is the second value, the eighth storage space is emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing result is stored in the first Eight storage spaces;

If the updated value of the counting unit is the first value or the third value, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not executed .
The method according to any one of claims 87 to 98, wherein the ninth storage space is composed of at least two storage spaces, and the at least two storage spaces are based on the target application when the target application is running. The first instruction of the program alternates between the first state and the second state; at the same time point, only one of the at least two storage spaces is in the first state, and the rest of the storage spaces are in the first state. the second state; when the at least two storage spaces are in the first state, the image frames in the at least two storage spaces can be transmitted to a display device for display; the at least two storage spaces are in the first state; In two states, the electronic device can draw in the at least two storage spaces; two image frames generated in two adjacent drawing cycles are stored in two different storage spaces in the ninth storage space, respectively. in space.
The method according to any one of claims 86-99, wherein,

The first condition is: the drawing instruction includes an instruction for enabling depth testing;

The second condition is: the drawing instruction includes an instruction to close the depth test.
The method according to any one of claims 86-100, wherein the time node of any two adjacent drawing cycles is the time point when the target application invokes the second instruction.
An electronic device, characterized in that the electronic device comprises: one or more processors and a memory;

The memory is coupled to the one or more processors for storing computer program code, the computer program code comprising computer instructions that the one or more processors invoke to cause the Electronic equipment performs:

During the first drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the first drawing result, and when the drawing instruction satisfies the second condition When conditions are met, the drawing result is stored in the eighth storage space as the second drawing result;

generating a seventh image frame according to the first rendering result and the second rendering result;

During the second drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the third drawing result. When the drawing instruction satisfies the second condition, no drawing is performed;

An eighth image frame is generated according to the third rendering result and the second rendering result.
The electronic device according to claim 102, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

In the third drawing cycle, according to the drawing instruction of the target application, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space as the fourth drawing result, when the drawing instruction satisfies the first condition, the drawing result is stored in the seventh storage space. When the second condition is met, the drawing result is stored in the eighth storage space as a fifth drawing result, and the third drawing cycle is located after the first drawing cycle;

generating a ninth image frame according to the fourth rendering result and the fifth rendering result;

During the fourth drawing cycle, generating a sixth drawing result according to the first drawing result and the fourth drawing result;

The tenth image frame is generated according to the sixth drawing result and the seventh drawing result; the seventh drawing result is the drawing obtained when the drawing instruction of the target application satisfies the second condition in the fifth drawing cycle As a result, the fifth drawing cycle precedes the fourth drawing cycle.
The electronic device according to claim 103, wherein, in the aspect of generating the seventh image frame according to the first rendering result and the second rendering result, the one or more processors are specifically configured to The computer instructions are invoked to cause the electronic device to perform:

The first drawing result and the second drawing result are stored in the ninth storage space;

The seventh image frame is generated in the ninth storage space according to the first rendering result and the second rendering result.
The electronic device according to claim 104, wherein in the aspect of generating the ninth image frame according to the fourth rendering result and the fifth rendering result, the one or more processors are specifically configured to The computer instructions are invoked to cause the electronic device to perform:

storing the fourth drawing result and the fifth drawing result into the ninth storage space;

The ninth image frame is generated in the ninth storage space according to the fourth rendering result and the fifth rendering result.
The electronic device according to claim 104 or 105, wherein in the aspect of generating the tenth image frame according to the sixth rendering result and the seventh rendering result, the one or more processors are specifically configured to The computer instructions are invoked to cause the electronic device to perform:

storing the sixth drawing result and the seventh drawing result into the ninth storage space;

The tenth image frame is generated in the ninth storage space according to the sixth rendering result and the seventh rendering result.
The electronic device according to any one of claims 103 to 106, wherein the seventh storage space is composed of a tenth storage space and an eleventh storage space, and the first drawing result is stored in the tenth storage space. In the storage space, the fourth drawing result is stored in the eleventh storage space, and in the aspect of generating the sixth drawing result according to the first drawing result and the fourth drawing result, the one or more drawing results are a processor, specifically configured to invoke the computer instructions to cause the electronic device to execute:

storing the first drawing result in the eleventh storage space;

The sixth drawing result is generated in the eleventh storage space according to the first drawing result and the fourth drawing result.
The electronic device according to any one of claims 103 to 106, wherein the seventh storage space is composed of at least three storage spaces, and the at least three storage spaces include a tenth storage space and an eleventh storage space. space and the twelfth storage space; the first drawing result is stored in the tenth storage space, and the fourth drawing result is stored in the eleventh storage space. In the aspect of generating the sixth drawing result from the fourth drawing result, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute:

storing the first drawing result and the fourth drawing result in the twelfth storage space;

The sixth drawing result is generated in the twelfth storage space according to the first drawing result and the fourth drawing result.
The electronic device according to any one of claims 103 to 108, wherein the third drawing period is adjacent to the fourth drawing period; the fifth drawing period and the third drawing period are the same In a drawing cycle, the fifth drawing result and the seventh drawing result are the same drawing result.
The electronic device according to claim 102, wherein, in the aspect of generating the seventh image frame according to the first rendering result and the second rendering result, the one or more processors are specifically configured to The computer instructions are invoked to cause the electronic device to perform:

storing the second drawing result in the seventh storage space;

The seventh image frame is generated in the seventh storage space according to the first rendering result and the second rendering result.
The electronic device according to any one of claims 103-110, wherein the electronic device comprises a counting unit; the initialization value of the counting unit is a first value, and the value of the counting unit is updated every time at the first value. Switch between a value and a second value once; the counting unit is updated at the moment when the drawing cycle starts;

During a drawing cycle,

If the updated value of the counting unit is the second value, the eighth storage space is emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing result is stored in the first Eight storage spaces;

If the updated value of the counting unit is the first value, the second storage unit is not emptied when the drawing cycle starts; and when the drawing instruction satisfies the second condition, the drawing is not performed.
The electronic device according to claim 102, wherein the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute:

During the third drawing cycle, generating a fourth drawing result according to the first drawing result and the third drawing result;

A ninth image frame is generated according to the second rendering result and the fourth rendering result.
The electronic device according to any one of claims 102-112, wherein,

The first condition is: the drawing instruction includes an instruction for enabling depth testing;

The second condition is: the drawing instruction includes an instruction to close the depth test.
A method for image frame prediction, applied to electronic equipment, is characterized in that, comprising:

When drawing the twenty-first drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-first drawing frame according to the first drawing range, and obtains a twenty-first drawing result. the size of the range is greater than the size of the twenty-first drawing frame of the first application;

When drawing the twenty-second drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range to obtain a twenty-second drawing result. The size of the twenty-two memory space is greater than the size of the twenty-second drawing frame, wherein the size of the twenty-first drawing frame is the same as the size of the twenty-second drawing frame;

The electronic device predicts and generates a twenty-third predicted frame of the first application according to the twenty-first rendering result and the twenty-second rendering result, wherein the size of the twenty-third predicted frame is The same size as the twenty-first drawing frame.
The method according to claim 114, wherein the electronic device draws the drawing instruction of the twenty-first drawing frame according to the first drawing range to obtain the twenty-first drawing result, which specifically includes:

The electronic device modifies the first parameter in the first drawing instruction of the twenty-first drawing frame issued by the first application to the first drawing range; the first parameter is used to set the value of the twenty-first drawing frame. drawing range size;

The electronic device draws the modified drawing instruction of the twenty-first drawing frame according to the first drawing range to obtain a twenty-first drawing result.
The method according to claim 115, wherein the size of the first drawing range is larger than the size of the twenty-first drawing frame of the first application, and specifically comprises:

The width of the first drawing range is K3 times the width of the twenty-first drawing frame, the height of the first drawing range is K4 times the height of the twenty-first drawing frame, and the K3. The K4 is greater than 1.
The method according to claim 116, wherein the K3 and the K4 are fixed values configured by a system of the electronic device, or a drawing instruction of the electronic device according to the twenty-first drawing frame The drawing parameters contained in are determined.
The method according to claim 117, wherein the electronic device draws the drawing instruction of the modified twenty-first drawing frame according to the first drawing range to obtain the twenty-first drawing result, which specifically includes:

The electronic device generates a first conversion matrix according to the K3 and the K4, and the electronic device draws the size of the drawing content in the drawing instruction of the modified twenty-first drawing frame to the first conversion matrix after adjustment. In the first drawing range, the twenty-first drawing result is obtained.
The method according to claim 114, wherein the electronic device draws the drawing instruction of the twenty-second drawing frame according to the second drawing range to obtain the twenty-first drawing result, which specifically includes:

The electronic device modifies the second parameter in the second drawing instruction of the twenty-second drawing frame issued by the first application to the second drawing range; the second parameter is used to set the second drawing range of the twenty-second drawing frame. drawing range size;

The electronic device draws the modified drawing instruction of the twenty-second drawing frame according to the second drawing range to obtain a twenty-second drawing result.
The method according to claim 119, wherein the size of the second drawing range is larger than the size of the twenty-second drawing frame of the first application, and specifically comprises:

The width of the second drawing range is K5 times the width of the twenty-second drawing frame, the height of the second drawing range is K6 times the height of the twenty-second drawing frame, and the K5. The K6 is greater than 1.
The method according to claim 120, wherein the K5 and the K6 are fixed values configured by a system of the electronic device, or a drawing instruction of the electronic device according to the twenty-first drawing frame The drawing parameters contained in are determined.
The method according to claim 121, wherein the electronic device draws the drawing instruction of the modified twenty-second drawing frame according to the second drawing range to obtain a twenty-second drawing result, which specifically includes:

The electronic device generates a second conversion matrix according to the K5 and the K6, and the electronic device draws the size of the drawing content in the drawing instruction of the modified twenty-second drawing frame to the second conversion matrix after adjustment. In the second drawing range, the twenty-first drawing result is obtained.
The method according to claim 114, wherein the electronic device predicts and generates a twenty-third predicted frame of the first application according to the twenty-first rendering result and the twenty-second rendering result , including:

The current device predicts and generates a twenty-third drawing result of the twenty-third predicted frame according to the twenty-first drawing result and the twenty-second drawing result;

The electronic device clips the twenty-third rendering result into the twenty-third predicted frame.
The method according to claim 123, wherein the electronic device predicts and generates the twentieth frame of the twenty-third predicted frame according to the twenty-first rendering result and the twenty-second rendering result Three drawing results, including:

The electronic device determines, according to the twenty-first rendering result and the twenty-second rendering result, a first motion vector of the twenty-second rendering result;

The electronic device predicts and generates a twenty-third rendering result of the twenty-third predicted frame according to the twenty-second rendering result and the first motion vector.
The method of claim 124, wherein the electronic device determines the first motion vector of the twenty-second rendering result according to the twenty-first rendering result and the twenty-second rendering result , including:

The electronic device divides the twenty-second rendering result into Q pixel blocks, and the electronic device takes out a first pixel block from the Q pixel blocks of the twenty-second rendering result;

The electronic device determines, in the twenty-first drawing result, a second pixel block that matches the first pixel block;

obtaining, by the electronic device, a motion vector of the first pixel block according to the displacement of the second pixel block to the first pixel block;

The electronic device determines a first motion vector of the twenty-second rendering result according to the motion vector of the first pixel block.
The method according to claim 125, wherein the electronic device determines, in the twenty-first drawing result, a second pixel block that matches the first pixel block, specifically comprising:

The electronic device determines a plurality of candidate pixel blocks in the twenty-first drawing result by using the first pixel points in the first pixel block;

The electronic device respectively calculates the difference between the color values of the plurality of candidate pixel blocks and the first pixel block;

The electronic device determines the second pixel block matching the first pixel block according to the difference between the color values of the first pixel block of the plurality of candidate pixel blocks, and the second pixel block is a multi-pixel block. The candidate pixel block with the smallest difference in color value from the first pixel block among the candidate pixel blocks.
The method according to any one of claims 114-126, wherein, when the twenty-first drawing frame of the first application is drawn, the electronic device converts the drawing instruction of the twenty-first drawing frame to the drawing instruction of the twenty-first drawing frame. Drawing according to the first drawing range, the twenty-first drawing result is obtained, which specifically includes:

When drawing the twenty-first drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-first drawing frame in the twenty-first memory space according to the first drawing range to obtain the twenty-first As a result of drawing, the size of the twenty-first memory space is greater than or equal to the size of the first drawing range.
The method according to any one of claims 114 to 127, wherein, when drawing the 22nd drawing frame of the first application, the electronic device converts the 22nd drawing frame The drawing instruction is drawn according to the second drawing range, and the twenty-second drawing result is obtained, which specifically includes:

When drawing the twenty-second drawing frame of the first application, the electronic device draws the drawing instruction of the twenty-second drawing frame in the twenty-second memory space according to the second drawing range to obtain the second drawing range. Twelve drawing results, the size of the twenty-second memory space is greater than or equal to the size of the second drawing range.
The method of claim 125, wherein the electronic device predicts and generates a twenty-third rendering of the twenty-third predicted frame according to the twenty-second rendering result and the first motion vector Results, including:

The electronic device predicts and generates a twenty-third drawing result according to the third drawing range according to the twenty-second drawing result and the first motion vector; wherein the size of the third drawing range is larger than that of the second drawing range Thirteen predicted frame sizes.
The method according to any one of claims 114-129, wherein, when the twenty-first drawing frame of the first application is drawn, the electronic device converts the drawing instruction of the twenty-first drawing frame to the drawing instruction of the twenty-first drawing frame. After drawing according to the first drawing range and obtaining the twenty-first drawing result, the method further includes:

The electronic device cuts the twenty-first rendering result into the twenty-first rendering frame.
The method according to any one of claims 114-130, wherein, when drawing the 22nd drawing frame of the first application, the electronic device converts the 22nd drawing frame After the drawing instruction is drawn according to the second drawing range, and the twenty-second drawing result is obtained, the method further includes:

The electronic device cuts the twenty-second rendering result into the twenty-second rendering frame.
An electronic device, characterized in that the electronic device comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used for storing computer program codes, the Computer program code comprising computer instructions invoked by the one or more processors to cause the electronic device to perform any of claims 1 to 26, 39 to 49, 59 to 70, 87 to 101, 114 to 131 one of the methods described.
A computer program product comprising instructions, characterized in that, when the computer program product is run on an electronic device, the electronic device is made to perform the operations of claims 1 to 26, 39 to 49, 59 to 70, and 87 to 101 , the method of any one of 114 to 131.
A computer-readable storage medium, comprising instructions, characterized in that, when the instructions are executed on an electronic device, the electronic device is made to perform the operations described in claims 1 to 26, 39 to 49, 59 to 70, and 87 to 101 , the method of any one of 114 to 131.