CN117314727A

CN117314727A - Image processing method, apparatus, device, storage medium, and program product

Info

Publication number: CN117314727A
Application number: CN202311318217.2A
Authority: CN
Inventors: 占克文; 冯晶; 周毅; 杨鲤豪
Original assignee: Glenfly Tech Co Ltd
Current assignee: Glenfly Tech Co Ltd
Priority date: 2023-10-11
Filing date: 2023-10-11
Publication date: 2023-12-29

Abstract

The present application relates to an image processing method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: determining a first target image to be displayed and a second target image to be displayed; acquiring a plurality of first processing instructions aiming at a first target to-be-displayed image and a plurality of second processing instructions aiming at a second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction; the method has the advantages that the target execution sequence of the first processing instructions and the second processing instructions is determined, so that the first processing instructions and the second processing instructions are executed based on the target execution sequence, and the first target to-be-displayed image and the second target to-be-displayed image are sequentially displayed.

Description

Image processing method, apparatus, device, storage medium, and program product

Technical Field

The present invention relates to the field of computer image processing technology, and in particular, to an image processing method, an image processing apparatus, a computer device, a storage medium, and a computer program product.

Background

GPU (Graphics Processing Unit, image processor) real-time rendering of the graphics pipeline is not separated, and developers generally divide the GPU graphics pipeline into four large parts, namely application (geometry processing), geometry processing (geometry processing), rasterization (rasterization) and pixel processing (pixel processing), which have respective pipeline structures, and can collectively refer to shaders required in the geometry processing before rasterization as geometry processing shaders or front-end shaders; the shaders required in the rasterized pixel processing are collectively referred to as a pixel shader (pixel shader) or a back-end shader. The geometry processing shader may include, among other things, vertex shader, subdivision control shader, subdivision computation shader, geometry shader, and the like.

With the rapid development of GPU image rendering technology, the processing instructions for displaying images are also becoming more and more complex, so that the number of ts# (shader resource and sampler, registers for shader resources and sampling) required by shaders in image rendering is in a trend of increasing.

In the conventional technology, the execution speed of the sampling instruction is increased only by setting a buffer to store the resource information corresponding to the TS# required by the shader.

However, since the processing instruction to be executed in the image display process is only acquired through one buffer area, it is necessary to wait until the processing instruction in one image is completely processed, and then the execution of the processing instruction of the next image can be started, which is time-consuming.

Disclosure of Invention

In view of the foregoing, it is desirable to provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can accelerate the image display speed, shorten the time required for image display, and improve the image display efficiency.

In a first aspect, the present application provides an image processing method. The method comprises the following steps:

determining a first target image to be displayed and a second target image to be displayed under the condition that the existence of the images to be displayed is detected;

acquiring a plurality of first processing instructions aiming at the first target to-be-displayed image and a plurality of second processing instructions aiming at the second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction;

determining a target execution order for the plurality of first processing instructions and the plurality of second processing instructions; the target execution sequence is characterized in that prefetch instructions belonging to different types and sampling instructions are executed in parallel;

And executing a plurality of first processing instructions and a plurality of second processing instructions based on the target execution sequence to sequentially display a first target to-be-displayed image and a second target to-be-displayed image.

In one embodiment, the executing the first processing instructions and the second processing instructions to sequentially display the first target to-be-displayed image and the second target to-be-displayed image based on the target execution order includes:

executing a plurality of first processing instructions based on the target execution sequence to sequentially acquire first geometric texture data and first pixel texture data for the first target image to be displayed;

displaying the first target image to be displayed according to the first geometric texture data and the second pixel texture data;

executing a plurality of second processing instructions to sequentially acquire second geometric texture data and second pixel texture data for the second target image to be displayed;

and displaying the second target to-be-displayed image based on the second geometric texture data and the second pixel texture data.

In one embodiment, the plurality of first processing instructions includes a first geometry prefetch instruction, a first geometry sample instruction, a first pixel prefetch instruction, and a first pixel sample instruction; the executing, based on the target execution order, a plurality of first processing instructions to sequentially obtain first geometric texture data and first pixel texture data for an image to be displayed of the first target, including:

Determining whether corresponding geometric data exists in a geometric cache region or not based on the first geometric prefetch instruction;

under the condition that the geometric type buffer area does not have corresponding geometric data, acquiring the corresponding geometric data from the second buffer area and writing the corresponding geometric data into the geometric type buffer area;

triggering and executing the first geometric sampling instruction and the first pixel prefetching instruction under the condition that corresponding geometric data exists in the geometric class cache region;

acquiring first geometric texture data based on the first geometric sampling instruction;

determining whether corresponding pixel data exists in a pixel type buffer area or not based on the first pixel prefetching instruction;

under the condition that the pixel type buffer area does not have corresponding pixel data, corresponding pixel data are obtained from the second buffer area and written into the pixel type buffer area;

under the condition that corresponding pixel data exists in the pixel type buffer area and the execution of the first geometric sampling instruction is completed, triggering and executing the first pixel sampling instruction and triggering and executing a geometric pre-fetching instruction of a next image to be displayed;

and acquiring first pixel texture data based on the first pixel sampling instruction.

In one embodiment, the method further comprises:

acquiring a currently running application program;

determining the proportion of a geometric type buffer area and a pixel type buffer area in a first buffer area based on the application program;

and adjusting the sizes of the geometric type buffer area and the pixel type buffer area based on the proportion of the geometric type buffer area and the pixel type buffer area in the first buffer area.

In one embodiment, the acquiring, based on the first geometry sampling instruction, first geometry texture data includes:

based on the first geometric sampling instruction, corresponding geometric data are obtained;

according to the geometric data, performing texture sampling processing to obtain first preliminary geometric texture data after the texture sampling processing;

and performing texture filtering processing on the first preliminary geometric texture data to obtain first geometric texture data.

In one embodiment, the performing texture sampling processing according to the geometric data to obtain first preliminary geometric texture data after the texture sampling processing includes:

determining whether corresponding preset texture data exists in a texture type cache region according to the geometric data and the first geometric sampling instruction;

writing the corresponding preset texture data of the second cache region into the texture type cache region under the condition that the corresponding preset texture data does not exist in the texture type cache region;

And taking the corresponding preset texture data acquired from the texture type buffer as first preliminary geometric texture data.

In one embodiment, the determining the first target to be displayed image and the second target to be displayed image includes:

and acquiring the display sequence of the plurality of images to be displayed, taking the first image to be displayed in the display sequence as a first target image to be displayed, and taking the next image to be displayed in the display sequence as a second target image to be displayed.

In a second aspect, the present application also provides an image processing apparatus. The device comprises:

the target to-be-displayed image determining module is used for determining a first target to-be-displayed image and a second target to-be-displayed image under the condition that the existence of the to-be-displayed image is detected;

a plurality of instruction acquisition modules, configured to acquire a plurality of first processing instructions for the first target to-be-displayed image and a plurality of second processing instructions for the second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction;

a target execution order determination module for determining a target execution order for the plurality of first processing instructions and the plurality of second processing instructions; the target execution sequence is characterized in that prefetch instructions belonging to different types and sampling instructions are executed in parallel;

And the image display module to be displayed is used for sequentially displaying the first target image to be displayed and the second target image to be displayed based on the target execution sequence.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

The image processing method, the image processing device, the computer equipment, the storage medium and the computer program product, and determine a first target image to be displayed and a second target image to be displayed when the existence of the image to be displayed is detected; acquiring a plurality of first processing instructions aiming at a first target to-be-displayed image and a plurality of second processing instructions aiming at a second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction; the method is used for determining target execution sequences of the first processing instructions and the second processing instructions, executing the first processing instructions and the second processing instructions based on the target execution sequences, and sequentially displaying the first target to-be-displayed image and the second target to-be-displayed image.

Drawings

FIG. 1 is a schematic diagram of a pipeline in which two images show instructions that need to be executed in the prior art;

FIG. 2 is a flow chart of an image processing method in one embodiment;

FIG. 3 is a flow diagram of the acquisition of geometric and pixel texture data in one embodiment;

FIG. 4 is a block diagram of the architecture of image processing in one embodiment;

FIG. 5 is a schematic diagram of a pipeline in which two target images show instructions that need to be executed, in one embodiment;

FIG. 6 is a flow chart of an image processing method according to another embodiment;

FIG. 7 is a block diagram showing the structure of an image processing apparatus in one embodiment;

fig. 8 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

With the rapid development of GPU image rendering technology, the more complex the sampling instruction that causes an image to be displayed is, the more the number of ts# (shader resource and sampler, registers for shader resources and sampling) required by a shader in image rendering is in a rising trend.

In the conventional technology, the execution speed of the sampling instruction is increased by setting a buffer to store the resource information corresponding to the ts# required by the shader. The display of an image requires execution of a plurality of processing instructions including a geometry prefetch instruction, a geometry sample instruction, a pixel prefetch instruction, and a pixel sample instruction. In one image, the execution sequence of the plurality of processing instructions is a geometry prefetch instruction, a geometry sampling instruction, a pixel prefetch instruction and a pixel sampling instruction, and the next instruction can be pointed after the execution of the previous instruction is completed. Specifically, as shown in fig. 1, a schematic diagram of a pipeline of instructions to be executed is shown for two image displays in the conventional art.

Then, in the conventional technique, it is necessary to wait until the processing instruction in one image is completely processed, and then start the execution of the processing instruction of the next image, which results in a relatively long time required for displaying the images.

In order to solve this problem, an image processing method of the present embodiment is proposed, and as shown in fig. 2, an image processing method is provided, and this embodiment is exemplified by the application of the method to a terminal, it will be understood that the method may also be applied to a server, and also to a system including a terminal and a server, including the steps of:

Step 202, determining a first target image to be displayed and a second target image to be displayed when the existence of the images to be displayed is detected.

The terminal may include, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and the like. The server may be implemented as a stand-alone server or as a server cluster composed of a plurality of servers.

The image to be displayed is an image which needs to be displayed on a display interface of the terminal. The first target to-be-displayed image and the second target to-be-displayed image are images to be displayed on a display interface of the terminal. The display order of the first target to-be-displayed image is better than the display order of the second target to-be-displayed image.

In specific practice, when the terminal detects that the image to be displayed exists, corresponding first target image to be displayed and second target image to be displayed can be obtained according to the detection sequence of the image detected by the terminal.

Step 204, acquiring a plurality of first processing instructions for the first target to-be-displayed image and a plurality of second processing instructions for the second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sample instruction, and a pixel type prefetch instruction and a sample instruction.

The plurality of first processing instructions may be processing instructions that need to be executed to display the first target image to be displayed. The plurality of second processing instructions may be processing instructions that need to be executed to display the second target image to be displayed. The plurality of first processing instructions and the plurality of second processing instructions may each include a geometry type prefetch instruction and a sample instruction, and a pixel type prefetch instruction and a sample instruction. The method comprises the steps that a geometric type pre-fetching instruction and a sampling instruction are used for obtaining geometric data of a first target image to be displayed; the pixel type pre-fetch instruction and the sampling instruction are used for obtaining pixel data of an image to be displayed aiming at the first target.

In an exemplary case where the first target to-be-displayed image and the second target to-be-displayed image are determined, a plurality of first processing instructions for the first target to-be-displayed image and a plurality of second processing instructions for the second target to-be-displayed image are acquired.

Step 206, determining a target execution sequence for the plurality of first processing instructions and the plurality of second processing instructions; the prefetch instructions belonging to different types in the target execution sequence are executed in parallel with the sampling instructions.

Wherein, for the first target image to be displayed, the execution order of the same type of prefetch instruction is better than the execution order of the sampling instruction. The execution order of the geometry type sampling instructions is better than the execution order of the pixel type sampling instructions.

The target execution order is an execution order derived based on the plurality of first processing instructions and the plurality of second processing instructions. The target execution order may include prefetch instructions belonging to different types being executed in parallel with the sample instructions. In particular, sampling instructions of the geometric type and prefetch instructions of the pixel type may be executed in parallel for the same image.

In specific practice, the terminal may include a processor, where the processor identifies that an image to be displayed exists, determines a first target image to be displayed and a second target image to be displayed, acquires a plurality of first processing instructions for the first target image to be displayed and a plurality of second processing instructions for the second target image to be displayed, and determines a target execution order based on the plurality of first processing instructions and the plurality of second processing instructions. Specifically, the first processing instruction includes an image identifier and an instruction identifier of a different first processing instruction. The image identification may be used to distinguish between images currently to be displayed. The instruction identification may be used to identify the type of data that the instruction needs to acquire.

In the case of acquiring a plurality of first processing instructions for a first target image to be displayed and a plurality of second processing instructions for a second target image to be displayed, the target execution order is determined based on the plurality of first processing instructions and the plurality of second processing instructions.

Step 208, executing a plurality of first processing instructions and a plurality of second processing instructions based on the target execution sequence, so as to sequentially display a first target to-be-displayed image and a second target to-be-displayed image.

The first target to-be-displayed image is displayed on a display interface of the terminal before the second target to-be-displayed image, namely after the first target to-be-displayed image is completely displayed, the second target to-be-displayed image is triggered to be displayed.

In the image processing method, under the condition that the existence of the image to be displayed is detected, the first target image to be displayed and the second target image to be displayed are determined; acquiring a plurality of first processing instructions aiming at a first target to-be-displayed image and a plurality of second processing instructions aiming at a second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction; the method is used for determining target execution sequences of the first processing instructions and the second processing instructions, executing the first processing instructions and the second processing instructions based on the target execution sequences, and sequentially displaying the first target to-be-displayed image and the second target to-be-displayed image.

In one embodiment, step 202 includes:

The image to be displayed is an image which is detected by the terminal and needs to be displayed on the display interface. The display order of the plurality of images to be displayed may be determined based on the detection order in which the terminal detected the images. The first target image to be displayed may be the first image to be displayed in the display order. The second target image to be displayed may be an ordered next image to be displayed in display order.

In specific practice, a processor in the terminal can be used for detecting whether an image needing to be displayed on a display interface exists or not, and determining that an image to be displayed exists under the condition that the image needing to be displayed on the display interface exists, and acquiring image information of the image to be displayed so as to determine the display sequence of a plurality of images to be displayed.

For example, a display order of the plurality of images to be displayed may be acquired, so that the first image to be displayed in the display order is taken as a first target image to be displayed, and the next image to be displayed in the display order is taken as a second target image to be displayed, based on the display order of the plurality of images to be displayed. In specific practice, the display order of the plurality of images to be displayed may be determined based on the detection order in which the processor in the terminal detects the images, thereby determining the first target image to be displayed and the second target image to be displayed.

In the above embodiment, based on the display sequence of the plurality of images to be displayed, the first target image to be displayed and the second target image to be displayed are determined, so that the display sequence of the first target image to be displayed is earlier than that of the second target image to be displayed.

In one embodiment, step 208 includes:

step 2082, based on the target execution sequence, executing a plurality of first processing instructions to sequentially acquire first geometric texture data and first pixel texture data for the first target to-be-displayed image.

Wherein the plurality of first processing instructions includes a geometry type prefetch instruction and a sample instruction, and a pixel type prefetch instruction and a sample instruction. Specifically, the first geometric texture data of the image to be displayed aiming at the first target can be acquired based on the pre-fetching instruction and the sampling instruction of the geometric type; the pre-fetch instruction and the sample instruction based on pixel type may obtain first pixel texture data for a first target image to be displayed.

Step 2084, displaying the first target image to be displayed according to the first geometric texture data and the second pixel texture data.

In specific practice, the display rendering process of the image includes a geometry process, a rasterization process, and a pixel process. Taking the first target image to be displayed as an example, the geometric processing process is that the first geometric texture data aiming at the first target image to be displayed can be obtained through a geometric type pre-fetching instruction and a sampling instruction. After the first geometric texture data is obtained, rasterization is required, and then the acquisition of the first pixel texture data is realized so as to display a first target image to be displayed.

Step 2086 executes a plurality of second processing instructions to sequentially obtain second geometric texture data and second pixel texture data for the second target image to be displayed.

The plurality of second processing instructions may include a geometry type prefetch instruction and a sample instruction, and a pixel type prefetch instruction and a sample instruction, among others. Specifically, the second geometric texture data for the image to be displayed of the second target may be acquired based on the geometry type prefetch instruction and the sampling instruction; the pre-fetch instruction and the sample instruction based on pixel type may obtain second pixel texture data for the second target image to be displayed.

Step 2088, displaying the second target image to be displayed based on the second geometric texture data and the second pixel texture data.

Specifically, the displaying process of the second target to-be-displayed image may refer to the displaying process of the first target to-be-displayed image, and the disclosure is not repeated herein.

In one embodiment, referring to FIG. 3, a flow diagram of the acquisition of geometric and pixel texture data in one embodiment is shown. Wherein the plurality of first processing instructions includes a first geometry prefetch instruction, a first geometry sampling instruction, a first pixel prefetch instruction, and a first pixel sampling instruction; step 2082 includes:

Step 302, determining whether there is corresponding geometric data in the geometric cache area based on the first geometric prefetch instruction.

The plurality of first processing instructions for the first target image to be displayed may include a first geometry prefetch instruction, a first geometry sampling instruction, a first pixel prefetch instruction, and a first pixel sampling instruction. The first geometry prefetch instruction may include identification information of geometry data that needs to be fetched. The geometry class cache may be used to cache geometry data.

The geometry data may be geometry processing resource data recorded by a TS# (shader resource and sampler, register for shader resources and samples) corresponding to the geometry processing shader. The geometry data may include resource and sample description information corresponding to the geometry processing shader. The resource and sample description information corresponding to the geometry processing shader includes identification information for identifying the corresponding geometry data. The resource and sample description information may be used to describe the characteristic information and address information of the object to be sampled.

In specific practice, whether all geometric data needed by the first geometric prefetch instruction exist in the geometric cache area can be determined through corresponding hit test (hit miss test) according to the identification information of the geometric data needed by the first geometric prefetch instruction.

Step 304, when the geometric cache area does not have the corresponding geometric data, the corresponding geometric data is obtained from the second cache area, and written into the geometric cache area.

In particular practice, the terminal may include a first buffer and a second buffer. The first buffer area is composed of a geometric buffer area and a pixel buffer area. The second buffer may be configured to store geometry processing resource data recorded by a ts# (shader resource and sampler, register for shader resources and samples) corresponding to the geometry processing shader, so that in the case that the geometry class buffer in the first buffer does not have all the geometry data required by the first geometry prefetch instruction, the corresponding geometry data is obtained from the second buffer.

In the case where the geometric class buffer does not have corresponding geometric data, the corresponding geometric data is acquired from the second buffer and written into the geometric class buffer.

And step 306, triggering to execute the first geometric sampling instruction and the first pixel prefetching instruction under the condition that corresponding geometric data exists in the geometric class cache region.

The terminal may include a controller for identifying whether the execution of the first geometric prefetch instruction is complete. The completion of execution of the first geometry prefetch instruction is marked by detecting that all of the geometry data required by the first geometry prefetch instruction is present in the geometry class cache. Specifically, based on the first geometry pre-fetching instruction, determining whether all geometry data required by the first geometry pre-fetching instruction exists in the geometry type cache region, and if all geometry data required by the first geometry pre-fetching instruction exists in the geometry type cache region, completing execution of the corresponding first geometry pre-fetching instruction; if the geometric type buffer area does not have all geometric data needed by the first geometric prefetch instruction, acquiring corresponding geometric data from the second buffer area, and writing the corresponding geometric data into the geometric type buffer area, so that the geometric type buffer area has all geometric data needed by the first geometric prefetch instruction, and determining that the execution of the corresponding first geometric prefetch instruction is completed.

Specifically, when the corresponding geometric data exists in the geometric cache region, that is, after the execution of the first geometric prefetch instruction is completed, the execution of the first geometric sampling instruction and the first pixel prefetch instruction is triggered. Specifically, the execution priority of the first geometry prefetch instruction is higher than the execution priority of the first geometry sample instruction; the first pixel prefetch instruction has an execution priority that is higher than the execution priority of the first pixel sample instruction.

Step 308, acquiring first geometric texture data based on the first geometric sampling instruction.

Illustratively, based on the first geometry sampling instruction, corresponding geometry data is acquired, and based on the corresponding geometry data, first geometry texture data for the first target image to be displayed is acquired.

Step 310, determining whether there is corresponding pixel data in the pixel buffer based on the first pixel prefetch instruction.

The first pixel prefetch instruction may include identification information of pixel data to be acquired. The pixel class buffer may be used to buffer pixel data.

The pixel data may be pixel processing resource data recorded for a TS# (shader resource and sampler, register for shader resources and samples) corresponding to the pixel shader. The pixel data may include corresponding resource and sample description information for the pixel shader. The resource and sample description information corresponding to the pixel shader includes identification information for identifying the corresponding pixel data.

In specific practice, whether all pixel data required by the first pixel prefetch instruction exist in the pixel type cache area can be determined through a corresponding hit test (hit miss test) according to the identification information of the pixel data required to be acquired in the first pixel prefetch instruction.

Step 312, in the case that the pixel type buffer area does not have the corresponding pixel data, obtaining the corresponding pixel data from the second buffer area, and writing the corresponding pixel data into the pixel type buffer area.

The first buffer area may include a pixel type buffer area. The second buffer may also be configured to store pixel processing resource data recorded by a ts# (shader resource and sampler, register for shader resources and samples) corresponding to the pixel shader, so as to obtain corresponding pixel data from the second buffer if the pixel class buffer in the first buffer does not have corresponding pixel data.

In step 314, under the condition that the pixel buffer has corresponding pixel data and the execution of the first geometry sampling instruction is completed, the execution of the first pixel sampling instruction and the execution of the geometry pre-fetching instruction of the next image to be displayed are triggered.

The mark that the execution of the first pixel prefetch instruction is completed is that all pixel data needed by the first pixel prefetch instruction is detected to exist in the pixel type cache area. Specifically, based on the first pixel prefetch instruction, determining whether all pixel data required by the first pixel prefetch instruction exists in the pixel type buffer area, and if all pixel data required by the first pixel prefetch instruction exists in the pixel type buffer area, completing execution of the corresponding first pixel prefetch instruction; if the pixel type buffer area does not have all the pixel data needed by the first pixel pre-fetching instruction, acquiring corresponding pixel data from the second buffer area, and writing the corresponding pixel data into the pixel type buffer area, so that the corresponding pixel data exists in the pixel type buffer area, and determining that the execution of the corresponding first pixel pre-fetching instruction is completed.

The first geometry sampling instruction execution completion flag is to obtain first geometry texture data.

In the case that the execution of the first pixel prefetch instruction is completed and the execution of the first geometry sampling instruction is completed, the execution of the first pixel sampling instruction and the execution of the geometry prefetch instruction of the next image to be displayed may be triggered.

The next image to be displayed of the first target image to be displayed is the second target image to be displayed. The plurality of second processing instructions for the second target image to be displayed may include a second geometry prefetch instruction, a second geometry sample instruction, a second pixel prefetch instruction, and a second pixel sample instruction.

Specifically, the execution sequence of the plurality of first processing instructions and the plurality of second processing instructions in the target execution sequence may be execution of a first geometry prefetch instruction, and after the execution of the first geometry prefetch instruction is completed, triggering the execution of the first geometry sampling instruction and the first pixel prefetch instruction; triggering and executing the first pixel sampling instruction and the second geometric prefetching instruction after the first geometric sampling instruction and the first pixel prefetching instruction are executed; triggering and executing the second geometric sampling instruction and the second pixel prefetching instruction after the first pixel sampling instruction and the second geometric prefetching instruction are executed; and triggering and executing the second pixel sampling instruction and the geometric prefetching instruction of the next image to be displayed after the execution of the second geometric sampling instruction and the second pixel prefetching instruction are completed.

Step 316, acquiring first pixel texture data based on the first pixel sampling instruction.

Illustratively, based on the first pixel sampling instruction, corresponding pixel data is acquired, and based on the corresponding pixel data, first pixel texture data for the first target image to be displayed is acquired.

In the embodiment, the time delay of acquiring geometric data or pixel data is reduced by prefetching the instruction; by setting different types of buffer areas, the parallel execution of the first geometric sampling instruction and the first pixel prefetching instruction is triggered, and the parallel execution of the geometric prefetching instruction of the first pixel sampling instruction and the next image is triggered, so that the image display speed can be increased, the time required by image display is shortened, and the image display efficiency is improved.

In one embodiment, the method further comprises:

acquiring a currently running application program;

The processor in the terminal can identify the currently running application program to determine the proportion of the geometric type buffer area and the pixel type buffer area in the first buffer area based on the application program.

In particular practice, there may be different amounts of geometric data and pixel data that are needed for the image to be displayed, as there may be different applications. For example, different kinds of application programs, or for the same application program, such as different stages of a game, the driver of the GPU can regulate and control the proportion of the geometric buffer area and the pixel buffer area in the first buffer area under the condition of identifying the application program, so that the buffer amount of the buffered geometric data or pixel data is increased, and the acquisition of the geometric data or pixel data is more convenient. Therefore, the ratio of the geometric class buffer and the pixel class buffer in the first buffer can be divided based on the application characteristics of the application program.

In specific practice, a corresponding relation between a plurality of groups of application programs and preset proportions of the geometric type buffer areas and the pixel type buffer areas in the first buffer area can be set, so that the sizes of the geometric type buffer areas and the pixel type buffer areas in the first buffer area are adjusted based on the application programs.

In the above embodiment, based on the application program, the ratio of the geometric class buffer area and the pixel class buffer area in the first buffer area is determined; based on the proportion of the geometric type buffer area and the pixel type buffer area in the first buffer area, the sizes of the geometric type buffer area and the pixel type buffer area are adjusted, so that the sizes of the buffer areas of corresponding types are adjusted in real time, and the distribution of geometric data and pixel data of images required to be displayed by an actual application program is more fit.

In one embodiment, step 308 comprises:

step 3082, obtaining corresponding geometric data based on the first geometric sampling instruction.

Wherein, the geometry data may include resource description information and sampling description information corresponding to the geometry processing shader. The sampling description information may be a sampling mode corresponding to the sampling instruction.

In specific practice, the resource description information may be used to describe the characteristic information and address information of the object to be sampled. The sampling mode corresponding to the sampling instruction may include, but is not limited to, point sampling or plane sampling.

In step 3084, texture sampling is performed according to the geometric data, so as to obtain first preliminary geometric texture data after the texture sampling.

Step 3086, performing texture filtering processing on the first preliminary geometric texture data to obtain first geometric texture data.

Illustratively, based on the first geometric sampling instruction, corresponding geometric data is obtained from the geometric class buffer area, so as to perform texture sampling processing according to the geometric data, and obtain first preliminary geometric texture data.

In specific practice, based on geometric data, preset texture data corresponding to the geometric data are obtained from a geometric cache region, so that first preliminary geometric texture data are obtained, and in order to enable the texture data in the first preliminary geometric texture data to be more fit with the texture of an image to be displayed actually, texture filtering processing is required to be carried out on the first preliminary geometric texture data, so that first geometric texture data are obtained.

In the above embodiment, based on the first geometry sampling instruction, corresponding geometry data is obtained, texture sampling processing is performed according to the geometry data, first preliminary geometry texture data after the texture sampling processing is obtained, and texture filtering processing is performed on the first preliminary geometry texture data, so as to obtain first geometry texture data. And performing texture filtering processing on the first preliminary geometric texture data to obtain first geometric texture data, so that the texture data is more attached to the textures of the image to be displayed actually.

In one embodiment, the acquiring of the first pixel texture data may be based on the first pixel sampling instruction, so as to acquire corresponding pixel data from the pixel buffer area, perform texture sampling processing according to the pixel data to obtain first preliminary pixel texture data, and perform texture filtering processing on the first preliminary pixel texture data to obtain the first pixel texture data.

Specifically, for a specific operation of acquiring the first pixel texture data, refer to the process of acquiring the first geometric texture data in step 3082-step 3086, and the present invention is not repeated here.

In one embodiment, step 3084 includes:

The second buffer area may also be used to store preset texture data corresponding to the geometric data and preset texture data corresponding to the pixel data.

The first preliminary geometric texture data obtaining process is to write the corresponding preset texture data of the second buffer into the texture type buffer under the condition that the corresponding preset texture data does not exist in the texture type buffer; and taking the corresponding preset texture data acquired from the texture type buffer as first preliminary geometric texture data.

In specific practice, according to the geometric data and the first geometric sampling instruction, the required texture data and the position information of the required texture data are determined, so as to determine whether the corresponding preset texture data exists in the texture type buffer according to the required texture data and the position information of the required texture data.

Specifically, the process of acquiring the first preliminary pixel texture data may refer to the process of acquiring the first preliminary geometric texture data, which is not repeated herein.

In the above embodiment, based on the geometric data and the first geometric sampling instruction, it is determined whether the texture buffer has corresponding preset texture data, and if the texture buffer does not have corresponding preset texture data, the corresponding preset texture data of the second buffer is written into the texture buffer, so as to facilitate the subsequent acquisition of the corresponding preset texture data from the texture buffer.

For a better understanding of the image processing process, an example is illustrated, referring to FIG. 4, which shows a block diagram of the image processing in one embodiment, comprising:

an execution unit 402, configured to obtain a currently running application program; determining the proportion of a geometric type buffer area and a pixel type buffer area in a first buffer area based on an application program; based on the proportion of the geometric type buffer area and the pixel type buffer area in the first buffer area, the sizes of the geometric type buffer area and the pixel type buffer area are adjusted; determining a first target image to be displayed and a second target image to be displayed under the condition that the existence of the images to be displayed is detected; the method comprises the steps of acquiring a plurality of first processing instructions aiming at a first target to-be-displayed image and a plurality of second processing instructions aiming at a second target to-be-displayed image, determining a target execution sequence of the processing instructions, and executing the plurality of first processing instructions and the plurality of second processing instructions based on the target execution sequence.

A first boundary detector (barrier detector) 404 is configured to monitor whether execution of the geometric prefetch instruction of the image is completed, and after it is monitored that execution of the geometric prefetch instruction of the image is completed, feedback execution of the geometric prefetch instruction to the execution unit 402, so that the execution unit 402 sends a corresponding geometric sampling instruction.

A second boundary detector (barrier detector) 406 is configured to monitor whether the execution of the pixel prefetch instruction of the image is completed, and after detecting that the execution of the pixel prefetch instruction of the image is completed, feed back the execution of the pixel prefetch instruction to the execution unit 402, so that the execution unit 402 sends a corresponding pixel sampling instruction.

A first hit test 408 determines if there is corresponding geometry data in the geometry class cache.

A second hit test 410 is used to determine if the pixel class buffer has corresponding pixel data.

The first buffer 412 (not shown in fig. 4) includes a geometry class buffer 4122 and a pixel class buffer 4124.

The geometry class buffer 4122 is used for storing a part of geometry data, and obtaining geometry data from the second buffer.

The pixel type buffer 4124 is used for storing a part of pixel data, and acquiring the pixel data from the second buffer.

The second buffer 414 is used for storing geometric data, pixel data and corresponding preset texture data.

The texture sampling module 416 is configured to identify whether the geometric data or the pixel data has corresponding preset texture data in the texture class buffer.

The texture class buffer 418 is used for storing geometric data, pixel data and corresponding preset texture data.

The texture filtering processing module 420 is configured to perform filtering processing on the obtained preset texture data.

In specific practice, the execution unit 402 in the terminal may be a processor and the first and second boundary detectors may be controllers.

For a better understanding of the image processing process, an example is described with reference to FIG. 5, which shows a schematic diagram of a pipeline in which two target images display instructions that need to be executed, in one embodiment, comprising:

the first geometric prefetch instruction, the first geometric sampling instruction, the first pixel prefetch instruction and the first pixel sampling instruction are processing instructions aiming at the first target to-be-displayed image. The second geometry pre-fetch instruction, the second geometry sampling instruction, the second pixel pre-fetch instruction, and the second pixel sampling instruction are processing instructions for the second target image to be displayed. The next image to be displayed is the image to be displayed after the second target image to be displayed is displayed.

The target execution sequence may be to execute a first geometry prefetch instruction, and trigger to execute a first geometry sampling instruction and a first pixel prefetch instruction after the execution of the first geometry prefetch instruction is completed; triggering and executing the first pixel sampling instruction and the second geometric prefetching instruction after the first geometric sampling instruction and the first pixel prefetching instruction are executed; triggering and executing the second geometric sampling instruction and the second pixel prefetching instruction after the first pixel sampling instruction and the second geometric prefetching instruction are executed; and triggering and executing the second pixel sampling instruction and the geometric prefetching instruction of the next image to be displayed after the execution of the second geometric sampling instruction and the second pixel prefetching instruction are completed.

In particular practice, the same type of prefetch instruction has a higher priority than the sample instruction for the same image, and the geometry type of sample instruction has a higher priority than the pixel type of sample instruction. And, in general, the prefetch instruction takes less time to execute than the sample instruction.

For a better understanding of the image processing process, an example is described, referring to fig. 6, which shows a flow diagram of an image processing method in one embodiment, comprising:

step 602, when it is detected that there are images to be displayed, a display sequence of a plurality of images to be displayed is acquired, the first image to be displayed in the display sequence is taken as a first target image to be displayed, and the next image to be displayed in the display sequence is taken as a second target image to be displayed.

Step 604, a plurality of first processing instructions for a first target to-be-displayed image and a plurality of second processing instructions for a second target to-be-displayed image are acquired.

Step 606 determines a target execution order for the plurality of first processing instructions and the plurality of second processing instructions.

Step 608, based on the target execution sequence, executing the first geometry pre-fetch instruction to determine whether the geometry class cache has corresponding geometry data; and under the condition that the geometric type buffer area does not have corresponding geometric data, acquiring the corresponding geometric data from the second buffer area, and writing the corresponding geometric data into the geometric type buffer area.

Step 610, triggering to execute a first geometry sampling instruction and a first pixel prefetching instruction in case that corresponding geometry data exists in the geometry class buffer;

step 612, obtaining first geometric texture data based on the first geometric sampling instruction.

Step 614, determining whether there is corresponding pixel data in the pixel buffer based on the first pixel prefetch instruction; and under the condition that the corresponding pixel data does not exist in the pixel type buffer area, acquiring the corresponding pixel data from the second buffer area, and writing the corresponding pixel data into the pixel type buffer area.

In step 616, when the pixel buffer has corresponding pixel data and the execution of the first geometry sampling instruction is completed, the execution of the first pixel sampling instruction and the execution of the geometry pre-fetching instruction of the next image to be displayed are triggered.

In step 618, based on the first pixel sampling instruction, the first pixel texture data is acquired to display the second target image to be displayed.

In this embodiment, when it is detected that an image to be displayed exists, a first target image to be displayed and a second target image to be displayed are determined; acquiring a plurality of first processing instructions aiming at a first target to-be-displayed image and a plurality of second processing instructions aiming at a second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sampling instruction, and a pixel type prefetch instruction and a sampling instruction; the method is used for determining target execution sequences of the first processing instructions and the second processing instructions, executing the first processing instructions and the second processing instructions based on the target execution sequences, and sequentially displaying the first target to-be-displayed image and the second target to-be-displayed image.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus for implementing the above-mentioned image processing method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the image processing apparatus provided below may refer to the limitation of the image processing method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 7, there is provided an image processing apparatus including: a target to-be-displayed image determination module 702, a plurality of instruction acquisition modules 704, a target execution order determination module 706, and a to-be-displayed image display module 708, wherein:

the target to-be-displayed image determining module 702 is configured to determine, when the presence of the to-be-displayed image is detected, a first target to-be-displayed image and a second target to-be-displayed image.

A plurality of instruction acquisition modules 704, configured to acquire a plurality of first processing instructions for the first target to-be-displayed image and a plurality of second processing instructions for the second target to-be-displayed image; the plurality of first processing instructions and the plurality of second processing instructions each include a geometry type prefetch instruction and a sample instruction, and a pixel type prefetch instruction and a sample instruction.

A target execution order determination module 706 for determining a target execution order for the plurality of first processing instructions and the plurality of second processing instructions; the prefetch instructions belonging to different types in the target execution sequence are executed in parallel with the sampling instructions.

The to-be-displayed image display module 708 is configured to sequentially display a first target to-be-displayed image and a second target to-be-displayed image based on the target execution order.

In some embodiments, the image display module 708 to be displayed includes:

a first geometric and pixel texture data acquisition sub-module for executing a plurality of first processing instructions based on the target execution order to sequentially acquire first geometric texture data and first pixel texture data for an image to be displayed of the first target;

the first target image to be displayed display sub-module is used for displaying the first target image to be displayed according to the first geometric texture data and the second pixel texture data;

a second geometric and pixel texture data acquisition sub-module for executing a plurality of second processing instructions to sequentially acquire second geometric texture data and second pixel texture data for the second target image to be displayed;

and the second target image to be displayed display sub-module is used for displaying the second target image to be displayed based on the second geometric texture data and the second pixel texture data.

In some embodiments, the plurality of first processing instructions includes a first geometry prefetch instruction, a first geometry sample instruction, a first pixel prefetch instruction, and a first pixel sample instruction; a first geometry and pixel texture data acquisition sub-module comprising:

A geometric data determining unit, configured to determine, based on the first geometric prefetch instruction, whether corresponding geometric data exists in a geometric cache area;

the geometric data acquisition unit is used for acquiring corresponding geometric data from the second cache region and writing the corresponding geometric data into the geometric cache region under the condition that the geometric cache region does not have the corresponding geometric data;

the first instruction triggering unit is used for triggering and executing the first geometric sampling instruction and the first pixel prefetching instruction under the condition that corresponding geometric data exists in the geometric class cache region;

the first geometric texture data acquisition unit is used for acquiring first geometric texture data based on the first geometric sampling instruction;

a pixel data determining unit, configured to determine, based on the first pixel prefetch instruction, whether corresponding pixel data exists in a pixel buffer area;

the pixel data acquisition unit is used for acquiring corresponding pixel data from the second buffer area and writing the corresponding pixel data into the pixel buffer area under the condition that the corresponding pixel data does not exist in the pixel buffer area;

the second instruction triggering unit is used for triggering and executing the first pixel sampling instruction and triggering and executing the geometric prefetching instruction of the next image to be displayed under the condition that the pixel type buffer area has corresponding pixel data and the execution of the first geometric sampling instruction is completed;

And the first pixel texture data acquisition unit is used for acquiring the first pixel texture data based on the first pixel sampling instruction.

In some embodiments, the first geometric texture data fetch unit comprises:

the geometric data acquisition subunit is used for acquiring corresponding geometric data based on the first geometric sampling instruction;

the primary geometric texture data acquisition subunit is used for performing texture sampling processing according to the geometric data to obtain first primary geometric texture data after the texture sampling processing;

and the first geometric texture data acquisition subunit is used for performing texture filtering processing on the first preliminary geometric texture data to obtain first geometric texture data.

In some embodiments, the preliminary geometric texture data obtaining subunit is specifically configured to determine, according to the geometric data and the first geometric sampling instruction, whether corresponding preset texture data exists in the texture class buffer;

In some embodiments, the target to-be-displayed image determination module 702 includes:

the target to-be-displayed image determining sub-module is used for acquiring the display sequence of the plurality of to-be-displayed images, taking the to-be-displayed image ranked first in the display sequence as a first target to-be-displayed image, and taking the to-be-displayed image ranked second in the display sequence as a second target to-be-displayed image.

The respective modules in the above-described image processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing geometric data, pixel data and corresponding preset texture data. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an image processing method.

The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. An image processing method, the method comprising:

2. The method of claim 1, wherein executing the first and second plurality of processing instructions to sequentially display the first and second target to-be-displayed images based on the target execution order comprises:

3. The method of claim 2, wherein the plurality of first processing instructions includes a first geometry prefetch instruction, a first geometry sample instruction, a first pixel prefetch instruction, and a first pixel sample instruction; the executing, based on the target execution order, a plurality of first processing instructions to sequentially obtain first geometric texture data and first pixel texture data for an image to be displayed of the first target, including:

4. A method according to claim 3, characterized in that the method further comprises:

acquiring a currently running application program;

5. A method according to claim 3, wherein said obtaining first geometry texture data based on said first geometry sampling instruction comprises:

6. The method of claim 5, wherein performing texture sampling based on the geometry data to obtain first preliminary geometry texture data after the texture sampling comprises:

7. The method of claim 1, wherein determining the first target image to be displayed and the second target image to be displayed comprises:

8. An image processing apparatus, characterized in that the apparatus comprises:

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.