CN117203661A

CN117203661A - Image processing apparatus and method

Info

Publication number: CN117203661A
Application number: CN202180096777.0A
Authority: CN
Inventors: 蒋垚; 钱伟锋
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-12-22
Filing date: 2021-12-22
Publication date: 2023-12-08
Also published as: WO2023115408A1

Abstract

The image processing device and the method provided by the embodiment of the application relate to the technical field of computers, and can improve the rejection rate of data to be rendered, thereby reducing the consumption of computing resources and improving the rendering speed. The device comprises a rendering pipeline, wherein a first rasterizer and a first depth testing module are arranged in the rendering pipeline. A first rasterizer for converting primitives input to a rendering pipeline into fragments to be rendered. The first depth test module is used for carrying out a first depth test on the fragment to be rendered according to global depth information, if the fragment to be rendered fails the first depth test, the fragment to be rendered is removed, and the global depth information comprises depth information of a plurality of pixel blocks obtained by updating all the primitives to be rendered.

Description

Image processing apparatus and method

Technical Field

The present application relates to the field of computer technology, and in particular, to an image processing apparatus and method.

Background

Rendering is a process of performing two-dimensional projection on a model in a three-dimensional scene according to set environment, lamplight, materials and rendering parameters to obtain a digital image. But the graphics processor (graphics processing unit, GPU) consumes significant computing resources during rendering. Therefore, the GPU can reject the data to be rendered which does not affect the rendering result in the rendering stage, so as to reduce the consumption of computing resources.

However, in the related art, the GPU can only reject a small amount of data to be rendered, which does not affect the rendering result, in the rendering stage, and cannot achieve a very high rejection rate.

Disclosure of Invention

The application provides an image processing device and method, which can improve the rejection rate of data to be rendered, thereby reducing the consumption of computing resources and improving the rendering speed.

In a first aspect, the present application provides an image processing apparatus comprising a rendering pipeline having a first rasterizer and a first depth test module disposed therein. A first rasterizer for converting primitives input to the rendering pipeline into fragments to be rendered. The first depth test module is used for carrying out a first depth test on the segment to be rendered according to global depth information, if the segment to be rendered fails the first depth test, the segment to be rendered is removed, and the global depth information comprises depth information of a plurality of pixel blocks obtained by updating all the primitives to be rendered.

It can be seen that the image processing device provided by the application can reject fragments to be rendered through the global depth information, and the global depth information is generated according to all the fragments to be rendered as the global depth information is determined according to all the primitives to be rendered and the fragments to be rendered are obtained by converting the primitives to be rendered. Compared with the prior art that the segment is removed only through the depth information of part of the segments to be rendered, the method provided by the application can improve the removal rate of the data to be rendered, thereby reducing the consumption of computing resources and improving the rendering speed of the GPU.

In one possible implementation, the first depth test module is specifically configured to: and carrying out the first depth test on the segment to be rendered according to the depth information of a target pixel block in the global information and the depth information of the segment to be rendered, wherein the target pixel block is a pixel block overlapping with the segment to be rendered in the plurality of pixel blocks.

In one possible implementation, the first depth test module is specifically configured to: and determining that the fragment to be rendered fails a first depth test under the condition that the depth value of the fragment to be rendered is larger than the depth value of the target pixel block.

Since there is overlap between the target pixel block and the fragment to be rendered, the fragment to be rendered will be rendered on the target pixel block. By the depth information of the target pixel block and the depth information of the segment to be rendered in the global information, it can be judged that the segment to be rendered is blocked by the visible segment on the target pixel block under the condition that the depth value of the segment to be rendered is larger than that of the target pixel block.

Alternatively, a block of pixels (e.g., a target block of pixels) may include a target region. The depth information of the pixel block may include a maximum depth value of a target area of the pixel block, a maximum depth value of a non-target area of the pixel block, and a range of the target area of the pixel block.

Alternatively, the depth information of a fragment (e.g., a fragment to be rendered) may include the depth value of the fragment. Wherein the depth value of the segment may be obtained by interpolation.

In one possible implementation manner, the target pixel block includes a target area, the depth value of the target pixel block includes a first depth value and a second depth value, and the first depth test module is specifically configured to:

and determining that the fragment to be rendered passes the first depth test under the condition that the fragment to be rendered is positioned in the target area of the target pixel block and the depth value of the fragment to be rendered is smaller than or equal to the first depth value.

And under the condition that the fragment to be rendered is positioned in the target area of the target pixel block and the depth value of the fragment to be rendered is larger than the first depth value, determining that the fragment to be rendered does not pass the first depth test.

And determining that the fragment to be rendered passes the first depth test under the condition that the fragment to be rendered is positioned in a non-target area of the target pixel block and the depth value of the fragment to be rendered is smaller than or equal to the second depth value.

And under the condition that the fragment to be rendered is positioned in the non-target area of the target pixel block and the depth value of the fragment to be rendered is larger than the second depth value, determining that the fragment to be rendered does not pass the first depth test.

It should be noted that, the segment to be rendered is located in the target area of the target pixel block, and the depth value of the segment to be rendered is greater than the first depth value, which indicates that the segment to be rendered is blocked by the visible segment in the target area of the target pixel block, so that the segment to be rendered can be removed, so as to reduce the consumption of computing resources and improve the rendering speed of the GPU.

The segment to be rendered is located in the target area of the non-target pixel block, and the depth value of the segment to be rendered is larger than the second depth value, which indicates that the segment to be rendered is shielded by the visible segment in the non-target area of the target pixel block, so that the segment to be rendered can be removed, the consumption of computing resources is reduced, and the rendering speed of the GPU is improved.

It can be appreciated that compared with performing depth test on the fragment to be rendered only through the depth value of the target pixel block, the target pixel block is divided into the target area and the non-target area, and the depth test is performed on the fragment to be rendered through the maximum depth values of the two areas, so that the accuracy of the depth test can be increased, and the probability of error rejection can be reduced.

In one possible implementation manner, the apparatus may further include a first buffer module, where the first buffer module is configured to buffer the global depth information.

In one possible implementation, the apparatus may further include a partition pipeline having a second rasterizer and a second depth test module disposed therein. And the second rasterizer is used for projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas. And the second depth module is used for carrying out a second depth test on the to-be-rendered primitive according to the plurality of projection areas, and eliminating the to-be-rendered primitive if the to-be-rendered primitive fails the second depth test. Wherein the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks.

It can be seen that the device provided by the application can reject not only the segment to be rendered which is not required to be rendered, but also the primitive to be rendered which is not required to be rendered, and compared with the method for rejecting only the segment to be rendered which is not required to be rendered, the device provided by the application also rejects the primitive to be rendered which is not required to be rendered, thereby further improving the rejection rate of the data to be rendered, further reducing the consumption of computing resources, and further improving the rendering speed of the GPU.

In one possible implementation, the second depth module is specifically configured to: determining whether the primitive to be rendered passes a depth test in a first pixel block according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block. If the primitive to be rendered passes the depth test in the first pixel block, determining that the primitive to be rendered passes the second depth test; and if the to-be-rendered primitive fails the depth test in the pixel blocks, determining that the to-be-rendered primitive fails the second depth test.

In one possible implementation, the second depth module is specifically configured to: and determining that the primitive to be rendered passes the depth test in the first pixel block under the condition that the minimum depth value of the first projection area is smaller than the depth value of the first pixel block.

Alternatively, the depth information of the projection area (e.g., the first projection area) may include a maximum depth value of the projection area, a minimum depth value of the projection area, and a range of the projection area.

In a possible implementation manner, the first pixel block includes a target area, the depth value of the first pixel block includes a maximum depth value of the target area of the first pixel block and a maximum depth value of a non-target area of the first pixel block, and the second depth module is specifically configured to:

and if the first projection area overlaps with the target area and the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value or the second depth value, determining that the primitive to be rendered passes the test in the first pixel block.

Determining that the primitive to be rendered fails the test in the first pixel block in the case where there is overlap of the first projection region with both the target region and the non-target region of the first pixel block and the minimum depth value of the first projection region is greater than the first depth value and the second depth value

And determining that the primitive to be rendered passes the test in the first pixel block under the condition that the first projection area only overlaps with the target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value.

And determining that the primitive to be rendered fails the test in the first pixel block under the condition that the first projection area only overlaps with the target area of the first pixel block and the minimum depth value of the first projection area is larger than the first depth value.

And determining that the primitive to be rendered passes the test in the first pixel block under the condition that the first projection area only overlaps with the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the second depth value.

In the case where the first projection area only overlaps with the non-target area of the first pixel block and the minimum depth value of the first projection area is greater than the second depth value, it is determined that the primitive to be rendered fails the test in the first pixel block.

In a possible implementation, the second depth module is further configured to update the global depth information according to the plurality of projection areas.

In one possible implementation, the second depth module is further specifically configured to: and updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.

In a possible implementation manner, the first pixel block includes a target area, and the second depth module is further specifically configured to: and updating the depth information of the first pixel block according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block and the maximum depth value of the non-target area of the first pixel block.

In one possible implementation, the second depth module is further specifically configured to:

and updating the depth information of the first pixel block under the condition that the first projection area overlaps with the target area and the non-target area of the first pixel block and the maximum depth value of the first projection area is smaller than the first depth value or the second depth value.

In the case where the first projection area overlaps both the target area and the non-target area of the first pixel block and the maximum depth value of the first projection area is greater than the first depth value and the second depth value, the depth information of the first pixel block is not updated.

And updating the depth information of the first pixel block under the condition that the first projection area only overlaps with the target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value.

In the case where the first projection area overlaps only the target area of the first pixel block and the minimum depth value of the first projection area is greater than the first depth value, the depth information of the first pixel block is not updated.

And updating the depth information of the first pixel block under the condition that the first projection area only overlaps with the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the second depth value.

In case that the first projection area only overlaps with the non-target area of the first pixel block and the minimum depth value of the first projection area is larger than the second depth value, the depth information of the first pixel block is not updated.

in the case where the first projection area is completely coincident with the target area of the first pixel block and the third depth value (i.e., the maximum depth value of the projection area) is smaller than the first depth value, the target area of the first pixel block is updated to the target projection area and the first depth value of the first pixel block is updated to the third depth value.

And updating the target area of the first pixel block into the non-target projection area and updating the second depth value of the first pixel block into the third depth value under the condition that the first projection area is completely overlapped with the non-target area of the first pixel block and the third depth value is smaller than the second depth value.

When the first projection area does not completely coincide with the target area and the non-target area of the first pixel block and the first condition and the second condition are satisfied, the target area of the first pixel block is updated to the first area and the first depth value of the first pixel block is updated to the first value. The first condition is that the third depth value is smaller than the first depth value but larger than the second depth value or the third depth value is smaller than the second depth value but larger than the first depth value. The second condition is that the first absolute value is smaller than the second absolute value, the first absolute value is an absolute value of a difference between the first depth value and the third depth value, and the second absolute value is an absolute value of a difference between the second depth value and the third depth value. The first region is a union region of the target region and the target projection region, and the first value is a larger value between the first depth value and the third depth value.

And updating the target area of the first pixel block to a second area and updating the second depth value of the first pixel block to a second value under the condition that the first projection area is not completely overlapped with the target area and the non-target area of the first pixel block and the first condition and the third condition are met. The third condition is that the first absolute value is larger than the second absolute value. The second region is an intersection region of the target region and the non-target projection region. The second value is a larger value between the second depth value and the third depth value.

And updating the target area of the first pixel block to the target projection area, updating the first depth value of the first pixel block to the third depth value, and updating the second depth value of the first pixel block to the third value under the condition that the first projection area is not completely overlapped with the target area and the non-target area of the first pixel block and the third depth value is smaller than the first depth value and the second depth value. Wherein the third value is a larger value between the first depth value and the second depth value.

Optionally, the target projection area (CurMask) includes a first target projection area (i.e., curMask 0) and a second target projection area (i.e., curMask 0)

In the case where the third depth value of the projection area is smaller than the second depth value of the pixel block, the first target projection area (i.e., curMask 0) may represent an overlapping area of the projection area and the non-target area of the pixel block.

In the case where the third depth value of the projection area is smaller than the first depth value of the pixel block, the second target projection area (i.e., curMask 0) may represent an overlapping area of the projection area and the target area of the pixel block.

The target projection area (CurMask) may be a union of the first target projection area (CurMask 0) and the second target projection area (CurMask 0).

In one possible implementation, the rendering pipeline of the image processing apparatus may be further provided with at least one of a front depth test module, a pixel shader, a rear depth test module, or a depth buffer module. And the front depth test module is used for carrying out front depth test on the first target segment according to the depth value in the depth buffer module. The pixel shader is used for shading and rendering the second target segment. And the back depth test module is used for carrying out back depth test on the third target segment according to the depth value in the depth buffer module. And the depth caching module is used for caching the depth value. The first target segment is a segment passing a first depth test, the second target segment is a segment passing a previous depth test, and the third target segment is a segment rendered by coloring by a pixel shader.

In a second aspect, the present application also provides another image processing apparatus, including a rendering pipeline in which a rasterizer, a rendering coarse-granularity depth test module, a front depth test module, a pixel shader, a depth buffer module, and a rear depth test module are disposed. The rasterizer is used for converting the primitive input into the rendering pipeline into a fragment to be rendered; the rendering coarse-granularity depth test module is used for performing a first depth test on the fragment to be rendered; the front depth test module is used for performing front depth test on the first target segment according to the depth value in the depth cache module; the pixel shader is used for shading and rendering the second target segment; the rear depth test module is used for performing rear depth test on the third target segment according to the depth value in the depth cache module; the depth caching module is used for caching the depth value. The first target segment is a segment passing a first depth test, the second target segment is a segment passing a previous depth test, and the third target segment is a segment rendered by coloring by a pixel shader.

The fragments are culled by only setting the front depth test module and the rear depth test module in the rendering pipeline as compared to the related art. The rendering pipeline of the image processing device is provided with a front depth testing module, a rear depth testing module and a rendering coarse-granularity depth testing module. The fragments are subjected to depth test by the rendering coarse-granularity depth test module, the front depth test module and the rear depth test module, so that the rejection rate of data to be rendered can be improved, the consumption of computing resources is reduced, and the rendering speed of the GPU is improved.

In one possible implementation manner, the rendering coarse granularity depth test module is specifically configured to: and carrying out the first depth test on the segment to be rendered according to the depth information of a target pixel block in global information and the depth information of the segment to be rendered, wherein the target pixel block is a pixel block overlapped with the segment to be rendered in the pixel blocks, and the global depth information comprises the depth information of the pixel blocks updated according to all the primitives to be rendered.

In one possible implementation, the apparatus further includes a coarse-granularity depth buffer module configured to buffer the global depth information.

In one possible implementation, the apparatus further comprises a partition pipeline provided with a coarse-granularity rasterizer and a partition coarse-granularity depth test module; the coarse granularity rasterizer is used for projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas, and the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks; and the partition coarse-granularity depth test module is used for carrying out a second depth test on the primitives to be rendered according to the plurality of projection areas.

In one possible implementation, the partition coarse-granularity depth test module is further configured to: and updating global depth information according to the plurality of projection areas, wherein the global depth information comprises depth information of a plurality of pixel blocks obtained by updating all the primitives to be rendered.

The user can input the scenes of the rendering pipeline in the sequence of the fragments from the small to the large in the depth value, and turn off the rendering coarse-granularity depth test module in the rendering pipeline so as to reduce the data processing amount of the rendering process.

The user can turn off all the depth test modules (i.e. the rendering coarse-granularity depth test module, the front depth test module and the rear depth test module) in the rendering pipeline in the scene (such as a planar scene, a two-dimensional scene and the like) which does not relate to the depth, so that the data to be rendered directly enter the pixel shader for shading rendering, and the data processing amount in the rendering process is reduced.

The user can turn off the post-depth test module in a scene where the depth value of the fragment is not affected by the pixel shader (i.e., the depth value of the fragment does not change before and after rendering by the pixel shader), so as to reduce the data throughput of the rendering process.

The user may turn off the pre-depth test module to reduce the data throughput of the rendering process when the depth value of a segment is affected by the pixel shader (i.e., the depth value of the segment changes before and after the pixel shader rendering) and the direction of the change of the depth value of the segment is not fixed (e.g., the depth value of a portion of the segment increases after the color rendering and the depth value of another portion of the segment decreases after the rendering).

In a third aspect, the present application also provides an image processing method, including: first, a primitive input to a rendering pipeline is converted into a fragment to be rendered. And then carrying out a first depth test on the fragment to be rendered according to the global depth information, and eliminating the fragment to be rendered if the fragment to be rendered fails the first depth test. The global depth information comprises depth information of a plurality of pixel blocks updated according to all the primitives to be rendered.

In one possible implementation manner, the global depth information includes depth information of a plurality of pixel blocks, and the performing a first depth test on the segment to be rendered according to the global depth information includes: and carrying out the first depth test on the fragment to be rendered according to the depth information of a target pixel block and the depth information of the fragment to be rendered, wherein the target pixel block is a pixel block which is overlapped with the fragment to be rendered in the plurality of pixel blocks.

In one possible implementation manner, the performing the first depth test on the segment to be rendered according to the depth information of the target pixel block and the depth information of the segment to be rendered includes: and determining that the fragment to be rendered fails a first depth test under the condition that the depth value of the fragment to be rendered is larger than the depth value of the target pixel block.

In one possible implementation, the target pixel block includes a target region, and the depth value of the target pixel block includes a first depth value and a second depth value, where the first depth value is a maximum depth value of the target region, and the second depth value is a maximum depth value of a non-target region in the target pixel block.

In one possible implementation, the method further includes: projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas, wherein the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks; performing a second depth test on the primitives to be rendered according to the plurality of projection areas; and if the primitive to be rendered does not pass the second depth test, rejecting the primitive to be rendered.

In one possible implementation manner, the performing a second depth test on the primitive to be rendered according to the plurality of projection areas includes: determining whether the primitive to be rendered passes a depth test in a first pixel block according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block; if the primitive to be rendered passes the depth test in the first pixel block, determining that the primitive to be rendered passes the second depth test; and if the to-be-rendered primitive fails the depth test in the pixel blocks, determining that the to-be-rendered primitive fails the second depth test.

In one possible implementation manner, the determining whether the primitive to be rendered passes a depth test in the first pixel block according to the depth information of the first pixel block and the depth information of the first projection area includes: and determining that the primitive to be rendered passes the depth test in the first pixel block under the condition that the minimum depth value of the first projection area is smaller than the depth value of the first pixel block.

In one possible implementation, the first pixel block includes a target region, and the depth value of the first pixel block includes a maximum depth value of the target region of the first pixel block and a maximum depth value of a non-target region of the first pixel block.

In one possible implementation, the method further includes: and updating the global depth information according to the plurality of projection areas.

In a possible implementation manner, the updating the global depth information according to the plurality of projection areas includes: and updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.

In one possible implementation manner, the updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of the first projection area includes: and updating the depth information of the first pixel block according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block and the maximum depth value of the non-target area of the first pixel block.

In one possible implementation, the method further includes: the fragments passing the first depth test are subjected to a pre-depth test.

In another possible implementation, the method further includes: post depth testing is performed on fragments that pass the first depth test.

In yet another possible implementation, the method further includes: front depth testing and back depth testing are performed on the fragments that pass the first depth testing.

In a fourth aspect, the present application also provides a graphics processor comprising an electronic device as described in the above aspects or any possible implementation thereof, the graphics processor being configured to implement the method as described in the above aspects or any possible implementation thereof.

In a fifth aspect, the present application also provides an electronic device, including: a memory, at least one processor, a transceiver, and instructions stored on the memory and executable on the processor. Further, the memory, the processor and the communication interface communicate with each other through an internal connection path. Execution of the instructions by the at least one processor causes the image processing apparatus to implement the method described in the aspects above or any possible implementation thereof.

The electronic device is illustratively a cell phone.

In a sixth aspect, the application also provides a computer readable storage medium storing a computer program comprising instructions for implementing the method described in the above aspects or any possible implementation thereof.

In a seventh aspect, the application also provides a computer program product comprising instructions which, when run on a computer, cause the computer to implement the method as described in the above aspects or any possible implementation thereof.

The image processing method, the image processing apparatus, the graphics processor, the electronic device, the computer storage medium and the computer program product provided in this embodiment are used to execute the image processing method provided above, so that the advantages achieved by the method can be referred to the advantages in the image processing method provided above, and will not be repeated here.

Drawings

FIG. 1 is a schematic depth view of an embodiment of the present application;

FIG. 2 is a schematic diagram of a rendering pipeline;

fig. 3 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 4 is a schematic flow chart of an image processing method according to an embodiment of the present application;

FIG. 5 is a flowchart of another image processing method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a projection process according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a pixel block according to an embodiment of the present application;

FIG. 8 is a schematic diagram of another pixel block according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a depth value according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 11 is a schematic diagram of another depth value according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 13 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 14 is a schematic view of yet another depth value provided by an embodiment of the present application;

FIG. 15 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 16 is a schematic view of yet another depth value provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 18 is a schematic diagram of yet another depth value provided by an embodiment of the present application;

FIG. 19 is a schematic diagram of a pixel block according to another embodiment of the present application;

FIG. 20 is a schematic diagram of a frame including pixel blocks according to an embodiment of the present application;

FIG. 21 is a schematic diagram of a pixel block according to another embodiment of the present application;

fig. 22 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The term "and/or" in the present application is merely an association relation describing the association object, and indicates that three kinds of relations may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone.

The terms "first" and "second" and the like in the description and in the drawings are used for distinguishing between different objects or between different processes of the same object and not for describing a particular order of objects.

Furthermore, references to the terms "comprising" and "having" and any variations thereof in the description of the present application are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed but may optionally include other steps or elements not listed or inherent to such process, method, article, or apparatus.

It should be noted that in the description of the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the description of the present application, unless otherwise indicated, the meaning of "a plurality" means two or more.

First, some terms in the present application will be explained in order to be understood by those skilled in the art.

Depth: for characterizing the distance between the object and the camera. As shown in fig. 1, the distance is used to represent the projected distance from the camera orientation to the line segment.

Depth test: and judging whether the depth value of the data to be rendered is larger than a depth threshold value.

Front depth test: depth testing of data (e.g., fragments) is performed before the data is rendered by the pixel shader.

Post depth test: depth testing of data after rendering of the data by pixel shader.

Primitive (primary): the combination of geometric vertices in the primitive of a geometric representation in a graphics rendering pipeline is referred to as a primitive (e.g., a point, line segment, or polygon, etc.).

Rasterizing: rasterization may convert a primitive into fragments that possess information about the pixels. The rasterization determines where all pixels need to be rendered, and interpolates to determine the locations of all pixels between the two vertices. Rasterization may not only interpolate pixels, but any vertex shader's output variables and fragment shader's input variables may be based on the corresponding pixel interpolation. Smooth color gradation, real illumination and other effects can be achieved through rasterization.

Partition phase (nesting pass): dividing a screen into a plurality of areas (bins), carrying out view angle transformation on all the primitives to be rendered to obtain the screen positions of the primitives, screening the primitives to the corresponding screen areas according to the screen positions of the primitives, and finally, independently rendering each area (bin), wherein only primitive sets falling into the areas in the process of rendering the partitions are rendered. This screening process is the zoning phase.

The GPU may consume a significant amount of computing resources during rendering. Therefore, the GPU can reject the data to be rendered which does not affect the rendering result in the rendering stage, so as to reduce the consumption of computing resources.

In the related art, depth test may be used to reject data to be rendered that does not affect the rendering result. For example, in a rendering pipeline of an image processor as shown in fig. 2, a front depth test module in the rendering pipeline may perform front depth testing (earlyate-detph-test) on a fragment input to the front depth test module according to a depth value in a depth cache. When the depth value of the segment is smaller than the depth value in the depth cache, determining that the segment passes a pre-depth test, inputting the segment into a pixel shader for shading rendering, and updating the depth value in the depth cache to the depth value of the segment; and when the depth value of the fragment is larger than the depth value in the depth cache, determining that the fragment fails the pre-depth test and rejecting the fragment. In some scenes, the depth value of the specific segment is affected by the pixel shader, where depth testing requires post-depth testing (late-detph-test) in a post-depth test mode.

It can be seen that in the related art, the depth value in the depth buffer is determined according to the part of the segment to be rendered that has passed the test, that is, the related art can only reject the segment to be rendered by using the depth information of the part of the segment during the depth test, so that a very high rejection rate cannot be achieved. For example, the segments to be rendered include a segment to be rendered 1, a segment to be rendered 2, a segment to be rendered 3, a segment to be rendered 4, and a segment to be rendered 5, and the depth values are 50, 40, 30, 20, and 10, respectively. If the depth values of the 5 fragments to be rendered are used for the depth test at the same time, only the fragment 5 with the smallest depth value to be rendered passes the depth test, and the rest 4 fragments to be rendered are rejected.

As shown in table 1, when the depth test is performed on the fragment 1 to be rendered according to the prior art, the depth value is not cached in the depth cache, and then the fragment 1 to be rendered enters the shader to be rendered according to the depth test, and the depth value in the depth cache is updated to be 50. When the depth test is performed on the fragment 2 to be rendered through the prior art, the depth value 40 of the fragment 2 to be rendered is smaller than the depth value 50 in the depth cache, and then the fragment 2 to be rendered enters the shader through the depth test to be subjected to coloring rendering, and the depth value in the depth cache is updated to 40.

When the depth test is performed on the fragment 3 to be rendered through the prior art, the depth value 30 of the fragment 3 to be rendered is smaller than the depth value 50 in the depth buffer, and then the fragment 3 to be rendered enters the shader to be colored and rendered through the depth test, and the depth value in the depth buffer is updated to be 30.

When the depth test is performed on the fragment to be rendered 4 through the prior art, the depth value 20 of the fragment to be rendered 4 is smaller than the depth value 50 in the depth buffer, and then the fragment to be rendered 3 enters the shader to be subjected to coloring rendering through the depth test, and the depth value in the depth buffer is updated to be 20.

When the depth test is performed on the fragment 5 to be rendered through the prior art, the depth value 10 of the fragment 5 to be rendered is smaller than the depth value 50 in the depth buffer, and then the fragment 3 to be rendered enters the shader to be colored and rendered through the depth test, and the depth value in the depth buffer is updated to be 10.

As can be seen from table 1, the to-be-rendered segment 2, the to-be-rendered segment 3, the to-be-rendered segment 4, and the to-be-rendered segment 5 can pass the depth test, and the related art can not achieve a higher rejection rate by rejecting the to-be-rendered segment only by using the depth information of the partial segment passing the depth test.

TABLE 1

Therefore, the embodiment of the application provides an image processing method which can improve the rejection rate of data to be rendered.

The image processing method provided by the embodiment of the application is suitable for an image processing device. The image processing apparatus may be provided in a graphics processor.

In particular, the image processing apparatus may be provided in a graphics processor of an electronic device. The embodiment of the application does not limit the specific type of the electronic equipment. For example, the electronic device may be a cell phone, tablet computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), or the like.

The image processing means may be provided in a graphics processor of the handset, for example.

Fig. 3 shows a schematic block diagram of the image processing apparatus comprising a partition pipeline, a global depth information buffer module and a rendering pipeline.

In a partition pipeline, the screen space is divided into pixel blocks (pixel blocks) onto which each primitive (private) in the graphics programming interface (draw-call) is projected after rasterization (rasterization) by each CPU. For example, the screen space may be divided into N pixel blocks.

A coarse granularity rasterizer and a partition coarse granularity depth test module are arranged in the partition pipeline.

A coarse-granularity rasterizer (also referred to as a second rasterizer) projects primitives (i.e., primitives to be rendered) in an input partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas. For example, the second rasterizer may project primitives in the input partition pipeline into N pixel blocks to generate N projected areas. Wherein the plurality of pixel blocks and the plurality of projection areas are in one-to-one correspondence.

A partition coarse-granularity depth test module (also referred to as a second depth test module) is configured to perform a partition coarse-granularity depth test (also referred to as a second depth test) on primitives to be rendered according to the plurality of projection areas. And if the primitive to be rendered does not pass the second depth test, eliminating the primitive to be rendered. And if the to-be-rendered primitive does not pass the second depth test, the to-be-rendered primitive passes the second depth test, and the to-be-rendered primitive is input into a rendering pipeline.

In one possible implementation, the partition pipeline may be further provided with a polygon list generation (polygon list generation) module, polygon list generation module for storing primitives to be rendered that pass the second depth test by memory, and packing the primitives to be rendered that pass the second depth test into a Polygon List (PL), which is input to the rendering pipeline in the form of a polygon stack (PL-Heap).

In one possible implementation, the second depth test module may update the depth information accumulated on each pixel block during the second depth test and store the depth information of each pixel block into the global depth information cache module.

The global depth information buffer module (may be simply referred to as a first buffer module) is configured to buffer depth information (may be simply referred to as global depth information) of the plurality of pixel blocks, where the depth information of the pixel blocks includes a pending mask (Pmask), a pending target depth value (pending destination z value, PZ), and an original target depth value (old Destination z value, OZ). For example, the first buffer module buffers global depth information, where the global depth information includes depth information of a plurality (e.g., N) of pixel blocks updated by all primitives to be rendered.

The global depth information buffer module may be a buffer in the image processing apparatus, or may be a buffer in another device (such as a buffer in a CPU).

Pmask is used to indicate a target area in a pixel block. Pmask can identify the range of a target region in a pixel block.

PZ is used to indicate the maximum depth value of the target area in the pixel block, i.e. the depth value of all visible segments in the target area of the pixel block is smaller than the PZ value.

OZ is used to indicate the maximum depth value of the non-target area in the pixel block, i.e. the depth value of all visible fragments in the non-target area of the pixel block is smaller than the OZ value.

The rendering pipeline is provided with a rasterizer, a render coarse-granularity depth test module, and a pixel shader.

A rasterizer (which may be referred to as a first rasterizer) for converting primitives input to the rendering pipeline (i.e., primitives that pass the second depth test) into fragments (i.e., fragments to be rendered).

A rendering coarse-granularity depth test module (which may be called a first depth test module) is configured to perform a first depth test on the fragment to be rendered according to the global depth information. And if the fragment to be rendered does not pass the first depth test, eliminating the fragment to be rendered. If the fragment to be rendered passes the first depth test, the fragment to be rendered is input to a pixel shader.

In one possible implementation, the render coarse-granularity depth test module may also perform the first depth test on the fragment to be rendered through other information.

And the pixel shader is used for shading and rendering fragments of the input pixel shader.

In one possible implementation, a front depth test module, a rear depth test module, and a depth cache module may also be provided in the rendering pipeline.

And the front depth test module is used for carrying out front depth test on the fragment to be rendered. The specific method of pre-depth testing may be handled by any method as will occur to those of skill in the art, and embodiments of the present application are not specifically limited in this regard.

And the rear depth test module is used for performing rear depth test on the colored and rendered fragments. The specific method of post depth testing may be handled by any method as will occur to those of skill in the art, and embodiments of the present application are not specifically limited in this regard.

In one possible implementation, the second depth module is specifically configured to: determining whether a primitive to be rendered passes a depth test in a first pixel block according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in a plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block. If the primitive to be rendered passes the depth test in the first pixel block, determining that the primitive to be rendered passes the second depth test; and if the primitives to be rendered do not pass the depth test in the pixel blocks, determining that the primitives to be rendered do not pass the second depth test.

In one possible implementation, the second depth module is further configured to update the global depth information based on the plurality of projection areas.

In one possible implementation, the second depth module is further specifically configured to: and updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of the first projection area, wherein the first pixel block is any pixel block in a plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.

In one possible implementation, the first pixel block includes a target area, and the second depth module is further specifically configured to: and updating the depth information of the first pixel block according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block and the maximum depth value of the non-target area of the first pixel block.

In one possible implementation, the first depth test module is specifically configured to: and carrying out a first depth test on the fragment to be rendered according to the depth information of the target pixel block in the global information and the depth information of the fragment to be rendered, wherein the target pixel block is a pixel block overlapping with the fragment to be rendered in a plurality of pixel blocks.

And determining that the fragment to be rendered does not pass the first depth test under the condition that the fragment to be rendered is not located in the non-target area of the target pixel block and the depth value of the fragment to be rendered is larger than the second depth value.

As shown in fig. 3, a user may turn off all depth test modules (i.e., a rendering coarse-granularity depth test module, a front depth test module, and a rear depth test module) in a rendering pipeline in an image processing apparatus in a scene (e.g., a planar scene, a two-dimensional scene, etc.) that does not involve depth, so that data to be rendered directly enters a pixel shader to be rendered, thereby reducing the data processing amount in the rendering process.

The user may turn off the pre-depth test module to reduce the data throughput of the rendering process when the depth value of a segment is affected by the pixel shader (i.e., the depth value of the segment may change before and after the pixel shader rendering) and the direction of change of the depth value of the segment is not fixed (e.g., the depth value of a portion of the segment increases after the shader rendering and the depth value of another portion of the segment decreases after the shader rendering).

Fig. 4 shows a schematic flow chart of an image processing method provided by an embodiment of the present application, which can be performed by the image processing apparatus shown in fig. 3. As shown in fig. 4, the image processing method includes:

s401, the image processing device performs a second depth test on the primitives to be rendered which are input into the partition pipeline, and eliminates the primitives to be rendered which do not pass the second depth test.

Taking the example that M primitives to be rendered are input into the partition pipeline, the image processing apparatus performs a second depth test on the 1 st primitive to be rendered, then performs a second depth test on the 2 nd primitive, and then performs a second depth test on the 3 rd, 4 th, … … th and M th primitives to be rendered. If the primitive to be rendered does not pass the second depth test, rejecting the primitive to be rendered; and if the to-be-rendered primitive passes the second test, inputting the to-be-rendered primitive into a rendering pipeline.

It should be noted that, the primitive to be rendered does not pass the second depth test, which indicates that the primitive to be rendered has a higher depth and is blocked by other primitives to be rendered, so that the primitive may not be rendered, but is rejected to reduce the consumption of computing resources.

In one possible implementation, the performing, by the image processing apparatus, the second depth test on each primitive to be rendered may include: the primitive to be rendered is projected to a plurality of pixel blocks divided by a screen space through a coarse granularity rasterizer to generate a plurality of projection areas, and then a second depth test is carried out on the primitive to be rendered according to the plurality of projection areas. Wherein the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks.

For example, as shown in fig. 6, the screen space may be divided into 16 pixel blocks (white areas in the drawing), the image processing apparatus may project the primitive to be rendered onto the 16 pixel blocks, respectively, to generate 16 projection areas (black areas in the drawing) corresponding to the 16 pixel blocks one by one, and then perform the second depth test on the primitive to be rendered according to the 16 projection areas.

In one possible implementation, the image processing apparatus may determine whether the primitive to be rendered passes the depth test in each pixel block according to the depth information of each pixel block and the depth information of the projection area corresponding to each pixel block. If the primitive to be rendered passes the depth test in any pixel block, determining that the primitive to be rendered passes a second depth test; and if the primitive to be rendered fails the depth test in the pixel blocks, determining that the primitive to be rendered fails the second depth test.

For example, as shown in fig. 6, the image processing apparatus may determine whether the primitive to be rendered passes the depth test in each of the 16 pixel blocks in fig. 6 based on the depth information of the 16 pixel blocks in fig. 6 and the depth information of the 16 projection areas in fig. 6. And if the primitive to be rendered passes the depth test in any pixel block, determining that the primitive to be rendered passes a second depth test. And if none of the pixels to be rendered passes the depth test, determining that the pixels to be rendered do not pass the second depth test.

Alternatively, the pixel block may include a target area, and the depth information of the pixel block may include a maximum depth value (may be simply referred to as a first depth value) of the target area of the pixel block, a maximum depth value (may be simply referred to as a second depth value) of a non-target area of the pixel block, and a range of the target area of the pixel block.

Alternatively, the depth information of the projection region may include a maximum depth value of the projection region and a minimum depth value of the projection region.

In one possible implementation manner, the image processing apparatus may determine whether the primitive to be rendered passes the depth test in each pixel block according to a maximum depth value of the target area of each pixel block, a maximum depth value of the non-target area of each pixel block, and a minimum depth value of the projection area corresponding to each pixel block.

Fig. 7 shows a pixel block and a projection area, in fig. 7, a gray area is a target area of the pixel block, a white area is a non-target area of the pixel block, and a black area is a projection area. As can be seen from fig. 7, there are three positional relationships between the pixel block and the projection area. Fig. 7 (a) illustrates that the first positional relationship, i.e., the projection region, overlaps with both the target region and the non-target region of the pixel block; fig. 7 (b) shows that the projection region overlaps only with the target region of the pixel block, which is the second positional relationship; fig. 7 (c) shows that the projection region overlaps only with the non-target region of the pixel block in the third positional relationship.

Taking the first pixel block and the first projection area as an example in conjunction with fig. 7, a specific process of determining whether the primitive to be rendered passes the depth test in each pixel block by the image processing apparatus according to the maximum depth value of the target area of each pixel block, the maximum depth value of the non-target area of each pixel block, and the minimum depth value of the projection area corresponding to each pixel block is described below. The first pixel block is any one of a plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.

As shown in fig. 7 (a), in the case where there is overlap of the first projection area with both the target area and the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value or the second depth value, it is determined that the primitive to be rendered passes the depth test in the first pixel block.

As shown in fig. 7 (a), in the case where there is overlap of the first projection area with both the target area and the non-target area of the first pixel block and the minimum depth value of the first projection area is greater than the first depth value and the second depth value, it is determined that the primitive to be rendered fails the depth test in the first pixel block.

As shown in fig. 7 (b), in the case where there is only overlap of the first projection area with the target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value, it is determined that the primitive to be rendered passes the depth test in the first pixel block.

As shown in fig. 7 (b), in the case where there is only overlap of the first projection area with the target area of the first pixel block and the minimum depth value of the first projection area is greater than the first depth value, it is determined that the primitive to be rendered does not pass the depth test in the first pixel block.

As shown in fig. 7 (c), in the case where there is only overlap of the first projection area with the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the second depth value, it is determined that the primitive to be rendered passes the depth test in the first pixel block.

As shown in fig. 7 (c), in the case where the first projection area overlaps only the non-target area of the first pixel block and the minimum depth value of the first projection area is greater than the second depth value, it is determined that the primitive to be rendered does not pass the test in the first pixel block.

In a possible implementation, the image processing apparatus may further update the global depth information according to the plurality of projection areas. Wherein the global depth information includes depth information of a plurality of pixel blocks divided by the screen space. For example, the image processing apparatus may update the depth information of each pixel block in the global depth information according to the depth information (current depth information) of each pixel block and the depth information of the projection area corresponding to each pixel block.

In the following, taking the first projection area and the first pixel block as an example in conjunction with fig. 7, a specific process of updating the depth information of the first pixel block by the image processing apparatus according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block (i.e., the first depth value), and the maximum depth value of the non-target area of the first pixel block (i.e., the second depth value) will be described.

As shown in fig. 7 (a), in the case where there is overlap of the first projection region with both the target region and the non-target region of the first pixel block and the maximum depth value of the first projection region is smaller than the first depth value or the second depth value, the depth information of the first pixel block is updated.

As shown in fig. 7 (a), in the case where there is overlap of the first projection region with both the target region and the non-target region of the first pixel block and the maximum depth value of the first projection region is greater than the first depth value and the second depth value, the depth information of the first pixel block is not updated.

As shown in fig. 7 (b), in the case where the first projection area overlaps only the target area of the first pixel block and the minimum depth value of the first projection area is smaller than the first depth value, the depth information of the first pixel block is updated.

As shown in fig. 7 (b), in the case where the first projection area overlaps only the target area of the first pixel block and the minimum depth value of the first projection area is greater than the first depth value, the depth information of the first pixel block is not updated.

As shown in fig. 7 (c), in the case where the first projection area overlaps only the non-target area of the first pixel block and the minimum depth value of the first projection area is smaller than the second depth value, the depth information of the first pixel block is updated.

As shown in fig. 7 (c), in the case where the first projection area overlaps only the non-target area of the first pixel block and the minimum depth value of the first projection area is greater than the second depth value, the depth information of the first pixel block is not updated.

In one possible implementation, the image processing apparatus may update the depth information of the first pixel block according to the target area. Wherein the target projection area (CurMask) includes a first target projection area (i.e., curMask 0) and a second target projection area (i.e., curMask 0).

CurMask0: in the overlapping area of the projection area of the pixel block to be rendered and the non-target area (also referred to as OZ area) of the pixel block, if the maximum depth (which may be referred to as a third depth value, also referred to as Zmax) of the projection area is smaller than the maximum depth (i.e. the second depth value, also referred to as OZ) of the non-target area of the pixel block, the depth values of all visible segments of the pixel block in the overlapping area are necessarily smaller than the Zmax value (otherwise masked by the pixel), and then the OZ value of the pixel block is conditionally updated to the Zmax value of the projection area of the pixel block to be rendered. In this case the coverage of this part of the area may be represented by a Curmask0 mask. That is, in the case where the third depth value of the projection area is smaller than the second depth value of the pixel block, curMask0 may represent an overlapping area of the projection area and the non-target area of the pixel block.

CurMask1: in the overlapping area of the projection area of the pixel block to be rendered and the target area (also referred to as PZ area) of the pixel block, if the maximum depth of the projection area is smaller than the maximum depth value (i.e. the first depth value, which may be simply referred to as PZ) of the target area of the pixel block, the depth values of all visible segments of the pixel block in the overlapping area are necessarily smaller than the Zmax value (otherwise masked by the pixel), and then the PZ value of the pixel block is conditionally updated to the Zmax value of the projection area of the pixel block to be rendered. In this case the coverage of this part of the area may be represented by a CurMask1 mask. That is, in the case where the third depth value of the projection area is smaller than the first depth value of the pixel block, curMask1 may represent an overlapping area of the projection area and the target area of the pixel block.

CurMask: curMask0 and CurMask 1.

layer0Mask: representing the areas in the original OZ area (i.e., in the OZ area prior to updating) that are not indicated by CurMask 0.

layer1Mask: representing the areas indicated by non-CurMask 1 in the original PZ region (i.e. in the PZ region before updating).

Specifically, the image processing apparatus updating the depth information of the first pixel block may include:

Illustratively, fig. 8 (a) illustrates a target area (gray area) and a non-target area (white area) before the first pixel block is updated, and fig. 8 (b) illustrates a first projection area (black area) and a first pixel block. It can be seen from fig. 8 (b) that the first projection area completely coincides with the target area of the first pixel block. Fig. 9 (a) illustrates a depth range of the first depth value (PZ), the second depth value (OZ), and the first projection region, and the horizontal axis in fig. 9 represents the depth value, increasing from left to right. It can be seen from the graph (a) in fig. 9 that the maximum depth value (Zmax) of the first projection area is smaller than the first depth value (PZ) and the second depth value (OZ), so that the target projection area includes the entire first projection area. Accordingly, as shown in fig. 8 (c), the image processing apparatus updates the target area of the first pixel block to the target projection area. As shown in fig. 9 (b), the image processing apparatus updates the first depth value (PZ) of the first pixel block to the third depth value (Zmax).

Illustratively, as can be seen with reference to fig. 10 (a) and (b), the first projection area is completely coincident with the non-target area of the first pixel block. Referring to fig. 11 (a), it can be seen that the third depth value is smaller than the second depth value but larger than the first depth value, and the target projection area includes an overlapping area (i.e., the entire non-target area) of the first projection area and the non-target area of the first pixel block. As shown in fig. 8 (c), since the non-target projection area is the same as the target area of the first pixel block, the image processing apparatus does not update the target area. As shown in fig. 11 (b), the image processing apparatus updates the second depth value (OZ) of the first pixel block to the third depth value (Zmax).

Also illustratively, as can be seen with reference to fig. 12 (a) and (b), the first projected area is fully coincident with the non-target area of the first pixel block. If the maximum depth value (Zmax) of the first projection area is smaller than the first depth value (PZ) and the second depth value (OZ), the target projection area comprises the entire first projection area. Then the image processing apparatus updates the target area of the first pixel block to the non-target projection area (i.e., the area of the first pixel block that does not overlap the first projection area) and updates the second depth value (OZ) of the first pixel block to the third depth value (Zmax) as shown in fig. 12 (c).

The first condition and the second condition are satisfied, which means that the third depth value is between the first depth value and the second depth value, and the third depth value is closer to the first depth value.

Also illustratively, as can be seen with reference to fig. 13 (a) and (b), the first projection area does not completely coincide with both the target area and the non-target area of the first pixel block. Referring to fig. 14 (a), it can be seen that the third depth value (Zmax) is between the first depth value (PZ) and the second depth value (OZ), and the third depth value is closer to the first depth value. Since the third depth value is smaller than the second depth value but larger than the first depth value, the target projection area includes only the overlapping area of the first projection area and the non-target area, and the first area includes the target projection area and the current target area. As shown in fig. 13 (c), the image processing apparatus updates the target area of the first pixel block to the first area. Since the third depth value is greater than the first depth value, the first value is the third depth value. As shown in fig. 14 (b), the image processing apparatus updates the first depth value (PZ) of the first pixel block to the third depth value (Zmax).

The first condition and the third condition are satisfied, which means that the third depth value is between the first depth value and the second depth value, and the third depth value is closer to the second depth value.

Also illustratively, as can be seen with reference to fig. 15 (a) and (b), the first projection area does not completely coincide with both the target area and the non-target area of the first pixel block. Referring to fig. 16 (a), it can be seen that the third depth (Zmax) value is between the first depth value (PZ) and the second depth value (OZ), and the third depth value is closer to the second depth value. Since the third depth value is smaller than the second depth value but larger than the first depth value, the target projection area includes only the overlapping area of the first projection area and the non-target area. The second region then includes an intersection region of the non-target projection region and the current target region. As shown in fig. 15 (c), since the second area is the same as the current target area, the image processing apparatus does not update the target area. Since the second depth value is greater than the third depth value, the second value is the second depth value. As shown in fig. 16 (b), since the second value is the same as the current second depth value (OZ), the image processing apparatus does not update the second depth value (OZ).

Also illustratively, as can be seen with reference to fig. 17 (a) and (b), the first projection area does not completely coincide with both the target area and the non-target area of the first pixel block. Referring to fig. 18 (a), it can be seen that the third depth value (Zmax) is smaller than the first depth value (PZ) and the second depth value (OZ). Since the third depth value is smaller than the first depth value and the second depth value, the target projection area includes the entire first projection area. As shown in fig. 17 (c), the image processing apparatus updates the target area of the first pixel block to the target projection area, that is, the entire first projection area. Since the first depth value is smaller than the second depth value, the third value is the second depth value. As shown in fig. 18 (a), the image processing apparatus updates the first depth value of the first pixel block to the third depth value. In addition, since the current second depth value of the first pixel block is the same as the third value, the image processing apparatus does not update the second depth value of the first pixel block.

Optionally, the image processing apparatus may generate (polygon list generation) a list of polygons storing primitives to be rendered that pass the second depth test in the partition pipeline.

S402, the image processing device inputs the primitives to be rendered which pass the second depth test into a rendering pipeline.

Illustratively, taking the example that K primitives to be rendered among the primitives to be rendered of the M primitive to be rendered input partition pipeline pass the second depth test, polygon list generation in the image processing apparatus may package the primitives to be rendered that pass the second depth test into a plurality of Polygon Lists (PLs) and then input the polygon lists (PL-Heaps) to the rendering pipeline in the form of a polygon stack.

S403, the image processing device converts the primitive input into the rendering pipeline into a fragment to be rendered.

Illustratively, the image processing apparatus may convert the primitives input to the rendering pipeline (i.e., primitives passing the second depth test) into fragments to be rendered by a rasterizer in the rendering pipeline.

The specific method for converting the primitive into the fragment may be any method that can be considered by those skilled in the art, and the embodiment of the present application is not limited in detail.

S404, the image processing device performs a first depth test on the fragment to be rendered and eliminates the fragment to be rendered which does not pass the first depth test.

In one possible implementation manner, the image processing apparatus may perform a first depth test on the fragment to be rendered according to the global depth information in the global depth information buffer module.

Specifically, the image processing apparatus may perform the first depth test on the fragment to be rendered according to the depth information of the target pixel block and the depth information of the fragment to be rendered in the global depth information. The target pixel block is a pixel block which is overlapped with the fragment to be rendered in a plurality of pixel blocks.

Illustratively, fig. 19 shows 9 pixel blocks and 1 fragment to be rendered (black square in the figure), and it can be seen that there is overlap of the pixel block in the upper left corner with the fragment to be rendered. And the image processing device performs a first depth test on the fragment to be rendered according to the depth information of the pixel block at the upper left corner in the global depth information and the depth information of the fragment to be rendered.

Alternatively, the depth information of the segment may include a depth value of the segment. Wherein the depth value of the segment may be obtained by interpolation.

In one possible implementation, the image processing apparatus performs a first depth test on the fragment to be rendered according to the depth value, the first depth value, or the second depth value of the fragment to be rendered.

In one possible implementation manner, the image processing apparatus may perform a first depth test on the fragment to be rendered according to the depth value, the first depth value, or the second depth value of the fragment to be rendered, and may include:

the image processing device determines that the fragment to be rendered passes the first depth test under the condition that the fragment to be rendered is positioned in a target area of the target pixel block and the depth value of the fragment to be rendered is smaller than or equal to the first depth value.

The image processing device determines that the fragment to be rendered fails the first depth test under the condition that the fragment to be rendered is located in the target area of the target pixel block and the depth value of the fragment to be rendered is larger than the first depth value.

The image processing device determines that the fragment to be rendered passes the first depth test under the condition that the fragment to be rendered is not located in the target area of the target pixel block and the depth value of the fragment to be rendered is smaller than or equal to the second depth value.

The image processing device determines that the fragment to be rendered fails the first depth test under the condition that the fragment to be rendered is not located in the target area of the target pixel block and the depth value of the fragment to be rendered is larger than the second depth value.

Illustratively, taking a 4x4 size block of pixels within the left frame of the picture shown in fig. 20 as an example, it can be seen that the objects rendered in the block of pixels include the sky, the roof, and a triangle. The gray area shown in fig. 21 (a) is a target area of the pixel block, and the black area is a non-target area of the pixel block. The maximum depth value (PZ) of the target area of the pixel block in the global depth information is 0.1, and the maximum depth value (OZ) of the non-target area is 1.0. After rasterizing the triangle, it can be seen from fig. 21 (b) that the two segments of the triangle (i.e., the white squares in the figure) are located in the target area, and from fig. 21 (c) that the two segments of the triangle (i.e., the light gray squares in the figure) are located in the non-target area. As shown in fig. 21 (d), since the depth value of the interpolation of the two segments located in the target area is greater than PZ, the two segments located in the target area are rejected; the depth value of the interpolation of the two fragments in the non-target area is smaller than OZ, so that the two fragments in the non-target area are not rejected and are input into a pixel shader for shading rendering.

It can be seen that, the method provided by the embodiment of the application can reject the segment to be rendered through the global depth information, and the global depth information is generated according to all the segments to be rendered because the global depth information is determined according to all the primitives to be rendered and the segment to be rendered is obtained by converting the primitives to be rendered. Compared with the prior art, the method and the device for eliminating the fragments by using the depth information of all the fragments to be rendered can improve the eliminating rate of the data to be rendered, thereby reducing the consumption of computing resources and improving the rendering speed of the GPU.

S405, the image processing device inputs the segment to be rendered which passes the first depth test into a pixel shader.

In one possible implementation, the image processing apparatus may perform the previous depth test on the segment to be rendered that passes the first depth test by the previous depth test module, and then input the segment to be rendered that passes the previous depth test to the pixel shader.

S406, the image processing device performs shading rendering on the fragments of the input pixel shader.

The specific method of rendering the above-described coloring may be handled by any method that can be conceived by those skilled in the art, and the embodiment of the present application is not particularly limited thereto.

In one possible implementation, the image processing apparatus may perform a post-depth test on the color-rendered segment by a post-depth test module.

Fig. 5 shows a schematic flow chart of another image processing method provided by an embodiment of the present application, which can be performed by the image processing apparatus shown in fig. 3. As shown in fig. 5, the image processing method includes:

s501, the image processing device converts the primitive input into the rendering pipeline into a fragment to be rendered.

S502, the image processing device performs a first depth test on the fragment to be rendered according to the global depth information

The global depth information comprises depth information of a plurality of pixel blocks updated according to all the primitives to be rendered.

It should be noted that, the specific implementation of S502 may refer to the description of S404, which is not described herein.

S503, if the fragment to be rendered does not pass the first depth test, the image processing device eliminates the fragment to be rendered.

The embodiment of the present application also provides a computer storage medium having stored therein computer instructions which, when executed on an image processing apparatus, cause the image processing apparatus to execute the above-described related method steps to implement the image processing method in the above-described embodiment.

The embodiment of the application also provides a computer program product which, when run on a computer, causes the computer to perform the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

The embodiment of the application also provides an image processing device which can be a chip, an integrated circuit, a component or a module. In particular, the apparatus may comprise a processor coupled to a memory for storing instructions, or the apparatus may comprise at least one processor for retrieving instructions from an external memory. When the apparatus is running, the processor may execute instructions to cause the chip to perform the image processing method in the above-described method embodiments.

Referring to fig. 22, fig. 22 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application. The image processing apparatus 2000 includes: the at least one CPU, GPU, memory, type of memory may include, for example, SRAM and ROM, microcontroller (Micro controller Unit, MCU), WLAN subsystem, bus, transmission interface, etc. Although not shown in fig. 22, the image processing apparatus 2000 may further include an application processor (Application Processor, AP), an NPU, or other dedicated processor, and other subsystems such as a power management subsystem, a clock management subsystem, and a power consumption management subsystem.

The various portions of the image processing apparatus 2000 are coupled by connectors, which are exemplary and include various types of interfaces, transmission lines or buses, etc., which are typically electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment.

Alternatively, the CPU may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor; alternatively, the CPU may be a processor group composed of a plurality of processors, and the plurality of processors are coupled to each other through one or more buses. In an alternative case, the CPU implements an image processing method as in any of the method embodiments described above by calling program instructions stored in the memory described above or in off-chip memory. In an alternative case, the CPU and MCU together implement any of the image processing methods described in the method embodiments above, e.g. the CPU performs part of the steps in the image processing method and the MCU performs the other steps in the image processing method. In an alternative case, an AP or other dedicated processor implements an image processing method as in any of the method embodiments described above by invoking program instructions stored in the on-chip or off-chip memory.

The transmission interface may be an interface of the processor chip for receiving and transmitting data, and typically includes a variety of interfaces, and in an alternative case, the transmission interface may include an Inter-Integrated Circuit (I2C) interface, a serial peripheral interface (Serial Peripheral Interface, SPI), a universal asynchronous receiver/transmitter (UART) interface, a General-purpose input/output (GPIO) interface, and the like. It should be appreciated that these interfaces may be different functions implemented by multiplexing the same physical interfaces.

In an alternative case, the transmission interface may also include a high definition multimedia interface (High Definition Multimedia Interface, HDMI), a V-By-One interface, an embedded Display Port (Embedded Display Port, eDP), a mobile industry processor interface (Mobile Industry Processor Interface, MIPI), or a Display Port (DP), etc.

In an alternative case, the above parts are integrated on the same chip; in another alternative, the memory may be a stand-alone chip.

The WLAN subsystem may include, for example, radio frequency circuitry and baseband.

The chips referred to in embodiments of the present application are systems fabricated on the same semiconductor substrate in an integrated circuit process, also known as semiconductor chips, which may be a collection of integrated circuits formed on a substrate (typically a semiconductor material such as silicon) using an integrated circuit process, the outer layers of which are typically encapsulated by a semiconductor encapsulation material. The integrated circuit may include various functional devices, where each functional device includes a logic gate, a Metal-Oxide-Semiconductor (MOS) transistor, a bipolar transistor, a diode, or other components such as a capacitor, a resistor, or an inductor. Each functional device can work independently or under the action of necessary driving software, and can realize various functions such as communication, operation or storage.

The image processing apparatus, the computer storage medium and the computer program product provided in this embodiment are used to execute the corresponding methods provided above, so that the advantages achieved by the image processing apparatus, the computer storage medium and the computer program product can refer to the advantages of the corresponding methods provided above, and are not described herein.

It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the above-described method embodiments, and are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the above-described division of units is merely a logical function division, and there may be another division manner in actual implementation, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The above functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or a part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above-described method of the various embodiments of the present application. And the storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

An image processing apparatus, comprising: a rendering pipeline in which a first rasterizer and a first depth test module are disposed;

the first rasterizer is used for converting the primitive input into the rendering pipeline into a fragment to be rendered;

the first depth test module is used for carrying out a first depth test on the fragment to be rendered according to global depth information;

and if the segment to be rendered does not pass the first depth test, rejecting the segment to be rendered, wherein the global depth information comprises depth information of a plurality of pixel blocks updated according to all the primitives to be rendered.
The apparatus of claim 1, wherein the first depth test module is specifically configured to:

and carrying out the first depth test on the segment to be rendered according to the depth information of a target pixel block in the global information and the depth information of the segment to be rendered, wherein the target pixel block is a pixel block overlapping with the segment to be rendered in the plurality of pixel blocks.
The apparatus of claim 2, wherein the first depth test module is specifically configured to:

and determining that the fragment to be rendered fails a first depth test under the condition that the depth value of the fragment to be rendered is larger than the depth value of the target pixel block.
The apparatus of claim 3, wherein the target block of pixels comprises a target region, and wherein the depth value of the target block of pixels comprises a first depth value that is a maximum depth value of the target region and a second depth value that is a maximum depth value of non-target regions in the target block of pixels.
The apparatus of any one of claims 1 to 4, further comprising a first caching module configured to cache the global depth information.
The apparatus according to any one of claims 1 to 5, further comprising: a partition pipeline in which a second rasterizer and a second depth test module are disposed:

the second rasterizer is used for projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas, and the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks;

The second depth module is used for carrying out a second depth test on the primitive to be rendered according to the plurality of projection areas;

and if the primitive to be rendered does not pass the second depth test, rejecting the primitive to be rendered.
The apparatus of claim 6, wherein the second depth module is specifically configured to:

determining whether the primitive to be rendered passes a depth test in a first pixel block according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block;

if the primitive to be rendered passes the depth test in the first pixel block, determining that the primitive to be rendered passes the second depth test;

and if the to-be-rendered primitive fails the depth test in the pixel blocks, determining that the to-be-rendered primitive fails the second depth test.
The apparatus of claim 7, wherein the second depth module is specifically configured to:

and determining that the primitive to be rendered passes the depth test in the first pixel block under the condition that the minimum depth value of the first projection area is smaller than the depth value of the first pixel block.
The apparatus of claim 8, wherein the first block of pixels comprises a target region, and wherein the depth value of the first block of pixels comprises a maximum depth value of the target region of the first block of pixels and a maximum depth value of a non-target region of the first block of pixels.
The apparatus of any of claims 6 to 9, wherein the second depth module is further configured to update the global depth information based on the plurality of projection areas.
The apparatus of claim 10, wherein the second depth module is further specifically configured to:

and updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.
The apparatus of claim 11, wherein the first block of pixels comprises a target region, and wherein the second depth module is further specifically configured to:

and updating the depth information of the first pixel block according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block and the maximum depth value of the non-target area of the first pixel block.
The apparatus of any one of claims 1 to 12, wherein the rendering pipeline is further provided with at least one of a front depth module, a pixel shader, a rear depth test module, or a depth cache module;

the front depth test module is used for performing front depth test on a first target segment according to the depth value in the depth cache module, wherein the first target segment is a segment passing the first depth test;

the pixel shader is used for shading and rendering a second target segment, and the second target segment is a segment passing the front depth test;

the back depth test module is used for performing a back depth test according to a third target segment of the depth value in the depth buffer module, wherein the third target segment is a segment rendered by the pixel shader;

the depth caching module is used for caching the depth value.
An image processing apparatus, comprising: the system comprises a rendering pipeline, wherein a rasterizer, a rendering coarse-granularity depth test module, a front depth test module, a pixel shader, a depth buffer module and a rear depth test module are arranged in the rendering pipeline;

The rasterizer is used for converting the primitive input into the rendering pipeline into a fragment to be rendered;

the rendering coarse-granularity depth test module is used for performing a first depth test on the fragment to be rendered;

the front depth test module is used for performing front depth test on a first target segment according to the depth value in the depth cache module, wherein the first target segment is a segment passing the first depth test;

the pixel shader is used for shading and rendering a second target segment, and the second target segment is a segment passing the front depth test;

the back depth test module is used for carrying out back depth test on a third target segment according to the depth value in the depth buffer module, wherein the third target segment is a segment rendered by the pixel shader;

the depth caching module is used for caching the depth value.
The apparatus of claim 14, wherein the render coarse-granularity depth test module is specifically configured to:

and carrying out the first depth test on the segment to be rendered according to the depth information of a target pixel block in global information and the depth information of the segment to be rendered, wherein the target pixel block is a pixel block overlapped with the segment to be rendered in the pixel blocks, and the global depth information comprises the depth information of the pixel blocks updated according to all the primitives to be rendered.
The apparatus of claim 15, further comprising a coarse-granularity depth caching module configured to cache the global depth information.
The apparatus according to any one of claims 14 to 16, further comprising a partition pipeline provided with a coarse-granularity rasterizer and a partition coarse-granularity depth test module;

the coarse granularity rasterizer is used for projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas, and the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks;

and the partition coarse-granularity depth test module is used for carrying out a second depth test on the primitives to be rendered according to the plurality of projection areas.
The apparatus of claim 17, wherein the partition coarse-granularity depth test module is further to:

and updating global depth information according to the plurality of projection areas, wherein the global depth information comprises depth information of a plurality of pixel blocks obtained by updating all the primitives to be rendered.
An image processing method, comprising:

Converting the primitives input into a rendering pipeline into fragments to be rendered;

performing a first depth test on the fragment to be rendered according to global depth information, wherein the global depth information comprises depth information of a plurality of pixel blocks obtained by updating all the primitives to be rendered;

and if the fragment to be rendered does not pass the first depth test, eliminating the fragment to be rendered.
The method of claim 19, wherein the global depth information comprises depth information for a plurality of pixel blocks, wherein the performing a first depth test on the segment to be rendered based on global depth information comprises:

and carrying out the first depth test on the fragment to be rendered according to the depth information of a target pixel block and the depth information of the fragment to be rendered, wherein the target pixel block is a pixel block which is overlapped with the fragment to be rendered in the plurality of pixel blocks.
The method of claim 20, wherein the performing the first depth test on the segment to be rendered based on the depth information of the target pixel block and the depth information of the segment to be rendered comprises:

and determining that the fragment to be rendered fails a first depth test under the condition that the depth value of the fragment to be rendered is larger than the depth value of the target pixel block.
The method of claim 21, wherein the target pixel block comprises a target region, and wherein the depth value of the target pixel block comprises a first depth value that is a maximum depth value of the target region and a second depth value that is a maximum depth value of a non-target region in the target pixel block.
The method according to any one of claims 19 to 22, further comprising:

projecting the primitive to be rendered input into the partition pipeline to a plurality of pixel blocks to generate a plurality of projection areas, wherein the plurality of projection areas are in one-to-one correspondence with the plurality of pixel blocks;

performing a second depth test on the primitives to be rendered according to the plurality of projection areas;

and if the primitive to be rendered does not pass the second depth test, rejecting the primitive to be rendered.
The method of claim 23, wherein the performing a second depth test on the primitives to be rendered based on the plurality of projection areas comprises:

determining whether the primitive to be rendered passes a depth test in a first pixel block according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block;

If the primitive to be rendered passes the depth test in the first pixel block, determining that the primitive to be rendered passes the second depth test;

and if the to-be-rendered primitive fails the depth test in the pixel blocks, determining that the to-be-rendered primitive fails the second depth test.
The method of claim 24, wherein determining whether the primitive to be rendered passes a depth test in the first pixel block based on depth information of the first pixel block and depth information of a first projection region comprises:

and determining that the primitive to be rendered passes the depth test in the first pixel block under the condition that the minimum depth value of the first projection area is smaller than the depth value of the first pixel block.
The method of claim 25, wherein the first block of pixels includes a target region, and wherein the depth value of the first block of pixels includes a maximum depth value of the target region of the first block of pixels and a maximum depth value of a non-target region of the first block of pixels.
The method according to any one of claims 23 to 26, further comprising:

And updating the global depth information according to the plurality of projection areas.
The method of claim 27, wherein the updating the global depth information from the plurality of projection areas comprises:

and updating the depth information of the first pixel block in the global depth information according to the depth information of the first pixel block and the depth information of a first projection area, wherein the first pixel block is any pixel block in the plurality of pixel blocks, and the first projection area is a projection area corresponding to the first pixel block.
The method of claim 28, wherein the updating the depth information of the first block of pixels in the global depth information based on the depth information of the first block of pixels and the depth information of the first projection area comprises:

and updating the depth information of the first pixel block according to the maximum depth value of the first projection area, the maximum depth value of the target area of the first pixel block and the maximum depth value of the non-target area of the first pixel block.
The method according to any one of claims 19 to 29, further comprising:

front depth testing and/or rear depth testing is performed on the fragments that pass the first depth testing.
An electronic device comprising at least one processor and interface circuitry, the at least one processor and the interface circuitry coupled, wherein the at least one processor executes programs or instructions stored in memory to cause the electronic device to implement the method of any one of claims 19-30.
A computer readable storage medium storing a computer program, characterized in that the computer program comprises instructions for implementing the method of any of the preceding claims 19 to 30.
A computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to carry out the method of any one of the preceding claims 19 to 30.