KR20200145669A - Motion based adaptive rendering - Google Patents

Abstract

A method of performing adaptive shading of image frames by a Graphics Processing Unit (GPU) includes: determining, by the GPU, a first shading rate based on determining that a change in a plurality of underlying assets between a first image frame and a second image frame is above a first threshold; determining, by the GPU, a second shading rate based on determining that one or more viewports in the second image frame is similar to one or more viewports in the first image frame; determining, by the GPU, a third shading rate based on determining that a quality reduction filter is used; and selecting, by the GPU, a shading rate from among the first shading rate, the second shading rate, and the third shading rate for the first image frame. The present invention provides motion-based adaptive rendering in which the number of samples rendered in a block of pixels is reduced.

Description

Motion-based adaptive rendering {MOTION BASED ADAPTIVE RENDERING}

Embodiments of the present invention relate to techniques for performing graphic processing in which the number of samples rendered in a block of pixels is reduced. More specifically, an embodiment of the present invention is directed to automatically analyzing motion and other elements in individual display screen tiles (blocks of pixels) and making a sampling decision on a tile-by-tile basis. About.

1 shows the main parts of a graphics pipeline 100 based on the OpenGL® 3.0 standard. An exemplary set of stages is a vertex shader operation stage 105, a primitive assembly and rasterization stage 110, and a fragment pixel shader operation. A fragment pixel shader operation stage 115, a frame buffer stage 120, and a texture memory 125. The pipeline operates to receive vertex data, shade vertices, assemble, and rasterized primitives, and to perform shading (shadow processing) operations on fragments/pixels.

In one aspect of the graphics pipeline 100, all regions of the image are rendered at the same minimum resolution. In particular, in a typical graphics pipeline, the sampling rate (average number of samples per pixel) is typically at least one sample for every pixel in the image.

In one aspect of a typical graphics pipeline it is wasteful and requires more pixel shading operations than desired. In particular, in order to reduce the sampling rate to less than one sample per pixel (sub-sampling/de-sampling) in local regions of the image, the graphics pipeline automatically performs strategic choices. There is no automation. In the context of mobile devices, this means that the amount of energy consumed is higher than desired.

Since the above-described information in the background art section is only to help understanding the background of the technology, it should not be interpreted as acknowledging the existence or relevance of the prior art.

It is an object of the present invention to provide a motion-based adaptive rendering in which the number of samples rendered in a block of pixels is reduced.

The graphics system adaptively renders individual parts of the frame based on the motion of underlying objects rendered relative to the camera's reference frame. In one embodiment, adaptive rendering is based at least in part on the speed of an object rendered on the screen over at least two frames. Measuring motion in screen space (through pixels) incorporates other motion sources including object motion and camera motion. If the speed of the basic motion is less than the quasi-static limit, it may be determined whether to reuse the fraction of pixels from the previous frame. In the medium rate regime, a full sampling rate is used. In at least one high-speed regime, it is determined whether to select a reduced sampling rate. The decisions can be made on a tile-to-tile basis, where a tile is a set of adjacent pixels within an image, typically in a block having a square or rectangular shape.

One embodiment of the method includes determining, on a tile-to-tile basis, the speed of pixels of objects in a current frame with respect to a previous frame. Each tile is classified into one of at least three speed categories, and the at least three speed categories include a semi-static speed category, a medium speed category, and a high speed category. The sampling decision for each tile is made based on at least a portion of the rate category associated with each tile. The sampling decision may include determining whether the tile is to be sampled at a full resolution sampling rate of at least one sample per pixel in the current frame or at a lower rate in the current frame. Then, the tiles are rendered. In an embodiment, the sampling decision may be further based on whether the tile is detected as having a high probability of including an edge of color or depth. In one embodiment, for tiles classified in a quasi-static velocity category, the method includes reusing pixel data from a previous frame by copying pixel data for at least one pixel of a previous frame to the tile. In one embodiment, for tiles classified in the medium rate regime, every pixel is sampled at least once. In one embodiment, for tiles classified in at least one high-speed regime, selection of a sampling pattern in which the number of samples is less than the number of pixels associated with the tile is performed, and interpolation is performed to determine color at non-sampled pixel locations. ) Is performed.

An embodiment of a graphics system includes a graphics processor and a graphics pipeline that includes an adaptive sampling generator and a pixel shader. The adaptive sampling generator determines a required sample rate for each tile based at least in part on the speed of the pixels of the objects in each tile and selects a sample pattern based on the required sample rate. In one embodiment, the adaptive sampling generator determines a sample pattern and sample rate for an individual tile based on a combination of the speed of objects in the individual tile and whether the individual tile contains an edge. In one embodiment, the graphics system includes a transfer unit for performing an advection, and for tiles with a speed below the quasi-static speed limit, a sample pattern with a reduced sampling rate is selected, and the transfer unit Fills the missing pixel data by reusing the pixel data from the previous frame through transfer. In one embodiment, the graphics system includes a reconstruction unit, and for tiles having a rate above a threshold rate, a reduced sampling rate is selected, and missing pixel data is interpolated by the reconstruction unit.

In some embodiments, a method of performing adaptive shading of image frames by a Graphics Processing Unit (GPU), by the GPU, includes a plurality of underlying assets between a first image frame and a second image frame ( determining a first shading rate based on determining that a change in the underlying asset) exceeds a first threshold, by the GPU, at least one viewport in the second image frame is the first image Determining a second shading rate based on determining that it is similar to one or more viewports in the frame, determining, by the GPU, a third shading rate based on determining that a quality reduction filter is to be used, and the And selecting, by a GPU, a shading rate from among the first shading rate, the second shading rate, and the third shading rate for the first image frame.

In some embodiments, the method further comprises determining, by the GPU, that the change in the plurality of underlying assets between the first image frame and the second image frame is less than the first threshold. And reusing pixel data from the first image frame to render the second image frame. In some embodiments, the method comprises, by the GPU, the first image frame to render the second image frame by reprojecting the pixel data from the first image frame to the second image frame. And reusing the pixel data from In some embodiments, the change in the plurality of underlying assets between the first image frame and the second image frame is associated with a camera and viewport.

In some embodiments, the method further comprises determining, by the GPU, that a change between the one or more viewports in the second image frame and the one or more viewports in the first image frame is less than a second threshold. And reusing pixel data from the first image frame to render the second image frame based on, the method further comprising, by the GPU, from the first image frame to the second image frame And reusing the pixel data from the first image frame to render the second image frame by reprojecting the pixel data.

In some embodiments, determining that the quality reduction filter is used determines, by the GPU, a decrease in quality of the second image frame, by the GPU, the output color of the second image frame is Determining whether it corresponds to a weighted sum of pixel data from a first image frame, and by the GPU, the output color of the second image frame is the output color of the pixel data from the first image frame. And determining the third shading rate for the first image frame based on determining that it corresponds to a weighted sum.

In some embodiments, determining that the output color of the second image frame corresponds to the weighted sum of the pixel data from the first image frame is, by the GPU, a higher order function of the second image frame ( functional). In some embodiments, a proportional relationship between a plurality of shaded samples in the first image frame and a plurality of input values for the higher order function is based on the workload of the GPU. In some embodiments, the higher order function is configured to receive the quality reduction filter as an input and determine the third shading rate for the first image frame.

In some embodiments, the weighted sum comprises the weighted sum of the pixel data of the first image frame over a plurality of input image frames exemplifying the first image frame. In some embodiments, the quality reduction filter includes one or more of a depth-of-field filter, a motion blur filter, and a smoothing filter. In some embodiments, the quality reduction filter is configured to average over multiple samples or taps from input textures rendered at the GPU.

In some embodiments, a system for performing adaptive shading of image frames includes a memory and a Graphics Processing Unit (GPU) connected to the memory, wherein the GPU includes a first image frame and a second image frame. A first shading rate is determined based on determining that a change in a plurality of underlying assets exceeds a first threshold, and at least one viewport in the second image frame is the first image frame. A second shading rate is determined based on determining that it is similar to one or more viewports in, and a third shading rate is determined based on determining that a quality reduction filter is used, and the second shading rate is determined for the first image frame. It is configured to select a shading rate from among 1 shading rate, the second shading rate, and the third shading rate.

In some embodiments, the GPU renders the second image frame based on determining that the change in the plurality of underlying assets between the first image frame and the second image frame is less than the first threshold. It is further configured to reuse pixel data from the first image frame for rendering. In some embodiments, the GPU retrieves the pixel data from the first image frame to render the second image frame by reprojecting the pixel data from the first image frame to the second image frame. It is further configured to reuse.

In some embodiments, the change in the plurality of underlying assets between the first image frame and the second image frame is associated with a camera and viewport, and the GPU comprises: the one or more viewports in the second image frame. And reprojecting pixel data from the first image frame to the second image frame, based on determining that the change between the one or more viewports in the first image frame is less than a second threshold, the second Further configured to reuse the pixel data from the first image frame to render the image frame.

In some embodiments, the GPU determines a decrease in quality of the second image frame, and determines whether the output color of the second image frame corresponds to a weighted sum of pixel data from the first image frame. And determine the third shading rate for the first image frame based on determining that the output color of the second image frame corresponds to the weighted sum of the pixel data from the first image frame. And determining that the output color of the second image frame corresponds to the weighted sum of the pixel data from the first image frame comprises determining a functional of the second image frame. Include.

In some embodiments, a proportional relationship between a plurality of shaded samples in the first image frame and a plurality of input values for the higher-order function is based on a workload of the GPU, and the higher-order function, Receive the quality reduction filter as an input and determine the third shading rate for the first image frame.

In some embodiments, the method of performing adaptive shading of image frames by a Graphics Processing Unit (GPU) is, by the GPU, one or more heuristics are applied to a pixel shader of the GPU. Determining whether a quality reduction filter is used by applying, determining, by the GPU, a higher-order function of a function in the pixel shader based on determining that the quality reduction filter is used in the GPU, Determining, by the GPU, a first shading rate of a first image frame based on the higher-order function, and rendering the first image frame based on the first shading rate, by the GPU Includes steps. In some embodiments, the rendering of the first image frame comprises rendering, by the GPU, at least a first portion of the first image frame based on the first shading rate, and the first The first portion of an image frame is a rendered subset of pixels in the first image frame.

According to an embodiment of the present invention, motion-based adaptive rendering in which the number of samples rendered in a block of pixels is reduced is provided.

In addition, according to an embodiment of the present invention, motion and other elements are automatically analyzed in individual display screen tiles (blocks of pixels), and a sampling decision is made on a tile-by-tile basis. A method and system for performing adaptive shading is provided.

1 shows a typical graphics pipeline.
2 illustrates a graphics pipeline according to an embodiment of the present invention.
3 illustrates an adaptive desampling generator according to an embodiment of the present invention.
4 exemplarily shows pixel speed considerations when performing adaptive rendering according to an embodiment of the present invention.
5 is a flowchart illustrating rendering and reconstruction options according to an embodiment of the present invention.
6A exemplarily shows dithering sampling patterns for reducing visual artifacts according to an embodiment of the present invention.
6B shows a general method of dithering sampling patterns according to an embodiment of the present invention.
7A shows an example of transfer according to an embodiment of the present invention.
7B illustrates a general method of performing transfer in a graphic system according to an embodiment of the present invention.
8 illustrates an example of using pre-computed weights for performing cubic spline interpolation according to an embodiment of the present invention.
9 shows an example of a sampling pattern related to considerations for determining pre-computed weights according to an embodiment of the present invention.
10 shows an example of a sampling pattern related to considerations for determining pre-computed weights according to an embodiment of the present invention.
11 shows a general method of adaptive desampling according to an embodiment of the present invention.
12 illustrates a general method of performing cubic spline interpolation in a graphic system according to an embodiment of the present invention.
13 illustrates a general method of performing cubic spline interpolation in a graphic system according to an embodiment of the present invention.
14 shows an example of differences between feed and spline reconstruction.
15A and 15B show an example in which different regions of a frame are adaptively rendered using different approaches based on the magnitude of the per-pixel rate.
16 illustrates an example of using transport for stereoscopic rendering according to an embodiment of the present invention.
17 illustrates adaptive rendering to which fobited rendering is applied according to an embodiment of the present invention.
18 shows an exemplary GPU pipeline.
19A and 19B show an example of reusing (or reprojecting) data from another viewport.
20 is an exemplary flow chart illustrating a shader analysis method for detecting a quality decrease in a rendered image and calculating a shading rate image.
21 illustrates an exemplary method of performing adaptive shading of image frames by a Graphics Processing Unit (GPU).
22 shows an exemplary arrangement of pixels.
23 shows an exemplary pixel mapping.

Overview of an exemplary graphics pipeline system

2 shows a graphics pipeline 200 according to an embodiment of the present invention. The graphics pipeline 200 may be implemented using a Graphics Processing Unit (GPU) including graphics hardware. The graphics pipeline 200 includes several new stages (but stages) and functions to assist in automatically determining regions of the frame, and regions of the frame provide an acceptable viewing experience for a human user. It does not require that all pixels in individual tiles (blocks of pixels) be sampled and rendered to achieve. As used herein, a tile is an adjacent set of pixels within an image, typically within a block having a square shape. The term frame is generally used to describe a set of operations performed to render an image read by a display at a preset frequency. However, the term frame is also used to refer to a rendered image derived from a set of actions used to render the image.

In one embodiment, the adaptive desampling (AD) sample generator step 205 is provided to assist in adjusting the sampling pattern in local regions of the image, where the local region is a block of pixels (e.g. For example, a block of 4 x 4 pixels, 16 x 16, or another size). Desampling is the reduction in the number of samples per tile sampled and rendered in the current frame. For example, desampling may involve on average less sampling and rendering than one sample per pixel in a tile, and thus is also described as sub-sampling. To maintain full image resolution, two different approaches can be used to obtain values of the missing pixel data. The reconstruction and transfer step 210 supports two different options to reduce the number of pixels that need to be sampled and rendered in the tile while maintaining the user's visual quality. The reconfiguration and transfer step 210 includes a reconfiguration submodule 211 and a transfer submodule 212. In one embodiment, a first option to reduce the number of rendered pixels in a tile is to reconstruct the tile through higher-order polynomial interpolation and filtering to generate missing pixel data for that tile. A second option to reduce the number of rendered pixels in a tile is advection, which is to identify the location of one or more pixels in the previous frame and from the previous frame for a selected fraction of pixels in the tile. It involves reusing the pixels of

In one embodiment, pixel data 215 of frame n of objects 220 from frame “n” is stored for possible reuse of pixel data in next frame “n+1”. In addition, vertex coordinate data is stored for use in determining a frame-to-frame motion vector of pixels. In one embodiment, pixel data and vertex coordinates from frame n are stored in a buffer memory for use in the next frame n+1.

3 shows an AD sample step 205 according to an embodiment of the present invention. In an embodiment, desampling decisions may be made in local tile regions based on speed and edge detection (eg, edge detection in depth/Z). The velocity buffer 310 receives per vertex coordinates from the current frame and from the previous frame. The speed of an individual pixel may be determined by comparing the vertex coordinates of the pixel in the current frame with the vertex coordinates of the pixel in the previous frame. In one embodiment, the forward splatting approach is used by rendering a “velocity image” with primitives in the scene and using the per-vertex velocity as a vertex attribute. Many graphics applications render the Z-buffer with a technique that reduces the number of pixel shader instances during rendering passes. The velocity buffer/image can be rendered as a Z-buffer. During the Z-pass where the Z/Depth buffer is created, in addition to splatting and updating the depth, the speed is also updated for each pixel. Rendering the speed buffer creates per-pixel speed values in screen space, the size of which corresponds to speed. Thus, a tile, such as a 4 x 4 tile, has a pixel rate associated with each pixel. Thus, the tile has a maximum pixel rate, a mean pixel rate, a median pixel rate, and a minimum pixel rate. In one embodiment, the average pixel rate is used to make desampling decisions, but more generally a maximum pixel rate or an average pixel rate may also be used.

Visual artifacts in moving objects are less perceived by the human eye. Thus, one factor as to whether the sampling rate in the tile can be reduced is whether the rate exceeds the threshold rate.

However, certain types of visual artifacts tend to be more perceptible at the edge of the color. Strictly speaking, it is impossible to detect color edges in the final image without first rendering the image. However, prior to rendering, it may be possible to detect a high likelihood of an edge of color. That is, in one embodiment, the edge detection module 305 detects the possibility of an edge of color in local blocks of pixels. That is, by assuming that there is a high likelihood of color fluctuation across objects, areas with high likelihood of an edge of color are detected. In one embodiment, the Z values from rasterization of the current frame are analyzed to perform edge detection. The laplace edge detector may be defined as a stencil centered on the current pixel. Any pixel in the tile is marked as having an edge if the laplacian of the Z-buffer in the pixel is greater than the threshold multiplied by the Z-value in the pixel. This defines one bit value per tile. More generally, any type of edge detection can be used.

In one embodiment, an edge mask is generated for an individual tile and an edge status bit may be generated to indicate whether the tile includes at least one edge. In one implementation, more generally other tile sizes may be used, but an edge mask is created for each block of 4 x 4 pixels. Information about the velocity and the presence of an edge is used by the sample generator 315 to determine the sample pattern of the tile. In one embodiment, when an edge is detected, the full sampling resolution is used. If no edge is detected and the tile has a rate greater than the first threshold rate, then the first reduced sampling rate is used. If no edge is detected and the tile has a rate exceeding the second threshold rate, the second reduced sampling rate is used. Other additional optional factors may also be considered in making the sampling rate determination. In one embodiment, the sample pattern options are full sample resolution (at least one sample per pixel), one-half (1/2) resolution (half of the pixels sampled in each tile), and one-quarter ( one-quarter, 1/4) resolution (one of 4 pixels sampled from each tile). More generally, multiple sampling rates may be provided that are controlled by threshold parameters for each sample rate. Also, the selected sample rates can be optimized for the selected block/tile size. Thus, the exemplary embodiment includes 3 sample rates of 4, 8, and 16 samples for 4×4 blocks, but the approach is of the sampling rates each controlled by the threshold parameters for each sample rate. It can be changed based on block size or other considerations having a set. Thus, the number of sampling rates, N, may be greater than 3 depending on implementation details such as block/tile size and other factors.

In one embodiment, a dithering module 320 is provided to adjust the sampling pattern from the selection of sampling patterns having the same effective sampling rate. Dithering may be a repetitive sequence (eg, sample pattern 1, sample pattern 2, sample pattern 3, sample pattern 4) or may include aspects of randomization.

Dithering of the sampling pattern by the dithering module 320 reduces visual perception of sampling artifacts by human users. The human eye and human brain begin to blend images into a video sequence when the rate is faster than the biological threshold. That is, when images change at a rate faster than the biometric threshold, the human eye blends the images over time and perceives them as a continuously changing sequence similar to a video. There is some debate about the exact number of biometric thresholds. At a frame rate of about 12 frames per second, the human eye and brain begin to see a sequence of moving images instead of individual images. However, a rather high frame rate of about 15 frames per second is required to experience relatively fluid (non-rumbling) beginnings of movement. However, the nature of the underlying images is also an additional factor as to whether a human observer will perceive fluid motion at a given frame rate. Thus, the human eye tends to average dithered visual artifacts at a frame rate of about 12 frames per second or more. In one embodiment, the dithering is performed such that every pixel is rendered at least 15 frames per second, which is faster than the human eye can identify individual images. At 60 frames per second, dithering the sample pattern in the tile every 4 frames corresponds to rendering each pixel at least 15 frames per second.

Example motion speed regimes

4 shows examples of speed regimes according to an embodiment of the present invention. Motion is a combination of object motion and camera motion. The velocity corresponds to the size of the motion vector in the tile. In this example, the speed is an indicator of the number of samples required in a block of pixels to have an acceptable visual quality. If the motion exceeds a certain threshold speed (K _fast1 ) for a block of pixels, the human eye cannot perceive high frequencies in the moving object, so the number of samples decreases (e.g., 8 samples in a 4 x 4 tile). It may be an indicator that it is). If the speed exceeds the higher threshold speed (K _fast2 ), it may be an indicator that the number of samples in the tile is further reduced (eg, 4 samples in a 4 x 4 tile). On the other hand, if the motion in the tile is very slow, below the velocity (K _stat ) (or no motion), then reuse the pixel data from the previous frame (e.g., via transfer, reuse 8 color values from the previous frame There may be an opportunity to render 8 samples in a 4 x 4 tile). Reuse of pixel data from the previous frame also requires a graphics state that does not change from the previous frame to the current frame, and the graphics state includes the shaders used, the constants provided to the shaders, and the geometry provided to the frames. . There may be rate regimes that require full sampling resolution. For example, there may be an intermediate rate regime between K _stat and K _fast1 , and full sampling resolution may be required to achieve high visual quality in the medium rate regime. In addition, there may be scenarios in which super-sampling is applied to individual tiles. As an exemplary embodiment, an option may be provided to support super-sampling in a Z-edge case.

In one embodiment, _desampling (changing the sample pattern to reduce the sampling rate to less than one sample per pixel) is allowed if the rate exceeds a first threshold rate (K _fast1 ). In an embodiment, the sampling rate may be further reduced when the rate exceeds the second threshold rate K _fast2 . The decision as to whether to perform desampling may further depend on other conditions such as whether an edge has been detected.

In one embodiment, motion in camera screen space is obtained by distinguishing vertex position data from the current frame and the previous frame. The tile velocity regime is classified on a tile-to-tile basis by calculating the size of a motion vector based on how far a pixel of an object has moved from one frame to another. As described above, in one embodiment, splatting is used in the Z-pass to determine per-pixel motion vectors. In one embodiment, velocity thresholds are defined and used as inputs to determine whether adaptive desampling or transfer will be used in the current frame. A velocity regime is a quasi-static regime in which an object moves slowly enough so that the pixels of the object do not differ significantly from their previous image counterparts. If the speed is within the quasi-static speed limit, a decision can be made as to whether or not the transfer will be used to reuse pixels from the previous frame. In one embodiment, the upper limit of the quasi-static speed, K _stat is that pixels in a given tile (tile m) in frame n are kept on the same tile in frame n+1. In one embodiment, if the rate is less than K _stat , additional checks are performed to determine whether pixels from the previous frame can be used in the current frame. This may include checking whether the transfer in the previous frame produced acceptable results. Also, a check can be performed to check if the pixel values for the tile in the current frame match a small shift over the previous frame, which can be described as a discrepancy check. The transfer mismatch status bit may be associated with the tile, which may indicate that the tile has passed one or more discrepancy checks to ensure that the tile is suitable for transfer of at least some of the pixel data.

5 is a flow chart illustrating exemplary adaptive rendering selections based on speed, edge detection, dithering, spline reconstruction and transport. Some common graphics pipeline features are omitted for clarity. 5 shows specific examples in which 4 x 4 tiles according to an embodiment of the present invention are used. A color pass may be performed after an initial pre-pass is performed to render the pixel data. The scene geometry 505 of the image is provided by the application. The Z-buffer is calculated (510) and edge detection is performed (515). Motion vectors are computed 520 for the scene geometry. A per-pixel motion vector is calculated (525). The range of motion in the tile is calculated 530. Based on this information, a decision 535 is made as to whether to 1) render 4, 8, or 16 samples in a 4 x 4 block and perform interpolation, or 2) render 8 and transfer 8. I can. Dithering 540 is performed on the sampling patterns. Spline reconstruction 545 is used to reconstruct the pixel data. If transfer is used, transfer 550 is used to obtain 8 of the pixel values, and the rest is obtained by rendering.

Example sampling patterns and dithering

6A shows an example of sampling patterns and dithering. In this example, the tile size is a block of 4 x 4 pixels. The full resolution corresponds to 16 samples. Half-resolution (8 samples) and one-quarter resolution (4 samples) allow a change in the pattern of samples. That is, in the case of 8 samples, the arrangement of samples may have a first sample pattern, a second sample pattern, a third sample pattern, and the like. Having pre-defined sampling patterns supports dithering of the sample pattern for temporal color averaging. Pre-defined sampling patterns are selected to rotate the sampling so that every pixel position is rendered every few frames. Dithering of the sample pattern can be achieved by other techniques. In an embodiment, selection of a sample pattern in an individual frame may be selected by the dithering module 320 in a sequence by a modulo k counter. Dithering the sample positions over time in multiple frames renders the error harder for a human observer to see. In one embodiment, the sample patterns are selected to ensure that each pixel is rendered at least once every k frames, where (n*n)/k is the minimum number of samples per n x n tile. In another embodiment, temporal dithering is implemented using a stochastic approach to selecting a sample pattern.

6B illustrates a method of dithering according to an embodiment of the present invention. The tiles of the current frame are selected 605 for sub-sampling at a reduced average sampling rate. For each tile, the sampling pattern is selected 610 to change over the previous frame. Rendering and reconstruction is performed (615). If additional frames have to be rendered, the process continues.

Transfer example

7A shows an example of transfer. In a tile area, such as a 4 x 4 tile 700, the transfer involves copying pixel data from a pixel at a given location in the previous frame to a corresponding location in the current frame. For example, an individual object (for example, a ball slowly moving across the ground) can move across the screen so that all pixels of the ball move at a certain speed. In this example, there is a high level of temporal coherence between the pixels of the ball slowly moving from one frame to another. In this case, the changes are primarily motion. By determining the motion of individual pixels of the ball across frames, pixel data can be copied across frames. In this example, the motion is slow enough so that the pixel data can be mapped from the current pixel location to a pixel in the same tile in the previous frame. The position of the pixel in the previous frame can be calculated as x(n-1) = x-mv(x), where mv(x) is a motion vector. Consequently, this allows pixel data to be copied from x(n-1) to x(n). That is, if the movement (motion) of pixels between frames is small, the pixel position of the current frame is now projected back to the pixel of the frame, and pixel data from the previous frame is copied. If x(n-1) has decimal elements, a bilinear or arbitrary high-order interpolation method can be used.

In the example of Fig. 7A, transport is mixed with rendering. In one embodiment, transfer is used for half of the pixels 705 in the tile and the other half of the pixels 710 are rendered. Mixing transport and rendering in a single frame reduces the visual artifacts associated with performing only transport. That is, the possibility of visual errors due to transport detectable by a typical human observer is minimized. In conjunction with temporal dithering, this ensures that errors do not accumulate over time, thereby reducing the likelihood of visual errors that will be found by typical human observers. The 1:1 ratio of rendered pixels and transferred pixels is one option, but more generally other ratios may be used.

As mentioned above, in one embodiment, the maximum speed is used as a condition as to whether or not transfer is allowed. In one embodiment, the criterion is that the critical speed is sufficiently low that the local transformation of small neighboring pixel positions can be classified as a rigid transformation, and the change in the position of the pixels in the rigid transformation is the entire set of pixels. It can be expressed with the desired accuracy using either transform and rotation for. For example, the maximum speed for transport may be that the magnitude of the pixel movement is less than a threshold value of k pixels. Rigid body transformation can occur at any speed, but the likelihood decreases as the speed increases, so that the speed threshold can be used as a criterion for cases where feed is advantageous. A discrepancy check can be performed on individual tiles to determine whether the transfer produces acceptable results. This discrepancy check can be performed in the current frame and recorded as a 1-bit value for each tile.If the check indicates that the transfer results are inaccurate, a decision on whether to disable the transfer to the neighbor of the tile that failed the discrepancy check is made. It can be done in the next frame. That is, in this implementation, the transfer is performed on the tile of frame n, and the discrepancy check is performed on frame n and consumed (used) by frame n+1. Then, frame n+1 uses a discrepancy check (calculated in frame n) to determine whether to perform a neighbor transfer for the tile in frame n+1. If the discrepancy check in frame n indicates that the transfer result was acceptable, transfer is allowed in frame n+1. Otherwise, the transfer is turned off for the selected number of frames. The discrepancy check is checked based on whether there is a large change in the pixel values of the tile that do not match the underlying assumptions of valid transport. If the object's pixels move slowly, the tile is not expected to change significantly between the two frames. That is, if the state of the tile changes significantly, the discrepancy check fails. The tile state mismatch bit (eg, 0 or 1) can be used to indicate whether the mismatch check has passed. The degree to which a tile state change is allowed can be determined heuristically or empirically, for example, based on a trade-off between computational benefits of transport and minimization of the occurrence of visual artifacts.

Other methods of performing mismatch checks may be used. There are computational advantages of performing the transfer in the tile in the current frame n, performing a discrepancy check, and then deciding whether to perform the transfer in the frame n+1 using the discrepancy check. However, it will be appreciated that an alternative implementation of the discrepancy check could be used, i.e. the discrepancy check is performed in frame n and can be used to reuse the pixels from the previous frame by determining whether or not the transfer will be used in frame n. .

If desired, accuracy can be improved using various enhancement functions. In one embodiment, back and forth error correction and compensation (BFECC) may be used. BFECC uses the position determined from the Semi-Lagrangian advection, and adds the velocity at the corresponding coordinates to get a new position in the current frame. If there is no error, the coordinates should be the same as the original position (x, y). Otherwise, by subtracting half of the error from (x-vx, y-vy), assuming that the speed of the pixel is correct, a second-order accurate estimate of the position, accurate to half the pixel, is obtained.

7B shows a general method of performing transfer according to an embodiment of the present invention. A determination 1405 is made as to whether the tile is suitable for transport. Suitability is based on whether there is a speed range within the quasi-static range, reinforced by passing any additional discrepancy checks. If the tile is suitable for transport, a decision 1410 is made in the block of corresponding pixel locations in the previous frame. The selected portion of pixels from the tile of the previous frame is reused (1420). The remaining pixels are rendered 1425.

Image interpolation and reconstruction examples

8 shows an example of image interpolation and reconstruction of pixel color values for a desampling situation. In one embodiment, the weighted sum of color values is used to reconstruct the unrendered pixels. When a given weight function w is selected, a set of normalized weights for each configuration of pixels generated in a specific sampling pattern can be calculated in advance. For example, if 4 pixels are rendered in a 4 x 4 block, the remaining 12 pixels may be expressed using a weighted sum of pixels rendered in the same block as well as neighboring blocks. Further, since the set of possible pixel configurations in neighboring blocks is limited by the set of sampling patterns, in this case, all possible sets of weights can be calculated in advance.

Traditionally, GPUs use bilinear interpolation. However, bilinear interpolation has various disadvantages. In one embodiment, higher order polynomials of at least third order, such as a piece-wise cubic polynomial (also referred to as a cubic spline), are used for efficient reconstruction of sparse samples.

Higher order polynomials, such as cubic splines, can map a higher frequency spectrum than bilinear interpolation, and provide high accuracy (fidelity) of reconstructed data from sub-sampled blocks. Also, with bilinear interpolation, samples are preferred for both sides of the pixel, since the linear interpolation of one side may be inaccurate and exceed the color spectrum range. In contrast, higher order polynomials using wide support (more than one pixel apart) are more likely to accurately approximate the functional, high-order function form of the rendered image data. Although various higher-order polynomials can be used, cubic splines have better continuity properties than quadratic polynominals. Due to the edge-detection step performed prior to desampling, a tile undergoing reconstruction will not have abrupt discontinuities, and higher order polynomial reconstruction may perform poorly at abrupt discontinuities.

According to an aspect of performing sub-sampling, there is sparse sample data at runtime. In an individual block area, such as a k x k pixel area, desampling can derive a subset of rendered pixels, such as 4 or 8 pixels from a block of 4 x 4 pixels. The missing pixel data needs to be reconstructed. As a result of having predetermined sample patterns, there is a finite set of possible sample positions. In this way, a fixed set of local stencils is created, stored, and used to reconstruct pixel data using cubic splines or other higher order polynomials prior to runtime. Common methods of evaluating higher order polynomials in hardware are computationally expensive. In contrast, in embodiments of the present invention, the use of a fixed set of precomputed stencils eliminates computational overhead during runtime, performing general higher order polynomial evaluation. The use of a static set of samples allows the determination of possible configurations of pixels that may need to be reconstructed, so the necessary stencils can be pre-computed.

In one embodiment, higher order polynomial interpolation is implemented as static stencil operations using pre-computed weights. In one embodiment, the table of stencils is stored and made available for spatial reconfiguration for the reconfiguration submodule 211 in the reconstruction and transfer step 210. The table of stencils provides weights based on known sample locations. In one embodiment, the table of stencils has all pre-computed stencil weights for each pixel location within a defined sample pattern. The pre-computed weights can cause higher order polynomial reconstruction to be performed using static stencil operations.

In one embodiment, a set of 5 x 5 stencils for all possible pixel positions in a tile (eg, 4 x 4 tile) that may need to be interpolated during runtime is determined. Each 5 x 5 stencil is computed for each pixel location and neighboring configuration. Each stencil provides a list of weight values and corresponding locations of sample points. The stencils are stored in a constant memory table available for reconfiguration purposes for the reconfiguration submodule 211 of the reconstruction and transfer step 210. In one embodiment, at runtime, for each pixel that needs to be interpolated, an index is computed into this table using the pixel coordinates and the sampling mask. In one implementation, each stencil is addressed using (a) the location of the pixel within the tile and (b) the sampling mask used for rendering. Thus, if dithering is used, the stencil selected will depend on the selected sample pattern for a given degree of subsampling.

In one embodiment, higher order polynomial interpolation is performed using a multiplier/adder to accumulate the products of weights and sample color values. The accumulated values are normalized by division, which in many cases can be done by bit shifting (movement) in integer format, or by subtraction of floating point format. Thus, the use of pre-computed weights and stencils allows higher order polynomial interpolation to be computed at runtime with relatively little computational effort.

An example of a cubic spline function used to calculate and reconstruct pixel colors as a weighted sum of known pixel color values is as follows.

In one embodiment, the formula representing the weighted sum to determine the pixel color value is based on the following weights w().

In this case, c(i, j) is the color value at the pixel position (i, j), w() is the two-dimensional spline function, and “Filled” is the set of rendered pixels. The 2D spline function is the product of two 1D spline functions, or w(i, j)=k(i)k(j), where the 1D spline function k() is based on the cubic filter formula, and the cubic filter formula The paper by Don P. Mitchell and Arun N. Netravali, “Reconstruction Filters in Computer Graphics,” Computer Graphics, Volume 22, Number 4, August 1988, pp. It is stated in 221-228 as follows:

In the paper by Mitchell and Netravali, distances are defined in scaled pixel space as:

By limiting the relative position of the sample points, the weights and denominators can be pre-computed into stencils. Since the spline function is defined in a limited way, scaling of the size of x can be used to extend the functions to a desired support radius, such as two pixel support radii.

For a tile of size n x n, it is possible to arrange a k x k square in possible configurations of (n/k)*(n/k). Since s squares are required at a sampling rate of 4*s, sampling patterns of (n*n)/(k*k*s) occur.

9 shows an example of a sampling pattern in a 4×4 tile where Xs mark rendered samples and O marks the interpolation position. A 5 x 5 stencil centered on O is used. Assuming that any access outside this 4x4 tile is not valid, the stencil has weights of 0 for any locations outside the 4x4 tile, which are removed from the stencil table. Assuming the top left pixel is (0, 0), then the table entry is (0, 0), (2, 0), with appropriate weights (w0, w1, w2, w3) and normalization factor (w) Read the required positions as, (0, 2), (2, 2). Then, the weighted sum is 1/w (w0*c(0, 0) + w1*c(2, 0) + w2*c(0, 2) for each color component using multiply-and-accumulate motion. ) + w3*c(2, 2)). However, more generally, the reconstruction is not limited to one tile, but the area of influence of the stencil can also be extended to neighboring 4 x 4 blocks.

Assuming a 5 x 5 stencil, there are all 24 values that need to be pre-calculated (the pixel itself has no color value, so the center is always 0). Of these, if 8 samples per 4 x 4 tile are used, up to half can be rendered, leaving 12 values. In one embodiment, each stencil contains a 4-bit count of the number of non-zero weights, followed by 8-bit weights stored in one chunk, and 3- for x and y coordinates for the center. Two chunks of bit coordinate offsets follow.

In one embodiment, the stencils are stored in order of the sampling patterns. In one embodiment, different sampling patterns for the same sampling rate are rotations of each other, so there are two sets of patterns. With a list of indexes pointing to the data for pixels (i, j), these can be stored in row major order within 4 x 4 tiles. For rotations of the sampling mask, the coordinates are converted appropriately.

Referring to FIG. 10, a case of a 4 x 4 tile of pixels in which 8 samples out of 16 possible samples are rendered is considered. In this example, stencils are defined for each unknown pixel given a weight function. These stencils can be retrieved at runtime from a set of pre-defined stencils. In the exemplary case of cubic stencils with a supporting radius of 2 pixels, the size of these stencils is 5 x 5 unless super-sampling is performed. If accesses to the k x k tile region must be essentially restricted, the stencils can be appropriately modified to have weights of zero for those pixels that fall out of the tile. It is important to note that the number of samples need not be less than the number of pixels. In areas where super-sampling is required for anti-aliasing, the number of samples may exceed the number of pixels (eg, 32 samples for a 16 pixel 4 x 4 tile). Appropriate pre-computed stencils will be added for these cases.

In one embodiment, each sampling pattern is defined as a combination of sparse square patterns (eg, four samples to be rendered as a square pattern). Choosing square patterns is useful in applications where a group of 4 pixels (quad) is the default processing unit. However, more generally, other arrangements of sampling locations may be used in the sampling patterns. In one embodiment, the sample patterns are squares of size 3 x 3 in 4 x 4 tiles. Thus, adjacent vertices are 2 pixels apart along each axis.

In one embodiment, the same sampling pattern is used for all regions of an individual frame that are sub-sampled at a given sampling rate. In this embodiment, the same sampling pattern is used in all tiles sub-sampled at a given sampling rate, because the spacing of the sample positions in every frame is kept constant to simplify the reconstruction routine.

In one embodiment, the sampling patterns are based on quads to utilize single instruction multiple data (SIMD) processing units. The consistent spacing of the samples provides robust interpolation and helps to achieve full pixel resolution in the final image.

11 shows a general method of adaptive desampling and spline interpolation according to an embodiment of the present invention. A determination is made 1505 for the presence of edges and a determination of whether the speed range of the tile is within the speed range for sub-sampling. A determination 1510 of the sub-sampling rate and the selected sample pattern is made. Pixels of the tile are shaded 1515 based on the sampling pattern. Reconstruction is performed 1520 to interpolate the missing pixel values for which spline interpolation may be performed.

12 illustrates a method of performing cubic spline interpolation according to an embodiment of the present invention. Tiles are selected for sparse sampling (1705). A sample pattern is selected (1710). Pixels are rendered 1715 for the sampled locations. Reconstruction of missing pixel data is performed through cubic spline interpolation based on the pre-computed weights (1720).

13 illustrates a method of using stencils including pre-computed weights according to an embodiment of the present invention. Pre-computed weights are generated (1805) for each missing pixel location in the sample pattern. A stencil containing pre-computed weights is stored 1810. The stored stencil is accessed 1815 during runtime. The accessed stencil is used 1820 to perform cubic spline interpolation.

Comparison of examples of transfer and reconstruction

14 shows an example of aspects of transport and reconstruction through cubic splines. The tile size is 4 x 4 tile size. The pixel pattern in the previous frame is a checkerboard pattern. The rendered pixel values are denoted by R. In the example on the left, the transfer is performed to reuse half of the pixel color data from the previous frame in a 4 x 4 tile. The speed associated with the tile is very slow, and half of the pixels are transferred by copying them from the pixel values of the previous frame. Arrows for four pixels are shown to indicate reuse of pixel data from the same tile in the previous frame. In this case, color information is copied without color bleeding. In the example on the right, there is a significant tile rate that corresponds to half-pixel displacement per frame. In this example, the reconstruction is performed based on cubic spline interpolation. The speed according to x0.5 pixels ensures that each rendered pixel has an exact gray color between black and white. Therefore, the reconstructed pixels have the same value. That is, the color values are correct, and full resolution rendering also produces the same values.

Example of automatic tile-to-tile adaptive rendering

15A shows an example of a frame in a scene where the pixel rate is different from other areas and some areas have areas including color edges. By way of example, the scene includes static objects such as plants (vegetation) slowly moving in the wind as well as riders on a motorcycle and semi-static objects. Thus, there are areas that can be classified into different speed regimes. As a result, as indicated by the boxes in Fig. 15B, different regions of the scene have different pixel velocities, and some regions provide different opportunities for adaptive rendering. As a result, in individual frames, the system automatically analyzes individual tiles, desampling and transports on a tile-to-tile basis, whether to perform desampling and cubic spline interpolation, or a general default sampling scheme. Decide whether to use it. Individual decisions can also be made as to whether or not to perform super-sampling on a tile-based basis. Since the system automatically performs this optimization, no special inputs from the application developer are required, assuming that the relative parameter values are each defined.

Stereoscopic rendering example

Embodiments of the present invention may be used to create a single (non-stereoscopic) display. However, stereoscopic rendering can also be applied to virtual reality applications. Referring to FIG. 16, a case in which separate images are generated for each eye corresponding to a left-eye image and a right-eye image is considered. Transfer can be used to improve the efficiency of stereoscopic rendering. In one embodiment, the left image is created. Translation Motiontrans is defined as a translation that converts a part of a left-eye image into a right-eye image. In one embodiment, the sample generator (pseudo) decision augments the sampling (pseudo) decision for the right image to attempt to transfer pixel values from the left image. In one embodiment, the sampling is Z-based, and a test is performed as to whether the minimum Z of the left image and the right image is greater than the threshold Z. If the minimum Z of the left image and the right image is greater than the threshold Z (min(Zleft, Zright)> Zthresh), then pixels are transferred from the left frame to the right using a translation motion (motiontrans). Otherwise, rendering is based on a motion based sampling rate. As shown in Fig. 16, this causes the right eye image to be a combination of transferred pixels and rendered pixels from the left eye image.

Foveated rendering with adaptive rendering

17 illustrates an embodiment in which adaptive rendering is applied to fobited rendering. The structure of the human retina of the eye has a portion of the fovea that provides the highest visual acuity in a healthy human eye. The highest visual acuity of a healthy human eye is within a small angular cone and falls with increasing angular distance. Fobited rendering renders high detail near where the user is looking and lowers the detail as it moves away from the focus point. 17 shows the focal point (x, y) 1725. The sampling rate decreases as the radial distance from the focal point increases (eg 1/(distance from focus)). The reduction can be done in a stepwise manner at a certain radial distance. For example, a specific number of samples may be rendered from a circular area 1720 to a radius distance r0 1715. Fewer samples are rendered in the annular region 1710 from r0 to r1 (1705). A much smaller number of samples are rendered in an area with a radial distance greater than r1. As an exemplary embodiment, 16 samples may be rendered in the region between (x, y) and r0, 8 samples may be rendered in the region between r0 and r1, and 4 samples may be in the region after r1. . More generally, other radially varying sampling functions may be used.

While the present invention has been described in connection with specific embodiments, it will be understood that the invention is not limited to the described embodiments. On the contrary, this is intended to include alternatives, modifications, and equivalents, as included in the technical spirit of the present invention as defined by the following claims. The present invention may be practiced without all or part of these specific embodiments. In addition, well-known features may not be described in detail in order not to unnecessarily obscure the present invention. According to an embodiment of the present invention, components, process steps, and/or data structures may be implemented using various types of operating systems, programming languages, computing platforms, computer programs, and/or computing machines. I can. In addition, those of ordinary skill in the art to which the present invention pertains recognizes hardware devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or such devices. In addition, these devices can be used without departing from the scope of the technical idea disclosed in the present specification. Further, the present invention can be explicitly implemented as a set of computer instructions stored on a computer readable medium such as a memory device.

Example of adaptive rendering using intra-frame and inter-frame information

Pixel shading (PS) may be defined as calculating the color of each pixel of an image in a display. For example, in computer graphics, a pixel shader (or fragment shader) may be a computer program that helps to detect the color, brightness, contrast, and/or other characteristics of a single pixel. In exemplary embodiments of the present disclosure, “pixel sampling” and “pixel shading” may mean the same operation and may be used interchangeably.

Graphics processing units (GPUs) that perform pixel shading operations are increasingly limited depending on their energy consumption. As discussed with respect to FIG. 1, pixel shading may be an expensive operation with respect to energy consumed during operations performed for pixel shading operations. Limiting shading cost in real-time rendering applications can be difficult, so it may be desirable to reduce the number of pixel shading (PS) invocations during rendering. Some GPUs may have at least 1 PS call per pixel, which may be wasteful with regard to the GPU's energy consumption and performance. For example, for each pixel on the screen, pixel shaders can be called to calculate the color of the pixel.

The shading rate may refer to the resolution at which pixel shaders are called. In some cases, the shading rate may be different from the full screen resolution. A high shading rate can provide higher visual accuracy, but can increase the cost associated with the GPU (eg, processing power, energy consumption, etc.). On the other hand, a low shading rate can provide low visual accuracy at low GPU cost. During rendering, if a shading rate is set, the shading rate is applied to all pixels in the frame. However, all pixels in a frame may not require the same level of visual accuracy (eg, pixels that have not changed significantly in the current frame compared to the previous frame).

However, some GPUs may have additional functionality that allows applications to reduce the shading rate (eg, the sampling or shading rate reduction described in connection with FIG. 4 ). In some cases, GPUs of the related technology also allow applications running on the GPU (e.g. rendering applications) to provide a shading rate, or to define methods, and by means of methods to a GPU (e.g. For example, the software (SW) layer of the GPU) may calculate a shading rate used for pixel shading. For example, some methods of related technology can selectively reduce the shading rate in areas of the frame that do not affect the visual quality, so that additional performance enhancement can be achieved, and as a result, GPU cost can be reduced. . However, the methods of these related technologies that allow the application to provide a shading rate or define methods to calculate the shading rate may not be automatic or transparent, and the application running on the GPU (e.g., rendering application) You may be required to select them explicitly.

The ideal number of pixel shading calls can be determined by Nyquist sampling of the image signal. For example, according to the Nyquist sampling of the image signal, the number of samples for accurately reconstructing the image must be at least twice the maximum frequency in the image signal.

Some embodiments of the present disclosure may provide hardware and software additions to the GPU that may be required to approach Nyquist optimality in rendering. In addition, some embodiments of the present disclosure may provide not only methods of controlling the shading rate of an image, but also heuristics and mechanisms for automatically controlling the shading rate to minimize shading costs while maintaining final image quality.

Some embodiments of the present disclosure may use sets of heuristics, and the sets of heuristics are software (SW) that may use reverse and/or forward error analysis within a frame to determine a desired shading rate for various shading operations. And/or hardware (HW). For example, a reverse error analysis approach can be used to determine scenarios where it is not necessary to shade all samples, e.g., a reduction in shading requirements, e.g., Depth-of-Field (DoF) , Motion blur, camera blur, or other similar quality reduction steps in the rendering pipeline. For example, Depth-of-Field (DoF) can reduce image quality, so fewer shaded samples may be required to achieve similar quality. In this case, a combination of driver-compiler interaction can be used to reduce the shading rate.

For example, as discussed in connection with FIG. 7A, in one embodiment, the front and rear error correction compensation (BFECC) may use a position determined in a Semi-Lagrangian advection and a new position in the current frame. You can add the velocity at that coordinate to get it. If there is no error, the coordinates should be the same as the original position (x, y). Otherwise, subtract half of that error from (x-vx, y-vy), assuming the velocity is correct (e.g., the pixel is correct), a second-order accurate estimate of the position, accurate to half the pixel. Can be obtained.

For example, some embodiments of the present disclosure are similar to any image in the current or previous frame (e.g., similar to the embodiments discussed with respect to FIGS. 2, 4, 7A, 7B, and 14). ), a set of heuristics can be used to reuse (or reproject) some or all of the pixel (or rendered) data. For example, some embodiments of the present disclosure may detect, within a frame, that an image to be rendered uses one or more viewports that are substantially (or sufficiently) similar to one or more viewports of another image, or within an image. , One or more of the viewports may detect that they are similar to each other, or that if they exceed a certain distance from the camera, it may be established that the rendered results will be sufficiently identical to these similar viewports (e.g., similar embodiments are shown in FIG. To 17). Further, some embodiments of the present disclosure may reuse (or reproject) some or all of the pixel (or rendered) data from any image in the current or previous frame based on the determination of similarity of viewports.

Some embodiments of the present disclosure reuse a portion of pixel data from any image in the current or previous frame to collectively reduce the number of shading samples by hardware and software analysis using hardware for API features. Reverse and/or forward error analysis can be combined within a frame to determine the desired shading rate for various shading operations. For example, some embodiments of the present disclosure may track objects across frames using application assistance with driver resource tracking to reuse intermediate rendered results from one or more previous frames that aid in rendering of the current frame. . Moreover, some embodiments of the present disclosure, across frames, in which certain regions within certain images contribute to the final displayed image (eg, also including certain images), do not change across frames, and are out of date. It can be detected that it can be reused (or reprojected) from the rendered copies.

Fine-grained shading rate control has been discussed, for example, in connection with the exemplary embodiments of FIGS. 3, 4, and 5 of the present disclosure. The exemplary embodiment of FIG. 18 shows the concept and functionality of the exemplary methods of selecting the reduced sampling rate described above with additional heuristics to further reduce the rendering cost (e.g., FIGS. 3, 4, 5, 11, and With respect to 17, for example, choosing a reduced sampling rate based on speed and edge detection, and in some cases (e.g., increasing the radial distance from the focal point, in Figure 17) are practiced. . In addition, the exemplary embodiment of FIG. 18 (e.g., an exemplary method of generating a shading rate image) may be coded with API extensions, which may be any, e.g., a block of rendered 4 x 4 pixels. It can be attached to any image to control the number of pixels shaded for.

18 shows an augmented GPU pipeline that uses exemplary sets of heuristics to reduce the rendering work performed in both the shading and fixed function steps of the GPU pipeline by reducing the shading rate. In the exemplary embodiment of FIG. 18, the detection and calculation of the shading rate image may be performed by any combination of hardware and/or software components of the GPU.

The GPU pipeline of FIG. 18 includes an input stage 1802, a GPU pipeline stage 1804 with shading-rate images and fine control support, a shading reduction pipeline stage 1806, a shading rate image stage 1808, and An output step 1810 may be included.

The input stage 1802 provides input rendering calls or input image geometry (e.g., three-dimensional coordinates of the input image, color, texture, etc.) to the GPU pipeline stage 1804. I can. The GPU pipeline step 1804 may be similar to the graphics pipeline 200 of FIG. 2 in terms of functions. The GPU pipeline stage 1804 includes a fixed function geometry pipeline stage 1812, a geometry related shaders stage 1814, a fixed function rasterization or interpolation pipeline stage 1816, and pixel or sample shading. Shaders step 1818 may be included.

The input image geometry from the input step 1802 may be received in the fixed function geometry pipeline step 1812 of the GPU pipeline step 1804. The output from the fixed function geometry pipeline step 1812 is the number of shaders associated with the geometry (e.g., vertex, hull, domain, geometry, etc.) to serve as input to step 1814. have. Step 1814 (e.g., vertex shader (e.g., vertex shader 105 in Fig. 2)) can process vertices from the received input, transform, skinning, and morphing. Per-vertex operations such as, and per-vertex lighting may be performed. In addition, step 1814 may convert low-detail subdivision surfaces into high-detail primitives on the GPU, and perform tessellation (or break-up) tiles to convert higher-order surfaces into structures suitable for rendering, etc. I can do it. The output from step 1814 can be used as an input to a fixed function rasterization or interpolation pipeline step 1816.

The rasterization step 1816 converts vector information (e.g., consisting of shapes or primitives) received from the geometry-related shaders step 1814 for the purpose of displaying a real-time 3D graphic into a raster image (e.g., Consisting of pixels). The rasterization step 1816 may be the same as the rasterization step 110 of FIG. 2. The output from the rasterization step 1816 can be used as an input to the shaders step 1818 related to pixel or sample shading. Step 1818 (e.g., pixel-shader) may be the same as the fragment or pixel shader operation step 115 of FIG. 2, and shading techniques such as per-pixel lighting and post-processing (e.g., Rich shading techniques). In some embodiments, the pixel shader is a program that combines constant variables, texture data, interpolated per-vertex values, and other data to generate per-pixel outputs. The rasterizer step (e.g., 1816) calls the pixel shader once for each pixel covered by the primitive.

The output from step 1818 can be used as an input to an output step 1810. The output stage 1810 includes pipeline stages (e.g., 1812, 1814, 1816), pixel data generated by pixel shaders (e.g., 1818), the contents of the rendering targets, and a depth or stencil buffer. A combination of their contents can be used to create the final rendered pixel color. The output step 1810 may be the final step for determining the visible pixels (eg, with a depth-stencil test) and mixing the final pixel colors.

In some embodiments, the rasterization step 1816 is based on the vector information received from the geometric shader step 1814 as well as its output (e.g., a raster image) based on the shading rate image generated from the shading rate image step 1808. ). The shading rate image step 1808 may generate a shading rate image based on the output from the shading reduction pipeline step 1806. In some embodiments, shading reduction pipeline stage 1806 may be a set of software or hardware that automatically generates a shading rate image without application knowledge to feed extensions to any GPU that supports the API. .

Shading reduction pipeline stage 1806 is a sampling rate stage (or heuristic) 1820 (e.g., the required sampling rate as a function of a quality reduction filter (depth-of-field, motion blur, etc.)). , Reuse from previous frames vs. redraw analysis step (or heuristic) 1822, reuse from other viewports vs. readrow analysis step (or heuristic) 1824, and shading rate higher order function determination step ( 1826).

Sampling rate step 1820 is characterized by averaging multiple samples (or taps) from input textures rendered on the GPU, quality reduction filters (e.g., DoF, motion blur, smoothing). (smoothing) filters, or the like). If such a pattern is detected, the pattern can be used to determine the desired (or necessary) shading rate to achieve an acceptable (or sufficient) quality, which is discussed with respect to FIG. 20 in the latter part of this disclosure.

In some embodiments, the shading rate reduction may also include cross-frame changes of the underlying asset in the rendered image (e.g., the underlying asset (e.g., camera and viewport) between the input image frame and the output image frame. ) Of changes).

In one embodiment, the GPU can check for changes in underlying assets, e.g., cameras and viewports (e.g., anything that changes how the scene is viewed by the observer), and from previous images. Reprojection can be used to reuse data. If any other assets including geometry, shaders, states, or transformations change, it is impossible to reuse or reproject previous image data into the current image.

For example, in one embodiment, in step 1822, the underlying assets in the area of the rendered image do not change (or the change is less than a threshold), or the change is a linear transformation (e.g., affine ) Transform), the image data from the previous frame(s) can be reused instead of being rendered (or read-rowed) again. In these regions of the rendered image, the shading rate can be reduced to zero, and the pixel data can be copied from the previous frame(s) with small transformations (or transformations) if necessary (e.g. Reusing a fraction (some) of pixels from a frame is discussed with respect to FIGS. 2, 4, 7A, and 14). In some embodiments, reprojection may be used instead of reuse of pixel data.

In some embodiments, reprojection may be an example of reuse (eg, specialization or use case). In some embodiments, reuse may imply that data is copied from the previous image to the current image. Reprojection may include other example use cases, e.g., if the camera is moved, the pixels can be distorted, so to reuse pixel data from the previous image, the pixel data from the previous image is Can be reprojected into.

In some embodiments, reuse of pixel data from a previous frame may require that the graphics state does not change from the previous frame to the current frame, and the graphics state is the shaders used, constants provided to the shaders, and provided to the frames. Includes geometry.

In some embodiments, the shading rate reduction is the detection of the same image or multiple similar viewports within the same frame (e.g., the change between the first viewport of the current image frame and the second viewport of the current image frame is a threshold. Less than). For example, at step 1824, the GPU may detect, within the frame, that the image to be rendered uses one or more viewports that are substantially (or sufficiently) similar to one or more viewports of other images, or within the image, If more than one viewport can detect something similar to each other, or beyond a certain distance from the camera, it can be established that the rendered results will be sufficiently similar or identical to these similar viewports (e.g., similar embodiments are Discussed in connection with FIG. 16 of the disclosure). Some embodiments of the present disclosure may reuse (or re-project) all or part of pixel data from an arbitrary image in a current or previous frame based on determining the similarity of viewports.

For example, FIGS. 19A and 19B show an example of reusing (or reprojecting) data from a different viewport in stereo rendering cases, as well as an example of reusing (or reprojecting) pixel data across cameras. In the exemplary embodiment of FIGS. 19A-19B, the transformation matrix maps any pixel (x, y) in viewport 2 1902 to other pixel coordinates (x', y') in viewport 1 1904. Can be defined for Thus, if (x', y') is within the boundary and has the same occlusion characteristics, the data (eg, pixel data) can be copied from viewport 1 (1904) to viewport 2 (1902). . In some embodiments, occlusion characteristics may imply that data copied across viewports comes from the same object or the same class of objects, and the class may be defined by the depth range of the objects. In some embodiments, if (x', y') is out of range in viewport 1 1904, then pixel (x, y) will be rendered in viewport 2 1902 since there is no reference data to copy. In some embodiments, the reuse vs. readrow analysis step (or heuristic) 1824 from different viewports can be applied to the camera 1906 as lighting can vary significantly (or relatively large) across the viewports. Close objects can be identified and handled individually.

For objects or backgrounds far from the camera, differences in viewports may be considered relatively large or small enough to not cause sufficient differences in lighting, thus allowing reuse (or reprojection) of pixel data. In some embodiments of the present disclosure, the shading rate of areas with nearby objects (or foreground) may be the maximum required, while those with distant objects (or background) may be the minimum supported. And the remaining unrendered pixels/samples are filled by copying (or reusing or reprojecting) the data across the viewports.

For example, in FIG. 19B, in viewport 1 1904, both foreground 1908 and background 1910 have been fully rendered. However, in viewport 2 1902, only the foreground region 1912 and region 1916 were rendered. As the difference in the foregrounds 1098 and 1912 of the viewports 1904 and 1902 can be considered considerably (or sufficiently high), the foreground region 1912 in viewport 2 1902 was rendered. Area 1916 in viewport 2 1902 was rendered because that area was never sampled in viewport 1 1904. The rendered data from the background 1910 of viewport 1 1904 can be reused (reprojected in some cases) in the background 1914 of viewport 2 1902, because viewport 1 1904 and viewport 2 ( 1902)'s backgrounds 1910 and 1914 are far from the camera, so the difference in the backgrounds 1910 and 1914 of the viewports 1904 and 1902 does not cause a relatively large or significant difference in lighting. Can be considered small. In FIGS. 19A-19B, viewport 1 1904 and viewport 2 1902 may correspond to a left eye image and a right eye image (eg, discussed with respect to FIG. 16 ).

Returning to Figure 18, each of the heuristics 1820, 1822, and 1824 can determine a proposed shading rate based on their respective analysis. For example, based on their respective analysis, the heuristic 1820 may suggest or determine a first shading rate of the input image, and the heuristic 1822 may suggest or determine a second shading rate of the input image and And, the heuristic 1824 may suggest or determine a third shading rate of the input image.

In some embodiments, each of the heuristics (e.g., 1820, 1822, and 1824) can analyze a specific region of the rendered image and provide a shading rate accordingly, so that the GPU can, as needed, or As desired, many of these heuristics (eg, 1820, 1822, and 1824) can be applied in shading reduction pipeline stage 1806. For example, in an embodiment, the heuristic 1820 may analyze a first region of a rendered image or an output image and provide a first shading rate of an input image for the first region. Similarly, in some embodiments, the heuristic 1822 may analyze a second region of the rendered image or output image and provide a second shading rate of the input image for the second region, and the heuristic 1824 ) May analyze a third area of the rendered image or output image and provide a third shading rate of the input image for the third area.

In some embodiments, the first region of the rendered image or output image is a rendered subset (e.g., a first subset) of pixels in the rendered image or output image, and the second region of the rendered image or output image Is the rendered image or another rendered subset of pixels in the output image (e.g., a second subset), and a third region of the rendered image or output image is another rendered image or another of the pixels in the output image. This is the rendered subset (eg, the third subset).

If the shading rates are determined according to the heuristics (e.g., 1820, 1822, and 1824), at 1825, the GPU is to determine the shading rate image (e.g., at 1808), a resolution resolving filter ( For example, a maximum filter that can result in a maximum shading rate across all heuristics (eg, 1820, 1822, and 1824) can be used. For example, the GPU may select a maximum shading rate from among a first shading rate, a second shading rate, and a third shading rate for the input image using the maximum filter.

In some embodiments, based on the shading rates according to the heuristics (e.g., 1820, 1822, and 1824), the GPU determines a shading rate functional in step 1826 and shading in step 1808. Rate images can be computed or generated. In some embodiments, the maximum (or minimum) shading rate from the shading rates suggested by each of the heuristics (e.g., 1820, 1822, and 1824) is to generate a shading rate image in step 1808. May be selected, which may be input in step 1816 to produce an output rendered image in step 1810.

In some embodiments, two or more shading rates from the shading rates suggested by the respective heuristics (eg, 1820, 1822, and 1824) may be selected to render different portions of the image. However, the decision to choose a maximum or minimum shading rate can be detailed and context dependent. For example, if the quality degradation in the rendered image is due to DoF or motion blur, the minimum shading rate from the shading rates suggested by the respective heuristics (e.g., 1820, 1822, and 1824) is It may be selected to generate a shading rate image at 1808. The method of determining the shading rate higher order function is discussed in connection with FIG. 20.

FIG. 20 is an exemplary flow chart illustrating a shader analysis method for detecting quality reduction in a rendered image and calculating a shading rate image (eg, calculating a shading rate for an input image). The shader analysis method 2000 may be performed in the GPU of the shading reduction pipeline 1806 in step 1820.

As described above with respect to FIG. 18, the sampling rate step 1820 has the feature of averaging a number of samples (or taps) from input textures rendered on the GPU, for example, It relies on the detection of DoF, motion blur, smoothing filters, or similar quality reduction filters. If such a pattern is detected, as will be described later in connection with FIG. 20, the pattern is the desired (or required) shading rate (e.g., the required) to achieve an acceptable (or sufficient) quality (e.g., the quality of the output image). , May be used to determine the shading rate for the input image).

For example, FIG. 20 provides a detailed description of how the heuristic 1820 operates. For example, in the exemplary embodiment of FIG. 20, the processor or GPU may determine the rendering associated with averaging the input taps from the image or texture. Based on this determination, the processor or GPU has the higher the number of taps, the more pixels the processor or GPU needs to render in the input image (eg, “inputcolor” discussed in connection with 2006 in FIG. 20). You can decide what is going to be fewer.

For example, in 2002, a processor or GPU may analyze shading operations when rendering an output image (eg, outputcolor). In one embodiment, in 2002, pixel or fragment shaders may be analyzed to determine the presence of a quality reduction in the output image (eg, output color).

If, in 2002, it is determined that there is no reduction in the quality of the output image, then in 2004, the GPU does not modify the current shading rate image (eg, shading rate for the input image). However, if, in 2002, the quality reduction of the output image is determined, in 2006, the GPU responds to the weighted sum of pixel data from one or more input images in which the shading operation of the output image (e.g., output color) is determined. For example, it may be determined whether outputcolor (x,y) = sum_i (weight_i (x_i, y_i) * inputcolor_i (x_i, y_i))). For example, in one embodiment, in 2006, the GPU may determine whether the output color of the output image corresponds to a weighted sum of pixel data from one or more input images.

For example, in one embodiment, in 2006, the GPU may determine a higher order function for the final or output image (e.g., output color), and the final or output image (e.g., output color) or final or The formula expressing the weighted sum of pixel data to determine the pixel color value of the output image (for example, the output color) is as follows.

outputcolor (x,y) = sum_i (weight_i (x_i, y_i) * inputcolor_i (x_i, y_i))

At this time, i is [0,… (N-1)], N is the number of input values (eg, the number of input pixels), and (x, y) is the pixel coordinate in the final or output image. In some embodiments of the present disclosure, N may be 4. In one embodiment, from a higher order function of the final or output image, the “output color” at a particular pixel can be concluded as a weighted sum of the coordinates of the other pixels.

In example embodiments of the present disclosure, a higher-order function may be defined as a function that receives another function as an input. For example, in the case of a Depth-of-Field (DoF) filter, different applications can program or define their Depth-of-Field (DoF) filters in different ways. The higher-order function may receive an application defining a blur function as an input and then calculate a shading rate (eg, a shading rate for an input image) as an output. In particular, the higher-order function may be logic that determines a shading rate (eg, a shading rate for an input image) based on a specific filter (eg, Depth-of-Field (DoF), motion blur, etc.) However, higher-order functions are independent of how the filter is programmed or defined by the application.

In 2006, the processor or GPU corresponds to the shading operation of the output image (e.g., output color) to a weighted sum of data from one or more input images (e.g., outputcolor (x,y) = sum_i (weight_i ( If we determine that x_i, y_i) * inputcolor_i (x_i, y_i))), then in 2008, shading a specific subset of samples in the input image (e.g., input color) is the output image (e.g., output color). The processor or GPU reduces the shading (e.g., the shading rate, e.g. the shading rate for the input image) of the input image (e.g. Shading rate image can be generated).

In an embodiment, in 2008, the processor or GPU may generate a shading rate image (eg, a shading rate for the input image) for the input image or images (eg, input color). For example, in 2008, the GPU generates a shading rate image for the input image or images (eg, input color), and the number of shaded samples is inversely proportional to the number “N” of input values in the higher order function. do. However, the proportional relationship (or exact proportion) between the number of shaded samples and the number “N” of input values in the higher-order function can be adjusted based on the workload of the GPU and the quality reduction determined in 2002. For example, in some embodiments, the weighted sum comprises a weighted sum of the pixel data of the input images over the N input image frames when the input image (e.g., input color) is a specific example. can do.

In 2010, the GPU uses steps 1826 or 1808 to present any other (e.g., 1822, 1824) and shading rate image (e.g., input image from 2008). And shading rate).

In one embodiment, the output from the method of FIG. 20 may be used as the input of the max() function at 1825 in 1806.

In one embodiment, the reduction in shading (eg, a shading rate for an input image) may not be for the same image. In some embodiments, using information from one image (eg, output image), the shading rate in the previous image (eg, input image) may be reduced (in this case “previous image”). Is an image not in terms of time, but in terms of dependency with the current image).

FIG. 21 shows an exemplary method of performing adaptive shading of image frames by a GPU, as discussed in connection with FIG. 18.

At 2102, the GPU determines whether a quality reduction filter has been used by applying one or more heuristics to the pixel shaders in the GPU (eg, as discussed with respect to FIG. 18).

At 2104, the GPU determines a higher-order function of the function in the pixel shader based on determining that the quality reduction filter is used in the GPU (discussed with respect to FIG. 20).

At 2106, the GPU determines a first shading rate for the previous image frame (e.g., the input image frame) (or at least a portion of the previous image frame) based on the higher order function (as discussed in connection with FIG. 20). . In some embodiments, a portion of the previous image frame (eg, input image frame) is a rendered subset (eg, first subset) of pixels in the previous image frame.

At 2108, the GPU renders the previous image frame (eg, part of the previous image frame) based on the first shading rate (discussed with respect to FIGS. 18 and 20).

In some embodiments, at 2110, the GPU checks for changes in underlying assets, e.g., camera and viewport (e.g., anything that changes the scene observed by the observer), and uses reprojection. Reuse data from the previous image or previous image frame. If any other assets including geometry, shaders, states, or transformations change, it may be impossible to reuse or reproject the previous image onto the current image. In one embodiment, at 2110, the GPU changes in underlying assets between the current image frame and the previous image frame (as discussed with respect to 1822 in Figure 18, e.g., between the input image frame and the output image frame. Determine the underlying assets (eg, changes in camera and viewport).

In some embodiments, at 2112, the GPU is based on a determination that the change in underlying assets between the current image frame and the previous image frame is less than the first threshold, based on the second shading rate or image data from the previous image frame. Render the current image frame (or part of the current image frame) by reusing or reprojecting (discussed with respect to FIGS. 18, 19, and 20). In some embodiments, a portion of the current image frame is a rendered subset of pixels in the current image frame.

An exemplary embodiment of quality reduction based on depth-of-field (DoF)

In some exemplary embodiments, the depth-of-field (DoF) experiment may be an example of the embodiment of FIG. 20 in which a specific function and a higher order function can be used, and the use of shading rate reduction results in a significant decrease in quality. It may be determined that it may not (eg, as measured by the industry standard metric peak signal to noise ratio (PSNR)).

In one embodiment, the shader uses per-pixel depth and the per-pixel depth and focal length (e.g., the distance the camera is focused, e.g., the human visual system is Is approximate with a 50-55 mm lens). The difference between the per-pixel depth and focal length can determine the degree of blurry of the result that can be copied from the input image using the level of detail (LOD). If the per-pixel depth is close to the focal length, the selected LOD may be close to zero (eg, most detailed). However, if the per-pixel depth is far from the focal length, high LOD levels may be selected.

The high LOD levels may be the average of the low LOD levels (eg, level 1 is the M x N averaged version of 2M x 2N level 0). Therefore, the processor or GPU, the outputcolor (x, y) is a number of input color level 0 to a specific distance from (x, y) whose weight is radially decreased based on the distance from (x, y). We can create a function, which is the weighted sum of the pixels of.

Therefore, in one embodiment, if the per-pixel depth of (x, y) is far from the focal length, the processor or GPU can calculate the shading rate of the input color image level 0 since the error in shading can be averaged over several pixels. Can be reduced. Since input color level 0 may be the most expensive image to render in that particular workload, saving may be relatively high or significant compared to methods in the related art. Therefore, while keeping the quality (measured by PSNR) within acceptable levels (e.g., less than visible levels), shading is reduced (and the amount of work required for shading) is reduced using this logic. Can be) can be.

In example embodiments of the present disclosure, PSNR may be the log of a mean-square error (i.e., the per-pixel difference in color values is (1) full-resolution rendering and (2)). ) Compute from rendering using the exemplary approach described above, then sum the squares of these per-pixel differences and calculate log(sqrt(sum)) to determine PSNR). It may be desirable to make the PSNR as high as possible.

In one embodiment, if the PSNR is greater than or equal to 40, this may be visually indistinguishable from full resolution rendering for most people, and if the PSNR is greater than or equal to 50 or 60, it may not be visually distinguishable to all people. have. If the PSNR is greater than or equal to 40, savings of about 25% of shading operations in the input image (eg, input color level 0 image) can be achieved.

As discussed in connection with FIG. 2, in the adaptive desampling (AD), information on the spatial profile of the final image detail is back-propagate to the initial stages of the graphics pipeline and thus is small. Rendering quads and interpolating between them may be a lossy quad culling technique, identifying areas that may cause minimal visual artifacts. In some cases, by utilizing the parameter settings of the level of the level of detail (LOD) features (e.g. in texture samplers), areas that require greater rendering detail than others can be identified and reduce rendering work. Can be used by AD for harm.

In some cases, for example in areas of the image where rendering may not be necessary, AD can be used to save rendering work. For example, areas of a rendering target (eg, an image) that do not require fine-scale details can often be rendered at low resolution without experiencing noticeable visual artifacts. In some cases, the AD can sparsely render a full resolution image and reconstruct the pixels in between. To achieve these results, AD uses a rendering scheme so that the quads (e.g., a block of 2 x 2 pixels, which is the basic unit of work in rendering) are 4 x 4 blocks (e.g., MultiQuad). To a non-contiguous set of 4 pixels within a 3 x 3 area (eg, a 3 x 3 quad) to be re-mapping.

In some cases, as shown in FIG. 22, arranging the pixels in the manner described above may cause any 3 x 3 quad that has not been rendered to be reconstructed from the rendered neighboring pixels. In some cases, as long as at least one of the quads is rendered, any unrendered pixels will have at least one rendered neighbor and are reconstructed from this neighbor. In some cases, the AD also identifies the order of which 3 x 3 quads are progressively rendered as the rendering density increases. In some embodiments, this order may define three available rendering densities for a multiquad, such as the sets of rendered quads below, the sets of rendered quads as follows. {0}, {0, 3}, and {0, 1, 2, 3}. In some cases, each of these densities can be represented as a one-hot encoded 4-bit value, with each bit indicating whether the corresponding quad is rendered. In some cases, this code is called an adaptive desampling sample pattern (ADSP) code. Fig. 23 shows the correspondence between these codes (eg ADSP codes) and rendered pixels within a multiquad. In some embodiments, other reconstruction schemes may be used depending on the rendering density.

In some embodiments, the AD can be used to determine which rendering density to use while minimizing information loss and where to maximize rendering operation savings. To determine a desirable point in the trade-off between maximizing rendering operation savings while minimizing information loss, in some embodiments, the AD may render infrequently in areas of the image with less information loss (e.g. , Small details are present in the final image), densely rendering can be achieved in areas where fine details may be desirable. In some embodiments, information loss may be estimated using a PSNR metric (metric) between an AD rendered image and a full rendered version of the image. In some embodiments of the present disclosure, the AD may identify which areas of the image require more or less detail when the final image itself is unknown, which may involve two passes of the AD. For example, in the first pass, depth edges can be identified using edge detection operations in the depth buffer, and the rendering density can be set to a relatively high value when edges occur. Then, the image can be rendered at the corresponding density. The second pass can identify the color edges (eg, using the available quads rendered in the first pass) and increase the density if necessary. In some embodiments, the AD can infer areas that require high or low density by examining user specific parameters that specify how the target image is sampled downstream.

As an alternative to the two pass approach, in another embodiment of AD, higher-order function analysis can be used (discussed with respect to FIG. 20) to infer the sampling function of the image in the shader program as a function of external dependencies. For example, in some embodiments, the texture method in glsl (eg, OpenGL Shading Language) may accept an LOD parameter that specifies how the target texture is sampled. For higher values, the texture can be sampled with a lower LOD, and a blurred version of the texture can be returned at the requested coordinates, and vice versa.

Although terms such as “first”, “second”, “third”, etc. are used herein to describe various elements, components, regions, layers, and/or sections, these elements It will be understood that, components, regions, layers, and/or sections are not limited by these terms. These terms are used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Accordingly, a first element, component, region, layer, or section described herein may be referred to as a second element, component, region, layer, or section without departing from the scope of the technical idea of the present invention.

In this specification, spatially relative terms such as “lower”, “lower”, “lower”, “above”, “upper”, and similar terms are used for other component(s) shown in the drawings. Or, in relation to the feature(s), it may be used to easily describe one component or feature. Spatial and relative terms are intended to include other directions of the device in use or operation in addition to the directions shown in the figures. For example, if the device is turned over in the drawings, components depicted as “below” or “bottom” of other components or features will face “above” the other components or features. That is, the exemplary term “below” may include both an upward and downward direction. The device may be oriented differently (rotated 90 degrees or oriented differently) and the spatially relative descriptions used herein should be interpreted accordingly. It will also be understood that when a layer is referred to as being “between” two layers, it may be the only layer between the two layers, or there may be more than one intervening layer.

The terms used herein are only used for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used herein, the terms “usually”, “about”, and similar terms are used as approximate terms, not used as terms of degree, and are not used by those skilled in the art to which the present invention belongs. It is intended to take into account the inherent variation of the measured or calculated values that are identified.

Singular expressions used in this specification are intended to include plural expressions unless the context clearly dictates otherwise. As used herein, the terms “comprise” and/or “comprising” are understood to specify the presence of the recited features, integers, steps, actions, elements, and/or components, It is understood that it does not exclude the presence or addition of one or more other features, integers, steps, elements, components, and/or combinations thereof. As used herein, the term “and/or” includes any and all combinations of one or more related listed items. Expressions such as “at least one”, when used before an item of elements, modify the entire item of elements and do not modify individual elements of the item. In addition, the use of “can” refers to “one or more embodiments of the present disclosure” when describing embodiments of the present disclosure. Also, the term “exemplary” is intended to refer to an example or illustration. As used herein, the terms “use”, “use”, and “used” may be considered synonymous with “use”, “use”, and “used”, respectively.

When an element or layer is referred to as “on it”, “connected”, “joined”, or “adjacent” in relation to another element or layer, it is a connected element or layer located directly on it in relation to another element or layer. , Combined, or contiguous, or the presence of one or more intervening elements or layers. In contrast, when an element or layer is referred to as “directly located”, “directly connected”, “directly joined”, or “directly adjacent” in relation to another element or layer, an intervening element or layer is present. It should be understood as not.

Any numerical range recited herein is intended to include the estimation of all sub-ranges of the same numerical precision within the recited range. For example, a range of “1.0 to 10.0” is intended to include all subranges between the recited minimum value of 1.0 and the recited maximum value of 10.0, ie, from 2.4 to 7.6. As such, it is intended to include a minimum value greater than or equal to 1.0 and a maximum value less than or equal to 10.0. Any maximum numerical limit recited herein is intended to include all lower numerical limits recited therein, and any minimum numerical limit recited herein is intended to include all high numerical limits recited therein.

In some embodiments, one or more outputs of other embodiments of the methods and systems of the present disclosure may include a display device for displaying one or more outputs or information regarding one or more outputs of other embodiments of the methods and systems of the present disclosure. It may be transmitted to a combined or electronic device including the same.

Electronic or electrical devices and/or any other related devices or elements according to embodiments of the present disclosure described herein may be any suitable hardware, firmware (e.g., an application-specific integrated circuit), It can be implemented using software, or a combination of software, firmware, and hardware. For example, the various elements of these devices may be formed on one integrated circuit (IC) chip or separate IC chips. In addition, various elements of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a single substrate. In addition, various elements of these devices may be executed on one or more processors, within one or more computing devices, that interact with other system elements and execute computer program instructions to perform the various functions described herein. Or it may be a thread. Computer program instructions are stored in memory, which can be implemented in a computing device using standard memory devices, such as, for example, random access memory (RAM). Further, the computer program instructions may be stored in, for example, a CD-ROM, a flash drive, or similar non-transitory computer readable media. In addition, those of ordinary skill in the art to which the present invention pertains are not departing from the technical spirit of exemplary embodiments of the present disclosure, functions of various computing devices may be combined with a single computing device or integrated into a single computing device, Alternatively, it should be appreciated that the functionality of a particular computing device may be distributed across one or more other computing devices.

Claims (10)

In a method of performing adaptive shading of image frames by a Graphics Processing Unit (GPU):
Determining, by the GPU, a first shading rate based on determining that a change in a plurality of underlying assets between a first image frame and a second image frame exceeds a first threshold value;
Determining, by the GPU, a second shading rate based on determining that one or more viewports in the second image frame are similar to one or more viewports in the first image frame;
Determining, by the GPU, a third shading rate based on determining that a quality reduction filter is used; And
And selecting, by the GPU, a shading rate from among the first shading rate, the second shading rate, and the third shading rate for the first image frame. The method of claim 1,
Rendering, by the GPU, the second image frame based on determining that the change in the plurality of underlying assets between the first image frame and the second image frame is less than the first threshold. And reusing the pixel data from the first image frame for processing. The method of claim 2,
Re-using, by the GPU, the pixel data from the first image frame to render the second image frame by reprojecting the pixel data from the first image frame to the second image frame How to further include. The method of claim 1,
The method wherein the change in the plurality of underlying assets between the first image frame and the second image frame is associated with a camera and a viewport. The method of claim 1,
Rendering the second image frame based on determining, by the GPU, that a change between the one or more viewports in the second image frame and the one or more viewports in the first image frame is less than a second threshold Reusing the pixel data from the first image frame to do so; And
Re-projecting, by the GPU, the pixel data from the first image frame to the second image frame, thereby reusing the pixel data from the first image frame to render the second image frame How to. The method of claim 1,
Determining that the quality reduction filter is to be used is:
Determining, by the GPU, a decrease in quality of the second image frame;
Determining, by the GPU, whether the output color of the second image frame corresponds to a weighted sum of pixel data from the first image frame; And
The third shading rate for the first image frame is determined by the GPU based on determining that the output color of the second image frame corresponds to the weighted sum of the pixel data from the first image frame. A method that involves determining. The method of claim 6,
Determining that the output color of the second image frame corresponds to the weighted sum of the pixel data from the first image frame:
And determining, by the GPU, a higher order function of the second image frame. The method of claim 7,
The proportional relationship between a plurality of shaded samples in the first image frame and a plurality of input values for the higher-order function is based on a workload of the GPU. The method of claim 7,
The higher order function is:
Receiving the quality reduction filter as input, and
A method configured to determine the third shading rate for the first image frame. The method of claim 6,
The weighted sum comprises the weighted sum of the pixel data of the first image frame over a plurality of input image frames exemplifying the first image frame.

Images