EP3405907B1

EP3405907B1 - Performing anti-aliasing operations in a computing system

Info

Publication number: EP3405907B1
Application number: EP17741813.4A
Authority: EP
Inventors: Evgene Fainstain
Original assignee: Advanced Micro Devices Inc
Current assignee: Advanced Micro Devices Inc
Priority date: 2016-01-18
Filing date: 2017-01-17
Publication date: 2022-05-04
Anticipated expiration: 2037-01-17
Also published as: US10354365B2; US10152772B2; KR102635452B1; JP6992005B2; US20170206626A1; EP3405907A1; JP7361089B2; CN108701235B; KR20180102617A; WO2017127363A1; US20170206638A1; EP3405907A4; JP2019505939A; CN108701235A; EP4040395A1; JP2022031880A

Description

BACKGROUND

Priority Information

This application claims benefit of priority of U.S. Provisional Application Serial Number 62/279,889 titled "Advanced Multisampling Techniques For Realtime 3D Graphics" filed January 18, 2016 , whose inventor was Evgene Fainstain.

Description of the Related Art

Three-dimensional (3D) rendering is a computationally-intensive process, and to produce realistic (i.e., cinematic) high quality real-time rendering, a significant amount of computing power is required. The 3D hardware manufacturers continually create more powerful devices (e.g., graphical processing units (GPUs)) to perform rendering tasks. A GPU is a complex integrated circuit that is configured to perform graphics-processing tasks. For example, a GPU can execute graphics-processing tasks required by an end-user application, such as a video-game application. The GPU can be a discrete device or can be included in the same device as another processor, such as a central processing unit (CPU).
Unfortunately, modern real-time 3D rendering hardware is not powerful enough to properly simulate the physical world in real-time. For example, ray tracing-like algorithms are too computationally expensive to implement in real-time. Accordingly, various efficiency improvement techniques are used to produce an image that seems real enough although the image is not entirely physically accurate. A common problem of 3D generated images is the inherent "zipper" effect (i.e., "jaggies" or "aliasing"). Up until now, the techniques which have been developed to minimize these aliasing effects on rendered images have been ineffective and/or too computationally expensive
Maule M. et al "Transparency and Anti-Aliasing Techniques for Real-Time Rendering", XXV SIBGRAPI Conference on Graphics, Patterns and Images Tutorials, 22. August 2012, Ouro Preto, Brazil, pages 50-59, discloses different anti-aliasing methods.

SUMMARY OF EMBODIMENTS

Various embodiments of a system and method for performing anti-aliasing related operations are contemplated. In one embodiment, a system is contemplated that comprises at least a memory and a processor. The processor is configured to detect a request to perform an anti-aliasing resolve operation on a first image that is stored in the memory. The first image includes multiple sub-pixel samples per pixel and has a first set of dimensions. In response to detecting the request, the processor is configured to expand dimensions of the first image to create a second image that includes a single sample per pixel and has a second set of dimensions greater than the first set of dimensions, perform post-processing anti-aliasing on the second image to create a third image with the second set of dimensions, and reduce dimensions of the third image to create a fourth image, where the fourth image is a result of the anti-aliasing resolve operation. In various embodiments, performing post-processing anti-aliasing is performed prior to the averaging and comprises filtering pixels based on values of surrounding pixels. In various embodiments, expanding dimensions of the first image comprises converting sub-pixels of the first image into regular pixels of the second image. Further, in some embodiments the second image is created by expanding and rotating the first image. In some embodiments, the processor is configured to rotate the first image to align the sub-pixels into a vertical and horizontal grid pattern within the second image. Further embodiments include the processor rearranging a plurality of triangular portions from the first image into new locations within the second image, and rearranging any remaining portions of the first image into new locations within the second image in between the plurality of triangular portions which have been rearranged. Methods are also contemplated for performing the above described functions.
Also contemplated is a processor comprising a memory and a plurality of execution units. The the processor is configured to generate, based on a rotated grid, a plurality of sub-pixel sampling coordinates for each pixel of an image being rendered, wherein an amount of rotation is specified for the rotated grid; sample sub-pixel locations indicated by the plurality of sub-pixel sampling coordinates within each pixel of the image being rendered by the plurality of execution units; and store values of the sub-pixel locations in the memory. When generating the plurality of sub-pixel sampling coordinates, the processor is configured to calculate locations that correspond to a first set of parallel lines which pass diagonally through given locations of the image, and calculate locations that correspond to a second set of parallel lines perpendicular to the first set of parallel lines, wherein the first and second sets of parallel lines form the rotated grid. Additionally, the processor is configured shift the rotated grid to cause vertices of the first and second sets of parallel lines to have a center of gravity which is in a center of each corresponding pixel, and specify the plurality of sub-pixel sampling coordinates to coincide with locations of the vertices of the first and second sets of parallel lines within pixels of the image. The given locations are either corners of the pixels or centers of the pixels. Further, performing one or more of the above ensures that one or more of the following conditions are met: the sub-pixel sampling coordinates are regularly spaced both horizontally and vertically, there is only one sub-pixel sampling location per pixel on a given horizontal or vertical line which traverses the image, and a slope of each line of the first set of parallel lines is equal to a ratio of two mutually prime numbers.

BRIEF DESCRIPTION OF THE DRAWINGS

The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram of one embodiment of a computing system.
FIG. 2 is a block diagram of one embodiment of a graphics processing pipeline.
FIG. 3 illustrates diagrams of a sub-pixel sampling pattern.
FIG. 4 illustrates a diagram of the 5xMSAA sampling pattern.
FIG. 5 illustrates diagrams of a 10xMSAA sub-pixel sampling pattern.
FIG. 6 illustrates diagrams of a 13xMSAA sub-pixel sampling pattern.
FIG. 7 illustrates diagrams of a 17xMSAA sub-pixel sampling pattern.
FIG. 8 illustrates diagrams of a sampling pattern with 8 sub-pixel sampling points per original pixel.
FIG. 9 illustrates diagrams of pixel-sharing patterns.
FIG. 10 illustrates a diagram of one embodiment of a hybrid anti-aliasing scheme.
FIG. 11 illustrates a diagram of another embodiment of a hybrid anti-aliasing scheme.
FIG. 12 illustrates a diagram of a technique for reducing the size of an expanded image.
FIG. 13 is a generalized flow diagram illustrating one embodiment of a method for performing a hybrid anti-aliasing resolve operation.
FIG. 14 is a generalized flow diagram illustrating one embodiment of a method for rearranging image portions when expanding and rotating an image.
FIG. 15 is a generalized flow diagram illustrating another embodiment of a method for rearranging image portions when expanding and rotating an image.
FIG. 16 is a generalized flow diagram illustrating sampling sub-pixel locations of an image.
FIG. 17 is a generalized flow diagram illustrating generating a plurality of sub-pixel sampling coordinates for each pixel of an image being rendered.
FIG. 18 is a generalized flow diagram illustrating generating sub-pixel sampling coordinates
FIG. 19 is a generalized flow diagram illustrating one embodiment of a method for performing an anti-aliasing operation.
FIG. 20 is a generalized flow diagram illustrating generating a sampling grid.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.
Various systems, apparatuses, methods, and computer-readable mediums for performing hybrid anti-aliasing operations on a processor are disclosed. In one embodiment, a processor includes at least a memory and multiple execution units. In one embodiment, the processor is a GPU. In other embodiments, the processor is any of various other types of processors (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), multi-core processor). The processor detects a request to perform an anti-aliasing resolve operation on a first image stored in the memory. Responsive to detecting the request, the processor expands dimensions of the first image to create a second image, filters the second image with a post-processing anti-aliasing filter to create a third image, and then performs averaging of the third image to create a fourth image, wherein the fourth image is a result of the anti-aliasing operation. Expanding dimensions of the first image involves converting sub-pixels of the first image into regular pixels of the second image. The processor can also rotate the first image to align the sub-pixels into a vertical and horizontal grid pattern within the second image.
In one embodiment, the processor rearranges a plurality of triangular portions from the first image into new locations within the second image so as to reduce the memory overhead of storing and/or processing unused areas of the second image, wherein the unused areas are created as a result of rotating the first image. The processor also rearranges any remaining portions of the first image into new locations within the second image in between the plurality of triangular portions which have been rearranged. In one embodiment, the processor relocates a first triangular portion of the first image to the right-side of the second image. The processor also relocates a second triangular portion of the first image to the left-side of the second image. Still further, the processor relocates a third portion of the first image between the first triangular portion and the second triangular portion within the second image.
Referring now to FIG. 1, a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes system on chip (SoC) 105 coupled to memory 150. SoC 105 can also be referred to as an integrated circuit (IC). In one embodiment, SoC 105 includes processing units 115A-N, input/output (I/O) interfaces 110, shared caches 120A-B, fabric 125, graphics processing unit (GPU) 130, and memory controller(s) 140. SoC 105 can also include other components not shown in FIG. 1 to avoid obscuring the figure. Processing units 115A-N are representative of any number and type of processing units. In one embodiment, processing units 115A-N are central processing unit (CPU) cores. In another embodiment, one or more of processing units 115A-N are other types of processing units (e.g., specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP)). Processing units 115A-N are coupled to shared caches 120A-B and fabric 125.
In one embodiment, processing units 115A-N are configured to execute instructions of a particular instruction set architecture (ISA). Each processing unit 115AN includes one or more execution units, cache memories, schedulers, branch prediction circuits, and so forth. In one embodiment, the processing units 115A-N are configured to execute the main control software of system 100, such as an operating system. Generally, software executed by processing units 115A-N during use can control the other components of system 100 to realize the desired functionality of system 100. Processing units 115A-N can also execute other software, such as application programs.
GPU 130 includes at least sub-pixel coordinate table 135 and compute units 145A-N which are representative of any number and type of compute units that are used for graphics or general-purpose processing. Sub-pixel coordinate table 135 is a programmable table which stores the coordinates of sub-pixel sampling locations within the pixels of an image being rendered by GPU 130. It is noted that the term "sub-pixel sampling locations" is defined as the multiple locations for sampling the value of a parameter (e.g., color, depth, stencil, transparency) within a given pixel of an image being rendered. The term "sub-pixel sampling locations" indicates that there will be multiple sampling locations within each pixel of the image being rendered. Any of various different types of sub-pixel sampling patterns can be utilized in different embodiments. The sub-pixel sampling pattern is generated and programmed into sub-pixel coordinate table 135 to specify which locations within the pixels should be sampled when processing a given image.
Compute units 145A-N can also be referred to as "shader arrays", "shader engines", "shader units", "single instruction multiple data (SIMD) units", or "SIMD cores". Each compute unit 145A-N includes a plurality of execution units. GPU 130 is coupled to shared caches 120A-B and fabric 125. In one embodiment, GPU 130 is configured to execute graphics pipeline operations such as draw commands, pixel operations, geometric computations, and other operations for rendering an image to a display. In another embodiment, GPU 130 is configured to execute operations unrelated to graphics. In a further embodiment, GPU 130 is configured to execute both graphics operations and non-graphics related operations.
In one embodiment, GPU 130 is configured to detect a request to perform an anti-aliasing resolve operation on a first image. In one embodiment, the term "resolve operation" is defined as converting a sub-pixel sampled surface into a surface with one sample per pixel. Responsive to detecting the request, GPU 130 expands dimensions of the first image to create a second image, filters the second image with a post-processing anti-aliasing filter to create a third image, and then performs averaging of the third image to create a fourth image, wherein the fourth image is a result of the anti-aliasing operation. Expanding dimensions of the first image involves converting sub-pixels of the first image into regular pixels of the second image. GPU 130 can also rotate the first image to align the sub-pixels into a vertical and horizontal grid pattern within the second image.
I/O interfaces 110 are coupled to fabric 125, and I/O interfaces 110 are representative of any number and type of interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices can be coupled to I/O interfaces 110. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth.
SoC 105 is coupled to memory 150, which includes one or more memory modules. Each of the memory modules includes one or more memory devices mounted thereon. In some embodiments, memory 150 includes one or more memory devices mounted on a motherboard or other carrier upon which SoC 105 is also mounted. In one embodiment, memory 150 is used to implement a random access memory (RAM) for use with SoC 105 during operation. The RAM implemented can be static RAM (SRAM), dynamic RAM (DRAM), Resistive RAM (ReRAM), Phase Change RAM (PCRAM), or any other volatile or non-volatile RAM The type of DRAM that is used to implement memory 150 includes (but is not limited to) double data rate (DDR) DRAM, DDR2 DRAM, DDR3 DRAM, and so forth. Although not explicitly shown in FIG. 1, SoC 105 can also include one or more cache memories that are internal to the processing units 115A-N and/or compute units 145A-N. In some embodiments, SoC 105 includes shared caches 120A-B that are utilized by processing units 115A-N and compute units 145A-N. In one embodiment, caches 120A-B are part of a cache subsystem including a cache controller.
It is noted that the letter "N" when displayed herein next to various structures is meant to generically indicate any number of elements for that structure (e.g., any number of processing units 115A-N) including one. Additionally, different references within FIG. 1 that use the letter "N" (e.g., processing units 115A-N and compute units 145A-N) are not intended to indicate that equal numbers of the different elements are provided (e.g., the number of processing units 115A-N can differ from the number of compute units 145A-N).
In various embodiments, computing system 100 can be a computer, laptop, mobile device, server or any of various other types of computing systems or devices. It is noted that the number of components of computing system 100 and/or SoC 105 can vary from embodiment to embodiment. There can be more or fewer of each component/subcomponent than the number shown in FIG. 1. For example, in another embodiment, SoC 105 can include multiple memory controllers coupled to multiple memories. It is also noted that computing system 100 and/or SoC 105 can include other components not shown in FIG. 1. Additionally, in other embodiments, computing system 100 and SoC 105 can be structured in other ways than shown in FIG. 1.
Turning now to FIG. 2, a block diagram of one embodiment of a graphics processing pipeline 200 is shown. In one embodiment, graphics processing pipeline 200 is implemented by GPU 130 (of FIG. 1). In other embodiments, graphics processing pipeline 200 can be implemented using other types of processing hardware (e.g., FPGA, ASIC, DSP, multi-core processor). Generally speaking, graphics processing pipeline 200 can be implemented using any suitable combination of software and/or hardware. Pipeline architectures can perform long latency operations more efficiently by splitting up an operation into multiple stages, with the output of each stage feeding the input of the subsequent pipeline stage. The shader units 205A-N in pipeline 200 can include a vertex shader, geometry shader, fragment shader, pixel shader, and/or one or more other shaders coupled to cache/memory 210. Cache/memory 210 is representative of any type and number of cache or memory devices. One or more of the shader units 205A-N shown in pipeline 200 is configured to perform a hybrid anti-aliasing resolve operation as described in further detail below. This shader unit receives an image, texture, or other input pixel data, and the shader unit performs a hybrid anti-aliasing resolve operation on the image to produce an anti-aliased image as output. Depending on the embodiment, the anti-aliased image can be driven to the display, written back to memory, or processed by one or more additional shader units.
Referring now to FIG. 3, diagrams of a sub-pixel sampling pattern is shown. Diagram 305 is shown at the top of FIG. 3, with diagram 305 including a 4x4 pixel square grid with horizontal and vertical lines indicating the borders of the pixels. Each square in the grid represents a pixel of a source image. A sub-pixel sampling pattern is generated by drawing a first set of lines with a slope of 2x1 through the corners of the pixels. It is noted that first set of lines with a slope of 2x1 are parallel to the diagonal of a rectangle which is two pixels wide and one pixel high. As used herein, the term "slope of 2x1" corresponds to a slope of -1/2 according to the traditional mathematical definition of slope. Generally speaking, when a set of lines are described as having a slope of NxM, the traditional definition of slope for these lines will be -M/N. Then, a second set of lines are drawn perpendicular to the first set of lines, with the first and second sets of lines creating a rotated grid on top of the 4x4 pixel square grid. The sub-pixel sampling locations are selected to coincide with the vertices (i.e., intersections) of the first and second sets of lines of the rotated grid. It is noted that while the discussion herein describes "drawing" lines to create a sampling pattern, the first and second sets of parallel lines of the rotated grid do not actually have to be drawn to generate the sub-pixel sampling pattern. Rather, the lines can be mathematically determined (e.g., calculating equations for the lines) so as to generate the sub-pixel sampling pattern.
Then, as shown in diagram 310, the rotated grid is shifted to cause the center of gravity of the sub-pixel sampling locations for each original pixel to be in the center of the original pixel. For example, the grid is shifted such that the value of (X₁ + X₂ + ... X_N) / N is equal to the center of the original pixel for the x coordinate and using a similar equation for the Y coordinates of the sub-pixels, with N equal to the number of sub-pixel samples. It is noted that this step of shifting the rotated grid to cause the center of gravity of the sub-pixel sampling locations for each original pixel to be in the center of the original pixel can be omitted. The sub-pixel sampling locations are indicated in diagram 310 by the circles at the intersections of the first and second sets of lines. The sub-pixel sampling pattern is generated by drawing a rotated regular sampling grid through the centers of the original pixels and then optionally shifted. When the grid is drawn through the centers of the pixels, then one of the sub-pixel samples will be in the center of the pixel, the center of gravity of sub-pixel samples will be in the center of the pixel, and the sub-pixel sampling pattern will be symmetrical around the center of the pixel. For example, if the center of the pixel has coordinates (0,0), then for a sub-pixel sample at coordinates (a,b), there will be another sub-pixel sample at coordinates (-a,-b). The sub-pixel sampling locations for a single pixel are shown in diagram 315.
The sub-pixel sampling pattern shown in diagram 310 is used to generate sub-pixel sampling coordinates for the underlying image. These coordinates are then used to determine which sub-pixel locations to sample within the image being rendered by a processor (e.g., GPU). The coordinates are generated in accordance with a Cartesian coordinate system with each sampling point specified by a pair of numerical coordinates. The numerical coordinates are stored as floating point (e.g., single precision floating point, double precision floating point) values. The numerical coordinates are stored using other types of representations (e.g., fixed point, integer).
The technique of sampling multiple separate locations within a single pixel can be referred to as "sub-pixel sampling" or as "multisampling anti-aliasing" (MSAA). In some cases, sampling multiple separate locations within a single pixel can be referred to as "subsampling". It should be understood that "subsampling" in this context has a different meaning than when the term "subsampling" is used to describe signal processing techniques. Within the field of signal processing, the term "subsampling" is defined as reducing the sampling rate of a signal. Within this disclosure, the term "sub-pixel sampling" will be used to describe sampling multiple separate locations within a single pixel to avoid any confusion.
For example, the pattern shown in diagram 315 can be referred to as a 5xMSAA sampling pattern. This pattern is constructed by drawing a first set of lines with a 2x1 slope, with each line of the first set of lines drawn through the original pixel square grid traversing two horizontal pixels for each vertical pixel. Then, a second set of lines are drawn perpendicular to the first set of lines, with the intersections of the lines determining sub-pixel sampling locations, resulting in 2^2 + 1^2 = 5 sub-pixel samples per pixel. This sampling pattern can be shifted, rotated by 90 degrees, and mirrored. This sampling pattern, and any adjustments, can be utilized for various applications in temporal anti-aliasing.
Turning now to FIG. 4, a diagram 400 of the 5xMSAA sampling pattern is shown. The discussion of FIG. 4 is a continuation of the discussion regarding FIG. 3. The dots shown in diagram 400 of FIG. 4 represent the sub-pixel sampling locations for determining the locations at which to calculate the value of any of various parameters (e.g., color, depth, stencil, transparency) of an image. As shown in diagram 400, the vertical and horizontal lines which are drawn to intersect with these sub-pixel sampling locations illustrate that the 5xMSAA sampling pattern satisfies the regular rectangular grid condition since samples are regularly spaced both horizontally and vertically. The 5xMSAA sampling pattern also satisfies the one-sample-per-horizontal/vertical line condition, since there is only one sub-pixel location sampled per pixel on a given horizontal or vertical line which traverses the image. Satisfying the one-sample-per-horizontal/vertical line condition can enhance the anti-aliasing properties of the pattern by allowing improved representation of vertical and horizontal lines at the sub-pixel level within the image.
The 5xMSAA sampling pattern is superior to a standard 2x2 rotated grid super sampling (RGSS) pattern in multiple ways. For example, the 5xMSAA pattern produces four intermediate colors for horizontal and vertical lines instead of three. The 5xMSAA sampling pattern also ensures uniform sub-pixel sampling throughout the image. The 5xMSAA sampling pattern can be shifted, including shifted across pixel boundaries, and mirrored while still retaining both regular grid quality and the one-sample-per-horizontal/vertical-line quality of the pattern. The 5xMSAA sampling pattern can be advantageous when implementing various temporal anti-aliasing techniques.
Referring now to FIG. 5, diagrams of a 10xMSAA sub-pixel sampling pattern are shown. Diagrams 505, 510, and 515 are generated in a similar manner to the diagrams 305, 310, and 315 shown in FIG. 3. The 10xMSAA sampling pattern shown in diagrams 505, 510, and 515 is constructed from a first set of lines with a 3x1 slope and a second set of lines perpendicular to the first set of lines, resulting in 3^2 + 1^2 = 10 samples in each pixel of the image.
To create diagram 505, first a square pixel grid of the original pixels of an image is generated. The grid includes vertical and horizontal lines at the borders of the pixels of the source image. Then, a first set of lines are drawn through the corners of the pixel grid with a slope of -1/3, with each line passing through three horizontal pixels for each vertical pixel. Then, a second set of lines, perpendicular to the first set of lines, are drawn through the corners of the pixel grid. The second set of lines have a slope of 3, with each line passing through three vertical pixels for each horizontal pixel. The first set of lines and second set of lines can be referred to as a rotated grid.
Next, as shown in diagram 510, the rotated grid is shifted so that the vertices (i.e., intersections) of the first set of lines and second set of lines within each original pixel have a center of gravity at the center of the original pixel. Then, the coordinates of the vertices are programmed into the graphics hardware (e.g., GPU) to be utilized as the sub-pixel sampling coordinates of the image. Diagram 515 illustrates the sub-pixel sampling locations for a single pixel. Alternatively, the first and second sets of lines are drawn through the centers of the pixels of the grid rather than through the corners of the pixel grid.
Turning now to FIG. 6, diagrams of a 13xMSAA sub-pixel sampling pattern are shown. Diagrams 605, 610, and 615 are generated in a similar manner to the diagrams 305, 310, and 315 shown in FIG. 3 and diagrams 505, 510, and 515 shown in FIG. 5. The 13xMSAA sampling pattern shown in diagrams 605 and 610 is constructed based on lines with a 3x2 slope, resulting in 3^2 + 2^2 = 13 samples in each pixel of the image. Diagram 615 illustrates the 13 sub-pixel sample locations for a single pixel.
Referring now to FIG. 7, diagrams of a 17xMSAA sub-pixel sampling pattern are shown. Diagrams 705, 710, and 715 are generated in a similar manner to the previous diagrams. The 17xMSAA sampling pattern shown in diagrams 705 and 710 is constructed based on lines with a 4x1 slope, resulting in 4^2 + 1^2 = 17 samples in each pixel of the image. Diagram 715 illustrates the 17 sub-pixel sample locations for a single pixel.
Additional patterns can also be generated in a similar way to the patterns shown in FIGs 3 and 5-7. When generating additional patterns, the slope generating numbers (e.g., 3x1) are mutually prime (co-prime) so that the patterns satisfy the one-sample-per-horizontal/vertical-line condition. In other words, the greatest common divisor between the slope generating numbers should be one. For example, additional pattern options which meet this condition include 4x3 with 25 sub-pixel samples per pixel (sspp), 5x1 with 26 sspp, 5x2 with 29 sspp, 5x3 with 34 sspp, 5x4 with 41 sspp, 6x1 with 65 sspp, etc.
Turning now to FIG. 8, diagrams of a sampling pattern with 8 sub-pixel sampling points per original pixel are shown. Diagrams 805 and 810 illustrate a sub-pixel sampling pattern with non-mutually-prime slope generating numbers 2x2. Accordingly, the 8xMSAA sampling pattern shown in diagrams 805 and 810 is constructed based on lines with a 2x2 slope. The resulting pattern includes 8 sub-pixel samples per pixel as shown in diagram 815. For this 8 sub-pixel sampling pattern, there are 2 samples per pixel for each vertical and horizontal line. Accordingly, when anti-aliasing horizontal and vertical lines that are displayed in the source image, only 3 intermediate tones will be generated, which is less than the 4 intermediate tones generated by the 5xMSAA pattern shown in FIG. 3. However, the 8xMSAA pattern can be implemented efficiently for hybrid anti-aliasing resolve operations as will be described in further detail below. Another version of the 8xMSAA pattern can be generated from a non-rotated rectangular grid 4x2, with this other version also able to be implemented efficiently for hybrid anti-aliasing resolve operations.
Referring now to FIG. 9, diagrams of pixel-sharing patterns are shown. The previously described patterns shown in FIGs 3 and 5-8 can be shifted in such a way that one sub-pixel sampling location falls onto a corner or in the center of the source pixel. For example, diagram 905 shows a 5x sub-pixel sampling pattern, based on a slope of 2x1, with the sub-pixel sampling locations shifted so that one of the locations coincides with the lower right corner of the source pixel. The shifting of the pattern allows the sub-pixel location on the lower right corner of the pixel to be shared among four adjacent pixels. Diagram 910 illustrates an alternate configuration for the 5x sub-pixel sampling pattern. Diagram 910 illustrates the 5x sub-pixel sampling pattern shifted so that one of the sub-pixels is aligned with the center of the pixel.
Diagram 915 illustrates a 10x sub-pixel sampling pattern with a slope of 3x1, with the sub-pixel sampling pattern shifted so that one of the sub-pixel sampling locations falls on the lower-right corner of the pixel. Diagram 920 illustrates a 13x sub-pixel sampling pattern with a slope of 3x2, with the sub-pixel sampling pattern shifted so that one of the sub-pixel sampling locations falls on the lower-right corner of the pixel. Diagram 925 illustrates a 17x sub-pixel sampling pattern with a slope of 4x1, with the sub-pixel sampling pattern shifted so that one of the sub-pixel sampling locations falls on the lower-right corner of the pixel. Each of these sampling patterns allows one of the sub-pixel sampling locations to be shared among four adjacent pixels. Accordingly, four fragments will be accessed during the resolve operation rather than one fragment.
Creating a surface that contains 5, 10, 13, or 17 samples per fragment might not be optimal in terms of the actual hardware implementation. However, different types of workarounds can be implemented allowing these patterns with 5, 10, 13, or 17 samples per fragment to be efficiently implemented on existing or future hardware. For example, an 8x coverage sampling anti-aliasing (CSAA) pattern or a 4x enhanced quality anti-aliasing (EQAA) pattern each have eight coverage samples. The 8xCSAA pattern or the 4xEQAA pattern can be implemented to generate the 5x sub-pixel sampling pattern illustrated shown in diagrams 310 and 315 of FIG. 3. For 8xCSAA pattern or the 4xEQAA pattern, the extra three coverage samples are positioned to coincide with three of the first five positions of the 5x pattern. These extra sub-pixel samples are then depth-culled in the early stages of the rendering pipeline and thus will not waste rendering time. Alternatively, the hardware is configured to not calculate the unnecessary sub-pixel samples. This same approach can be implemented by using an existing 16x sub-pixel sampling mode (e.g., 16x CSAA, 16x Quincunx CSAA (QCSAA), 16x EQAA) to support a 10x or 13x sub-pixel sampling pattern.
When implementing a 5x sub-pixel sampling pattern, two surfaces are utilized for storage, with one surface implementing a 4x sub-pixel sampling pattern and the other surface implementing a 1x sub-pixel sampling pattern. In a similar manner, a 10x sub-pixel sampling pattern can be utilized by implementing an 8x sub-pixel sampling pattern surface and a 2x sub-pixel sampling pattern surface. Additionally, a 13x sub-pixel sampling pattern can consist of a 12x sub-pixel sampling pattern surface and a 1x sub-pixel sampling pattern surface, or the 13x sub-pixel sampling pattern can consist of a 8x sub-pixel sampling pattern surface, a 4x sub-pixel sampling pattern surface, and a 1x sub-pixel sampling pattern surface. Also, a 17x sub-pixel sampling pattern can be implemented using a 16x sub-pixel sampling pattern surface and a 1x sub-pixel sampling pattern surface.
One of the samples in a sub-pixel sampling pattern can be omitted from the pattern. For example, for a 17x sub-pixel sampling pattern, only 16 sub-pixel sample locations can be calculated. The omitting of sampling locations can cause holes in the regular grid pattern, but these holes could be programmatically interpolated during the resolve operation. A GPU is able to programmatically change the sub-pixel sampling patterns, but there could be a specific granularity to the positions of the samples. For example, a grid of 16x16 possible positions inside of the pixel can be available in one embodiment. Accordingly, some of the patterns described herein might not be capable of being implemented with absolute procession, but an approximated pattern where sample positions are adjusted to the closest grid points can be implemented and yield suitable results.
Turning now to FIG. 10, a diagram of one embodiment of a hybrid anti-aliasing scheme is shown. Generally speaking, there are typically two families of anti-aliasing algorithms: multi-sampling anti-aliasing and post-processing anti-aliasing. These two methods are typically mutually exclusive. A pass of post-processing anti-aliasing can be applied to an image resolved from a multi-sampled image. Typically, the resultant image will become blurry and possibly worse than before. However, the techniques described herein for implementing hybrid anti-aliasing schemes are an improvement on the prior art.
In one embodiment, a hybrid anti-aliasing scheme starts with a multi-sampled image, where the sub-pixel sampling pattern generates a regular grid within the image. Any of the previously described sub-pixel sampling patterns (e.g., 5x, 10x, 13x, 17x) can be utilized. Next, the multi-sampled image is expanded to an image at the sub-pixel sampling resolution. Then, a post-processing anti-aliasing algorithm is used to filter the expanded image. Finally, the filtered expanded image is averaged back to an image at the original resolution. This approach can be implemented on current graphics processing hardware (e.g., a GPU). For example, the sub-pixel sampling locations can be programmed in the GPU to support the new sub-pixel sampling pattern. This approach works with rotated and non-rotated grid patterns. In some embodiments, this approach can be combined with temporal filter techniques as well as with reprojection techniques. In one embodiment, existing games can utilize the hybrid anti-aliasing scheme by updating graphics driver software.
In one embodiment, a hybrid anti-aliasing scheme can be implemented for the scenario shown in the block diagram of FIG. 10. In this embodiment, image 1005 with dimensions width (W) by height (H) utilizes multi-sampling (i.e., sub-pixel sampling) based on a 2x2 ordered grid (OG) as shown in diagram 1010. Image 1005 is expanded by treating the sub-pixels as if they were regular pixels. The resultant image 1015 has dimensions 2W by 2H. In one embodiment, tone mapping for high dynamic range (HDR) textures is integrated into the expansion step.
Next, image 1015 is processed by a post-processing anti-aliasing filter to generate image 1020. Also, in some embodiments, additional post-processing, such as sharpening, is applied to the expanded image 1015. Then, an averaging filter is utilized to generate image 1025 from image 1020. It is noted that images 1005, 1015, 1020, and 1025 can also be referred to as textures. In one embodiment, the averaging filter is a box averaging filter (e.g., 2x2 box averaging filter). In other embodiments, other types of averaging filters (e.g., tent filter) can be utilized. Additionally, the averaging filter can include inverse tone mapping for high dynamic range (HDR) textures, sharpening, and/or others. It is noted that image 1025 has the same dimensions, W by H, as the original image 1005. In other embodiments, the averaging filter can change the resolution of image 1025 to a different resolution than the original image 1005. Alternatively, the downscaling step using the averaging filter can be omitted, with the resultant image having a larger resolution than the starting image.
Referring now to FIG. 11, a diagram of another embodiment of a hybrid anti-aliasing scheme is shown. In one embodiment, a hybrid post-processing, anti-aliasing scheme is applied to image 1105, which is representative of any type of image, texture, surface, or other pixel data. Image 1105 has dimensions of W by H, and image 1105 utilizes a x5 MSAA sub-pixel sampling pattern based on a slope of 2x1 as shown in diagram 1110. The sub-pixel samples of image 1105 are expanded and then rotated for alignment to generate image 1115, which is the image at its natural resolution without multi-sampling. In other words, the sub-pixels of image 1105 are treated as pixels in image 1115. Also, the pixels are rotated so that the pixels in image 1115 form a regular, rectangular grid.
Next, a post-processing, anti-aliasing filtering step is performed on image 1115 to generate image 1120. Then, an averaging filter step is performed on image 1120 to generate image 1125, which has the same dimensions WxH as the original image 1105. In another embodiment, the averaging filter step can be omitted, and image 1120 can be the result of the hybrid anti-aliasing resolve operation. Alternatively, in a further embodiment, the averaging filter can average image 1120 so that the resolution of image 1125 is greater than the original resolution WxH of image 1105.
Turning now to FIG. 12, a diagram of a technique for reducing the size of an expanded image is shown. Image 1205 is an example of an expanded image similar to image 1115 in FIG. 11. Image 1205 includes an expanded and rotated image created from a sub-pixel sampled image (not shown). However, each of the areas containing an "X" within image 1205 corresponds to unused space that will consume additional memory.
Accordingly, in one embodiment, instead of processing image 1205 in its current state, the portions of image 1205 are rearranged to create image 1210 which results in a more efficient usage of space. To rearrange portions into a more space-efficient image 1210, triangular portion 1215 is moved from the right-side of image 1205 to the left-side of image 1210. Also, triangular portion 1220 is moved from the top of image 1205 to the bottom right of image 1210. In the upper right portion of image 1210 there is still a small portion of unused space which is marked with an "X". However, this is much smaller than the amount of unused space in image 1205. Within image 1210, portions 1215, 1220, and 1225 fit together in a more efficient arrangement with less wasted space. This will allow the next post-processing, anti-aliasing step to be performed more efficiently on image 1210 than on image 1205.
It is noted that when rearranging the image by moving the portions around, some overlap can be included with each portion to allow the post-processing algorithms to be performed. For example, if the post-processing algorithm samples five pixels around the center pixel, then an overlap of five pixels is added to the portions 1215, 1220, and 1225 as they are combined to form image 1210. It is also noted that the rearranging of portions 1215, 1220, and 1225 can be performed during the expansion of the sub-pixels stage. Accordingly, the rearranging of portions 1215, 1220, and 1225 does not need an additional rendering pass. It is further noted that the use of alternate triangle subdivision and/or alternate rearrangement patterns are possible and are contemplated.
Referring now to FIG. 13, one embodiment of a method 1300 for performing a hybrid anti-aliasing resolve operation is shown. For purposes of discussion, the steps in this embodiment and those of FIG. 14-17 are shown in sequential order. However, it is noted that in various embodiments of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems, apparatuses, or computing devices described herein are configured to implement method 1300.
A processor detects a request to perform an anti-aliasing resolve operation on a first image with multiple sub-pixel samples per pixel and a first set of dimensions (block 1305). In one embodiment, the processor is a GPU. In other embodiments, the processor is any of various other types of processors ((e.g., DSP, FPGA, ASIC, multi-core processor). Depending on the embodiment, the first image can be a texture, surface, or other type of pixel data being processed by the processor.
In response to detecting the request, the processor expands dimensions of the first image to create a second image that includes a single sample per pixel and has a second set of dimensions greater than the first set of dimensions (block 1310). In one embodiment, expanding dimensions of the first image involves converting sub-pixels of the first image into regular pixels of the second image. In other words, the sub-pixels of the first image will be treated as if they were actual pixels of the second image. The expansion of the dimensions of the first image will vary according to the number of sub-pixels sampling locations per pixel of the first image. It is noted that expanding dimensions of the first image to create a second image can be performed virtually by performing subsequent calculations as if the first image were expanded without actually doing the expansion and without creating a second image. In some embodiments, in addition to expanding dimensions of the first image, the first image is also rotated. In these embodiments, the rotation is performed to align the sub-pixels of the first image into a vertical and horizontal grid pattern within the second image.
Next, the processor performs post-processing anti-aliasing on the second image to create a third image with the second set of dimensions (block 1315). In some embodiments, block 1315 can be performed virtually without actually creating the third image. The processor utilizes an anti-aliasing filter to perform the anti-aliasing processing. The anti-aliasing filter can also be referred to as a post-processing, anti-aliasing filter. In one embodiment, the anti-aliasing filter is based on the fast approximate anti-aliasing (FXAA) algorithm. In other embodiments, the anti-aliasing filter can utilize other algorithms for filtering the second image to create the third image. Generally speaking, an anti-aliasing filter analyzes a central pixel and then the surrounding pixels to determine if there any unwanted aliasing effects (i.e., jaggies) that will be displayed in the vicinity of the central pixel. If the anti-aliasing filter detects the presence of artifacts for the central pixel, the value of the central pixel can be averaged with the neighboring pixels.
Then, the processor reduces dimensions of the third image to create a fourth image at the desired resolution (block 1320). In one embodiment, reducing dimensions involves performing an averaging of the third image to create the fourth image. In one embodiment, the averaging of the third image can be performed in-place without creating a fourth image. It is noted that depending on the embodiment, subsequent to block 1320, the fourth image can be written back to memory, the fourth image can undergo additional processing, and/or the fourth image can be driven to a display. In one embodiment, the fourth image has a same resolution as the first image. In another embodiment, the fourth image has a higher resolution than the first image. In some embodiments, block 1320 can be omitted from method 1300, with the third image being the result of the anti-aliasing resolve operation. For example, in some applications (e.g., virtual reality (VR) applications), having access to a higher resolution image can be useful when warping (i.e., reprojecting) the rendered image before sending it to the display to correct for head movement which occurred after the rendering. After block 1320, method 1300 ends.
It should be understood that the terminology of first image, second image, third image, and fourth image utilized in describing method 1300 is intended to distinguish between the different steps of method 1300. Each such term is meant to indicate that some processing has been applied to a source image to create a modified source image. For example, when expanding dimensions of the first image to create the second image, the processor can expand the dimensions virtually by performing subsequent calculations as if the first image were expanded and without actually creating a second image. Additionally, when the processor performs an averaging of the third image, the processor can overwrite the third image with the averaged version of the third image. From this perspective, the third image can be averaged and then the output of the averaging can still be called the third image. However, for the purposes of clarity, the output of averaging the third image is referred to as the fourth image in method 1300 to clearly distinguish between the input and output of the averaging step. Similarly, the other steps of method 1300 can actually overwrite or modify the source image rather than creating a new image.
Turning now to FIG. 14, one embodiment of a method 1400 for rearranging image portions when expanding and rotating an image is shown. A processor rearranges a plurality of triangular portions from a first image into new locations within a second image (block 1405). In one embodiment, the processor rearranges the plurality of triangular portions from the first image into new locations within the second image as part of a hybrid anti-aliasing resolve operation. When the processor expands and rotates the first image into the second image as part of the hybrid anti-aliasing resolve operation, the processor also rearranges the plurality of triangular portions from the first image into new locations within the second image. Also, the processor rearranges any remaining portions of the first image into new locations within the second image in between the plurality of triangular portions (block 1410). After block 1410, method 1400 ends. By rearranging the image portions of the first image into new locations in the second image, subsequent processing operations on the second image can be performed more efficiently by the graphics hardware.
Referring now to FIG. 15, another embodiment of a method 1500 for rearranging image portions when expanding and rotating an image is shown. A processor moves a first triangular portion from the left-side of a source image to the right-side of an expanded version of the source image (block 1505). Also, the processor moves a second triangular portion from the right-side of the source image to the left-side of the expanded version of the source image (block 1510). Still further, the processor moves a third portion of the source image to fit between the first triangular portion and the second triangular portion within the expanded version of the source image (block 1515). It is noted that in one embodiment, the processor includes an overlap of sub-pixels on the edges of the first, second, and third portions when moving the first, second, and third portions to their new positions in the expanded version of the source image. After block 1515, method 1500 ends.
Turning now to FIG. 16, a method 1600, which is not part of the invention, for sampling sub-pixel locations of an image is shown. A processor generates, based on a rotated grid, a plurality of sub-pixel sampling coordinates for each pixel of an image being rendered, wherein an angle of rotation is specified for the rotated grid (block 1605). The angle of rotation can also be referred to more generally as an amount of rotation. Next, the processor samples sub-pixel locations indicated by the plurality of sub-pixel sampling coordinates within each pixel of the image being rendered (block 1610). Then, the processor stores values of the sub-pixel locations in a memory (block 1615). It is noted that the memory represents a cache, local data share, global data share, memory device, or any other suitable collection of storage elements. After block 1615, method 1600 ends.
Referring now to FIG. 17, a method 1700, which is not part of the invention, for generating a plurality of sub-pixel sampling coordinates for each pixel of an image being rendered is shown. A processor calculates locations that correspond to a first set of parallel lines which pass diagonally through given locations of an image (block 1705). It is noted that to "pass diagonally" is defined as traversing the image at an angle other than 90 or 180 degrees. In other words, the first set of parallel lines are not parallel with either the horizontal or vertical lines of a square grid marking the boundaries of the pixels of the image. The given locations are corners of the pixels. Alternatively, the given locations are centers of the pixels. Alternatively, the slope of each line of the first set of parallel lines is generated from a ratio of two mutually prime numbers. For example, alternatively, the slope of each line of the first set of parallel lines is two pixels by one pixel. Alternatively, the slope can be other suitable ratios. Alternatively, the distance between adjacent lines is constant for each pair of adjacent lines of the first set of parallel lines.
Next, the processor calculates locations that correspond to a second set of parallel lines perpendicular to the first set of parallel lines, and wherein the first and second sets of parallel lines form a rotated grid (block 1710). The distance between adjacent lines is constant for each pair of adjacent lines of the second set of parallel lines. Alternatively, the distance between adjacent lines of the first set of parallel lines is equal to the distance between adjacent lines of the second set of parallel lines. Then, the processor shifts the rotated grid to cause the vertices of the first and second sets of parallel lines to have a center of gravity which is in a center of each corresponding pixel (block 1715). Next, the processor utilizes the locations of the vertices of the first and second sets of parallel lines to specify the plurality of sub-pixel sampling coordinates within pixels of the image (block 1720). After block 1720, method 1700 ends. Implementing method 1700 will cause the sub-pixel sampling coordinates to be regularly spaced both horizontally and vertically. Additionally, implementing method 1700 will result in there being only one sub-pixel sampling location per pixel on a given horizontal or vertical line which traverses the image. It is noted that other methods can be implemented to achieve similar sub-pixel sampling patterns as those generated by method 1700. In general, for a square grid that has a density of "a^2+b^2" sub-samples per pixel (wherein a and b are integers), if the grid is rotated or mirrored at an angle of "atan(a/b)+90^∗n" (wherein n is an integer), then the sampling pattern within each pixel will be identical.
Turning now to FIG. 18, a method, which is not part of the invention, for generating sub-pixel sampling coordinates is shown. A processor specifies an angle for a rotated grid (block 1805). For example, the processor could specify that a first set of lines of the rotated grid have a slope of 2x1, which refers to lines which are parallel to a diagonal of a rectangle two pixels wide and one pixel high. A second set of lines of the rotated grid are perpendicular to the first set of lines. Next, the processor superimposes the rotated grid over an image being rendered (block 1810). Then, the processor generates, based on the rotated grid, a plurality of sub-pixel sampling coordinates for each pixel of the image being rendered (block 1815). After block 1815, method 1800 ends.
Referring now to FIG. 19, one embodiment of a method for performing an anti-aliasing operation is shown. A processor detects a request to perform an anti-aliasing operation on an image (block 1905). It is noted that the anti-aliasing operation can also be referred to as an anti-aliasing resolve operation. In response to detecting the request, the processor expands dimensions of the image (block 1910). In one embodiment, the processor performs a virtual expansion of the image in block 1910. The virtual expansion can be performed by creating a new image that is larger or by treating the original image as a larger image. In one embodiment, the processor treats the original image as a larger image that is rotated. In one embodiment, expanding dimensions of the image is defined as converting the sub-pixels into regular pixels. In some embodiments, the image is also rotated to cause the regular pixels to be aligned in a rectangular grid. Next, the processor performs anti-aliasing processing on the image (block 1915). Then, the processor performs an averaging of the image to create the final result of the anti-aliasing operation at the desired resolution (block 1920). After block 1920, method 1900 ends.
Turning now to FIG. 20, a method, which is not part of the invention, for generating a sampling grid is shown. A slope (e.g., 2x1) is selected for a sampling grid (block 2005). Next, lines at the selected slope are drawn passing through all grid crossings of a pixel grid (block 2010). Alternatively, the lines at the selected slope are drawn passing through all pixel centers of the pixel grid in block 2010. Then, perpendicular lines are drawn passing through all grid crossings of the pixel grid (block 2015). Alternatively, the perpendicular lines are drawn passing through all pixel centers of the pixel grid in block 2015. The lines drawn in blocks 2010 and 2015 will generate a rotated regular grid. This rotated regular grid can also be referred to as the sampling grid.
Next, in an optional step, the sampling grid is shifted, rotated by a multiple of 90 degrees, and/or mirrored (block 2020). The pattern created by the grid crossings of the sampling grid within a single pixel is utilized as a sampling pattern for an image (block 2025). The sampling pattern within a single pixel will be identical for all pixels in the image for the sampling pattern created by method 2000. After block 2025, method 2000 ends. It is noted that any of the systems and/or apparatuses described herein can be configured to perform method 2000.
In various embodiments, program instructions of a software application are used to implement the methods and/or mechanisms previously described. The program instructions describe the behavior of hardware in a high-level programming language, such as C. Alternatively, a hardware design language (HDL) is used, such as Verilog. The program instructions are stored on a non-transitory computer readable storage medium. Numerous types of storage media are available. The storage medium is accessible by a computing system during use to provide the program instructions and accompanying data to the computing system for program execution. The computing system includes at least one or more memories and one or more processors configured to execute program instructions.
It should be emphasized that the above-described embodiments are only nonlimiting examples of implementations. The invention is defined by the appended claims.

Claims

A system for performing anti-aliasing related operations comprising:
a memory; and

a processor;

wherein the processor is configured to:
detect a request to perform an anti-aliasing resolve operation (1305) on a first

image stored in the memory, wherein the first image includes multiple sub-pixel samples per pixel and has a first set of dimensions;
responsive to detecting the request:
expand dimensions of the first image (1310) to create a second image

such that each pixel of the second image has a single sample represented by one of the sub-pixel samples of the first image and has a second set of dimensions greater than the first set of dimensions;
perform post-processing anti-aliasing (1315) on the second image to

create a third image with the second set of dimensions; and
reduce dimensions of the third image (1320) to create a fourth image,

wherein the fourth image is a result of the anti-aliasing resolve operation.
The system as recited in claim 1, wherein performing post-processing anti aliasing comprises filtering pixels based on values of surrounding pixels, and wherein the anti-aliasing processing is performed prior to the averaging.
The system as recited in claim 1, wherein expanding dimensions of the first image comprises converting sub-pixels of the first image into regular pixels of the second image.
The system as recited in claim 1, wherein responsive to the request, the processor is further configured to rotate the first image such that the multiple sub-pixel samples align with pixels of the second image.
The system as recited in claim 4, wherein the processor is configured to rotate the first image to align the sub-pixels into a vertical and horizontal grid pattern within the second image.
The system as recited in claim 5, wherein the processor is configured to:
rearrange a plurality of triangular portions (1405) from the first image into new locations within the second image; and

rearrange any remaining portions (1410) of the first image into new locations within the second image in between the plurality of triangular portions which have been rearranged.
The system as recited in claim 6, wherein the processor is configured to:
move a first triangular portion (1505) from a left-side of the first image to a right-side of the second image; and

move a second triangular portion (1510) from a right-side of the first image to a left-side of the second image; and

move a third portion (1515) of the first image to fit between the first triangular portion and the second triangular portion within the second image.
A method for use in a computing device, in accordance with the system of claims 1 - 7.