JP2014219739A - Image processing apparatus which processes graphics by dividing space, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which processes graphics efficiently at a high speed in space-partitioning drawing, and an image processing method.SOLUTION: A compiler processing unit outputs a shader program divided into a pre-processing shader formed by extracting a function required for generating a color/depth value and a post-processing shader. A spatial index addition unit generates a spatial index after vertex processing to create a vertex buffer with spatial index. A pre-processing unit performs rasterization to acquire a pixel for each space defined by the spatial index, and generates a color value and a depth value on the basis of the pre-processing shader. A post-processing unit performs texture shading processing based on the post-processing shader, on a pixel decided to be written on an output color buffer. A pixel output unit outputs a color value of the pixel obtained by the pre-processing unit and the post-processing unit to the output color buffer.

Description

  The present invention relates to an image processing apparatus and an image processing method capable of performing graphics drawing processing after dividing a space. More specifically, according to the present invention, when graphics are drawn by dividing a space, the drawing needs to be updated when the drawing needs to be updated by a user's screen operation or the like. Polygon drawing update can be executed for only space. As a result, the image processing apparatus of the present invention speeds up the drawing process and minimizes the drawing delay.

  Conventionally, in fields such as game machines and car navigation devices, 3D image display devices that draw 3D computer graphics in real time have become widespread. In the three-dimensional computer graphics, a polygon model that represents an object in a three-dimensional space with a large number of polygons is generally used. In the polygon model drawing process, shading is performed to shade the polygon surface in consideration of the light source, viewpoint position, line-of-sight direction, reflectance of the object surface, and the like. In addition, in order to generate an image with high realism, texture mapping is performed by attaching a texture image to the surface of the polygon model.

  The three-dimensional image display device is realized by, for example, a graphic processing chip. This graphics processing chip is equipped with a processor, and this processor has a function specialized for image processing by incorporating a shader in which a programmer defines an image generation algorithm. A shader is primarily a combination of software instructions used on graphic resources to perform lighting (light source calculations), shading (shading), rendering (pixelation), and the like. As a method of displaying a three-dimensional image on a graphic processing chip, processing is performed on the CPU side, and a shader mounted on the graphic processing chip is configured to be programmable, and an object finally rendered through each shader is processed. It is also possible to perform processing.

  In recent years, a system for performing graphics processing on a tile basis has been developed (Patent Document 1, etc.). In order to display each frame, the tile-based graphics system divides the frame into a plurality of areas called tiles. The system also has a binning section for displaying each graphics primitive (point, line, triangle, etc.) in the frame. The binning unit is used to perform a binning operation for determining the intersection of graphics primitives among various tiles. In other words, the binning unit basically designates a plurality of bins (for example, one bin for each tile) and, based on the coordinate data for the graphics primitive, the tile (single or plural) that the graphics primitive intersects with. ) And the process of recording the identifier of the graphics primitive in the associated bin for each intersecting tile. In this graphics system, after the binning operation is finished, graphics data is generated for each tile. That is, when the binning operation is completed, a specific tile is assigned to the rasterization unit of the graphics system, and graphics data for the assigned tile is generated according to a tile list applicable to the tile.

JP 2012-168951 A

  Incidentally, the conventional tile-based graphics system divides each frame uniformly into a plurality of areas called tiles. For example, in the system disclosed in Patent Document 1, each frame is divided into 256 tiles, and each tile is determined to have a resolution of 256 × 256 pixels. However, the space area that requires high-speed processing differs depending on the nature of graphics for which rendering is updated. For example, depending on the frame space, there are areas where primitives are densely populated and areas where primitives are scattered. On the other hand, if each frame is divided and processed as in the conventional system, the spatial area that requires high-speed processing is equal even in areas that do not require high-speed processing. Since tiles are assigned to each other, graphics processing cannot be advanced especially in a space area that requires high-speed processing, which is inefficient. In addition, when performing tile-based graphics processing as in the conventional tile-based graphics system, if there are many primitives that straddle multiple tiles, this will lead to processing delay. Each tile has the same size, and the size of each tile cannot be changed. Therefore, in the conventional system, it is not possible to flexibly deal with such as changing the size of the division according to the size of the primitive, and this also causes a problem that the efficiency of graphics is lowered. there were. As described above, in the conventional technique, it is difficult to explicitly perform the space area division in order to distribute the processing of a heavy part of the application. Furthermore, when updating a part of an arbitrary screen, the conventional technology cannot execute the process by dividing only the arbitrary part of the space.

  For this reason, at the present time, an image processing device capable of processing graphics efficiently and at high speed is required for space division drawing.

Therefore, as a result of intensive studies on the means for solving the above-described problems of the conventional invention, the inventor of the present invention has performed vertex processing (coordinate conversion and illuminance calculation) on the vertex data constituting the polygon, and the user then Creates a spatial index based on given spatial information, etc., creates a spatial indexed vertex buffer that stores the polygon configuration information, processed vertex data, and spatial index in association, and creates this indexed vertex buffer By performing graphics processing using, it is possible to flexibly determine the position, size, and range of the space to be divided according to the nature of the graphics, and to render graphics efficiently and quickly The knowledge that processing was realizable was acquired. The inventor has conceived that the problems of the prior art can be solved based on the above knowledge, and has completed the present invention.
More specifically, the present invention has the following configuration.

A first aspect of the present invention relates to an image processing apparatus for computer graphics processing.
The image processing apparatus of the present invention includes a compiler processing unit 10, a spatial index addition unit 20, a pre-processing unit 30, a post-processing unit 40, and a pixel output unit 50.
The compiler processing unit 10 reads a shader program in a graphics application, extracts a function necessary for generating a color / depth value, and performs a texture / shading process not included in the preceding process shader. Output post-processing shader including it.
The spatial index adding unit 20 reads vertex data constituting a polygon in the graphics application, performs vertex processing including coordinate conversion, obtains processed vertex data, and generates a spatial index in the coordinate system after conversion. , A vertex buffer 21 with a spatial index in which the configuration information of the polygon, the processed vertex data, and the spatial index are stored in association with each other is created.
The pre-processing unit 30 reads the vertex buffer 21 with the spatial index to construct a polygon, performs rasterization processing on the polygon for each space defined by the spatial index, converts the polygon into pixels, and based on the pre-processing shader , Generate color values and depth values of rasterized pixels.
The post-processing unit 40 performs a texture shading process based on the post-processing shader on the pixel determined to be written to the output color buffer 51 with reference to the depth value of the rasterized pixel, and the texture shading process Generate the color value of the completed pixel.
The pixel output unit 50 outputs the pixel color values obtained by the pre-processing unit 30 and the post-processing unit 40 to the output color buffer 51.

  Here, in the image processing apparatus of the present invention, the pre-processing unit 30 preferably includes a pre-processor core 31 that performs rasterization processing, and the post-processing unit 40 preferably includes a post-processor core 41 that performs texture shading processing. . The front processor core 31 and the rear processor core 41 are preferably different hardware.

  In the image processing apparatus of the present invention, it is preferable that the space index adding unit 20 performs space division based on space information arbitrarily given by the user and generates a space index.

In the image processing apparatus of the present invention, the compiler processing unit 10 preferably includes a front end processing unit 11, a function extraction unit 12, and a back end processing unit 13.
The front-end processing unit 11 performs front-end processing on the shader program and converts it into an intermediate representation.
The function extraction unit 12 performs turn-match processing between the intermediate representation and a processing pattern relating to generation of the color / depth value prepared in advance, so that functions necessary for generating the color / depth value can be obtained from the intermediate representation. Extract.
The back-end processing unit 13 performs back-end processing on the intermediate representation, outputs functions necessary for generating the color / depth values extracted by the function extracting unit 12 as a pre-processing shader, Functions other than those necessary for generating color / depth values are output as a post-processing shader.

The second aspect of the present invention relates to an image processing method executed by an image processing apparatus for computer graphics processing.
In the image processing method of the present invention, the image processing apparatus performs a compiler processing step, a spatial index addition step, a pre-stage processing step, a post-stage processing step, and a pixel output step.
The compiler processing unit 10 reads a shader program in the graphics application, extracts a function necessary for generating a color / depth value, and performs a texture / shading process not included in the preceding process shader. Output post-processing shader including it.
In the spatial index addition process, the vertex data constituting the polygon in the graphics application is read and the vertex processing including coordinate transformation is performed to obtain the processed vertex data, and the spatial index in the coordinate system after the transformation is generated, A vertex buffer 21 with a spatial index that stores the configuration information of the polygon, the processed vertex data, and the spatial index in association with each other is created.
In the pre-processing unit process, the vertex buffer 21 with a spatial index is read to construct a polygon, and for each space defined by the spatial index, the polygon is rasterized and converted into pixels. Based on the pre-processing shader , Generate color values and depth values of rasterized pixels.
In the post-processing unit process, texture shading processing based on the post-processing shader is performed on the pixels determined to be written to the output color buffer 51 with reference to the depth value of the rasterized pixel, and the texture shading processing is performed. Generate the color value of the completed pixel.
In the pixel output unit process, the color value of the pixel obtained by the pre-processing process and the post-processing process is output to the output color buffer 51.

  Another aspect of the present invention relates to a computer including the image processing apparatus according to the first aspect.

  Another aspect of the present invention relates to a game machine including the image processing apparatus according to the first aspect.

  The present invention generates a space index and performs space-division drawing based on the space index, so that even when a drawing update is required due to a user's screen operation or the like, the space that needs to be updated is updated. Only the polygon drawing update can be executed. Accordingly, the present invention can speed up the drawing process and minimize drawing delay. The space division in the present invention does not uniformly divide a frame into a plurality of areas (tiles), but a plurality of spaces in a coordinate system after conversion by rasterization are divided into a plurality of areas based on arbitrarily given space indexes. Since it can be divided into regions, the size of space can be determined flexibly according to the nature of graphics. In addition, since the image processing apparatus according to the present invention has the above-described mechanism, drawing processing can be performed in a local buffer, so that access to an external memory can be reduced, resulting in power consumption. In addition, the overhead during memory access is reduced, and the graphics processing speed can be increased.

  Furthermore, the image processing apparatus of the present invention divides the shader program into a front-end processing shader mainly having functions necessary for generating color and depth values, and a rear-stage processing shader mainly required for texture shading processing. To do. Then, after processing the pixels in the pre-processing unit based on this pre-processing shader, only the pixels that require texture / shading processing are input to the post-processing unit and the processing based on the post-processing shader is executed. is there. Therefore, it is not necessary to perform unnecessary pixel processing in the post-processing unit, and the processing time can be reduced. Further, the shader processed by the pre-processing unit is extracted by a compiler limited to functions necessary for generating color / depth values. Therefore, since the pre-processing unit only needs to process a shader from which unnecessary processing is omitted, the size of the processor core for the pre-processing unit can be minimized, resulting in cost reduction.

  As described above, the present invention can provide an image processing apparatus and the like capable of processing graphics efficiently and at high speed when performing space division drawing.

FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to the present invention. FIG. 2 is a flowchart showing an example of processing in the compiler. FIG. 3 is a flowchart showing an example of processing in the spatial index adding unit. FIG. 4 shows an example of the data structure in the vertex buffer with spatial index. FIG. 5 is a block diagram showing a modification of the image processing apparatus according to the present invention. FIG. 6 is a block diagram illustrating an example of the configuration of a computer for graphics processing. FIG. 7 is a block diagram showing an example of the configuration of the game machine.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, but includes those appropriately modified by those skilled in the art from the following embodiments.

[1. Image processing apparatus and image processing method]
FIG. 1 shows a block diagram of an image processing apparatus 1 according to the present invention. The image processing apparatus 1 according to the present invention is used as hardware for three-dimensional computer graphics processing.
As shown in FIG. 1, the image processing apparatus 1 reads a graphics application 60, performs a predetermined graphics drawing process, and outputs the result of the drawing process to the output color buffer 51. The image processing apparatus 1 includes a compiler processing unit 10 shown in the upper part of FIG. 1 and a graphics pipeline shown in the lower part of FIG. The graphics pipeline includes a spatial index adding unit 20, a pre-processing unit 30, a post-processing unit 40, and a pixel output unit 50. The compiler processing unit 10 reads a shader program 61 from the graphics application 60 and divides the shader program 61 into a pre-processing shader 62 and a post-processing shader 63. In other words, the compiler processing unit 10 performs a process of assigning commands described in the shader program 61 to the pre-processing unit 30 and the post-processing unit 40. In the graphics pipeline, after the spatial index adding unit 20 performs processing for creating the spatial index-added vertex buffer 21, the pre-processing unit 30 reads the spatial index-added vertex buffer 21 and performs rasterization processing for each space. , Generate a color value and a depth value of the pixel based on the preprocessing shader. Thereafter, the post-processing unit 40 performs texture processing and shading processing on necessary pixels based on the post-processing shader 63. The pixel output unit 50 outputs the result of the pixel processing in the pre-processing unit 30 and the post-processing unit 40 to the output color buffer 51. The pixel color values stored in the output color buffer 51 are referred to when a moving image or a still image is displayed by a known display device.

  A shader program 61 in the graphics application 60 is input to the compiler processing unit 10. The compiler processing unit 10 analyzes and compiles the shader program 61 to output a pre-processing shader 62 and a post-processing shader 63, which are compiled binary files. The pre-processing shader 62 is a shader code that extracts functions necessary for generating pixel color values and depth values. The pre-processing shader 62 includes, for example, a depth test, an alpha test, a stencil test, and the like as processes related to generation of color values and depth values. The post-processing shader 63 is a shader code other than the pre-processing shader 62 in the shader program 61. The post-processing shader 63 includes at least functions necessary for texture processing and shading processing. For example, the post-processing shader 63 may include environment mapping processing, light refractive index calculation processing, and the like in addition to texture processing and shading processing.

  As shown in FIG. 1, the compiler processing unit 10 includes a front-end processing unit 11, a function extraction unit 12, and a back-end processing unit 13. In the compiler processing unit 10, the front-end processing unit 11 converts the programming language source code into an intermediate representation, and the back-end processing unit 13 further converts the intermediate representation into an output language binary code such as a machine language. Generate. At this time, the back end processing unit 13 generates optimized code that operates at high speed in the back end. In the compiler processing unit 10, a function extraction unit 12 is provided between the front end processing unit 11 and the back end processing unit 13. The function extraction unit 12 extracts a shader code (predecessor processing shader 62) from which functions necessary for generating the color value and depth value of the pixel are extracted from the commands described in the shader program 61, and other functions. Is output separately from the shader code (the post-processing shader 63) having

  When the shader program 61 in the graphics application 60 is input, the front-end processing unit 11 performs general front-end processing such as syntax analysis, and converts it into an intermediate representation indicating the intra-shader operation function. For example, in syntax analysis, the source program is converted into an intermediate representation (intermediate program) by performing lexical analysis and semantic analysis of the source program.

  The intermediate representation after the front-end processing is analyzed by the function extraction unit 12. The function extraction unit 12 holds in advance processing patterns for functions necessary for generating pixel color values and depth values. This processing pattern may be arbitrarily set by the user, or may be in accordance with the function of the processor core that generates the color value and the depth value. For example, the function extraction unit 12 may analyze the processing function of the processor core and hold the one that conforms to a format that can be processed by the processor core as the processing pattern. Here, the function extraction unit 12 performs a matching process between the intermediate representation and a processing pattern for color values and depth values prepared in advance. Then, the function extraction unit 12 extracts a portion (function) that matches the processing pattern in the intermediate representation as a function necessary for generating the color value and the depth value. The processing pattern required for matching is obtained by analyzing the capability of the processor core included in the image processing apparatus 1 and storing the processing pattern executable in the processor core in a local memory for graphics processing in advance. Good. For example, as processing patterns required for matching, for example, information including a conditional expression of a depth test or an alpha test can be used. In this way, the function extraction unit 12 extracts the functions necessary for generating the color value and the depth value from the shader program 61, thereby converting the intermediate representation after the front end processing into the color value and the depth value. Separate the intermediate representation for functions required for generation and the intermediate representation for functions other than this function.

  After the function extraction processing in the function extraction unit 12, the back end processing unit 13 performs back end processing and generates shader codes corresponding to the pre-processing unit 30 and the post-processing unit 40, respectively. That is, the back-end processing unit 13 performs the back-end processing on the functions (intermediate representation) necessary for generating the color value and the depth value extracted by the function extracting unit 12, and outputs the pre-processing shader 62. At the same time, the back-end processing unit 13 outputs a post-processing shader 63 by performing back-end processing for functions not extracted by the function extracting unit 12, for example, functions necessary for texture processing and shading processing (intermediate representation). To do. As described above, the back-end processing unit 13 performs back-end processing on each of intermediate representations for functions necessary for generating color values and depth values and intermediate representations for functions other than this function. A pre-processing shader 62 for the pre-processing unit 30 and a post-processing shader 63 for the post-processing unit 40 are output. In addition, the back-end processing unit 13 converts each intermediate representation into an optimized code that operates at high speed in the hardware provided in the pre-processing unit 30 or the hardware provided in the post-processing unit 40. What is necessary is just to perform backend processing.

  Here, FIG. 2 shows an embodiment of processing steps performed by the compiler processing unit 10. As shown in FIG. 2, the shader program input to the compiler processing unit 10 is subjected to front-end processing such as syntax analysis, and then converted to an intermediate representation indicating an intra-shader operation function. Thereafter, pattern matching processing is performed between the converted intermediate representation and processing patterns for color values and depth values prepared in advance. Then, the portion of the intermediate representation that matches the processing pattern for the color value and the depth value is subjected to output processing as the pre-stage processing shader 62 through back-end processing. On the other hand, the portion of the intermediate representation that does not match the processing pattern for the color value and the depth value is subjected to output processing as a back-end processing shader 63 through back-end processing.

  In the present embodiment, the compiler processing by the compiler processing unit 10 is performed when the graphics application 60 is executed. For example, the compiler processing may be performed in advance before executing the application. Good. For example, before executing the graphics application 60, the graphics application 60 may be analyzed, and a portion matching the processing pattern for the color value and the depth value may be extracted and held in advance.

  Thereafter, an actual graphics rendering process is performed by the graphics pipeline shown in the lower part of FIG. 1 using a graphics program including the pre-processing shader 62 and the post-processing shader 63 output from the compiler processing unit 10. Is done.

  As shown in FIG. 1, in the graphics pipeline, first, vertex data constituting a polygon in the graphics application 60 is input to the spatial index adding unit 20. The spatial index adding unit 20 has a function of performing vertex processing on each input vertex data, generating a spatial index, and creating a vertex buffer 21 with a spatial index.

  FIG. 3 shows a flow of processing performed by the spatial index adding unit 20. As shown in FIG. 3, the spatial index adding unit 20 first reads vertex data and polygon configuration data from a graphics application and performs vertex processing. In other words, for a graphics application, for example, an object such as a character or a structure in a game space, a polygon is formed by connecting vertex data including vertex data including vertex coordinate information when the object is modeled with polygons. The configuration information including the information indicating whether or not is included. For this reason, the spatial index adding unit 20 reads these vertex data and polygon configuration information from the graphics application. Then, the spatial index adding unit 20 executes vertex processing based on the read vertex data and polygon configuration information. This vertex processing includes, for example, coordinate conversion processing and illuminance calculation processing.

  Specifically, the vertex processing will be described. The spatial index adding unit 20 first acquires vertex data including local coordinates of the modeled object from the graphics application, and uses the acquired local coordinates of the vertex data as world coordinates. Convert. This first coordinate transformation is performed to place an object at a predetermined position in one world coordinate system, and coordinate transformation from the local coordinate system to the world coordinate system is executed. In the first coordinate conversion, parameters such as the rotation angle for designating the direction of the object, the translation vector for designating the position, and the scale ratio for designating the size are read from the memory. Next, the space index adding unit 20 converts the coordinates of the object in the world coordinate system into world coordinates in the camera coordinate system viewed from the camera with the camera position as the origin. By performing this second coordinate transformation, the viewpoint is determined as the camera position, and the screen on which the object is projected is determined by designating the camera direction and zoom level. In addition, the spatial index adding unit 20 calculates the illuminance at each vertex when light irradiated from a specific light source position is applied to the object. In addition, the spatial index adding unit 20 similarly calculates other characteristics such as the color of each vertex when light irradiated from a specific light source position is applied to the object. Finally, the spatial index adding unit 20 reads display information including the display resolution and the like, and performs coordinate conversion for projecting three-dimensional world coordinates onto the screen. By this third coordinate conversion, the coordinates of the vertices on the three-dimensional virtual space are converted into two-dimensional screen coordinates. As described above, the spatial index adding unit 20 performs vertex processing, and includes vertex data of an object defined in the local coordinate system, including information indicating two-dimensional coordinates on the screen plane and vertex characteristics such as illuminance. Convert to vertex data. As a result, the spatial index adding unit 20 can obtain the vertex data subjected to the vertex processing.

  After the vertex processing, the spatial index adding unit 20 generates a spatial index in the coordinate system after conversion by the vertex processing based on the spatial information given by the user. As will be described later, in the present invention, the pre-processing unit 30 performs rasterization processing for each space, but the spatial index created here is a reference value that defines the space in which the pre-processing unit 30 performs rasterization processing. It becomes. That is, the space index is information that defines the position, size, and range of a space in which graphics processing is performed. Then, after the spatial index generation process, the spatial index adding unit 20 creates a vertex buffer 21 with a spatial index that stores the polygon configuration information, the vertex data that has been subjected to vertex processing, and the generated spatial index in association with each other. Note that the vertex buffer 21 with a spatial index may be provided in a local memory.

Here, FIG. 4 shows an example of the data structure in the vertex buffer 21 with the spatial index. As shown in FIG. 4, the data in the vertex buffer 21 with space index includes information indicating the vertex data constituting the polygon (polygon configuration data) and coordinate information of each vertex data, and the space to which the polygon belongs. Information (spatial index) to be indicated is registered. For example, referring to the example shown in FIG. 4, if the data in the spatial indexed vertex buffer 21 is viewed, the triangle 0 is composed of vertices of v ₀ , v ₁ , and v _2. Recognize. The coordinates of the vertex v ₀ are (x ₀ , y ₀ , z ₀ ), the coordinates of the vertex v ₁ are (x ₁ , y ₁ , z ₁ ), and the coordinates of the vertex v ₂ are (x ₂ , y ₂ , z ₂ ). Further, the coordinate values of the vertices v _{0 to} v ₂ are also registered in the vertex buffer 21 with a spatial index. Further, it can be seen that the triangle 0 belongs to the space defined as Space ₀ in the coordinate system after the conversion by the vertex processing. As described above, the vertex buffer 21 with space index indicates whether each polygon belongs to a space defined by the space index, and belongs to which space when the polygon belongs to the space. It shows what it is. Further, referring to the vertex buffer 21 with the space index, it can be seen that the space defined by the space index includes Space ₀ and Space ₁ . Thus, in this invention, it divides | segments into several space. Furthermore, the space Space ₀ and the space Space ₁ partially overlap each other. As described above, in the present invention, the plurality of spaces do not need to be uniformly divided from each other, and may be partially overlapped. Furthermore, in the example shown in FIG. 4, the triangle 1 belongs to both the space defined as Space ₀ and the space defined as Space ₁ in the coordinate system after conversion by the vertex processing. I understand that. Thus, in the present invention, it can be considered that one polygon (triangle) exists across a plurality of spaces.

  As described above, the present invention divides the coordinate system converted by the vertex processing into a plurality of spaces, and then performs subsequent graphics drawing processing. A spatial index defining each space can be arbitrarily determined by the user. For example, the user may specify the coordinate value of each vertex constituting each space or the space configuration information indicating a plurality of vertices constituting each space as a space index for partitioning the space. That is, the space index includes the coordinate value of each vertex constituting each space and the configuration information of each space. The coordinate values and configuration information of vertices constituting these spaces are stored in the vertex buffer 21 with space index. The shape of the space may be a circle, a triangle, or a quadrangle, or may be a pentagon or more polygon. Furthermore, as shown in FIG. 4, the plurality of spaces may partially overlap each other.

Further, when creating a spatial index, it may be determined whether or not the entire polygon or vertices constituting the polygon belong to the space defined by the spatial index. For example, if the space is rectangular,
−w ≦ x ≦ w,
−w ≦ y ≦ w,
−w ≦ z ≦ w,
A determination formula such as
On the other hand, when the space has an arbitrary shape, in order to define the clipping index, for example, the case where the space is defined by four sides will be described as an example.
a ₀ x + b ₀ y + c ₀ = 0
a ₁ x + b ₁ y + c ₁ = 0
a ₂ x + b ₂ y + c ₂ = 0
a ₃ x + b ₃ y + c ₃ = 0
By defining the space by three or more straight line equations such as, and substituting the x and y coordinates of the object to be judged (polygon vertices) into these expressions, the inside / outside judgment of the space is made by the sign of the calculation result Can do. Note that the z direction can be determined by a conventional determination formula such as -w <z <w. Further, the process of generating the spatial index may be performed by the spatial index adding unit 20 according to the vertex shader program in the graphics application.

  After the processing by the spatial index adding unit 20 is completed, the pre-processing unit 30 reads the vertex buffer with spatial index 21 and re-creates the polygon in the coordinate system after conversion based on the polygon configuration information and the vertex processed vertex data. To construct. Then, the pre-processing unit 30 performs rasterization processing on the reconstructed polygon for each space defined by the space index, and converts it into pixels. That is, the pre-processing unit 30 performs a rasterization process to perform view conversion in which a polygon in the three-dimensional space is converted into a fragment on the drawing plane by projection conversion, and the fragment on the drawing plane is further converted to a drawing plane. While scanning along the horizontal direction, each raster line is converted into a quantized pixel. In performing this rasterization process, the pre-processing unit 30 refers to the vertex buffer 21 with a spatial index and executes the rasterization process for each space defined by the spatial index. For this reason, the pre-processing unit 30 can perform the rasterization process only on the space that needs to be updated by drawing operation by the user. For example, the pre-processing unit 30 compares the currently processed frame with a processed frame (for example, the previous frame that has already been processed) and determines that there has been a change with respect to the processed frame. The rasterization process may be executed only for the space that has been processed. On the other hand, the pre-processing unit 30 compares the currently processed frame with the processed frame, and configures the processed frame for a space that is determined to have no change with respect to the processed frame. The rasterization process can be omitted by using the pixel information. As a result, it is not necessary to update the drawing of the entire frame, and the processing for the space that needs to be updated can be performed at high speed.

  As described above, the image processing apparatus according to the present invention generates a spatial index, and performs spatial division drawing based on the spatial index, so that it is necessary to update drawing due to a user's screen operation or the like. , The polygon drawing update can be executed only in the space where the drawing update is necessary. Accordingly, the present invention can speed up the drawing process and minimize drawing delay. The space division in the present invention does not uniformly divide a frame into a plurality of areas (tiles), but divides the space in the camera coordinate system into a plurality of areas based on a spatial index arbitrarily given. Therefore, the size of the space can be determined flexibly according to the nature of the graphics. In addition, since drawing processing can be performed in the local buffer, it is possible to reduce access to the external memory. As a result, power consumption and overhead during memory access are reduced, and graphics processing is reduced. High speed can be achieved.

  After the rasterization process, the pre-processing unit 30 loads the pre-processing shader 62 output from the compiler processing unit 10 and executes processing based on the pre-processing shader 62. That is, the pre-processing shader 62 describes functions necessary for generating color values and depth values. For this reason, the pre-processing unit 30 calculates at least a color value represented by RGB three primary colors and a depth value (Z value) indicating the depth for each pixel obtained by the rasterizing process according to the pre-processing shader 62. . At this time, the pre-processing shader 62 may obtain an alpha value indicating the transparency of the color. However, the pre-processing by the pre-processing unit 30 does not perform texture processing for pasting a texture image on the surface of the polygon, shading processing for shading the polygon surface, or the like. When these texture processing and shading processing are necessary, these processing is performed by a later stage processing unit 40 described later.

  The pre-processing unit 30 performs blending, alpha test, and depth test on the fragment converted from the polygon, obtains the color value and depth value of the pixel, and uses the rendering result as pixel data to obtain the color / depth. Write to the value buffer 32. The color / depth value buffer 32 is preferably provided in a local memory.

  After the processing by the pre-processing unit 30, the post-processing unit 40 reads the color / depth value buffer 32, refers to the depth value of the rasterized pixel, and outputs the color value for display on the screen. A pixel to be written to the buffer 51 is determined. That is, the post-processing unit 40 may perform a depth test based on the depth value of the rasterized pixel and determine that the color value of the pixel that has passed the depth test is written in the output color buffer 51. Note that the process for determining the pixels to be written in the output color buffer 51 may be performed by the pre-processing unit 30.

  Thereafter, the post-processing unit 40 loads the post-processing shader 63 output from the compiler processing unit 10 and, based on the loaded post-processing shader 63, for the pixels determined to write color values in the output color buffer 51. Perform the drawing process. That is, the post-processing shader 63 describes functions other than those necessary for generating color / depth values in the shader program. For example, the post-processing shader 63 includes at least texture processing and shading processing. Therefore, the post-processing unit 40 performs a shading process based on the pixel information calculated by the pre-processing unit 30 according to the post-processing shader 63, and calculates the color value of the pixel after the shading process. Furthermore, when performing texture mapping, the post-processing unit 40 synthesizes the color values of the texture image and calculates the final pixel color value.

  As described above, the image processing apparatus of the present invention uses the shader program 61 for the former stage processing shader 62 mainly having the functions necessary for generating the color and depth values, and the latter stage processing mainly necessary for the texture shading process. After being divided into shaders 63 and processing pixels by the pre-processing unit 30 based on the pre-processing shader 62, only the pixels that require texture shading processing are input to the post-processing unit 40 to input the post-processing shader. The process based on 63 is executed. Accordingly, it is not necessary for the post-processing unit 40 to perform unnecessary pixel processing, and thus the processing time can be reduced.

  Further, as shown in FIG. 1, the pre-processing unit 30 includes a pre-processor core 31 and the post-processing unit 40 includes a post-processor core 41. The upstream processor core 31 is hardware that executes rasterization processing in the upstream processor 30, and the downstream processor core 41 is hardware that executes texture processing and shading processing in the downstream processor 40. These pre-stage processor core 31 and post-stage processor core 41 are preferably different from each other in terms of hardware. In the present invention, the shader code (pre-stage processing shader 62) processed by the pre-stage processor core 31 of the pre-stage processing unit 30 is extracted by the compiler processing unit 10 limited to functions necessary for generating color values and depth values. Is. Therefore, the pre-processor core 31 only needs to process the shader code from which unnecessary processing is omitted, so that the size of the processor core 31 can be minimized, resulting in cost reduction. In addition, it is preferable that only the pixel data that has passed the depth test based on the pixel depth value stored in the color / depth value buffer 32 is input to the subsequent processor core 41 included in the subsequent processing unit 40. For example, the post-processing unit 40 may include other hardware for executing the depth test in addition to the post-processor core 41. Further, the process of executing the depth test and determining the pixel for writing the color value in the output color buffer 51 may be performed by hardware constituting the pre-processing unit 30, for example, the pre-processor core 31. The form in which the pre-processing unit 30 and the post-processing unit 40 are configured by different hardware is a preferred form of the present invention. In other embodiments, the pre-processing unit 30 and the post-processing unit 40 are substantially different. The same hardware may be used.

  After the processing by the post-processing unit 40, the final pixel color value obtained by the pixel processing by the pre-processing unit 30 and the post-processing unit 40 is input to the pixel output unit 50. The pixel output unit 50 receives the final color value of the pixel from the post-processing unit 40 and outputs it to the output color buffer 51. The output color buffer 51 is a buffer that stores pixel color values in association with screen coordinates, and the stored pixel color values are referred to when an image is displayed on a display device.

  Here, FIG. 5 shows a configuration of a modification of the image processing apparatus of the present invention. In the modification shown in FIG. 5, a space clip processing unit 70 is provided between the pre-processing unit 30 and the post-processing unit 40. The space clip processing unit 70 performs clipping processing after the rasterization processing by the pre-processing unit 30. The clipping process is a process of cutting the pixel data of the object for the protruding part when the projection image of the object onto the scoon protrudes from a predefined space. Information regarding the space for display on the screen is included in the graphics application 60. For this reason, the space clip processing unit 70 can perform the above-described clipping processing based on information included in the graphics application 60. The pixel information after the clipping processing is input to the post-processing unit 40, and the above-described texture processing and shading processing are performed. Since the portion protruding from the space does not need to be drawn on the display, the clipping processing described above eliminates unnecessary pixel processing by the post-processing unit 40.

[2. Computer]
FIG. 6 is a block diagram showing a computer according to an embodiment of the present invention. This embodiment relates to a computer based on computer graphics (such as a graphic computer). As shown in FIG. 6, this computer includes a central processing unit (CPU) 102, a geometry calculation unit such as a geometry calculation circuit 103, a rendering unit such as a renderer 104, a texture generation unit such as a texture generation circuit 105, and an illumination processing circuit 107. An illumination processing unit such as a display circuit 108, a display information generation unit such as a display circuit 108, a frame buffer 109, and a monitor 110. These elements are connected by a bus or the like and can transmit data to each other. In addition, the storage unit may include a main memory (not shown), various tables, a work memory 111 serving as a work area, a texture memory 112 that stores textures, and the like. The hardware constituting each unit is connected via a bus, for example. The storage unit may be configured by a RAM such as a VRAM, a CR-ROM, a DVD, a hard disk, or the like. The image processing apparatus according to the present invention may be provided and implemented in such a computer graphics computer.

  A central processing unit (CPU) 102 is a device for controlling a program or the like for generating an image. The work memory 111 may store data used by the CPU 102, a display list, and the like. Then, the CPU 102 may read a program stored in the main memory and perform predetermined processing. However, the predetermined processing may be performed only by hardware processing. For example, the CPU 102 reads polygon data as world coordinate three-dimensional object data from the work memory 111 and outputs the polygon data to the geometry calculation circuit 103. Specifically, those having a main processor, a coprocessor, a data processor, four arithmetic circuits, a general-purpose arithmetic circuit, and the like as appropriate. These are connected by a bus or the like so that signals can be exchanged. In addition, a data expansion processor for expanding compressed information may be provided.

  More specifically, the geometry calculation circuit 103 is a circuit for performing coordinate transformation or the like on the input polygon data to data in the camera coordinate system having the viewpoint as the origin. The geometry calculation circuit 103 outputs the processed polygon data to the renderer 104. Specific geometry operation circuits include a geometry processor, a coprocessor, a data processor, four arithmetic operation circuits, or a general-purpose operation circuit that are connected to the main processor through a bus or the like.

  The renderer 104 is a circuit or device for converting polygon unit data into pixel unit data. The renderer 104 outputs the pixel unit data to the texture generation circuit 105. Specific examples of the renderer 104 include a data processor connected to the main processor through a bus, a four arithmetic operation circuit, or a general-purpose operation circuit.

  The texture generation circuit 105 is a circuit for generating a texture color in units of pixels based on the texture data stored in the texture memory 112. The texture generation circuit 105 outputs pixel unit data having texture color information to the illumination processing circuit 107. Specific examples of the texture generation circuit 105 include a data processor connected to the main processor through a bus, a four arithmetic operation circuit, a general-purpose operation circuit, and the like.

  The illumination processing circuit 107 is a circuit for performing shading on a polygon having texture color information using a normal vector, a barycentric coordinate, and the like in pixel units. The illumination processing circuit 107 outputs the shaded image data to the display circuit 108. Specific examples of the illumination processing circuit 107 include a data processor connected to the main processor through a bus, four arithmetic operation circuits, or a general-purpose operation circuit. Then, information relating to light may be appropriately read out from a table stored in the memory and the like to perform shading.

  The display circuit 108 is a circuit for writing the image data input from the illumination processing circuit 107 to the frame buffer 109, reading the image data written in the frame buffer 109, and controlling it to obtain display image information. The display circuit 108 outputs display image information to the monitor 110. Specific examples of the display circuit include a drawing processor, a data processor, a four arithmetic operation circuit, and a general-purpose arithmetic circuit connected to the main processor through a bus.

  The monitor 110 is a device for displaying a computer graphics image in accordance with input display image information.

[3. game machine]
FIG. 7 is a block diagram of a game machine according to another embodiment of the present invention. The embodiment represented by this block diagram can be suitably used particularly as a portable, home or business game machine. Therefore, in the following, it will be described as a game machine. Note that the game machine shown in the figure may include at least the processing unit 200 (or may include the processing unit 200 and the storage unit 270, or the processing unit 200, the storage unit 270, and the information storage medium 280). These blocks (for example, the operation unit 260, the display unit 290, the sound output unit 292, the portable information storage device 294, and the communication unit 296) can be arbitrary constituent elements. The image processing apparatus according to the present invention may be provided and mounted on such a game machine.

  The processing unit 200 performs various processing such as control of the entire system, instruction instruction to each block in the system, game processing, image processing, and sound processing. The function of the processing unit 200 can be realized by hardware such as various processors (CPU, DSP, etc.) or ASIC (gate array, etc.), or a given program (game program).

  The operation unit 260 is for a player to input operation data. The function of the operation unit 260 can be realized by, for example, a controller including a lever, a button, an outer frame, and hardware. In particular, in the case of a portable game machine, the operation unit 260 may be formed integrally with the game machine body. Processing information from the controller is transmitted to the main processor or the like via a serial interface (I / F) or a bus.

  The storage unit 270 is a work area such as the processing unit 200 or the communication unit 296. In addition, programs and various tables may be stored. The storage unit 270 may include, for example, a main memory 272, a frame buffer 274, and a texture storage unit 276, and may store various tables. The function of the storage unit 270 can be realized by hardware such as ROM and RAM. Examples of the RAM include VRAM, DRAM, and SRAM, and may be appropriately selected depending on the application. A VRAM or the like constituting the frame buffer 274 is used as a work area for various processors.

  The information storage medium (storage medium usable by a computer) 280 stores information such as programs and data. The information storage medium 280 can be sold as so-called game software. The functions of the information storage medium 280 can be realized by hardware such as an optical disk (CD, DVD), a magneto-optical disk (MO), a magnetic disk, a hard disk, a magnetic tape, or a memory (ROM). The processing unit 200 performs various processes based on information stored in the information storage medium 280. The information storage medium 280 stores information (program or program and data) for executing the means of the present invention (this embodiment) (particularly, blocks included in the processing unit 200).

  The display unit 290 outputs an image generated according to the present embodiment, and functions thereof are CRT (CRT), LCD (Liquid Crystal), OEL (Organic Electroluminescent Device), PDP (Plasma Display Panel), or HMD. It can be realized by hardware such as (head mounted display).

  The sound output unit 292 outputs sound. The function of the sound output unit 292 can be realized by hardware such as a speaker. The sound output is processed by a sound processor connected to the main processor or the like via a bus, for example, and output from a sound output unit such as a speaker.

  The portable information storage device 294 stores player personal data, save data, and the like. Examples of the portable information storage device 294 include a memory card and a portable game device. The functions of the portable information storage device 294 can be achieved by known storage means such as a memory card, flash memory, hard disk, USB memory and the like.

  The communication unit 296 is an arbitrary unit that performs various controls for communicating with the outside (for example, a host device or other image generation system). The function of the communication unit 296 can be realized by various processors, hardware such as a communication ASIC, a program, or the like.

  The program or data for executing the game machine may be distributed from the information storage medium included in the host device (server) to the information storage medium 280 via the network and the communication unit 296.

  An example of the processing unit 200 includes a game processing unit 220, an image processing unit 230, and a sound processing unit 250. Specific examples include a main processor, a coprocessor, a geometry processor, a drawing processor, a data processor, four arithmetic operation circuits, or a general-purpose operation circuit. These are appropriately connected by a bus or the like so that signals can be exchanged. In addition, a data expansion processor for expanding compressed information may be provided.

  Here, the game processing unit 220 accepts coins (price), sets various modes, progresses the game, sets the selection screen, and positions and rotation angles of objects (rotation angles around the X, Y, or Z axes). , Processing to move the object (motion processing), processing to determine the position of the viewpoint (virtual camera position) and line-of-sight angle (virtual camera rotation angle), processing to place objects such as map objects in the object space, Operation data from the operation unit 260 includes various game processes such as a hit check process, a process for calculating game results (results, results), a process for a plurality of players to play in a common game space, or a game over process. Or based on personal data, stored data, game programs, etc. from portable information storage device 294 Do.

  The image processing unit 230 performs various types of image processing in accordance with instructions from the game processing unit 220 and the like. The sound processing unit 250 performs various types of sound processing in accordance with instructions from the game processing unit 220 and the like.

  All of the functions of the game processing unit 220, the image processing unit 230, and the sound processing unit 250 may be realized by hardware, or all of them may be realized by a program. Alternatively, it may be realized by both hardware and a program. The image processing unit 230 includes one including a geometry calculation unit 232 (three-dimensional coordinate calculation unit) and a drawing unit 240 (rendering unit).

  The geometry calculation unit 232 performs various geometry calculations (three-dimensional coordinate calculation) such as coordinate conversion, clipping processing, perspective conversion, or light source calculation. Then, the object data (object vertex coordinates, vertex texture coordinates, luminance data, etc.) after the geometry processing (after perspective transformation) is stored and saved in the main memory 272 of the storage unit 270, for example.

  The drawing unit 240 draws an object in the frame buffer 274 based on the object data after the geometry calculation (after perspective transformation) and the texture stored in the texture storage unit 276.

  Examples of the drawing unit 240 include a texture mapping unit 242 and a shading processing unit 244. Specifically, it can be implemented by a drawing processor. The drawing processor is connected to a texture storage unit, various tables, a frame buffer, a VRAM, and the like via a bus and further connected to a display.

  The texture mapping unit 242 reads the environment texture from the texture storage unit 276 and maps the read environment texture to the object.

  The shading processing unit 244 performs shading processing on the object. For example, the geometry processing unit 232 performs light source calculation, and obtains the luminance (RGB) of each vertex of the object based on the light source information for shading processing, the illumination model, the normal vector of each vertex of the object, and the like. Based on the luminance of each vertex, the shading processing unit 244 obtains the luminance of each dot on the primitive surface (polygon, curved surface) by, for example, phone shading or goo shading.

  An example of the geometry calculation unit 232 includes a normal vector processing unit 234. The normal vector processing unit 234 may perform a process of rotating the normal vector of each vertex of the object (in a broad sense, the normal vector of the surface of the object) using a rotation matrix from the local coordinate system to the world coordinate system. .

  If the configuration of the game machine is used as appropriate, it is possible to obtain a device that can also function as an image display device in a slot machine, a pachinko gaming machine, or the like.

  The present invention relates to an image processing apparatus and an image processing method for processing graphics by dividing a space. Therefore, the present invention can be suitably used in the computer industry.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 10 ... Compiler process part 11 ... Front end process part 12 ... Function extraction part 13 ... Back end process part 20 ... Spatial index addition part 21 ... Vertex buffer 30 with space index ... Pre-stage process part 31 ... Pre-stage processor core 32 ... Color / depth value buffer 40 ... Subsequent processing unit 41 ... Subsequent processor core 50 ... Pixel output unit 51 ... Output color buffer 60 ... Graphics application 61 ... Shader program 62 ... Pre-stage processing shader 63 ... Post-stage processing shader 70 ... Space clip processing section

Claims (7)

An image processing apparatus for computer graphics processing,
A shader for pre-processing that reads the shader program in the graphics application and extracts the functions necessary for generating color and depth values, and a shader for post-processing that includes texture shading that is not included in the pre-processing shader. A compiler processing unit (10) for outputting,
Reads the vertex data that constitutes the polygon in the graphics application, performs vertex processing including coordinate transformation, obtains the processed vertex data, generates a spatial index in the coordinate system after the transformation, A spatial index addition unit (20) for creating a vertex buffer (21) with a spatial index that stores the processed vertex data and the spatial index in association with each other;
The vertex buffer (21) with the spatial index is read to construct a polygon, and for each space defined by the spatial index, the polygon is rasterized and converted into pixels. Based on the pre-processing shader, A pre-processing unit (30) for generating color values and depth values of rasterized pixels;
The pixel determined to be written to the output color buffer (51) with reference to the depth value of the rasterized pixel is subjected to texture shading processing based on the post-processing shader, and texture shading processed A post-processing unit (40) for generating pixel color values;
An image processing apparatus comprising: a pixel output unit (50) that outputs a color value of a pixel obtained by the pre-stage processing unit (30) and the post-stage processing unit (40) to an output color buffer (51). The pre-processing unit (30) includes a pre-processor core (31) that performs the rasterization process.
The post-processing unit (40) includes a post-processor core (41) that performs the texture / shading process,
The image processing apparatus according to claim 1, wherein the former processor core (31) and the latter processor core (41) are different hardware. The image processing apparatus according to claim 1, wherein the spatial index adding unit (20) performs spatial division based on spatial information arbitrarily given by a user and generates the spatial index. The compiler processing unit (10)
A front-end processing unit (11) for performing front-end processing on the shader program and converting it into an intermediate representation;
A function extraction unit that extracts functions necessary for generating color / depth values from the intermediate representation by performing turn-match processing between the intermediate representation and a processing pattern relating to generation of color / depth values prepared in advance. (12),
Back-end processing is performed on the intermediate representation, and the functions necessary for generating the color / depth values extracted by the function extraction unit (12) are output as a pre-processing shader, and the shader program And a back-end processing unit (13) for outputting functions other than those necessary for generating color / depth values as a post-processing shader.
The image processing apparatus according to claim 1. An image processing method executed by an image processing apparatus for computer graphics processing,
The image processing device is
A shader for pre-processing that reads the shader program in the graphics application and extracts the functions necessary for generating color and depth values, and a shader for post-processing that includes texture shading that is not included in the pre-processing shader. The compiler processing step to output,
Reads the vertex data that constitutes the polygon in the graphics application, performs vertex processing including coordinate transformation, obtains the processed vertex data, generates a spatial index in the coordinate system after the transformation, A spatial index adding step for creating a vertex buffer (21) with a spatial index that stores the processed vertex data and the spatial index in association with each other;
The vertex buffer (21) with the spatial index is read to construct a polygon, and for each space defined by the spatial index, the polygon is rasterized and converted into pixels. Based on the pre-processing shader, A pre-processing step for generating color values and depth values of rasterized pixels;
A texture shading process is performed on the pixel determined to be written to the output color buffer (51) by referring to the depth value of the rasterized pixel, and the texture shading process is performed based on the post-processing shader. A post-processing step to generate a color value for each pixel;
A pixel output step of outputting a color value of a pixel obtained by the pre-stage processing step and the post-stage processing step to an output color buffer (51).   A computer comprising the image processing apparatus according to claim 1.   A game machine comprising the image processing apparatus according to claim 1.

Images