JP6424329B2 - Image data processing method in image processing processor and program thereof - Google Patents

Description

  The present invention relates to an image data processing method and program thereof in an image processing processor, and more particularly, to an image data processing method and program thereof in an image processor equipped with a programmable shader.

  In three-dimensional computer graphics, it is common to generate an object to be drawn using a polygon model represented by a large number of polygons. In the drawing process of the polygon model, a shading process for shading the surface of the polygon and a texture mapping process for attaching a texture image to the surface of the polygon model are performed.

  Although these processes were initially realized by fixed pipelines using dedicated hardware circuits for high-speed processing, in order to further improve their expressiveness as technology development progresses, Programmability of transformation from three-dimensional coordinates to two-dimensional coordinates and shading processing has progressed, and has become a character of "processor" as a whole. Accordingly, the CPU is called an image processing processor (for example, a GPU (Graphic Processing Unit)).

  In such a GPU, the functional units that perform the processing from the above seat transformation to the texture mapping are generally also commonly referred to as "shaders" as a whole, but the programmization of each processing that constitutes them is in progress, and fixed processing Instead, the user can select and execute various shading processes freely and variably. Specifically, after programmable from polygon vertex units to programmable pixel units after rasterization, recently, programmable functional parts in the GPU can be used not only for image processing but also for general-purpose calculations What has appeared to have a configuration called "compute shader".

Japanese Patent Application Publication No. 2002-520748 JP, 2013-251717, A

X. Yu, B. S. Morse, and T. W. Sederberg, "Image Reconstruction Using Data-Dependent Triangulation", IEEE Computer Graphics and Applications, Vol. 21, No. 3, pp. 62-68 May / Jun. (2001)

  By the way, in the rendering process in the above-described GPU, in the process of attaching a texture image to a polygon, a process of enlarging, reducing or deforming the texture image is performed, and these enlargements Bilinear filtering and bicubic filtering are used for each processing of reduction and deformation (see, for example, Patent Document 1).

  However, in each process of bilinear filtering and bicubic filtering, there is a problem that so-called shaggy occurs at the edge in the diagonal direction of the object and the entire image is blurred.

  On the other hand, as a high-quality enlargement method, a triangle division method has been proposed in which the image area is divided into triangles and the pixels are interpolated according to the cost function, the edge direction, etc. (See, for example, Non-Patent Document 1). This is based on the cost function in which division and so-called "edge swap" are repeated based on the cost function across each lattice-like area having each pixel as a vertex to divide the area and calculate the value of the pixel to be interpolated. This is a method of determining sampling (reference) pixel points.

  Patent Document 2 is an example of applying this "triangle division method" to image enlargement processing. In this document, a triangle division method is adopted as interpolation processing at the time of drawing in two-dimensional computer graphics processing, and the image data expanded by the decoder corresponds to the division pattern of each area determined by the triangle division evaluation circuit. And store in the image memory, and when there is a pixel whose value is to be determined by interpolation, information of the division pattern for the area of the grid including the pixel to be interpolated and a predetermined pixel value related to the grid An image display processing apparatus and an image display processing method are disclosed which are read from the above to obtain values of pixels to be interpolated.

  However, in the technique disclosed in Patent Document 2, two-dimensional image data obtained in units of lines or blocks by the decoder is sequentially input, and data obtained by processing the input data by applying the triangle division method is used. Since it is stored in a memory such as a work buffer and the drawing circuit accesses the data stored in the work buffer to obtain interpolation data, it is possible to use a specific address for texture mapping like a drawing circuit of three-dimensional computer graphics. The configuration of the premise is different from the process of specifying a texture by a UV value.

  That is, as in the case of texture mapping in three-dimensional computer graphics, when interpolation data at arbitrary coordinates is randomly required, image data can not be sequentially processed as in the technique disclosed in Patent Document 2, and A work buffer for storing division pattern information is necessary. For example, when providing an interpolation method as a pixel shader to a 3D renderer material such as Unity, which is an engine for game development, due to limitations of the renderer etc. When the work buffer can not be used, the technology disclosed in Patent Document 2 can not be used.

  Also, in a mode in which interpolation data of arbitrary block coordinates is randomly requested in block units as in the case where a compute shader is mounted and used, the premise is different from Patent Document 2, and new Need to find out how to apply.

  The present invention has been made under the circumstances as described above, and it is an object of the present invention to provide an image processing processor having a programmable shader function capable of obtaining a smoother enlarged image by employing a triangulation method. An image data processing method and its program are provided.

  In order to achieve the above object, according to an image data processing method in an image processor of the present invention, there is provided a vertex shader for obtaining a position on a display unit corresponding to the vertex data based on vertex data of input polygons; From the position of the vertex on the display unit determined by the shader, the pixel on the display unit required to display an image having the specified texture mapped to the polygon on the display unit is determined In order to display on the display unit a rasterizer and an image in which a specified texture is mapped to the polygon, the rasterizer gives information on the position of each pixel on the display unit which requires a pixel value. Pixel shape at each position is determined by interpolating the values of texture pixels at predetermined positions around the position. An image data processing method in an image processing processor, wherein the pixel shader configures a triangle division pattern in a triangle division method according to the pixel grid of the texture including the position, the pixel grid The present invention is characterized in that the value of the predetermined surrounding texture pixel is determined based on the determined triangle division pattern, and the pixel value of the position is determined based on the determined pixel and the surrounding pixels.

  Here, when the pixel shader determines the triangle division pattern, it is preferable that a low pass filter is applied to each value of pixels constituting the pixel grid and pixels therearound to obtain a gradient value.

  Also, in detail, when the pixel shader determines the triangle division pattern, the pixel shader relates to the triangle division pattern related to the pixel grid of the texture and the grids above and below and to the left and right of the pixel grid. The triangle division pattern according to the pixel grid of the texture, which temporarily determines the triangle division pattern and includes the position determined temporarily, and the triangle division pattern according to the top, bottom, left, and right grids of the pixel grid include In the case of misalignment, the final determination of the triangle division pattern of the texture pixel grid including the position is performed by matching.

  Further, in order to achieve the above object, an image data processing program in an image processor of the present invention executes vertex shader processing for obtaining a position on a display unit corresponding to the vertex data based on vertex data of an input polygon. A position on the display unit, which is necessary for displaying on the display unit an image in which a designated texture is mapped to the polygon from the position of the vertex on the display unit determined by the vertex shader processing Rasterizer processing for obtaining the pixels of each of the pixels, and individual pixels on the display unit requiring pixel values from the rasterizer processing to display on the display unit an image in which a designated texture is mapped to the polygon Each time the information on the position is given, the pixel value at the position is interpolated by the value of the texture pixel at a predetermined periphery of the position. An image data processing program for causing an image processing processor to perform rendering processing, the pixel shader processing includes triangle division according to the pixel grid of the texture in which the position is included The triangle division pattern in the method is determined from the pixels constituting the pixel grid and pixels in the periphery thereof, and the values of the predetermined surrounding texture pixels are interpolated based on the determined triangle division pattern, and the pixels at the corresponding position The point is to determine the value.

  Further, in order to achieve the above object, an image data processing method in an image processing processor of the present invention is an image data processing method in an image processing processor that includes a compute shader having a plurality of compute units and performs rendering processing, Each compute unit takes charge of each divided texture obtained by dividing the texture and processes it in parallel, and has a plurality of threads operated in parallel by a program, and using the plurality of threads, From the values of each pixel constituting the divided texture, for a predetermined number of grids of each grid formed by each pixel, a triangle division pattern to be used in the triangle division method is determined at once, and the texture is mapped to polygons Stores the enlarged image in order to display the displayed image on the display unit. The predetermined number of grid positions corresponding to the pixel positions on the grid are determined, and the values of the pixels on the display unit related to the positions included in each grid are used for each thread corresponding to each grid. The gist of the present invention is to interpolate and obtain the values of peripheral pixels in the divided texture based on the triangle division pattern determined for each grid.

  Here, when the plurality of threads collectively determine the triangle division pattern, it is preferable that the processing is performed while storing values obtained in the process of processing in a memory common to the plurality of threads. is there.

  Further, when each of the compute units collectively determines the triangle division pattern using the plurality of threads, a low pass filter is applied to each value of pixels constituting each grid and pixels in the periphery thereof. While the gradient value is determined, the number of obtained triangle division patterns is reduced with respect to the number of pixels of the divided textures to be used by the filtering process, so that each compute unit has a boundary portion of each divided texture. It is effective that is assigned redundantly.

  Further, in detail, when each of the compute units collectively determines the triangle division pattern using the plurality of threads, the triangle division pattern according to the grid corresponding to each thread, and the upper and lower sides of the grid The triangle division patterns for the left and right grids are temporarily determined, and the temporarily determined triangle division patterns for the grid corresponding to the thread and the triangle division patterns for the upper, lower, left, and right grids of the grid are not correct. In the case of matching, by matching, the triangulation pattern of the lattice corresponding to the thread is finally determined.

  Further, to achieve the above object, an image data processing program in an image processing processor according to the present invention comprises an image processing processor comprising a plurality of parallel operable compute units each having a plurality of parallel operable threads, An image data processing program that performs rendering processing based on information of a three-dimensional model configured by polygons and information of a texture to be attached, wherein each of divided textures obtained by dividing the texture into a plurality of pieces is allocated to each of the compute units Processing and, based on the values of the pixels constituting the divided texture, the triangle division patterns used in the triangle division method collectively for a predetermined number of grids of each grid formed of the pixels in the plurality of threads And the texture is mapped to the polygon. Processing for obtaining the positions on the predetermined number of grids corresponding to the positions of the pixels on the buffer storing the enlarged image, and displaying each of the threads corresponding to the grids, in order to display the scanned image on the display unit; Processing for interpolating values of surrounding pixels in the divided texture on the basis of the triangle division pattern determined for each lattice based on the triangle division pattern determined for each lattice; Make it a gist.

  According to the image data processing method and program thereof in the image processing processor of the present invention, in the image processing processor having a programmable shader function, it is possible to obtain a smoother enlarged image by adopting a triangle division method.

  In particular, in an image processing processor provided with a compute shader, a plurality of triangle division patterns in the triangle division method can be obtained at once, thereby obtaining values of each pixel necessary for display on the display unit at once. it can.

  In addition, when determining the triangle division pattern for each grid, since the gradient value is obtained while applying low-pass filtering for each value of the pixels constituting the grid and the pixels around it, noise and the like can be removed. .

It is a block diagram showing composition of an image processing device with which an embodiment of an image data processing method of the present invention and its program is embodied. FIG. 2A is a flowchart showing the procedure of processing in the rasterizer 22, and FIG. 2B is a diagram for explaining the rasterization processing. It is a flowchart which shows the procedure of the process in the pixel shader 23 performed whenever there is a request of calculation of a pixel value from the rasterizer 22. FIG. It is a flowchart which shows the detail of the procedure of the determination process of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the interpolation process based on the determined triangle division | segmentation pattern. It is a block diagram showing composition of an image processing device with which another embodiment in an image data processing method of the present invention and its program is embodied. FIG. 6 is a block diagram showing a detailed configuration of each compute unit 251. It is a figure which shows the example of how to divide | segment and process a texture. It is a flow chart which shows the processing procedure of each compute unit 251. It is a flowchart which shows the detailed procedure of the process which determines 9x9 division patterns collectively from 16x16 division | segmentation texture pixel value TPV in step S32. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern. It is a figure for demonstrating the decision processing of a triangle division pattern.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<Embodiment with GPU Having Programmable Pixel Shader Function>
FIG. 1 is a block diagram showing a configuration of an image processing apparatus in which an embodiment of an image data processing method and a program thereof according to the present invention is embodied.

  The image processing apparatus shown in the figure should be attached with the display unit 5 on which a three-dimensional graphics image is displayed, the VRAM 3 in which data of a plurality of textures are stored, and vertex data of a 3D model composed of polygons Based on the CPU 1 supplying information specifying each texture, the vertex data supplied from the CPU 1 and the information specifying the texture, each pixel corresponding to the range displayed on the display unit 5 is obtained from the vertex data, A GPU 2A that can read the data of the corresponding texture stored in the VRAM 3 and calculate and output the value of each corresponding pixel based on it, and each pixel output from the GPU 2A located in front of the display unit 5 And a frame buffer 4 for storing data.

  The GPU 2 A includes a vertex shader 21, a rasterizer 22, a pixel shader 23, and a texture fetch unit 24. Hereinafter, so-called rendering processing in the GPU 2A will be described in detail with reference to FIGS. 2 to 10 together with FIG. The vertex shader 21, the rasterizer 22, and the pixel shader 23 are all programmable.

  The vertex shader 21 performs various coordinate transformations on the vertex data supplied from the CPU 1 and obtains the display position (coordinates) on the display unit 5.

  FIG. 2A is a flowchart showing the procedure of processing in the rasterizer 22, and FIG. 2B is a diagram for explaining the rasterization processing. Therefore, the rasterizer 22 first performs rasterization processing as shown in FIG. 2B from the vertex data on the display unit 5 sent from the vertex shader 21 and derives pixels necessary for the value (step S11). ). FIG. 2B shows an example in which the pixel value is obtained when the center of the pixel is located inside the boundary at the boundary derived from the vertex data. Next, the rasterizer 22 gives the position of each pixel to the pixel shader 23 as the value UV (0 to 1, 0 to 1) of the texture UV coordinate system, for each pixel for which a value is required. The pixel shader 23 is requested to calculate the pixel value (step S12). Then, a pixel value is acquired from the pixel shader 23 for each pixel requiring a value (step S13). When the pixel values are obtained for all the pixels requiring values obtained in the process of S11 (affirmative determination in step S14), the process proceeds to step S15, and the acquired pixel values are written to the frame buffer 4 to display 5 Provide for display in Here, for convenience of explanation, the procedure is such that all pixel values are obtained and then collectively supplied to the frame buffer 4, but each time each pixel value is obtained, the frame buffer 4 is individually provided. It is also conceivable to supply, which may be any.

  FIG. 3 is a flow chart showing the procedure of processing in the pixel shader 23 performed each time there is a request for calculation of pixel values from the rasterizer 22, and FIG. 4 is a process of determining a triangle division pattern which is a part of the processing. FIG. 5 to FIG. 10 are diagrams for explaining the details of the procedure, and FIGS. 5 to 10 are diagrams for explaining the determination processing of the triangle division pattern and the interpolation processing based on the determined division pattern. The processing in the pixel shader 23 shown in FIG. 3 is a portion that is characteristically programmed by the user this time. The programming language is, for example, HLSL (High Level Shading Language) or Cg.

  Referring to FIG. 3, first, the pixel shader 23 requests the calculation of the value of the pixel requiring the value from the pixel shader 23 with the input of the UV coordinate value UV from the rasterizer 22. If there is a request (Yes in step S21), one triangle division pattern corresponding to a pixel requiring the value is determined to obtain the value using the triangle division method. It determines (step S22).

  For details of the process of determining the triangle division pattern, referring to FIG. 4, the pixel shader 23 first converts UV coordinate values UV received from the rasterizer 22 into texture pixel coordinate values (step S 221). This can easily be done by multiplying the resolution of the texture image. Next, as shown in FIG. 5A, the pixel shader 23 uses, as a reference point (0, 0) for convenience, the upper left pixel of the four texture pixels surrounding the position from the obtained texture pixel coordinate value. Of the pixels from (-3, -3) to (4, 4), that is, 8.times.8 pixels, by designating the texture address TA to the texture fetch unit 24. It instructs to read (step S222). The values of 8 × 8 pixels of (-3, -3) to (4, 4) are given as the UV coordinate values UV, and the values of the necessary pixels of the values are interpolated by the triangle division method. It is information necessary to determine a triangle division pattern to be determined. Also, the reason why the 8 × 8 pixel value is required in this way is to perform the low-pass filter process described below to enhance the noise resistance.

  Therefore, when the value of 8 × 8 pixels as shown in FIG. 5A is received, the pixel shader 23 next pre-covers the 3 × 3 pixels as shown in FIG. 5B. As shown in FIG. 6C, the filter PF is prepared, and the 8 × 8 pixels are sequentially scanned, and each time the values of the 3 × 3 pixels are averaged, the scanning is completed as shown in FIG. A 6 × 6 pixel value as shown in a) is obtained (step S223).

  Next, the pixel shader 23 prepares a gradient filter GF that covers 2 × 2 pixels as shown in FIG. 6B with respect to the obtained 6 × 6 pixel values, as shown in FIG. As shown in c), the 6 × 6 pixels are sequentially scanned, and each time the gradient is determined based on the value of the 2 × 2 pixel, in other words, by determining the gradient value corresponding to each grid, At the end of the scan, 5 × 5 gradient values corresponding to the 5 × 5 grid as shown in FIG. 7A are obtained (step S224).

  Next, the pixel shader 23 prepares an averaging filter AF that covers the 3 × 3 gradient values as shown in FIG. 7B with respect to the obtained 5 × 5 gradient values, As shown in FIG. 9C, 5 × 5 gradient values are sequentially scanned, and in each case, 3 × 3 gradient values are averaged to obtain 3 as shown in FIG. A gradient value of × 3 is obtained (step S225).

  From the 3 × 3 gradient values obtained as described above, we determine the triangulation pattern for the grid that contains the position we want to obtain, ie, the grid composed of (0,0) to (1,1) As a premise, first, the triangle division pattern will be described with reference to FIG. Here, the case where the inclination of the pixel value is approximated to an accuracy of 360 degrees / 16, that is, an accuracy of 22.5 degrees will be described. When the inclination of the pixel value is approximated to an accuracy of 22.5 degrees, the division patterns determined for each of the lattice regions formed by the four pixel points of interest are fifteen shown in FIG.

  Therefore, from the 3 × 3 gradient values obtained as shown in FIG. 9A, in the central grid, in other words, texture pixels (0, 0), (0, 1), (1, 0), ((0) For the grid constructed in 1, 1), any one of the division patterns shown in FIG. 8 is determined, then, from the nine gradient values, in addition to the center grid, the grids above, below, left and right The division pattern is also obtained for (step S226). At this time, if the connection of the divided pattern for the central grid and the connection of the upper, lower, left, and right divided patterns is matched (affirmative determination in step S227), the divided pattern is finally obtained also for the central grid at that time. It is the division pattern that you want to obtain. For example, when the division pattern is obtained as shown in FIG. 9B, since the alignment is made, the division pattern for the central grid to be obtained is determined as shown in FIG.

  On the other hand, if the connection between the division pattern for the central grid and the division patterns on the upper, lower, left, and right sides is mismatched (negative determination in step S227), for example, as shown in FIG. Are corrected (step S228).

  Returning to FIG. 3, next, using the triangle division pattern obtained in step S22 as described above, the position where the value is desired to be obtained by interpolation based on the triangle division method, in other words, the UV coordinate value UV by the rasterizer 22 To calculate the value of the pixel requested to calculate the value (step S23).

At this time, according to the triangle division method, surrounding texture pixels used for interpolation are determined in accordance with the obtained division pattern. This will be briefly described with reference to FIG.
FIGS. 10 (a) to 10 (h) are diagrams showing examples of division patterns showing how to take reference pixels. For example, as shown in FIG. 6A, in the case of a pattern without division, linear interpolation of pixels p (ul), p (ur), p (dl) and p (dr) at each vertex of the grid is performed. Calculate the required position rp of the value. Also, in the case of the two divisions of FIG. 7B, when the required position rp of the values is as shown in the same drawing, based on the values of the pixels p (ul), p (ur) and p (dl). To calculate the value of the position rp. In addition, in the case of one pattern of three divisions shown in (c) of the same figure and the required position rp of the values is as shown in the same figure, the values of the pixels p (ur), p (dl), p (dll) The value of the position rp is calculated based on In addition, in the case of one pattern of three divisions shown in (d) of the same figure and the required position rp of the values is as shown in the same figure, the values of the pixels p (ur), p (dl), p (urr) The value of the position rp is calculated based on In addition, in the case of one pattern of three divisions shown in (e) of the same figure and the required position rp of the values is as shown in the same figure, the values of the pixels p (ur), p (dl), p (uru) The value of the position rp is calculated based on Further, in the case of one pattern of three divisions shown in (f) of the figure and the required position rp of the values is as shown in the same figure, the values of the pixels p (ur), p (dl), p (dld) The value of the position rp is calculated based on The same applies to each of the divided areas also for each of the four divided patterns shown in (g) and (h) in the same figure. Furthermore, the same applies to the other seven division patterns not illustrated in the figure.

  Therefore, after all, if there are 12 texture pixel values shown in FIG. 10I, values can be determined for the position to be interpolated in the grid. Since these 12 are completely included in the 8 × 8 pixels shown in FIG. 5A which are read to determine the triangle division pattern, it is not necessary to read them from VRAM 3 again. Become.

  Referring back to FIG. 3, finally, the pixel value obtained in step S23 is returned to the rasterizer 22 as the pixel value requested by the rasterizer 22 (step S24).

  The above is the embodiment of the GPU having the programmable pixel shader function.

  When the texture fetch unit 24 incorporates filtering processing such as bilinear or bicubic as a standard specification, the function is canceled. This is in order to prevent competition with the filtering process in the pixel shader 23 as described above, and an unintended blurry image or the like. Therefore, as described above, the read pixel value is given to the pixel shader 23 as it is.

  In addition, each filter process mentioned above can be changed according to the kind and the magnitude | size of the noise assumed.

  As described above, in the embodiment of the GPU having the programmable pixel shader function, the rasterizer 22 notifies the pixel shader 23 of the pixel position on the display unit 5 for which the value is necessary by notifying the pixel shader 23 in pixel units. Because a value is required, and the pixel shader 23 performs processing of calculating the value and returning it to the rasterizer 22 by interpolation based on the triangle division method using texture pixel values, a smoother enlarged image You can get

<Embodiment with GPU having Compute Shader Function>
FIG. 11 is a block diagram showing the configuration of an image processing apparatus in which another embodiment of the image data processing method and the program thereof of the present invention is embodied.

  The image processing apparatus shown in the figure includes a display unit 5 on which a three-dimensional graphics image is displayed, a VRAM 3 in which data of a plurality of textures are stored, and at least vertex data and textures of a 3D model composed of polygons. The CPU 1 supplies information related to the division when processing each texture together with the information specifying the image, the vertex data supplied from the CPU 1, the information for specifying the texture, and the VRAM 3 based on the division information. The corresponding texture stored is determined collectively for each divided unit, and the triangle division pattern is determined collectively, and the value of the display pixel in the display unit 5 is determined according to the enlargement factor based on the vertex data, all division Based on the pattern, the GPU 2 B, which is obtained by interpolation at a time using the triangle division method, and is output to the front stage of the display unit 5 And location, and a frame buffer 4 each pixel data outputted from GPU2A is stored, the.

  Further, as shown in FIG. 11, the GPU 2B has a compute shader 25 and a dispatcher 26, and further, each of the compute shaders 25 is referred to as each of divided textures (hereinafter referred to as "divided texture") , And a plurality of compute units 251a, 251b, 251c,...

  FIG. 12 is a block diagram showing the detailed configuration of each compute unit 251. As shown in FIG. As shown in the figure, each compute unit 251 has a texture fetch unit 2511 that reads in each pixel value TPV of the divided texture in charge from the VRAM 3, and each pixel value of the divided texture read via the texture fetch unit 2511. A plurality of threads 2512a, 2512b, 2512c,... Which respectively process TPVs in parallel, and a shared memory 2513 in which information and process results of processing in each thread 2512 are stored. The number of threads 2512 is, for example, 768. The shared memory 2513 is, for example, a cache memory of 32 k bytes.

Next, the operation and processing procedure of the image processing apparatus shown in FIG. 11 will be described.
How the texture stored in the VRAM 3 is divided and processed is designed in advance. FIG. 13 is a diagram showing an example of how to divide and process a texture. Here, one divided texture is configured by 16 × 16 texture pixels, and the texture is divided as 3 × 2 divided textures ((0, 0) to (1, 2)). Here, the reason why each divided texture is overlapped for six grids is to be able to obtain a divided pattern without omission at the boundary of division in the process of determining a triangular divided pattern to be described later (if there is no such overlapping (For example, a drop will occur according to the filter processing described later).

  Therefore, the CPU 1 sends, to the GPU 2 B, “division information”, that is, the number of pixels constituting the division texture (for example, 16 × 16 as described above), and division information together with vertex data and information for specifying texture. The number information etc. are notified in advance. The dispatcher 26 of the GPU 2B recognizes the required number of compute units 251 based on the information supplied from the CPU 1 and notifies the compute units 251 of divided textures to be handled and processed by the group ID. . Thus, for example, in the example illustrated in FIG. 13, for example, the compute units 251a, 251b, 251c, 251d, 251e, and 251f respectively generate divided textures (0, 0), (0, 1), and (0, 2). , (1, 0), (1, 1), and (1, 2), and group IDs (0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2) are passed.

  FIG. 14 is a flowchart showing the processing procedure of each compute unit 251. The processing is incorporated by program into each compute unit 251, and each compute unit 251 can operate in parallel. The program language is, for example, HLSL or Cg.

  Therefore, the processing procedure will be described along the same figure. First, the texture fetch unit 2511 of each compute unit 251 takes charge of divided textures according to the range of UV coordinate values calculated from the group ID notified from the dispatcher 26. The pixel values TPV (16 × 16 pixels) of are read in (step S31). Next, 9 × 9 triangle division patterns are collectively determined from the 16 × 16 divided texture pixel values TPV read in (step S32).

  FIG. 15 is a flowchart showing a detailed procedure of a process of collectively determining 9 × 9 divided patterns from 16 × 16 divided texture pixel values TPV in step S32. 16 to 20 are diagrams for explaining the procedure.

  As a premise, the 16 × 16 pixel values TPV of the divided textures in charge that have been read in are locally represented as pixels (x, y) (x = −3 to 12, y = −3 to 12) as shown in FIG. On the other hand, 256 of the threads 2512a, 2512b,... Are selected, and they are made to correspond to two-dimensional pixels (x, y), and the thread 2512 [i, j] (i = 1 to 16 and j = 1 to 16), and each thread 2512 [i, j] takes charge of processing each pixel (x, y) or an average value on a one-to-one basis. It has become. In the specific example shown in FIG. 16 and thereafter, the pixel (-3, -3) corresponds to the thread 2512 [1, 1], and the pixel (12, -3) corresponds to the thread 2512 [16, 1], Pixel (-3, 12) corresponds to thread 2512 [1, 16] and finally pixel (12, 12) corresponds to thread 2512 [16, 16], but this is merely illustrative It is the correspondence of convenience.

  Therefore, first, each of the threads 2512 [i, j] (i = 1 to 16, j = 1 to 16) corresponds to the corresponding pixel (x, y) (x = −3 to 12, y = −3) 12) is stored in the shared memory 2513 (step S321). Then, the process waits until the memory barrier process, ie, all storage in each thread 2512 is completed (step S322).

  Next, as shown in FIG. 16, each of 14 × 14 threads 2512 [i, j] (i = 2 to 15, j = 2 to 15) has its own pixel in charge from shared memory 2513. The value of 3 × 3 pixels can be read from the shared memory 2513, and one value is calculated by prefiltering and stored in the shared memory 2513 (step S323). This pre-filtering process is the same as the process described with reference to FIGS. 5B and 5C, but is different from the previous embodiment in that each thread 2512 performs the pre-filtering process in parallel. Then, the process waits until all storage by the thread 2512 [i, j] (i = 2 to 15, j = 2 to 15) is completed (step S 324). Upon leaving the step, 14 × 14 pieces of information are stored in the shared memory 2513 at this time.

  Next, as shown in FIG. 17, each of the 13 × 13 threads 2512 [i, j] (i = 2 to 14, j = 3 to 15) has a grid including a pixel in charge of itself as a lower left pixel. The value of 2 × 2 pixels to be configured can be read from the shared memory 2513, one value is calculated by gradient filter processing, and stored in the shared memory 2513 (step S325). This gradient filter processing is the same as the processing described with reference to FIGS. 6B and 6C, but is different from the previous embodiment in that each thread 2512 performs the gradient filter processing in parallel. Then, the process waits until all storage by the thread 2512 [i, j] (i = 2 to 14, j = 3 to 15) is finished (step S 326). When this step is exited, 13 × 13 pieces of information are stored in the shared memory 2513 at this time.

  Next, as shown in FIG. 18, each of the 11 × 11 threads 2512 [i, j] (i = 3 to 13, j = 4 to 14) has its own pixel at the lower left side of the lattice center Then, 3 × 3 values included as pixels closest to the lattice center can be read from the shared memory 2513, one value is calculated by averaging filter processing, and stored in the shared memory 2513 (step S327). Although this averaging filter process is the same as the process described with reference to FIGS. 7B and 7C, the place where each thread 2512 performs the averaging filter process in parallel is the same as the previous embodiment. It is different. Then, the process waits until all storage by the thread 2512 [i, j] (i = 3 to 13, j = 4 to 14) is finished (step S 328). When this step is exited, 11 × 11 pieces of information are stored in the shared memory 2513 at this time.

Next, as shown in FIG. 19, each of 9 × 9 threads 2512 [i, j] (i = 4 to 12, j = 5 to 13) is centered on the value of its own grid 3. A value of × 3 can be read from shared memory 2513, and based on those values, in addition to the grid in charge of itself, divided patterns (see FIG. 8) are obtained for the upper, lower, left, and right grids. Similarly to the above, if the division patterns for the upper, lower, left, and right grids are matched, the central divided pattern is determined as the divided pattern for the grid for which it is responsible, and if not matched, it is corrected. The determined central division pattern is stored in the shared memory 2513 as a division pattern for the grid in charge of itself (pattern number etc. is actually stored (step S329)). This process is the same as the process described with reference to FIG. 9, but the process performed by each thread 2512 in parallel is different from the previous embodiment. Then, the process waits until all storage by the thread 2512 [i, j] (i = 4 to 12, j = 5 to 13) is completed (step S330). After the step, finally, information related to the 9 × 9 triangle division pattern is stored in the shared memory 2513.
The reason why the divided textures are overlapped by six grids as shown in FIG. 13 is that, as shown in FIG. 20, it is finally divided into 9 × 9 triangles as described above from 16 × 16 texture pixels. This is for obtaining information related to the pattern, and if the divided textures are not overlapped, an area where information related to the triangle divided pattern can not be obtained occurs.

  Returning to FIG. 14, each of the compute units 251a, b, c,... (In the case of FIG. 13, the compute units 251a to 251f) enlarges the image secured according to the enlargement ratio of the texture supplied from the CPU 1 The display pixels in the buffer for storing, where their positions are UV coordinate values, those positions included in the UV coordinate value range of the divided textures for which they are responsible (positions for which it is desired to obtain RGB values) And the pixels of the divided textures in charge of itself (when the enlargement ratio is large, each position where the value is to be obtained becomes finer) (step S33).

  Then, 9 × 9 threads 2512 [i, j] (i = 4 to 12, j) included in each of the compute units 251a, b, c,... (In the case of FIG. 13, the compute units 251a to 251f). Each of (5) to (5) is, in parallel, the triangle division pattern determined in step S329 above for the grid for which it is responsible, for the plurality of position values to be determined, which are included in the grid for which it is responsible. Based on the above, using the peripheral pixels (see FIG. 10) corresponding to the pattern, it is determined by the triangle division method (step S34).

  Finally, each of the compute units 251a, b, c,... (In the case of FIG. 13, compute units 251a to 251f) includes 9 × 9 threads 2512 [i, j] (i = 4 to 12). , J = 5 to 13) and outputs all the values for the pixels on the display unit 5 to the frame buffer 4 (step S35). A plurality of threads can be allocated to each of 9 × 9 pieces of triangle division information in order to increase the specification efficiency of the threads.

  In the above-described embodiment using a GPU having a compute shader function, the number of texture pixels handled by each compute unit 251 is 16 × 16 pixels, but the present invention is not limited to this. It should be decided. However, naturally, the number of threads 2512 included in each compute unit 251 is limited.

  In the GPU embodiment having the compute shader function described above, each compute unit 251 processes in parallel each split texture obtained by splitting the texture, and is included in each compute unit 251 Since each thread 2512 takes charge of one pixel each, finds a triangle division pattern in parallel, and performs interpolation in parallel by a triangle division method based on those triangle division patterns, so a high-speed and smooth enlarged image You can get

  The image data processing method and the program thereof in the image processing processor of the present invention can be adopted, for example, as a GPU mounted on a game machine, a general-purpose PC, a game machine, a so-called smartphone or the like.

1 CPU
2A, 2B GPU
Reference Signs List 21 vertex shader 22 rasterizer 23 pixel shader 24 texture fetch unit 25 compute shader 251 compute unit 2511 texture fetch unit 2512 thread 2513 shared memory 26 dispatcher 3 VRAM
4 Frame buffer 5 Display

Claims (9)

A vertex shader for obtaining a position on the display unit corresponding to the vertex data of the input polygon based on the vertex data of the polygon;
Pixels on the display unit that are necessary for displaying on the display unit an image in which a designated texture is mapped to the polygon from the position of the vertex on the display unit determined by the vertex shader A rasterizer to determine
Each time the rasterizer gives information on the position of each pixel on the display unit for which a pixel value is required, in order to display on the display unit an image in which a designated texture is mapped to the polygon. A pixel shader which obtains the pixel value of the position by interpolating the values of texture pixels around predetermined positions of the position;
A method of processing image data in an image processing processor comprising:
The pixel shader determines a triangle division pattern in a triangle division method according to the texture grid of pixels including the position, from the pixels constituting the pixel grid and the peripheral pixels thereof, and is determined A method of processing image data comprising: interpolating values of the predetermined surrounding texture pixels based on a pattern to obtain pixel values at the corresponding positions.   The pixel shader is characterized in that, when determining the triangle division pattern, gradient values are obtained while performing low-pass filtering for each value of pixels constituting the pixel grid and pixels in the periphery thereof. The image data processing method according to 1.   The pixel shader determines, when determining the triangle division pattern, the triangle division pattern according to the pixel grid of the texture, which includes the position, and the triangle division pattern according to the top, bottom, left, and right grids of the pixel grid. If the triangle division pattern according to the pixel grid of the texture, which has been temporarily determined and temporarily determined, includes the position, and the triangle division pattern according to the grid above, below, left and right of the pixel grid is mismatched The image data processing method according to claim 1, wherein the triangle division pattern of the texture pixel grid including the position is finally determined by matching. Vertex shader processing for obtaining a position on the display unit corresponding to the vertex data based on vertex data of the input polygon;
From the position of the vertex on the display unit obtained by the vertex shader processing, it is necessary to display on the display unit an image in which the specified texture is mapped to the polygon, on the display unit Rasterizer processing to determine pixels,
Every time the rasterizer processing provides information on the position of each pixel on the display unit for which a pixel value is required, in order to display on the display unit an image in which a designated texture is mapped to the polygon. A pixel shader process for determining the pixel value of the position by interpolating the values of texture pixels around predetermined positions of the position;
And an image data processing program for causing an image processing processor to perform rendering processing,
The pixel shader process determines a triangle division pattern in a triangle division method according to the texture grid of pixels including the position, from the pixels constituting the grid of pixels and the pixels in the periphery thereof An image data processing program comprising: interpolating values of the predetermined surrounding texture pixels based on a division pattern; and determining a pixel value at the position. A method of processing image data in an image processing processor, comprising a compute shader having a plurality of compute units, which performs rendering processing,
Each compute unit is
It takes charge of each divided texture which divided the texture and processes it in parallel, and has a plurality of threads operated in parallel by the program,
From the value of each pixel making up the divided texture using the plurality of threads, for a predetermined number of lattices in each lattice made up of each pixel, a triangular division pattern used in the triangular division method is collectively Decide
In order to display on the display unit an image in which the texture is mapped to a polygon, positions on the predetermined number of grids corresponding to positions of pixels on a buffer storing the enlarged image are obtained.
The value of the pixel on the display unit according to the position included in each grid is used in each of the threads corresponding to each grid, based on the triangle division pattern determined for each grid, in the periphery of the divided texture. An image data processing method characterized by interpolating the pixel values of   The plurality of threads perform processing while storing values obtained in the course of processing in a memory common to the plurality of threads when collectively determining the triangle division pattern. 5. The image data processing method according to 5. When each of the compute units collectively determines the triangle division pattern using the plurality of threads, gradients are applied while low-pass filtering is performed on the values of pixels constituting each grid and pixels in the periphery thereof. Find the value,
Since the number of obtained triangle division patterns is reduced with respect to the number of pixels of the divided textures to be used by the filtering process, the boundaries of the divided textures are redundantly allocated to the respective compute units. The image data processing method according to claim 5, characterized in that:   When each of the compute units collectively determines the triangle division pattern using the plurality of threads, the triangle division pattern according to a grid corresponding to each thread and the grid according to upper, lower, left, and right grids of the grid If the triangle division pattern of the lattice corresponding to the thread, which is temporarily determined with the triangle division pattern, and the triangle division pattern of the upper, lower, left, and right lattices of the lattice mismatch, The image data processing method according to claim 5, wherein the triangle division pattern of the lattice corresponding to the thread is finally determined by matching. An image processing processor comprising a plurality of parallel operable compute units each having a plurality of parallel operable threads, rendering processing based on information of a three-dimensional model composed of polygons and information of a texture to be attached An image data processing program to be executed,
Assigning each of the divided textures obtained by dividing the texture into a plurality to each of the compute units;
Process of causing the plurality of threads to collectively determine a triangle division pattern to be used in a triangle division method from a value of each pixel constituting the divided texture, for a predetermined number of lattices among the lattices constituted by each pixel When,
A process of determining positions on the predetermined number of grids corresponding to positions of pixels on a buffer storing an enlarged image, in order to display an image in which the texture is mapped to the polygon on a display unit;
In each thread corresponding to each grid, the value of the pixel on the display unit according to the position included in each grid is calculated based on the triangle division pattern determined for each grid, of surrounding pixels of the divided texture. An image data processing program comprising processing for interpolating and obtaining values.

Images