JP2017123034A - Image processing apparatus including cip and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which draws an image in accordance with a shader program achieving acceleration and low power consumption of processing by efficiently using a fixed function pipeline.SOLUTION: An image processing apparatus includes: a main unit 10 which can program calculation processing to be executed; a sub-unit 20 which has a plurality of calculation units for performing predetermined calculation processing and in which the calculation units are connected in series so that the calculation result by the calculation unit on the former stage is input to the calculation unit on the latter stage; and a compilation processing unit 30 which extracts a code that can be executed by the sub-unit 20 from a shader program in accordance with a connection pattern of the calculation units constituting the sub-unit 20 and rewrites the extracted code into a code for the sub-unit that can be executed by the sub-unit 20. The main unit 10 causes the sub-unit 20 to execute the code for the sub-unit in accordance with the shader program rewritten by the compilation processing unit 30 and executes other codes by itself.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus including a CIP (Combined Instruction Processor) functioning as a fixed function pipeline, and an image processing method executed by the apparatus. More specifically, the present invention allows computer graphics to be consumed at high speed and with low consumption by causing a part of arithmetic processing executed in the programmable pipeline (main unit) to be executed by the fixed function pipeline (sub unit: CIP). The present invention relates to an apparatus and a method for drawing with electric power.

  Traditionally, computer graphic drawing has been performed with a fast, small, fixed-function pipeline that is easy to implement. However, with the development of computers, image processing by fixed function pipelines has been replaced by image processing by programmable pipelines that are more flexible and capable of various processing, and most of them are now performed by programmable pipelines (Patent Documents). 1, Patent Document 2). The programmable pipeline incorporates a programmable shader in which a programmer defines an image generation algorithm, for example, and can be used for various purposes. As an example of a programmable shader, a vertex shader, a geometry shader, a fragment shader (pixel shader), and the like can be individually configured, and these can be integrated to form an integrated shader. As a method of mounting the programmable shader processor on the hardware, it is possible to adopt a mode in which a plurality of processors on which a plurality of programmable shaders are mounted are set on a chip or a board in consideration of necessary processing capability. It is common. Since a free shader program can be used in such a programmable pipeline, flexible expression is possible, and it is possible to draw complicated shadows that could not be realized in the conventional fixed function pipeline.

  However, when a programmable pipeline is installed in a processor, the programmable pipeline can perform various operations on a general-purpose basis, so the processor is larger than the configuration with only a fixed-function pipeline. Tend to. In addition, it is necessary to further increase the size of the programmable pipeline when attempting to obtain a predetermined calculation capability, but this is a particularly significant problem in portable game machines with limited mounting space. Therefore, the applicant of the present invention is an image processing apparatus to which information processed by the CPU is input, and includes a fixed function pipeline that performs predetermined arithmetic processing and a programmable pipeline that can program the arithmetic processing to be executed. A hybrid type image processing apparatus equipped with both has been developed (Patent Document 3). In this image processing apparatus, information processed by the CPU is input to either a fixed function pipeline or a programmable pipeline and selectively processed. As a result, high-speed image processing was realized while reducing power consumption and chip mounting space.

JP 2005-322224 A JP 2010-250625 A JP 2012-164238 A

  However, the image processing apparatus disclosed in Patent Document 3 mounts both a fixed function pipeline and a programmable pipeline, but operates only one of them when performing image processing. That is, in this image processing apparatus, information processed by the CPU is selectively input to either the fixed function pipeline or the programmable pipeline. Complicated processes such as those requiring processing are executed in the programmable pipeline, while other simple processes are executed in the fixed function pipeline. As described above, this image processing apparatus processes an input shader program in either a fixed function pipeline or a programmable pipeline, and does not have a function of operating these simultaneously. In addition, since this image processor selectively operates the programmable pipeline and the fixed function pipeline, the programmable pipeline always operates when the shader program exceeds the processing capability of the fixed function pipeline. However, if such processing continues, the fixed function pipeline will not function at all, and not only can the image processing speed not be increased by using the fixed function pipeline, but also the fixed function pipeline is installed. There was a risk of wasting space.

  Therefore, the present invention achieves further speeding up of image processing and lower power consumption by efficiently using the fixed function pipeline in the image processing apparatus having both the fixed function pipeline and the programmable pipeline. This is the solution issue.

  The inventors of the present invention, as a result of diligent research on the means for solving the above problems, extract code (lines) that can be executed by the fixed function pipeline (subunit) from the original shader program by compile processing, and extract the code Code that can be executed in the fixed-function pipeline, and the compiled shader program is input to the programmable pipeline (main unit), and the rewritten code portion is controlled under the programmable pipeline. I got the knowledge that the fixed function pipeline is executed. As a result, even when a shader program that exceeds the processing capability of the fixed function pipeline is input, a part of the shader program can be executed by the fixed function pipeline, and at the same time, other parts can be executed. Can be executed in the programmable pipeline, so that the fixed-function pipeline can be used efficiently to achieve higher speed image processing and lower power consumption. Then, the present inventors have conceived that the problems of the prior art can be solved based on the above knowledge, and have completed the present invention. If it demonstrates concretely, this invention has the following structures and processes.

A first aspect of the present invention relates to an image processing apparatus that draws an image according to a shader program. The image processing apparatus according to the present invention includes a main unit 10, a subunit 20, and a compile processing unit 30.
The main unit 10 is a programmable pipeline, and can program the arithmetic processing to be executed.
The subunit 20 is a fixed function pipeline, and has a plurality of arithmetic units that perform predetermined arithmetic processing, and each of the sub units 20 is input so that the arithmetic result of the preceding arithmetic unit is input to the subsequent arithmetic unit. Arithmetic units are connected in series. One or more subunits 20 are provided in the image processing apparatus. A configuration in which a plurality of arithmetic units are connected in series in this manner is referred to as a CIP (Combined Instruction Processor) in this specification.
The compile processing unit 30 extracts code that can be executed by the subunit 20 from the shader program in accordance with the connection pattern of the arithmetic units constituting the subunit 20, and uses the extracted code for the subunit that can be executed by the subunit 20. Compile the code to rewrite it. The shader program compiled by the compile processing unit 30 is input to the main unit 10.
The main unit 10 causes the subunit 20 to execute the subunit code in accordance with the shader program that has been rewritten by the compile processing unit 30, and executes the other codes by the main unit itself.

  In the present invention, a shader program includes a plurality of codes (lines) that define variables obtained from operations according to instructions. For example, in the example shown in FIG. 2A, in the code “A = ADD B, C”, the part “A” means “variable” and the part “ADD B, C” means “instruction”. Means. A plurality of codes included in the shader program have a dependency relationship between codes in which a variable of a certain code is included in an instruction of another code. In this case, the compile processing unit 30 extracts from the shader program those in which the dependency between the codes coincides with the dependency between the arithmetic units constituting the subunit 20 in the compile processing. Rewrite to executable subunit code.

  In the present invention, when the compile processing unit 30 extracts the code to be rewritten as the subunit code, all the sets of instructions that use the variable of the code are extracted from the shader program as the subunit code. It is preferable that it is limited to the case where it is possible.

  The image processing apparatus of the present invention preferably includes a plurality of subunits 20. In this case, it is preferable that the connection patterns of the arithmetic units constituting each subunit 20 are different. In this way, a plurality of settings for the subunit 20 are prepared, and when the processing is requested from the main unit 10 to the subunit 20, it is designed to specify which setting is to be used in the shader program. The processing of the subunit 20 can be assigned to a plurality of locations, and the usage frequency of the subunit 20 can be further increased.

  In the present invention, it is preferable that the main unit 10 has a plurality of contexts that store values of variables used in the arithmetic processing, and the contents of the arithmetic processing can be switched by switching the context to be used. As described above, the main unit 10 has a switching function for switching contexts, so that the main unit 10 itself can perform another calculation process while the subunit 20 executes a specific process. Thereby, it is possible to hide the waiting time of the calculation result by the subunit 20, and it is possible to further improve the efficiency of the image processing.

  In the image processing apparatus of the present invention, it is preferable that the number of arithmetic units constituting the subunit 20 is four. In consideration of the size and performance of the circuits constituting the image processing apparatus, it can be said that the number of arithmetic units of the subunit 20 is preferably four in terms of statistics.

  In the image processing apparatus of the present invention, the arithmetic unit constituting the subunit 20 is preferably composed of an adder and a multiplier. In image processing of computer graphics, addition and multiplication are often performed, and the subunit 20 can be used more efficiently by causing the subunit 20 to execute this.

  In the image processing apparatus of the present invention, the main unit 10 includes a processing unit (for example, a texture processing unit) that does not perform arithmetic processing at the same time as the subunit 20, and constitutes a computing unit and the subunit 20 that constitute the processing unit. It is preferable that at least a part of the arithmetic units is shared. In this way, by sharing a part of the arithmetic unit constituting the main unit 10 with the arithmetic unit constituting the subunit 20, the circuit scale of the entire image processing apparatus can be further reduced.

  In the image processing apparatus according to the present invention, the above-described compile processing unit 30 may not be provided in the image processing apparatus itself. That is, the shader program is a code for subunits that can be executed by the subunit 20 in advance according to the connection pattern of the computing units constituting the subunit 20 by another computer or the like. It may have been rewritten. In this case, the main unit 10 causes the subunit 20 to execute the subunit code in accordance with the shader program that has been rewritten in advance, and executes the other codes in the main unit.

The second aspect of the present invention relates to an image processing method executed by an image processing apparatus that draws an image according to a shader program. Similar to the first aspect described above, the image processing apparatus includes a main unit 10 that can program arithmetic processing to be executed, and a plurality of arithmetic units that perform predetermined arithmetic processing. Is provided with one or a plurality of subunits 20 in which the respective arithmetic units are connected in series so as to be input to the post-stage arithmetic unit.
Here, the image processing method includes a compile processing step and an execution step.
In the compile processing step, code that can be executed by the subunit 20 is extracted from the shader program in accordance with the connection pattern of the arithmetic units constituting the subunit 20, and the extracted code is executed by the subunit 20. Rewrite to
In the execution process, the main unit 10 causes the subunit code to be executed by the subunit 20 in accordance with the shader program that has been rewritten in the compile processing process, and other codes are executed by the main unit itself.

  According to the present invention, in an image processing apparatus equipped with both a fixed function pipeline and a programmable pipeline, the fixed function pipeline is efficiently used to further increase the speed of image processing and reduce power consumption. be able to.

  More specifically, in the present invention, code that can be executed by the subunit 20 (fixed function pipeline) is extracted from the original shader program in the compiling process, and the extracted code is executed by the subunit 20. Rewrite to Then, the compiled shader program is input to the main unit 10 (programmable pipeline), and the rewritten code portion is executed by the subunit 20 under the control of the main unit 10. As a result, even when a shader program exceeding the processing capability of the subunit 20 is input as a whole, a part of the shader program can be executed by the subunit 20, and at the same time, other shader programs can be executed. The part can be executed in the main unit 10. As described above, since it is possible to perform image processing by operating the main unit 10 and the subunit 20 at the same time, it is possible to realize high-speed image processing by increasing the calculation capability while suppressing an increase in circuit scale. . Further, since the subunit 20 that is a fixed function pipeline consumes much less power than the main unit 10, the subunit 20 is actively responsible for the executable portion, thereby consuming the entire image processing apparatus. Electric power can be kept low.

  Even when a shader program exceeding the processing capability of the subunit 20 is input, a part of the processing can be assigned to the subunit 20, so that the subunit 20 is actively Thus, the mounting space for the subunit 20 is not wasted.

  Furthermore, in the present invention, in the compile process, the original shader program is analyzed according to the connection pattern of the arithmetic units constituting the subunit 20, so that the code executable by the subunit 20 is extracted, The code for the unit 20 can be automatically rewritten. For this reason, even if the programmer does not create a shader program assuming the connection pattern of the arithmetic units in the subunit 20, the shader program is automatically rewritten to one that can be executed in the subunit 20, so that the burden on the programmer is reduced. Can be reduced. Even when an existing shader program created in the past is directly input to the image processing apparatus of the present invention, the shader program is automatically rewritten, so that any shader program can be used for general purposes.

  In addition, by using the subunit 20 in which a plurality of arithmetic units are connected in series, the number of times of reading and writing the intermediate variables in the arithmetic processing is reduced, so that the size of the context can be kept small. In addition, the load on the memory access bus can be reduced by reducing the number of times the variable is read from and written to the context. Furthermore, if the bus load is reduced, the competition for accessing textures and other images decreases, so that each unit operates smoothly.

FIG. 1 is a block diagram showing the basic concept of an image processing apparatus and an image processing method. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus. FIG. 3 is a block diagram illustrating a configuration example of the subunit. Fig. 4 shows the basic concept of processing by the compilation processing unit. FIG. 5 is a block diagram showing a specific configuration of the subunit. FIG. 6 shows a specific example of processing by the compile processing unit. FIG. 7 is an explanatory diagram regarding processing conditions by the compilation processing unit. FIG. 8 shows an example of optimization processing by the compile processing unit. FIG. 9 shows an example of the hardware configuration of the subunit. FIG. 10 is a block diagram showing an embodiment of the image processing apparatus. FIG. 11 is a block diagram showing another embodiment of the image processing apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. This invention is not limited to the form demonstrated below, The thing suitably changed in the range obvious to those skilled in the art from the following forms is also included.

  FIG. 1 is a block diagram showing the basic concept of an image processing apparatus and an image processing method according to the present invention. As shown in FIG. 1, an original shader program composed of a plurality of codes (lines) is input to the compile processing unit 30. The compile processing unit 30 performs compile processing for rewriting a shader program (source code) written in a programming language into a machine language shader program (binary code) that can be directly executed by a computer. In this compile processing, the compile processing unit 30 extracts a code that can be executed by the subunit 20 (CIP) having a fixed arithmetic function from the original shader program, and can execute the extracted code by the subunit 20. Rewrite the code to the correct subunit code. Details of the compiling process will be described later. The compiled shader program is input to the GPU 100 (Graphics Processing Unit). The GPU 100 includes a main unit 10 that can arbitrarily program the arithmetic processing to be executed, and one or a plurality of subunits 20 (CIP) having a fixed arithmetic function. When the main unit 10 receives the compiled shader program, the main unit 10 causes the subunit 20 to execute the subunit code and executes the other codes by itself according to this program. Then, the main unit 10 collectively outputs the calculation result by itself and the calculation result by the subunit 20 to the external memory 200 or other external circuit.

  Here, the compile processing unit 30 is preferably mounted in the same image processing apparatus (computer) as the GPU 100. However, the compile processing unit 30 can be mounted on a computer different from the computer on which the GPU 100 is mounted. In this case, the shader program compiled by the compile processing unit 30 provided in another computer is input to the same image processing apparatus as the GPU 100 via a communication network such as a recording medium or the Internet.

  FIG. 2 is a block diagram showing in more detail the configuration of the GPU 100 shown in FIG. The GPU 100 includes a main unit 10 and one or a plurality of subunits 20 (CIP). The subunit 20 operates based on the control of the main unit 10. For example, the main unit 10 is OpenGL2. Hardware that functions as a programmable pipeline corresponding to the x system and later can be implemented, and the subunit is the conventional OpenGL1. It is possible to implement hardware that functions as a fixed function pipeline corresponding to the x system.

  The main unit 10 constituting the GPU 100 typically includes a management unit 11, a vertex processing unit 12, a rasterization processing unit 13, a fragment processing unit 14, a texture processing unit 15, a color update processing unit 16, And an internal memory 17. The main unit 10 is also connected to an external memory 200 placed outside the GPU 100.

  The management unit 11 receives a compiled shader program from the compilation processing unit 30 and controls the processing units 12 to 16 according to the shader program. The management unit 11 reads vertex information of polygon data (three-dimensional model) from the external memory 200 and transfers it to the vertex processing unit 12.

  The vertex processing unit 12 performs vertex processing including coordinate conversion and illumination calculation on each vertex constituting the polygon data. In vertex processing, for example, coordinate transformation processing that converts the coordinate values of vertices expressed in the modeling coordinate system into the world coordinate system, camera coordinate system, or projected coordinate system, the distance between each vertex and the light source, and the distance between each vertex and the light source Illumination calculation processing for calculating the luminance of the vertex based on the angle or texture processing for calculating texture coordinate values for texture mapping is performed. The vertex processing unit 12 passes the vertex information obtained as a result of the vertex processing to the rasterization processing unit 13.

  The rasterization processing unit 13 performs rasterization processing for generating data (fragment) for each pixel of polygon data based on the vertex information obtained by the vertex processing. The result calculated by the vertex processing is only the result for that vertex. To actually draw computer graphics, draw for all the pixels in the polygon data consisting of multiple vertices. The color needs to be calculated. For this reason, the rasterization processing unit 13 performs a rasterization process in which a figure represented by a combination of a vertex and a line segment (vector) is replaced with a set of pixels. In addition, interpolation of the luminance value inside the polygon from the luminance value of the vertex in glow shading may be performed in the rasterizing process. The rasterization processing unit 13 delivers the fragment generated by the rasterization processing to the fragment processing unit 14.

  The fragment processing unit 14 performs arithmetic processing (fragment processing) on the fragment generated by the rasterization processing. The fragment processing unit 14 performs various calculations related to colors such as color value, luminance, and transparency for each fragment. In addition, the fragment processing unit 14 can perform an operation based on the luminance value of each calculated fragment and the texture value that can be referred to. In this case, a mixed value (addition value) of the texture value with respect to the luminance value of the fragment. Or multiplication value). Specifically, the fragment processing unit 14 can perform texture processing for pasting a texture image for giving a texture to the surface of a three-dimensional object. Through texture processing, texture coordinates (u, v) corresponding to the texture image are added to the pixel data. In addition, the fragment processing unit 14 can perform a shading process that changes the color value (tone or gradation) of each pixel constituting the object in consideration of the angle of light and the distance from the light source. For example, the vertex processing unit 12 performs light source calculation in the vertex processing, obtains information on a light source for shading processing, a lighting model, a normal vector of each vertex of the object, and the like based on the vertex information. The color value (RGB) of each vertex is obtained. Then, the fragment processing unit 14 can obtain the color value of each pixel corresponding to the polygon by, for example, phone shading or glow shading based on the color value of each vertex obtained by the vertex processing. Thereafter, the fragment processing unit 14 writes the result of the calculation related to the color of the fragment in the external memory 200 for the display screen.

  When the texture processing unit 15 needs the texture data stored in the external memory 200 to perform the image processing of the three-dimensional model, the texture processing unit 15 reads the necessary texture data from the external memory 200, and the vertex processing unit 12, the rasterization processing unit 13, or the fragment processing unit 14. Further, the texture data read from the external memory 200 by the texture processing unit 15 can be temporarily stored in the internal memory 17. In this case, the vertex processing unit 12 and the like read necessary texture data from the internal memory 17. be able to.

  The color update processing unit 16 is an arbitrary element. The color update processing unit 16 merges the result of the fragment processing into the content stored in the frame buffer on the external memory 200 (or the internal memory 17), or updates the information regarding the content for each fragment.

  The internal memory 17 is basically connected to the processing units 12 to 16 and writes and reads information necessary for the processing units 12 to 16 to perform arithmetic processing. The internal memory 17 can temporarily store all or part of the shader program executed by the GPU 100, and can store the values of variables that appear during the calculation. Also has a role. Finally, all information such as a shader program, a final result image, a referenced texture image, and vertex information is stored in the external memory 200.

  In addition, the structure of the main unit 10 demonstrated here is only an example, and can employ | adopt a well-known structure suitably. For example, the vertex processing unit 12 and the fragment processing unit 14 are implemented as the same device, and the vertex processing and the fragment processing can be performed as the same device and can be executed in a time-sharing manner. Various configurations are conceivable, such as whether the fragment processing unit 14 and the texture processing unit 15 are directly connected, or whether data can be transmitted / received via the internal memory 17. The details of these configurations do not limit the specification conditions of the present invention.

  The subunit 20 (CIP) that constitutes the GPU 100 executes a part of the processing that can be executed by the main unit 10 in order to increase the processing speed and reduce the power consumption. That is, while the sub unit 20 is executing a process, the main unit 10 can also execute another process at the same time. The subunit 20 is connected to one or a plurality of processing units (vertex processing unit 12, rasterization processing unit 13, fragment processing unit 14, texture processing unit 15, and color update processing unit 16) constituting the main unit 10. be able to. In the embodiment shown in FIG. 2, the subunit 20 is connected to the vertex processing unit 12 and the fragment processing unit 14. As described above, the subunit 20 is preferably one that shares and executes part of the vertex processing and / or fragment processing executed by the vertex processing unit 12 and the fragment processing unit 14. However, the processing unit to which the subunit 20 is connected is not limited to these. In the example shown in FIG. 2, one subunit 20 is mounted in the image processing apparatus, but two or more subunits 20 can be mounted.

  FIG. 3 shows an example of the configuration of the subunit 20. As shown in FIG. 3, the subunit 20 includes a plurality of arithmetic units 21, 22, and 23 that perform predetermined arithmetic processing. The plurality of calculators 21 to 23 are connected in series so that the calculation result of the preceding calculator is input to the succeeding calculator. That is, in the example shown in FIG. 3, the calculation result of the first calculator 21 is input to the second calculator 22, and the second calculator 22 is based on the calculation result of the second calculator 22. Perform the operation. Further, the calculation result of the second calculator 22 is input to the third calculator 23, and the third calculator 23 performs a calculation based on the calculation result of the second calculator 22. Then, the calculation result by the third calculator 23 is returned to the main unit 10. The configuration of the subunit 20 (fixed function pipeline) having the plurality of arithmetic units 21 to 23 connected in series in this way is referred to as a CIP (Combined Instruction Processor) in the present specification. Further, the subunit 20 does not have an internal memory for writing out the calculation results by the respective arithmetic units or reading out data when performing the arithmetic operations by the individual arithmetic units. Since each arithmetic unit is connected in series, the subunit 20 can perform computation without using an internal memory. Since writing to and reading from the internal memory are not performed, the subunit 20 can complete the arithmetic processing at high speed and with low power consumption. In addition, the number of stages of the arithmetic units constituting the subunit 20 may be two, or may be three or more. In particular, the number of stages of the arithmetic unit is preferably 4 in terms of statistics.

  A well-known thing can be employ | adopted as a calculator which comprises the subunit 20. FIG. For example, the arithmetic unit preferably includes at least an adder (ADD) and a multiplier (MUL). In addition, one type or two or more types of arithmetic units may be employed from MADD, EQUAL, NEQUAL, LESS, LEQUAL, SUM, RSQ, SEL, EXP, and LOG. Further, as shown in FIG. 3, each computing unit may have two inputs, or may have one input or three or more inputs. For example, a 2-input computing unit is preferably one that performs an addition or multiplication operation, and a 3-input computing unit is preferably one that performs a multiplication count instruction. Further, it is preferable that the one-input computing unit performs a computation of “RSQ (x) = 1 / sqrt (x)”. Which calculation process each calculator performs is set in advance according to the driver program before the drawing process is started. Such a calculator setting may be created when the compile processing unit 30 converts the original shader program. Further, when a plurality of subunits 20 are provided in the image processing apparatus, it is preferable that the patterns of the arithmetic units constituting each subunit 20 are different. By providing a plurality of subunits 20 with different settings, the processing of the subunits 20 can be assigned to a plurality of locations in the shader program, and the usage frequency of the subunits 20 can be further increased.

  Further, as shown in FIG. 2, the subunit 20 may share a part or all of the computing units and the processing units constituting the main unit 10. That is, the main unit 10 includes a processing unit that does not perform arithmetic processing simultaneously with the subunit 20. In this case, by sharing a part or all of the arithmetic units constituting the processing unit of the main unit 10 and the sub unit 20, the number of arithmetic units mounted in the image processing apparatus can be reduced. Therefore, it is possible to further reduce the size of the image processing apparatus. In the example shown in FIG. 2, the main unit 10 includes the texture processing unit 15, but the texture processing unit 15 basically does not operate simultaneously with the subunit 20. For this reason, even if the arithmetic unit constituting the texture processing unit 15 and the arithmetic unit constituting the subunit 20 are shared, the image processing is not affected. Therefore, in the example shown in FIG. 2, the computing unit can be shared between the texture processing unit 15 and the subunit 20.

  Here, the compile processing unit 30 grasps the connection pattern of the arithmetic units constituting the subunit 20 as described above, and the original shader program is generated by the subunit 20 according to the connection pattern of the arithmetic units. Compile processing, including processing to rewrite it to include executable code. Here, the “computation pattern of computing units” is information including information on the number of computing units constituting the subunit 20 and the types of computing units constituting each stage. For example, when the number of arithmetic units constituting the subunit 20 is three and each arithmetic unit is connected as “ADD → ADD → ADD”, the compile processing unit 30 determines that “the connection of the arithmetic units”. As the “pattern”, it is understood that the number of stages of the arithmetic units is three and that the combination (dependency) of the arithmetic units is “ADD → ADD → ADD”.

  FIG. 4 is an explanatory diagram showing the concept of code extraction / rewriting processing performed by the compile processing unit 30. As shown in FIG. 4, the shader program includes a plurality of codes that define “variables” obtained from operations according to “instructions”. That is, in the example shown in FIG. 4A, the program 1 (shader program) includes a total of three lines of code including the first line of code “A = ADD B, C” and the other two lines of code. In the code on the first line, the part “A” means “variable” and the part “ADD B, C” means “instruction”. That is, the code in the first line means that an operation for obtaining the variable A by adding the constant B and the constant C is performed. Here, in the program 1 shown in FIG. 4A, the variables defined in each code are not incorporated in the instructions of other codes. For this reason, in the program 1, there is no dependency relationship between the codes. In order to perform an operation according to the program 1, the GPU 100 needs to refer to a context in which a constant value is already stored and a space for storing a variable value is provided, as in the context 1. In the program 1, since each code has no dependency relationship, the subunit 20 having the CIP configuration as shown in FIG. 3 cannot be in charge of processing. For this reason, the compile processing unit 30 cannot rewrite the program 1 for the subunit 20. Here, a table storing variable values is called a context.

  On the other hand, as shown in FIG. 4B, in the program 2 (shader program), the variable “A” of the code on the first line is incorporated in the instruction of the code on the second line, and the second line The code variable “E” is incorporated in the code instruction on the third line. Therefore, it can be said that the three lines of code constituting the program 2 have a dependency relationship such as the first line → the second line → the third line. However, in the statement description method such as program 2, firstly, the first line operation is performed to obtain the variable “A”, the value is stored in the context 2, and the second line operation is performed. , Read the value of the variable “A” by referring to the context 2 to obtain the variable “E”, store the value in the context 2, and refer to the context 2 when performing the operation on the third line. A process of reading the value of the variable “E”, obtaining the variable “G”, and storing the value in the context 3 is performed. As described above, when writing or reading of data with respect to the context is necessary, it is necessary to perform arithmetic processing by the main unit 10 having the internal memory 17 and to perform arithmetic processing by the subunit 20 having no internal memory. I can't.

  Therefore, the compile processing unit 30 performs a process of rewriting the program 2 so that even the subunit 20 having no internal memory can be executed. For example, as shown in FIG. 3, in the case where the subunit 20 is composed of three stages of arithmetic units, and all the arithmetic units are multipliers (ADD), the arithmetic units constituting the subunit 20 are connected to each other. The dependency relationship and the dependency relationship between the codes constituting the program 2 are completely matched. In this case, the compile processing unit 30 extracts all the codes of the program 2, rewrites the extracted codes into codes that can be executed by the subunit 20, and outputs a compiled program 3 (shader program). In the example shown in FIG. 4B, in the program 3, the variable parts in the first and second lines are deleted, and [PREV] is described in the instruction parts in the second and third lines. This [PREV] means to directly substitute the calculation result of the preceding code (calculation expression). For this reason, in the program 3, the operation result of the first line is assigned as it is to the instruction part of the second line, and the operation result of the second line is assigned as it is to the instruction part of the third line. As described above, since the calculation result between the codes can be directly assigned to another code, the program 3 does not require a context for reading and writing the calculation result. Since the code dependency in the program 3 matches the dependency relationship of the arithmetic units in the subunit 20, the program 3 can be executed in the subunit 20. In this way, the compile processing unit 30 has a function of rewriting part or all of the original shader program so as to match the coupling pattern of the arithmetic units constituting the subunit 20.

  Further, with reference to FIGS. 5 and 6, the code extraction / rewriting process by the compile processing unit 30 will be described in more detail. FIG. 5 shows a coupling pattern of arithmetic units constituting the subunit 20. In the example shown in FIG. 5, the subunit 20 has a configuration in which three stages of computing units 21 to 23 are connected in series, the first computing unit 21 is an adder (ADD), and the second computing unit 22. Is a multiplier (MUL), and the third computing unit 23 is an adder (ADD). The compile processing unit 30 grasps the coupling pattern of the arithmetic units constituting such a subunit 20.

  FIG. 6 shows the flow of code extraction / rewriting processing by the compile processing unit 30. First, the program 4 is input to the compile processing unit 30 as an original shader program. The program 4 is composed of a total of 8 lines of code. The compile processing unit 30 analyzes the program 4 and extracts, from among a plurality of codes, ones whose dependency relationships between the codes coincide with the dependency relationships between the arithmetic units constituting the subunit 20. Specifically, as described above, the dependency relationship between the arithmetic units constituting the subunit 20 is “ADD → MUL → ADD”. Therefore, the compile processing unit 30 includes this dependency relationship from the program 4. Extract matching code. In the example shown in FIG. 6, it can be seen that the dependency relationships of the codes in the first line, the fifth line, and the seventh line in the program 4 correspond to the dependency relations between the arithmetic units. Therefore, the compile processing unit 30 extracts the first line, the fifth line, and the seventh line, and rewrites the program 5 in an appropriate order. In Program 4, the three lines of code in the 1st, 5th, and 7th lines that were arranged separately in Program 4 have been rewritten so that they are aligned in the 5th to 7th lines in Program 5. I understand. Furthermore, the compile processing unit 30 can generate the program 6 by rewriting the codes in the fifth to seventh lines in the program 5 into one line of code that can be easily executed by the subunit 20 (CIP). it can. In the program 6, the code on the fifth line is rewritten as “R15 = CIP (R2, R1, R12, R12)”. As shown in FIG. 5, this code inputs “R2” to input 1 of the subunit (CIP), “R1” to input 2, “R12” to input 3, and “R12” to input 4. This means that an operation for obtaining the variable “R15” is performed by the subunit (CIP). As described above, the code on the fifth line in the program 6 corresponds to the connection pattern of the arithmetic units in the subunit 20 and can therefore be executed by the subunit 20. In addition, since the codes in the first to fourth lines and the sixth line in the program 6 cannot be executed in the subunit 20, they are executed by the main unit 10.

  In this way, by rewriting the original shader program (program 4) into a program (program 6) that includes code executable by the subunit 20 and other codes (program 6), the subunit 20 and the main unit 10 operate simultaneously. Thus, both units can be operated in parallel. As a result, high-speed arithmetic processing can be realized. Further, since the subunit 20 composed of the fixed function pipeline has much lower power consumption than the main unit 10, the subunit 20 is responsible for a part of the arithmetic processing described in the shader program. Thus, the power consumption of the entire image processing apparatus can be suppressed. Further, since the subunit 20 composed of a plurality of arithmetic units is smaller than the main unit 10 having a programmable circuit scale, a large number of necessary calculation units such as adders and multipliers are mounted in a small area. Can do. For this reason, the circuit scale of the entire image processing apparatus can be reduced while maintaining the processing performance.

  Next, conditions for rewriting the code included in the shader program so that it can be executed by the subunit 20 will be described with reference to FIGS. In the shader program (program 7) shown in FIG. 7, the codes in the 5th to 7th lines coincide with the 5th to 7th lines of the program 6 shown in FIG. For this reason, it is also conceivable to rewrite the fifth to seventh lines of the program 7 with a code that can be executed by the subunit 20 shown in FIG. However, in the program 7, it is necessary to use the variable “R3” obtained by the operation on the fifth line in the instruction part of the code on the eighth line. Here, when the code in the 5th to 7th lines in the program 7 is rewritten to the code for the subunit 20 and is executed in the subunit 20, the variable “R3” is not written in the context in the subunit 20. Will only be consumed. Then, the variable “R3” cannot be used for the code on the eighth line in the program 7, and the calculation result of the code on the eighth line cannot be obtained. Therefore, in such a case, even if the program 7 includes a code that matches the dependency between the arithmetic units constituting the subunit 20, the code is used as the code for the subunit 20. Cannot be rewritten. In this way, when the compile processing unit 30 extracts the code to be rewritten as the subunit code, all the instruction sets that use the code variables are extracted from the shader program as the subunit code. Limited to when possible.

In other words, the code group that the compile processing unit 30 selects from the shader program to be executed by the subunit 20 needs to satisfy the following two conditions.
(Condition 1) The entire shader program must be assigned to the same variable only once.
(Condition 2) A set of instructions that use the variables of the code group to be selected is not taken into the code group for the subunit.

  Here, the condition 1 and the condition 2 need only satisfy the shader program after optimization by the compile processing unit 30. More specifically, in the shader program (program 9) shown in FIG. 8, the same variable “R3” is assigned to the first and second lines. For this reason, the program 9 before optimization does not satisfy the above condition 1. However, since the code on the first line is a dead code, the compile processing unit 30 should delete the code on the first line in the course of the optimization process and omit the assignment to the variable “R3”. Then, after the code on the first line is deleted, the above condition 1 is satisfied. In the program 9, the variable “R3” is used in the instruction part of the code for obtaining the last variable “R18”. For this reason, the program 9 before optimization does not satisfy the above condition 2. However, since the code on the last line is also dead code, the compile processing unit 30 should delete the code on the last line in the course of the optimization process. Then, after deleting the code in the last line, the above condition 2 is satisfied. As described above, the program 9 cannot be assigned to the subunit 20 in the pre-optimal state, but by performing the optimization and deleting the dead code, the condition 9 and the condition 2 are satisfied. Can be assigned to unit 20

  FIG. 9 shows a preferred example of the hardware configuration of the subunit 20. As shown in FIG. 9, the subunit 20 has a plurality of arithmetic units (STAGE 0 to STAGE N) connected in series so that the arithmetic result of the preceding arithmetic unit is directly input to the subsequent arithmetic unit. It is configured. The sub unit 20 further includes a plurality of input registers (IN 0 to IN 3) for temporarily holding the operation values (particularly variables) supplied from the main unit 10 and constants supplied from a driver program or the like. A plurality of constant registers (CONST 0 to CONST 7) are provided for temporary storage. The values held by the input register and the constant register are input to a plurality of arithmetic units as necessary and used for arithmetic operations by each arithmetic unit. However, as shown in FIG. 9, the subunit 20 does not include a memory or a register for storing a calculation result by each calculator. The value calculated through each calculator is output to the main unit 10 as it is without being stored in a memory or the like. As described above, since the configuration of the subunit 20 is simple without a memory or the like, it consumes less power and can perform arithmetic processing at high speed.

  In the image processing apparatus of the present invention, the main unit 10 has a plurality of contexts that store the values of variables used in the arithmetic processing, and the contents of the arithmetic processing can be switched by switching the context to be used. It is preferable that For example, when there is an instruction that takes a long time until the operation result by the subunit 20 is output, the context switch function of the main unit 10 performs other operation processing using another context during the waiting time. It is a function to perform. The main unit 10 simultaneously executes a plurality of processes with different variable contents even in the same shader program. For this reason, the main unit 10 is provided with a plurality of contexts (variable group tables) describing the contents of variables, and when waiting time occurs for processing based on a certain context, switch to another context. Efficiency can be achieved by executing other processes. In the image processing apparatus of the present invention, by using the main unit 10 having such a context switching function, the main unit 10 and the plurality of subunits 20 can be effectively cooperated. In other words, since the main unit 10 has a switching function for switching contexts, the main unit 10 itself can perform another calculation process while the subunit 20 is executing a specific process. Thereby, it is possible to hide the waiting time of the calculation result by the subunit 20, and it is possible to further improve the efficiency of the image processing.

  Next, with reference to FIG. 10 and FIG. 11, a part where the compile processing unit 30 is provided will be described. As shown in FIGS. 10 and 11, a computer (image processing apparatus) 1000 having an image processing function generally includes a GPU 100 (graphics processing apparatus), an external memory 200 (storage apparatus), and a CPU 300 (central These devices are connected to each other through a bus or the like. The CPU 300 executes the operating system program and also appropriately reads and executes application programs and driver programs stored in the external memory 200. When it is necessary to draw graphics, the CPU 300 executes a driver program for the GPU 100 and performs necessary settings (such as setting constants) in the GPU 100 according to the driver program. Thereafter, the GPU 100 draws desired graphics on the external memory 200 in accordance with a shader program stored in the external memory 200. The CPU 300 sends the image stored in the external memory 200 to the display device as necessary.

  Here, in a graphics processing system such as OpenGL, when a shader program executed by the GPU 100 is provided to the computer (image processing apparatus) 1000 in the state of binary code after completion of the compiling process, There are two cases when provided in the state of the previous source code.

  FIG. 10 shows an example in which the shader program is provided to the image processing apparatus 1000 from another computer 2000 in the state of binary code that has been compiled. The compiled shader program can be provided to the image processing apparatus 1000 via a recording medium such as a CD-ROM, or can be provided to the image processing apparatus 1000 through an information communication line such as the Internet. is there. Such an aspect is often seen in portable game machines and the like. When the processing capabilities of the GPU 100 and the CPU 300 are limited as in a portable game machine, a shader program is provided in a binary code state to the image processing apparatus 1000 (portable game machine). In this case, since it is not necessary to compile the shader program in the image processing apparatus 1000 itself, the GPU 100 can load and execute the shader program stored in the external memory 200 as it is. In the case of the mode shown in FIG. 10, it is another computer 2000 that has the function of compiling the shader program. For this reason, in such an aspect, the compile processing unit 30 is provided in another computer 2000.

  On the other hand, FIG. 11 shows an example in which a shader program is provided to the image processing apparatus 1000 from another computer 2000 in the state of source code before compilation. In a processing device such as a desktop PC, a graphics shader program is generally provided as source code, and a compiled program for the GPU 100 is included in a driver program. This is because in a processing device such as a desktop PC, the GPU 100 and the CPU 300 are generally provided by different manufacturers, and it is necessary to unify both interfaces. In such an embodiment, the shader program used by the GPU 100 is converted into a binary program by a unique compile program provided by the GPU manufacturer and incorporated in the driver immediately before the drawing is executed. That is, the CPU 300 executes a compiling program for a shader bundled with the driver program, reads a shader program in a source code state stored in the external memory 200, and performs a compiling process for rewriting it into binary code. . Then, the CPU 300 provides the shader program in the binary code state to the GPU 100, and the GPU 100 executes graphics processing according to the compiled shader program. As described above, in the case of the mode shown in FIG. 11, the CPU 300 in the image processing apparatus 1000 is responsible for compiling the shader program. Therefore, in such an embodiment, the compile processing unit 30 is provided in the CPU 300 in the image processing apparatus 1000.

  As described above, since compiling for GPU is a very heavy process, it is possible to compile a shader program by a CPU immediately before drawing on a desktop PC or the like. However, in a portable game machine or the like, a shader program is often compiled and converted into a binary program in advance because the CPU capability is low or the GPU model is determined in advance. In the present invention, as shown in FIG. 10, the GPU shader program compilation process may be executed by a computer different from the image processing apparatus, and the compiled shader program may be stored in the image processing apparatus. Alternatively, as shown in FIG. 11, it may be executed immediately before drawing by a CPU provided in the image processing apparatus. That is, the present invention includes both aspects of FIG. 10 and FIG.

  As mentioned above, in this specification, in order to express the content of this invention, embodiment of this invention was described, referring drawings. However, the present invention is not limited to the above-described embodiments, but includes modifications and improvements obvious to those skilled in the art based on the matters described in the present specification.

  The present invention relates to an image processing apparatus and an image processing method for computer graphics. Therefore, the present invention can be suitably used in the computer related industry.

DESCRIPTION OF SYMBOLS 10 ... Main unit 11 ... Management part 12 ... Vertex processing part 13 ... Rasterization processing part 14 ... Fragment processing part 15 ... Texture processing part 16 ... Color update processing part 17 ... Internal memory 20 ... Sub unit 21 ... First computing unit 22 ... second computing unit 23 ... third computing unit 30 ... compile processing unit 100 ... GPU
200 ... External memory 300 ... CPU
1000: Image processing apparatus 2000: Another computer

Claims (10)

An image processing apparatus for drawing an image according to a shader program,
A main unit (10) capable of programming the arithmetic processing to be executed;
One or a plurality of sub-units having a plurality of arithmetic units that perform predetermined arithmetic processing and in which the respective arithmetic units are connected in series so that the arithmetic result of the former stage arithmetic unit is input to the subsequent stage arithmetic unit. 20),
Codes that can be executed by the subunit (20) are extracted from a shader program according to the connection pattern of the computing units constituting the subunit (20), and the extracted codes can be executed by the subunit (20). A compile processing unit (30) for rewriting the code for the subunit,
The main unit (10) causes the subunit unit (20) to execute the subunit code in accordance with the shader program that has been rewritten by the compile processing unit (30), and transmits other codes to the main unit (10). ) Is executed. A shader program includes a plurality of codes that define variables obtained from operations according to instructions, and the plurality of codes include a dependency relationship between codes in which a variable of a certain code is included in an instruction of another code. Exists,
The compile processing unit (30) can extract from the shader program that the dependency relationship between codes matches the dependency relationship between the arithmetic units constituting the subunit, and can be executed by the subunit (20). The image processing apparatus according to claim 1, wherein the sub-unit code is rewritten. When the code to be rewritten as the subunit code is extracted by the compile processing unit (30), it is assumed that all of the instruction set using the code variable is rewritten from the shader program to the subunit code. The image processing apparatus according to claim 2, which is limited to a case where extraction is possible. The image processing apparatus according to claim 1, wherein a plurality of the subunits (20) are provided, and a connection pattern of computing units constituting each subunit (20) is different. The main unit (10) has a plurality of contexts that store values of variables used in arithmetic processing, and the contents of arithmetic processing can be switched by switching the context to be used. Image processing apparatus. The image processing device according to claim 1, wherein the number of stages of the computing units constituting the subunit (20) is four. The image processing apparatus according to claim 1, wherein the computing unit constituting the subunit (20) includes an adder and a multiplier. The main unit (10) includes a processing unit that does not perform arithmetic processing simultaneously with the subunit (20), and an arithmetic unit that constitutes the processing unit and an arithmetic unit that constitutes the subunit (20) are at least The image processing apparatus according to claim 1, wherein a part is shared. An image processing apparatus for drawing an image according to a shader program,
A main unit (10) capable of programming the arithmetic processing to be executed;
One or a plurality of sub-units having a plurality of arithmetic units that perform predetermined arithmetic processing and in which the respective arithmetic units are connected in series so that the arithmetic result of the former stage arithmetic unit is input to the subsequent stage arithmetic unit. 20), and
The shader program is a code for subunits that can be executed by the subunit (20) according to a connection pattern of computing units constituting the subunit (20). It has been rewritten as
The main unit (10) causes the subunit unit (20) to execute the subunit code in accordance with the rewritten shader program, and executes the other codes in the main unit (10). . An image processing method executed by an image processing apparatus for drawing an image according to a shader program,
The image processing apparatus includes:
A main unit (10) capable of programming the arithmetic processing to be executed;
One or a plurality of sub-units having a plurality of arithmetic units that perform predetermined arithmetic processing and in which the respective arithmetic units are connected in series so that the arithmetic result of the former stage arithmetic unit is input to the subsequent stage arithmetic unit. 20), and
The image processing method includes:
Codes that can be executed by the subunit (20) are extracted from a shader program according to the connection pattern of the computing units constituting the subunit (20), and the extracted codes can be executed by the subunit (20). A compilation process to rewrite the code for the subunit,
The main unit (10) causes the subunit unit (20) to execute the subunit code in accordance with the shader program that has been rewritten in the compile processing step, and the other units execute other codes. An image processing method comprising:

Images