KR100696198B1 - Embedded system of supporting graphics operation based on programmable shader - Google Patents

Abstract

An embedded system for supporting a graphic operation based on a programmable shader is provided to enable a user to develop a program for processing 3D(Dimensional) data and transfer the developed program to graphic hardware, and make the graphic hardware process vertexes/pixels with the received program. A system memory(500) stores one or more than one program and data generated by linking with one or more than one shader program inputted through an API(Application Program Interface). The graphic hardware(300) loads and executes the program/data selected by the API among the programs/data stored in the system memory. The system memory includes more than one shader object area(510a,510b) storing shader ID information, a shader program source, a compiled code compiling the shader program source, and a parameter declared in the shader program source, and a program object area(520) storing the shader ID information of the shader programs forming the program, the program generated by linking the compile codes, and the data referred when the program is executed.

Description

Embedded system of supporting graphics operation based on programmable shader

1 is a view showing a three-dimensional data processing process having a fixed pipeline structure.

2 illustrates a memory structure of graphics hardware and system memory of an embedded system.

3 illustrates a three dimensional data process with a programmable shader structure in accordance with the present invention.

4 illustrates the role of an application program interface in using a programmable shader.

5 is a diagram illustrating a memory structure of a system memory and a graphics hardware of an embedded system according to an exemplary embodiment of the present invention.

6 is a flowchart of a method of generating a shader object area according to an exemplary embodiment of the present invention.

7 is a flowchart illustrating a method of generating a program object area according to an exemplary embodiment of the present invention.

8 is a flow chart of a memory operation method in accordance with one preferred embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

500: system memory

510a, 510b: shader object area

520 program object area

300: graphics hardware

The present invention relates to an embedded system, and more particularly, to an embedded system that supports programmable shader-based graphics operations for processing three-dimensional graphics operations for processing three-dimensional graphics operations.

OpenGL ES (Open Graphics Library Embedded System) is a cross-platform application program interface (API) for two-dimensional and three-dimensional graphics functions on embedded systems including royalty-free vehicles, equipment and portable devices. )to be. It is a subset of OpenGL (Open Graphics Liabrary), a three-dimensional graphics standard for PC environments, that provides a flexible yet powerful low-level interface between software applications and the graphics engine of hardware or software.

OpenGL ES is an embedded system such as mobile communication devices, personal digital assistants (PDAs), mobile devices such as portable multimedia players (PMPs), automotive control devices, refrigerator control devices, and factory robot control devices. It is a software solution that processes 3D graphics operations to provide 3D games and various advanced 3D graphics functions.

Graphics hardware that supports OpenGL ES is a hardware implementation of the three-dimensional algorithm provided by OpenGL ES, and is a device for processing three-dimensional graphics operations in real time. Conventional graphics hardware processed three-dimensional data according to a fixed algorithm.

FIG. 1 is a diagram illustrating a 3D data processing process having a fixed function pipeline structure, and FIG. 2 is a diagram illustrating a memory structure of graphics hardware and a system memory of an embedded system.

Referring to FIG. 1, 3D data is processed according to OpenGL ES 1.x.

The graphics hardware 200 processes the 3D data in real time by the application program interface 100 and outputs the result to the frame buffer 290. The 3D data may be vertex data including a position, a normal vector, a color, a texture coordinate, and the like.

The application program interface 100 is an interface for specifying vertex data and state data and giving a rendering command. Rendering refers to a process or a technique for creating realistic images while considering shadows and color concentrations that appear differently depending on external information such as shape, position, and lighting in a flat picture. The shape of the object to be expressed is divided into triangular polygon sets, and the three vertex data constituting the polygon is processed through a process to be described later to create a three-dimensional image.

Vertex data is stored in Vertex Buffer Objects 205. The primitive processing module 210 generates points to apply transform and lighting from three-dimensional data stored in the vertex buffer object 205. Each point is the three vertices of the triangle that make up the polygon.

The transform and light effects module 215 determines the location on the screen through matrix operations for the points generated by the primitive processor module 210 and the brightness of the points according to the lighting model (eg, phong illlumination model, etc.). Determine.

The primitive assembly module 220 collects the converted points and the lighting calculations to form a triangle. Rasterizer 225 determines the pixels that the triangle occupies on the screen.

The texture environment module 230 obtains color information from the texture image and determines the color of each pixel according to the designated state information, and the color sum module 235 diffuses and reflects the color information. specular) color values, and the fog module 240 mixes fog colors as the distance increases. This ensures that each pixel has a final color.

After that, each pixel is not processed if the alpha value (transparency) is less than the user-specified standard. The distance stencil does not process pixels farther than the distance stored in the depth buffer. ), A color buffer blend that blends the color in the color buffer with the color of the current pixel to determine what color is stored in the color buffer, and converts it to match the bit depth of the current screen. After dithering, the data is stored in a frame buffer 290.

Referring to FIG. 2, system memory 10 is used by application program interface 100. The system memory 10 may be divided into a conversion state region 13, a color state region 16, an illumination state region 19, and the like. In the transformation state area 13, various parameters such as a model view matrix MODELVIEW_MATRIX and a projection matrix PROJECTION_MATRIX are stored. Each state area 13, 16, 19 of the system memory 10 can be changed by a user. Each parameter determines conversion state data, color state data, and illumination state data.

The values of various parameters stored in the system memory 10 are stored in the state area 63 of the graphics hardware 60 in accordance with the designation of the application program interface 100, and the respective modules shown in FIG. Each operation is performed as shown.

Here, the operation in the transform and light effect module 215 is determined by a first fixed function (eg, a matrix for operation, etc.) stored in the transform and light effect area 66. Then, the operations in the texture calculation module 230, the color sum module 235, and the fog effect module 240 are determined by the second fixed function stored in the texture calculation / color sum / fog effect area 69.

In the transformation and lighting effect module 210, transformations can be represented only by matrices, and the equations for lighting effects are already fixed per graphics hardware. The user can only change the parameters stored in each state area 13, 16, 19 of the system memory 10.

Therefore, the user is limited to expressing 3D graphics using only functions provided by the graphics hardware. In addition, if the user wants to obtain another transformation and lighting effect, there is a problem of changing the graphics hardware itself or changing the first function stored in the transformation and lighting effect area 66. In addition, the matrix operation only mirrors each point on the plane to a point on another plane, and there is a problem that symmetry is not possible with a new plane that is not a plane (for example, when a flag is fluttering or water is waving).

Therefore, in OpenGL ES 2.0, you can develop a program that processes three-dimensional data, send it to graphics hardware, and in graphics hardware, use a program that you send to process vertices and pixels. We propose a programmable shader structure. Graphics hardware that supports OpenGL ES 2.0 requires a new memory structure because the memory structure shown in FIG. 2 is not suitable.

Accordingly, the present invention provides an embedded system that allows a user to develop a program for processing three-dimensional data, transmit the program to graphics hardware, and process graphics and vertices using the program transmitted by the user.

In addition, the present invention provides an embedded system that can improve the speed of the three-dimensional graphics operation by eliminating the basic complex lighting calculation or coordinate conversion calculation if it is not necessary.

Other objects of the present invention will be readily understood through the following description.

In order to achieve the above object, according to an aspect of the present invention, in an embedded system that supports a programmable shader-based graphics operation, input via an application program interface (Application Program Interface) A system memory for storing one or more programs and data generated by linking one or more shader programs; And a graphics hardware for loading and executing a program and data selected by the application program interface among the one or more programs and data.

Preferably, the system memory includes shader identification information for identifying the shader program, a shader program source that is a source of the shader program, compiled code that is a result of the shader program source being compiled, and the shader program source. One or more shader object areas that store declared constants; And the program generated by linking each of the shader identification information of the one or more shader programs constituting the program and the compilation code of the one or more shader object regions corresponding to the shader identification information, and data referenced when the program is executed. It may include a program object area for storing the.

Here, the shader object area may include an identification area storing the shader identification information; A source area for storing the shader program source; A code area for storing the compiled code; And a constant region for storing the constant.

The identification area may be an integer storage area, and the shader identification information may be an integer value that does not overlap each other between the shader object areas. The source region may be a string storage region, and the shader program source may be written in a high-level language. The code region may be a binary data type storage region, and the compiled code may be machine code. The constant region may be a real array storage region.

In addition, the program object area may include a shader identification area storing one or more shader identification information; A program area for storing the program; And a data area for storing the data.

The shader identification area may be an integer array type storage area, and the shader identification information may be an integer value that does not overlap between shaders. The program area may be a binary data type storage area, and the program may be machine code in which the compiled code is linked. The data area may be a real array storage area, and the data may include constants of the shader object area.

Here, the program object area may include at least two program identification areas for storing program identification information which is an integer value which is two or more and does not overlap each other between the program object areas.

Advantageously, said graphics hardware further comprises: an application program area for loading a program selected from said one or more programs; A data area for loading data corresponding to the selected program; And an output data area for outputting a result of performing a predetermined operation by the program loaded in the application program area to the data loaded in the data area.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the embedded system and memory operation method for supporting the programmable shader-based graphics operation in accordance with the present invention. In describing the present invention, if it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Numbers (eg, first, second, etc.) used in the description of the present specification are merely identification symbols for sequentially distinguishing identical or similar entities.

3 is a view showing a three-dimensional data processing process having a programmable shader structure according to the present invention.

Programmable shader structures are proposed in OpenGL ES 2.0. The transform and light effect module 215 shown in FIG. 1 is a vertex shader 310, and the texture calculation module 230, the color sum module 235, and the fog effect module 240 are fragment shaders. ; 320). That is, the graphics hardware 300 according to the present invention, unlike the graphics hardware 200 illustrated in FIG. 1, loads the vertex shader 310 and the fragment shader 320. Here, the vertex shader 310 and the fragment shader 320 have a shader structure that the user can program in a high-level language. The replaced vertex shader 310 and the fragment shader 320 may freely express 3D graphics by executing a user-developed program instead of the fixed function of the graphics hardware 300.

The vertex shader 310 receives vertex data positioned in three dimensions and outputs position, color, and additional information on the screen of each vertex to which an effect by a user is applied. Here, the vertex data is three-dimensional data, and includes three-dimensional positions, colors, normal vectors, texture coordinates, and the like.

The user loads the vertex shader 310 in the predetermined area on the graphics hardware 200 via the application program interface 100 and stores the vertex data in the vertex buffer object 205. The processing of the vertex data is then performed by the loaded vertex shader 310.

The fragment shader 320 determines the color of the fragment using information such as the color and texture coordinates transmitted to the fragment. The fragment is a collection of pixels determined by the rasterizer 225. The rasterizer 225 determines pixels included in a triangular plane having three vertices of each pixel corresponding to the position of each vertex determined by the vertex shader 310 on the screen on which the 3D image is to be expressed.

The fragment shader 320 determines the color of the fragment using colors, texture coordinates, and the like included in the vertex data corresponding to three vertices constituting the fragment.

4 illustrates the role of an application program interface in using a programmable shader.

Referring to FIG. 4, a user creates a 3D data processing program 410 that processes 3D data, that is, vertex data. The three-dimensional data processing program 410 may be programmed such that various effects are applied for the location transformation and / or lighting effects of the vertices.

For example, there is vertex data on a plane representing a flag. Conventionally, only the matrix operation is used, so that the plane is transformed into another plane only, and thus the three-dimensional shape of the flag fluttering could not be realized. However, if the vertex shader 310 is used to transform each vertex data by a sine function, cosine function, log function, and exponential function, a complex form of conversion It is possible to implement a three-dimensional shape in the form of a flag fluttering.

The application program interface 410 translates the 3D data processing program 410 into a program that can be processed by the graphics hardware 420. The form that the graphics hardware 420 can process is preferably machine code composed of binary data. The application program interface 410 is based on OpenGL ES 2.0, but can of course be applied to the extension of OpenGL ES later.

When using OpenGL ES 2.0, two kinds of objects are used, shader objects and program objects, to use programmable shaders. The shader object represents a vertex shader or fragment shader module and stores the shader program source and its compilation result. A program object represents a shader program of a single execution unit consisting of vertex shaders and fragment shaders. Linking one or more shader objects forming a shader program generates and stores execution code of a single execution unit.

The application program interface 410 is composed of functions for creating and managing shader objects and program objects, and for delivering data and instructions, that is, shader programs suitable for the graphics hardware 300.

The graphics hardware 300 receives a shader program corresponding to a vertex shader and / or a fragment shader from the application program interface 410, and executes each shader to perform a rendering operation. Graphics hardware 300 according to the present invention is preferably a programmable shader-based hardware that can receive and execute each shader program. In addition, the graphics hardware 300 receives data necessary for executing the shader program from the application program interface 410.

Programmable shader-based graphics hardware 300, application program interface 410, etc., which can produce a user-programmed effect by the above process, require a different memory structure than the system memory 10 shown in FIG. do. 5 will be described with reference to the drawings the structure of the system memory and the memory operation method that can be used in the graphics hardware 300, the application program interface 410 according to the present invention.

5 is a diagram illustrating a memory structure of the system memory 500 and the graphics hardware 300 of the embedded system according to an exemplary embodiment of the present invention.

The embedded system includes system memory 500 and graphics hardware 300. In the present invention, the embedded system refers to a mobile device such as a mobile communication terminal, a personal digital assistant (PDA), a portable multimedia terminal (PMP), a vehicle control device, a refrigeration degree control device, a factory robot control device, and the like.

The system memory 500 is utilized by the application program interface 410 to store the translated and translated code for the 3D data processing program 400 to be processed by the graphics hardware 300.

The system memory 500 includes one or more shader object areas 510a and 510b and a program object area 520. In FIG. 5, it is assumed that there are two shader object regions 510a and 510b, but the present invention is not limited thereto and there may be several shader object regions. The shader object regions 510a and 510b have the same structure. Hereinafter, the shader object regions 510a will be described.

The shader object area 510a includes an identification area 512a, a source area 514a, a code area 516a, and a constant area 518a.

The identification area 512a stores shader identification information for distinguishing each shader object. The shader identification information is an integer value that does not overlap each other to distinguish between shader objects. Therefore, the identification area 512a is preferably an integer storage area. If there is only one shader program, no distinction is required, and thus the identification area 512a may not be allocated in the shader object area 510.

The source region 514a stores a shader program source, which is a source of a shader program corresponding to each shader object. Shader program sources can be written in a high-level language that is easy for the user to work with. The shader program source may be the above-described three-dimensional data processing program 400 itself, or may be one of a plurality of shader programs constituting the three-dimensional data processing program 400. Therefore, the source area 514a is preferably a string type storage area capable of storing shader program source data written in a high level language.

The code region 516a stores compiled code that is a result of compiling a shader program source stored in the source region 514a. The compiled code is machine-readable machine code, that is, binary data, so that the shader program source stored in the source area 514a can be executed in a computer, program execution hardware, or digital device. Therefore, the code area 516a is preferably a binary data type storage area.

The constant region 518a stores constants declared in the shader program source and referenced later when the shader program is executed. Constants can be natural numbers, integers, real numbers, and the like, but are preferably floating types that can cover the widest range. Therefore, the constant region 518a is preferably a real array storage region so that several constants are stored.

Here, the code region 516a and / or the constant region 518a store the result of compiling the machine-readable contents of the shader program source stored in the source region 514a. Thus, code region 516a and / or constant region 518a may not be preallocated but later allocated so that the shader program source can be compiled and the result stored when the corresponding shader object is called to construct the program. .

The shader object area 510a may be a storage area for the vertex shader 310 for processing the vertex data described above or the fragment shader 320 for determining the color of the fragment. In addition, it may be a storage area for a shader that replaces a module of the graphics hardware 300 having a fixed function by an extension of OpenGL ES.

The program object area 520 stores a program executed in the graphics hardware 300.

A program consists of one or more shader programs. When only the vertex shader 310 is replaced, only the vertex shader program is configured, and when the fragment shader 320 is replaced only, the fragment shader program is configured, and the vertex shader 310 and the fragment shader 320 are both replaced. Consists of a link between the vertex shader program and the fragment shader program. Here, one or more vertex shader programs and / or fragment shader programs may exist according to the type of effect desired by a user.

The program object area 520 includes a program identification area 522, a shader identification area 524, a program area 526, and a data area 528.

The program identification area 522 stores program identification information for distinguishing each program object. The program identification information is an integer value that does not overlap each other to distinguish between program objects. Therefore, the program identification area 522 is preferably an integer storage area.

The shader identification area 524 stores shader identification information that can distinguish one or more shader objects 510a or 510b corresponding to one or more shader programs constituting a program corresponding to the program object area 520. As described above, the shader identification information is assigned an integer value which is not duplicated to distinguish between the shader objects. The shader identification area 524 is preferably an integer array type storage area.

The program area 526 stores a program, which is machine code generated by linking compiled codes stored in the code area 516a or 516b of each shader object area 510a or 510b constituting an executable program in the graphics hardware 300. do. The program is binary data readable by the machine, and the program area 524 is preferably a binary data type storage area. The shader identification information stored in the shader identification area 524 of the program object area 520 accesses the shader object area 510a or 510b to read and link the compiled code stored in the code area 516a or 516b.

The data area 528 stores constants stored in the constant area 518a or 518b of each shader object area 510a or 510b constituting the program and / or data that the program refers to when operating in the graphics hardware 300. do. Constants or data can be natural numbers, integers, real numbers, etc., but are preferably real numbers that can cover the widest range. Therefore, the data area 528 is preferably a real array storage area so that several constants or data can be stored. The shader identification information stored in the shader identification area 524 of the program object area 520 is accessed to read the constants stored in the constant area 518a or 518b by accessing the corresponding shader object area 510a or 510b.

Graphics hardware 300 includes a data area 552, an application area 554, and an output data area 556. The output data area 556 stores a result of an operation performed by a program loaded in the application program area 554 and outputs the result to the outside.

The data area 552 loads constant and / or data stored in the data area 528 of the program object area 520 selected from one or more program object areas stored in the system memory 500. The constants and / or data need to be stored in the memory of the graphics hardware 300 because they are values referenced by the graphics hardware 300 to execute the program.

The application program area 554 loads a program, that is, machine code, stored in the program area 526 of the program object area 520 selected from one or more program object areas stored in the system memory 500. The loaded machine code then refers to the value stored in the data area 552 on the graphics hardware 300 and performs a predetermined operation (eg, rendering).

6 is a flowchart of a method of generating a shader object region according to an exemplary embodiment of the present invention, FIG. 7 is a flowchart of a method of generating a program object region according to an exemplary embodiment of the present invention, and FIG. A flowchart of a memory operation method according to a preferred embodiment.

Referring to FIG. 6, in operation S610, shader identification information and shader object area 510a space corresponding to the shader program source input through the application program interface 100 are generated in the system memory 500. The shader identification information should not overlap with the identification information of the shader object area 510a previously existing on the system memory 500. When the shader object region 510a is generated, an identification region 512a, a source region 514a, a code region 516a, and a constant region 518a are automatically allocated therein. In the OpenGL ES 2.0 application interface, this is glCreateShader.

In operation S620, the shader program source input to the source region 514a is stored. In the OpenGL ES 2.0 application program interface, this is glShaderSource.

In operation S630, the compiled code and the constant generated by compiling the shader program source are stored in the code region 516a and the constant region 518a, respectively. In the OpenGL ES 2.0 application interface, this is glCompileShader.

One shader object area 510a corresponding to a specific shader program is generated through the above-described steps S610 to S630. A plurality of shader object areas 510a or 510b may be generated by repeating steps S610 to S630 for each shader program source input by a user.

Referring to FIG. 7, in operation S710, a space of program identification information and a program object area 520 corresponding to a program having a specific three-dimensional effect is generated in the system memory 500 according to a command of the application program interface 100. The program identification information should not overlap with the identification information of the program object area 520 previously existing on the system memory 500. When the program object area 520 is generated, the program identification area 522, the shader identification area 524, the program area 526, and the data area 528 are automatically allocated therein. In the OpenGL ES 2.0 application program interface, this is glCreateProgram.

In operation S720, the shader identification information of the shader object area 510a or 510b in which the shader program constituting the program is stored is stored in the shader identification area 524. In the OpenGL ES 2.0 application interface, this is glAttachShader.

In operation S730, the program generated by linking the compiled code stored in the code area 516a or 516b of the shader object area 510a or 510b corresponding to the shader identification information stored in the shader identification area 524 in the program area 526 is obtained. Save it. In the OpenGL ES 2.0 application program interface, this is glLinkProgram.

In operation S740, constants and / or programs stored in the constant area 518a or 518b of the shader object area 510a or 510b corresponding to the shader identification information stored in the shader identification area 524 in the data area 528 may be operated. Stores data that is referenced at time. In the OpenGL ES 2.0 application interface, glGetUniformLocation and glUniform {1234} {if} [v].

One program object area 520 corresponding to a specific program is generated through the above-described steps S710 to S740. The plurality of program object areas 520 may be generated by repeating steps S710 to S740.

Referring to FIG. 8, in operation S810, one or more shader object regions 510a or 510b are generated. This is illustrated in detail in FIG. 6 as described above.

In operation S820, one or more program object regions 520 are generated. This is also shown in detail in FIG. 7 as described above.

In operation S830, a program object to be used for a specific rendering is selected. The selection of the program object is possible by an operation such as writing the program identification information to a predetermined address of the system memory 500 or the graphics hardware 300. In the OpenGL ES 2.0 application program interface, this is glUseProgram.

The program of the program object area 520 corresponding to the program object selected in step S840 is loaded into the application program area 554 of the graphics hardware 300. In operation S850, constants and / or data of the program object area 520 corresponding to the selected program object are loaded into the data area 552 of the graphics hardware 300. Steps S840 and S850 may be performed in any order and may be performed simultaneously.

In operation S860, the graphics hardware 300 starts rendering using the loaded program and data. In the OpenGL ES 2.0 application interface, this is glDrawArray, glDrawElement, and so on.

As described above, the embedded system according to the present invention may develop and transmit a program for processing 3D data to graphics hardware, and the graphics hardware may process vertices and pixels using a program transmitted by the user.

In addition, the present invention can improve the speed of the three-dimensional graphics operation by omitting this if the basic complex lighting calculation or coordinate transformation operation is not necessary.

In addition, as the sine function, cosine function, logarithmic function, and exponential function can be applied in the vertex transformation process, various planes can be represented using vertex data on a plane.

In addition, by developing the structure of system memory according to the OpenGL ES 2.0 high-level shader application program interface, it is possible to apply to extensions of OpenGL ES or shaders that can replace fixed functions of graphics hardware in addition to vertex shaders and fragment shaders. .

Although the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art to which the present invention pertains without departing from the spirit and scope of the present invention as set forth in the claims below It will be appreciated that modifications and variations can be made.

Claims (11)

In an embedded system that supports graphical shader-based graphics operations, A system memory configured to store one or more programs and data generated by linking one or more shader programs input through an application program interface; And Embedded graphics comprising graphics hardware to load and execute a program and data selected by the application program interface from among the one or more programs and data. The system of claim 1, wherein the system memory is At least one shader identification information for identifying the shader program, a shader program source that is a source of the shader program, compiled code that is a result of the shader program source being compiled, and a constant declared in the shader program source Shader object area; And The program generated by linking the shader identification information of the one or more shader programs constituting the program and the compilation code of the one or more shader object regions corresponding to the shader identification information, and data referred to when the program is executed. Embedded system that includes a program object area for storing. The method of claim 2, wherein the shader object area is An identification area for storing the shader identification information; A source area for storing the shader program source; A code area for storing the compiled code; And Embedded system including a constant region for storing the constant. The method of claim 3, And the identification area is an integer storage area, and the shader identification information is an integer value that does not overlap each other between the shader object areas. The method of claim 3, And the constant region is a real array storage region. The method of claim 2, wherein the program object area is A shader identification area for storing one or more shader identification information; A program area for storing the program; And Embedded system comprising a data area for storing the data. The method of claim 6, The shader identification area is an integer array type storage area, and the shader identification information is an integer value that does not overlap between each shader. The method of claim 6, The program region is a binary data type storage region, and the program is a machine language code to which the compiled code is linked. The method of claim 6, And the data area is a real array storage area, and the data includes constants of the shader object area. The method of claim 6, And at least two program object regions, and a program identification region for storing program identification information which is an integer value which does not overlap each other between the program object regions. The system of claim 1, wherein the graphics hardware is An application program area for loading a program selected from the one or more programs; A data area for loading data corresponding to the selected program; And And an output data area configured to output a result of performing a predetermined operation by the program loaded in the application program area with reference to the data loaded in the data area.

Images