CN114663272A

CN114663272A - Image processing method and electronic equipment

Info

Publication number: CN114663272A
Application number: CN202210163839.1A
Authority: CN
Inventors: 孙黎; 刘金晓; 尹朝阳; 麦倩屏
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2022-02-22
Filing date: 2022-02-22
Publication date: 2022-06-24
Anticipated expiration: 2042-02-22
Also published as: CN114663272B

Abstract

The embodiment of the application discloses an image processing method and electronic equipment, relates to the field of image processing, and can simplify shading operation and reduce running time and power consumption overhead by running a corresponding simplified shader. The specific scheme is as follows: and determining an original shader according to the rendering instruction stream, matching the original shader, and determining a simplified shader corresponding to the original shader. The first shader corresponds to a fifth shader, and the fifth shader is a shader stored in the electronic device and has the same code length as the second shader. And binding the simplified shader to the first program indicated by the rendering instruction stream, so that when the first program is called subsequently, the simplified shader can be called to execute the corresponding high shading operation.

Description

Image processing method and electronic equipment

Technical Field

The present application relates to the field of image processing, and in particular, to an image processing method and an electronic device.

Background

With the development of electronic devices, images to be displayed become finer and finer. In the process of rendering a fine image, as the model in the image needs to be accurately colored, the running time and the power consumption of the shader are increased more and more. This may cause frame loss, jamming, etc. during image display.

Disclosure of Invention

The embodiment of the application provides an image processing method and an electronic device, which can simplify shading operation by operating a corresponding simplified shader when the shader (such as a to-be-simplified shader) needs to be called, and reduce operation time consumption and power consumption overhead. The embodiment of the application further provides a specific determination mode of the simplified shader and a simplified shader matching mode in the running process of the application program, so that replacement of the shader and the simplified process are achieved.

In order to achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

in a first aspect, an image processing method is applied to an electronic device, where the electronic device is installed with an application program, and the electronic device executes a rendering instruction stream issued by the application program, and the method includes: the electronic device calls a Graphics Processing Unit (GPU) to execute a shading operation through a first shading instruction, wherein a shader bound to a first item (program) indicated by the first shading instruction comprises a first shader, the first shader is different from a second shader, the second shader is a shader bound to the first program indicated by a second shading instruction, and the second shading instruction is included in the rendering instruction stream. The first shader corresponds to a fifth shader, and the fifth shader is a shader stored in the electronic device and has the same code length as the second shader.

Based on this scheme, an example of a scheme for simplifying the coloring operation is provided. In this scheme, the application program may instruct the electronic device to perform rendering operations including shading operations through a rendering instruction stream. In response to the stream of rendering instructions, the electronic device can perform a reduction operation using a corresponding reduction shader. For example, an application may instruct a shading operation to be performed using a second shader bound to a first program. Correspondingly, the electronic device may send a second shading instruction to the GPU instructing the GPU to perform the shading operation. The second shading instruction may instruct the GPU to perform a shading operation using a simplified shader bound to the first program (e.g., the first shader). In some embodiments, the first shader may be a simplified shader corresponding to the second shader. For example, a first shader may implement the same or similar shading functionality as a second shader, while the first shader may have a smaller amount of code than the second shader. Therefore, the GPU can normally complete the corresponding shading operation by operating the first shader according to the second shading instruction, and meanwhile, can obtain lower time consumption and power consumption compared with the operation of the second shader. In the scheme, a scheme for determining the simplified shader corresponding to the second shader is also provided. For example, a fifth shader corresponding to the second shader is determined according to the length matching and the code matching, and then the simplified shader corresponding to the fifth shader, that is, the first shader may be used as the simplified shader of the second shader. The correspondence between the fifth shader and the first shader may be stored in the electronic device in a simplified correspondence.

In one possible design, the method further includes: the electronic device obtains the second shader based on the stream of rendering instructions. The electronic device determines the first shader based on the second shader, the first shader corresponding to the second shader. Based on the scheme, an example of a scheme for determining a first shader is provided. In this scenario, the electronic device may obtain a shader (e.g., an original shader, or a second shader) to be used by the application from the rendering instruction stream. The electronic device can also determine a corresponding first shader from the second shader. Therefore, the determined first shader can correspond to the second shader, so that the shading operation which is required by the application program can be realized by running the first shader, and meanwhile, the time consumption and the power consumption are low.

In one possible design, the method further includes: the electronic device replaces the second shader with the first shader. Based on the scheme, a scheme implementation for performing shading operations using a first shader is provided. In this scheme, the first shader is used to replace the second shader, so that when the application program needs to call the second shader, the GPU can call the corresponding first shader to perform the shading operation.

In one possible design, the replacing the second shader with the first shader is: the electronic device binds the first shader to the first program. Based on the scheme, a specific implementation of shader replacement is provided. It will be appreciated that when an application calls a shader, it can be implemented by calling the program to which the shader is bound. In this example, a first shader may be bound to a first program. Therefore, when the application program needs to call the second shader through the first program, the electronic device can correspondingly call the first shader currently bound on the first program, and then the first shader is used for performing more efficient shading operation.

In one possible design, this further includes: and binding the storage address of the binary code of the first shader on the first program. Based on the scheme, the method comprises the following steps of,

in one possible design, the method further includes: the identification of the first shader is bound to the first program. Based on the scheme, a specific implementation of shader replacement is provided. In this scenario, the identity of the first shader (e.g., the first shader's ID) may be bound to the first program. Therefore, when the application program issues the instruction to call the first program, the first shader (such as a storage address) bound to the first program can be found according to the ID of the first shader bound to the first program, and then the shading operation is performed according to the found first shader.

In one possible design, before the binding the identification of the first shader to the first program, the method further includes: unbinding the second shader from the first program based on the identification of the second shader. Based on the scheme, a scheme implementation of binding the first shader is provided. A second shader may be bound to the first program before the first shader is bound. Then, if the first shader is directly bound to the first program, the binding of the first shader to the first program may not be achieved. In this example, the first program may be obtained without any shader bound by unbinding the second shader from the first program. Then, a first shader can be bound to a first program that is not bound to any shader, so that only the first shader is bound to the first program.

In one possible design, the method further includes: and compiling the source code of the first shader into binary code. The binary code of the first shader is stored. Based on the scheme, the implementation scheme in the calling process of the first shader is provided. In this example, the source code of the first shader may be compiled to obtain the corresponding binary code prior to performing the shading operation using the first shader. Thus, when the first shader is used, the binary code can be directly called and run, and therefore efficiency is improved.

In one possible design, before the electronic device binds the first shader to the first program, the method further includes: the electronic equipment acquires the identification of the first program based on the rendering instruction stream. Based on the scheme, the scheme implementation of the electronic device binding simplified shader is provided. It should be appreciated that in order to properly implement the replacement of the second shader by the first shader, it is necessary to explicitly bind the first program of the first shader. In this example, different programs may be identified by their identities (e.g., program IDs). The electronic device may obtain the identifier of the first program according to an instruction in the rendering instruction stream, so as to perform subsequent replacement accurately.

In one possible design, the electronic device obtains code for a plurality of third shaders, the code for the plurality of third shaders including code for a first shader, the determining the first shader based on the code for the second shader includes determining the first shader code based on the code for the second shader. Based on the scheme, an example of a scheme is provided for determining a first shader based on a second shader. Before the first shader needs to be acquired, the electronic device may acquire the code for multiple simplified shaders (i.e., the third shader). The third shaders may include a shader corresponding to (or called as a match) the code of the second shader, and the matched shader may be the first shader.

In one possible design, the electronic device stores codes of a plurality of third shaders, and the obtaining the codes of the plurality of third shaders includes: and reading the codes of the plurality of third shaders from the address for storing the codes of the third shaders. Based on the scheme, an example of a scheme for acquiring codes of a plurality of third shaders is provided. Code for a plurality of third shaders may be stored in the electronic device. Thus, when the first shader needs to be determined by using a plurality of third shaders, the plurality of third shaders can be read from corresponding storage addresses. In other embodiments, part or all of the code of the third shaders may also be obtained from the cloud, or may be determined by the electronic device itself.

In one possible design, the obtaining the second shader based on the rendering instruction stream includes: and acquiring the identification and the storage address of the second shader based on the rendering instruction stream, wherein the storage address stores the code of the second shader, and the code comprises source code or binary code. Based on the scheme, a specific scheme example for acquiring the second shader is provided. In this example, the identifier and the storage address of the shader bound to the first program (i.e., the second shader) may be obtained according to the rendering instruction stream. The storage address may be an address for storing a source code of the second shader, or an address for storing a binary code of the second shader. For example, the ID and the storage address of the second shader can be determined by determining the shader resource bound to the first program according to the ID of the first program.

In one possible design, the electronic device further stores codes of a plurality of fourth shaders, the codes of the plurality of fourth shaders correspond to the codes of the plurality of third shaders one to one, and the codes of the plurality of fourth shaders include source codes or binary codes. Based on this scheme, a specific example of a scheme for determining the first shader is provided. For example, the fourth shader (i.e., the shader to be simplified) may have a one-to-one correspondence with the simplified third shader. Then, the code of the fourth shader can determine the first shader meeting the requirements in the third shader.

In one possible design, the codes of the fourth shaders include a fifth shader, the fifth shader is matched with the second shader, and the fifth shader corresponds to the first shader. The determining the first shader based on the second shader includes: and acquiring the code of the second shader based on the storage address of the second shader. And matching codes of the second shader with codes of the plurality of fourth shaders to obtain the fifth shader. And acquiring the first shader through the fifth shader. Based on this approach, a specific example of an approach is provided for determining a first shader that matches a second shader. In this example, a fifth shader that matches the second shader may be found among the plurality of fourth shaders based on the code of the second shader. The fifth shader may have a simplified shader corresponding to one of the third shaders. The simplified shader corresponding to the fifth shader may be the first shader to be determined.

In one possible design, the code of the fifth shader matches the code of the second shader by: the code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader. Based on the scheme, a specific matching mechanism is provided. Under the condition of being in line with the matching mechanism, the matching can be considered to be successful, and the corresponding matching item can be used as a matching result. It should be noted that, in this example, whether the code of the shader performing the matching (e.g., the code of the second shader, the code of the fourth shaders, etc.) is the source code or the binary code, the corresponding shaders meeting the requirement may be screened according to the matching mechanism.

In one possible design, the first shader and the second shader are of the same type. Based on this approach, a corresponding definition of the first shader and the second shader is provided. Therefore, after the first shader is used for replacing the second shader, correct coloring operation can be guaranteed.

In one possible design, the first shader and the second shader are both vertex shaders, or the first shader and the second shader are both fragment shaders. Based on this scheme, two types of examples of the first shader and the second shader are provided. In other embodiments, the first shader and the second shader may also be other types of shaders different from the vertex shader or the fragment shader.

In one possible design, the rendering instruction stream includes a first instruction and a second instruction, the first instruction is used for indicating the identifier of the first program, and the second instruction is used for indicating the shader bound on the first program. The electronic equipment comprises an interception module and an acquisition module, and the method further comprises the following steps: the interception module acquires the first instruction and the second instruction. The interception module sends the first instruction and the second instruction to the acquisition module. The obtaining module obtains the identifier of the first program according to the first instruction. The obtaining module obtains the identifier and the storage address of the second shader according to the second instruction. Based on the scheme, an example for realizing the scheme of the embodiment of the application based on modularization is provided. In this example, the interception module may be configured to intercept the instruction, such as may be configured to intercept the first instruction and the second instruction. For other instructions in the rendering instruction stream, the interception module may call back directly to the graphics library to implement native logic.

In one possible design, the electronic device further includes a matching module, and the method further includes: and the acquisition module sends the identifier and the storage address of the second shader to the matching module. Based on the scheme, the obtaining module can send the obtained identifier and storage address of the second shader to the matching module, so as to facilitate subsequent operations.

In a possible design, the electronic device further includes a memory, where codes of a plurality of third shaders are stored in the memory, and the codes of the plurality of third shaders include the code of the first shader. The memory also stores a plurality of fourth shader codes, and the fourth shader codes correspond to the third shader codes one by one. The code of the plurality of fourth shaders comprises source code or binary code. The method further comprises the following steps: the matching module reads the codes of the fourth shaders from the memory. Based on the scheme, the matching module can read a plurality of to-be-simplified shaders (such as a fourth shader) stored in the memory, so that the to-be-simplified shaders corresponding to the codes of the second shader are determined by subsequent matching, and then the corresponding first shader is determined from the plurality of third shaders.

In one possible design, the codes of the plurality of fourth shaders include a code of a fifth shader, the fifth shader is matched with the second shader, the fifth shader corresponds to the first shader, and the method further includes: and the matching module reads the code of the second shader according to the storage address of the second shader. The matching module matches the codes of the second shader with the codes of the plurality of fourth shaders to obtain the fifth shader. The matching module obtains the first shader through the fifth shader. Based on the scheme, the matching module can match the codes of the plurality of shaders to be simplified according to the codes of the second shader. The match may be a source code match or a hash value match based on the source code. The match may also be a binary match or a hash value match of the binary-based code. Through the matching mechanism, a shader to be simplified, such as a fifth shader, corresponding to the second shader can be determined.

In one possible design, the electronic device further includes a replacement module, and the method includes: the replacement module binds the memory address of the binary code of the first shader to the first program. Based on this scheme, the replacement module may bind the storage address of the simplified shader corresponding to the second shader (i.e., the first shader) to the first program. For example, the replacement module may replace a pointer indicating a memory address of the binary code of the first shader onto the first program so that the binary code of the first shader may be directly called by calling the first program.

In one possible design, the code is source code. The replacement module unbinds the second shader from the first program, and the replacement module binds the identification of the first shader to the first program. Based on the scheme, the matching module can perform source code-based matching according to the source codes of the second shaders and the obtained source codes of the fourth shaders. The replacement module may then perform a source code based replacement operation. For example, through matching, the replacement module may know, from the matching module, a storage address and an ID of a source code of a first shader that is successfully matched, so that the replacement module may bind the ID of the first shader to the first program, and since a unique mapping relationship exists between the ID of the first shader and the storage address of the source code of the first shader, a code of the first shader may be smoothly obtained when the first program is called.

In one possible design, the electronic device is further provided with a compiling module, and the method further includes: the replacing module or the obtaining module sends the storage address of the first shader to the compiling module, and the compiling module compiles the code of the first shader to obtain the binary code of the first shader. The compiling module stores the binary code of the first shader in the memory. Based on the scheme, on the basis of replacing source code information (such as the ID of the first shader) on the first program, the source code of the first shader may also be compiled, so as to facilitate subsequent quick invocation of the first shader.

In one possible design, the method further includes: the compiling module sends the storage address of the binary code of the first shader to the replacing module. Based on the scheme, the compiling module can send the address of the binary code of the first shader to the replacing module after compiling, so that the replacing module can bind the binary information to the first program.

In a possible design, the first instruction and the second instruction are the same, the first instruction is used to indicate an identifier of the first program and a shader bound to the first program, and the obtaining module obtains the identifier of the first program and an identifier and a storage address of the second shader according to the first instruction. Based on the scheme, a scheme example of the relationship between the first instruction and the second instruction is provided. For example, the first instruction may be the same instruction or multiple instructions as the second instruction, and the ID of the first program may be confirmed by the instruction, and may also be used to confirm (the ID and/or the storage address of) the shader bound to the first program.

In a second aspect, an electronic device is provided that includes one or more processors and one or more memories; one or more memories coupled with the one or more processors, the one or more memories storing computer instructions; the computer instructions, when executed by the one or more processors, cause the electronic device to perform the image processing method of the first aspect and any of the various possible designs as described above.

In a third aspect, a chip system is provided, where the chip system includes an interface circuit (or referred to as a communication interface) and a processor; the interface circuit and the processor are interconnected through a line; the interface circuit is used for receiving signals from the memory and sending signals to the processor, and the signals comprise computer instructions stored in the memory; when the processor executes the computer instructions, the system-on-chip performs the image processing method of the first aspect and any of the various possible designs as described above.

In a fourth aspect, there is provided a computer readable storage medium comprising computer instructions which, when executed, perform the image processing method of the first aspect and any one of the various possible designs as described above.

In a fifth aspect, there is provided a computer program product comprising instructions which, when run on a computer, enable the computer to perform the image processing method of the first aspect and any of the various possible designs as described above, in accordance with the instructions.

A sixth aspect provides an image processing method, where the method is used to obtain a simplified correspondence, where the simplified correspondence is stored in an electronic device, and the simplified correspondence is used for determining, by the electronic device, a corresponding first shader according to a second shader indicated by a second shading instruction issued by an application program, where the first shader is different from the second shader. The method comprises the following steps: and determining at least one to-be-simplified shader by all rendering instruction streams issued in the running process of the application program of the electronic equipment, wherein the at least one to-be-simplified shader comprises the second shader. The electronic equipment determines an abstract syntax tree of the shader to be simplified according to the code of the shader to be simplified. And the electronic equipment simplifies the codes of the shaders to be simplified based on the abstract syntax tree according to a preset simplification algorithm so as to obtain at least one shader variety. The at least one shader variant includes the first shader. The electronic equipment screens the at least one shader variant and determines a simplified shader corresponding to the shader to be simplified. When the to-be-simplified shader is the second shader, the simplified shader is the first shader.

In one possible design, the simplified correspondence includes: the length of the shader is to be reduced. The shader to be simplified code or a hash value of the code, the code comprising source code or binary code. And the storage position of the simplified shader corresponding to the shader to be simplified. When the electronic device runs the second shading instruction according to the simplification corresponding relation, the electronic device searches and obtains a fifth shading device which has the same length as the second shading device and codes or hash values of the codes from the at least one shading device to be simplified according to the second shading device. The simplified shader corresponding to the fifth shader in the simplified correspondence is the first shader. The electronic device binds the first shader to a first program corresponding to the second shading instruction instead of the second shader to obtain a first shading instruction. The electronic device instructs the GPU to execute the first shading instruction.

In one possible design, the to-be-simplified shader includes a vertex shader and a fragment shader.

In a seventh aspect, an electronic device is provided that includes one or more processors and one or more memories; one or more memories coupled with the one or more processors, the one or more memories storing computer instructions; the computer instructions, when executed by the one or more processors, cause the electronic device to perform the image processing method of the first aspect and any of the various possible designs as described above.

In an eighth aspect, a chip system is provided, where the chip system includes an interface circuit (or referred to as a communication interface) and a processor; the interface circuit and the processor are interconnected through a line; the interface circuit is used for receiving signals from the memory and sending signals to the processor, and the signals comprise computer instructions stored in the memory; when the processor executes the computer instructions, the system-on-chip performs the image processing method of the first aspect and any of the various possible designs as described above.

A ninth aspect provides a computer readable storage medium comprising computer instructions which, when executed, perform the image processing method of the first aspect and any one of the various possible designs as described above.

A tenth aspect provides a computer program product comprising instructions which, when run on a computer, enable the computer to perform the image processing method of the first aspect and any of the various possible designs as described above, in accordance with the instructions.

It should be understood that the technical solutions provided by the above sixth aspect, seventh aspect, eighth aspect, ninth aspect and tenth aspect can be used to enable the electronic device to obtain the simplified correspondence. For example, the process may be performed offline and the results stored in the electronic device. Therefore, the electronic device can execute the image processing method provided in the first aspect and the possible design thereof based on the simplified corresponding relationship, and therefore, the beneficial effects that can be achieved are similar, and are not described herein again.

Drawings

FIG. 1 is a schematic diagram of an image rendering process;

FIG. 2 is a logic diagram of a rendering instruction stream;

FIG. 3 is a logic diagram illustrating the issue of coloring related functions during loading;

FIG. 4 is a diagram illustrating the issue of coloring related functions during loading;

fig. 5 is a schematic composition diagram of an electronic device according to an embodiment of the present disclosure;

fig. 6 is a schematic flowchart of a method for determining a simplified correspondence relationship according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a screening protocol provided in an embodiment of the present application;

FIG. 8 is a schematic illustration of yet another screening protocol provided in an embodiment of the present application;

FIG. 9 is a schematic illustration of yet another screening protocol provided in an embodiment of the present application;

fig. 10 is a schematic flowchart of an image processing method according to an embodiment of the present application;

fig. 11 is a schematic flowchart of an image processing method according to an embodiment of the present application;

fig. 12 is a schematic flowchart of an image processing method according to an embodiment of the present application;

fig. 13 is a schematic composition diagram of an electronic device according to an embodiment of the present application;

fig. 14 is a schematic composition diagram of a chip system according to an embodiment of the present disclosure.

Detailed Description

Currently, an application installed in an electronic device may instruct the electronic device to perform a corresponding image rendering operation by issuing an instruction stream including a plurality of image rendering commands, so as to obtain related data of a display image (e.g., data of a color, a shape, and the like of each part of a model included in the image), and then display different images to a user through a display screen of the electronic device according to the obtained related data.

Illustratively, Shader (Shader) -related instructions may be included in the stream of rendering instructions to instruct the electronic device to perform shading operations on vertices (vertices) and/or fragments (fragments) of a model included in the image using respective shaders. The shader may refer to an applet running on a Graphics Processing Unit (GPU), and by running the applet, the GPU may perform shading of vertices or fragments.

Wherein a segment may be a unit that one or more points in an image may constitute. For example, a graphic (e.g., a triangle) included in an image may correspond to a segment. In the data corresponding to the segment, corresponding color, depth, texture data, and the like may be included. Shaders may be editable programs used to implement image rendering. Shaders may include Vertex shaders (Vertex shaders), and Fragment shaders (Fragment shaders). Fragment shaders may also be referred to as Pixel shaders (Pixel shaders). The shader information sent by the application program may include pairs of vertex shaders and fragment shaders, so that the electronic device may implement complete rendering of the model. In the embodiments of the present application, the vertex shader and the fragment shader for the same model may be referred to as a shader pair (shader pair).

It should be noted that the vertex shader is used to draw vertex coordinates of an object to be rendered, and the vertex shader is used to perform model transformation, view transformation, projection transformation, illumination, and the like. The fragment shader is used for calculating colors, obtaining textures and filling colors of each pixel point. The processing mechanisms of the vertex shader and the fragment shader are described below in conjunction with FIG. 1. As shown in fig. 1, the output of the vertex shader may be used as input data to the fragment shader. For example, vertex data from a stream of rendering instructions issued by an application may be input to a vertex shader for processing. The rendering process in the vertex shader may include primitive assembly and rasterization operations. As one possible example, the process of primitive assembly may include performing clipping, perspective splitting, and Viewport (Viewport) transformation, among other operations. In the example of fig. 1, the process of primitive assembly may be implemented by a Tessellation Control Shader (TCS) and a Tessellation Evaluation Shader (TES). The rasterized output data (e.g., the pixel coordinates that can be rendered) can be passed to the fragment shader to perform the corresponding shading operation.

That is, in the vertex shader and the fragment shader which are arranged in pair, the output data of the vertex shader can be used as the input data of the fragment shader to perform the rendering of the corresponding model.

Taking an application program as an example of a game application installed in an electronic device, the process of issuing an instruction to complete rendering by the game application is exemplified by combining the shader use mechanism of fig. 1. As shown in fig. 2, the game application may issue a series of rendering instruction streams (e.g., execution S201) including shading instructions, which may include one or more functions issued in sequence. The rendering instruction stream may be passed to a Graphics library in the electronic device, so that the Graphics library calls a corresponding Application Programming Interface (API) to instruct a rendering component (e.g., a Graphics Processing Unit (GPU)) in the electronic device to perform a corresponding shading operation (e.g., perform S202). In the following description, an instruction stream for instructing a shading operation in a rendering instruction stream is referred to as a shading instruction stream.

In some embodiments, in conjunction with fig. 3, the shader instruction stream issued by the game application may include a function of creating a shader object, a function of creating a project (program), a function of compiling a shader, a function of binding a shader object, a function of linking a program, a function of using a program, and the like, which are issued in sequence. Thus, the electronic device can implement the coloring operation of the model in the image according to the coloring instruction stream.

The command functions included in the rendering instruction stream are exemplified below by taking an Open Graphics Library (OpenGL) rendering platform as an example. Wherein the commands may implement their functions through different functions. Illustratively, the shading instruction stream may include a glCreateShader function, a glCreateProgram function, a glCompileShader function, a glAttachShader function, a glLinkProgram, and the like. These functions are briefly described below.

1. glcreateshader (shadertype). The glCreateShader function may be used to create a shader object. One shader object may correspond to a piece of shader code. In this example, the glCreateShader function may carry parameters, such as a shadertype. The shadertype may be carried in the glCreateShader function in numerical form. Different values of the shadertype may be used to indicate the gl _ vertex _ shader or the gl _ fragment _ shader.

Wherein the glCreateShader function indicating gl _ vertex _ shader is used to create a vertex shader. By calling the vertex shader object, the vertex shader object can be used to perform vertex shading operations using the corresponding vertex shader code.

Correspondingly, the glCreateShader function indicating the gl _ fragment _ shader is used to create a fragment shader. By calling the fragment shader object, the fragment shader operation can be performed by using the code of the corresponding fragment shader.

In some implementations, the code of the Shader may be stored in a file corresponding to the Shader resource (Shader Source). For example, the corresponding file may be a text file with the extension of. One shader object may correspond to one shader resource. Alternatively, shader resources can be used to define shader objects.

2. glcreatprogram (void). The glCreatProgram function may be used to create a null item (program) and return a non-zero value (program ID) that may be referenced. The method can realize one-time calling of the vertex shader and the fragment shader by calling a program bound with a shader pair consisting of at least one vertex shader and one fragment shader.

In this example, the program object created by the glCreatProgramm function can be used for additional shader objects. That is, by binding the shader object to the program, the electronic device can smoothly call the corresponding vertex shader and fragment shader according to the instruction of the game application, and by running the code stored in the shader resource corresponding to the shader, the purpose of performing the shading operation by using the shader is achieved.

3. glCompi leShader (shader). The glCompileShader function can be used to compile source code for the shader.

It should be understood that the source code (e.g., code in the format of C + +, Java, etc.) of the corresponding shader may be stored in the file of the shader resource corresponding to the shader. Since the source code cannot be directly recognized by the electronic device, the source code is usually executed by executing the binary code after being interpreted (i.e., the source code is input into a compiler and converted into the binary code). To facilitate quick invocation, the gaming application may compile and store the source code of most (or all) shaders that need to be used into code for binary code prior to using the shaders, such as during loading. Therefore, in the subsequent operation process, the electronic equipment can call and operate the compiled binary code when needing to call the shader, so that the delay caused by the interpretation process is avoided, and the efficiency is improved.

After the compiling and storing of the binary code are completed, the electronic device may add a pointer indicating an ID and a storage address of the binary code to the corresponding shader ID, so that the corresponding binary code may be acquired according to the shader ID in a subsequent calling process.

In this example, the game application may implement compilation of the shader through glcompileshader (shader). The parameter shader carried by the function can be identified by the ID of the created shader object.

It should be noted that the glCompileShader function may be issued after an instruction for creating a corresponding shader object is issued by the game application. In different implementations, there is no restriction on the order of operations of compiling the shader and operations of creating the program. For example, the command of glCompi leShader (shader) may be issued before the glCreatProgram (void) command. As another example, the glCreatProgramm (void) instruction may be issued after the glCreatProgramm (void) instruction. As another example, the glCreatProgramm (void) instruction may be issued concurrently with the glCreatProgramm (void) instruction.

4. glAttachShader (program, shader). The glAttachShader function may bind (or append) the already created shader to the already created program to facilitate use of subsequent game applications. In some implementations, the parameter program carried by the function may be identified by an ID of the program. The parameter shader carried by the function can be identified by the ID of the shader.

In combination with the foregoing description, since both the vertex shader and the fragment shader are used during the process of drawing a model, the electronic device may bind at least one vertex shader and one fragment shader to the same program by issuing the above-mentioned glAttachShader function. For example, the vertex shader is bound to the program by the glAttachShader (program ID, VertexShader ID). As another example, the vertex shader is bound to the program via a glAttachShader (program ID, FragmentShader ID).

5. gllinkprogram (program). The glLinkProgram function may be used to link the corresponding program to the current gaming application.

Thus, as illustrated in the flow of fig. 3, the game application may issue an instruction stream including the above 5 functions in a game loading process, and in a subsequent game application running process, perform a shading operation by calling a program (such as the issue gluuseprogam function) to directly use binary codes of the corresponding vertex shader and fragment shader.

It should be noted that, in the embodiment of the present application, the game loading process may refer to a process in which, after the game application is started, the game application prepares data in a normal running process. For example, during the loading process, the game application may read data to be used from the external storage to the internal memory, so as to make a quick call during the running process. The gaming application may also perform precompiled processes, such as compiling source code in frequently used shader resources into binary code, and the like. In the game loading process, the electronic device may display a prompt word such as "loading in progress" on the interface, or the electronic device may display a loading progress bar on the interface. Dynamic or static images may also be displayed during the process. After the loading is completed, the electronic device may enter a main interface that enables interaction with the user.

In addition, in some scenarios, the game application may also implement a function of binding the shader-bound program to an application program by issuing part of instructions in functions 1 to 5. For example, prior to compilation (e.g., such as the glcompileshader (shader) function), the gaming application may determine whether the shader's source code has been compiled. For example, the game application may check to see if the shader's binary code is stored in the electronic device, and determine if it has been compiled. If the code of the binary code of the shader is stored, the compiling instruction for the shader does not need to be issued.

The process from the creation of the shader to the shading operation using the shader will be described below with reference to a specific example and fig. 4. For example, the source code of the shader needs to be compiled. In the following example, a plurality of correspondences are described. In different implementations, the corresponding relationship may be indicated by a code, the corresponding relationship may be indicated by a corresponding entry, and the corresponding relationship may also be indicated by setting a pointer. The specific indication manner of the corresponding relationship is not limited in the present application.

As shown in fig. 4, the game application may issue a glcreateshader (a) for creating a vertex shader object shader (V1), and the electronic device may further map the shader (V1) to the source code segment V1. The game application may also issue a glcreateshader (b) for creating a fragment shader (F1), and the electronic device may also map the shader (F1) to the source code segment F1. Among them, V1 may be the ID of the loader (V1), and F1 may be the ID of the loader (F1).

In some implementations, the electronic device can store a shader ID and a pointer to a storage address of the corresponding source code. Alternatively, the electronic device may store a correspondence between the shader ID and the storage address of the corresponding source code.

Illustratively, table 1 shows an example of a correspondence relationship between storage shader IDs and storage addresses corresponding to source codes. The storage address of the source code may include a first address where the source code is stored and a length of the source code. Alternatively, the storage address of the source code may include the first address and the last address where the source code is stored. The full amount of codes corresponding to the stored source codes can be obtained through the storage address. For simplicity, address 1 identifies the storage address of V1, address 2 identifies the storage address of F1, and so on.

TABLE 1

Shader ID	Corresponding to the storage address of the source code
		V1	Address
1
	F1	Address 2
……			……

As shown in FIG. 4, the game application may also issue a glCreateProgram (void) for creating a new program. The electronic device may create a new program in response to the instruction, assign a new ID to the program, and feed back the ID to the game application. For example, the ID of the new program is 1, and the new program is program (1).

The game application can also issue glCompileShader (V1), and correspondingly, the electronic device can compile the source code corresponding to the shader (V1) to obtain the corresponding binary code. In some embodiments, according to table 1, the electronic device may determine that the storage address of the source code of the shader with the shader ID V1 is address 1, read the source code segment V1 of the shader (V1) from address 1, and input the source code into the compiler for compiling to obtain a corresponding binary code (e.g., binary code V1'). Similarly, the game application may issue glCompileShader (F1) for compiling the shader (F1), thereby obtaining the corresponding binary code (e.g., binary code F1').

It should be noted that, in some embodiments, after the binary code is obtained by the compiling, the binary code may be stored in the original file storing the source code, and the address of the binary code may not be changed. For example, the binary code V1' may be stored in the memory space corresponding to address 1. As another example, the binary code F1' may be stored in the memory space corresponding to the address 2. In conjunction with the foregoing description of the shader resources, in this example, the compiled binary code V1' may be stored in a file that stores the source code segment V1 corresponding to the shader resources. Similarly, compiled binary code F1' may be stored in a file that stores the shader resources of source code segment F1.

In other embodiments, after the binary code is compiled, the binary code may be stored in a new file different from the original source code, and the address of the binary code may be different from the address of the source code. For example, binary code V1 'may be stored in the memory space corresponding to address 1'. As another example, binary code F1 'may be stored in the memory space corresponding to address 2'. At this time, the electronic device may update the table entry in table 1, for example, add a new data column for identifying the address storing the binary code. Illustratively, Table 2 shows an example of an updated table entry.

TABLE 2

Shader ID	Storage address of source code	Memory address of binary code
			V1	Address
1	Address 1'
		F1	Address 2	Address 2'
……	……				……

In the example of Table 2, the shader with shader ID V1 may have its source code stored at address 1 and its binary code stored at address 1'. Similarly, for a shader with shader ID F1, its source code may be stored at address 2 and its binary code may be stored at address 2'.

With the table 2, when the loader needs to be called (V1), the electronic device may obtain the storage address of the corresponding binary code as the address 1', and then read the binary code from the address to run. Similarly, when the loader needs to be called (F1), the electronic device may obtain the storage address of the corresponding binary code as the address 2', and then read the binary code from the address to run.

The gaming application may then issue a glAttachShader (1, V1) binding the vertex shader (V1) to program (1). The game application may also issue a glAttachShader (1, F1) binding the vertex shader (F1) to program (1). Thus, one vertex shader and one fragment shader are bound to program (1). Similar to the above operation, the electronic device may further update the correspondence table, and update the correspondence between the loader (V1) and the loader (F1) and the program (1) into the table entry. For example, Table 3 shows yet another example of updating an entry.

TABLE 3

Item (program) ID	Shader ID	Storage address of source code	Memory address of binary code
				1	V1	Address	1	Address 1'
1	F1	Address 2	Address 2'

In the example of table 2, an item with an item ID of 1 may have a corresponding relationship, such as a binding relationship, with a shader with an ID of V1 and a shader with an ID of F1. As can be seen from the correspondence relationship shown in table 3, the source code of the shader with ID V1 can be stored at address 1, and the binary code of the shader with ID V1 can be stored at address 1'. As can be seen from the correspondence shown in table 3, the source code of the shader with ID F1 can be stored at address 2, and the binary code of the shader with ID F1 can be stored at address 2'.

After issuing the shader-to-program binding instruction, the game application may issue a glLinkProgram instruction 1 (e.g., glLinkProgram (1)), linking the shader (V1) and shader (F1) -bound program to the current application (i.e., the game application). Thus, during subsequent runs, the game application can use the shader (V1) and the shader (F1) to perform coloring by calling the program (1).

Illustratively, the game application may call program (1) by sending a gluseprogam (1). Therefore, according to the corresponding relationship shown in table 3, when the program (1) is called, the electronic device may determine that the corresponding shader includes a shader (V1) and a shader (F1), and determine a storage address of a binary code of each shader at the same time. The purpose of calling the program (1) to perform the coloring operation can be realized by reading and running the code of the memory address.

Therefore, the game application can achieve the purpose of performing coloring operation through the coloring device.

It should be noted that, with reference to the description in table 1, the correspondence relationship shown in tables 2 and 3 above may also implement the correspondence of each entry therein by adding a pointer to the source code storage address (or the source code storage address and the binary code storage address) on the code segment corresponding to the shader ID and/or the program ID.

It can be appreciated that the shading operation, as an integral part of the rendering operation, has a significant impact on the rendering operation's time consumption and power consumption overhead. As the degree of fineness of the displayed image of the electronic device is higher and higher, the coloring operation is more complicated.

In order to meet the complex shading requirements, the code amount of the shaders required to be used is gradually increased, and the time consumption of the corresponding shading operation is obviously increased. This represents a time-consuming process of rendering the image and an increase in power consumption overhead.

In order to reduce the rendering overhead of the shading operation, the scheme provided by the embodiment of the application replaces an original shader (i.e., a shader used by a game application instruction) with a simplified shader, and realizes the same or similar rendering function through simpler and faster shading operations. In addition, due to the fact that the number of shaders in the running process of the game is large, the complex situations that the same vertex shaders correspond to different fragment shaders, or the different vertex shaders correspond to the same fragment shaders exist, and the like, the scheme provided by the embodiment of the application can ensure that the original shaders can be accurately matched with the corresponding simplified shaders through an accurate matching scheme, and therefore the rendering effect is guaranteed.

The scheme provided by the embodiment of the application is described in detail below with reference to the accompanying drawings. It should be noted that, in the following description, an application (e.g., a game application) may instruct an electronic device to perform rendering operations including a shading operation by issuing a rendering instruction stream. In rendering the instruction stream, the original shader to which the indicated program is bound may be the second shader. The simplified shader corresponding to the second shader may be the first shader. The plurality of to-be-simplified shaders stored in the electronic device may be fourth shaders, and the plurality of simplified shaders corresponding to the plurality of to-be-simplified shaders may also be referred to as third shaders. Among the fourth plurality of shaders, the shader that matches the second shader may also be referred to as a fifth shader. Further, the second coloring instruction may correspond to the first instruction or the second instruction, and the first coloring instruction may correspond to the fourth instruction. The first program may be program (1).

The image processing method provided by the embodiment of the application can be applied to electronic equipment of a user. For example, the electronic device may be a portable mobile device with image processing capability, such as a mobile phone, a tablet computer, a Personal Digital Assistant (PDA), an Augmented Reality (AR) \ Virtual Reality (VR) device, or a media player, and may also be a wearable electronic device, such as a smart watch. The embodiment of the present application does not specifically limit the specific form of the apparatus.

For example, an electronic device to which the solution provided in the embodiment of the present application is applied may include a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, an audio module, a sensor module, a key, a motor, an indicator, a camera, a display screen, and a Subscriber Identity Module (SIM) card interface. Wherein, the sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like. In some embodiments, the electronic device may further include a speaker, a receiver, a microphone, a headphone jack, and the like for implementing audio-related functions of the electronic device.

As an example, the Processor may include a Central Processing Unit (CPU), an Application Processor (AP), a modem Processor, a GPU, an Image Signal Processor (ISP), and the like.

It is to be understood that the above examples do not constitute a specific limitation on the electronic device. In other embodiments, an electronic device may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In other embodiments, the electronic device may have a layered architecture of software, and different layers may be provided with respective corresponding modules or components for implementing corresponding functions.

Illustratively, fig. 5 shows a schematic diagram of a software architecture provided in an embodiment of the present application. In this example, several layers may be included in the electronic device, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. Operated by electronic equipment

The (Android) operating system is an example. The system can be divided into multiple layers, which are an application layer (abbreviated as application layer 501), an application framework layer 502 (abbreviated as framework layer 502), a system library 503, and a hardware layer 504 from top to bottom.

The application layer 501 may include a series of application packages, among other things. The application package may include camera, gallery, calendar, phone, map, navigation, WLAN, bluetooth, music, video, short message, etc. applications. In some embodiments of the present application, the applicationLayer 501 may include an application that provides multimedia streaming functionality to a user. For example, various gaming applications (e.g., gaming applications) may be included in the application layer 501

Etc.). For another example, various types of video applications may also be included in the application layer 501. In the process of running the application programs needing image rendering (or coloring processing in image rendering), the scheme provided by the embodiment of the application can be used for obtaining corresponding beneficial effects. The following examples are described with the game application running as an example.

Framework layer 502 provides an application programming interface and programming framework for applications of application layer 501. The framework layer 502 includes some predefined functions. Framework layer 502 can include a window manager, content provider, view system, resource manager, notification manager, activity manager, input manager, and the like. The Window Manager provides a Window Management Service (WMS), which may be used for Window management, Window animation management, a surface Manager, and a relay station as an input system. The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, and the view system can be used to build applications. The display interface may be composed of one or more views. The resource manager provides various resources for the application. The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short dwell, and does not require user interaction. The notification manager may also be a notification that appears in the form of a chart or scroll bar text at the top status bar of the system. The campaign Manager may provide a campaign Manager Service (AMS), which may be used for the start-up, switching, scheduling of system components (e.g., campaigns, services, content providers, broadcast receivers), and the management and scheduling of application processes. The Input Manager may provide an Input Manager Service (IMS) that may be used to manage inputs to the system, such as touch screen inputs, key inputs, sensor inputs, and the like. The IMS takes the event from the input device node and assigns the event to the appropriate window by interacting with the WMS.

In this embodiment of the present application, in order to implement the image processing method provided in this embodiment of the present application, an interception module, an acquisition module, a matching module, a compiling module, a replacement module, and the like may be disposed in the frame layer 502. In other embodiments, the framework layer 502 may further include more or fewer modules for implementing the solution provided in the embodiment of the present application, for example, in a case that a binary code of the simplified shader is already stored in the electronic device, the framework layer 502 may also be implemented without a compiling module, and the simplified shader is directly executed by calling the stored binary code.

The system library 503 may include a plurality of functional modules. For example: surface manager, Media Framework (Media Framework), Standard C library (libc), OpenGL, open graphics library for Embedded Systems (OpenGL ES), Vulkan, sqlite, Webkit, and the like. In the embodiment of the present application, a rendering mechanism of a rendering platform provided by OpenGL is taken as an example. In the embodiment of the present application, the system library 503 may further include a RunTime (RunTime). For example, the android system-based runtime can be an android runtime. The runtime may provide the functionality of a virtual machine. When the electronic device needs to run various types of code, the respective code can be executed in a virtual environment provided at the time of the running. For example, in the compiling (or pre-compiling) process, the electronic device may input the source code into a compiler in the runtime, and obtain the binary code corresponding to the source code in the current virtual environment through the processing of the compiler. For example, in the case of rendering using an OpenGL platform, the electronic device may compile source codes using a FXC HLSL compiler provided by Microsoft (Microsoft) during android runtime, and then convert a result obtained by the compilation into a GLSL language format using a shader bytecode cross compiler (HLSLcc), so as to facilitate subsequent invocation of the graphics library and the GPU.

In this example, a hardware layer 504 may also be included in the electronic device. For example, various hardware components for implementing the functions of the electronic device may be included in the hardware layer 504. For example, the hardware layer 504 may include a GPU for performing rendering processing, a CPU for performing data processing, a memory for performing data storage, and the like. The memory may include an internal storage medium embedded in the CPU or the GPU, a flash memory having a fast read/write function and provided in the electronic device, a nonvolatile storage medium for storing a large file in the electronic device, and the like.

With reference to the example of fig. 4, in the embodiment of the present application, in the software architecture shown in fig. 5, a game application having an application layer may issue a rendering instruction stream to an electronic device after a game starts to run. Correspondingly, the rendering instruction stream calls an interface of the graphics library through the framework layer to execute corresponding rendering operation. This enables the process of creating a shader to link the shader-bound program to a game application as shown in FIG. 4.

It should be noted that, in the embodiment of the present application, a part of the code of the shader to be simplified (for example, simply referred to as the shader to be simplified) and the simplified shader corresponding to the shader to be simplified may be stored in the memory of the electronic device. The shader to be simplified can be part or all of shaders needing to be called in the running process of the application program. The code of the simplified shader stored in the memory may be a source code or a binary code. For these simplified shaders, the electronic device may replace the original shader bound on the program with one of these simplified shaders by the solution provided by the embodiments of the present application. The specific matching replacement process will be set forth in detail in the following description. Therefore, in the subsequent game application running process, when the shader needs to be called, the corresponding program can be called to run the code of the simplified shader to perform the shading operation, and therefore the power consumption and the time overhead of the shading operation are saved.

In various implementations of the present application, the simplified shader corresponding to the shader to be simplified may be determined in an offline manner. In this way, the code of the simplified shader can be stored in the electronic device before the game starts to run, so that the replacement and use of the simplified shader can be directly realized through the matching mechanism in the running process of the game. In other implementations of the present application, the process of determining the code of the shader to be simplified may also be performed in an electronic device. For example, when the electronic device is in a sleep state or the processing load is low, the codes of the shaders to be simplified are determined and stored in the electronic device, so as to facilitate subsequent replacement use.

The following describes the determination process of the shader to be simplified by taking the offline determination of the code of the shader to be simplified as an example. In some embodiments, as shown in fig. 6, the electronic device can determine the code of the simplified shader to be simplified by the shader as follows.

S601, determining a rendering instruction stream issued by a game application in the game running process.

The rendering instruction stream may include information such as the shader source code, the call instruction for the shader. As an example, the electronic device may record a pat file during the game running through a tool such as Patrace, so as to determine the information through the obtained pat file.

S602, determining a shader to be simplified according to the rendering instruction stream.

For example, the electronic device determines a heavily loaded shader pair in a game running scene according to a pat file storing information such as a shader calling instruction and a shader source code, and marks an object drawn by the shader.

The electronic device may analyze the pat file through a pat processing tool (pat processor) or other tool, so as to implement the function of this step. It should be understood that a plurality of scenes may be included during the game, and the load is different in different scenes. The load may refer to a power consumption load, an operation load, and the like. The electronic device can determine a scene with a load greater than a preset load threshold value as a heavy load scene by analyzing the pat file. And taking the pair of shaders in the heavy load scene as the shader to be simplified. One or more heavy load scenes may be used.

S603, obtaining and determining a corresponding abstract syntax tree according to the shader to be simplified.

In this example, the electronic device may take the shader to be simplified as input, and generate an Abstract Syntax Tree (AST) of the corresponding vertex shader and fragment shader. For each vertex shader or fragment shader that is to be simplified, the electronic device may generate a corresponding AST, and perform the following steps to determine the code of the respective simplified shader.

S604, code simplification is carried out on the shader to be simplified according to a preset simplification algorithm to obtain at least one shader variety.

In this step, the electronic device simplifies the shader code according to a preset simplification algorithm and the AST of the shader to be simplified. Wherein the preset simplified algorithm may include any one of the following: expression deletion, code movement, etc. Shader variants (shaders variants) of the shaders to be simplified are thus obtained. It will be appreciated that the predetermined simplified algorithm may include a plurality, such as a plurality of expression deletion mechanisms and/or a plurality of code movement mechanisms. Thus, each shader to be simplified can obtain a plurality of different shader variants of the shader to be simplified.

It should be appreciated that since the acquisition of each variant is obtained by simplifying the code of the shader to be simplified, the running overhead of all variants is less than the running of the shader to be simplified. For example, the amount of code for some variants is less than the amount of code for the shader to be simplified, and thus replacing the shader to be simplified with these variants for rendering can reduce the computational overhead. As another example, a variant may run less time than the shader to be simplified, and thus replacing the shader to be simplified with the variants for rendering may reduce the overhead on the time duration.

S605, screening at least one shader variety, and determining a simplified shader to be simplified.

For example, in this example, the electronic device may perform shader variant filtering based on rendering quality errors and/or temporal gains. The screening process may be performed based on a pareto model.

Wherein the time gain may be determined by replacing the shader to determine the running time difference.

As a possible implementation, in conjunction with FIG. 7, a frame image stream including frames 0-Q is taken as an example during the game. Where frame P may include the shader to be simplified. After the process of S604, X variants of the frame P can be acquired. Such as frame P1, frame P2, … …, frame PX, etc. The original frame and the running time after replacement of each variant can be run and recorded separately. For example, the time of the running frame 0-frame Q of the original frame (i.e., including frame P) is duration 0. The running time of variant 1 (i.e. including frame P1) is duration 1. By analogy, the running time of the variant X (i.e. comprising the frame PX) is duration X.

In this example, the rendering quality error may be determined by an image quality error (error) evaluation of deferred rendering.

As a possible implementation, in conjunction with FIG. 8, the example of including frame image streams of frame 0-frame Q during the game play process is continued. Where frame P may include the shader to be simplified. After the process of S604, X variations of the frame P can be acquired. Such as frame P1, frames P2, … …, frame PX, etc. The original frame and the rendering result after replacing each variety can be run separately. The rendering result of the original frame is used as the reference image quality, and the image quality difference between each variety and the original frame can be obtained through image quality evaluation. If the variance 1 corresponds to the average image quality 1, the variance 2 corresponds to the average image quality 2, and so on, the variance X corresponds to the average image quality X. In different implementations, the mode of image quality evaluation may be different, and for example, any of the following methods may be adopted: peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Multi-Scale structural Similarity (MS-SSIM), Information-Weighted structural Similarity scheme (Evaluation of Information Content-Weighted SSIM, IW-SSIM), and the like.

In some embodiments of the present application, the simplified shader is determined based on a time gain as an example. The electronic device may use the variant with the shortest duration among the variants as the simplified shader to be simplified, based on the scheme shown in fig. 7. Thus, by replacing the simplified shader, maximum time gain can be achieved.

In other embodiments of the present application, the simplified shader is determined based on rendering quality errors. The electronic device may use the variant with the smallest difference in image quality among the variants as the simplified shader to be simplified based on the scheme shown in fig. 8. Thus, by replacing the simplified shader, the image quality can be guaranteed to the maximum extent.

In other embodiments of the present application, simplified shaders may be determined based on temporal gains and rendering quality errors, for example. The electronic device may obtain the time gain and the image quality difference corresponding to each variation based on the schemes of fig. 7 and 8. Based on this, the electronic device can acquire a frame difference map as shown in fig. 9. Where the abscissa is the temporal gain and the ordinate is the picture quality error, each variant stream may include the rendering of a plurality of frame images, and thus the gain of each frame image may be shown in fig. 9. Based on different scene requirements, the variant meeting the requirement can be selected as the simplified shader. For example, as shown in fig. 9, most of the frame images of the variant 1 have smaller image quality errors, so the variant 1 can be selected as the simplified shader in a scene requiring image quality assurance. Correspondingly, the time gain of most frame images of the variant 2 is higher than that of the variant 1, so that the variant 2 can be selected as the simplified shader in a scene needing to shorten the rendering time.

Thus, through the operations in S601-S605, the electronic device can determine the simplified shader corresponding to the shader to be simplified.

In this example, the electronic device may store the determined information about each shader to be simplified and the corresponding simplified shader in the memory, so as to facilitate replacement and use during the game running process.

For example, the information stored in the memory may include the length of the original shader (i.e., the shader to be simplified), the code of the original shader, the storage location of the corresponding simplified shader, and so on.

The length of the original shader and the code of the original shader can be used for shader matching, so that the electronic device can accurately find the shader needing simplified replacement from a rendering instruction stream issued by a game application after a game runs. After determining the shader needing to be replaced, the electronic device can obtain the simplified shader according to the storage address of the corresponding simplified shader, so that the simplified shader can be used for replacing the original shader, and simplification of the shader is achieved. It should be noted that in different embodiments, the code of the original shader may have different implementations. For example, taking the example of determining the shader to be simplified in the rendering instruction stream through source code matching, the code of the original shader in the correspondence relationship may be the source code of the original shader and/or a hash (hash) value corresponding to the source code of the original shader. For another example, taking the example of determining the shader to be simplified in the rendering instruction stream through binary code matching, the code of the original shader in the correspondence relationship may be the binary code of the original shader and/or a hash (hash) value corresponding to the binary code of the original shader.

The electronic device may store the information about each of the to-be-simplified shaders and the corresponding simplified shader in the form of a file in the electronic device. For example, the file storing the related information of each shader to be simplified and the corresponding simplified shader may be referred to as a map file. In some embodiments, the map file may be stored under a root directory of the game application for normal use when the corresponding game application is running.

In some embodiments, for a pair of shaders that is a pair of a vertex shader and a fragment shader, information corresponding to each of the vertex shader and the fragment shader may be stored in the electronic device. Illustratively, table 4 shows an example of a correspondence relationship between the to-be-simplified shader and the simplified shader included in the map file.

TABLE 4

In the corresponding relationship of table 4, an original vertex shader length vslen and an original fragment shader length fslen can be stored. Taking source code matching as an example, the corresponding relationship may further include original vertex shader source code and/or hash value matching. For example, the original vertex shader source code may be a shader. The original fragment shader source code may be a shader.

As shown in table 4, the corresponding relationship may further include a storage location vert _ x _ x.glsl of the simplified vertex shader (i.e., the simplified shader). The corresponding relationship may further include a storage location frag _ x _ x.glsl of the simplified fragment shader (i.e., the fragment shader).

Thus, after the length and the source code (and/or the hash value) are successfully matched, it can be determined that the storage location in the current table entry stores the corresponding simplified shader.

The shader simplification mechanism provided by the embodiments of the present application will be described in detail below with reference to the to-be-simplified shader in the foregoing description. In an example where the electronic device stores a map file including each shader to be simplified and a corresponding simplified shader correspondence (corresponding relation shown in table 4), the electronic device has a configuration shown in fig. 4.

Exemplarily, in connection with fig. 10. The game application may issue a rendering instruction stream during the loading process. Included in the stream of rendering instructions may be instructions for creating a shader, instructions for creating a program, instructions for compiling a shader, instructions for binding a shader to a program, and instructions for linking a program to a game application. In other embodiments, instructions for compiling the shader may not be included in the rendering instruction stream.

In the stream of rendering instructions, a first instruction, a second instruction, and a third instruction may be included. The first instruction may be one or more instructions carrying an ID of the program or capable of being used to obtain the ID of the program. The second instruction may be one or more instructions carrying the ID and/or memory address of the shader bound to the program, or capable of being used to obtain the ID and/or memory address of the shader bound to the program. The third instruction may be one or more instructions in the rendering instruction stream that are different from the first instruction or the second instruction.

It should be appreciated that in some scenarios, there may be some instructions that can both carry program IDs (i.e., included in the first instruction) and be used to obtain IDs and/or storage addresses of corresponding shaders (i.e., included in the second instruction). Thus, in different implementations, at least partial coincidence may be included in the first instructions and the second instructions. In contrast, in other scenarios, the instructions included in the first instruction and the second instruction may be completely different.

In the solution provided by the present application, after the game starts to load, each instruction in the rendering instruction stream may be input to the interception module, respectively. The intercepting module can be used for intercepting a first instruction and a second instruction in a rendering instruction stream issued by a game application. The intercept module may also pass the first instruction and the second instruction to the fetch module.

For other instructions (e.g., a third instruction) in the stream of rendering instructions, such as the glCreateShader function, the glCreatProgram function, the glCompileShader function, the glattachsharer function, etc., the intercept module may callback the third instruction to the graphics library to ensure implementation of the native logic.

In addition, the interception module can also transmit the first instruction to the graphics library in addition to transmitting the first instruction to the acquisition module.

As an example, the first instruction is intercepted. The interception module may intercept the first instruction by Hook (Hook) technology. The interception module can specifically intercept the instruction in the following manner: when the game application program is started, the interception module modifies a function pointer list in a Thread Local Storage (TLS) of a game Thread, and replaces a graphical function pointer recorded in the list with a replacement pointer. The graphic function pointer is a function pointer pointing to a realization function of the graphic library; instead of pointers, are function pointers that point to implementation functions of other modules (e.g., acquisition modules, etc.). For example, the function pointer of the first instruction in the function pointer list to the implementation function of the graphics library may be replaced with a function pointer to the implementation function of the acquisition module. Therefore, other modules can execute corresponding operations according to the instructions intercepted by the intercepting module. Then, through Hook (Hook) API, the interception module can intercept the instruction and transmit the instruction to other modules when the application program issues the corresponding instruction to be intercepted. The interception of the second instruction is similar to the interception of the first instruction, and is not described herein again.

The obtaining module may be configured to obtain an ID of the current program according to the first instruction from the intercepting module. The obtaining module may be further configured to obtain, according to the second instruction from the intercepting module, an ID and a storage address of the original shader bound to the current program. For example, the obtaining module may parse the first instruction, so as to confirm that the ID of the current program is 1. For another example, the obtaining module may parse the second instruction to obtain that the shader bound to the current program includes the original shader (V1) with ID V1 and source code storage address 1, and the original shader (F1) with ID F1 and source code storage address 2.

In combination with the relationship between the first instruction and the second instruction, in this embodiment of the application, when the first instruction is different from the second instruction, the obtaining module may directly call back the first instruction to the graphics library after obtaining the current program ID. When the first instruction is the same as the second instruction, no callback to the first instruction is performed (as in the example of fig. 5), but subsequent operations continue.

In other embodiments of the present application, the obtaining module may be further configured to obtain the number and length of shaders mounted in the first instruction. The number and length of the shaders can be used for subsequent shader matching. In some implementations, the acquisition module may first determine the number of shaders. When the number of the shaders is 2, it indicates that the shaders can form a pair of the shaders, and then the obtaining module may further determine the types and respective lengths of the mounted pair of the shaders.

The obtaining module can transmit information such as the ID, the storage address and the length of the shader pair of the original shader to the matching module. For example, the fetch module may transmit the ID of the loader (V1) and the source code storage address (address 1) to the matching module. The obtaining module may also transmit the ID of the loader (F1) and the source code storage address (address 2) to the matching module. For another example, the fetch module may transmit the length (length V1) of the loader (V1) to the match module. The acquisition module may also transmit the length (length F1) of the loader (F1) to the matching module.

And the matching module can be used for matching according to the information such as the ID, the storage address, the length of the shader pair and the like of the original shader from the acquisition module, and determining the simplified shader corresponding to the original shader.

In this example, the matching mechanism may be source code matching based on source codes.

For example, in conjunction with the foregoing determination mechanism of the simplified shader, the electronic device may store codes of a plurality of shaders to be simplified, and the codes may be uncompiled source codes. Simplified shaders corresponding to all the to-be-simplified shaders can be stored in the electronic equipment. The correspondence may be stored in a map file including the one shown in table 4. In the following description, the correspondence relationship in the map file may also be referred to as a simplified correspondence relationship.

That is, the electronic device may maintain the correspondence between the code of each simplified shader and the code of the original shader before simplification by simplifying the correspondence. Illustratively, table 5 below shows a specific illustration of one such simplified correspondence.

TABLE 5

In the example of table 5, it can be understood that table 4 corresponds to a specific implementation of the example. In this example, the original vertex shader may be V1 in length, its source code may be shader Source code segment 1 to be simplified, and the corresponding simplified vertex shader may have address 3a as its storage location. The length of the original fragment shader may be F1, the source code may be the shader source code segment 2 to be simplified, and the corresponding storage location of the simplified fragment shader may be address 4 a.

It should be noted that, in the above example, the source code segment of the original shader is directly stored in the simplified correspondence as an example. In other embodiments of the present application, the source code of the original shader may also be obtained by pointer jumping through the stored address. For example, table 6 gives an example of indicating the source code by the memory address.

TABLE 6

Based on the simplified correspondence shown in table 6, the original vertex shader source code can be read from address 3 when it is needed. Correspondingly, when the original fragment shader source code needs to be used, it can be read from address 4.

In some embodiments, the matching module may read all of the to-be-simplified shader source code segments in the simplified correspondence from the memory before performing the matching operation.

The matching module may read a corresponding code segment from the memory according to the storage address of the original shader bound to the current program from the obtaining module. For example, the matching module may read the source code of the shader (V1), i.e., the source code segment V1, from address 1. The matching module may read the source code of the shader (F1), i.e., the source code segment F1, from the address 2.

Then, the matching module can perform a matching operation using the source code segment of the original shader and the source code segment of the shader to be simplified.

In some embodiments of the present application, the matching mechanism may include: and the matching module performs length matching according to the length of the code segment, and performs code matching under the condition of successful length matching.

Wherein, successful length matching may mean that the code lengths of the two matched code segments are completely the same. Successful code matching may refer to source code matching, hash value matching, and the like. Taking source code matching as an example, when all codes of two code segments being compared are identical, the matching is considered to be successful. Conversely, when the codes of the two code segments being compared comprise at least partially different, the match is considered to be a failure.

Take matching the source code segment V1 as an example. The matching module can match the source code segment V1 with all the shader code segments to be simplified in the map file by length and source code respectively. If there is a matching success item, the simplified shader corresponding to the matched shader code segment to be simplified may be the simplified shader corresponding to the source code segment V1.

The matching process for source code segment F1 is similar.

In order to more clearly explain the matching process provided in the embodiments of the present application, the example of table 6 is combined.

Take the example that the shader to be simplified stored at address 3 matches the source code segment V1 and the shader to be simplified stored at address 4 matches the source code segment F1. After the matching process, the matching module may confirm that the source code storage address of the simplified shader corresponding to the shader (V1) is address 3 a. The matching module may determine that the source code storage address of the simplified shader corresponding to the shader (F1) is address 4 a.

In the following example, the simplified shader corresponding to the shader (V1) is the shader (V2) whose memory address is address 3a, and the simplified shader corresponding to the shader (F1) is the shader (F2) whose memory address is address 4 a.

In the case of successful matching, the matching module may transmit the ID and the storage address of the simplified shader to the replacement module for subsequent operations. For example, the matching module may transmit the ID (e.g., V2) of the simplified shader corresponding to the shader (V1), and the memory address (e.g., address 3a) to the replacement module. The matching module may transmit the ID (e.g., F2) of the simplified shader corresponding to the shader (F1), and the memory address (e.g., address 4a) to the replacement module.

Correspondingly, in the case of a matching failure, the table need not be simplified with the loader (V1) and the loader (F1) currently bound to the program (1). The matching module may transmit information of the matching failure to the intercepting module. In this way, the interception module may transmit the second instruction included in the rendering instruction stream to the graphics library, so that when the program (1) is subsequently called, the corresponding shading operation may be performed according to the shader (V1) and the shader (F1).

And the replacing module can be used for replacing the shader information currently bound on the program (1) according to the related information (such as the ID of the simplified shader and the storage address of the simplified shader) of the simplified shader sent by the matching module.

It is understood that when the matching module transmits the information related to the simplified shader to the replacement module, at least one ID and a storage address may be received for the replacement module, and the ID may be, for example, an ID of the simplified shader, and the storage address may store a source code of the simplified shader. Even if the matching module does not inform the replacement module of the information of the simplified shader, the replacement module can consider that the received information of the simplified shader is the information of the simplified shader, and executes corresponding replacement operation.

In some embodiments of the present application, the replacement module may further acquire information of the program from the acquisition module in order to confirm the current program. With reference to the foregoing description, the intercepting module may transmit the first instruction to the obtaining module, and the obtaining module may obtain the ID (for example, the ID is 1) of the current program according to the first instruction. In this example, the replacement module may acquire the ID of the current program as 1 from the acquisition module. In other embodiments of the present application, the replacing module may further obtain the first instruction from the obtaining module or the intercepting module, so as to determine that the ID of the current program is 1 according to the first instruction. Then, the replacement module may confirm that the simplified shader sent by the matching module is replaced on program (1).

As a possible implementation, the replacement module can realize the replacement of the simplified shader through the operation processes of unbinding and binding.

For example, the replacement module may unbind the shader (V1) and the shader (F1) on the current program (1) from the program (1) through the glDetachShader function. Thereby obtaining a program (1) to which no shader is bound.

The replacement module may bind the loader (V2) and the loader (F2) to the program (1) through the glAttachShader function.

Through the above replacement operation, the shaders currently bound to program (1) can be simplified shader shaders (V2) and shaders (F2).

After the replacement operation, the correspondence relationship as in table 3 above may be updated to the following correspondence relationship as shown in table 7.

TABLE 7

Project ID	Shader ID	Storage address of source code	Memory address of binary code
				1	V2	Address 3a	Address 1'
1	F2	Address 4a	Address 2'

It can be seen that, in the updated correspondence, the shader bound on program (1) may include: the shader (V2), namely ID is V2, the source code storage address is address 3a, and the binary code storage address is address 1'. The shader bound on program (1) may further include: the shader (F2), namely ID is F2, the storage address of the source code is address 4a, and the storage address of the binary code is address 2'.

It should be noted that, in other embodiments of the present application, before performing the replacement for the source code, the replacement module may further replace the shader resources of the simplified shader on the program (1) through the glsharersource instruction, so as to facilitate other threads or function calls in the subsequent rendering process.

In the above example, the source code of the simplified shader currently bound to program (1) has not been compiled, and therefore the memory address of the binary code may be incorrect in the correspondence relationship after the above replacement. For example, in this example, the memory address of the binary is also the memory address of the binary of the original shader.

In the case where the memory address is indicated in the form of a pointer, it may appear that the pointer indicating the memory address of the binary code currently bound to the program (1) may be empty, or that the content indicated by the pointer is incorrect, or the like.

In order to avoid problems in the process of calling the binary code subsequently, the replacing module may transmit the storage address of the source code to the compiling module after completing the replacement of the source code information on the program (1).

For example, the replacement module may transmit information about the simplified shader (e.g., ID V2, address 3 a; and ID F2, address 4a) to the compiling module, respectively. The compiling module may read the source code segment of the shader (V2) from the address 3a to compile, and obtain the corresponding binary code segment to store, for example, at the address 3 b. The compiling module may read the source code segment of the shader (F2) from the address 4a to compile, and obtain the corresponding binary code segment to store, for example, at the address 4 b.

The compiling module may transmit the information related to the compiled binary code (e.g., ID V2, address 3b of the binary code, and ID F2, address 4b of the binary code) to the replacement module, so that the replacement module updates the aforementioned relationship. Such as updating a pointer to a memory address of the binary code, or updating a program indicating a memory address of the simplified shader binary code, etc. Illustratively, the correspondence after updating is shown in table 8 below:

TABLE 8

Item (program) ID	Shader ID	Storage address of source code	Memory address of binary code
				1	V2	Address 3a	Address 3b
1	F2	Address 4a	Address 4b

Thus, comparing the correspondence between before and after the compiling module compiles the binary, it can be seen that the storage address for storing the binary code has been updated to the correct address 3b for storing the binary code of the shader (V2) and the correct address 4b for storing the binary code of the shader (F2) in the correspondence.

In other embodiments, instead of the replacement module sending the address of the simplified shader to the compilation module to instruct the compilation module to perform compilation, the matching module may send the ID of the simplified shader and the storage address to the compilation module to perform compilation after determining the simplified shader. The compiling module can obtain the code segments of the simplified shaders from the corresponding addresses for compiling, and store the corresponding binary systems in the memory. The compiling module may transmit the storage address of the binary code and the corresponding shader ID to the replacing module after the compiling is completed, so that the replacing module replaces the pointer of the storage address of the binary code of the shader bound to the program.

Through the replacement process, the replacement module can bind the simplified shader to program (1). The replacement module may then send a new second instruction (e.g., referred to as a fourth instruction) to the graphics library with the updated binding relationship.

For example, in some embodiments, the replacement module may send a fourth instruction to the graphics library. The fourth instruction may carry an ID and/or a storage address of a shader bound to the current program, or the fourth instruction may be used to obtain an ID and/or a storage address of a shader bound to the current program.

In conjunction with the foregoing description, according to the fourth instruction, the shader bound on the current program (1) includes a shader (V2), the ID of the shader (V2) is V2, the source code storage address is 3a, and the binary code storage address is 3 b. The shader bound on the current program (1) further includes a shader (F2), the ID of which (F2) is F2, the source code storage address is 4a, and the binary code storage address is 4 b. The shader (V2) may be a simplified shader of the shader (V1). The shader (F2) may be a simplified shader of the shader (F1).

In other embodiments, the replacement module may send a fourth instruction to the intercept module, which may send the fourth instruction to the graphics library after receiving the information.

Thus, in the case of successful matching, the graphics library may determine, based on the fourth instruction, that the shader bound on program (1) includes the simplified shader (V2) and the simplified shader (F2). When program (1) is called later, the simplified shader (V2) and shader (F2) can be used for shading operations. Thereby increasing coloring efficiency.

Correspondingly, due to the callback mechanism provided in the embodiment of the present application, in the case of a failure in matching, the graphics library may determine, according to the native second instruction, that the shader bound on the program (1) includes the original shader (V1) and the original shader (F1). Correspondingly, when the program (1) is called subsequently, the original shader (V1) and the shader (F1) can be used for shading.

In the above example, the scheme provided by the embodiment of the present application is described from the perspective of module functions with reference to fig. 10. The following description is continued with reference to the interaction between the modules shown in fig. 11.

For example, referring to fig. 11, the game application may issue a rendering instruction stream including a first instruction, a second instruction, and a third instruction, respectively.

The interception module may call back the third instruction to the graphics library to implement the native logic.

In addition, the modules in the framework layer may cooperate with each other, and the scheme provided by the embodiment of the present application is implemented according to the following interaction flow.

S1101, after the game application issues the rendering instruction stream, the interception module intercepts the first instruction and the second instruction.

Illustratively, the stream of rendering instructions may include a first instruction, a second instruction, and a third instruction. The first instruction may be an instruction or instructions carrying an ID of the program or capable of being used to obtain the ID of the program. The second instruction may be one or more instructions carrying the ID and/or memory address of the shader bound to the program, or capable of being used to obtain the ID and/or memory address of the shader bound to the program. The third instruction may be one or more instructions in the rendering instruction stream that are different from the first instruction or the second instruction.

In different implementations, the first instruction and the second instruction may implement their corresponding functions through the same function or multiple functions, that is, the first instruction and the second instruction may be the same. In other embodiments, the function for the first instruction and the function for the second instruction may be different. In connection with the example in fig. 5, in case the first instruction and the second instruction are the same or different, the callback mechanism for the instructions may be different in this scenario. For example, in the case that the first instruction and the second instruction are the same, the electronic device may only execute a callback of the third instruction before the whole process is completed (for example, the fourth instruction is issued), thereby ensuring smooth implementation of the native logic. Correspondingly, under the condition that the first instruction and the second instruction are different, the electronic device may call back the first instruction by using the corresponding module after the first instruction is used, for example, after the ID of the current program is acquired according to the first instruction.

In the following example, the first instruction and the second instruction are the same as an example.

S1102, the interception module sends the first instruction and the second instruction to the acquisition module.

S1103, the obtaining module obtains the ID of the current program according to the first instruction, and obtains the ID, the storage address and the length of the bound original shader according to the second instruction. Among them, the original shader may include a shader (V1) and a shader (F1).

It should be appreciated that the creation, compilation, binding, etc. of the shader is completed before the second instruction is issued by the gaming application. That is, the program (1) has bound thereto the loader (V1) and the loader (F1), and can store in the memory the correspondence shown in table 3 by the pointer, and/or the look-up table, and/or the correspondence of the fixed field.

In this example, the obtaining module may obtain the ID and the storage address of the bound original shader according to the second instruction. For example, the obtaining module may obtain the storage addresses of shaders of the shader (V1) and the shader (F1) through a glgetattachedshares function. Shaders (V1) and shaders (F1) are the original shaders in this example. In other embodiments of the present application, the obtaining module may also look up and obtain the ID, the storage address, and the length of the original shader according to the program ID indicated by the first instruction.

S1104, the obtaining module transmits the ID, the storage address and the length of the original shader to the matching module.

For example, the fetch module may transmit the ID of the original vertex shader (V1) (i.e., V1), and the memory address (Address 1) to the match module. The fetch module may also transmit the ID of the original fragment shader (F1) (i.e., F1), and the memory address (address 2) to the match module.

The obtaining module may also transmit the length V1 of the shader (V1) and the length F1 of the shader (F1) to the matching module.

S1105, the matching module matches the length of the original shader.

In the case that the length matching is successful, the following S1106 is performed. In the case of a failure in matching, the following S1113-S1114 are performed.

S1106, the matching module matches codes of the original shaders.

In the case where the code matching is successful, the following S1107 is executed. In the case of a failure in matching, the following S1113-S1114 are performed.

In this example, the code matching may include source code matching or hash value matching, etc.

For example, the matching module may read from the memory the stored source code of the plurality of shaders to be simplified. The matching module can also acquire the source code of each original shader according to the storage address of the original shader from the acquisition module. And matching the source code of the original shader with the source code of the shader to be simplified.

The matching mechanism may include source code matching and/or hash matching, for example. The successful matching means that the source code of one shader to be simplified is identical to the source code of the original shader in the source codes of a plurality of shaders to be simplified. Correspondingly, the failure of matching means that, in the source codes of the plurality of shaders to be simplified, the source code of any shader to be simplified is not identical to the source code of the original shader. Take hash matching as an example. Each code fragment (e.g., source code of the original shader, or source code of the shader to be simplified) may correspond to a hash value. When the codes included in the code segments are different, the hash values are different. Then, a successful match may mean that the hash values of the two code segments that match are the same. Correspondingly, a matching failure may mean that the hash values of the two code segments that are matched are different.

And if the matching is successful, determining that the simplified shader exists in the current original shader and the simplified processing is required to be performed. Take the example that the loader (V1) and the loader (F1) are both successfully matched. The simplified shader of the shader (V1) may be the shader (V2), and the source code may be stored at address 3 a. The simplified shader of the shader (F1) may be the shader (F2), and the storage address of the source code may be the address 4 a. The matching module may proceed to S1106 below.

Correspondingly, when the matching fails, it is determined that the current original shader does not need to be simplified.

S1107, the matching module sends the ID and the storage address of the simplified shader to the replacement module.

For example, the matching module may send the ID of the loader (V2) (i.e., V2) and its storage address (address 3a) to the replacement module. The matching module may send the ID of the loader (F2) (i.e., F2) and its storage address (address 4a) to the replacement module.

S1108, the replacement module binds the simplified shader to the current program. The current program may be program (1). Before the simplified shader is bound to program (1), the program (1) may have the original shader bound to it.

For example, the replacement module may acquire information of the current program from the acquisition module before executing the S1107. For example, the ID (i.e., 1) of the current program is acquired from the acquisition module. The replacement module may bind the simplified shader to program (1) by a unbinding operation for the shader (V1) and the shader (F1) and a binding operation for the shader (V2) and the shader (F2). The replacement of the simplified shader with the original shader is also achieved.

S1109, the replacing module transmits the ID and the storage address of the simplified shader to the compiling module.

For example, the replacement module may transmit the ID and memory address (address 3a) of the shader (V2), the ID and memory address (address 4a) of the shader (F2) to the compilation module to perform the source code to binary code conversion.

S1110, the compiling module compiles the source code of the simplified shader.

S1111, the compiling module stores the binary code of the simplified shader, and sends the ID of the simplified shader and the storage address of the binary code to the replacing module.

For example, the compiling module may obtain the source code of the shader (V2) from the address 3a in the memory and compile the source code. The compiling module may store the binary code obtained by compiling at the address 3b of the memory.

The compiling module may obtain the source code of the shader (F2) from the address 4a in the memory and compile the source code. The compiling module may store the binary code obtained by compiling at the address 4b of the memory.

In this example, the compiling module may also send information indicating the address 3b, identified as V2, to the replacement module. The compiling module may also send a message indicating the address of 4b, identified as F2, to the replacement module. In order to inform the replacement module of the storage location of the binary code.

S1112, the replacement module updates the identification pointing to the binary code of the simplified shader bound on the current program.

Illustratively, the shader and binary code storage locations are maintained by pointers, for example. The replacement module may update the pointer of the corresponding shader according to the ID of the simplified shader from the compilation module and the storage address of the binary code. The updated pointer may be: the pointer of the shader (V2) may point to address 3b so that the binary code of the shader (V2) may be obtained from address 3 b. The pointer of the shader (F2) may point to the address 4b, so that the binary code of the shader (F2) may be acquired from the address 4 b.

S1113, the replacement module sends a fourth instruction to the graphic library.

The program linked to the fourth instruction may still be the same program (1) as the first instruction or the second instruction. However, unlike the second instruction, the fourth instruction may link to a program (1) that is bound by a simplified vertex shader (e.g., shader (V2)) and a simplified fragment shader (e.g., shader (F2)).

S1114, the matching module sends matching failure information to the intercepting module.

S1115, the interception module transmits the first instruction and the second instruction to the graphics library.

Through the schemes of S1114 and S1115, native logic can be maintained to link the program bound with the original shader to the current game application for subsequent invocation without simplifying the original shader bound to the program.

In this way, the replacement of the tinters during the loading phase is completed. Therefore, the replaced simplified shader can be called to execute the shading operation in the subsequent game running process.

For example, the game application may issue a giuseprogam instruction. The giuseprogam instruction can be used to instruct a shading operation using program (1). The graphics library may call the corresponding application interface to instruct the GPU to execute the rendering instruction corresponding to the program. The rendering instruction corresponding to the program may be an instruction to perform a rendering operation using the program (1). The GPU may retrieve the program bound shader from memory. For example, the GPU may determine, through the correspondence between the program and the ID and the memory address of the shader bound to the program, that the memory address of the binary code of the vertex shader (V2) bound to the program (1) is address 3 b. The GPU may determine, through table 8, that the memory address of the binary code of the fragment shader (F2) to which program (1) is bound is address 4 b. The GPU may then perform shading operations using the program bound shaders. For example, the GPU may perform vertex coloring of the model according to vertex color data delivered by the game application by running a code segment whose address is address 3 b. The GPU may perform pixel rendering of the model according to pixel color data delivered by the game application by running the code segment whose address is address 4 b.

In this way, since the binary codes stored at the address 3b and the address 4b are the binary codes of the simplified shader, the effect of less load and time consumption than the operation of the original shader can be obtained.

A person skilled in the art should be able to fully understand the scheme provided by the embodiments of the present application through the above description of the scheme in fig. 10 and fig. 11. In which, the code matching mechanism is taken as the source code matching as an example. In other embodiments of the present application, matching may also be based on binary codes.

For example, in the simplified correspondence relationship stored in the map file, the length information may be the length of the binary system of the shader to be simplified, and the code of the corresponding simplified shader may also store the binary code. Therefore, after the simplified shaders are determined through the length matching and the binary matching, the corresponding binary codes of the simplified shaders can be directly called without compiling through a compiling module. Therefore, the overhead of the rendering duration can be further increased.

It is understood that in different scenarios, a binary-based alternative or a source code-based alternative may be selected as needed. Wherein, under the scene of paying attention to the improvement of rendering efficiency, a binary alternative scheme can be adopted. In a scenario where storage space is concerned, since the code amount of the source code is generally smaller than the corresponding binary code amount, an alternative scheme based on the source code may be adopted.

With reference to fig. 12, a flowchart of an image processing method is provided in an embodiment of the present application based on the descriptions of fig. 10 to fig. 11. The scheme can be applied to electronic equipment. As shown in fig. 12, the scheme may include:

s1201, the electronic device acquires a second shader based on the rendering instruction stream. The second shader may be the original shader. For example, the electronic device can obtain an identification and a storage address of the second shader based on the stream of rendering instructions. The storage address of the second shader may store a code of the second shader, and the code may be a source code and/or a binary code.

S1202, the electronic device acquires the identifier of the first program based on the rendering instruction stream. Illustratively, the identification of the first program may be an ID of program (1), such as 1.

S1203, the electronic device obtains codes of the second shader based on the storage address of the second shader.

S1204, the electronic device performs length matching and code matching on the codes of the second shader and the codes of the fourth shaders to obtain a fifth shader.

The code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader. Then, the length of the fifth shader is also the same as the length of the second shader.

The fifth shader corresponds to the first shader. That is, the first shader may be a simplified shader of the fifth shader. In this example, the fourth shader may be a to-be-simplified shader. The codes of the fourth shaders include the code of the fifth shader. The codes of the fourth shaders can be stored in the electronic device, the codes of the fourth shaders correspond to the codes of the third shaders one by one, and the codes of the fourth shaders comprise source codes or binary codes.

And S1205, the electronic equipment acquires the first shader through the fifth shader.

Illustratively, the electronic device obtains code for a plurality of third shaders. The third shader may be a simplified shader corresponding to the shader to be simplified. In this example, the code of the plurality of third shaders may include code of the first shader. For example, the electronic device may have stored therein code for a plurality of third shaders. Then, the electronic device may read the codes of the plurality of third shaders from addresses where the codes of the plurality of third shaders are stored, respectively.

S1206, the electronic device replaces the second shader with the first shader.

Illustratively, the electronic device binds a first shader to a first program. For example, the electronic device binds a memory address of a binary code of a first shader to a first program. The electronic device can also unbind the second shader from the first program based on the identification of the second shader, and then the electronic device can bind the identification of the first shader to the first program. In some implementations, the electronic device can compile source code of the first shader into binary code and store the binary code of the first shader.

This allows the simplified first shader to be bound to the first program.

S1207, the electronic device calls the GPU to execute the coloring operation through the first coloring instruction. The shader bound by the first program indicated by the first shading instruction includes the first shader.

The example of fig. 12 is described with the matching success taken as an example. Therefore, when the program is used subsequently, the simplified shader can be directly called, so that the simplification of the shading process is realized, and the time consumption and the power consumption overhead are reduced. It should be appreciated that if a match fails, a callback to the corresponding instruction may be made in accordance with the scheme provided in the previous embodiment in order to implement native logic. Such as performing corresponding shading operations from the original shader.

The scheme provided by the embodiment of the application is mainly introduced from the perspective of each module. To implement the above functions, it includes hardware structures and/or software modules for performing the respective functions. Those of skill in the art would readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Please refer to fig. 13, which is a schematic diagram illustrating an electronic device according to an embodiment of the present disclosure. The electronic device may be the above-described electronic device installed with the game application.

As shown in fig. 13, the electronic device 1300 may include: a processor 1301 and a memory 1302. The memory 1302 is used to store computer-executable instructions. For example, in some embodiments, the processor 1301, when executing the instructions stored in the memory 1302, may cause the electronic device 1300 to perform any of the image processing methods described in the above embodiments.

It should be noted that all relevant contents of each step related to the above method embodiment may be referred to the functional description of the corresponding functional module, and are not described herein again.

Fig. 14 shows a schematic diagram of a chip system 1400. The chip system 1400 may include: a processor 1401 and a communication interface 1402 to enable the electronic device to carry out the functions referred to in the above embodiments. In one possible design, the system-on-chip further includes a memory for storing necessary program instructions and data for the terminal. The chip system may be constituted by a chip, or may include a chip and other discrete devices. It should be noted that, in some implementations of the present application, the communication interface 1402 may also be referred to as an interface circuit.

The functions or actions or operations or steps, etc., in the above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application are all or partially generated upon loading and execution of computer program instructions on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or can comprise one or more data storage devices, such as a server, a data center, etc., that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

Although the present application has been described in conjunction with specific features and embodiments thereof, it will be evident that various modifications and combinations can be made thereto without departing from the spirit and scope of the application. Accordingly, the specification and figures are merely exemplary of the present application as defined in the appended claims and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present application. It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include such modifications and variations.

Claims

1. An image processing method is applied to an electronic device, the electronic device is provided with an application program, the electronic device executes a rendering instruction stream issued by the application program, and the method comprises the following steps:

the electronic device calls a Graphics Processor (GPU) to perform a shading operation through a first shading instruction, wherein shaders bound by a first item program indicated by the first shading instruction comprise a first shader,

wherein the first shader is different from a second shader that is a shader bound on the first program indicated by a second shading instruction included in the stream of rendering instructions;

the first shader corresponds to a fifth shader, and the fifth shader is a shader stored in the electronic device and having the same code length as the second shader.

2. The method of claim 1, further comprising:

the electronic device obtaining the second shader based on the stream of rendering instructions;

the electronic device determines the first shader based on the second shader, the first shader corresponding to the second shader.

3. The method according to claim 1 or 2, characterized in that the method further comprises:

the electronic device replaces the second shader with the first shader.

4. The method of claim 3, wherein the replacing the second shader with the first shader is specifically:

the electronic device binds the first shader to the first program.

5. The method according to any one of claims 1-4, further comprising:

binding a memory address of the binary code of the first shader to the first program.

6. The method of claim 5, further comprising:

binding the identification of the first shader to the first program.

7. The method of claim 6, wherein prior to binding the identity of the first shader to the first program, the method further comprises:

unbinding the second shader from the first program based on the identification of the second shader.

8. The method according to any one of claims 1-7, further comprising:

compiling source code of the first shader into binary code;

storing binary code for the first shader.

9. The method of any of claims 4-8, wherein before the electronic device binds the first shader to the first program, the method further comprises:

and the electronic equipment acquires the identification of the first program based on the rendering instruction stream.

10. The method of any of claims 2-9, wherein the electronic device obtains code for a plurality of third shaders, the code for the plurality of third shaders including code for a first shader, the determining the first shader based on the second shader comprising:

determining code of the first shader based on code of the second shader.

11. The method of claim 10, wherein the electronic device has a plurality of codes of the third shaders stored therein, and wherein obtaining the plurality of codes of the third shaders comprises:

reading the codes of the plurality of third shaders from addresses storing the codes of the plurality of third shaders.

12. The method according to any of claims 2-11, wherein the fetching the second shader based on the rendering instruction stream is specifically:

and acquiring the identification and the storage address of the second shader based on the rendering instruction stream, wherein the storage address stores the code of the second shader, and the code comprises source code or binary code.

13. The method of any of claims 1-12, wherein the electronic device further stores a plurality of fourth shader codes, the plurality of fourth shader codes corresponding one-to-one to the plurality of third shaders, the plurality of fourth shader codes comprising source codes or binary codes.

14. The method of claim 13, wherein the codes of the plurality of fourth shaders include a code of the fifth shader, the code of the fifth shader matching the code of the second shader; the determining the first shader based on the second shader comprises:

obtaining code of the second shader based on a storage address of the second shader;

matching codes of a second shader with codes of the plurality of fourth shaders to obtain the fifth shader;

obtaining, by the fifth shader, the first shader.

15. The method of claim 14, wherein the code of the fifth shader matches the code of the second shader by:

the code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader.

16. The method of any one of claims 1-15,

the first shader and the second shader are of the same type.

17. The method according to any one of claims 1 to 16,

the first shader and the second shader are both vertex shaders, or,

the first shader and the second shader are both fragment shaders.

18. The method according to any of claims 1-17, wherein the rendering instruction stream includes a first instruction and a second instruction, the first instruction indicating an identification of the first program, the second instruction indicating a shader bound on the first program;

the electronic equipment comprises an interception module and an acquisition module, and the method further comprises the following steps:

the interception module acquires the first instruction and the second instruction;

the interception module sends the first instruction and the second instruction to the acquisition module;

the obtaining module obtains the identifier of the first program according to the first instruction;

and the acquisition module acquires the identifier, the storage address and the code length of the second shader according to the second instruction.

19. The method of claim 18, wherein the electronic device further comprises a matching module, the method further comprising:

and the acquisition module sends the identifier, the storage address and the code length of the second shader to the matching module.

20. The method according to claim 18 or 19, wherein the electronic device further comprises a memory, wherein codes of a plurality of third shaders are stored in the memory, and the codes of the plurality of third shaders include the code of the first shader; codes of a plurality of fourth shaders are also stored in the memory, and the codes of the plurality of fourth shaders correspond to the codes of the plurality of third shaders one by one; the code of the plurality of fourth shaders comprises source code or binary code; the method further comprises the following steps:

the matching module reads code of the fourth plurality of shaders from the memory.

21. The method of claim 20, wherein code of a fifth shader is included in code of the plurality of fourth shaders, the fifth shader matching the second shader, the fifth shader corresponding to the first shader, the method further comprising:

the matching module reads the code of the second shader according to the storage address of the second shader;

the matching module matches codes of a second shader with codes of the plurality of fourth shaders to obtain a fifth shader;

wherein, the matching module is the same as the second shader in length and the same as the codes or the hash values of the codes according to the plurality of fourth shaders; the code is a source code or a binary code;

the matching module acquires the first shader through the fifth shader.

22. The method of any one of claims 18-21, wherein the electronic device further comprises a replacement module, the method comprising:

the replacement module binds a memory address of the binary code of the first shader to the first program.

23. The method of any one of claims 18-22, wherein the code is a source code;

the replacement module unbinds the second shader from the first program,

the replacement module binds the identification of the first shader to the first program.

24. The method of claim 23, wherein the electronic device is further provided with a compiling module, and wherein the method further comprises:

the replacing module or the obtaining module sends the storage address of the first shader to the compiling module, and the compiling module compiles the code of the first shader to obtain the binary code of the first shader;

the compilation module stores binary code for the first shader in the memory.

25. The method of claim 24, further comprising:

and the compiling module sends the storage address of the binary code of the first shader to the replacing module.

26. The method according to any of claims 18-25, wherein the first instruction and the second instruction are the same, the first instruction is used to indicate an identifier of the first program and a shader bound to the first program, and the obtaining module obtains the identifier of the first program and an identifier and a storage address of the second shader according to the first instruction.

27. An image processing method is used for obtaining a simplified corresponding relation, wherein the simplified corresponding relation is stored in an electronic device, the simplified corresponding relation is used for determining a corresponding first shader by the electronic device according to a second shader indicated by a second shading instruction issued by an application program, and the first shader is different from the second shader; the method comprises the following steps:

determining at least one shader to be simplified by all rendering instruction streams issued in the running process of the application program of the electronic equipment, wherein the at least one shader to be simplified comprises the second shader;

the electronic equipment determines an abstract syntax tree of the shader to be simplified according to the code of the shader to be simplified;

the electronic equipment simplifies the codes of the shaders to be simplified based on the abstract syntax tree according to a preset simplification algorithm to obtain at least one shader variety; the at least one shader variant includes the first shader;

the electronic equipment screens the at least one shader variant and determines a simplified shader corresponding to the shader to be simplified; when the shader to be simplified is the second shader, the simplified shader is the first shader.

28. The method of claim 27, wherein the simplified correspondence comprises:

length of the shader to be simplified; code of the shader to be simplified or a hash value of the code, the code comprising source code or binary code; a storage position of a simplified shader corresponding to the shader to be simplified;

when the electronic device runs the second coloring instruction according to the simplified correspondence relationship,

the electronic equipment searches and acquires a fifth shader which has the same length as the second shader and the same code or hash value of the code from the at least one shader to be simplified according to the second shader; the simplified shader corresponding to the fifth shader in the simplified correspondence is the first shader;

the electronic equipment replaces the second shader with the first shader and binds the second shader on a first program corresponding to the second shading instruction to obtain a first shading instruction;

the electronic device instructs a GPU to execute the first shading instruction.

29. The method of claim 27 or 28, wherein the to-be-simplified shader comprises a vertex shader and a fragment shader.

30. A chip system, comprising a processor and a communication interface; the processor is configured to call up and run a computer program stored in a storage medium from the storage medium, perform the image processing method according to any one of claims 1 to 26, or perform the image processing method according to any one of claims 27 to 29.