CN114663272B

CN114663272B - Image processing method and electronic equipment

Info

Publication number: CN114663272B
Application number: CN202210163839.1A
Authority: CN
Inventors: 孙黎; 刘金晓; 尹朝阳; 麦倩屏
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2022-02-22
Filing date: 2022-02-22
Publication date: 2024-04-09
Anticipated expiration: 2042-02-22
Also published as: CN114663272A

Abstract

The embodiment of the application discloses an image processing method and electronic equipment, which relate to the field of image processing, and can realize the simplification of coloring operation by running a corresponding simplified coloring device, and reduce the running time consumption and the power consumption expenditure. The specific scheme is as follows: and determining an original shader according to the rendering instruction stream, matching the original shader, and determining a simplified shader corresponding to the original shader. The first shader corresponds to a fifth shader, which is a shader stored in the electronic device that has the same code length as the second shader. The simplified shader is bound to a first program indicated by the rendering instruction stream, such that when the first program is subsequently invoked, the simplified shader may be invoked to perform a corresponding highly shading operation.

Description

Image processing method and electronic equipment

Technical Field

The present disclosure relates to the field of image processing, and in particular, to an image processing method and an electronic device.

Background

As electronic devices develop, the images that need to be displayed become finer and finer. In the process of rendering a fine image, the operation time of the shader and the power consumption cost are increased due to the need of accurately coloring the model in the image. This may lead to a frame loss, a jam, etc. during the image display.

Disclosure of Invention

The embodiment of the application provides an image processing method and electronic equipment, which can realize the simplification of coloring operation by operating a corresponding simplified coloring device when a coloring device (such as a coloring device to be simplified) is required to be called, and reduce the operation time consumption and the power consumption expenditure. Embodiments of the present application further provide a specific determination manner of a simplified shader, and a simplified shader matching manner in an application running process, so as to implement a replacement of a shader and a simplified process.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical scheme:

in a first aspect, an image processing method is provided, applied to an electronic device, where an application program is installed on the electronic device, and the electronic device executes a rendering instruction stream issued by the application program, where the method includes: the electronic device invokes a Graphics Processor (GPU) to perform a shading operation via a first shading instruction, the first item (program) bound shader indicated by the first shading instruction including a first shader, wherein the first shader is different from a second shader that is a shader bound on the first program indicated by a second shading instruction, the second shading instruction included in the rendering instruction stream. The first shader corresponds to a fifth shader, which is a shader stored in the electronic device that has the same code length as the second shader.

Based on this scheme, a scheme example of simplifying the coloring operation is provided. In this scheme, the application program may instruct the electronic device to perform rendering operations including shading operations through a rendering instruction stream. In response to the rendering instruction stream, the electronic device may perform a reduced operation using the corresponding reduced shader. For example, the application may instruct to perform a shading operation using a second shader that is bound to the first program. Correspondingly, the electronic device may send a second shading instruction to the GPU, instructing the GPU to perform the shading operation. The second shading instruction may instruct the GPU to perform shading operations using a simplified shader (e.g., a first shader) that is bound to the first program. In some embodiments, the first shader may be a simplified shader corresponding to the second shader. For example, the first shader may implement the same or similar shading functions as the second shader, while the code size of the first shader may be less than the second shader. In this way, the GPU can normally complete the corresponding shading operation by running the first shader according to the second shading instruction, and at the same time, can obtain lower time consumption and power consumption compared with running the second shader. In this scheme, a scheme of determining a simplified shader corresponding to the second shader is also provided. For example, a fifth shader corresponding to the second shader is determined according to the length match and the code match, and then a simplified shader corresponding to the fifth shader, that is, the first shader, may be used as a simplified shader of the second shader. The correspondence of the fifth shader and the first shader may be stored in the electronic device in a simplified correspondence.

In one possible design, the method further comprises: the electronic device obtains the second shader based on the rendering instruction stream. The electronic device determines the first shader based on the second shader, the first shader corresponding to the second shader. Based on this scheme, an example of a scheme for determining the first shader is provided. In this scenario, the electronic device may obtain a shader (e.g., an original shader, or a second shader) to be used by the application from the rendering instruction stream. The electronic device may also determine a corresponding first shader based on the second shader. The first shader is thus determined to correspond to the second shader, ensuring that the shading operations intended by the application can be achieved by running the first shader, while having lower time and power consumption.

In one possible design, the method further comprises: the electronic device replaces the second shader with the first shader. Based on this scheme, a scheme implementation of a shading operation using a first shader is provided. In the scheme, the first shader is used for replacing the second shader, so that when the application program needs to call the second shader, the GPU can call the corresponding first shader to execute the shading operation.

In one possible design, the replacing the second shader with the first shader is specifically: the electronic device binds the first shader to the first program. Based on this scheme, a specific implementation of shader replacement is provided. It will be appreciated that when an application invokes a shader, it may be implemented by invoking a program to which the shader is bound. In this example, the first shader may be bound to the first program. Therefore, when the application program needs to call the second shader through the first program, the electronic device can correspondingly call the first shader currently bound on the first program, and further perform more efficient shading operation by using the first shader.

In one possible design, the method further comprises: binding a memory address of binary code of the first shader to the first program. Based on this scheme of the present invention,

in one possible design, the method further comprises: binding the identification of the first shader to the first program. Based on this scheme, a specific implementation of shader replacement is provided. In this scheme, the identification of the first shader (e.g., the ID of the first shader) may be bound to the first program. Thus, when the application program issues a command to call the first program, the first shader (such as a storage address) bound on the first program can be found according to the ID of the first shader bound on the first program, and then the shading operation is performed according to the found first shader.

In one possible design, before the binding the identification of the first shader on the first program, the method further includes: unbinding the second shader from the first program based on the identification of the second shader. Based on this scheme, a scheme implementation of binding a first shader is provided. The first program may have a second shader bound to it before the first shader is bound. Then, if the first shader is directly bound to the first program, the binding of the first shader to the first program may not be achieved. In this example, the first program of any unbound shader may be obtained by unbinding the second shader from the first program. Then, the first shader may be bound to the first program that does not bind any shader, thereby allowing only the first shader to be bound to the first program.

In one possible design, the method further comprises: the source code of the first shader is compiled into binary code. The binary code of the first shader is stored. Based on the scheme, an implementation scheme in the calling process of the first shader is provided. In this example, the source code of the first shader may be compiled prior to the shading operation using the first shader, thereby obtaining corresponding binary code. Thus, when the first shader is used, the binary code can be directly called to run, so that the efficiency is improved.

In one possible design, before the electronic device binds the first shader to the first program, the method further includes: the electronic device obtains an identification of the first program based on the rendering instruction stream. Based on the scheme, a scheme implementation of the electronic device binding simplified shader is provided. It should be appreciated that in order to be able to properly implement the replacement of the second shader by the first shader, then the first program of the first shader needs to be bound explicitly. In this example, different programs may be indicated by their identity (e.g., program ID). The electronic device may obtain, according to an instruction in the rendering instruction stream, an identification of the first program, so as to facilitate accurate execution of the subsequent replacement.

In one possible design, the electronic device obtains codes of a plurality of third shaders, the codes of the plurality of third shaders including codes of a first shader, the first shader being determined based on the second shader, including determining the first shader code based on the codes of the second shader. Based on this scheme, an example of a scheme for determining a first shader based on a second shader is provided. The electronic device may obtain codes for a plurality of simplified shaders (i.e., a third shader) before the first shader needs to be obtained. A shader corresponding to (or referred to as a match) the code of the second shader may be included in the plurality of third shaders, and the matched shader may then be the first shader.

In one possible design, the electronic device has stored therein a plurality of codes of the third shader, the acquiring the codes of the plurality of third shaders includes: the codes of the plurality of third shaders are read from an address where the codes of the third shaders are stored. Based on this scheme, a scheme example of acquiring codes of a plurality of third shaders is provided. A plurality of codes of the third shader may be stored in the electronic device. Thus, when it is desired to determine the first shader using a plurality of third shaders, the plurality of third shaders may be read from the corresponding memory addresses. In other embodiments, some or all of the code of the plurality of third shaders may also be acquired from the cloud, or some or all of the code of the plurality of third shaders may also be self-determined by the electronic device.

In one possible design, the obtaining the second shader based on the rendering instruction stream is specifically: the method further includes obtaining an identification of the second shader and a memory address based on the rendering instruction stream, the memory address storing code of the second shader, the code including source code or binary code. Based on this scheme, a specific scheme example of acquiring the second shader is provided. In this example, the identification of the shader (i.e., the second shader) that is bound on the first program and the storage address may be obtained from the rendering instruction stream. The storage address may be an address for storing the source code of the second shader, or may be an address for storing the binary code of the second shader. For example, the shader resource bound to the first program may be determined by the ID of the first program, and thus the ID of the second shader and the memory address.

In one possible design, the electronic device further stores a plurality of fourth shader codes, the plurality of fourth shader codes corresponding one-to-one to the plurality of third shaders, the plurality of fourth shader codes including source code or binary code. Based on this scheme, a specific example of a scheme for determining the first shader is provided. For example, the fourth shader (i.e., the shader to be simplified) may have a one-to-one correspondence with the simplified third shader. Then, by the code of the fourth shader, the satisfactory first shader of the third shader can be determined.

In one possible design, the codes of the fourth plurality of shaders include a code of a fifth shader, the code of the fifth shader matching the code of the second shader, the fifth shader corresponding to the first shader. The determining the first shader based on the second shader includes: the code of the second shader is obtained based on the memory address of the second shader. Matching the codes of the second shader with the codes of the fourth shaders to obtain the fifth shader. The first shader is acquired by the fifth shader. Based on this scheme, a specific example of a scheme for determining a first shader that matches a second shader is provided. In this example, a fifth shader that matches the second shader may be found among the plurality of fourth shaders based on the code of the second shader. The fifth shader may correspond to one simplified shader among the plurality of third shaders. The simplified shader corresponding to the fifth shader may be the first shader to be determined.

In one possible design, the code of the fifth shader matches the code of the second shader, specifically: the code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader. Based on this scheme, a specific matching mechanism is provided. Under the condition of conforming to the matching mechanism, the matching can be considered to be successful, and the corresponding matching item can be used as a matching result. It should be noted that, in this example, whether the code of the shader performing matching (such as the code of the second shader, the code of the fourth shaders, and so on) is source code or binary code, the corresponding shader meeting the requirements may be screened according to the matching mechanism.

In one possible design, the first shader and the second shader are the same type. Based on this scheme, a corresponding definition of a first shader and a second shader is provided. In this way, after the second shader is replaced by the first shader, the correct execution of the shading operation can be ensured.

In one possible design, the first shader and the second shader are both vertex shaders, or the first shader and the second shader are both fragment shaders. Based on this scheme, two examples of the types of the first shader and the second shader are provided. In other embodiments, the first shader and the second shader may also be other types of shaders that are different from the vertex shader or the fragment shader.

In one possible design, the rendering instruction stream includes a first instruction for indicating an identification of the first program and a second instruction for indicating a shader bound on the first program. The electronic equipment comprises an interception module and an acquisition module, and the method further comprises the steps of: the interception module acquires the first instruction and the second instruction. The interception module sends the first instruction and the second instruction to the acquisition module. The acquisition module acquires the identification of the first program according to the first instruction. The acquisition module acquires the identification and the storage address of the second shader according to the second instruction. Based on this scheme, an example of implementation of the embodiment scheme of the application based on modularization is provided. In this example, the interception module may be used to intercept instructions, such as may be used to intercept a first instruction and a second instruction. For other instructions in the rendering instruction stream, the interception module may call back directly to the graphics library to implement the native logic.

In one possible design, the electronic device further includes a matching module, the method further comprising: the acquisition module sends the identification and the storage address of the second shader to the matching module. Based on the scheme, the acquisition module can send the acquired identification and storage address of the second shader to the matching module so as to facilitate subsequent operation.

In one possible design, the electronic device further includes a memory, where the memory stores codes of a plurality of third shaders, where the codes of the plurality of third shaders include the codes of the first shader. The memory also stores codes of a plurality of fourth shaders, and the codes of the fourth shaders are in one-to-one correspondence with the codes of the third shaders. The code of the plurality of fourth shaders includes source code or binary code. The method further comprises the steps of: the matching module reads codes of the plurality of fourth shaders from the memory. Based on the scheme, the matching module can read a plurality of to-be-simplified shaders (such as a fourth shader) stored in the memory, so that the to-be-simplified shaders corresponding to codes of the second shader are determined by subsequent matching, and corresponding first shaders are determined from a plurality of third shaders.

In one possible design, the code of the plurality of fourth shaders includes code of a fifth shader, the fifth shader matching the second shader, the fifth shader corresponding to the first shader, the method further comprising: the matching module reads the code of the second shader according to the memory address of the second shader. The matching module matches the codes of the second shader with the codes of the fourth shaders to obtain the fifth shader. The matching module obtains the first shader through the fifth shader. Based on the scheme, the matching module can match codes of a plurality of to-be-simplified shaders according to codes of the second shader. The match may be a source code match or a hash value match based on source code. The match may also be a binary match or a hash value match of a binary-based code. By means of the matching mechanism, a shader to be simplified, such as a fifth shader, corresponding to the second shader can be determined.

In one possible design, the electronic device further includes a replacement module, the method comprising: the replacement module binds a memory address of the binary code of the first shader to the first program. Based on this scheme, the replacement module may bind the memory address of the simplified shader (i.e., the first shader) corresponding to the second shader to the first program. For example, the replacement module may replace a pointer indicating a storage address of the binary code of the first shader onto the first program so that the binary code of the first shader may be directly called by calling the first program.

In one possible design, the code is source code. The replacement module unbinds the second shader from the first program, and the replacement module binds the identification of the first shader to the first program. Based on the scheme, the matching module can perform matching based on the source codes according to the source codes of the second shader and the acquired source codes of the fourth shaders. The replacement module may then perform a source code based replacement operation. For example, through matching, the replacement module can know the storage address and the ID of the source code of the first shader that is successfully matched from the matching module, so that the replacement module can bind the ID of the first shader to the first program, and because the unique mapping relationship exists between the ID of the first shader and the storage address of the source code of the first shader, the code of the first shader can be successfully acquired when the first program is called.

In one possible design, the electronic device is further provided with a compiling module, the method further comprising: the replacing module or the acquiring module sends the storage address of the first shader to the compiling module, and the compiling module compiles the code of the first shader to acquire the binary code of the first shader. The compiling module stores binary code of the first shader in the memory. Based on the scheme, the source code of the first shader can be compiled on the basis of replacing source code information (such as the ID of the first shader) on the first program, so that the subsequent quick call to the first shader is facilitated.

In one possible design, the method further comprises: the compiling module sends a storage address of the binary code of the first shader to the replacing module. Based on the scheme, the compiling module can send the address of the binary code of the first shader after compiling to the replacing module so that the replacing module binds the binary information to the first program.

In one possible design, the first instruction and the second instruction are the same, the first instruction is used for indicating the identifier of the first program and the shader bound on the first program, and the obtaining module obtains the identifier of the first program and the identifier and the storage address of the second shader according to the first instruction. Based on this scheme, an example scheme of the relation between the first instruction and the second instruction is provided. For example, the first instruction may be the same instruction or instructions as the second instruction, by which the ID of the first program may be validated, and may also be used to validate (the ID and/or the memory address of) the shader bound on the first program.

In a second aspect, an electronic device is provided, the electronic device comprising one or more processors and one or more memories; one or more memories coupled to the one or more processors, the one or more memories storing computer instructions; the computer instructions, when executed by one or more processors, cause the electronic device to perform the image processing method of the first aspect and any of the various possible designs described above.

In a third aspect, a chip system is provided, the chip system comprising an interface circuit (or referred to as a communication interface) and a processor; the interface circuit and the processor are interconnected through a circuit; the interface circuit is used for receiving signals from the memory and sending signals to the processor, and the signals comprise computer instructions stored in the memory; when the processor executes the computer instructions, the system-on-chip performs the image processing method as described above in the first aspect and any of various possible designs.

In a fourth aspect, there is provided a computer readable storage medium comprising computer instructions which, when executed, perform the image processing method of the first aspect and any of the various possible designs described above.

In a fifth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the image processing method of the first aspect and any of the various possible designs described above in accordance with the instructions.

In a sixth aspect, an image processing method is provided, where the method is used to obtain a simplified correspondence, where the simplified correspondence is stored in an electronic device, and the simplified correspondence is used to determine a corresponding first shader according to a second shader indicated by a second shading instruction issued by an application program by the electronic device, where the first shader is different from the second shader. The method comprises the following steps: and determining at least one to-be-simplified shader by all the issued rendering instruction streams in the running process of the application program of the electronic equipment, wherein the at least one to-be-simplified shader comprises the second shader. The electronic device determines an abstract syntax tree of the shader to be simplified according to the code of the shader to be simplified. The electronic device performs simplification processing on the code of the shader to be simplified based on the abstract syntax tree according to a preset simplification algorithm so as to obtain at least one shader variant. The at least one shader variant includes the first shader. The electronic device screens the at least one shader variant and determines a simplified shader corresponding to the shader to be simplified. When the to-be-simplified shader is the second shader, the simplified shader is the first shader.

In one possible design, the simplified correspondence includes: the length of the shader is to be simplified. The code of the shader to be simplified, or the hash value of the code, includes source code or binary code. The storage location of the simplified shader corresponding to the to-be-simplified shader. And when the electronic equipment runs the second coloring instruction according to the simplified corresponding relation, the electronic equipment searches and acquires a fifth coloring device with the same length as the second coloring device and the same code or hash value of the code from the at least one coloring device to be simplified according to the second coloring device. The simplified shader corresponding to the fifth shader in the simplified correspondence is the first shader. The electronic device binds the first shader to replace the second shader on a first program corresponding to the second shading instruction so as to acquire the first shading instruction. The electronic device instructs the GPU to execute the first shading instruction.

In one possible design, the shader to be simplified includes a vertex shader and a fragment shader.

In a seventh aspect, an electronic device is provided, the electronic device comprising one or more processors and one or more memories; one or more memories coupled to the one or more processors, the one or more memories storing computer instructions; the computer instructions, when executed by one or more processors, cause the electronic device to perform the image processing method of the first aspect and any of the various possible designs described above.

In an eighth aspect, a chip system is provided, the chip system comprising an interface circuit (or referred to as a communication interface) and a processor; the interface circuit and the processor are interconnected through a circuit; the interface circuit is used for receiving signals from the memory and sending signals to the processor, and the signals comprise computer instructions stored in the memory; when the processor executes the computer instructions, the system-on-chip performs the image processing method as described above in the first aspect and any of various possible designs.

In a ninth aspect, there is provided a computer readable storage medium comprising computer instructions which, when executed, perform the image processing method of the first aspect and any of the various possible designs described above.

In a tenth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the image processing method of the first aspect and any of the various possible designs described above in accordance with the instructions.

It should be appreciated that the technical solutions provided in the sixth aspect, the seventh aspect, the eighth aspect, the ninth aspect and the tenth aspect above may be used to enable the electronic device to obtain a simplified correspondence. For example, the process may be performed offline and the results stored in the electronic device. Therefore, the electronic device can execute the image processing method provided in the first aspect and the possible designs thereof based on the simplified corresponding relation, and thus the beneficial effects can be similar, and the description thereof is omitted herein.

Drawings

FIG. 1 is a schematic diagram of an image coloring process;

FIG. 2 is a logic diagram of a rendering instruction stream;

FIG. 3 is a logic diagram illustrating the issue of coloring correlation functions during loading;

FIG. 4 is a detailed diagram of coloring correlation function issuing in the loading process;

fig. 5 is a schematic diagram of the composition of an electronic device according to an embodiment of the present application;

FIG. 6 is a flowchart illustrating a method for determining a simplified correspondence according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a screening protocol according to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of yet another screening protocol provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of yet another screening protocol provided in an embodiment of the present application;

fig. 10 is a schematic flow chart of an image processing method according to an embodiment of the present application;

fig. 11 is a schematic flow chart of an image processing method according to an embodiment of the present application;

fig. 12 is a flowchart of an image processing method according to an embodiment of the present application;

fig. 13 is a schematic diagram of the composition of an electronic device according to an embodiment of the present application;

fig. 14 is a schematic diagram of a system on chip according to an embodiment of the present application.

Detailed Description

Currently, an application program installed in an electronic device may instruct the electronic device to perform a corresponding image rendering operation by issuing an instruction stream including a plurality of image rendering commands, so as to obtain related data of a display image (such as data of colors, shapes, etc. of each part of a model included in the image), and further display different images to a user through a display screen of the electronic device according to the obtained related data.

For example, shader (loader) -related instructions may be included in the rendering instruction stream to instruct the electronic device to perform shading operations on vertices (vertexes) and/or fragments (fragments) of the model included in the image using the corresponding Shader. The shader may refer to an applet running on a graphics processor (Graphics Process ing Unit, GPU) by which the GPU may implement shading of vertices or fragments.

Wherein a segment may be a unit in which one or more points in an image may be composed. For example, a graphic (e.g., triangle) included in an image may correspond to a segment. The data corresponding to the segments may include corresponding color, depth, texture data, and the like. The shader may be an editable program to implement image rendering. The shaders may include Vertex shaders (Vertex shaders), and Fragment shaders (Fragment shaders). The fragment Shader may also be referred to as a Pixel Shader (Pixel Shader). Pairs of vertex shaders and fragment shaders may be included in shader information issued by an application so that the electronic device may implement a complete rendering of the model. In the embodiments of the present application, vertex shaders and fragment shaders for the same model may be referred to as shader pairs (pairs).

It should be noted that, the vertex shader is used to draw vertex coordinates of an object to be rendered, and the vertex shader is used to perform model transformation, visual transformation, projective transformation, illumination, and the like. The fragment shader is used to compute color, fetch texture, and color fill for each pixel. The processing mechanisms of the vertex shader and fragment shader are described below in conjunction with FIG. 1. As shown in FIG. 1, the output of the vertex shader may be used as input data to the fragment shader. For example, vertex data from a rendering instruction stream issued by an application may be input to a vertex shader for processing. Rendering processing in a vertex shader may include primitive assembly, as well as rasterization. As one possible example, the process of primitive assembly may include performing operations such as clipping, perspective segmentation, and Viewport (Viewport) transformation. In the example of fig. 1, this process of primitive assembly may be implemented by tessellation (Tessellat ion Control Shader, TCS) and tessellation evaluation shader (Tessellation Evaluat ion Shader, TES). The output data (e.g., the renderable pixel coordinates) after the rasterization process may be passed to a fragment shader for execution of the corresponding shading operations.

That is, in the vertex shader and the fragment shader arranged in pairs, the output data of the vertex shader can be used as the input data of the fragment shader to render the corresponding model.

Taking an application program as an example of a game application installed in the electronic device, in combination with the shader usage mechanism of fig. 1, a process from issuing instructions to completing rendering by the game application is exemplified. As shown in FIG. 2, the gaming application may issue a series of rendering instruction streams (e.g., execute S201) including shading instructions, which may include one or more functions issued in sequence. The rendering instruction stream may be passed to a graphics library in the electronic device, so that the graphics library invokes a corresponding application program interface (Applicat ion Programming Interface, API) instructing a rendering component (e.g., graphics processor (Graphics Process ing Unit, GPU)) in the electronic device to perform a corresponding shading operation (e.g., to perform S202). In the following description, an instruction stream for indicating a shading operation among rendering instruction streams is referred to as a shading instruction stream.

In some embodiments, in connection with FIG. 3, the stream of shading instructions issued by the gaming application may include sequentially issued functions to create shader objects, create items (programs), compile shader functions, bind shader objects, link programs, and use programs, among others. Therefore, the electronic equipment can realize the coloring operation of the model in the image according to the coloring instruction stream.

The command functions included in the above-described shading instruction stream are exemplified below by an open graphics library (Open Graphics Library, openGL) rendering platform. Wherein the command may implement its function by different functions. Illustratively, the colored instruction stream may include a glCreateHader function, a glCreatProgram function, a glCompileHader function, a glAttachShader function, a glLinkProgram, and the like. These functions are briefly described below.

1. glCreateShader (shadertype). The glCreateShader function may be used to create a shader object. One shader object may correspond to a piece of code of a shader. In this example, the glCreateStader function may carry parameters such as shadatype. The shadatype may be carried in numeric form in the glCreateShader function. Different values of the shadatype may be used to indicate gl_vertex_shader or gl_fragment_shader.

Wherein the glCreateStader function, which indicates gl_vertex_loader, is used to create a vertex shader. By invoking the vertex shader object, the vertex shading operations may be performed using the code of the corresponding vertex shader.

Correspondingly, the glCreateStader function, which indicates gl_fragment_loader, is used to create a fragment shader. By invoking the fragment shader object, fragment shading operations can be performed using the code of the corresponding fragment shader.

In some implementations, the code of the Shader can be stored in a file corresponding to a Shader resource (loader Source). For example, the corresponding file may be a text file with an extension of. Reader. One shader object may correspond to one shader resource. Alternatively, shader resources can be used to define shader objects.

2. glCreatProgram (void). The glCreatProgram function may be used to create a null item (program) and return a non-zero value (program ID) that may be referenced. One-time invocation of vertex and fragment shaders may be accomplished by invoking a program bound with a pair of shaders consisting of at least one vertex shader and one fragment shader.

In this example, the program object created by the glCreatProgram function may be used to append a shader object. That is, by binding the shader object to the program, the electronic device can smoothly call the corresponding vertex shader and fragment shader according to the instruction of the game application, and by running the code stored in the shader resource corresponding to the shader, the purpose of using the shader to perform the shading operation is achieved.

3. glCompi lesader (loader). The glcompilehader function may be used to compile source code for the shader.

It should be appreciated that the source code (e.g., code in c++, java, etc.) for the corresponding shader may be stored in the file of the shader resource for the shader. Since the source code cannot be directly recognized by the electronic device, the source code is usually required to be interpreted (i.e., the source code is input into a compiler and converted into binary code) before the execution of the binary code can be realized. To facilitate quick invocation, the gaming application may compile the source code of most (or all) of the shader that needs to be used into code of binary code and store it before using the shader, such as during loading. In this way, in the subsequent operation process, the electronic device can avoid the delay caused by the interpretation process by calling and operating the compiled binary code when the shader needs to be called, thereby improving the efficiency.

After the compiling and storing of the binary code are completed, the electronic device may newly add a pointer for indicating the ID and the storage address of the binary code on the corresponding shader ID, thereby enabling the corresponding binary code to be acquired according to the shader ID in the subsequent call process.

In this example, the game application may implement the compilation of the shader through glCompileShader (shader). The parameter loader carried by the function can be identified by the ID of the created shader object.

It should be noted that the glCompileShader function may be issued after an instruction issued by the game application to create a corresponding shader object. In a different implementation, the order of the operations of the compiling shader and the operations of creating the program is not limited. For example, the glCompi leslader instruction may be issued before the glCreatProgram (void) instruction. As another example, the instruction of glCreatProgram (void) may be issued after the glCreatProgram (void) instruction. As another example, the instruction of glCreatProgram (void) may be issued concurrently with the glCreatProgram (void) instruction.

4. glatatchloader (program, loader). The glatatchloader function may bind (or append) an already created shader to an already created program to facilitate use of subsequent gaming applications. In some implementations, the parameter program carried by the function may be identified by the program's ID. The parameter loader carried by the function can be identified by the ID of the loader.

In connection with the foregoing description, since both vertex and fragment shaders are used in drawing a model, the electronic device may bind at least one vertex and one fragment shader to the same program by issuing the glatatachloader function described above. For example, the vertex shader is bound to the program by glatatchloader (program ID, vertexShader ID). As another example, the vertex shader is bound to the program by a glatatchloader (program ID, fragmentShader ID).

5. glLinkProgram (program). The glLinkProgram function may be used to link the corresponding program to the current gaming application.

Thus, as shown in the flowchart of fig. 3, the game application may issue an instruction stream including the 5 functions during the game loading process, and may directly use binary codes of the corresponding vertex shader and fragment shader to perform a shading operation by calling a program (e.g., the following glusepgram function) during the running process of the subsequent game application.

It should be noted that, in the embodiment of the present application, the game loading process may refer to a process in which after the game application is started, the game application prepares data in the normal running process. Illustratively, during this loading process, the gaming application may read the data that needs to be used from external storage into memory for quick recall during execution. The gaming application may also perform precompiled processes such as compiling source code in frequently used shader resources into binary code, and the like. In the game loading process, the electronic device can display a prompt word such as loading on the interface, or the electronic device can display a loading progress bar on the interface. Dynamic or static images may also be displayed in the process. After loading is completed, the electronic device may enter a main interface that is capable of interacting with the user.

In addition, in some scenarios, the game application may also implement the function of binding the shader-bound program to the application by issuing some of the instructions in functions 1 through 5. Illustratively, the gaming application may determine whether the source code of the shader is compiled prior to compiling (e.g., issuing glCompileShader (shader) functions). For example, the gaming application may look to see if the shader's binary code is stored in the electronic device, determining if it is compiled. If the code of the binary code of the shader is stored, the compiling instruction for the shader may not need to be issued.

The procedure from creation of a shader to shading operations using a shader described above is described below with reference to a specific example and fig. 4. Taking the example of compiling the source code of the shader. In the following examples, description of a plurality of correspondence relationships is referred to. In different implementations, the correspondence may be indicated by a code, the correspondence may be indicated by a correspondence table entry, and the correspondence may also be indicated by setting a pointer. The specific indication mode of the correspondence is not limited in the present application.

As shown in FIG. 4, the gaming application may issue glCreateShader (A) a vertex shader object, a loader (V1), which the electronic device may also correspond to the source code segment V1. The gaming application may also issue glCreateShader (B) for creating a fragment shader segment, which the electronic device may also correspond to the source code segment, F1. Wherein V1 may be the ID of the reader (V1), and F1 may be the ID of the reader (F1).

In some implementations, the electronic device can store the shader ID and a pointer to a storage address of the source code. Alternatively, the electronic device may store the correspondence of the shader ID and the storage address corresponding to the source code.

By way of example, table 1 shows an example of a correspondence of a storage shader ID and a storage address corresponding to source code. The storage address of the source code may include a first address of the source code storage and a length of the source code. Alternatively, the storage address of the source code may include a first address and a last address of the source code storage. The full code of the corresponding stored source code can be acquired through the storage address. For brevity, address 1 identifies the storage address of V1, address 2 identifies the storage address of F1, and so on.

TABLE 1

Shader ID	Memory address corresponding to source code
		V1	Address 1
F1	Address 2
		……	……

As shown in fig. 4, the gaming application may also issue glCreateProgram (void) for creating a new program. The electronic device may create a new program in response to the instruction and assign a new ID to the program, which is fed back to the gaming application. For example, taking the ID of a new program as 1, the new program is the program (1).

The game application can also issue a glCompileShader (V1), and the corresponding electronic equipment can compile source codes corresponding to the sharer (V1) to obtain corresponding binary codes. In some embodiments, the electronic device may determine, according to table 1, that the storage address of the source code of the shader having the shader ID V1 is address 1, read the source code segment V1 of the loader (V1) from address 1, and input the source code into the compiler for compiling to obtain the corresponding binary code (e.g., binary code V1'). Similarly, the game application may issue a glCompileShader (F1) for compiling the loader (F1), thereby obtaining a corresponding binary code (e.g., binary code F1').

It should be noted that, in some embodiments, after the binary code is compiled and obtained, the binary code may be stored in the original file storing the source code, and its address may be unchanged. For example, the binary code V1' may be stored in the memory space corresponding to the address 1. As another example, the binary code F1' may be stored in the memory space corresponding to the address 2. In connection with the foregoing explanation regarding shader resources, in this example, the binary code V1' after compilation may be stored in a file corresponding to the shader resources storing the source code segment V1. Similarly, the compiled binary code F1' may be stored in a file corresponding to the shader resource storing the source code segment F1.

In other embodiments, after the binary code is compiled, it may be stored in a new file that is different from the original stored source code, and its address may be different from the address of the stored source code. For example, the binary code V1 'may be stored in a memory space corresponding to the address 1'. As another example, the binary code F1 'may be stored in the memory space corresponding to the address 2'. At this time, the electronic device may update the entries in table 1 synchronously, such as adding a data column for identifying the address where the binary code is stored. Illustratively, table 2 shows an example of updating an entry.

TABLE 2

Shader ID	Storage address of source code	Memory address of binary code
			V1	Address 1	Address 1'
F1	Address 2	Address 2'
			……	……	……

In the example of Table 2, the shader ID is V1, its source code may be stored at address 1, and its binary code may be stored at address 1'. Similarly, the shader ID is F1, its source code may be stored at address 2, and its binary code may be stored at address 2'.

Through table 2, the electronic device can obtain the storage address of the corresponding binary code as address 1' when the loader (V1) needs to be called, and then read the binary code from the address to run. Similarly, the electronic device may obtain the storage address of the corresponding binary code as address 2' when the loader (F1) needs to be called, and then read the binary code from the address to run.

The game application may then issue a glatatchloader (1, V1), binding the vertex shader loader (V1) to program (1). The game application may also issue a glatatchloader (1, F1), binding the vertex shader loader (F1) to program (1). Thus, a vertex shader and a fragment shader are bound to program (1). Similar to the above operation, the electronic device may also update the correspondence table, and update the correspondence of the loader (V1) and the loader (F1) with the program (1) into the table entry. For example, table 3 shows an example of yet another update entry.

TABLE 3 Table 3

Project (program) ID	Shader ID	Storage address of source code	Memory address of binary code
				1	V1	Address 1	Address 1'
1	F1	Address 2	Address 2'

In the example of table 2, the item ID 1 may have a correspondence relationship, such as a binding relationship, with the shader ID V1 and the shader ID F1. It can also be seen from the correspondence shown in table 3 that the source code of the shader with ID V1 can be stored at address 1 and the binary code of the shader with ID V1 can be stored at address 1'. It can also be seen from the correspondence shown in table 3 that the source code of the shader with ID F1 can be stored at address 2 and the binary code of the shader with ID F1 can be stored at address 2'.

After issuing the instruction that the shader binds to the program, the game application may issue a glLinkProgram instruction 1 (e.g., glLinkProgram (1)), linking the program with bound loader (V1) and loader (F1) to the current application (i.e., the game application). Thus, during subsequent runs, the game application may use the loader (V1) and the loader (F1) to color by calling the program (1).

Illustratively, the game application may call program (1) by sending glauseprogram (1). Thus, the electronic device can determine that the corresponding shader includes a loader (V1) and a loader (F1) according to the corresponding relation shown in the table 3 when the program (1) is called, and determine the storage address of the binary code of each shader. By reading and running the code of the storage address, the purpose of calling the program (1) to perform coloring operation can be achieved.

Thus, the game application can achieve the purpose of performing a coloring operation by the shader.

It should be noted that, in combination with the description in table 1, the correspondence relationships shown in the foregoing tables 2 and 3 may also implement the correspondence between the entries in the foregoing table by adding pointers to the source code storage addresses (or the source code storage addresses and the binary code storage addresses) on the code segments corresponding to the shader IDs and/or the program IDs.

It will be appreciated that the shading operations are an integral part of the rendering operations, and have a significant impact on the time and power consumption overhead of the rendering operations. As the fineness of the displayed image of the electronic device increases, the coloring operation required is more complicated.

To meet the complex shading requirements, the amount of code that needs to be used by the shader is gradually increased, and the time consumption of the corresponding shading operation is also significantly increased. This manifests itself as time consuming procedures for rendering the image and an increase in power consumption overhead.

In order to reduce the rendering overhead of the shading operation, the scheme provided by the embodiment of the application replaces the original shader (namely, the shader used by the game application instruction) with a simplified shader, and the same or similar rendering functions are realized through simpler and faster shading operation. In addition, as the number of the shaders in the game running is large, and the situation that the same vertex shaders correspond to different fragment shaders or the different vertex shaders correspond to the same fragment shaders exists, the scheme provided by the embodiment of the application can ensure that the original shaders can be accurately matched to the corresponding simplified shaders through an accurate matching scheme, so that the rendering effect is ensured.

The following describes the schemes provided in the embodiments of the present application in detail with reference to the accompanying drawings. In the following description, an application program (such as a game application) may instruct an electronic device to perform a rendering operation including a coloring operation by issuing a rendering instruction stream. In the rendering instruction stream, the original shader to which the indicated program is bound may be a second shader. The simplified shader corresponding to the second shader may be the first shader. The plurality of to-be-simplified shaders stored in the electronic device may be fourth shaders, and the plurality of simplified shaders corresponding to the plurality of to-be-simplified shaders, respectively, may also be referred to as a plurality of third shaders. Among the plurality of fourth shaders, a shader that matches the second shader may also be referred to as a fifth shader. Further, the second shading instruction may correspond to the first instruction or the second instruction, and the first shading instruction may correspond to the fourth instruction. The first program may be program (1).

The image processing method provided by the embodiment of the application can be applied to the electronic equipment of the user. For example, the electronic device may be a mobile device with image processing capability, such as a mobile phone, a tablet computer, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR), a Virtual Reality (VR) device, a media player, or a wearable electronic device, such as a smart watch. The embodiment of the present application does not particularly limit the specific form of the apparatus.

By way of example, the electronic device for application of the solution provided in the embodiments of the present application may include a processor, an external memory interface, an internal memory, a universal serial bus (universal serial bus, USB) interface, a charge management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, an audio module, a sensor module, a key, a motor, an indicator, a camera, a display screen, a subscriber identity module (subscriber identification module, SIM) card interface, and the like. The sensor module may include, among other things, a pressure sensor, a gyroscope sensor, a barometric sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc. In some embodiments, the electronic device may further include a speaker, a receiver, a microphone, an earphone interface, and the like for implementing audio-related functions of the electronic device.

As one example, the Processor may include a central processing unit (Central Processing Unit/Processor, CPU), an application Processor (application Processor, AP), a modem Processor, a GPU, an image signal Processor (image signal Processor, ISP), and the like.

It will be appreciated that the above examples do not constitute a particular limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In other embodiments, the electronic device may have a layered architecture of software, and the different layers may be provided with respective corresponding modules or components for implementing the corresponding functions.

Exemplary, fig. 5 shows a schematic diagram of a software architecture provided in an embodiment of the present application. In this example, several layers may be included in the electronic device, each layer having a distinct role and division of labor. The layers communicate with each other through a software interface. Operating with electronic equipmentAn (Android) operating system is an example. The system may be divided into multiple layers, from top to bottom, an application layer (abbreviated as application layer 501), an application framework layer 502 (abbreviated as framework layer 502), a system library 503, and a hardware layer 504, respectively.

The application layer 501 may include a series of application packages, among other things. The application package may include camera, gallery, calendar, talk, map, navigation, WLAN, bluetooth, music, video, short message, etc. applications. In some embodiments of the present application, the application layer 501 may include an application that provides multimedia stream presentation functionality to a user. For example, various types of gaming applications (e.g. Etc.). As another example, various types of video applications may also be included in the application layer 501. In the process of running these application programs that need to perform image rendering (or coloring processing in image rendering), the solutions provided by the embodiments of the present application may be used to obtain corresponding beneficial effects. The following examples are each described by way of example as running a gaming application.

Framework layer 502 provides an application programming interface and programming framework for the application programs of application layer 501. Framework layer 502 includes some predefined functions. The framework layer 502 may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like. The window manager provides a window management service (Window Manager Service, WMS) that may be used for window management, window animation management, surface manager (surface manager), and as a transfer station for an input system. The content provider is used to store and retrieve data and make such data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebooks, etc. The view system includes visual controls that can be used to build applications. The display interface may be composed of one or more views. The resource manager provides various resources to the application. The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction. The notification manager may also be a notification that appears in the system top status bar in the form of a chart or scroll bar text. The activity manager may provide activity management services (Activity Manager Service, AMS) that may be used for system component (e.g., activity, service, content provider, broadcast receiver) start-up, handoff, scheduling, and application process management and scheduling tasks. The input manager may provide input management services (Input Manager Service, IMS), which may be used to manage inputs to the system, such as touch screen inputs, key inputs, sensor inputs, and the like. The IMS retrieves events from the input device node and distributes the events to the appropriate windows through interactions with the WMS.

In the embodiment of the present application, in order to implement the image processing method provided in the embodiment of the present application, an interception module, an acquisition module, a matching module, a compiling module, a replacement module, and the like may be provided in the framework layer 502. In other embodiments, more or fewer modules may be further included in the framework layer 502 to implement the solution provided in the embodiments of the present application, for example, in a case where the binary code of the simplified shader is already stored in the electronic device, the framework layer 502 may also be not provided with a compiling module, and the operation of the simplified shader may be directly implemented by calling the stored binary code.

The system library 503 may include a plurality of functional modules. For example: surface manager, media Framework, standard C library (Standard C library, libc), openGL, open graphics library of embedded systems (OpenGL for Embedded Systems, openGL ES), vulkan, sqlite, webkit, etc. In the embodiment of the application, taking a rendering mechanism of a rendering platform provided based on OpenGL as an example. In the embodiment of the present application, a RunTime (RunTime) may also be included in the system library 503. For example, the android-based runtime may be An Zhuoyun rows. The runtime may provide the functionality of a virtual machine. When the electronic device needs to run various types of code, the various codes can be executed in a virtual environment provided by the runtime. For example, in the compiling (or precompiled) process, the electronic device may input the source code into a compiler in the runtime, and obtain, through processing of the compiler, the binary code corresponding to the source code in the current virtual environment. Taking rendering using the OpenGL platform as an example, the electronic device may compile source code using a FXC HLSL compiler provided by Microsoft (Microsoft) in An Zhuoyun row, and then convert the result obtained by compiling into GLSL language format using a shader bytecode cross compiler (HLSLcc) to facilitate subsequent calls of the graphics library and GPU.

In this example, a hardware layer 504 may also be included in the electronic device. For example, hardware components for implementing electronic device functions may be included in hardware layer 504. For example, the hardware layer 504 may include a GPU for performing rendering processing, a CPU for performing data processing, a memory for performing data storage, and the like. The memory may include an internal storage medium embedded in the CPU or the GPU, a flash memory with a fast read-write function provided in the electronic device, a nonvolatile storage medium for storing a larger file in the electronic device, and the like.

In connection with the example of fig. 4, in the embodiment of the present application, a game application having an application layer in the software architecture shown in fig. 5 may issue a rendering instruction stream to an electronic device after a game starts to run. Correspondingly, the rendering instruction stream calls an interface of the graphic library to execute corresponding rendering operation through the framework layer. Thereby implementing the process of creating a shader as shown in fig. 4 to linking the program with the shader bound to the game application.

It should be noted that, in the embodiment of the present application, a portion of a shader that needs to be simplified (for example, a shader to be simplified) and a code of a simplified shader corresponding to the shader to be simplified may be stored in the memory of the electronic device. The shader to be simplified may be part or all of the shaders that need to be called in the running process of the application program. The code of the simplified shader stored in the memory may be source code or binary code. For these simplified shaders, the electronic device may replace the original shader bound on the program with one of these simplified shaders through the scheme provided by the embodiments of the present application. The specific matching substitution process will be set forth in detail in the following description. Therefore, in the running process of the follow-up game application, when the shader is required to be called, the code of the simplified shader can be run to carry out the shading operation by calling the corresponding program, so that the power consumption and time cost of the shading operation are saved.

In various implementations of the present application, the simplified shader corresponding to the to-be-simplified shader may be determined in an offline manner. In this way, the code of the simplified shader can be stored in the electronic device before the game starts to run, so that the replacement and use of the simplified shader can be realized directly through a matching mechanism during the running of the game. In other implementations of the present application, the process of determining the code of the shader to be simplified may also be performed in an electronic device. For example, when the electronic device is dormant or the processing load is low, the code of the shader to be simplified is determined and stored in the electronic device for subsequent replacement use.

The determination of the shader to be simplified will be described below taking the determination of the code of the shader to be simplified by way of example. In some embodiments, as shown in FIG. 6, the electronic device may determine the code of the simplified shader of the shader to be simplified by the following steps.

S601, determining a rendering instruction stream issued by a game application in the game running process.

The rendering instruction stream may include call instructions to shaders, loader source code, etc. information. As an example, the electronic device may record the pat file at the time of game running through a Patrace or the like, so that the information is determined through the acquired pat file.

S602, determining a shader to be simplified according to the rendering instruction stream.

The electronic device determines a heavy-load loader pair in a game running scene according to a pat file stored with information such as a call instruction of a shader and loader source codes, and marks objects drawn by the loader.

The electronic equipment can analyze the pat file through tools such as a pat processing tool (pat processor) and the like, and the function of the step is realized. It should be appreciated that multiple scenarios may be included during game play, with different loads in different scenarios. The load may refer to a power consumption load, an operational load, and the like. The electronic device may determine, by analyzing the pat file, a scenario with a load greater than a preset load threshold as a heavy load scenario. And takes the pairs of shaders in the heavy-load scene as shaders to be simplified. The heavy load scene may be one or a plurality of heavy load scenes.

S603, acquiring and determining a corresponding abstract syntax tree according to the shader to be simplified.

In this example, the electronic device may take as input the shader to be simplified, generating an abstract syntax tree (Abstract Syntax Tree, AST) of the corresponding vertex shader and fragment shader. For each vertex shader or fragment shader that is a shader to be simplified, the electronic device may generate a corresponding AST, thereby performing the following steps to determine the code of the respective corresponding simplified shader.

S604, performing code simplification on the to-be-simplified shader according to a preset simplification algorithm to obtain at least one shader variant.

In this step, the electronic device performs the loader code simplification according to the AST of the shader to be simplified according to the preset simplification algorithm. The preset simplifying algorithm may include any one of the following: expression deletion, code movement, etc. Thereby obtaining shader variants (shaders) of the shader to be simplified. It will be appreciated that the preset reduced algorithm may include a plurality of, for example, a plurality of expression deletion mechanisms and/or a plurality of code movement mechanisms. Thus, each shader to be simplified may obtain a plurality of different shader variants of the shader to be simplified.

It should be appreciated that since the acquisition of each variant is by simplifying the code acquisition of the shader to be simplified, the operating overhead of all variants is less than the operation of the shader to be simplified. For example, the code amount of some varieties is smaller than that of the shader to be simplified, so that the varieties are used for replacing the shader to be simplified to render, and the expenditure of computational effort can be reduced. As another example, one variant may have a smaller run time than the shader to be simplified, thereby replacing the shader to be simplified with those variants for rendering, which may reduce the overhead on the duration.

S605, screening at least one shader variant, and determining a simplified shader of the to-be-simplified shader.

Illustratively, in this example, the electronic device may perform the screening of shader variations based on rendering image quality errors and/or time gain. The screening process may be performed based on a pareto model.

Wherein the time benefit may be determined by replacing the shader to determine the run time difference.

As a possible implementation, in connection with fig. 7, a frame image stream including frames 0-Q is taken as an example during game running. Wherein the frame P may comprise a shader to be simplified. After the processing of S604, X varieties of the frame P can be acquired. Such as frame P1, frame P2, … …, frame PX, etc. The original frames and the run times after the replacement of the variants can be run and recorded separately. For example, run frames 0-Q of the original frame (i.e., including frame P) are time duration 0. The run time of variant 1 (i.e. comprising frame P1) is duration 1. Similarly, the run time of variant X (i.e., including frame PX) is duration X.

In this example, the drawing image quality error may be determined by an image quality error (error) evaluation of the delay rendering.

As a possible implementation, in connection with fig. 8, the example of a frame image stream including frames 0-Q during game operation is continued. Wherein the frame P may comprise a shader to be simplified. After the processing of S604, X varieties of the frame P can be acquired. Such as frame P1, frame P2, … …, frame PX, etc. The original frames may be run separately and the rendered results after each variant replaced. The rendering result of the original frame is used as a reference image quality, and the image quality difference between each variety and the original frame can be obtained through image quality evaluation. If variation 1 corresponds to the image quality average difference 1, variation 2 corresponds to the image quality average difference 2, and so on, variation X corresponds to the image quality average difference X. In different implementations, the image quality evaluation may be performed in different ways, for example, any of the following methods may be employed: image Peak Signal-to-Noise Ratio (PSNR), structural similarity (Structure Similarity Index Measure, SSIM), multi-scale structural similarity (Multi Scale Structure Similarity Index Measure, MS-SSIM), information-based Weighted structural similarity scheme (Evaluation of Information Content-Weighted SSIM, IW-SSIM), and the like.

In some embodiments of the present application, a simplified shader is exemplified based on time yield determination. The electronic device may take the shortest-duration variant of the variants as a simplified shader of the shader to be simplified based on the scheme shown in fig. 7. Thus, by replacing the simplified shader, the maximum time benefit can be obtained.

In other embodiments of the present application, a simplified shader is exemplified by a determination based on rendering image quality errors. The electronic device may use the variant with the smallest image quality difference among the variants as a simplified shader of the shader to be simplified based on the scheme shown in fig. 8. In this way, by replacing the simplified shader, the image quality can be ensured to the greatest extent.

In other embodiments of the present application, a simplified shader may be determined as an example based on time gain and rendering image quality errors. The electronic device may obtain the time gain and the image quality difference corresponding to each variety based on the schemes of fig. 7 and fig. 8. Based on this, the electronic device can acquire the frame difference map as shown in fig. 9. Where the abscissa is time gain and the ordinate is image quality error, each variant stream may include a rendering of multiple frame images, so the gain of each frame image may be shown in fig. 9. Based on different scene requirements, the variants meeting the requirements can be selected as simplified shaders. For example, as shown in fig. 9, since most of the frame images of variant 1 have small image quality errors, variant 1 can be selected as a simplified shader in a scene where the image quality needs to be ensured. Correspondingly, most of the frame images of variant 2 have higher time gain than variant 1, so that variant 2 can be selected as a simplified shader in a scene where the rendering time is required to be shortened.

Thus, through the operations of S601 to S605, the electronic device may determine the simplified shader corresponding to the to-be-simplified shader.

In this example, the electronic device may store in memory information related to the determined respective to-be-simplified shaders and corresponding simplified shaders to facilitate replacement and use during game play.

Illustratively, the information stored in memory may include the original shader (i.e., the shader to be simplified) length, the code of the original shader, the storage location of the corresponding simplified shader, and so forth.

Wherein the length of the original shader and the code of the original shader can be used to perform shader matching so that the electronic device can accurately find the shader requiring simplified replacement from the rendering instruction stream issued by the gaming application after the game is run. After determining the shader to be replaced, the electronic device can acquire the simplified shader according to the storage address of the corresponding simplified shader, so that the simplified shader can be used for replacing the original shader, and the simplification of the shader is realized. It should be noted that in different embodiments, the code of the original shader may have different implementations. For example, taking determining the shader to be simplified in the rendering instruction stream through source code matching as an example, the code of the original shader in the corresponding relationship may be the source code of the original shader and/or a hash (hash) value corresponding to the source code of the original shader. For another example, taking the determination of the shader to be simplified in the rendering instruction stream by binary code matching as an example, the code of the original shader in the correspondence may be the binary code of the original shader and/or a hash (hash) value corresponding to the binary code of the original shader.

The electronic device may store the above-mentioned related information of each of the to-be-simplified shaders and the corresponding simplified shader in the electronic device in the form of a file. For example, the file storing information about each simplified shader and the corresponding simplified shader may be referred to as a map file. In some embodiments, the map file may be stored under the root directory of the game application so that the corresponding game application can be used normally when running.

In connection with the foregoing description, in some embodiments, for a pair of shaders that are a pair of a vertex shader and a fragment shader, information corresponding to each of the vertex shader and the fragment shader may be stored separately in the electronic device. Illustratively, table 4 shows an example of correspondence of a shader to be simplified and a simplified shader included in a map file.

TABLE 4 Table 4

In the correspondence of table 4, the original vertex shader length vslen, and the original fragment shader length fslen may be stored. Taking source code matching as an example, the corresponding relationship can also comprise original vertex shader source code and/or hash value matching. For example, the original vertex shader source code may be a loader, overt, and the hash value corresponding to the original vertex shader source code may be a hash, overt. The original fragment shader source code may be a loader.frag, and the hash value corresponding to the original fragment shader source code may be a hash.frag.

As shown in Table 4, the storage location of the reduced vertex shader (i.e., reduced shader) is also included in the correspondence. Also included in this correspondence may be the storage location of the reduced fragment shader (i.e., fragment shader) frag x glsl.

Thus, after the length, source code (and/or hash value) match is successful, it may be determined that the storage location in the current entry stores the corresponding reduced shader.

The shader simplification mechanism provided by the embodiments of the present application will be described in detail below with reference to the shader to be simplified in the foregoing description. In this case, the map file including each of the to-be-simplified shaders and the corresponding relation (the corresponding relation shown in table 4) of the simplified shaders is stored in the electronic device, and the electronic device has the composition shown in fig. 4 as an example.

For example, in connection with fig. 10. The game application may issue a stream of rendering instructions during the loading process. Instructions for creating a shader, instructions for creating a program, instructions for compiling a shader, instructions for binding a shader to a program, and instructions for linking a program to a gaming application may be included in the rendering instruction stream. In conjunction with the foregoing description, in other embodiments, instructions for compiling a shader may not be included in the rendering instruction stream.

In the rendering instruction stream, a first instruction, a second instruction, and a third instruction may be included. The first instruction may be an ID carrying the program or one or more instructions that can be used to obtain the program ID. The second instruction may be one or more instructions carrying the ID and/or storage address of the shader bound to the program, or which can be used to obtain the ID and/or storage address of the shader bound to the program. The third instruction may be one or more instructions in the rendering instruction stream that are different from the first instruction or the second instruction.

It should be appreciated that in some scenarios, there are instructions that can be used to both carry a program ID (i.e., included in the first instruction) and to obtain the ID and/or memory address of the corresponding shader (i.e., included in the second instruction). Thus, in different implementations, the first instruction and the second instruction may include at least partial overlap therein. In contrast, in other scenarios, the instructions included in the first instruction and the second instruction may also be completely different.

In the scheme provided by the application, after the game starts to be loaded, each instruction in the rendering instruction stream can be respectively input to the interception module. The interception module may be configured to intercept a first instruction and a second instruction in a rendering instruction stream issued by the game application. The intercept module may also pass the first instruction and the second instruction to the fetch module.

For other instructions (e.g., a third instruction) in the rendering instruction stream, such as a glCreateHader function, a glCreatProgram function, a glCompileHader function, a glAttachShader function, etc., the interception module may call back the third instruction to the graphics library to ensure implementation of native logic.

In addition, the interception module may pass the first instruction to the graphics library in addition to passing the first instruction to the acquisition module.

As an example, the first instruction is intercepted. The interception module may implement interception of the first instruction through Hook (Hook) technology. The interception module can specifically intercept the instruction by the following modes: when the game application is started, the interception module modifies the function pointer list in the thread local store (Thread Local Storage, TLS) of the game thread, and replaces the graphic function pointer recorded by the list with the replacement pointer. A graphic function pointer which is a function pointer pointing to an implementation function of the graphic library; instead of pointers, function pointers are function pointers that point to implementation functions of other modules (e.g., acquisition modules, etc.). For example, the function pointer of the implementation function of the graphic library of the first instruction recorded in the function pointer list may be replaced with a function pointer of the implementation function of the acquisition module. Thereby facilitating other modules to execute corresponding operations according to the instructions intercepted by the interception module. Then, through Hook (Hook) API, the interception module can intercept the instruction and transmit the instruction to other modules when the application program issues the corresponding instruction to be intercepted. The interception of the second instruction is similar to the interception of the first instruction described above, and will not be repeated here.

The acquisition module can be used for acquiring the ID of the current program according to the first instruction from the interception module. The acquisition module is further configured to acquire an ID and a storage address of an original shader bound on the current program according to the second instruction from the interception module. For example, the obtaining module may parse the first instruction, thereby confirming that the ID of the current program is 1. For another example, the obtaining module may parse the second instruction, so as to obtain that the shader bound to the current program includes an original shader loader (V1) with an ID of V1 and a source code storage address of address 1, and an original shader loader (F1) with an ID of F1 and a source code storage address of address 2.

In combination with the relationship between the first instruction and the second instruction, in this embodiment of the present application, when the first instruction is different from the second instruction, the obtaining module may directly call back the first instruction to the graphics library after obtaining the current program ID. When the first instruction is the same as the second instruction, then no callback to the first instruction is executed (as illustrated in FIG. 5), but subsequent operations continue to be performed.

In other embodiments of the present application, the obtaining module may be further configured to obtain the number and length of the shapers mounted in the first instruction. The number and length of the shaders may be used for subsequent shader matching. In some implementations, the acquisition module may first determine the number of readers. When the number of the shaders is 2, the number indicates that the shaders can form a pair of shaders, and then the acquisition module can further determine the type and the respective length of the mounted pair of shaders.

The acquisition module may transmit information such as the ID, the storage address, the length of the loader pair, etc. of the original shader to the matching module. For example, the acquisition module may transmit the ID of the reader (V1) and the source code storage address (address 1) to the matching module. The acquisition module may also transmit the ID of the reader (F1) and the source code storage address (address 2) to the matching module. For another example, the obtaining module may transmit the length (length V1) of the reader (V1) to the matching module. The obtaining module may also transmit the length (length F1) of the reader (F1) to the matching module.

And the matching module can be used for matching according to the ID, the storage address, the length of the loader pair and other information of the original shader from the acquisition module, and determining the simplified shader corresponding to the original shader.

In this example, the matching mechanism may be source code matching based on source code.

For example, in conjunction with the foregoing determination mechanism for a simplified shader, the electronic device may store a plurality of codes for the simplified shader, which may be uncompiled source code. The electronic device may further store simplified shaders corresponding to the respective to-be-simplified shaders. The correspondence may be stored in a map file including the map shown in table 4. In the following description, the correspondence in the map file may also be referred to as a simplified correspondence.

That is, the electronic device can maintain the correspondence between the codes of the respective simplified shaders and the codes of the original shaders before simplification by simplifying the correspondence. By way of example, table 5 below shows a specific illustration of this simplified correspondence.

TABLE 5

In the example of table 5, one can understand a specific implementation of the corresponding example of table 4. In this example, the original vertex shader may be V1 in length, its source code may be shader source code segment 1 to be simplified, and the corresponding simplified vertex shader may be stored at address 3a. The original fragment shader may be F1 in length, the source code may be the shader source code segment 2 to be simplified, and the corresponding simplified fragment shader may be stored at address 4a.

In the above example, the source code segment of the original shader is directly stored in the simplified correspondence relationship. In other embodiments of the present application, the source code of the original shader may also be obtained by pointer jumping from the stored address. For example, table 6 gives one example of indicating source code by a storage address.

TABLE 6

Based on the simplified correspondence shown in this table 6, when the original vertex shader source code needs to be used, the source code can be read from address 3. Correspondingly, when the original fragment shader source code needs to be used, the source code can be read from address 4.

In some embodiments, the matching module may read all of the shader source code segments to be simplified in the simplified correspondence from memory before performing the matching operation.

The matching module may read the corresponding code segment from the memory according to the memory address of the original shader bound on the current program from the acquisition module. For example, the matching module may read the source code of the reader (V1), i.e., the source code segment V1, from address 1. The matching module may read the source code of the reader (F1), i.e., the source code segment F1, from address 2.

Then, the matching module may perform a matching operation with the source code segment of the shader to be simplified using the source code segment of the original shader.

In some embodiments of the present application, the matching mechanism may include: the matching module performs length matching according to the length of the code segment, and performs code matching under the condition that the length matching is successful.

Where a successful length match may refer to the code lengths of the two code segments being matched being identical. The successful code match may refer to source code match, hash value match, and the like. Taking source code matching as an example, when all codes of two code segments to be compared are completely consistent, the matching is considered to be successful. Conversely, when the codes of the two code segments being compared include at least partially different, then the match is deemed to have failed.

Take the example of matching the source code segment V1. The matching module may match the source code segment V1 with all the to-be-simplified shader code segments in the map file by length and source code respectively. When there is a successful match, the simplified shader corresponding to the matched code segment of the to-be-simplified shader may be the simplified shader corresponding to the source code segment V1.

The matching process is similar for source code segment F1.

In order to more clearly illustrate the matching process provided in the embodiments of the present application, the examples of table 6 are combined.

Taking the example that the shader to be simplified stored in address 3 matches with the source code segment V1, the shader to be simplified stored in address 4 matches with the source code segment F1. After the matching process, the matching module may confirm that the source code storage address of the simplified shader corresponding to the loader (V1) is address 3a. The matching module may confirm that the source code storage address of the simplified shader corresponding to the loader (F1) is address 4a.

In the following example, a simplified shader corresponding to the loader (V1) is referred to as a loader (V2), its storage address is referred to as an address 3a, a simplified shader corresponding to the loader (F1) is referred to as a loader (F2), and its storage address is referred to as an address 4a.

Under the condition that the matching is successful, the matching module can transmit the ID and the storage address of the simplified shader to the replacement module for subsequent operation. For example, the matching module may transmit the ID (e.g., V2) of the simplified shader corresponding to the loader (V1) and the memory address (e.g., address 3 a) to the replacement module. The matching module may transmit the ID (e.g., F2) of the simplified shader corresponding to the loader (F1) and the memory address (e.g., address 4 a) to the replacement module.

Correspondingly, in the case of a matching failure, the table is not required to be simplified by the loader (V1) and the loader (F1) currently bound to the program (1). The matching module may transmit information of the matching failure to the interception module. In this way, the interception module may transmit the second instruction included in the rendering instruction stream to the graphics library, so that when the program (1) is subsequently called, a corresponding coloring operation may be performed according to the loader (V1) and the loader (F1).

And the replacing module can be used for replacing the shader information currently bound on the program (1) according to the related information (such as the ID of the simplified shader and the storage address of the simplified shader) of the simplified shader sent by the matching module.

It will be appreciated that when the matching module transmits information about the reduced shader to the replacement module, at least one ID, which may be an ID of the reduced shader, and a memory address, which may store the source code of the reduced shader, may be received for the replacement module. Even if the matching module does not inform the replacement module of the information to simplify the information of the shader, the replacement module can consider the received information to simplify the information of the shader and perform the corresponding replacement operation.

In some embodiments of the present application, to confirm the current program, the replacement module may further acquire information of the program from the acquisition module. In combination with the foregoing description, the interception module may transmit the first instruction to the acquisition module, and the acquisition module may acquire the ID (e.g. ID is 1) of the current program according to the first instruction. In this example, the replacement module may acquire the ID of the current program as 1 from the acquisition module. In other embodiments of the present application, the replacing module may further acquire the first instruction from the acquiring module or the intercepting module, so as to determine that the ID of the current program is 1 according to the first instruction. Then the replacement module can confirm that the simplified shader sent by the matching module is replaced onto program (1).

As one possible implementation, the replacement module may implement simplified shader replacement through unbinding, binding operation procedures.

For example, the replacement module may unbind the loader (V1) and the loader (F1) on the current program (1) from the program (1) through a gldetachloader function. Thereby obtaining a program (1) that does not bind any shader.

The replacement module can bind a loader (V2) and a loader (F2) to the program (1) through a glatatchloader function.

Through the above replacement operation, the shader currently bound on the program (1) may be a simplified shader loader (V2) as well as a loader (F2).

After the replacement operation, the correspondence relationship as in the above table 3 may be updated to the correspondence relationship as shown in the following table 7.

TABLE 7

Project (program) ID	Shader ID	Storage address of source code	Memory address of binary code
				1	V2	Address 3a	Address 1'
1	F2	Address 4a	Address 2'

It can be seen that in the updated correspondence, the shader bound on program (1) may include: the loader (V2), namely the ID is V2, the source code storage address is address 3a, and the binary code storage address is address 1'. The shader bound on program (1) may further include: the loader (F2), i.e. ID is F2, the source code memory address is address 4a, and the binary code memory address is address 2'.

It should be noted that, in other embodiments of the present application, before the above-mentioned substitution for source code is performed, the substitution module may also substitute the shader resource of the simplified shader onto the program (1) through the glShaderSource instruction, so as to facilitate other threads or function calls in the subsequent rendering process.

In the above example, the source code of the simplified shader currently bound to program (1) has not been compiled yet, so the memory address of the binary code may be erroneous in the correspondence after the above substitution. For example, in this example, the memory address of the binary is also the memory address of the binary of the original shader.

In the case of indicating the storage address in the form of a pointer, it may be expressed that the pointer for indicating the storage address of the binary code currently bound on the program (1) may be empty, or that the content indicated by the pointer is erroneous, or the like.

In order to avoid problems in the process of subsequently calling the binary code, the replacement module can transmit the storage address of the source code to the compiling module after completing the replacement of the source code information on the program (1).

For example, the replacement module may transmit the related information (e.g., ID V2, address 3a, and e.g., ID F2, address 4 a) of the simplified shader to the compiling module, respectively. The compiling module may read the source code segment of the loader (V2) from the address 3a to compile, and obtain a corresponding binary code segment for storage, for example, in the address 3b. The compiling module may read the source code segment of the reader (F2) from the address 4a to compile, and obtain a corresponding binary code segment for storage, for example, in the address 4b.

The compiling module may transmit the related information of the binary code after compiling (e.g. the ID is V2, the storage address of the binary code is address 3b, and the storage address of the binary code is address 4 b) to the replacing module, so that the replacing module updates the foregoing relationship. Such as updating a pointer to a memory address of the binary code, or updating a program indicating that there is a memory address of the reduced shader binary code, etc. Exemplary, the correspondence after updating is shown in the following table 8:

TABLE 8

Project (program) ID	Shader ID	Source code storageStorage address	Memory address of binary code
				1	V2	Address 3a	Address 3b
1	F2	Address 4a	Address 4b

In this way, the correspondence relation before and after the binary update is compiled by the compiling module is compared, and it can be seen that the storage address storing the binary code has been updated to the address 3b storing the binary code of the loader (V2) correctly and the address 4b storing the binary code of the loader (F2) correctly in the correspondence relation.

In other embodiments, instead of the replacement module sending the address of the reduced shader to the compilation module to instruct the compilation module to perform the compilation, the matching module may also send the ID of the reduced shader and the memory address to the compilation module to perform the compilation after determining the reduced shader. The compiling module may obtain the code segments of the reduced shader from the corresponding address for compiling and store the corresponding binary in memory. The compiling module may transmit the storage address of the binary code and the corresponding shader ID to the replacing module after the compiling is completed, so that the replacing module replaces the pointer of the binary code storage address of the shader bound on the program.

By the foregoing replacement procedure, the replacement module can bind the simplified shader to the program (1). The replacement module may then send a new second instruction (e.g., called a fourth instruction) to the graphics library with the updated binding.

For example, in some embodiments, the replacement module may send a fourth instruction to the graphics library. The fourth instruction may carry the ID and/or the storage address of the shader bound on the current program, or the fourth instruction may be used to obtain the ID and/or the storage address of the shader bound on the current program.

In connection with the foregoing description, according to the fourth instruction, the shader currently bound to the program (1) includes a loader (V2), the ID of the loader (V2) is V2, the source code storage address is 3a, and the binary code storage address is 3b. The shader bound on the current program (1) further comprises a loader (F2), wherein the ID of the loader (F2) is F2, the source code storage address is 4a, and the binary code storage address is 4b. Wherein, the loader (V2) may be a simplified shader of the loader (V1). The loader (F2) may be a simplified shader of the loader (F1).

In other embodiments, the replacement module may send a fourth instruction to the interception module, which may send the fourth instruction to the graphics library after receiving the information.

Thus, in the event that the match is successful, the graphics library may determine, based on the fourth instruction, that the shader bound on program (1) includes a simplified shader loader (V2) and a simplified shader loader (F2). When program (1) is subsequently invoked, a simplified shader loader (V2) and loader (F2) may be used to perform the shading operation. Thereby improving coloring efficiency.

Correspondingly, due to the callback mechanism provided in the embodiment of the present application, in the case of a matching failure, the graphics library may determine, according to the native second instruction, that the shader bound on the program (1) includes the original shader loader (V1) and the original shader loader (F1). Correspondingly, when the program (1) is subsequently called, the original shader loader (V1) and the loader (F1) can be used for shading operation.

In the above example, the solution provided in the embodiment of the present application is described from the viewpoint of the module function with reference to fig. 10. The following description of the solution provided in the embodiment of the present application is continued with reference to the interaction between the modules shown in fig. 11.

For example, referring to fig. 11, the gaming application may issue a rendering instruction stream including a first instruction, a second instruction, and a third instruction, respectively.

The interception module may call back a third instruction to the graphics library to implement the native logic.

In addition, each module in the framework layer can be mutually matched, and the scheme provided by the embodiment of the application is realized according to the following interaction flow.

S1101, after the game application issues the rendering instruction stream, the interception module intercepts the first instruction and the second instruction.

For example, the rendering instruction stream may include a first instruction, a second instruction, and a third instruction. The first instruction may be an ID carrying the program or one or more instructions that can be used to obtain the program ID. The second instruction may be one or more instructions carrying the ID and/or storage address of the shader bound to the program, or which can be used to obtain the ID and/or storage address of the shader bound to the program. The third instruction may be one or more instructions in the rendering instruction stream that are different from the first instruction or the second instruction.

In different implementations, the first instruction and the second instruction may implement their corresponding functions through the same function or functions, that is, the first instruction and the second instruction may be the same. In other embodiments, the function corresponding to the first instruction and the function corresponding to the second instruction may also be different. In connection with the example in fig. 5, where the first instruction and the second instruction are the same or different, the callback mechanism for the instructions may be different in this scheme. For example, in the case that the first instruction and the second instruction are the same, the electronic device may only execute the callback to the third instruction before completing the whole flow (such as issuing the fourth instruction), thereby ensuring that the native logic is implemented smoothly. Correspondingly, when the first instruction and the second instruction are different, the electronic device may call back the first instruction by using the corresponding module after the first instruction is used, for example, after the ID of the current program is obtained according to the first instruction.

In the following examples, the first instruction and the second instruction are the same.

S1102, the interception module sends the first instruction and the second instruction to the acquisition module.

S1103, the acquisition module acquires the ID of the current program according to the first instruction, and acquires the ID, the storage address and the length of the original shader which is bound according to the second instruction. The original shader may include a loader (V1) and a loader (F1), among others.

It should be appreciated that the creation, compilation, binding, etc. of the shader has been completed before the game application issues the second instruction. That is, the program (1) has bound the loader (V1) and the loader (F1), and the correspondence relationship as shown in table 3 may be stored in the memory by pointers, and/or a lookup table, and/or correspondence of fixed fields.

In this example, the acquisition module may acquire the ID of the bound original shader and the storage address according to the second instruction. For example, the acquisition module may acquire the storage addresses of shaders of the loader (V1) and the loader (F1) through the glgetatachedshaders function. The loader (V1) and the loader (F1) are the original shaders in this example. In other embodiments of the present application, the obtaining module may also search for and obtain the ID, the storage address, and the length of the original shader according to the program ID indicated by the first instruction.

S1104, the acquisition module transmits the ID, the storage address and the length of the original shader to the matching module.

Illustratively, the fetch module may transmit the ID (i.e., V1) of the original vertex shader loader (V1), as well as the memory address (address 1) to the match module. The acquisition module may also transmit the ID (i.e., F1) of the original fragment shader loader (F1), as well as the memory address (address 2), to the matching module.

The obtaining module may also transmit the length V1 of the loader (V1) and the length F1 of the loader (F1) to the matching module.

S1105, the matching module performs length matching on the original shader.

In the case where the length matching is successful, the following S1106 is performed. In the case of a match failure, the following S1113-S1114 are performed.

S1106, the matching module performs code matching on the original shader.

In the case where the code matching is successful, the following S1107 is performed. In the case of a match failure, the following S1113-S1114 are performed.

In this example, the code matches may include source code matches or hash value matches, or the like.

Illustratively, the matching module may read the stored source code of the plurality of shader to be simplified from the memory. The matching module may further obtain source codes of the original shaders according to the storage addresses of the original shaders from the obtaining module. And matching the source code of the original shader with the source code of the shader to be simplified.

The matching mechanism may include source code matching and/or hash matching, for example. The successful matching means that among the source codes of the plurality of shaders to be simplified, the source code of one of the shaders to be simplified is identical to the source code of the original shading. Correspondingly, the failure of matching means that, among the source codes of the plurality of shaders to be simplified, no source code of the shader to be simplified is identical to the source code of the original shader. Taking hash matching as an example. Each code segment (e.g., source code of the original shader, or source code of the shader to be simplified) may correspond to a hash value. When the codes included in the code segments are different, the hash values are different. Then, a successful match may refer to the hash value of the two code segments that are being matched being the same. Correspondingly, a match failure may refer to the hash values of the two code segments that are being matched being different.

In case of successful matching, it is determined that the simplified shader exists in the current original shader, and a simplification process is required. Taking a successful match of the loader (V1) and the loader (F1) as an example. The simplified shader of the loader (V1) may be the loader (V2), and the storage address of the source code may be the address 3a. The simplified shader of the loader (F1) may be a loader (F2), and the storage address of the source code may be an address 4a. The matching module may continue to execute S1106 below.

Correspondingly, under the condition of matching failure, the current original shader is determined not to need to be subjected to simplification processing.

S1107, the matching module sends the ID of the simplified shader and the storage address to the replacement module.

For example, the matching module may send the ID of the reader (V2) (i.e., V2) and its storage address (address 3 a) to the replacement module. The matching module may send the ID of the reader (F2), i.e. F2, and its memory address (address 4 a) to the replacement module.

S1108, binding the simplified shader on the current program by the replacement module. The current program may be program (1). The simplified shader may be bound to the program (1) before the original shader is bound to the program (1).

For example, the replacement module may acquire information of the current program from the acquisition module before performing the S1107. For example, the ID of the current program is acquired from the acquisition module (i.e., 1). The replacement module may bind the simplified shader to the program (1) by unbinding operations for the loader (V1) and the loader (F1) and binding operations for the loader (V2) and the loader (F2). A simplified substitution of the shader with the original shader is achieved.

S1109, the replacement module transmits the ID of the simplified shader and the storage address to the compiling module.

Illustratively, the replacement module may transmit the ID and memory address (address 3 a) of the loader (V2), the ID and memory address (address 4 a) of the loader (F2) to the compilation module to effect conversion of source code into binary code.

S1110, compiling the source codes of the simplified shader by a compiling module.

S1111, the compiling module stores the binary code of the simplified shader, and sends the ID of the simplified shader and the storage address of the binary code to the replacing module.

Illustratively, the compiling module may obtain the source code of the loader (V2) from address 3a in memory and compile the source code. The compiling module may store the binary code obtained by compiling at address 3b of the memory.

The compiling module may obtain the source code of the loader (F2) from the address 4a in the memory and compile the source code. The compiling module may store the binary code obtained by compiling at address 4b of the memory.

In this example, the compiling module may also send information indicating that the identifier is V2 and the address is 3b to the replacing module. The compiling module may also send information indicating that the identity is F2 and the address is 4b to the replacing module. To inform the replacement module of the storage location of the binary code.

S1112, the replacement module updates the identification of the simplified shader bound on the current program, which points to the binary code.

Illustratively, the shader and binary code storage locations are maintained by pointers. The replacement module may update the pointer to the corresponding shader based on the ID of the reduced shader from the compilation module and the memory address of the binary code. The updated pointer may be: the pointer of the loader (V2) may point to the address 3b so that the binary code of the loader (V2) may be obtained from the address 3 b. The pointer of the loader (F2) may point to the address 4b so that the binary code of the loader (F2) may be obtained from the address 4 b.

S1113, the replacement module sends a fourth instruction to the graphic library.

The program linked to the fourth instruction may still be the same program (1) as the first instruction or the second instruction. However, unlike the second instruction, the fourth instruction may be bound to the linked program (1) by a reduced vertex shader (e.g., loader (V2)) and a reduced fragment shader (e.g., loader (F2)).

And 1114, the matching module sends matching failure information to the interception module.

S1115, the interception module transmits the first instruction and the second instruction to the graphic library.

Through the schemes of S1114 and S1115, in the case that the original shader bound to the program does not need to be simplified, the native logic can be maintained, and the program bound to the original shader is linked to the current game application, so as to facilitate the subsequent call.

In this way, the replacement of the shader during the loading phase is completed. Thus, the replaced simplified shader may be called to perform shading operations during subsequent game play.

For example, the game application may issue a glauseprogram instruction. The gluusepgram instruction may be used to instruct the use of program (1) for shading operations. The graphics library may invoke a corresponding application interface, instructing the GPU to execute the shading instructions corresponding to the program. The shading instruction corresponding to the program may be an instruction for performing a shading operation using the program (1). The GPU may obtain the program-bound shader from memory. For example, the GPU may determine that the storage address of the binary code of the vertex shader loader (V2) to which the program (1) is bound is the address 3b through the corresponding relationship between the program and the IDs and the storage addresses of the shaders bound to the program. The GPU may determine, through table 8, that the memory address of the binary code of the fragment shader loader (F2) to which program (1) is bound is address 4b. The GPU may then perform the shading operation using the program-bound shader. For example, the GPU may perform vertex shading of the model by running a code segment addressed to address 3b based on vertex color data issued by the gaming application. The GPU may perform pixel shading of the model by running a code segment addressed to address 4b, based on pixel color data issued by the gaming application.

Thus, since the binary codes stored by address 3b and address 4b are both binary codes of a simplified shader, less load and time consuming effects can be obtained than running the original shader.

The foregoing description of the embodiments of fig. 10 and 11 will enable those skilled in the art to fully understand the embodiments provided herein. The code matching mechanism is taken as a source code matching example. In other embodiments of the present application, matching may also be based on binary codes.

For example, in the simplified correspondence relation stored in the map file, the length information may be the binary length of the shader to be simplified, and the code of the corresponding simplified shader may also be stored with the binary code. Thus, after the simplified shader is determined through the length matching and the binary matching, the binary code of the corresponding simplified shader can be directly called without compiling through a compiling module. The cost of the rendering time can be further improved.

It will be appreciated that binary-based alternatives or source-code based alternatives may be chosen as desired in different scenarios. Wherein, in a scene of focusing on the improvement of rendering efficiency, a binary alternative scheme can be adopted. In the scenario where storage space is of concern, source-based alternatives may be employed, as the code amount of the source code is generally smaller than the corresponding binary code amount.

On the basis of the foregoing descriptions of fig. 10-11, please refer to fig. 12, an embodiment of the present application provides a flowchart of an image processing method. The scheme can be applied to electronic equipment. As shown in fig. 12, the scheme may include:

s1201, the electronic device acquires the second shader based on the rendering instruction stream. The second shader may be an original shader. For example, the electronic device may obtain an identification and a storage address of the second shader based on the stream of rendering instructions. The storage address of the second shader stores codes of the second shader, and the codes can be source codes and/or binary codes.

S1202, the electronic device obtains the identification of the first program based on the rendering instruction stream. Illustratively, the identity of the first program may be the ID of program (1), such as 1.

S1203, the electronic device acquires the code of the second shader based on the storage address of the second shader.

And S1204, the electronic equipment performs length matching and code matching on the codes of the second shader and the codes of the fourth shaders so as to acquire a fifth shader.

The code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader. Then the length of the fifth shader is also the same as the length of the second shader.

The fifth shader corresponds to the first shader. That is, the first shader may be a simplified shader of the fifth shader. In this example, the fourth shader may be a shader to be simplified. The codes of the fourth shaders include codes of the fifth shader. The codes of the fourth shaders can be stored in the electronic device, the codes of the fourth shaders are in one-to-one correspondence with the third shaders, and the codes of the fourth shaders comprise source codes or binary codes.

And S1205, the electronic equipment acquires the first shader through the fifth shader.

The electronic device obtains code for a plurality of third shaders. The third shader may be a simplified shader corresponding to the shader to be simplified. In this example, the code of the plurality of third shaders may include code of the first shader. For example, the code of the plurality of third shaders may be stored in the electronic device. Then, the electronic device may read the codes of the plurality of third shaders from the addresses storing the codes of the plurality of third shaders, respectively.

S1206, the electronic device replaces the second shader with the first shader.

The electronic device binds the first shader to the first program. For example, the electronic device binds a memory address of the binary code of the first shader to the first program. The electronic device may also unbind the second shader from the first program based on the identification of the second shader, and then the electronic device may bind the identification of the first shader to the first program. In some implementations, the electronic device can compile the source code of the first shader into binary code and store the binary code of the first shader.

In this way, the simplified first shader can be bound to the first program.

S1207, the electronic device calls the GPU to execute the shading operation through the first shading instruction. The shader bound by the first program indicated by the first shading instruction includes a first shader.

In the example of fig. 12, the matching success is described as an example. When the program is used subsequently, the simplified shader can be directly called, so that the simplified shading process is realized, and the time consumption and the power consumption cost are reduced. It should be appreciated that if a match fails, a callback of the corresponding instruction may be made according to the scheme provided in the previous embodiment in order to implement the native logic. Such as performing a corresponding shading operation according to the original shader.

The foregoing description of the solution provided in the embodiments of the present application has been mainly presented in terms of various modules. To achieve the above functions, it includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Fig. 13 is a schematic diagram of an electronic device according to an embodiment of the present application. The electronic device may be the above-described electronic device with the game application installed.

As shown in fig. 13, the electronic device 1300 may include: a processor 1301, and a memory 1302. The memory 1302 is used to store computer-executable instructions. For example, in some embodiments, the processor 1301, when executing instructions stored in the memory 1302, may cause the electronic apparatus 1300 to perform any of the image processing methods referred to in the embodiments above.

It should be noted that, all relevant contents of each step related to the above method embodiment may be cited to the functional description of the corresponding functional module, which is not described herein.

Fig. 14 shows a schematic diagram of the composition of a chip system 1400. The chip system 1400 may include: a processor 1401 and a communication interface 1402 for supporting the electronic device to implement the functions referred to in the above embodiments. In one possible design, the system on a chip also includes memory to hold the necessary program instructions and data for the terminal. The chip system can be composed of chips, and can also comprise chips and other discrete devices. It should be noted that, in some implementations of the present application, the communication interface 1402 may also be referred to as an interface circuit.

The functions or acts or operations or steps and the like in the embodiments described above may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, data centers, etc. that can be integrated with the medium. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Although the present application has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present application. It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to include such modifications and variations as well.

Claims

1. The image processing method is characterized by being applied to electronic equipment, wherein an application program is installed on the electronic equipment, and the electronic equipment executes a rendering instruction stream issued by the application program; the method comprises the following steps:

during the application loading process, the electronic device receives a second shading instruction in the rendering instruction stream, the second shading instruction including an indication to bind a second shader to the first program;

The electronic device determines an identification ID of the first program, and an ID and a code of the second shader;

the electronic equipment reads the first map file, and acquires a plurality of fourth shaders and simplified shaders corresponding to each of the fourth shaders;

wherein the first map file includes a plurality of fourth shaders, each of the plurality of fourth shaders corresponding to one of the reduced shaders; invoking at least one of a plurality of fourth shaders in the running process of the application program;

the electronic device determining a fifth shader that matches the second shader, the fifth shader included in the plurality of fourth shaders;

the second shader matching with the fifth shader includes: the second shader is identical to the code of the fifth shader; the code identity comprises the code identity length and the code identity hash value; the code includes source code and/or binary code;

the electronic device replaces the ID of the second shader bound on the first program with the ID of the first shader; the first shader is a simplified shader corresponding to the fifth shader and stored in the first map file;

In the running process of the application program, the electronic equipment calls a Graphic Processor (GPU) to execute shading operation through a first shading instruction, wherein a program indicated by the first shading instruction comprises the first program;

and the GPU calls the first shader bound on the first program to execute the shading operation according to the first shading instruction.

2. The method according to claim 1, wherein the method further comprises:

the electronic device obtains the second shader based on the rendering instruction stream;

the electronic device determines the first shader based on the second shader, the first shader corresponding to the second shader.

3. The method according to claim 1 or 2, characterized in that the method further comprises:

binding a memory address of binary code of the first shader to the first program.

4. A method according to claim 3, characterized in that the method further comprises:

binding the identification of the first shader to the first program.

5. The method of claim 4, wherein prior to binding the identification of the first shader on the first program, the method further comprises:

Unbinding the second shader from the first program based on the identification of the second shader.

6. The method of claim 1 or 2 or 4 or 5, further comprising:

compiling source code of the first shader into binary code;

storing binary code of the first shader.

7. The method of claim 6, wherein the electronic device is further comprised of, prior to binding the first shader on the first program:

the electronic device obtains an identification of the first program based on the rendering instruction stream.

8. The method of claim 2, wherein the electronic device obtains code for a plurality of third shaders, the code for the plurality of third shaders including code for a first shader, the determining the first shader based on the second shader comprising:

the code of the first shader is determined based on the code of the second shader.

9. The method of claim 8, wherein the electronic device has stored therein a plurality of codes of the third shader, the obtaining codes of the plurality of third shaders comprising:

Reading the codes of the plurality of third shaders from addresses storing the codes of the plurality of third shaders.

10. The method according to claim 2, wherein the obtaining the second shader based on the rendering instruction stream is in particular:

and acquiring an identification and a storage address of the second shader based on the rendering instruction stream, wherein the storage address stores codes of the second shader, and the codes comprise source codes or binary codes.

11. The method according to claim 8 or 9, wherein the electronic device further stores codes of a plurality of fourth shaders, the codes of the plurality of fourth shaders being in one-to-one correspondence with the plurality of third shaders, the codes of the plurality of fourth shaders including source codes or binary codes.

12. The method of claim 2 or 8, wherein the codes of the fourth plurality of shaders include codes of the fifth shader, the codes of the fifth shader matching the codes of the second shader; the determining the first shader based on the second shader includes:

acquiring code of the second shader based on a storage address of the second shader;

Matching the codes of the second shader with the codes of the fourth shaders to obtain the fifth shader;

the first shader is acquired by the fifth shader.

13. The method according to claim 12, wherein the code of the fifth shader is matched to the code of the second shader, in particular:

the code of the fifth shader is the same as the code of the second shader, or the hash value of the code of the fifth shader is the same as the hash value of the code of the second shader.

14. The method according to claim 1 or 2 or 4 or 5 or 7 to 10, wherein,

the first shader and the second shader are of the same type.

15. The method according to claim 1 or 2 or 4 or 5 or 7 to 10, wherein,

the first shader and the second shader are both vertex shaders, or,

the first shader and the second shader are both fragment shaders.

16. The method of claim 1, wherein the rendering instruction stream includes a first instruction and a second instruction, the first instruction being for indicating an identity of the first program, the second instruction being for indicating a shader bound on the first program;

The electronic device comprises an interception module and an acquisition module, and the method further comprises the steps of:

the interception module acquires the first instruction and the second instruction;

the interception module sends the first instruction and the second instruction to the acquisition module;

the acquisition module acquires the identification of the first program according to the first instruction;

and the acquisition module acquires the identification, the storage address and the code length of the second shader according to the second instruction.

17. The method of claim 16, wherein the electronic device further comprises a matching module, the method further comprising:

and the acquisition module sends the identification, the storage address and the code length of the second shader to the matching module.

18. The method of claim 17, further comprising a memory in the electronic device, wherein the memory stores a plurality of third shader codes, wherein the plurality of third shader codes includes the first shader code; the memory is also stored with codes of a plurality of fourth shaders, and the codes of the fourth shaders are in one-to-one correspondence with the codes of the third shaders; the code of the plurality of fourth shaders includes source code or binary code; the method further comprises the steps of:

The matching module reads the codes of the plurality of fourth shaders from the memory.

19. The method of claim 18, wherein the code of the plurality of fourth shaders includes code of a fifth shader, the fifth shader matching the second shader, the fifth shader corresponding to the first shader, the method further comprising:

the matching module reads codes of the second shader according to the storage address of the second shader;

the matching module matches codes of a second shader with codes of the fourth shaders to obtain the fifth shader;

wherein the matching module is according to a plurality of the fourth shaders, the length of the matching module is the same as that of the second shaders, and the code or the hash value of the code is the same; the code is a source code or a binary code;

the matching module obtains the first shader through the fifth shader.

20. The method of claim 18 or 19, wherein the electronic device further comprises a replacement module, the method comprising:

the replacement module binds a storage address of binary code of the first shader to the first program.

21. The method of claim 20, wherein the code is source code;

the replacement module unbinds the second shader from the first program,

the replacement module binds an identification of the first shader to the first program.

22. The method of claim 20, wherein the electronic device is further provided with a compiling module, the method further comprising:

the replacing module or the acquiring module sends the storage address of the first shader to the compiling module, and the compiling module compiles the codes of the first shader to acquire binary codes of the first shader;

the compilation module stores binary code of the first shader in the memory.

23. The method of claim 22, wherein the method further comprises:

the compiling module sends the storage address of the binary code of the first shader to the replacing module.

24. The method of any one of claims 16-19 or 21-23, wherein the first instruction and the second instruction are the same, the first instruction is used to instruct an identification of the first program and a shader bound on the first program, and the obtaining module obtains the identification of the first program and the identification and the storage address of the second shader according to the first instruction.

25. An image processing method is characterized in that the method is used for acquiring a simplified corresponding relation, and the simplified corresponding relation is stored in a first map file;

the simplified corresponding relation is stored in the electronic equipment, and is used for determining a corresponding first shader according to a second shader indicated by a second shading instruction issued by an application program by the electronic equipment, wherein the first shader is different from the second shader; the method comprises the following steps:

the electronic equipment acquires all rendering instruction streams in the process from the starting operation to the exiting operation of the application program;

the electronic equipment determines at least one shader to be simplified according to all rendering instruction streams, wherein the at least one shader to be simplified comprises the second shader; the at least one shader to be simplified is a shader used when the application program runs under a heavy load scene, wherein the heavy load scene is a scene with load larger than a load threshold value;

the electronic equipment determines an abstract syntax tree of the shader to be simplified according to the code of the shader to be simplified;

the electronic equipment performs simplification processing on the codes of the to-be-simplified shader based on the abstract syntax tree according to a preset simplification algorithm so as to obtain at least one shader variant; the at least one shader variant includes the first shader therein;

The electronic equipment screens the at least one shader variant according to the time gain and/or drawing image quality error of each shader variant, and determines a simplified shader corresponding to the shader to be simplified; when the to-be-simplified shader is the second shader, the simplified shader is the first shader.

26. The method of claim 25, wherein the simplified correspondence comprises:

the length of the shader is to be simplified; the code of the shader to be simplified or the hash value of the code, wherein the code comprises source code or binary code; the storage position of the simplified shader corresponding to the shader to be simplified;

when the electronic device runs the second coloring instruction according to the simplified corresponding relation,

the electronic equipment searches and obtains a fifth shader with the same length as the second shader and the same code or hash value of the code from the at least one shader to be simplified according to the second shader; the simplified shader corresponding to the fifth shader in the simplified corresponding relation is the first shader;

the electronic equipment binds the first shader to replace the second shader on a first program corresponding to the second shading instruction so as to acquire the first shading instruction;

The electronic device instructs the GPU to execute the first shading instruction.

27. The method of claim 25 or 26, wherein the shader to be simplified comprises a vertex shader and a fragment shader.

28. A chip system, wherein the chip system comprises a processor and a communication interface; the processor is configured to call up and execute a computer program stored in a storage medium from the storage medium, to perform the image processing method according to any one of claims 1 to 24, or to perform the image processing method according to any one of claims 25 to 27.