WO2020220953A1 - 用于图形渲染的方法及装置 - Google Patents

用于图形渲染的方法及装置 Download PDF

Info

Publication number
WO2020220953A1
WO2020220953A1 PCT/CN2020/083772 CN2020083772W WO2020220953A1 WO 2020220953 A1 WO2020220953 A1 WO 2020220953A1 CN 2020083772 W CN2020083772 W CN 2020083772W WO 2020220953 A1 WO2020220953 A1 WO 2020220953A1
Authority
WO
WIPO (PCT)
Prior art keywords
rendering
electronic device
information
graphics
instruction
Prior art date
Application number
PCT/CN2020/083772
Other languages
English (en)
French (fr)
Inventor
张攀
黎亮
黄巍
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP20799150.6A priority Critical patent/EP3951595A4/en
Publication of WO2020220953A1 publication Critical patent/WO2020220953A1/zh
Priority to US17/516,479 priority patent/US20220058766A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/541—Client-server
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/544—Remote
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/545—Gui
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/38—Creation or generation of source code for implementing user interfaces

Definitions

  • This application relates to the field of information technology, and more specifically, to a method and device for graphics rendering.
  • a mobile phone application runs on an electronic device, and the electronic device can only use the APP after downloading, updating, and other operations.
  • APP mobile phone application
  • the size of the APP is large, there will be problems of long downloading time and large storage space of electronic devices, and APP function redundancy/size expansion is the current trend, this problem is becoming more and more inevitable.
  • game APP it is installed locally on the electronic device, and various game plug-ins are emerging in an endless stream, which is difficult to fundamentally ban, affecting the fairness and experience of the game.
  • a method is proposed to separate the logical operation environment of the APP in the electronic device from the rendering environment, that is, the APP runs on the cloud side (for example, the server), and the rendering instructions are redirected to the end side (for example, the electronic device).
  • the cloud phone/cloud game operating mode of the cloud is called remote rendering (RR).
  • the prior art proposes a solution for separate rendering of video streams, that is, a graphics processing unit (GPU) on the cloud side performs graphics rendering to generate a video stream, and then encodes the video stream and sends it to the electronics Devices, for electronic devices, only need to have video decoding capabilities.
  • a graphics processing unit GPU
  • the image generated by the cloud side GPU for graphics rendering is compressed, resulting in lower picture quality. Poor, resulting in a poor user experience.
  • the present application provides a method, a server, and an electronic device for graphics rendering.
  • the electronic device performs graphics rendering through the acquired rendering instructions to obtain an image with the same rendering effect as an application program, thereby improving picture quality and user experience.
  • a method for graphics rendering is provided, and the method is applied to a server.
  • the method includes: starting an application program; acquiring a rendering instruction sent by the application program; sending the rendering instruction to an electronic device through a network, and the rendering instruction is used for the electronic device to perform graphics rendering, so as to facilitate the display on the electronic device
  • An image related to the application program such as a program screen or configuration interface of the application program, is displayed on the screen.
  • the electronic device executes the rendering instruction on the GPU to display a game screen and the like on the display device of the electronic device.
  • the server can obtain rendering instructions sent (or generated) by the application, and send these rendering instructions to the electronic device connected to the server, and the electronic device that receives the rendering instruction
  • the device performs graphics rendering, which avoids that the server performs graphics rendering to generate a video stream adapted to a fixed screen size in the solution of video stream separation rendering. If the screen size rendered by the electronic device and the server do not match, the image stretch or compression will occur .
  • the server can send the rendering instruction stream for graphics rendering to the electronic device. On the one hand, it can reduce the load of the server's graphics processing unit (GPU). On the other hand, it can send the rendering instructions to the electronic device.
  • the electronic device performs graphics rendering on the end side according to the rendering instruction, which can better adapt to the screen size of the electronic device, improve the picture quality, and enhance the user experience.
  • the launching of the application program may include: the server launches the application program in response to a first instruction, the first instruction may be sent by the electronic device to the server, may also come from another device, or may be generated by the server itself of.
  • the foregoing rendering instruction may include rendering information required by the rendering context, and the rendering information may include state information and data information.
  • the data information may be multiple vertex data
  • the state information may be vertex data bound in the current rendering context.
  • this application also provides a method for breakpoint transmission, which should be used in the above instruction stream separation rendering solution, and can achieve the same or better breakpoint resuming transmission effect as the video stream separation rendering solution.
  • the method includes: the server saves graphics rendering information, the graphics rendering information is the information required by the electronic device for graphics rendering, and sending the graphics rendering information to the electronic device in a timely manner, so that the electronic device The graphics rendering information performs graphics rendering.
  • the server receives a second instruction sent by the electronic device, and the second instruction is used to request to restore communication with the server
  • the server sends the graphics rendering information to the electronic device, and the graphics rendering information is used by the electronic device to perform graphics rendering.
  • the graphics rendering information may include the rendering instruction itself; in other implementation manners, the graphics rendering information may include the rendering instruction and/or a modified form of other information.
  • the storage of the graphics rendering information by the server is executed in response to the second instruction sent by the electronic device.
  • the graphics rendering information saved by the server is only the graphics rendering information during the time when the electronic device requests to restore the connection with the server (and a period of time thereafter).
  • the information is usually restored to The information of the latest state of the running of the application can be saved in a targeted manner and the storage space occupancy rate can also be reduced.
  • the server may also save graphics rendering information at any time or save graphics rendering information according to other rules.
  • the graphics rendering information is information used by the electronic device to restore the current latest rendering context of the application, and the current latest rendering context may refer to the rendering context of the application when the server receives the second instruction.
  • the application is a parkour game.
  • the parkour game runs on a server (similar to a cloud game), but the game can be played on your own electronic device.
  • the electronic device is disconnected from the server, the user can no longer enter the game interface, but the game is still running on the server.
  • the server sends the rendering information it saves to the electronic device, so that the electronic device is restored to the current latest game interface of the parkour game, and the user can start from the current latest game interface Continue to play the game, similar to the breakpoint resuming of the video streaming solution.
  • the application is corporate WeChat, which runs on the server, and the user uses an electronic device to obtain messages sent by the corporate WeChat.
  • the electronic device When the electronic device is disconnected from the server, the electronic device can no longer receive messages.
  • the method provided in this application can directly render and display the message interface containing the latest message sent by the enterprise WeChat.
  • the server can continue to send the video stream to the electronic device to achieve Reconnect after disconnection.
  • the rendering context does not have the feature of breakpoint transmission due to dependencies between the rendering contexts used for graphics rendering. Therefore, in this application, the graphics rendering information can be saved in the server.
  • the server receives the instruction sent by the electronic device requesting to restore the connection with the server, the server can send the saved graphics rendering information to the electronic device, so that the electronic device has recovery The rendering information required by the rendering context can realize the disconnection and reconnection between the server and the electronic device.
  • the graphics rendering information can be stored in the server where the application is running, or in other servers or other types of devices, or the graphics rendering information can also be stored in multiple servers in a distributed manner. On the device or using remote storage to store on other devices.
  • the graphics rendering information is the rendering instruction.
  • the server may intercept the rendering instructions sent by the application program, copy the rendering instructions, send one rendering instruction to the electronic device, and save the other rendering instruction in the server.
  • the graphics rendering information is a rendering information set
  • the rendering information set is used by the electronic device to restore the rendering context of the application program.
  • the server may process the intercepted rendering instructions to save the minimum set of information required by the electronic device to restore the application rendering context, thereby saving storage space.
  • the rendering information set may be the minimum rendering information required by the electronic device to restore the application rendering context. For example, assuming that 100 vertex data can be included in the rendering instruction, and the electronic device can perform graphic rendering to build a graph based on 80 vertex data in the 100 vertex data, the 80 vertex data required to construct the graph in the 100 vertex data can be For the minimum rendering information, if 81 vertex data is sent to the electronic device, there is redundant information; if 79 vertex data is sent to the electronic device, the electronic device cannot construct graphics.
  • the above are examples and do not limit the application in any way.
  • the method further includes: determining the rendering information set according to a pre-configured information list and the rendering instruction, and the pre-configured information list includes the information required to restore the connection between the electronic device and the server Rendering information.
  • the pre-configured information list includes rendering information whose retransmission times are less than a predetermined threshold.
  • the pre-configured information list can be the minimum set of the rendering context or the optimal set. Too much information in the pre-configured information list will cause redundancy of saved data, and too little data is not enough to restore the rendering context for graphics rendering .
  • the pre-configured information list may include rendering information that will not be retransmitted/reset in N frames (N can be set) graphics.
  • the pre-configured information list can be obtained by analyzing several engines and typical APPs that currently occupy most of the markets. By continuously capturing scene changes, touch feedback, and no input, the rendering instructions and rendering The change of context, after repeated comparison and analysis, a pre-configured information list is obtained, and the information list can include data or status that will not be retransmitted/reset in the N frame (N can be set) graphics.
  • the minimum rendering information required by the electronic device to restore the application rendering context can be determined according to the acquired rendering instructions and the pre-configured information list, that is, the rendering information set is determined, so as to save the rendering information set and re-install the rendering information on the electronic device.
  • the rendering information set is sent to the electronic device, so that the electronic device can resume the graphics rendering when the electronic device is disconnected from the server.
  • the server may send a set of rendering information to the electronic device.
  • the set of rendering information may be the minimum set of information required by the electronic device to restore the application rendering context. .
  • the method further includes: reordering the rendering information in the rendering information set according to the rendering order, where the rendering order refers to the order of rendering information when the rendering pipeline of the electronic device runs graphics rendering ;
  • Sending the graphics rendering information to the electronic device includes: sending the reordered set of rendering information to the electronic device.
  • the rendering information in the rendering information set can be sorted, that is, the rendering information set can be sorted according to the order of the rendering information when the rendering pipeline of the electronic device is running graphics rendering, and the reordered rendering can be sent to the electronic device
  • the information collection can ensure that the electronic device does not need to care about the sequence of rendering information when performing graphics rendering, thereby reducing the complexity of graphics rendering on the electronic device side.
  • the server can send all graphics rendering information to the electronic device; or, the server can send part of the graphics rendering information to the electronic device, and then determine the electronic device according to the needs of the electronic device, such as the cache list of the electronic device.
  • the device is missing the rendering information, and then continues to send the missing rendering information to the electronic device, that is, sending on-demand; or, the server can send the missing rendering information to the electronic device from the beginning according to the needs of the electronic device.
  • the method further includes: receiving a cache list sent by the electronic device, where the cache list is a list of rendering information cached by the electronic device; sending the graphics rendering information to the electronic device, The method includes: sending part or all of the graphics rendering information to the electronic device according to a cache list, and the sent graphics rendering information is graphics rendering information that does not exist in the cache list.
  • the server determines the missing rendering information of the electronic device according to one or more of the rendering instructions; searching for the missing rendering information in the cache list, and if the missing rendering information exists in the cache list Rendering information, send a third instruction to the electronic device, where the third instruction is used to instruct the electronic device to obtain the missing rendering information from a local cache; or, if the cache list does not exist
  • the missing rendering information is sent to the electronic device, and the rendering information set includes the missing rendering information.
  • the rendering instructions obtained by the server include draw call, and draw call may include the following instructions (only examples, not limited to the following instructions):
  • glDrawArrays glDrawArrays, glDrawElements, glDrawArraysInstanced, glDrawElementsInstancedg, lDrawRangeElements, glDrawArraysIndirect, and glDrawElementsIndirect.
  • the rendering instructions obtained by the server include key instructions, and the key instructions may include but are not limited to the following instructions:
  • the server can obtain the cache list of the electronic device, monitor the above draw call instruction or key instruction, determine the missing rendering information of the electronic device according to the monitored rendering instruction, and search for the missing rendering information in the cache list.
  • the missing rendering information is sent to the electronic device, which can speed up the process of the electronic device to restore the rendering context and shorten the waiting time for disconnection and reconnection between the server and the electronic device.
  • sending the graphics rendering information to the electronic device includes: determining a sequence of instructions for the graphics processor of the electronic device to identify the set of rendering information; Sending the instruction sequence and the rendering information set.
  • different instruction sequences can be constructed according to the state information in the rendering information. For example, if the state information is linked, the constructed instruction sequence may include instructions such as create, compile, and mount; if the state information is not linked, the constructed instruction sequence may include remount instructions.
  • a method for graphics rendering is provided, and the method is applied to an electronic device.
  • the method includes: receiving a rendering instruction sent by a server, where the rendering instruction is used for the electronic device to perform graphics rendering; performing graphics rendering and displaying an image related to the application program according to the rendering instruction.
  • the solution server sends rendering instructions, and then performs graphics rendering on the side of the electronic device to display application-related interfaces, etc.
  • the beneficial effects can be referred to the foregoing.
  • the method further includes: sending a first instruction to the server, where the first instruction is used to instruct the server to run the application.
  • the aforementioned rendering instructions may include a rendering context for graphics rendering.
  • the rendering context may be a resource set required for graphics rendering.
  • the resource set may include state information and data information.
  • the data information may be multiple vertex data
  • the state information may be Is the vertex data bound in the current rendering context.
  • separate rendering can be achieved through the instruction flow scheme, that is, the server can obtain the rendering instruction sent by the application, and send the rendering instruction to the electronic device connected to the server, and the electronic device that receives the rendering instruction performs graphics rendering, thereby It avoids that in the solution of separate rendering of video streams, the server performs graphics rendering to generate a video stream adapted to a fixed screen size. If the size of the screen rendered by the electronic device and the server do not match, the phenomenon of image stretching or compression will occur.
  • the server can send the rendering instruction stream for graphics rendering to the electronic device.
  • the load of the graphics processor in the server can be reduced, and on the other hand, the rendering instruction can be sent to the electronic device. Performing graphics rendering on the end side according to rendering instructions can better adapt to the screen size of the electronic device, improve the picture quality, and enhance the user experience.
  • the method further includes: sending a second instruction to the server, the second instruction being used to request to restore the connection between the electronic device and the server; receiving the graphics rendering information sent by the server, the graphics The rendering information is the information stored by the server and required for the electronic device to render graphics.
  • the server can continue to send the video stream to the electronic device to realize the disconnection reconnection. even.
  • the rendering context does not have the feature of breakpoint transmission.
  • the graphics rendering information can be saved in the server.
  • the server receives the instruction sent by the electronic device requesting to restore the connection with the server, the server can send the saved graphics rendering information to the electronic device, so that the electronic device has recovery
  • the rendering information required by the rendering context can realize the disconnection and reconnection between the server and the electronic device.
  • the graphics rendering information is the rendering instruction.
  • the server can intercept the rendering instructions sent by the application program and copy the rendering instructions.
  • One rendering instruction can be sent to the electronic device, and the other rendering instruction can be stored in the server.
  • the graphics rendering information is a rendering information set
  • the rendering information set is used by the electronic device to restore the application rendering context
  • the server can process the intercepted rendering instructions and save the minimum rendering information required by the electronic device to restore the application rendering context, that is, the server can save the rendering information set, thereby saving storage space in the server.
  • the rendering information set may be the minimum rendering information required by the electronic device to restore the application rendering context.
  • the rendering instruction can include 100 vertex data
  • the electronic device can perform graphics rendering to build a graph based on 80 of the 100 vertex data
  • the minimum rendering information of 80 vertex data required to construct the graph out of the 100 vertex data If 81 vertex data is sent to an electronic device, there is redundant information; if 79 vertex data is sent to an electronic device, the electronic device cannot construct a graph.
  • receiving the graphics rendering information sent by the server includes: receiving a set of rendering information sent by the server.
  • the server may send a rendering information set to the electronic device, and the rendering information set may be the minimum rendering required by the electronic device to restore the application rendering context information.
  • receiving the rendering information set sent by the server includes: receiving the reordered rendering information set, and the reordered rendering information set is to compare the rendering information according to the rendering order.
  • the rendering information in the set is reordered to generate a rendering information set, and the rendering order refers to the order of rendering information when the rendering pipeline of the electronic device runs graphics rendering.
  • the rendering information in the rendering information set can be sorted, that is, the rendering information set can be sorted according to the order of the rendering information when the rendering pipeline of the electronic device runs graphics rendering, and the reordered rendering information set can be sent to the electronic device Therefore, it can be ensured that the electronic device does not need to care about the sequence of rendering information when performing graphics rendering, thereby reducing the complexity of graphics rendering.
  • the method further includes: determining a sequence of instructions for a graphics processor of the electronic device to identify the set of rendering information.
  • an instruction sequence may be further determined, and the instruction sequence is used for the graphics processor of the electronic device to identify the rendering information set.
  • the rendering information set includes first identification information and first rendering information
  • the first identification information is identification information used by the server to identify the first rendering information
  • the method further include:
  • mapping relationship is determined, where the mapping relationship is a correspondence between the first identification information and the second identification information, and the second identification information is identification information used by the electronic device to identify the first rendering information.
  • the electronic device receives a set of rendering information sent by a server, the set of rendering information includes first rendering information, and the first rendering information includes an identifier set by the server, for example, the identifier is #5; the GPU of the electronic device performs processing according to the first rendering information.
  • the graphics are rendered and an identification, such as identification #4, is fed back to the electronic device, and the electronic device can determine the mapping relationship, that is, the corresponding relationship between #5 of the server and #4 of the electronic device.
  • a method for graphics rendering including: receiving a fourth instruction sent by an electronic device, where the fourth instruction is used to instruct the electronic device to disconnect from a server; and obtaining the application sent
  • the rendering instruction is used for the server to perform graphics rendering; the second instruction sent by the electronic device is received, and the second instruction is used for requesting to restore the connection between the electronic device and the server; save the graphics Rendering information, the graphics rendering information is used to restore the rendering context, the rendering context is the rendering context of the application when the second instruction is received; graphics rendering is performed according to the graphics rendering information to generate a video stream;
  • the electronic device sends the video stream.
  • the server can perform on-demand rendering according to the connection between the electronic device and the server, that is, if the electronic device and the server are in a connected state, the graphics processor of the server performs graphics rendering according to the rendering instructions to generate video Streaming; if the electronic device and the server are in an offline connection state, the graphics processor in the server does not perform graphics rendering.
  • the server when the electronic device is disconnected from the server, the server may not perform graphics rendering, but may save graphics rendering information in the server.
  • the server receives an instruction sent by the electronic device to restore the connection with the server, the server may Performing graphics rendering according to the graphics rendering information to generate a video stream, and sending the video stream to the electronic device, so that on-demand graphics rendering is implemented between the server and the electronic device, thereby reducing the load of the server.
  • the graphics rendering information is the rendering instruction.
  • the server can intercept the rendering instructions sent by the application program and copy the rendering instructions.
  • One rendering instruction can be sent to the electronic device, and the other rendering instruction can be stored in the server.
  • the graphics rendering information is a set of rendering information
  • the graphics rendering information is a set of rendering information obtained by processing the rendering instructions.
  • the server can save the rendering information set, that is, the server can save the minimum rendering information for the electronic device to restore the application rendering context, thereby saving storage space in the server.
  • the determining graphics rendering information according to the rendering instruction includes: determining the rendering information set according to a pre-configured information list and the rendering instruction, and the pre-configured information list includes that the number of retransmissions is less than Rendering information of a predetermined threshold.
  • the pre-configured information list can be the minimum set of the rendering context or the optimal set. Too much information in the pre-configured information list will cause redundancy of saved data, and too little data is not enough to restore the rendering context for graphics rendering .
  • the pre-configured information list may include rendering information that will not be retransmitted/reset in N frames (N can be set) graphics.
  • the pre-configured information list can be obtained by analyzing several engines and typical APPs that currently occupy most of the markets. By continuously capturing scene changes, touch feedback, and no input, the rendering instructions and rendering The change of context, after repeated comparison and analysis, a pre-configured information list is obtained, and the information list can include data or status that will not be retransmitted/reset in the N frame (N can be set) graphics.
  • the minimum rendering information for the electronic device to restore the application rendering context can be determined according to the acquired rendering instructions and the pre-configured information list, that is, the rendering information set is determined, so as to save the rendering information set in the server.
  • the method further includes: determining an instruction sequence for the graphics processor of the server to identify the rendering information set.
  • the instruction sequence in order to ensure that the rendering context is restored, that is, to ensure that the graphics processor of the server can recognize the rendering information set for rendering to generate a video stream, the instruction sequence can be further determined, and the instruction sequence is used for the graphics processor of the server to recognize the rendering information set.
  • a graphics rendering device which includes modules corresponding to the methods/operations/steps/actions described in the first aspect.
  • the foregoing device may be a server, or a device (for example, a chip, or a device that can be used with the server) for performing graphics rendering in the server.
  • the modules included in the above-mentioned graphics rendering apparatus may be hardware circuits, software, or hardware circuits combined with software.
  • a graphics rendering device which includes modules corresponding to the methods/operations/steps/actions described in the second aspect.
  • the foregoing device may be a server, or a device (for example, a chip, or a device that can be used with the server) for performing graphics rendering in the server.
  • the modules included in the above-mentioned graphics rendering apparatus may be hardware circuits, software, or hardware circuits combined with software.
  • a graphics rendering device which includes modules corresponding to the methods/operations/steps/actions described in the third aspect.
  • the foregoing device may be an electronic device, or a device for performing graphics rendering in an electronic device (for example, a chip, or a device that can be used with the electronic device).
  • the modules included in the above-mentioned graphics rendering apparatus may be hardware circuits, software, or hardware circuits combined with software.
  • a graphics rendering device which includes modules corresponding to the methods/operations/steps/actions described in the fourth aspect.
  • the foregoing device may be a server, or a device (for example, a chip, or a device that can be used with the server) for performing graphics rendering in the server.
  • the modules included in the above-mentioned graphics rendering apparatus may be hardware circuits, software, or hardware circuits combined with software.
  • a graphics rendering device in a ninth aspect, includes a processor, and the processor is configured to call program codes stored in a memory to perform part or all of the operations in any one of the above-mentioned first aspects.
  • the memory storing the program code may be located inside the graphics rendering device (the graphics rendering device may also include a memory in addition to the processor), or may be located outside the graphics rendering device (which may be the memory of other devices).
  • the aforementioned memory is a non-volatile memory.
  • the graphics rendering apparatus includes a processor and a memory
  • the processor and the memory may be coupled together.
  • a graphics rendering device in a tenth aspect, includes a processor, and the processor is configured to call a program code stored in a memory to perform some or all of the operations in any one of the above second aspects.
  • the memory storing the program code may be located inside the graphics rendering device (the graphics rendering device may also include a memory in addition to the processor), or may be located outside the graphics rendering device (which may be the memory of other devices).
  • the aforementioned memory is a non-volatile memory.
  • the graphics rendering apparatus includes a processor and a memory
  • the processor and the memory may be coupled together.
  • a graphics rendering device includes a processor, and the processor is configured to call a program code stored in a memory to perform some or all of the operations in any one of the above-mentioned third aspects.
  • the memory storing the program code may be located inside the graphics rendering device (the graphics rendering device may also include a memory in addition to the processor), or may be located outside the graphics rendering device (which may be the memory of other devices).
  • the aforementioned memory is a non-volatile memory.
  • the graphics rendering apparatus includes a processor and a memory
  • the processor and the memory may be coupled together.
  • a graphics rendering device in a twelfth aspect, includes a processor, and the processor is configured to call a program code stored in a memory to perform part or all of the operations in any one of the above-mentioned fourth aspects.
  • the memory storing the program code may be located inside the graphics rendering device (the graphics rendering device may also include a memory in addition to the processor), or may be located outside the graphics rendering device (which may be the memory of other devices).
  • the aforementioned memory is a non-volatile memory.
  • the graphics rendering apparatus includes a processor and a memory
  • the processor and the memory may be coupled together.
  • a computer-readable storage medium stores program code, where the program code includes instructions for performing part or all of the operations in the method described in any of the above aspects.
  • the aforementioned computer-readable storage medium is located in an electronic device, and the electronic device may be a device capable of performing graphics rendering.
  • the embodiments of the present application provide a computer program product, which when the computer program product runs on a communication device, causes the communication device to perform some or all of the operations in the method described in any of the foregoing aspects.
  • a chip in a fifteenth aspect, includes a processor, and the processor is configured to perform part or all of the operations in the method described in any of the foregoing aspects.
  • a system including the aforementioned electronic device and the aforementioned server.
  • Fig. 1 is a schematic diagram of a system architecture for graphics rendering according to an embodiment of the present application
  • Fig. 2 is a schematic diagram of a system architecture for graphics rendering according to another embodiment of the present application.
  • FIG. 3 is a schematic flowchart of the electronic device performing graphics rendering according to the acquired rendering instruction in this application;
  • FIG. 4 is a schematic flowchart of a method for graphics rendering provided according to an embodiment of the present application.
  • FIG. 5 is a schematic flowchart of a method for graphics rendering according to another embodiment of the present application.
  • Fig. 6 is a schematic flowchart of determining a snapshot of a state machine according to an embodiment of the present application
  • FIG. 7 is a schematic flowchart of an on-demand loading method according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a graphics rendering device provided according to an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of a graphics rendering apparatus according to another embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a server provided according to an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application.
  • the method for graphics rendering in the embodiment of the present application may be executed by an electronic device.
  • the electronic device may be a mobile terminal (for example, a smart phone), a computer, a personal digital assistant, a wearable device, a vehicle-mounted device, an Internet of Things device or other devices capable of performing graphics rendering processing.
  • the electronic device may be a device running Android system, IOS system, windows system and other systems.
  • the method for graphics rendering in the embodiments of the present application can implement a solution for separate rendering of instruction streams.
  • the solution for separate rendering of instruction streams means that the rendering instructions sent by the APP can be intercepted on the cloud side (for example, the server), and the rendering instructions are used for The electronic device performs graphics rendering.
  • the rendering instructions can include rendering information required by the rendering context.
  • the rendering information can include state information and data information.
  • the data information can be multiple vertex data, and the state information can be bound in the current rendering context. The vertex data.
  • the method for graphics rendering in the embodiment of the present application can be applied to the system architecture shown in FIG. 1.
  • the system architecture may include a server 110 and an electronic device 120.
  • the server 110 may include one or more application programs.
  • the server 110 may also include an interceptor, which can be used to intercept rendering instructions sent by the APP.
  • the rendering instruction is reset to the electronic device through the transmission control protocol (TCP) or proxy agent.
  • TCP transmission control protocol
  • the rendering pipeline in the electronic device can establish and maintain the rendering context and perform graphics rendering operations.
  • the interceptor may be a program running on the server to obtain APP rendering instructions.
  • the rendering instructions obtained by the interceptor may be OpenGL ES 2.0, OpenGL ES 3.0, or EGL.
  • graphics rendering can be performed on the GPU of the electronic device, that is, rendering instructions for graphics rendering can be redirected to the electronic device.
  • the GPU of the electronic device can perform graphics rendering according to the rendering instruction.
  • a graphic adapted to the screen size of the electronic device can be generated according to the rendering instruction, so that the clarity of the graphic can be improved, and the user experience can be improved.
  • the GPU of the server can obtain the rendering instruction sent by the application to perform graphics rendering to generate a video stream. Since there is no state dependency between the images in the video stream, the electronic device and the service can be disconnected and reconnected. That is, when the electronic device and the server switch from the offline state to the connected state, the electronic device can continue to obtain the video stream sent by the server to realize the disconnected connection with the server.
  • the electronic device when the electronic device is disconnected from the server, the electronic device can no longer receive the rendering instructions sent by the server, because the electronic device lacks part of the rendering information required for graphics rendering and the rendering information exists. Reliance, resulting in electronic equipment unable to continue graphics rendering.
  • the system architecture shown in FIG. 2 may be adopted.
  • the saving module and the recovery module can be set in the system architecture.
  • the saving module can be located in the server and can be used by any APP. All rendering instructions intercepted by the interceptor can be sent to the electronic device, and the rendering can be The instructions are copied and saved to the save module.
  • the saving module may also generate a snapshot, which refers to a copy of the information collection used by the electronic device to restore the rendering context.
  • the recovery module may be located in the server or in the electronic device, and the recovery module is used for the electronic device to reconstruct the rendering context.
  • the electronic device shown in FIG. 3 may be the electronic device 110 in the system architecture shown in FIG. 1, or may also be the electronic device 140 in the system architecture shown in FIG. 2.
  • the electronic device 100 may include: a central processing unit (CPU) 101, a graphics processing unit (GPU) 102, a display device 103 and a memory 104.
  • the electronic device 100 may further include at least one communication bus 110 (not shown in FIG. 3) for implementing connection and communication between various components.
  • the various components in the electronic device 100 may also be coupled through other connectors, and other connectors may include various interfaces, transmission lines, or buses.
  • the components in the electronic device 100 may also be connected in a radioactive manner with the processor 101 as the center.
  • coupling refers to being electrically connected or connected to each other, including direct connection or indirect connection through other devices.
  • the central processing unit 101 and the graphics processing unit 102 in the electronic device 100 may be located on the same chip, or may be separate chips.
  • the functions of the central processing unit 101, the graphics processing unit 102, and the display device 103 are briefly introduced below.
  • the Central processing unit 101 used to run an operating system 105 and application programs 107.
  • the application program 107 may be a graphic application program, such as a game, a video player, and so on.
  • the operating system 1005 provides a system graphics library interface through which the application program 107 can receive an instruction stream (for example, rendering instructions) for rendering graphics or image frames sent by the server.
  • the driver provided by the operating system 105 such as a graphics library user mode driver and/or a graphics library kernel mode driver, generates an instruction stream that can be recognized by the rendering pipeline in the graphics processor 102 and related rendering data required.
  • the system graphics library includes but is not limited to: embedded open graphics library (open graphics library for embedded system, OpenGL ES), the khronos platform graphics interface (the khronos platform graphics interface) or Vulkan (a cross-platform graphics application) Program interface) and other system graphics libraries.
  • the instruction stream contains a series of instructions, which are usually calls to the system graphics library interface.
  • the central processing unit 101 may include at least one of the following types of processors: an application processor, one or more microprocessors, a digital signal processor (digital signal processor, DSP), and a microcontroller (microcontroller unit, MCU) or artificial intelligence processor, etc.
  • processors an application processor, one or more microprocessors, a digital signal processor (digital signal processor, DSP), and a microcontroller (microcontroller unit, MCU) or artificial intelligence processor, etc.
  • the central processing unit 101 may further include necessary hardware accelerators, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an integrated circuit for implementing logic operations.
  • the processor 101 may be coupled to one or more data buses for transmitting data and instructions between various components of the electronic device 100.
  • the graphics processor 102 is used to receive the graphics instruction stream sent by the processor 101, generate a rendering target through a rendering pipeline, and display the rendering target on the display device 103 through the layer composition display module of the operating system.
  • the graphics processor 102 may include a general graphics processor that executes software, such as a GPU or other types of dedicated graphics processing units.
  • Display device 103 used to display various images generated by the electronic device 100, the image may be a graphical user interface (GUI) of the operating system or image data (including still images and video) processed by the graphics processor 1002 data).
  • GUI graphical user interface
  • the display device 103 may include any suitable type of display screen.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • the rendering pipeline is a series of operations performed by the graphics processor 102 sequentially during the process of rendering graphics or image frames. Typical operations include: vertex processing, primitive processing, rasterization, and fragmentation. Fragment processing and so on.
  • the server may be the server shown in FIG. 1 or FIG. 2, and the electronic device may be the electronic device shown in any one of FIG. 1 to FIG. 3.
  • the method shown in FIG. 4 includes steps 210 to 240, which are respectively described in detail below.
  • the server receives a first instruction sent by the electronic device, where the first instruction is used to instruct the server to run an application program.
  • the server may include one or more application programs, and the client of the application program may be installed on the electronic device.
  • the electronic device can send instructions to the server to start the operation of the application in the server.
  • the server can start the application program by itself, without triggering by the electronic device.
  • the server obtains the rendering instruction sent by the application program, and the rendering instruction is used for the electronic device to perform graphics rendering.
  • the interceptor in the server is used to intercept the rendering instructions sent by the application program, where the interceptor may be a program running on the server to obtain the APP rendering instructions.
  • the server sends the rendering instruction to the electronic device.
  • the electronic device performs rendering according to the rendering instruction to display the application program.
  • the server may send the rendering instruction of the application program obtained by the interceptor to the electronic device.
  • the server can send rendering instructions to the electronic device through TCP or an agent.
  • the server may save graphics rendering information, and the graphics rendering information is information required by the electronic device for graphics rendering.
  • the graphics rendering information may be rendering instructions sent by the intercepted application, or the graphics rendering information may also be a set of rendering information obtained after processing the rendering instructions sent by the intercepted application.
  • the intercepted rendering instructions of the application program can be processed in the server 130, one copy can be sent to the electronic device, so that the electronic device performs graphics rendering according to the rendering instruction, and the other copy can be saved to the electronic device in.
  • the server may process the intercepted rendering instructions and save the minimum rendering information required by the electronic device to restore the application rendering context, that is, the server may save the rendering information set.
  • the rendering information set may be the minimum rendering information required by the electronic device to restore the application rendering context. For example, assuming that 100 vertex data can be included in the rendering instruction, and the electronic device can perform graphic rendering to build a graph based on 80 vertex data in the 100 vertex data, the 80 vertex data required to construct the graph in the 100 vertex data can be For the minimum rendering information, if 81 vertex data is sent to the electronic device, there is redundant information; if 79 vertex data is sent to the electronic device, the electronic device cannot construct graphics.
  • the above are examples and do not limit the application in any way.
  • the server may receive a second instruction sent by the electronic device, and the second instruction is used to request to restore the connection with the server; the server may send the graphic to the electronic device Rendering information.
  • the rendering information set may be determined by the server.
  • the server may determine the rendering information set (for example, the key resource set shown in FIG. 5) according to the configured information list (for example, the key resource list shown in FIG. 5) and the rendering instructions, and the pre-configured information list may include restoring all information.
  • the pre-configured information list can be obtained by analyzing several engines and typical APPs that currently occupy most of the markets. By continuously capturing scene changes, touch feedback, and no input, the rendering instructions and rendering The change of context, after repeated comparison and analysis, a pre-configured information list is obtained, and the information list can include data or status that will not be retransmitted/reset in a multi-frame (definite threshold) graph.
  • the transmission process of the data information of the multi-frame graphics can be data A-data B-data C-data B-data C, and it can be seen that the data B and data C will be For repeated transmission, data A is data information that will not be retransmitted/reset in the multi-frame graphics. Therefore, in this case, the data A or the index value of the data A may be included in the information list.
  • the pre-configured information list can be the minimum set of the rendering context or the optimal set. Too much information in the pre-configured information list will cause redundancy of saved data, and too little data is not enough to restore the rendering context for graphics rendering .
  • the server may send the graphics rendering information saved by the server to the electronic device.
  • the electronic device may receive the rendering instruction saved by the server, and the rendering instruction may be the rendering instruction sent by the server intercepting the application.
  • the electronic device may receive the rendering information set sent by the server, and the rendering information set may be the minimum rendering information required by the electronic device to restore the application rendering context.
  • the rendering instructions saved by the server may be instructions sent by unprocessed applications, or the server may further save storage space and process the acquired rendering instructions to save the rendering information set.
  • the instruction sequence in order to ensure the restoration of the rendering context, that is, to ensure that the electronic device can recognize the rendering information set, the instruction sequence can be further determined, for example, to construct the instruction sequence, where the instruction sequence is used for the graphics processing of the electronic device The device recognizes the set of rendering information.
  • the instruction sequence can be determined in the server. After the server sends the instruction sequence to the electronic device, the GPU of the electronic device can identify the information used for graphics rendering in the instruction sequence.
  • the instruction sequence can be determined in the electronic device.
  • the server may send the set of rendering information to the electronic device.
  • the CPU of the electronic device determines the instruction sequence and sends the instruction sequence from the CPU of the electronic device to the GPU of the electronic device, so that the electronic device
  • the GPU performs graphics rendering based on the set of rendering information.
  • the set of rendering information may include first identification information and first rendering information.
  • the first identification information is identification information that may be used by the server to identify the first rendering information.
  • the electronic device may determine the mapping relationship. The relationship is a correspondence between the first identification information and the second identification information, and the second identification information is identification information used by the electronic device to identify the first rendering information.
  • the electronic device receives a set of rendering information sent by a server, the set of rendering information includes first rendering information, and the first rendering information includes an identifier set by the server, for example, the identifier is #5; the GPU of the electronic device performs processing according to the first rendering information.
  • the graphics are rendered and an identification, such as identification #4, is fed back to the electronic device, and the electronic device can determine the mapping relationship, that is, the corresponding relationship between #5 of the server and #4 of the electronic device.
  • the server can reorder the rendering information in the rendering information set according to the rendering order.
  • the rendering order refers to the rendering when the rendering pipeline of the electronic device runs graphics rendering.
  • the order of the information; the server may send the reordered set of rendering information to the electronic device.
  • the electronic device may send a cache list to the server, and the cache list includes rendering information cached locally by the electronic device; further, when the server receives a request for restoration and After the server is connected, the server may first monitor the rendering instructions to determine the rendering information that may be missing from the electronic device. Search for missing rendering information in the cache list of the electronic device, and if there is missing rendering information in the cache list, send a third instruction to the electronic device, the third instruction is used to instruct the electronic device to obtain the missing rendering information from the local cache; Alternatively, the missing rendering information is searched for in the cache list of the electronic device, and if the missing rendering information does not exist in the cache list, the missing rendering information is sent to the electronic device.
  • the server may first monitor the draw call or key instruction sent by the application to determine the resource required by the electronic device to restore the rendering context, and the server will further determine whether the resource has been restored; If the server determines that the resource has not been restored, the server can search in the cache list of the electronic device to determine whether the resource exists in the cache list of the electronic device. When the resource exists in the cache list, the server can instruct the electronic The device obtains the resource from its local cache; when the resource does not exist in the cache list, the server can send the resource to the electronic device. If the server determines that the resource has been restored, the server does not perform any operation.
  • the resource required to restore the rendering context is the rendering information required by the electronic device to restore the rendering context, that is, the rendering information missing from the electronic device.
  • the rendering information cached locally by the electronic device may include data information but not state information, that is, the rendering information cached locally by the electronic device may be data information for graphics rendering.
  • draw call can include but is not limited to the following commands:
  • glDrawArrays glDrawElements, glDrawArraysInstanced, glDrawElementsInstanced, glDrawRangeElements, glDrawArraysIndirect, glDrawElementsIndirect.
  • key commands can include but are not limited to the following commands:
  • glBindBuffer glBufferData, glBufferSubData, glBindTexture, glTexImage2D, glTexSubImage2D, glTexImage3D, glTexSubImage3D, glCompressedTexImage2D, glCompressedTexSubImage2D, glCompressedTexImage3D, glCompressedTexSubImage3D.
  • separate rendering can be achieved through the instruction flow scheme, that is, the server can obtain the rendering instruction sent by the application, and send the rendering instruction to the electronic device connected to the server, and the electronic device that receives the rendering instruction performs graphics rendering, thereby It avoids that in the solution of separate rendering of video streams, the server performs graphics rendering to generate a video stream adapted to a fixed screen size. If the size of the screen rendered by the electronic device and the server do not match, the phenomenon of image stretching or compression will occur.
  • the server can send the rendering instruction stream for graphics rendering to the electronic device.
  • the load of the graphics processor in the server can be reduced, and on the other hand, the rendering instruction can be sent to the electronic device. Performing graphics rendering on the end side according to rendering instructions can better adapt to the screen size of the electronic device, improve the picture quality, and enhance the user experience.
  • This application also provides a method for graphics rendering, which can enable the solution of separate rendering of video streams to achieve on-demand rendering, that is, the server can perform on-demand rendering according to the connection between the electronic device and the server, that is, if the electronic device and the When the server is in the connected state, the graphics processor of the server can perform graphics rendering to generate a video stream according to the rendering instructions; if the electronic device and the server are offline, the graphics processor in the server does not perform graphics rendering.
  • this application proposes a method for graphics rendering, including: a server receives a fourth instruction sent by an electronic device, where the fourth instruction is used to instruct the electronic device to disconnect from the server; the server obtains The rendering instruction sent by the application program is used for the graphics processor of the server to perform graphics rendering to generate a video stream; the server determines graphics rendering information according to the rendering instruction, and the graphics rendering information is generated by the server Rendering information required by the video stream; the server saves the graphics rendering information.
  • the above-mentioned embodiments of the present application are also applicable to the video stream separated rendering solution.
  • the difference between the video stream separated rendering solution and the instruction stream separated rendering solution is that the instruction stream separated rendering solution saves graphics
  • the rendering information is sent to the electronic device, and the electronic device performs the graphics rendering according to the graphics rendering information.
  • the saved graphics rendering information can be sent to the graphics processor of the server, and the graphics processor of the server can perform graphics rendering according to the graphics.
  • the rendering instruction performs graphics rendering to generate a video stream, and sends the video stream to the electronic device.
  • the process of saving the graphics rendering information and restoring the rendering context is also applicable to the solution of separate rendering of the video stream, and will not be repeated here.
  • the server when the server receives the second instruction sent by the electronic device, the second instruction is used to request to restore the connection between the electronic device and the server; the server renders according to the graphics Information is rendered graphically to generate a video stream; the server sends the video stream to the electronic device.
  • the server when the electronic device is disconnected from the server, the server may not perform graphics rendering, but may save graphics rendering information in the server.
  • the graphics rendering information may be information required by the server for graphics rendering.
  • the server receives After the request sent by the electronic device resumes the connection with the server, the server can perform graphics rendering according to the graphics rendering information, generate a video stream, and send the video stream to the electronic device, so that on-demand rendering between the server and the electronic device can be achieved, which can further reduce the server GPU Therefore, the instance density of server operation in the solution of video stream separation rendering is significantly increased.
  • the graphics rendering information may be a rendering instruction, that is, the rendering information saved in the server may be a rendering instruction sent by an acquired application.
  • the graphics rendering information may be a set of rendering information, which is used by the electronic device to restore the application rendering context.
  • the determining the graphics rendering information according to the rendering instruction includes: determining the rendering information set according to a pre-configured information list and the rendering instruction, and the pre-configured information The list includes rendering information required to restore the connection between the electronic device and the application program.
  • the server may also determine a sequence of instructions for the graphics processor of the server to identify the set of rendering information.
  • the central processing unit of the server determines the instruction sequence, and the instruction sequence is used for the graphics processor of the server to recognize the rendering information set, so that the graphics processor of the server performs graphics rendering to generate a video stream according to the rendering information set.
  • FIG. 5 is a schematic flowchart of a method for graphics rendering provided by an embodiment of the present application.
  • the method shown in FIG. 5 may include steps 301 to 310, and steps 301 to 310 will be described in detail below.
  • the end side/rendering end can be any electronic device shown in Figures 1 to 4, the APP, the state machine saving module, and the state machine can be located in the server, and the state machine recovery module can be located in the electronic device or the server. It should be understood that the state machine in this application may be a rendering context.
  • Step 301 the terminal side/rendering terminal is connected for the first time. That is, it can be the first connection between the end/rendering end and the APP in the server.
  • Step 302 The end side/rendering end may send a start instruction to the APP.
  • it may be the first instruction in FIG. 4.
  • Step 303 The cloud side updates and maintains the state machine.
  • the cloud-side update and maintenance state machine can be that the server uses the corresponding data structure to save and update the data and state necessary for graphics rendering in real time according to the rendering instructions sent by the APP received.
  • Step 304 Perform remote rendering, that is, send the rendering instruction sent by the APP to the electronic device, and the electronic device renders graphics according to the acquired rendering instruction.
  • Step 305 The state machine saving module receives an instruction to request state machine restoration/reconstruction sent by the end side/rendering end.
  • the instruction for requesting the restoration/reconstruction of the state machine may be the second instruction in the foregoing embodiment, and the second instruction is used for requesting to restore the connection between the electronic device and the server.
  • the cloud side After the cloud side receives the request state machine recovery/rebuild trigger command or other trigger command sent by the end side, it extracts the key resources in the state machine according to the preset key resource list; further, the extracted key resources
  • the key resources are rearranged according to the resource dependency relationship (which may be the rendering order in the foregoing embodiment), and a snapshot is generated.
  • a snapshot can be a copy of a key resource collection at a certain moment.
  • Fig. 6 is a schematic diagram of a server generating a snapshot provided by an embodiment of the present application.
  • Fig. 6 includes steps 410 to 440, and steps 410 to 440 are respectively described in detail below.
  • Step 410 Obtain a key resource list and state machine.
  • the resource required by the rendering context is the rendering information.
  • the resource is composed of data information and status information.
  • resources can be divided into EGL layer and OpenGL ES layer.
  • OpenGL ES resources are composed of the following parts:
  • OpenGL ES status information can include but is not limited to the following:
  • Binding of various global states such as the binding of buffer and target, the current settings of various parameters, etc.
  • binding relationships may include but not limited to the following:
  • the key resource list can be a pre-configured information list, and the key resource list can be abstracted by analyzing the rendering instruction sequence of each game engine in different situations.
  • the key resource list can be the smallest set of rendering contexts. It is also an optimal set, where too much information in the key resource list will swell the snapshot, and too little in the key resource list will not be enough to restore the rendering context, so its abstraction and definition are very important.
  • the abstraction of the key resource list can be a static process, that is, the acquisition of the key resource list may not be obtained from the CPU or GPU of the current server through a specific API or method.
  • the changes in rendering instructions and rendering context resources can be continuously captured during scene changes, touch feedback, and no input.
  • the resulting key resource list includes data or states that will not be retransmitted/reset in multiple frames (preset threshold).
  • determining the list of key resources can include but not limited to the following steps:
  • Step 1 Classify the engines and determine the rendering instructions for capturing different scenes
  • Step 2 Determine a scene and set the preset number of frames, such as 100 frames;
  • Step 3 Determine the rendering information with a smaller number of retransmissions in the image with the preset number of frames
  • Step 4 In different scenes, repeat the above steps 1 to 3, and finally the union of the rendering information with less transmission times obtained in different scenes is determined as the key resource list.
  • the key resource list can also be obtained in combination with touch feedback, no input and other conditions of rendering information transmission.
  • the key resource list may also be obtained based on artificial intelligence (AI) technology, where AI may be a digital computer or a machine controlled by a digital computer to simulate, extend, and extend humans.
  • AI artificial intelligence
  • the key resource list can be obtained based on a neural network algorithm, or a computer algorithm model. The above is an example and does not limit the source of the key resource list in any way.
  • the key resource list is usually obtained offline, but if there is an efficient way to obtain it, the key resource list can also be obtained online in real time through this method, which is not limited in this application.
  • the key resources may include but are not limited to the following resources:
  • attachment point status (binding) of the frame buffer/render buffer object and its corresponding data does not include the parameters set by glreadbuffer/gldrawbuffers;
  • the state machine can implement the complete EGL/OpenGL ES standard, and will intercept all the rendering commands sent by the APP to call.
  • the resources contained in the rendering commands may belong to different contexts. After determining the current rendering command belongs to the OpenGL ES version, distribute it to The corresponding Context is then finally distributed to different Object Resources according to the resource type contained in the rendering instruction.
  • Each Object Resource can be composed of many Object Resource instances.
  • the server can intercept the rendering instructions sent by the APP, one copy is normally sent to the electronic device, and the other is redirected to the server's state machine to modify the rendering context resources and states, such as executing resource creation, deletion, and binding relationships Operations such as maintenance will be synchronized to each ObjectResource of the Context to ensure that the state machine in the server is complete and the state transition can continue.
  • a state machine can be maintained in the server. Its purpose is to continuously receive the rendering instructions intercepted from the APP and update it to the corresponding data structure. When the electronic device requests the state machine to recover, pass The key resource list is a collection of key resources extracted from the state machine.
  • Step 420 The key resource set, that is, the server determines the key resource set according to the key resource list and the state machine in step 410.
  • the key resource list may be a pre-configured information list.
  • the server can extract a set of key resources from the state machine through the key resource list.
  • Step 430 Resource rearrangement.
  • the dependency relationship may be a rendering order, and the rendering order refers to the order of rendering information when the rendering pipeline of the electronic device runs graphics rendering.
  • the storage of global resources must be performed after various Object Data; the storage of textures/Images must be performed before FrameBuffer, etc.
  • the order of saving resources can be in the following order: EGL layer resources-ObjectResource ⁇ Buffer>-ObjectResource ⁇ VertexArray>-ObjectResource ⁇ Texture>-ObjectResource ⁇ Sampler>-ObjectResource ⁇ Program/Shader>-ObjectResource ⁇ RenderBuffer>-ObjectResource ⁇ FrameBuffer>-Global status.
  • Step 440 Determine the state machine snapshot.
  • key resources can be determined according to resource rearrangement, and a snapshot of the state machine can be generated, which can be persisted or choose to reside in cloud-side memory.
  • step 306 in the schematic flowchart of the method for graphics rendering shown in FIG. 5 may be executed to send the state machine snapshot.
  • the state machine snapshot may be the reordered rendering information set in the embodiment shown in FIG. 4.
  • Step 307 Trigger a rendering context reconstruction operation.
  • reconstruction operation of the rendering context may be performed in the electronic device, or may be performed in the server.
  • the following takes the reconstruction operation of the rendering context performed in the electronic device for illustration.
  • the server can rearrange the key resource collections according to the rendering order, so when the rendering context is rebuilt, that is, the rendering context is restored, there is no need to pay attention to the resource order problem and cause the resource restoration failure problem.
  • Different instruction sequences need to be generated according to the current state of each type of resource, that is, an instruction sequence that can be recognized by the GPU of the electronic device needs to be constructed according to the current state of each type of resource.
  • Each Object Resource is composed of many Object Resource instances, and each Object Resource maintains a map, such as the map ⁇ serverHandle,program> of Program/Shader ObjectResource.
  • map ⁇ serverHandle,program> of Program/Shader ObjectResource To save the mapping between the instance identifier of this type of Object Resource and the instance itself, each instance will be in a certain state when the state machine save action is triggered.
  • ObjectResource For Program/Shader ObjectResource, traverse ObjectResource ⁇ Program/Shader>. Regardless of the state of the current instance, the identifier must be used as a parameter, and the construction instruction will be sent to the terminal side, which will trigger the terminal side to establish a terminal-side resource identifier and cloud
  • the mapping map ⁇ serverHandle, clientHandle> of the side resource identification they identify the same data, after the state machine is restored, it is used to operate the side resource.
  • the reconstruction process of the rendering context can refer to the process of disassembling the data and state of the rendering context saved on the cloud side, according to the current state of each resource, in accordance with the GPU API standard and certain rules and order, Construct an instruction sequence that conforms to the GPU pipeline architecture and send it to the rendering thread (RenderThread) on the end side to perform graphics rendering. This process reloads the resources contained in the instruction sequence to the end GPU. Accordingly, the end The GPU on the side will also regenerate an identifier for operating the resource. In the process of rendering context reconstruction, all the resources of the GPU that are reloaded into the end side will be re-identified. If you want to perform graphics rendering in the reconstructed rendering context, you need to "identify" these "new" resources, so you need a
  • the resource mapping mechanism is the mapping relationship between the rendering context resource identifier on the cloud side and the rendering context resource identifier.
  • the electronic device can determine the resource mapping, that is, the rendering instruction constructed on the cloud side contains the resource and the identification number of the resource.
  • the end-side executes the instruction when it is restored, it will also generate an identification of the resource on the end-side. It establishes a mapping relationship map ⁇ serverHandle, clientHandle> with the resource identifier sent from the cloud side.
  • the cloud side uses its identification of the resource to operate the resource, and the end-side can use the resource mapping back-end identification instead of the cloud-side identification to perform graphics rendering operations, that is, the end-side Any data or status on the cloud side can be accessed.
  • the state of the instance in the snapshot resource can be read. Because the state machine needs to save the rendering context at the moment before the operation is triggered, different instruction sequences can be determined according to the state . If the current program instance has been linked (linked), create, compile, and mount the shaders instance on the mount point, and then set a few vertex attributes that are not frequently updated, and then use the program instance (through the example Mark to use), set the uniform variables in the key resources; and if the current program instance is not linked, you only need to remount the shaders on the mount point of the program instance.
  • the above are examples and do not limit the application in any way.
  • ObjectResource ⁇ Program/Shader> is used as an example above.
  • ObjectResource ⁇ Program/Shader> it can be customized similar to ObjectResource ⁇ Program/Shader> according to the state of each instance saved before the state machine save operation is triggered. Structured instruction sequence.
  • FIG. 7 is a schematic flowchart of a method for accelerating restoration of a rendering context provided by an embodiment of the present application.
  • Figure 7 includes steps 501 to 511, and steps 501 to 511 will be described in detail below.
  • the end side may be any electronic device shown in FIGS. 1 to 4, and the cloud side may be any server shown in FIGS. 1, 2 and 3.
  • Step 501 Resource on-demand loading mode is triggered.
  • the on-demand loading mode can take effect at the beginning of the construction of the instruction sequence, that is, the state machine recovery module.
  • the cloud side can monitor the draw call or key instructions on the end side, and only load/restore the strong dependent resources of the current frame or key instructions Load other resources later.
  • the rendering can be performed according to the on-demand recovery/resource cache acceleration algorithm while continuing.
  • the cloud side can continuously monitor the draw call or key instructions, and continue to be the subsequent current frame or key instruction The strong dependence on resource loading services.
  • Step 502 Obtain a list of end-side cache resources.
  • the rendering context When the rendering context is restored, it can be directly from the end side cache Reading from the database and loading it into the GPU on the end-side can significantly reduce the amount of network data transmission during the rendering context restoration process, thereby reducing the waiting time required to restore the rendering context.
  • Step 503 Construct an instruction sequence, that is, the determination instruction sequence in the foregoing embodiment.
  • the reconstruction process of the rendering context can refer to the process of disassembling the data and state in the rendering context saved on the cloud side, and constructing a conformity according to the current state of each resource, in accordance with the GPU API standard and certain rules and order.
  • Step 504 The main process of rendering context restoration is completed.
  • the completion of the main process of rendering context restoration may correspond to step 305 to step 309 shown in FIG. 5.
  • Step 505 Load on demand has been running, and continuously monitor the draw call/key instructions of the electronic device.
  • draw call can include but is not limited to the following commands:
  • glDrawArrays glDrawElements, glDrawArraysInstanced, glDrawElementsInstanced, glDrawRangeElements, glDrawArraysIndirect, glDrawElementsIndirect.
  • key commands can include but are not limited to the following commands:
  • glBindBuffer glBufferData, glBufferSubData, glBindTexture, glTexImage2D, glTexSubImage2D, glTexImage3D, glTexSubImage3D, glCompressedTexImage2D, glCompressedTexSubImage2D, glCompressedTexImage3D, glCompressedTexSubImage3D.
  • Step 506 Determine whether the acquired rendering instruction includes the monitored draw call/key instruction.
  • the texture resource or vertex resource required by the end-side to restore the rendering context can be determined according to the monitored draw call/key instruction, and in step 502, the end-side may cache part of the texture resources and vertex resources on the end-side. Therefore, if the draw call/key instruction is included in the rendering instruction on the end side, step 507 is executed; if the draw call/key instruction is not included in the rendering instruction on the end side, step 508 is executed, and the game runs in normal rendering mode.
  • the normal rendering mode can be command interception, one is sent to the state machine module, and the other is sent to the end side, the same as before the state machine recovery process is not triggered.
  • the cloud side can run in the on-demand loading mode, that is, continuous monitoring of drawcall and key instructions. If the rendering instructions intercepted by the cloud side include drawcall and key instructions, It will trigger a cache resource lookup, that is, it is monitored that some or all of the resources indicated by the draw call/key instruction may exist in the local cache of the electronic device, so the search can be performed in the cache list of the electronic device. Otherwise, it will only enter the normal rendering mode of the game.
  • Step 507 The cloud side obtains all resource identifiers indicated by the draw call/key instruction.
  • Step 508 Look up the resource identifier in the cache resource list sent by the end side.
  • the cache resource lookup can be from the list of all the resources in its cache (the mapping between the sha256sum value of the resource and the resource) obtained from the end side when the on-demand loading mode is triggered, to find the resource indicated by the draw call and the key instruction, and calculate the The sha256sum value of the resource is used as the key to search in the resource list. If the search is successful in the resource list cached on the end side, it means that the end side has cached the resource, and the cloud side directly sends the sha256sum value of the resource to the end side and informs it The end side can load the resource from the cache database to the GPU. Otherwise, the cloud side can send the resource to the end side via the network.
  • Step 509 Trigger the end side to load rendering information from its local cache.
  • Step 510 Obtain the resource corresponding to the identifier from the acquired state machine snapshot according to the resource identifier.
  • Step 511 Send the resource to the end side through the network.
  • the recovery speed of the rendering context can be accelerated according to the above-mentioned on-demand loading method, that is, the rendering information cached in the electronic device and the rendering information related to the current frame graphics can be restored, thereby reducing the active rendering context.
  • the waiting time required.
  • FIG. 8 is a schematic block diagram of a graphics rendering apparatus 600 according to an embodiment of the present application. It should be understood that the apparatus 600 can execute each step in the methods in FIGS. 4 to 7, and in order to avoid repetition, details are not described herein again.
  • the graphics rendering apparatus 600 may be a server, or the graphics rendering apparatus 600 may be a chip configured in the server.
  • the graphics rendering apparatus 600 includes: a processing unit 610 and a transceiving unit 620.
  • the processing unit 610 is used to start an application program; to obtain a rendering instruction sent by the application program; the transceiver unit 620 is used to send the rendering instruction to the electronic device, and the rendering instruction is used for the electronic device to perform graphics rendering To display images related to the application.
  • the graphics rendering apparatus 600 shown in FIG. 8 may be the server shown in FIG. 1 or FIG. 2, wherein the transceiver unit 620 may implement the corresponding functions of the server communication interface shown in FIG. 1 or FIG. 2.
  • the processing unit may be corresponding to implement the corresponding functions in the interceptor shown in FIG. 1, or the processing unit 610 may also be corresponding to implement the functions corresponding to the interceptor, the save module, and the recovery module shown in FIG. 2; the transceiver unit 620 may Corresponds to Figure 1 or Figure 2 to achieve the TCP/Proxy function.
  • the processing unit 610 is further configured to: save graphics rendering information, where the graphics rendering information is information required by the electronic device for graphics rendering.
  • the transceiver unit 620 is further configured to receive a second instruction sent by the electronic device, where the second instruction is used to request to restore the connection with the server; The second instruction is to send the graphics rendering information to the electronic device, the graphics rendering information is used by the electronic device to restore a rendering context, and the rendering context is the status of the application when the second instruction is received Rendering context.
  • the graphics rendering information is the rendering instruction.
  • the graphics rendering information is a rendering information set
  • the graphics rendering information is a rendering information set obtained by processing the rendering instructions.
  • processing unit 610 is further configured to:
  • the rendering information set is determined according to a pre-configured information list and the rendering instruction, and the pre-configured information list includes rendering information whose retransmission times are less than a predetermined threshold.
  • processing unit 610 is further configured to:
  • the transceiver unit 620 is specifically configured to:
  • the transceiver unit 620 is further configured to:
  • the processing unit 610 is also used for:
  • processing unit 610 is further configured to:
  • the transceiver unit 620 is specifically configured to:
  • the graphics rendering apparatus 600 includes: a processing unit 610 and a transceiving unit 620.
  • the transceiving unit 620 is configured to receive a fourth instruction sent by the electronic device, and the fourth instruction is used to instruct the electronic device to disconnect from the server;
  • the processing unit 610 is configured to receive a second instruction sent by the electronic device. Instruction, the second instruction is used to request to restore the connection between the electronic device and the server; save graphics rendering information, the graphics rendering information is used to restore the rendering context, the rendering context is when the second instruction is received
  • the rendering context of the application program performing graphics rendering according to the graphics rendering information to generate a video stream; the transceiver unit 620 is further configured to send the video stream to the electronic device.
  • the graphics rendering apparatus 600 shown in FIG. 8 may be the server shown in FIG. 1 or FIG. 2, where the transceiver unit 620 may implement the corresponding function of the server communication interface.
  • the processing unit 610 may be corresponding to implementing the corresponding functions in the interceptor shown in FIG. 1, or the processing unit 610 may also be corresponding to implementing the functions corresponding to the interceptor, the saving module, and the recovery module shown in FIG. 2; the transceiver unit 620 It can correspond to the function of TCP/Proxy in Figure 1 or Figure 2.
  • the graphics rendering information is the rendering instruction.
  • the graphics rendering information is a set of rendering information obtained by processing the rendering instructions.
  • the processing unit 610 is further configured to: determine the rendering information set according to a pre-configured information list and the rendering instruction, and the pre-configured information list includes retransmission Rendering information whose times are less than a predetermined threshold.
  • the processing unit 610 is further configured to: determine an instruction sequence, where the instruction sequence is used by the graphics processor of the server to identify the rendering information set.
  • the graphics rendering apparatus 600 here is embodied in the form of a functional unit.
  • the term "unit” herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” may be a software program, a hardware circuit, or a combination of the two that realize the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 9 is a schematic block diagram of a graphics rendering apparatus 700 according to an embodiment of the present application. It should be understood that the apparatus 700 can execute each step in the method of FIG. 4 to FIG. 7, and in order to avoid repetition, it will not be described in detail here.
  • the graphics rendering apparatus 700 may be an electronic device, or the graphics rendering apparatus 700 may be a chip configured in the electronic device.
  • the graphics rendering apparatus 700 includes: a transceiver unit 710 and a processing unit 720.
  • the transceiver unit 710 is configured to receive a rendering instruction sent by the server, where the rendering instruction is used for the electronic device to perform graphics rendering; the processing unit 7720 is configured to perform graphics rendering according to the rendering instruction.
  • the transceiver unit 710 is further configured to send a second instruction to the server, where the second instruction is used to request to restore the connection between the electronic device and the server;
  • the processing unit 720 is further configured to restore a rendering context according to the graphics rendering information, where the rendering context is the rendering context of the application when the second instruction is sent.
  • the graphics rendering information is the rendering instruction.
  • the graphics rendering information and the graphics rendering information are a set of rendering information obtained by processing the rendering instructions.
  • the processing unit 720 is further configured to determine an instruction sequence, where the instruction sequence is used by a graphics processor of the electronic device to identify the rendering information set.
  • the rendering information set includes first identification information and first rendering information
  • the first identification information is used by the server to identify the first rendering information
  • the processing unit 720 is further configured to determine a mapping relationship, the mapping relationship is the corresponding relationship between the first identification information and the second identification information, and the second identification information is used by the electronic device To identify the identification information of the first rendering information.
  • the graphics rendering apparatus 700 here is embodied in the form of a functional unit.
  • the term "unit” herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” may be a software program, a hardware circuit, or a combination of the two that realize the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 10 shows a schematic block diagram of a server 800 provided by an embodiment of the present application.
  • the server 800 includes a processor 820, a memory 860, a communication interface 840, and a bus 850.
  • the processor 820, the memory 860, and the communication interface 840 communicate through the bus 850, and may also communicate through other means such as wireless transmission.
  • the memory 860 is used to store instructions, and the processor 820 is used to execute instructions stored in the memory 860.
  • the memory 860 stores program codes 811, and the processor 820 can call the program codes 811 stored in the memory 860 to execute the methods for graphics rendering shown in FIGS. 4 to 7.
  • the processor 820 may be configured to execute the rendering instructions sent by the application program obtained in 220 in FIG. 4; or, may execute the process of determining the state machine snapshot shown in FIG.
  • the memory 860 may include a read-only memory and a random access memory, and provides instructions and data to the processor 820.
  • the memory 860 may also include a non-volatile random access memory.
  • the memory 860 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electronic Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be random access memory (RAM), which is used as an external cache.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • Double data rate synchronous dynamic random access memory double data date SDRAM, DDR SDRAM
  • enhanced SDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • direct rambus RAM direct rambus RAM
  • bus 850 may also include a power bus, a control bus, and a status signal bus.
  • bus 850 may also include a power bus, a control bus, and a status signal bus.
  • various buses are marked as bus 850 in FIG. 8.
  • the processor 820 can execute the steps/functions corresponding to the processing unit 610 shown in FIG. 8, and the communication interface 840 can execute the steps/functions corresponding to the transceiver unit 620 shown in FIG.
  • server 800 shown in FIG. 10 can implement various processes performed by the device in the method embodiments shown in FIGS. 4 to 7.
  • the operations and/or functions of the various modules in the server 800 are respectively intended to implement the corresponding processes of the devices in the foregoing method embodiments.
  • FIG. 11 is a schematic structural diagram of an electronic device 900 according to an embodiment of the present application.
  • the electronic device 900 can be, but is not limited to, a laptop computer, a desktop computer, a mobile phone, a smart phone, a tablet computer, a multimedia player, an e-reader, a smart car device, a smart home appliance, an artificial intelligence device, a wearable device, a thing Networked equipment, or virtual reality/augmented reality/mixed reality equipment, etc.
  • the electronic device 900 in FIG. 11 includes a communication module 910, a sensor 920, a user input module 930, an output module 940, a processor 950, a memory 970, and a power supply 980.
  • the processor 950 may include one or more CPUs.
  • the electronic device shown in FIG. 11 can execute each step of the graphics rendering method of the embodiment of the present application.
  • one or more CPUs in the processor 950 can execute each of the graphics rendering methods of the embodiment of the present application. step.
  • the various modules of the electronic device 900 in FIG. 11 will be described in detail below. It should be understood that the modules shown in FIG. 11 are only examples, and the electronic device 900 may include more or fewer modules in a specific implementation.
  • the communication module 910 may include at least one module that enables communication between the electronic device and other electronic devices.
  • the communication module 910 may include one or more of a wired network interface, a broadcast receiving module, a mobile communication module, a wireless Internet module, a local area communication module, and a location (or positioning) information module.
  • This embodiment does not limit the communication generation. For example, it may be 2G, 3G, 4G, 5G, or other communication generations that appear with technological development.
  • the communication module 910 can obtain the rendering instructions required for rendering the game screen from the game server in real time.
  • the sensor 920 can sense some operations of the user, and the sensor 920 can include a distance sensor, a touch sensor, and so on.
  • the sensor 920 can sense operations such as the user touching the screen or approaching the screen.
  • the senor 920 can sense some operations of the user on the game interface.
  • the user input module 930 is used to receive inputted digital information, character information, or contact touch operations/non-contact gestures, and receive signal input related to user settings and function control of the system.
  • the user input module 930 includes a touch panel and/or other input devices. For example, the user can control the game through the user input module 3030.
  • the output module 940 includes a display panel for displaying information input by the user, information provided to the user, various menu interfaces of the system, and the like.
  • the display panel may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).
  • the touch panel can cover the display panel to form a touch display screen.
  • the output module 940 may also include a video output module, an alarm, and a haptic module.
  • the video output module can display the game screen after graphics rendering.
  • the power supply 980 may receive external power and internal power under the control of the processor 950, and provide power required by the various modules of the entire electronic device during operation.
  • the processor 950 may include one or more CPUs, and the processor 950 may also include one or more GPUs.
  • the processor 950 includes multiple CPUs
  • the multiple CPUs may be integrated on the same chip or separately integrated on different chips.
  • the processor 950 includes multiple GPUs
  • the multiple GPUs may be integrated on the same chip or separately integrated on different chips.
  • the processor 950 includes both a CPU and a GPU
  • the CPU and the GPU may be integrated on the same chip.
  • the processor of the smart phone is generally related to image processing by a CPU and a GPU. Both the CPU and GPU here can contain multiple cores.
  • the memory 970 may store a computer program, and the computer program includes an operating system program 972, an application program 971, and the like.
  • typical operating systems such as Microsoft’s Windows, Apple’s MacOS, etc. are used in desktop or notebook systems, as well as systems developed by Google based on Android System and other systems used in mobile terminals.
  • the memory 970 may be one or more of the following types: flash memory, hard disk type memory, micro multimedia card type memory, card type memory (such as SD or XD memory), random access memory (random access memory) , RAM), static random access memory (static RAM, SRAM), read-only memory (read only memory, ROM), electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), programmable Read-only memory (programmable ROM, PROM), magnetic memory, magnetic disk or optical disk.
  • the memory 970 may also be a network storage device on the Internet, and the system may perform operations such as updating or reading the memory 970 on the Internet.
  • the foregoing memory 970 may store a computer program (the computer program is a program corresponding to the graphics rendering method in an embodiment of the present application).
  • the processor 950 executes the computer program, the processor 950 can execute the embodiment of the present application.
  • the memory 970 also stores other data 973 in addition to the computer program.
  • the memory 970 may store data during the processing of the graphics rendering method of the present application.
  • connection relationship of each module in FIG. 11 is only an example.
  • the electronic device provided by any embodiment of the present application can also be applied to electronic devices of other connection modes, for example, all modules are connected through a bus.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).
  • the term "and/or” is merely an association relationship describing an associated object, indicating that there may be three relationships.
  • a and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone.
  • the character "/" in this text generally indicates that the associated objects before and after are in an "or" relationship.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本申请提供了一种用于图形渲染的方法和装置等，其中，服务启动应用程序并获取所述应用程序发送的渲染指令；向所述电子设备发送所述渲染指令，所述电子设备根据所述渲染指令执行图形渲染以显示与所述应用程序相关的图像。本申请提供的技术方案能够使电子设备根据渲染指令进行图形渲染，而不是服务器进行图形渲染，从而提高画面质量，进而提升用户体验。

Description

用于图形渲染的方法及装置
本申请要求于2019年04月30日提交中国专利局、申请号为201910364218.8、申请名称为“用于图形渲染的方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及信息技术领域，并且更具体地，涉及一种用于图形渲染的方法及装置。
背景技术
传统模式下，手机应用程序(application，APP)运行在电子设备，电子设备要经过下载、更新等操作后才能使用APP。若APP尺寸大，就会存在下载时间长、占用电子设备存储空间大的问题，且APP功能冗余/尺寸膨胀是当前趋势，这种问题越来越不可避免。如果是游戏APP的话，安装在电子设备本地，各种游戏外挂层出不穷，难以从根本上禁绝，影响游戏公平性和体验。因此，提出了一种将电子设备中APP的逻辑运行环境与渲染环境分离，即APP运行在云侧(例如，服务器)，将渲染指令重定向到端侧(例如，电子设备)，二者分离的云手机/云游戏运行模式，也就是所谓的分离渲染(remote rendering，RR)。
基于分离渲染技术，现有技术中提出了视频流分离渲染的方案，即可以是在云侧的图形处理器(graphics processing unit，GPU)进行图形渲染生成视频流，将视频流进行编码发送至电子设备，对于电子设备而言只需要具备视频解码能力即可。但是，对于现有的视频流分离渲染的方案，由于为了降低云侧向端侧传输视频流所使用的带宽，会对云侧GPU进行图形渲染所生成的图片进行压缩，导致图片的画面质量较差，从而造成用户的体验感较差。
发明内容
本申请提供一种用于图形渲染的方法、服务器以及电子设备，电子设备通过获取的渲染指令进行图形渲染，得到与应用程序具有相同渲染效果的图像，从而提高画面质量提高用户的体验感。
第一方面，提供了一种用于图形渲染的方法，该方法应用于服务器。该方法包括：启动应用程序；获取所述应用程序发送的渲染指令；通过网络向电子设备发送所述渲染指令,所述渲染指令用于所述电子设备进行图形渲染,以便于在所述电子设备上显示与所述应用程序相关的图像，例如所述应用程序的程序画面或配置界面等等。以游戏应用为例，所述电子设备在GPU上执行所述渲染指令以在所述电子设备的显示设备上显示游戏画面等。
在本申请中，可以通过指令流方案实现分离渲染，即服务器可以获取应用程序发送(或者说生成)的渲染指令，将这些渲染指令发送至与服务器连接的电子设备，由接收到渲染 指令的电子设备进行图形渲染，从而避免了视频流分离渲染的方案中服务器进行图形渲染生成适配于固定屏幕尺寸的视频流，若电子设备与服务器渲染的画面尺寸不匹配则出现图像拉伸或压缩的现象。在本申请的技术方案中，服务器可以将用于图形渲染的渲染指令流发送至电子设备，一方面可以降低服务器图形处理器(GPU)的负载，另一方面将渲染指令发送至电子设备，由电子设备在端侧根据渲染指令进行图形渲染，可以更好的适配于电子设备的屏幕尺寸，提高画面质量从而提升用户的体验感。
在一些实现方式中，所述启动应用程序可以包括：服务器响应于第一指令启动应用程序，所述第一指令可以是电子设备向服务器发送的，也可以来自其他设备，也可以是服务器自己生成的。
上述渲染指令中可以包括渲染上下文所需的渲染信息，渲染信息可以包括状态信息和数据信息，例如，数据信息可以是多个顶点数据，状态信息可以是当前渲染上下文中绑定的顶点数据。
第二方面，本申请还提供一种断点传输的方法，该方法应可以用于以上指令流分离渲染的方案中，能够达到与视频流分离渲染方案相同或更好的断点续传效果。具体的，该方法包括：服务器保存图形渲染信息，所述图形渲染信息是所述电子设备用于图形渲染所需的信息，并适时向电子设备发送该图形渲染信息，使得所述电子设备根据所述图形渲染信息执行图形渲染。具体的，在电子设备断开连接(可能是网络故障也可以能设备故障)又重新连接之后，服务器接收所述电子设备发送的第二指令，所述第二指令用于请求恢复与所述服务器的连接；响应于所述第二指令，服务器向所述电子设备发所述图形渲染信息，所述图形渲染信息用于所述电子设备执行图形渲染。
在一些实现方式中，所述图形渲染信息可以包括渲染指令本身；在另一些实现方式中，所述图形渲染信息可以包括渲染指令和/或其它信息的变形形式。
在一些实现方式中，所述服务器保存图形渲染信息是响应于所述电子设备发送的所述第二指令执行的。换句话说，服务器保存的图形渲染信息仅是所述电子设备请求恢复与所述服务器连接时(以及之后的一段时间)内的图形渲染信息，对于电子设备而言，这些信息通常就它恢复到所述应用程序运行的最新状态的信息，这样保存有针对性也可以降低存储空间占用率。在另一些实现方式中，服务器也可以随时保存图形渲染信息或根据其他规则保存图形渲染信息。
在一些实现方式中，图形渲染信息为用于电子设备恢复所述应用程序当前最新的渲染上下文的信息，当前最新的渲染上下文可以是指服务器接收第二指令时所述应用程序的渲染上下文。
例如，应用程序是跑酷游戏，跑酷游戏在服务器中运行(类似云游戏的形式)，但用于可以在自己的电子设备上玩该游戏。当电子设备与服务器断开连接后，用户无法再进入游戏界面，但是游戏仍然在服务器中运行。当电子设备向服务器发送请求恢复连接的第二指令时，服务器向电子设备发送它保存的渲染信息，以使得电子设备恢复到跑酷游戏的当前最新的游戏界面，用户可以从当前最新游戏界面开始继续玩游戏，类似于视频流方案的断点续传。
再例如，应用程序是企业微信，企业微信在服务器中运行，用户使用电子设备获取企业微信发送的消息，当电子设备与服务器断开连接后，电子设备无法再接收消息，当电子 设备恢复与服务器的连接后，通过本申请提供的方法可以直接渲染并显示包含企业微信发送的最新的消息的消息界面。
对于视频流分离渲染的方案，由于视频流具有断点传输(或称断点续传)的特性，在服务器与电子设备断开连接进行重连时，服务器可以继续向电子设备发送视频流，实现断线重连。但是，对于指令流分离渲染的方案而言，由于用于图形渲染的渲染上下文之间存在依赖，所以渲染上下文不具备断点传输的特性。因此，在本申请中可以在服务器中保存图形渲染信息，当服务器接收到电子设备发送的请求恢复与服务器连接的指令后，服务器可以向电子设备发送保存的图形渲染信息，从而使得电子设备具有恢复渲染上下文所需的渲染信息，实现服务器和电子设备之间的断线重连。
在一些实现方式中，所述图形渲染信息可以保存在应用程序所运行的服务器中，也可以保存在其他的服务器或其他类型的设备中，或者图形渲染信息也可以采用分布式方式存储在多台设备上或采用远程存储方式存储在其它设备上。
在一些实现方式中，所述图形渲染信息为所述渲染指令。在本申请某些实施例中，服务器可以截取应用程序发送的渲染指令，对渲染指令进行拷贝，一份渲染指令发送至电子设备，另一份渲染指令则保存在服务器。
在一些实现方式中，所述图形渲染信息为渲染信息集合，所述渲染信息集合用于所述电子设备恢复所述应用程序的渲染上下文。在本申请某些实施例中，服务器可以对截取的渲染指令进行处理，以保存电子设备恢复应用程序渲染上下文所需的信息的最小集合，从而能够节约存储空间。
应理解，渲染信息集合可以是电子设备恢复应用程序渲染上下文所需的最小渲染信息。例如，假设渲染指令中可以包括100个顶点数据，电子设备可以根据100个顶点数据中的80个顶点数据进行图形渲染构建图形，则100个顶点数据中构建图形所需的80个顶点数据可以是最小渲染信息，若向电子设备发送81个顶点数据则存在冗余信息；若向电子设备发送79个顶点数据则电子设备无法构建图形。上述为举例说明，并不对本申请作任何限定。
在一些实现方式中，所述方法还包括：根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括恢复所述电子设备与所述服务器连接所需的渲染信息。例如，预配置的信息列表包括重传次数小于预定阈值的渲染信息。
应理解，预配置的信息列表可以是渲染上下文的最小集也可以是最优集，预配置的信息列表中信息过多会造成保存数据的冗余，数据过少不足以恢复渲染上下文进行图形渲染。
可选地，预配置的信息列表可以包括在N帧(N可以设置)图形中不会重传/重置的渲染信息。例如，预配置的信息列表可以是通过分析当前占据绝大多数市场的几款引擎以及典型APP得到的，通过不断抓取场景变换时、触摸反馈时、无输入时等情况下，渲染指令及渲染上下文的变化，经过反复对比和分析得到预配置的信息列表，在该信息列表中可以包括在N帧(N可以设置)图形中不会重传/重置的数据或状态。
在本申请实施例中，可以根据获取的渲染指令以及预配置的信息列表确定电子设备恢复应用程序渲染上下文所需的最小渲染信息，即确定渲染信息集合，从而保存渲染信息集合，在电子设备重新连接到服务器时，向所述电子设备发所述渲染信息集合，以便于所述 电子设备恢复断开与所述服务器连接时的图形渲染。
在本申请实施例中，为了节约断线重连所需要的等待时间，服务器可以向电子设备发送渲染信息集合，渲染信息集合可以是电子设备用于恢复应用程序渲染上下文所需的信息的最小集合。
在一些实现方式中，所述方法还包括：根据渲染顺序对所述渲染信息集合中的渲染信息进行重新排序，所述渲染顺序是指所述电子设备的渲染管线运行图形渲染时渲染信息的顺序；向所述电子设备发所述图形渲染信息，包括：向所述电子设备发送重新排序的所述渲染信息集合。
在本申请实施例中，可以对渲染信息集合中的渲染信息进行排序，即可以根据电子设备的渲染管线运行图形渲染时渲染信息的顺序对渲染信息集合进行排序，向电子设备发送重新排序的渲染信息集合，从而能够确保电子设备进行图形渲染时无需关心渲染信息的顺序，降低电子设备侧图形渲染的复杂化。
在一些实现方式中，服务器可以向电子设备发送全部图形渲染信息；或者，可以是服务器向电子设备发送图形渲染信息中的部分信息，然后根据电子设备的需求，例如电子设备的缓存列表，确定电子设备缺失的渲染信息，然后继续向电子设备发送该缺失的渲染信息，即实现按需发送；或者，服务器可以从一开始就根据电子设备的需求向电子设备发送缺失的渲染信息。
在一些实现方式中，所述方法还包括：接收所述电子设备发送的缓存列表，所述缓存列表是所述电子设备缓存的渲染信息的列表；向所述电子设备发所述图形渲染信息，包括：根据缓存列表向所述电子设备发送部分或全部所述图形渲染信息，发送的所述图形渲染信息为在所述缓存列表中不存在的图形渲染信息。具体的，服务器根据所述渲染指令中的一个或多个确定所述电子设备缺失的渲染信息；在所述缓存列表中查找所述缺失的渲染信息，若所述缓存列表中存在所述缺失的渲染信息，则向所述电子设备发送第三指令，所述第三指令用于指示所述电子设备从本地缓存中获取所述缺失的渲染信息；或者，若所述缓存列表中不存在所述缺失的渲染信息，则向所述电子设备发送所述缺失的渲染信息，所述渲染信息集合包括所述缺失的渲染信息。
例如，以OpenGL ES为例，服务器获取的渲染指令包括draw call，draw call又可以包括以下指令(仅示例，不限于以下指令)：
glDrawArrays、glDrawElements、glDrawArraysInstanced、glDrawElementsInstancedg、lDrawRangeElements、glDrawArraysIndirect、和glDrawElementsIndirect。
再例如，以OpenGL ES为例，服务器获取的渲染指令包括关键指令，关键指令可以包括但不限于以下指令：
glBindBuffer、glBufferData、glBufferSubData、glBindTexture、glTexImage2D、glTexSubImage2D、glTexImage3D、glTexSubImage3、glCompressedTexImage2D、glCompressedTexSubImage2D、glCompressedTexImage3D、glCompressedTexSubImage3D。在本申请中，服务器可以获取电子设备的缓存列表，并且监测上述draw call指令或关键指令，根据监测的渲染指令确定电子设备缺失的渲染信息，在缓存列表中查找是否存在缺失的渲染信息，当电子设备的缓存列表中不存在缺失的渲染信息时，再向电子设备发送缺失的渲染信息，从而能够加速电子设备恢复渲染上下文的过程，缩短服务器和电子设备之 间断线重连的等待时间。
在一些实现方式中，向所述电子设备发所述图形渲染信息，包括：确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合；向所述电子设备发送所述指令序列、以及所述渲染信息集合。
在本申请实施例中，为了确保电子设备实现渲染上下文的恢复，先要确保电子设备能够识别渲染信息集合，进一步可以确定指令序列，指令序列用于电子设备的图形处理器识别渲染信息集合。
在一些实现方式中，可以根据渲染信息中状态信息构造不同的指令序列。例如，若状态信息为已经链接过(linked)，则构造的指令序列可以包括创建、编译、挂载等指令；若状态信息为未链接过，则构造的指令序列可以包括重新挂载指令。
第三方面，提供了一种用于图形渲染的方法，该方法应用于电子设备。该方法包括：接收服务器发送的渲染指令，所述渲染指令用于电子设备进行图形渲染；根据所述渲染指令进行图形渲染显示与所述应用程序相关的图像。相比于视频流的云应用方案，该方案服务器发送的是渲染指令，然后在电子设备侧执行图形渲染，从而显示与应用相关的界面等，有益效果可参考前述。
在一些实现方式中，所述方法还包括：向服务器发送第一指令，所述第一指令用于指示所述服务器运行应用程序。
上述渲染指令可以包括用于图形渲染的渲染上下文，渲染上下文可以是图形渲染所需的资源集合，资源集合中可以包括状态信息和数据信息，例如，数据信息可以是多个顶点数据，状态信息可以是当前渲染上下文中绑定的顶点数据。
在本申请中，可以通过指令流方案实现分离渲染，即服务器可以获取应用程序发送的渲染指令，将渲染指令发送至与服务器连接的电子设备，由接收到渲染指令的电子设备进行图形渲染，从而避免了视频流分离渲染的方案中服务器进行图形渲染生成适配于固定屏幕尺寸的视频流，若电子设备与服务器渲染的画面尺寸不匹配则出现图像拉伸或压缩的现象。在本申请的技术方案中，服务器可以将用于图形渲染的渲染指令流发送至电子设备，一方面可以降低服务器中图形处理器的负载，另一方面将渲染指令发送至电子设备，由电子设备在端侧根据渲染指令进行图形渲染，可以更好的适配于电子设备的屏幕尺寸，提高画面质量从而提升用户的体验感。
在一些实现方式中，所述方法还包括：向服务器发送第二指令，所述第二指令用于请求恢复电子设备与所述服务器的连接；接收所述服务器发送的图形渲染信息，所述图形渲染信息是所述服务器保存的所述电子设备用于图形渲染所需的信息。
需要说明的是，对于视频流分离渲染的方案，由于视频流具有断点传输的特性，在服务器与电子设备断开连接进行重连时，服务器可以继续向电子设备发送视频流，实现断线重连。但是，对于指令流分离渲染的方案而言，由于用于图形渲染的渲染上下文之间存在依赖，即渲染上下文不具备断点传输的特性。
因此，在本申请中可以在服务器中保存图形渲染信息，当服务器接收到电子设备发送的请求恢复与服务器连接的指令后，服务器可以向电子设备发送保存的图形渲染信息，从而使得电子设备具有恢复渲染上下文所需的渲染信息，实现服务器和电子设备之间的断线重连。
在一些实现方式中，所述图形渲染信息为所述渲染指令。
在本申请中，服务器可以截取应用程序发送的渲染指令，对渲染指令进行拷贝，一份渲染指令可以发送至电子设备，另一份渲染指令可以在服务器中保存。
在一些实现方式中，所述图形渲染信息为渲染信息集合，所述渲染信息集合用于所述电子设备恢复所述应用程序渲染上下文。
在本申请中，服务器可以对截取的渲染指令进行处理保存电子设备恢复应用程序渲染上下文所需的最小渲染信息，即服务器可以保存渲染信息集合，从而能够节约服务器中的存储空间。
应理解，渲染信息集合可以是电子设备用于恢复应用程序渲染上下文所需的最小渲染信息。例如，假设渲染指令中可以包括100个顶点数据，电子设备可以根据100个顶点数据中的80个进行图形渲染构建图形，则100个顶点数据中构建图形所需的80个顶点数据最小渲染信息，若向电子设备发送81个顶点数据则存在冗余信息；若向电子设备发送79个顶点数据则电子设备无法构建图形。上述为举例说明，并不对本申请作任何限定。
在一种可能的实现方式中，接收所述服务器发送的所述图形渲染信息，包括：接收所述服务器发送的渲染信息集合。
在本申请中，为了节约电子设备与服务器断线重连所需要的等待时间，服务器可以向电子设备发送渲染信息集合，渲染信息集合可以是电子设备用于恢复应用程序渲染上下文所需的最小渲染信息。
在一种可能的实现方式中，接收所述服务器发送的渲染信息集合，包括：接收重新排序的所述渲染信息集合，所述重新排序的所述渲染信息集合是根据渲染顺序对所述渲染信息集合中的渲染信息进行重新排序生成的渲染信息集合，所述渲染顺序是指所述电子设备的渲染管线运行图形渲染时渲染信息的顺序。
在本申请中，可以对渲染信息集合中的渲染信息进行排序，即可以根据电子设备的渲染管线运行图形渲染时渲染信息的顺序对渲染信息集合进行排序，向电子设备发送重新排序的渲染信息集合，从而能够确保电子设备进行图形渲染时无需关心渲染信息的顺序，降低图形渲染的复杂化。
在一些实现方式中，所述方法还包括：确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合。
在本申请中，为了确保实现渲染上下文的恢复，即为了确保电子设备能够识别渲染信息集合，进一步可以确定指令序列，指令序列用于电子设备的图形处理器识别渲染信息集合。
在一些实现方式中，所述渲染信息集合中包括第一标识信息和第一渲染信息，所述第一标识信息为所述服务器用于标识所述第一渲染信息的标识信息，所述方法还包括：
确定映射关系，所述映射关系是所述第一标识信息和所述第二标识信息的对应关系，所述第二标识信息为所述电子设备用于标识所述第一渲染信息的标识信息。
例如，电子设备接收服务器发送的渲染信息集合，渲染信息集合中包括第一渲染信息，第一渲染信息中包括服务器设置的标识，例如标识为#5；电子设备的GPU根据该第一渲染信息进行图形渲染并向电子设备反馈标识，例如标识#4，则电子设备可以确定映射关系，即服务器的#5和电子设备的#4之间的对应关系。
第四方面，提供了一种用于图形渲染的方法，包括：接收电子设备发送的第四指令，所述第四指令用于指示所述电子设备与服务器断开连接；获取所述应用程序发送的渲染指令，所述渲染指令用于所述服务器进行图形渲染；接收所述电子设备发送的第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；保存图形渲染信息，所述图形渲染信息用于恢复渲染上下文，所述渲染上下文为接收所述第二指令时所述应用程序的渲染上下文；根据所述图形渲染信息进行图形渲染，生成视频流；向所述电子设备发送所述视频流。
应理解，在本申请的实施例中，服务器可以根据电子设备与服务器的连接情况进行按需渲染，即若电子设备与服务器处于连接态，则服务器的图形处理器根据渲染指令进行图形渲染生成视频流；若电子设备与服务器处于离线连接态，则服务器中的图形处理器不进行图形渲染。
在本申请中，当电子设备与服务器断开连接后，服务器可以不进行图形渲染，可以在服务器中保存图形渲染信息，当服务器接收到电子设备发送的请求恢复与服务器连接的指令后，服务器可以根据图形渲染信息进行图形渲染生成视频流，向电子设备发送视频流，使得服务器和电子设备之间实现按需图形渲染，从而能够减少服务器的负载。
在一些实现方式中，所述图形渲染信息为所述渲染指令。
在本申请中，服务器可以截取应用程序发送的渲染指令，对渲染指令进行拷贝，一份渲染指令可以发送至电子设备，另一份渲染指令可以在服务器中保存。
在一些实现方式中，所述图形渲染信息为渲染信息集合，所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
在本申请中，服务器可以保存渲染信息集合，即服务器中可以保存电子设备恢复应用程序渲染上下文的最小渲染信息，从而能够节约服务器中的存储空间。
在一些实现方式中，所述根据所述渲染指令确定图形渲染信息，包括：根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括重传次数小于预定阈值的渲染信息。
应理解，预配置的信息列表可以是渲染上下文的最小集也可以是最优集，预配置的信息列表中信息过多会造成保存数据的冗余，数据过少不足以恢复渲染上下文进行图形渲染。
可选地，预配置的信息列表可以包括在N帧(N可以设置)图形中不会重传/重置的渲染信息。
例如，预配置的信息列表可以是通过分析当前占据绝大多数市场的几款引擎以及典型APP得到的，通过不断抓取场景变换时、触摸反馈时、无输入时等情况下，渲染指令及渲染上下文的变化，经过反复对比和分析得到预配置的信息列表，在该信息列表中可以包括在N帧(N可以设置)图形中不会重传/重置的数据或状态。
在本申请中，可以根据获取的渲染指令以及预配置的信息列表确定电子设备恢复应用程序渲染上下文的最小渲染信息，即确定渲染信息集合，从而在服务器中保存渲染信息集合。
在一些实现方式中，所述方法还包括：确定指令序列，所述指令序列用于所述服务器的图形处理器识别所述渲染信息集合。
在本申请中，为了确保实现渲染上下文的恢复，即为了确保服务器的图形处理器能够识别渲染信息集合进行渲染生成视频流，进一步可以确定指令序列，指令序列用于服务器的图形处理器识别渲染信息集合。
第五方面，提供一种图形渲染装置，该装置包括用于执行第一方面中所描述的方法/操作/步骤/动作所对应的模块。
上述装置可以是服务器，也可以是服务器中的用于执行图形渲染的装置(例如，芯片，或者是能够和服务器匹配使用的装置)。
上述图形渲染装置包括的模块可以是硬件电路，也可是软件，也可以是硬件电路结合软件实现。
第六方面，提供一种图形渲染装置，该装置包括用于执行第二方面中所描述的方法/操作/步骤/动作所对应的模块。
上述装置可以是服务器，也可以是服务器中的用于执行图形渲染的装置(例如，芯片，或者是能够和服务器匹配使用的装置)。
上述图形渲染装置包括的模块可以是硬件电路，也可是软件，也可以是硬件电路结合软件实现。
第七方面，提供了一种图形渲染装置，该装置包括用于执行第三方面中所描述的方法/操作/步骤/动作所对应的模块。
上述装置可以是电子设备，也可以是电子设备中的用于执行图形渲染的装置(例如，芯片，或者是能够和电子设备匹配使用的装置)。
上述图形渲染装置包括的模块可以是硬件电路，也可是软件，也可以是硬件电路结合软件实现。
第八方面，提供一种图形渲染装置，该装置包括用于执行第四方面中所描述的方法/操作/步骤/动作所对应的模块。
上述装置可以是服务器，也可以是服务器中的用于执行图形渲染的装置(例如，芯片，或者是能够和服务器匹配使用的装置)。
上述图形渲染装置包括的模块可以是硬件电路，也可是软件，也可以是硬件电路结合软件实现。
第九方面，提供了一种图形渲染装置，该装置包括处理器，该处理器用于调用存储器存储的程序代码以执行上述第一方面中的任意一种方式中的部分或全部操作。
上述装置中，存储程序代码的存储器既可以位于图形渲染装置内部(图形渲染装置除了包括处理器之外，还可以包括存储器)，也可以位于图形渲染装置外部(可以是其他设备的存储器)。
可选地，上述存储器为非易失性存储器。
当图形渲染装置包括处理器和存储器时，该处理器和存储器可以耦合在一起。
第十方面，提供了一种图形渲染装置，该装置包括处理器，该处理器用于调用存储器存储的程序代码以执行上述第二方面中的任意一种方式中的部分或全部操作。
上述装置中，存储程序代码的存储器既可以位于图形渲染装置内部(图形渲染装置除了包括处理器之外，还可以包括存储器)，也可以位于图形渲染装置外部(可以是其他设备的存储器)。
可选地，上述存储器为非易失性存储器。
当图形渲染装置包括处理器和存储器时，该处理器和存储器可以耦合在一起。
第十一方面，提供了一种图形渲染装置，该装置包括处理器，该处理器用于调用存储器存储的程序代码以执行上述第三方面中的任意一种方式中的部分或全部操作。
上述装置中，存储程序代码的存储器既可以位于图形渲染装置内部(图形渲染装置除了包括处理器之外，还可以包括存储器)，也可以位于图形渲染装置外部(可以是其他设备的存储器)。
可选地，上述存储器为非易失性存储器。
当图形渲染装置包括处理器和存储器时，该处理器和存储器可以耦合在一起。
第十二方面，提供了一种图形渲染装置，该装置包括处理器，该处理器用于调用存储器存储的程序代码以执行上述第四方面中的任意一种方式中的部分或全部操作。
上述装置中，存储程序代码的存储器既可以位于图形渲染装置内部(图形渲染装置除了包括处理器之外，还可以包括存储器)，也可以位于图形渲染装置外部(可以是其他设备的存储器)。
可选地，上述存储器为非易失性存储器。
当图形渲染装置包括处理器和存储器时，该处理器和存储器可以耦合在一起。
第十三方面，提供了一种计算机可读存储介质，计算机可读存储介质存储了程序代码，其中，程序代码包括用于执行上述任一方面所描述的方法中的部分或全部操作的指令。
可选地，上述计算机可读存储介质位于电子设备内，该电子设备可以是能够进行图形渲染的装置。
第十四方面，本申请实施例提供一种计算机程序产品，当计算机程序产品在通信装置上运行时，使得通信装置执行上述任一方面所描述的方法中的部分或全部操作。
第十五方面，提供了一种芯片，所述芯片包括处理器，所述处理器用于执行上述任一方面所描述的方法中的部分或全部操作。
第十六方面，提供了一种系统，包括前述的电子设备以及前述的服务器。
附图说明
图1是根据本申请一实施例提供的用于图形渲染的系统架构的示意图；
图2是根据本申请另一实施例提供的用于图形渲染的系统架构的示意图；
图3是本申请中电子设备根据获取的渲染指令进行图形渲染的示意性流程图；
图4是根据本申请一实施例提供的用于图形渲染的方法的示意性流程图；
图5是根据本申请另一实施例提供的用于图形渲染的方法的示意性流程图；
图6是根据本申请一实施例提供的确定状态机快照的示意性流程图；
图7是根据本申请已实施例提供的按需加载方法的示意性流程图；
图8是根据本申请一实施例提供的图形渲染装置的结构性示意图；
图9是根据本申请另一实施例提供的图形渲染装置的结构性示意图；
图10是根据本申请一实施例提供的服务器的结构性示意图；
图11是根据本申请一实施例提供的电子设备的结构性示意图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
本申请实施例中用于图形渲染方法可以由电子设备来执行。该电子设备可以是移动终端(例如，智能手机)，电脑，个人数字助理，可穿戴设备，车载设备，物联网设备或者其他能够进行图形渲染处理的设备。该电子设备可以是运行安卓系统、IOS系统、windows系统以及其他系统的设备。
本申请实施例用于图形渲染的方法可以实现指令流分离渲染的方案，其中，指令流分离渲染的方案是指可以在云侧(例如，服务器)截取APP发送的渲染指令，该渲染指令用于电子设备进行图形渲染，渲染指令中可以包括渲染上下文所需的渲染信息，渲染信息可以包括状态信息和数据信息，例如，数据信息可以是多个顶点数据，状态信息可以是当前渲染上下文中绑定的顶点数据。
本申请实施例的用于图形渲染的方法可以应用于图1所示系统架构中。
如图1所示，系统架构中可以包括服务器110和电子设备120，服务器110中可以包括一个或者多个应用程序APP，服务器110中还包括拦截机，拦截机可以用于截取APP发送的渲染指令，通过网络传输控制协议(transmission control protocol，TCP)或代理agent将渲染指令重定至电子设备，电子设备接收到渲染指令后，电子设备中的渲染管线可以建立、维护渲染上下文，进行图形渲染操作。其中，拦截机可以是在服务器运行用于获取APP渲染指令的一段程序，例如，拦截机获取的渲染指令可以是OpenGL ES 2.0、OpenGL ES 3.0或者EGL。
应理解，如图1所示的指令流分离渲染的方案中，当电子设备与服务器连接时，可以在电子设备的GPU中进行图形渲染，即可以将用于图形渲染的渲染指令重定向至电子设备，从而电子设备的GPU可以根据渲染指令进行图形渲染。例如，可以根据渲染指令生成适配于电子设备的屏幕尺寸的图形，从而能够提高图形的清晰度，提升用户的体验感。
在视频流分离渲染的方案中，服务器的GPU可以获取应用程序发送的渲染指令进行图形渲染，生成视频流。由于视频流中的图像之间无状态依赖，因此电子设备与服务之间可以实现断线重连。即当电子设备与服务器之间从离线状态切换至连接状态时，电子设备可以继续获取服务器发送的视频流，实现与服务器的断线连接。但是，对于指令流分离渲染的方案，当电子设备与服务器断开连接时，电子设备无法再继续接收到服务器发送的渲染指令，由于电子设备缺失部分图形渲染所需的渲染信息且渲染信息存在状态依赖，导致电子设备无法继续进行图形渲染。为了在指令流分离渲染的方案中能够实现电子设备与服务器的断线重连，可以采用如图2所示的系统架构。
如图2所示，可以在系统架构中设置保存模块和恢复模块，其中，保存模块可以位于服务器中并且可以被任意APP使用，拦截机截取的所有渲染指令可以发送至电子设备，同时可以将渲染指令进行拷贝保存至保存模块中。此外，保存模块还可以生成快照，快照是指电子设备用于恢复渲染上下文的信息集合的副本。恢复模块可以位于服务器中，或者，也可以位于电子设备中，恢复模块用于电子设备重建渲染上下文。
下面结合图3对电子设备的具体结构进行详细的介绍。图3所示的电子设备可以是图1所示的系统架构中的电子设备110，或者，也可以是图2所示的系统架构中的电子设备140。
在一个实施例中，如图3所示，电子设备100可以包括：中央处理器(CPU)101、图形处理器(GPU)102、显示设备103和存储器104。可选地，该电子设备100还可以包括至少一个通信总线110(图3中未示出)，用于实现各个组件之间的连接通信。
应当理解，电子设备100中的各个组件还可以通过其他连接器相耦合，其他连接器可包括各类接口、传输线或总线等。电子设备100中的各个组件还可以是以处理器101为中心的放射性连接方式。在本申请的各个实施例中，耦合是指通过相互电连接或连通，包括直接相连或通过其他设备间接相连。
中央处理器101和图形处理器102的连接方式也有多种，不局限于图3所示的方式。电子设备100中的中央处理器101和图形处理器102可以位于同一个芯片上，也可以分别为独立的芯片。
下面对中央处理器101、图形处理器102、显示设备103的作用进行简单的介绍。
中央处理器101：用于运行操作系统105和应用程序107。应用程序107可以为图形类应用程序，比如游戏、视频播放器等等。操作系统1005提供了系统图形库接口，应用程序107通过该系统图形库接口，应用程序107可以接收服务器发送的用于渲染图形或图像帧的指令流(例如，渲染指令)。操作系统105提供的驱动程序，比如图形库用户态驱动和/或图形库内核态驱动，生成图形处理器102中渲染管线能够识别的指令流以及所需的相关渲染数据。其中，系统图形库包括但不限于：嵌入式开放图形库(open graphics library for embedded system，OpenGL ES)、柯罗诺斯平台图形界面(the khronos platform graphics interface)或Vulkan(一个跨平台的绘图应用程序接口)等系统图形库。指令流包含一些列的指令，这些指令通常为对系统图形库接口的调用指令。
可选地，中央处理器101可以包括以下至少一种类型的处理器：应用处理器、一个或多个微处理器、数字信号处理器(digital signal processor,DSP)、微控制器(microcontroller unit，MCU)或人工智能处理器等。
中央处理器101还可进一步包括必要的硬件加速器，如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)、或者用于实现逻辑运算的集成电路。处理器101可以被耦合到一个或多个数据总线，用于在电子设备100的各个组件之间传输数据和指令。
图形处理器102：用于接收处理器101发送的图形指令流，通过渲染管线(pipeline)生成渲染目标，并通过操作系统的图层合成显示模块将渲染目标显示到显示设备103。
可选地，图形处理器102可以包括执行软件的通用图形处理器，如GPU或其他类型的专用图形处理单元等。
显示设备103：用于显示由电子设备100生成的各种图像，该图像可以为操作系统的图形用户界面(graphical user interface，GUI)或由图形处理器1002处理的图像数据(包括静止图像和视频数据)。
可选地，显示设备103可以包括任何合适类型的显示屏。例如液晶显示器(liquid crystal display，LCD)或等离子显示器或有机发光二极管(organic light-emitting diode，OLED)显示器等。
渲染管线是图形处理器102在渲染图形或图像帧的过程中顺序执行的一系列操作，典型的操作包括：顶点处理(vertex processing)、图元处理(primitive processing)、光栅化 (rasterization)、片段处理(fragment processing)等等。
上文中介绍了本申请实施例适用的系统架构以及电子设备内部执行图形渲染的流程。下面结合附图4对本申请实施例中的用于图形渲染的方法进行详细的介绍。其中，服务器可以是图1或图2所示的服务器，电子设备可以是图1至图3中任一个所示的电子设备。
图4所示的方法包括步骤210至240，下面分别对这些步骤进行详细的描述。
210、服务器接收电子设备发送的第一指令，第一指令用于指示服务器运行应用程序。
在本申请的实施例中，服务器中可以包括一个或多个应用程序，应用程序的客户端可以安装在电子设备上。电子设备可以向服务器发送指令，启动服务器中应用程序的运行。在其他实施例中，服务器可以自行启动应用程序，不需要电子设备触发。
220、服务器获取所述应用程序发送的渲染指令，渲染指令用于电子设备进行图形渲染。
例如，可以如图1或者图2所示，服务器中的拦截机用于拦截应用程序发送的渲染指令，其中，拦截机可以是在服务器运行用于获取APP渲染指令的一段程序。
230、服务器将渲染指令发送至电子设备。
240、电子设备根据渲染指令执行渲染以显示所述应用程序。
在本申请的实施例中，服务器可以将拦截机获取的应用程序的渲染指令发送至电子设备。例如，在图1或图2所示的系统架构中，服务器可以通过TCP或代理(agent)将渲染指令发送至电子设备。
进一步地，在本申请的实施例中，服务器可以保存图形渲染信息，所述图形渲染信息是所述电子设备用于图形渲染所需的信息。其中，图形渲染信息可以是截取的应用程序发送的渲染指令，或者，图形渲染信息也可以是对截取的应用程序发送的渲染指令进行处理后得到的渲染信息集合。
例如，如图2所示，在服务器130中可以对截取的应用程序的渲染指令进行处理，一份发送至电子设备，使得电子设备根据渲染指令进行图形渲染，此外另一份可以保存至电子设备中。
例如，服务器可以对截取的渲染指令进行处理保存电子设备恢复应用程序渲染上下文所需的最小渲染信息，即服务器可以保存渲染信息集合。
应理解，渲染信息集合可以是电子设备恢复应用程序渲染上下文所需的最小渲染信息。例如，假设渲染指令中可以包括100个顶点数据，电子设备可以根据100个顶点数据中的80个顶点数据进行图形渲染构建图形，则100个顶点数据中构建图形所需的80个顶点数据可以是最小渲染信息，若向电子设备发送81个顶点数据则存在冗余信息；若向电子设备发送79个顶点数据则电子设备无法构建图形。上述为举例说明，并不对本申请作任何限定。
可选地，在本申请的实施例中，服务器可以接收所述电子设备发送的第二指令，所述第二指令用于请求恢复与所述服务器的连接；服务器可以向电子设备发所述图形渲染信息。
在本申请的实施例中，渲染信息集合可以是由服务器确定的。例如，服务器可以根据配置的信息列表(例如，图5所示的关键资源列表)和渲染指令确定渲染信息集合(例如，图5所示的关键资源集合)，预配置的信息列表可以包括恢复所述电子设备与所述服务器 连接所需的渲染信息。
例如，预配置的信息列表可以是通过分析当前占据绝大多数市场的几款引擎以及典型APP得到的，通过不断抓取场景变换时、触摸反馈时、无输入时等情况下，渲染指令及渲染上下文的变化，经过反复对比和分析得到预配置的信息列表，在该信息列表中可以包括在多帧(可定阈值)图形中不会重传/重置的数据或状态。
例如，对于同一状态信息，多帧图形的数据信息的传输过程可以是数据A-数据B-数据C-数据B-数据C，则可以看出在该多帧图形数据中数据B和数据C会进行重复的传输，则其中数据A是在多帧图形中不会重传/重置的数据信息。因此，在这种情况下，信息列表中可以包括数据A或者数据A的索引值。
应理解，预配置的信息列表可以是渲染上下文的最小集也可以是最优集，预配置的信息列表中信息过多会造成保存数据的冗余，数据过少不足以恢复渲染上下文进行图形渲染。
可选地，在本申请的实施例中服务器接收到电子设备发送的用于请求恢复与服务器连接的指令后，可以向电子设备发送服务器保存的图形渲染信息。
在一个示例中，电子设备可以接收服务器保存的渲染指令，渲染指令可以是服务器截取应用程序发送的渲染指令。
在一个示例中，为了节约断线重连所需要的等待时间，电子设备可以接收到服务器发送的渲染信息集合，渲染信息集合可以是电子设备用于恢复应用程序渲染上下文所需的最小渲染信息。
应理解，在本申请的实施例中，服务器保存的渲染指令可以是未经处理的应用程序发送的指令，或者，服务器可以进一步节约存储空间对获取的渲染指令进行处理保存得到的渲染信息集合。
在本申请的实施例中，为了确保渲染上下文的恢复，即为了确保电子设备能够识别渲染信息集合，进一步地可以确定指令序列，例如，构造指令序列，其中，指令序列用于电子设备的图形处理器识别渲染信息集合。
例如，可以在服务器中确定指令序列，当服务器向电子设备发送指令序列后，电子设备的GPU可以识别指令序列中的用于图形渲染的信息。
例如，可以在电子设备中确定指令序列。具体地，可以是服务器向电子设备发送渲染信息集合，电子设备的CPU接收渲染信息集合后，电子设备的CPU确定指令序列，将指令序列由电子设备的CPU发送至电子设备的GPU，使得电子设备的GPU根据渲染信息集合进行图形渲染。
在一个示例中，渲染信息集合中可以包括第一标识信息和第一渲染信息，第一标识信息为可以是服务器用于标识所述第一渲染信息的标识信息，电子设备可以确定映射关系，映射关系是第一标识信息和第二标识信息的对应关系，第二标识信息为电子设备用于标识第一渲染信息的标识信息。
例如，电子设备接收服务器发送的渲染信息集合，渲染信息集合中包括第一渲染信息，第一渲染信息中包括服务器设置的标识，例如标识为#5；电子设备的GPU根据该第一渲染信息进行图形渲染并向电子设备反馈标识，例如标识#4，则电子设备可以确定映射关系，即服务器的#5和电子设备的#4之间的对应关系。
在一个示例中，为了确保电子设备恢复应用程序渲染上下文时降低复杂化，服务器可以根据渲染顺序对渲染信息集合中的渲染信息进行重新排序，渲染顺序是指电子设备的渲染管线运行图形渲染时渲染信息的顺序；服务器可以向电子设备发送重新排序的所述渲染信息集合。
可选地，在本申请的实施例中，电子设备可以向服务器发送缓存列表，缓存列表中包括电子设备在本地缓存的渲染信息；进一步地，当服务器接收到电子设备发送的用于请求恢复与所述服务器的连接后，服务器可以先监测渲染指令，确定电子设备可能缺失的渲染信息。在电子设备的缓存列表中查找缺失的渲染信息，若缓存列表中存在缺失的渲染信息，则向电子设备发送第三指令，第三指令用于指示电子设备从本地缓存中获取缺失的渲染信息；或者，在电子设备的缓存列表中查找缺失的渲染信息，若缓存列表中不存在缺失的渲染信息，则向电子设备发送缺失的渲染信息。
需要说明的是，在本申请的实施例中，服务器可以先监测应用程序发送的draw call或者关键指令，确定电子设备恢复渲染上下文所需的资源，进一步服务器会判断是否对该资源进行过恢复；若服务器确定未对该资源进行过恢复，则服务器可以在电子设备的缓存列表中进行查找，从而确定电子设备的缓存列表中是否存在该资源，当缓存列表中存在该资源时，服务器可以指示电子设备从其本地缓存中获取该资源；当缓存列表中不存在该资源时，服务器可以向电子设备发送该资源。若服务器确定对该资源进行过恢复，则服务器不执行任何操作。其中，恢复渲染上下文所需的资源也就是电子设备恢复渲染上下文所需的渲染信息，即可以是电子设备缺失的渲染信息。
应理解，电子设备在本地缓存的渲染信息可以包括数据信息且不包括状态信息，也就是说，电子设备在本地缓存的渲染信息可以是用于图形渲染的数据信息。
例如，以OpenGL ES为例，draw call可以包括但不限于以下指令：
glDrawArrays、glDrawElements、glDrawArraysInstanced、glDrawElementsInstanced、glDrawRangeElements、glDrawArraysIndirect、glDrawElementsIndirect。
例如，以OpenGL ES为例，关键指令可以包括但不限于以下指令：
glBindBuffer、glBufferData、glBufferSubData、glBindTexture、glTexImage2D、glTexSubImage2D、glTexImage3D、glTexSubImage3D、glCompressedTexImage2D、glCompressedTexSubImage2D、glCompressedTexImage3D、glCompressedTexSubImage3D。
在本申请中，可以通过指令流方案实现分离渲染，即服务器可以获取应用程序发送的渲染指令，将渲染指令发送至与服务器连接的电子设备，由接收到渲染指令的电子设备进行图形渲染，从而避免了视频流分离渲染的方案中服务器进行图形渲染生成适配于固定屏幕尺寸的视频流，若电子设备与服务器渲染的画面尺寸不匹配则出现图像拉伸或压缩的现象。在本申请的技术方案中，服务器可以将用于图形渲染的渲染指令流发送至电子设备，一方面可以降低服务器中图形处理器的负载，另一方面将渲染指令发送至电子设备，由电子设备在端侧根据渲染指令进行图形渲染，可以更好的适配于电子设备的屏幕尺寸，提高画面质量从而提升用户的体验感。
本申请还提供了一种用于图形渲染的方法，该方法可以使得视频流分离渲染的方案实现按需渲染，即服务器可以根据电子设备与服务器的连接情况进行按需渲染，即若电子设备与服务器处于连接态，则服务器的图形处理器可以根据渲染指令进行图形渲染生成视频 流；若电子设备与服务器中处于离线状态，则服务器中的图形处理器不进行图形渲染。
在一个示例中，本申请提出了一种用于图形渲染的方法，包括：服务器接收电子设备发送的第四指令，所述第四指令用于指示所述电子设备与服务器断开连接；服务器获取所述应用程序发送的渲染指令，所述渲染指令用于所述服务器的图形处理器进行图形渲染生成视频流；服务器根据所述渲染指令确定图形渲染信息，所述图形渲染信息是所述服务器生成视频流所需的渲染信息；服务器保存所述图形渲染信息。
需要说明的是，上述本申请的实施例也适用于视频流分离渲染的方案，对于视频流分离渲染的方案与指令流分离渲染的方案存在的差异在于，指令流分离渲染的方案将保存的图形渲染信息发送至电子设备，电子设备根据图形渲染信息进行图形渲染；而对于视频流分离渲染的方案，可以将保存的图形渲染信息发送至服务器的图形处理器中，由服务器的图形处理器根据图形渲染指令进行图形渲染生成视频流，向电子设备发送视频流。上述实施例中，保存图形渲染信息以及恢复渲染上下文的过程同样适用于视频流分离渲染的方案，此处不再赘述。
可选地，在本申请的实施例中，当服务器接收到电子设备发送的第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；服务器根据所述图形渲染信息进行图形渲染，生成视频流；服务器向所述电子设备发送所述视频流。
在本申请中，当电子设备与服务器断开连接后，服务器可以不进行图形渲染，可以在服务器中保存图形渲染信息，图形渲染信息可以是服务器用于图形渲染所需的信息，当服务器接收到电子设备发送的请求恢复与服务器连接后，服务器可以根据图形渲染信息进行图形渲染，生成视频流，向电子设备发送视频流，使得服务器和电子设备之间实现按需渲染，进一步能够减轻服务器GPU的压力，从而在视频流分离渲染的方案中服务器运行的实例密度得到显著提升。
在一个示例中，图形渲染信息可以是渲染指令，即服务器中保存的渲染信息可以是获取的应用程序发送的渲染指令。
在一个示例中，图形渲染信息可以是渲染信息集合，渲染信息集合用于所述电子设备恢复所述应用程序渲染上下文。
可选地，在本申请的实施例中，所述根据所述渲染指令确定图形渲染信息，包括：根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括恢复所述电子设备与所述应用程序连接所需的渲染信息。
可选地，在本申请的实施例中，服务器还可以确定指令序列，所述指令序列用于所述服务器的图形处理器识别所述渲染信息集合。
例如，服务器的中央处理器确定指令序列，指令序列用于服务器的图形处理器能够识别渲染信息集合，从而使得服务器的图形处理器根据渲染信息集合进行图形渲染生成视频流。
图5是本申请实施例提供的一种用于图形渲染的方法的示意性流程图。图5所示的方法可以包括步骤301至310，下面分别对步骤301至310进行详细描述。其中，端侧/渲染端可以是图1至图4中所示的任意一个电子设备，APP、状态机保存模块、状态机可以位于服务器中，状态机恢复模块可以位于电子设备或服务器。应理解，在本申请中状态机可以是渲染上下文。
步骤301：端侧/渲染端首次连接。即可以是端侧/渲染端与服务器中的APP首次连接。
步骤302：端侧/渲染端可以向APP发送启动指令。
例如，可以是图4中的第一指令。
步骤303：云侧更新和维护状态机。
需要说明的是，云侧更新和维护状态机可以是服务器根据接收到APP发送的渲染指令，使用相应的数据结构对图形渲染所必须的数据和状态进行实时保存和更新。
步骤304：进行远程渲染，即可以是将APP发送的渲染指令发送至电子设备，电子设备根据获取的渲染指令进行图形的渲染。
步骤305：状态机保存模块接收到端侧/渲染端发送的请求状态机恢复/重建的指令。
其中，请求状态机恢复/重建的指令可以是上述实施例中的第二指令，第二指令用于请求恢复电子设备与服务器的连接。
在本申请中，云侧接收到端侧发送的请求状态机恢复/重建触发命令或其它触发命令后，根据预设的关键资源列表提取状态机中的关键资源；进一步地，可以对提取出的关键资源，根据资源依赖关系(可以是上述实施例中的渲染顺序)进行资源重排，并生成快照。其中，快照可以是某一时刻关键资源集合的副本。
图6是本申请实施例提供的服务器生成快照的示意性图。图6中包括步骤410至440，下面分别对步骤410至440进行详细描述。
步骤410：获取关键资源列表和状态机。
其中，渲染上下文所需的资源即渲染信息，资源是由数据信息和状态信息组成，以android分离渲染为例(也可以适用于IOS)，又可把资源可以分为EGL层和OpenGL ES层，其中OpenGL ES资源又由以下部分组成：
OpenGL ES状态信息可以包括但不限于以下内容：
1、各种全局状态的绑定，如buffer与target的绑定，当前各类参数的设置等；
2、顶点属性指针的设定；
3、所有Texture、EGLImage、Sampler的绑定；
4、Framebuffer object、RenderBuffer object的绑定；
5、Program和Shader的绑定及其当前状态；
6、vertex Attribute顶点属性和uniform变量的设定。
这些绑定关系(即状态)对应的数据，可以包括但不限于以下内容：
1、buffer data；
2、sampler data；
3、Texture data；
4、FrameBuffer/RenderBuffer data；
5、shader string data。
在本申请的实施例中，关键资源列表可以是预配置的信息列表，可以通过分析各游戏引擎在不同情形下的渲染指令序列，抽象出关键资源列表，关键资源列表可以是渲染上下文的最小集也是最优集，其中，关键资源列表中信息过多会让快照膨胀，关键资源列表中过少会不足以恢复渲染上下文，因此对其的抽象与定义至关重要。
对关键资源列表的抽象可以是个静态的过程，即对关键资源列表的获取可能无法通过 特定的API或方法从当前服务器的CPU或GPU中获得。可以通过分析当前占据绝大多数市场的几款引擎以及典型APP，不断抓取场景变换时、触摸反馈时、无输入时等情况下，渲染指令及渲染上下文资源的变化，经过反复对比和分析，得出关键资源列表包括在多帧(预设阈值)中不会重传/重置的数据或状态。
例如，确定关键资源列表可以包括但不限于以下步骤：
步骤1：对引擎进行分类，确定抓取不同场景的渲染指令；
步骤2：确定某一场景，设置预设帧数，如100帧；
步骤3：确定预设帧数量的图像中重传次数较少的渲染信息；
步骤4：在不同的场景下，重复执行上述步骤1至步骤3，最终不同场景下获取的传次数较少的渲染信息的并集被确定为关键资源列表。
除以上步骤之外，关键资源列表还可以结合触摸反馈、无输入等其他情况下渲染信息传输情况来获得。
应理解，在本申请的实施例中，关键资源列表也可以是基于人工智能(artificial intelligence，AI)技术得到的，其中，AI可以是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。例如，关键资源列表可以是基于神经网络算法，或者，计算机算法模型得到的。上述为举例说明，并不对关键资源列表的来源方式进行任何限定。
为了提高效率，关键资源列表通常是线下获取的，但是如果有效率很高的获取方式，关键资源列表也可以通过该方式在线上实时获取，本申请对此不做限定。
在一个示例中，关键资源可以包括但不限于以下资源：
1、少数或者仅在某个场景第一次切换进入才会更新的全局状态；
2、会调用sub指令进行更新的所有指令对应的数据，即上述五类数据中的buffer和texture数据，但不包含绝大多数纹理属性的设置状态；
3、program/shader string的所有资源和当前状态，包含compiled状态、attached状态、uniform设置变量以及少数不会频繁更新的vertex Attribute顶点属性；
4、frame buffer/render buffer object的attach点状态(绑定)及其对应的数据，但不包含glreadbuffer/gldrawbuffers所设置的参数；
对于大部分全局状态的设置，以及program对应的attributes顶点属性指针设置等信息，都会在每个draw call中不断进行更新，因此这部分状态或数据显然可以属于非关键资源，无需恢复，可以在后续的draw call中获取。
在本申请中，状态机可以实现完整的EGL/OpenGL ES标准，将拦截所有APP发送的渲染指令进行调用，渲染指令所包含资源可能分属不同Context，判别当前渲染指令所属OpenGL ES版本后分发到相应的Context，然后再根据该渲染指令包含的资源类型，最终分发到不同的Object Resource中，每种Object Resource可以由很多个Object Resource实例组成。
在服务器中可以截获APP发送的渲染指令，一份正常发往电子设备中，一份重定向到服务器的状态机中，对渲染上下文资源和状态的修改，如执行资源创建，删除，绑定关系维护等操作，都会同步到Context的各ObjectResource中，确保服务器中的状态机完备，状态迁移能够持续进行。
需要说明的是，服务器中可以维护一个状态机，它存在的目的是能持续接收从APP拦截得来的渲染指令，并能更新到相应的数据结构中，在电子设备请求状态机恢复时，通过关键资源列表，从状态机中抽取出关键资源集合。
步骤420：关键资源集合，即服务器根据步骤410中关键资源列表和状态机确定关键资源集合。
应理解的是，关键资源列表可以是预配置的信息列表。在端侧请求状态机恢复时，服务器可以通过关键资源列表从状态机中抽取出关键资源集合。
步骤430：资源重排。
在本申请中，由于不同种类的资源存在之间依赖关系，因此可以重排相互依赖的资源，确保恢复渲染上下文时无需关心资源依赖关系而令恢复时指令构造复杂化。其中，依赖关系可以是渲染顺序，渲染顺序是指电子设备的渲染管线运行图形渲染时渲染信息的顺序。
例如，全局资源的保存须在各类Object Data之后进行；纹理/Images的资源保存须在FrameBuffer之前进行等。资源的保存顺序(以Android状态机保存为例)可以按照以下顺序：EGL层资源-ObjectResource<Buffer>-ObjectResource<VertexArray>-ObjectResource<Texture>-ObjectResource<Sampler>-ObjectResource<Program/Shader>-ObjectResource<RenderBuffer>-ObjectResource<FrameBuffer>-全局状态。
步骤440：确定状态机快照。
在本申请中可以根据资源重排确定关键资源(数据+状态)，生成状态机快照，可以持久化也可以选择常驻云侧内存。
通过图6所示的服务器生成快照的示意性图之后，可以执行图5所示的用于图形渲染的方法的示意性流程图中的步骤306，发送状态机快照。其中，状态机快照可以是图4所示的实施例中的重新排序的所述渲染信息集合。
步骤307：触发渲染上下文重建操作。
应理解，渲染上下文的重建操作可以在电子设备中执行，也可以在服务器中执行。
下面以渲染上下文的重建操作在电子设备中执行举例说明。
在上述步骤430中，服务器可以根据渲染顺序对关键资源集合进行重排，故渲染上下文的重建即渲染上下文恢复时，可以无需关注资源顺序问题而造成资源恢复失败的问题。需要根据当前每类资源状态，产生不同的指令序列即可，即需要根据当前每类资源状态构造电子设备的GPU能够识别的指令序列，以Program/Shader相关资源的恢复为例进行说明：
Figure PCTCN2020083772-appb-000001
Figure PCTCN2020083772-appb-000002
上述代码可以是构造指令序列的伪代码，其中，每种Object Resource都由很多个Object Resource实例组成，每种Object Resource都维护着一个map，如Program/Shader ObjectResource的map<serverHandle,program>，用于保存该类Object Resource的实例标识和实例本身的映射，对于每个实例在状态机保存动作触发时，都会处于一种确定的状态。
对Program/Shader ObjectResource，遍历ObjectResource<Program/Shader>，无论当前实例所处状态如何，都要将该标识作为参数，构造指令发往端侧，这会触发端侧建立一个端侧资源标识与云侧资源标识的映射map<serverHandle,clientHandle>，他们标识的是同样的数据，在状态机恢复完成后，用于操作端侧资源。
需要说明的是，渲染上下文的重建过程可以是指将云侧保存的渲染上下文中的数据和状态进行拆解后，根据当前每种资源的状态，按照GPU API的标准及一定的规则和顺序，构造出符合GPU pipeline(渲染管线)架构的指令序列发送给端侧的渲染线程(RenderThread)执行图形渲染，这个过程会将指令序列中包含的资源重新加载至端侧的GPU中，相应地，端侧的GPU也会重新生成一个用于操作该资源的标识。渲染上下文重建的过程中会对所有重新加载进端侧的GPU的资源重新标识，若想在重建出的渲染上下文中接着进行图形渲染，就需要“识别”这些“新的”资源，因此需要一个资源映射机制，即云侧的渲染上下文资源标识与渲染端上下文资源标识之间的映射关系。
例如，电子设备可以确定资源映射，即云侧构造的渲染指令中包含资源及该资源的标识号，端侧在恢复时执行该指令，也会在端侧生成一个对该资源的标识，并将其与云侧发送过来的资源标识建立一个映射关系map<serverHandle,clientHandle>。渲染上下文恢复完成，进入正常渲染阶段时，云侧通过其对该资源的标识来操作该资源，端侧则可以通过资源映射后端侧标识代替云侧标识后再进行图形渲染操作，即端侧可以访问云侧任意的数据或状态。
建立端侧和云侧之间资源标识的映射后，可以读取快照资源中该实例所处的状态，因为需要状态机保存操作触发前那一刻的渲染上下文，所以可以根据状态确定不同的指令序列。如当前program实例已经链接过(linked)，就要对挂载点上的shaders实例进行创建、编译、挂载，然后设置少数不会频繁更新的vertex Attribute顶点属性，然后使用该program实例(通过实例标识来使用)，对关键资源中的uniform变量进行设置；而若当前program实例是未链接过，那就只需要将program实例挂载点上的shaders重新挂载即可。上述为举例说明，并不对本申请作任何限定。
需要说明的是，上述以ObjectResource<Program/Shader>进行举例说明，对于其他种类的ObjectResource，均可以采用类似于ObjectResource<Program/Shader>根据状态机保存操作触发前保存的每个实例的状态进行定制化的指令序列构造。
图7是本申请实施例提供的加速恢复渲染上下文的方法的示意性流程图。图7中包括步骤501至511，下面分别对步骤501至511进行详细描述。
需要说明的是，在本申请的实施例中，端侧可以是图1至图4所示的任意一个电子设备，云侧可以是图1、图2以及图3所示的任意一个服务器。
步骤501：资源按需加载模式触发。
需要说明的是，按需加载模式可以在构造指令序列即状态机恢复模块运转初期便生效，云侧可以监测端侧的draw call或关键指令，只加载/恢复当前帧或关键指令的强依赖资源而延后加载其他资源。另外，可以是端侧渲染上下文恢复/重建完成后，继续渲染的同时可以根据按需恢复/资源缓存加速算法运行，云侧可以持续监测draw call或关键指令，持续为后续的当前帧或关键指令的强依赖资源加载服务。
步骤502：获取端侧缓存资源列表。
例如，端侧中可以缓存在状态机中占比较大的纹理资源和顶点资源(75％～99.7％)，将这两部分资源缓存在端侧，当恢复渲染上下文时可以直接从端侧的缓存数据库中读取并加载到端侧的GPU中，可显著降低渲染上下文恢复过程中网络的数据传输量，从而可以降低恢复渲染上下文所需等待的时间。
步骤503：构造指令序列，即上述实施例中的确定指令序列。
例如，渲染上下文的重建过程可以是指将云侧保存的渲染上下文中的数据和状态进行拆解后，根据当前每种资源的状态，按照GPU API的标准及一定的规则和顺序，构造出符合端侧的GPU pipeline(渲染管线)架构的指令序列。
步骤504：渲染上下文恢复主要流程完成。
其中，渲染上下文恢复主要流程完成可以对应于图5所示的步骤305至步骤309。
步骤505：按需加载一直运行，持续监测电子设备的draw call/关键指令。
以OpenGL ES为例，draw call可以包括但不限于以下指令：
glDrawArrays、glDrawElements、glDrawArraysInstanced、glDrawElementsInstanced、glDrawRangeElements、glDrawArraysIndirect、glDrawElementsIndirect。
以OpenGL ES为例，关键指令可以包括但不限于以下指令：
glBindBuffer、glBufferData、glBufferSubData、glBindTexture、glTexImage2D、glTexSubImage2D、glTexImage3D、glTexSubImage3D、glCompressedTexImage2D、glCompressedTexSubImage2D、glCompressedTexImage3D、glCompressedTexSubImage3D。
步骤506：确定获取的渲染指令中是否包含监测的draw call/关键指令。
应理解，根据监测的draw call/关键指令可以确定端侧恢复渲染上下文所需的纹理资源或者顶点资源，而在步骤502中端侧可以将部分纹理资源和顶点资源缓存在端侧。因此，若端侧的渲染指令中包含draw call/关键指令，则执行步骤507；若端侧的渲染指令中不包含draw call/关键指令，则执行步骤508，游戏正常渲染模式运行，其中，游戏正常渲染模式可以是指令拦截，一份发往状态机模块，一份发往端侧与未触发状态机恢复流程前相同。
也就是说，端侧触发了状态机恢复流程后，则云侧可以运行在按需加载模式下，即对draw call和关键指令持续监测，若云侧拦截的渲染指令包含draw call和关键指令，则会触发缓存资源查找，即监测draw call/关键指令指示的资源中的部分或者全部可能存在于电子设备的本地缓存中，因此可以在电子设备的缓存列表中进行查找。反之不会，只会进入游戏正常渲染模式。
步骤507：云侧获取draw call/关键指令指示的所有资源标识。
步骤508：在端侧发送的缓存资源列表中查找资源标识。
例如，缓存资源查找可以是从按需加载模式触发时从端侧获取的其缓存的所有资源列 表(资源的sha256sum值与资源的映射)中，查找draw call和关键指令指示的资源，计算出该资源的sha256sum值，作为key去资源列表中查找，如果能在端侧缓存的资源列表中查找成功，说明端侧已经缓存了该资源，云侧直接将该资源的sha256sum值发送给端侧并告知端侧从缓存数据库中加载该资源到GPU中即可。否则，云侧可以通过网络发送该资源到端侧。
步骤509：触发端侧从其本地缓存中加载渲染信息。
步骤510：根据资源标识在获取的状态机快照中获取标识对应的资源。
步骤511：将资源通过网络发送至端侧。
在本申请的实施例中，根据上述按需加载的方式可以加速渲染上下文的恢复速度，即可以根据电子设备中缓存的渲染信息以及根据恢复当前帧图形相关的渲染信息，从而减少活肤渲染上下文所需的等待时间。
应理解，上述举例说明是为了帮助本领域技术人员理解本申请实施例，而非要将本申请实施例限于所例示的具体数值或具体场景。本领域技术人员根据所给出的上述举例说明，显然可以进行各种等价的修改或变化，这样的修改或变化也落入本申请实施例的范围内。
上文结合图1至图7，详细描述了本申请实施例提供的用于图形渲染的方法，下面将结合图8至图，详细描述本申请的装置实施例。应理解，本申请实施例中的图形渲染装置可以执行前述本申请实施例的各种方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。
图8是本申请实施例的图形渲染装置600的示意性框图。应理解，装置600能够执行图4至图7的方法中的各个步骤，为了避免重复，此处不再详述。该图形渲染装置600可以是服务器，或者，图形渲染装置600可以是配置于服务器中的芯片。
在一个示例中，图形渲染装置600包括：处理单元610和收发单元620。其中，处理单元610用于启动应用程序；获取所述应用程序发送的渲染指令；收发单元620用于向所述电子设备发送所述渲染指令，所述渲染指令用于所述电子设备执行图形渲染以显示与所述应用程序相关的图像。
需要说明的是，图8所示的图形渲染装置600可以是图1或者图2所示的服务器，其中，收发单元620可以实现图1或者图2所示的服务器通信接口的相应功能。处理单元可以对应于实现图1所示的拦截机中的相应功能，或者，处理单元610还可以对应于实现图2所示的拦截机、保存模块以及恢复模块所对应的功能；收发单元620可以对应于图1或者图2中实现TCP/代理的功能。
可选地，在一种可能的实现方式中，所述处理单元610还用于：保存图形渲染信息，所述图形渲染信息是所述电子设备用于图形渲染所需的信息。
可选地，在一种可能的实现方式中，所述收发单元620还用于接收所述电子设备发送的第二指令，所述第二指令用于请求恢复与所述服务器的连接；响应于所述第二指令，向所述电子设备发送所述图形渲染信息，所述图形渲染信息用于所述电子设备恢复渲染上下文，所述渲染上下文为接收所述第二指令时所述应用程序的渲染上下文。
可选地，在一种可能的实现方式中，所述图形渲染信息为所述渲染指令。
可选地，在一种可能的实现方式中，所述图形渲染信息为渲染信息集合，所述图形渲 染信息为对所述渲染指令进行处理得到的渲染信息集合。
可选地，在一种可能的实现方式中，所述处理单元610还用于：
根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括重传次数小于预定阈值的渲染信息。
可选地，在一种可能的实现方式中，所述处理单元610还用于：
根据渲染顺序对所述渲染信息集合中的渲染信息进行重新排序，所述渲染顺序是指所述电子设备的渲染管线运行图形渲染时渲染信息的顺序；
所述收发单元620具体用于：
向所述电子设备发送重新排序的所述渲染信息集合。
可选地，在一种可能的实现方式中，所述收发单元620还用于：
接收所述电子设备发送的缓存列表，所述缓存列表是所述电子设备缓存的渲染信息的列表；
处理单元610还用于：
根据缓存列表向所述电子设备发送部分或全部所述图形渲染信息，发送的所述图形渲染信息为在所述缓存列表中不存在的图形渲染信息。
可选地，在一种可能的实现方式中，所述处理单元610还用于：
确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合；
所述收发单元620具体用于：
向所述电子设备发送所述指令序列和所述渲染信息集合。
在一个示例中，图形渲染装置600包括：处理单元610和收发单元620。其中，收发单元620，用于接收电子设备发送的第四指令，所述第四指令用于指示所述电子设备与服务器断开连接；处理单元610，用于接收所述电子设备发送的第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；保存图形渲染信息，所述图形渲染信息用于恢复渲染上下文，所述渲染上下文为接收所述第二指令时所述应用程序的渲染上下文；根据所述图形渲染信息进行图形渲染，生成视频流；收发单元620，还用于向所述电子设备发送所述视频流。
需要说明的是，图8所示的图形渲染装置600可以是图1或者图2所示的服务器，其中，收发单元620可以实现服务器通信接口的相应功能。处理单元610可以对应于实现图1所示的拦截机中的相应功能，或者，处理单元610还可以对应于实现图2所示的拦截机、保存模块以及恢复模块所对应的功能；收发单元620可以对应于图1或者图2中实现TCP/代理的功能。
可选地，在一种可能的实现方式中，所述图形渲染信息为所述渲染指令。
可选地，在一种可能的实现方式中，所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
可选地，在一种可能的实现方式中，所述处理单元610还用于：根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括重传次数小于预定阈值的渲染信息。
可选地，在一种可能的实现方式中，所述处理单元610还用于：确定指令序列，所述 指令序列用于所述服务器的图形处理器识别所述渲染信息集合。
应理解，这里的图形渲染装置600以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图9是本申请实施例的图形渲染装置700的示意性框图。应理解，装置700能够执行图4至图7的方法中的各个步骤，为了避免重复，此处不再详述。该图形渲染装置700可以是电子设备，或者，图形渲染装置700可以是配置于电子设备中的芯片。图形渲染装置700包括：收发单元710和处理单元720。其中，收发单元710，用于接收所述服务器发送的渲染指令，所述渲染指令用于电子设备进行图形渲染；处理单元7720，用于根据所述渲染指令进行图形渲染。
可选地，在一种可能的实现方式中，收发单元710还用于向服务器发送第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；接收所述服务器发送的图形渲染信息；处理单元720还用于根据所述图形渲染信息恢复渲染上下文，所述渲染上下文为发送所述第二指令时所述应用程序的渲染上下文。
可选地，在一种可能的实现方式中，所述图形渲染信息为所述渲染指令。
可选地，在一种可能的实现方式中，所述图形渲染信息所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
可选地，在一种可能的实现方式中，所述处理单元720，还用于确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合。
可选地，在一种可能的实现方式中，所述渲染信息集合中包括第一标识信息和第一渲染信息，所述第一标识信息为所述服务器用于标识所述第一渲染信息的标识信息，所述处理单元720，还用于确定映射关系，所述映射关系是所述第一标识信息和所述第二标识信息的对应关系，所述第二标识信息为所述电子设备用于标识所述第一渲染信息的标识信息。
应理解，这里的图形渲染装置700以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的 特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图10示出了本申请一个实施例提供的服务器800的示意性框图。该服务器800包括处理器820、存储器860、通信接口840和总线850。其中，处理器820、存储器860、通信接口840通过总线850进行通信，也可以通过无线传输等其他手段实现通信。该存储器860用于存储指令，该处理器820用于执行该存储器860存储的指令。该存储器860存储程序代码811，且处理器820可以调用存储器860中存储的程序代码811执行图4至图7所示的用于图形渲染的方法。
例如，处理器820可以用于执行上述图4中的220中获取应用程序发送的渲染指令；或者，可以执行图6所示的确定状态机快照的流程。
该存储器860可以包括只读存储器和随机存取存储器，并向处理器820提供指令和数据。存储器860还可以包括非易失性随机存取存储器。该存储器860可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data date SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
该总线850除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图8中将各种总线都标为总线850。
如图10所示的服务器800，其中，处理器820可以执行图8所示的处理单元610对应的步骤/功能，通信接口840可以执行图8所示的收发单元620对应的步骤/功能。
应理解，图10所示的服务器800能够实现图4至图7所示方法实施例中设备执行的各个过程。服务器800中的各个模块的操作和/或功能，分别为了实现上述方法实施例中设备的相应流程。具体可参见上述方法实施例中的描述，为避免重复，此处适当省略详细描述。
图11是本申请实施例的电子设备900的结构示意图。该电子设备900可以但不限于是膝上型计算机、台式计算机、移动电话、智能手机、平板电脑、多媒体播放器、电子阅读器、智能车载设备、智能家电、人工智能设备、穿戴式设备、物联网设备、或虚拟现实/增强现实/混合现实设备等。
应理解，上文中图9所示的装置700的具体结构可以如图11所示。
图11中的电子设备900包括通信模块910、传感器920、用户输入模块930、输出模块940、处理器950、存储器970以及电源980。其中，处理器950可以包括一个或者多个CPU。
图11所示的电子设备可以执行本申请实施例的用于图形渲染方法的各个步骤，具体地，处理器950中的一个或者多个CPU可以执行本申请实施例的同于图形渲染方法的各个步骤。
下面对图11中的电子设备900的各个模块进行详细的介绍。应理解的是，图11所示的模块仅是举例说明，具体实现中电子设备900可以包含更多或更少的模块。
通信模块910可以包括至少一个能使该电子设备与其他电子设备之间进行通信的模块。例如，通信模块910可以包括有线网络接口、广播接收模块、移动通信模块、无线因特网模块、局域通信模块和位置(或定位)信息模块等其中的一个或多个。本实施例不限定通信代别，例如可以是2G、3G、4G、5G或随着技术发展出现的其它通信代别。
例如，通信模块910能够从游戏服务器端实时获取用于渲染游戏画面所需的渲染指令。
传感器920可以感知用户的一些操作，传感器920可以包括距离传感器，触摸传感器等等。传感器920可以感知用户触摸屏幕或者靠近屏幕等操作。
例如，传感器920能够感知用户在游戏界面的一些操作。
用户输入模块930，用于接收输入的数字信息、字符信息或接触式触摸操作/非接触式手势，以及接收与系统的用户设置以及功能控制有关的信号输入等。用户输入模块930包括触控面板和/或其他输入设备。例如，用户可以通过用户输入模块3030对游戏进行控制。
输出模块940包括显示面板，用于显示由用户输入的信息、提供给用户的信息或系统的各种菜单界面等。
可选的，可以采用液晶显示器(liquid crystal display，LCD)或有机发光二极管(organic light-emitting diode,OLED)等形式来配置显示面板。在其他一些实施例中，触控面板可覆盖显示面板上，形成触摸显示屏。
另外，输出模块940还可以包括视频输出模块、告警器以及触觉模块等。该视频输出模块可以显示图形渲染后的游戏画面。
电源980可以在处理器950的控制下接收外部电力和内部电力，并且提供整个电子设备各个模块运行时需要的电力。
处理器950可以包括一个或者多个CPU，处理器950还可以包括一个或者多个GPU。
当处理器950包括多个CPU时，该多个CPU可以集成在同一块芯片上，也可以分别集成在不同的芯片上。
当处理器950包括多个GPU时，该多个GPU既可以集成在同一块芯片上，也可以分别集成在不同的芯片上。
当处理器950既包括CPU又包括GPU时，CPU和GPU可以集成在同一块芯片上。
例如，当图11所示的电子设备为智能手机时，智能手机的处理器内部一般与图像处理相关的是一个CPU和一个GPU。这里的CPU和GPU均可以包含多个核。
存储器970可以存储计算机程序，该计算机程序包括操作系统程序972和应用程序971等。其中，典型的操作系统如微软公司的Windows，苹果公司的MacOS等用于台式机或笔记本的系统，又如谷歌公司开发的基于
Figure PCTCN2020083772-appb-000003
的安卓
Figure PCTCN2020083772-appb-000004
系统等用于移动终端的系统。
存储器970可以是以下类型中的一种或多种：闪速(flash)存储器、硬盘类型存储器、 微型多媒体卡型存储器、卡式存储器(例如SD或XD存储器)、随机存取存储器(random access memory,RAM)、静态随机存取存储器(static RAM,SRAM)、只读存储器(read only memory,ROM)、电可擦除可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)、可编程只读存储器(programmable ROM,PROM)、磁存储器、磁盘或光盘。在其他一些实施例中，存储器970也可以是因特网上的网络存储设备，系统可以对在因特网上的存储器970执行更新或读取等操作。
例如，上述存储器970可以存储一种计算机程序(该计算机程序是本申请实施例的用于图形渲染方法对应的程序)，当处理器950执行该计算机程序时，处理器950能够执行本申请实施例的用于图形渲染的方法。
存储器970还存储有除计算机程序之外的其他数据973，例如，存储器970可以存储本申请的图形渲染方法处理过程中的数据。
图11中各个模块的连接关系仅为一种示例，本申请任意实施例提供的电子设备也可以应用在其它连接方式的电子设备中，例如所有模块通过总线连接。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(Solid State Disk,SSD))等。
应理解，在本申请实施例中，术语“第一”、“第二”、“第三”等仅仅是为了指代对象，并不表示相应对象间的先后次序。
应理解，在本申请实施例中，术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系。例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本 申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围准。

Claims (23)

  1. 一种用于图形渲染的方法，其特征在于，包括：
    启动应用程序；
    获取所述应用程序发送的渲染指令；
    向所述电子设备发送所述渲染指令，所述渲染指令用于所述电子设备执行图形渲染以显示与所述应用程序相关的图像。
  2. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    保存图形渲染信息，所述图形渲染信息是所述电子设备用于图形渲染所需的信息。
  3. 如权利要求2所述的方法，其特征在于，所述方法还包括：
    接收所述电子设备发送的第二指令，所述第二指令用于请求恢复与所述服务器的连接；
    响应于所述第二指令，向所述电子设备发送所述图形渲染信息，所述图形渲染信息用于所述电子设备恢复渲染上下文，所述渲染上下文为接收所述第二指令时所述应用程序的渲染上下文。
  4. 如权利要求2或3所述的方法，其特征在于，所述图形渲染信息为所述渲染指令。
  5. 如权利要求2或3所述的方法，其特征在于，所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
  6. 如权利要求5所述的方法，其特征在于，所述方法还包括：
    根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括重传次数小于预定阈值的渲染信息。
  7. 如权利要求5或6所述的方法，其特征在于，所述方法还包括：
    根据渲染顺序对所述渲染信息集合中的渲染信息进行重新排序，所述渲染顺序是指所述电子设备的渲染管线运行图形渲染时渲染信息的顺序；
    向所述电子设备发所述图形渲染信息，包括：
    向所述电子设备发送重新排序的所述渲染信息集合。
  8. 如权利要求3至7中任一项所述的方法，其特征在于，所述方法还包括：
    接收所述电子设备发送的缓存列表，所述缓存列表为所述电子设备缓存的渲染信息的列表；
    向所述电子设备发送所述图形渲染信息，包括：
    根据所述缓存列表向所述电子设备发送部分或全部所述图形渲染信息，发送的所述图形渲染信息为在所述缓存列表中不存在的图形渲染信息。
  9. 根据权利要求5至7中任一项所述的方法，其特征在于，向所述电子设备发送所述图形渲染信息，包括：
    确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合；
    向所述电子设备发送所述指令序列和所述渲染信息集合。
  10. 一种用于图形渲染的方法，其特征在于，包括：
    接收服务器发送的渲染指令，所述渲染指令用于电子设备进行图形渲染；
    根据所述渲染指令进行图形渲染显示与所述应用程序相关的图像。
  11. 如权利要求10所述的方法，其特征在于，所述方法还包括：
    向服务器发送第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；
    接收所述服务器发送的图形渲染信息；
    根据所述图形渲染信息恢复渲染上下文，所述渲染上下文为发送所述第二指令时所述应用程序的渲染上下文。
  12. 如权利要求10或11所述的方法，其特征在于，所述图形渲染信息为所述渲染指令。
  13. 如权利要求10或11所述的方法，其特征在于，所述图形渲染信息所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
  14. 如权利要求13所述的方法，其特征在于，所述方法还包括：
    确定指令序列，所述指令序列用于所述电子设备的图形处理器识别所述渲染信息集合。
  15. 如权利要求13或14所述的方法，其特征在于，所述渲染信息集合中包括第一标识信息和第一渲染信息，所述第一标识信息为所述服务器用于标识所述第一渲染信息的标识信息，所述方法还包括：
    确定映射关系，所述映射关系是所述第一标识信息和所述第二标识信息的对应关系，所述第二标识信息为所述电子设备用于标识所述第一渲染信息的标识信息。
  16. 一种用于图形渲染的方法，其特征在于，包括：
    接收电子设备发送的第四指令，所述第四指令用于指示所述电子设备与服务器断开连接；
    获取所述应用程序发送的渲染指令，所述渲染指令用于所述服务器进行图形渲染；
    接收所述电子设备发送的第二指令，所述第二指令用于请求恢复所述电子设备与所述服务器的连接；
    保存图形渲染信息，所述图形渲染信息用于恢复渲染上下文，所述渲染上下文为接收所述第二指令时所述应用程序的渲染上下文；
    根据所述图形渲染信息进行图形渲染，生成视频流；
    向所述电子设备发送所述视频流。
  17. 如权利要求16所述的方法，其特征在于，所述图形渲染信息为所述渲染指令。
  18. 如权利要求16所述的方法，其特征在于，所述图形渲染信息为对所述渲染指令进行处理得到的渲染信息集合。
  19. 如权利要求18所述的方法，其特征在于，所述方法还包括：
    根据预配置的信息列表和所述渲染指令确定所述渲染信息集合，所述预配置的信息列表包括重传次数小于预定阈值的渲染信息。
  20. 根据权利要求18或19所述的方法，其特征在于，所述方法还包括：
    确定指令序列，所述指令序列用于所述服务器的图形处理器识别所述渲染信息集合。
  21. 一种图形渲染装置，其特征在于，包括存储器和处理器，所述存储器用于存储计 算机程序，所述处理器用于从所述存储器中调用并运行所述计算机程序，以执行如权利要求1至9中任一项所述的方法。
  22. 一种图形渲染装置，其特征在于，包括存储器和处理器，所述存储器用于存储计算机程序，所述处理器用于从所述存储器中调用并运行所述计算机程序，以执行如权利要求10至15中任一项所述的方法。
  23. 一种图形渲染装置，其特征在于，包括存储器和处理器，所述存储器用于存储计算机程序，所述处理器用于从所述存储器中调用并运行所述计算机程序，以执行如权利要求16至20中任一项所述的方法。
PCT/CN2020/083772 2019-04-30 2020-04-08 用于图形渲染的方法及装置 WO2020220953A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20799150.6A EP3951595A4 (en) 2019-04-30 2020-04-08 METHOD AND DEVICE FOR GRAPHIC RENDERING
US17/516,479 US20220058766A1 (en) 2019-04-30 2021-11-01 Graphics rendering method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910364218.8A CN111861854A (zh) 2019-04-30 2019-04-30 用于图形渲染的方法及装置
CN201910364218.8 2019-04-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/516,479 Continuation US20220058766A1 (en) 2019-04-30 2021-11-01 Graphics rendering method and apparatus

Publications (1)

Publication Number Publication Date
WO2020220953A1 true WO2020220953A1 (zh) 2020-11-05

Family

ID=72966690

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/083772 WO2020220953A1 (zh) 2019-04-30 2020-04-08 用于图形渲染的方法及装置

Country Status (4)

Country Link
US (1) US20220058766A1 (zh)
EP (1) EP3951595A4 (zh)
CN (1) CN111861854A (zh)
WO (1) WO2020220953A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112839184A (zh) * 2020-12-31 2021-05-25 深圳追一科技有限公司 图像处理方法、装置、电子设备及存储介质
CN113140029A (zh) * 2021-05-07 2021-07-20 贺之娜 一种基于5g的三维实时云渲染模拟系统
WO2023104186A1 (zh) * 2021-12-09 2023-06-15 许磊 一种高效低成本的云游戏系统

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112843680A (zh) * 2021-03-04 2021-05-28 腾讯科技（深圳）有限公司 画面显示方法、装置、终端设备及存储介质
CN113244614B (zh) * 2021-06-07 2021-10-26 腾讯科技（深圳）有限公司 图像画面展示方法、装置、设备及存储介质
CN113453073B (zh) * 2021-06-29 2023-01-06 北京百度网讯科技有限公司 一种图像渲染方法、装置、电子设备及存储介质
CN115883614A (zh) * 2021-09-26 2023-03-31 华为技术有限公司 一种云手机的实现方法及装置
CN114092590B (zh) * 2022-01-19 2023-04-07 荣耀终端有限公司 电子设备及其图像渲染性能的评估方法、介质
CN114827186A (zh) * 2022-02-25 2022-07-29 阿里巴巴（中国）有限公司 云应用处理方法和系统
CN114820272A (zh) * 2022-03-18 2022-07-29 维塔科技(北京)有限公司 数据交互的方法、装置、存储介质及电子设备
CN116051704A (zh) * 2022-08-29 2023-05-02 荣耀终端有限公司 渲染方法及装置
CN117472371A (zh) * 2023-10-09 2024-01-30 北京趋动智能科技有限公司 远程渲染方法、装置及存储介质
CN117710183A (zh) * 2023-12-14 2024-03-15 摩尔线程智能科技(北京)有限责任公司 渲染指令的传输方法、操作系统、电子设备、存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013028202A1 (en) * 2011-08-25 2013-02-28 Intel Corporation Collaborative graphics rendering using mobile devices to support remote display
CN105850097A (zh) * 2013-12-23 2016-08-10 高通股份有限公司 用于无线扩展坞的无线带宽的高效使用的远程渲染
US20170091160A1 (en) * 2015-09-30 2017-03-30 Samsung Display Co. Ltd. Display system and virtual web device in the cloud
CN107977272A (zh) * 2016-10-25 2018-05-01 腾讯科技（深圳）有限公司 应用运行的方法及装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2813679B1 (fr) * 2000-09-01 2006-09-29 Airsys Atm S A Systeme informatique multiprocess
US8106909B2 (en) * 2007-10-13 2012-01-31 Microsoft Corporation Common key frame caching for a remote user interface
US8572251B2 (en) * 2008-11-26 2013-10-29 Microsoft Corporation Hardware acceleration for remote desktop protocol
CN103023872B (zh) * 2012-11-16 2016-01-06 杭州顺网科技股份有限公司 一种云游戏服务平台
CN104796393A (zh) * 2014-05-30 2015-07-22 厦门极致互动网络技术有限公司 一种基于服务器实时渲染的网络游戏系统及方法
CN105727556B (zh) * 2016-01-29 2019-05-31 腾讯科技（深圳）有限公司 一种图像绘制的方法、相关设备及系统
CN107483382B (zh) * 2016-06-08 2020-08-25 成都云创互联信息技术有限公司 一种云渲染方法
US10979447B2 (en) * 2018-08-30 2021-04-13 Ordr Inc. Presenting, at a graphical user interface, device photos and risk categories associated with devices in a network
US11089081B1 (en) * 2018-09-26 2021-08-10 Amazon Technologies, Inc. Inter-process rendering pipeline for shared process remote web content rendering
CN109358936B (zh) * 2018-09-29 2021-09-10 Oppo广东移动通信有限公司 信息处理方法、装置、存储介质、电子设备及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013028202A1 (en) * 2011-08-25 2013-02-28 Intel Corporation Collaborative graphics rendering using mobile devices to support remote display
CN105850097A (zh) * 2013-12-23 2016-08-10 高通股份有限公司 用于无线扩展坞的无线带宽的高效使用的远程渲染
US20170091160A1 (en) * 2015-09-30 2017-03-30 Samsung Display Co. Ltd. Display system and virtual web device in the cloud
CN107977272A (zh) * 2016-10-25 2018-05-01 腾讯科技（深圳）有限公司 应用运行的方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3951595A4

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112839184A (zh) * 2020-12-31 2021-05-25 深圳追一科技有限公司 图像处理方法、装置、电子设备及存储介质
CN112839184B (zh) * 2020-12-31 2022-02-01 深圳追一科技有限公司 图像处理方法、装置、电子设备及存储介质
CN113140029A (zh) * 2021-05-07 2021-07-20 贺之娜 一种基于5g的三维实时云渲染模拟系统
CN113140029B (zh) * 2021-05-07 2023-05-09 北京千种幻影科技有限公司 一种基于5g的三维实时云渲染模拟系统
WO2023104186A1 (zh) * 2021-12-09 2023-06-15 许磊 一种高效低成本的云游戏系统

Also Published As

Publication number Publication date
EP3951595A1 (en) 2022-02-09
US20220058766A1 (en) 2022-02-24
EP3951595A4 (en) 2022-05-25
CN111861854A (zh) 2020-10-30

Similar Documents

Publication Publication Date Title
WO2020220953A1 (zh) 用于图形渲染的方法及装置
US20240095043A1 (en) Execution of sub-application processes within application program
CN111433743B (zh) 一种app远程控制方法及相关设备
WO2021169236A1 (zh) 渲染的方法、装置
US9946894B2 (en) Data processing method and data processing device
US20130073600A1 (en) Remote process execution management
US20230215076A1 (en) Image frame display method, apparatus, device, storage medium, and program product
US9344528B2 (en) Data compressor using hints
KR20070110127A (ko) 압축을 이용하여 분산 프로세싱 환경에서 그래픽디스플레이를 업데이트하는 방법 및 장치
CN109309842B (zh) 直播数据处理方法和装置、计算机设备和存储介质
US20130073601A1 (en) Remote process execution management
CN113286005B (zh) 抓屏方法、装置、电子设备及存储介质
JP2012195629A (ja) 情報処理装置、画像送信プログラム、画像送信方法および画像表示方法
CN111580988B (zh) 开放平台的实现方法、装置、存储介质和计算机设备
US20120113103A1 (en) Apparatus and method for executing 3d application program using remote rendering
US20190325043A1 (en) Method, device and computer program product for replicating data block
CN114820272A (zh) 数据交互的方法、装置、存储介质及电子设备
CN114581580A (zh) 渲染图像的方法、装置、存储介质及电子设备
WO2023216621A1 (zh) 云桌面的图像处理方法、装置、服务器和存储介质
WO2023035619A1 (zh) 一种场景渲染方法、装置、设备及系统
RU2635255C2 (ru) Системный когерентный кэш с возможностью фрагментации/дефрагментации
CN113411661B (zh) 用于记录信息的方法、装置、设备、存储介质及程序产品
JP5536093B2 (ja) 改良されたコマンドリモーティング技術
CN113835816A (zh) 一种虚拟机桌面显示方法、装置、设备及可读存储介质
US9972064B1 (en) Non-intrusive and low-power recording

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20799150

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020799150

Country of ref document: EP

Effective date: 20211105