CN112169322B - Remote rendering method and device, electronic equipment and readable storage medium - Google Patents
Remote rendering method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112169322B CN112169322B CN201910599248.7A CN201910599248A CN112169322B CN 112169322 B CN112169322 B CN 112169322B CN 201910599248 A CN201910599248 A CN 201910599248A CN 112169322 B CN112169322 B CN 112169322B
- Authority
- CN
- China
- Prior art keywords
- game
- sequence
- instruction
- game server
- intercepted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application provides a remote rendering method and device, electronic equipment and a readable storage medium. The electronic equipment intercepts a graphics API instruction sequence initiated by an installed game client based on a control instruction input by a user and sends the intercepted graphics API instruction sequence to the game server according to the interception sequence, so that a GPU of the game server executes the graphics API instruction sequence to obtain a rendered game picture. Therefore, the decoupling of the CPU processing logic and the GPU processing logic of the game client can be realized, the CPU processing logic is independent from the game server, and the program development cost in the game server is reduced.
Description
Technical Field
The application relates to the technical field of game picture rendering, in particular to a remote rendering method, a remote rendering device, electronic equipment and a readable storage medium.
Background
In a conventional client-side game architecture, a whole game client runs on a user-side device, in order to render a game picture smoothly, the user-side device must have a Graphics card supporting hardware acceleration, and a core of the Graphics card is a GPU (Graphics processing unit), which limits wide application of the game client.
In order to relieve the computing power requirement of a game client on a GPU, some common methods deploy a complete game client and an agent program in a cloud server, where the agent program is used to receive a control instruction of a user-side device and forward the control instruction to the game client for processing, and after a corresponding game picture is obtained by rendering, encode the game picture into a video code stream and return the video code stream to the user-side device for decoding and playing. However, the hardware cost and development cost of this approach are very high.
Disclosure of Invention
In view of the above, an object of the present application includes providing a remote rendering method, apparatus, electronic device and readable storage medium, which can reduce hardware cost and development cost of a game server.
To achieve the above object, embodiments of the present application may be implemented as follows:
in a first aspect, an embodiment of the present application provides a remote rendering method, which is applied to an electronic device installed with a game client, where the electronic device is in communication connection with a game server, and the method includes:
intercepting a graphic API instruction sequence initiated by the game client based on a control instruction input by a user, wherein the graphic API instruction sequence comprises a calling instruction, a calling parameter and a calling sequence of the game client to a graphic API;
and sending the intercepted graphic API instruction sequence to the game server according to the interception sequence, so that the GPU of the game server executes the graphic API instruction sequence to obtain a rendered game picture.
In a second aspect, an embodiment of the present application provides a remote rendering apparatus, which is applied to an electronic device installed with a game client, where the electronic device is in communication connection with a game server, and the apparatus includes:
the intercepting module is used for intercepting a graphic API instruction sequence initiated by the game client based on a control instruction input by a user, wherein the graphic API instruction sequence comprises a calling instruction, a calling parameter and a calling sequence of the game client to a graphic API;
and the rendering module is used for sending the intercepted graphic API instruction sequence to the game server according to the interception sequence, so that the GPU of the game server executes the graphic API instruction sequence to obtain a rendered game picture.
In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and machine executable instructions stored in the memory and executed in the processor, where the machine executable instructions, when executed by the processor, implement the remote rendering method in any one of the foregoing embodiments.
In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon machine executable instructions that, when executed, implement the remote rendering method of any one of the preceding embodiments.
According to the remote rendering method, the device, the electronic equipment and the readable storage medium, the game client side intercepts the graphic API instruction sequence initiated based on the control instruction input by the user, and sends the intercepted graphic API instruction to the game server according to the interception sequence, so that the GPU of the game server executes the graphic API instruction sequence to obtain a rendered game picture, logic which can be processed by the CPU is separated from the game server and is put to the electronic equipment at the user side to run, the hardware cost of the game server is reduced, subsequent program upgrading, compatibility, environment isolation and the like are easier, and the development cost is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic diagram of an architecture of a cloud game system according to the present embodiment;
fig. 2 is a schematic diagram of a communication relationship framework between an electronic device and an internal module of a game server according to this embodiment;
fig. 3 is a schematic structural diagram of an electronic device provided in this embodiment;
fig. 4 is a schematic flowchart of a remote rendering method according to this embodiment;
fig. 5 is a schematic flowchart of a remote rendering method according to the present embodiment;
fig. 6 is a schematic flowchart of a remote rendering method according to the present embodiment;
fig. 7 is a functional block diagram of a remote rendering apparatus provided in this embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Research shows that the processing flow of the game client during running is as follows: processing a control instruction input by a user according to a game logic, determining a game picture triggered by the control instruction, then determining a graphics API (application programming interface) required for rendering the game picture, and calling the determined graphics API to communicate with a driver of bottom hardware (such as a GPU) of a device where the graphics API is located, so as to start a corresponding function of the GPU to render the game picture. In the above processing flow, processing the control instruction to determine that the graphics API to be called is implemented by a Central Processing Unit (CPU), and calling the graphics API to implement the corresponding rendering depends on a GPU.
Based on the above findings, in the embodiment of the present application, the processing flow of the game client is skillfully decoupled into the CPU processing flow and the GPU processing flow, the CPU processing flow is deployed on the electronic device on the user side, the GPU processing flow is deployed on the server, and the GPU required for rendering is deployed on the server. Therefore, compared with the existing mode of deploying a complete game on the server, the processing operation required to be executed by the server can be reduced under the condition of not changing the hardware configuration of the server, and the performance requirement on the server is greatly reduced. Due to the fact that the complexity of the decoupled program is reduced, subsequent program upgrading and maintenance are simpler. In addition, the calculation force required for processing the control command according to the game logic is small, and the control command can be satisfied by only general electronic equipment (such as a personal computer, an intelligent terminal and the like), so that the equipment cost of the user side is hardly increased by the scheme.
Based on this, embodiments of the present application provide a remote rendering method, apparatus, electronic device and readable storage medium, which will be described below.
It should be noted that the above research and discovery on the existing game client and the proposed solution based on the discovery are the results of the practical and careful research by the inventor, and should be the contribution of the inventor to the present application in the course of the invention.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic diagram illustrating an architecture of a cloud game system according to an embodiment of the present disclosure. The cloud game system includes a plurality of electronic devices 100 and a game server 200 communicatively connected to the electronic devices 100, and fig. 2 is a schematic view of a communication relationship between one of the electronic devices 100 and an internal module of the game server 200.
The electronic device 100 may be any device having a data processing function and a display function and communicatively connected to the game server 200, for example, a notebook computer, a tablet computer, a television, an intelligent terminal, and the like, have simple data processing and video decoding and playing capabilities, but have weak rendering capability and are difficult to directly run a large game.
The electronic device 100 includes a game client 110, a proxy program 120, and a first graphics API (application programming interface) library 130. Among them, the game client 110 may be a 3D game application developed based on a 3D (3-dimensional) engine, which may be, for example, UE4(unreal engine 4), Unity, or the like. The agent 120 is provided in the game client 110, in other words, the game client 110 is a host program of the agent 120.
The game server 200 includes a second graphics API library 230, a GPU240, a hardware driver 250, and a stub program that communicates with the agent program of each game client. Such as the stub program 220 in communication with the agent program 120 shown in fig. 1. Wherein, the GPU240 may support hardware acceleration, and particularly, when the game client 110 is a 3D game application, the GPU240 may support 3D hardware acceleration.
In this embodiment, the first graphic API library 130 and the second graphic API library 230 are the same graphic API library, such as OpenGL, DirectX, Vulkan, etc., and include APIs for rendering two-dimensional and three-dimensional vector graphics, and the APIs are called to communicate with a driver of the underlying hardware (e.g., GPU), so as to start the graphics processing function of the underlying hardware.
Referring to fig. 3, fig. 3 is a schematic diagram of an exemplary hardware structure of the electronic device 100 provided in this embodiment, where the electronic device 100 includes a memory 101 and a processor 102, and the memory 101 and the processor 102 are communicatively connected via a system bus 103 to implement data transmission. The memory 101 stores machine executable instructions that the processor 102 may implement the remote rendering methods described below by reading and executing. Among other things, the machine executable instructions in memory 101 may include instructions for implementing game client 110, agent 120, and first graphics API library 130 described above.
It should be noted that the structure shown in fig. 3 is merely an illustration, and the electronic device 100 may include more or less components than those shown in fig. 3, or have a completely different configuration than that shown in fig. 3.
Referring to fig. 4, fig. 4 is a flowchart illustrating a remote rendering method according to the present embodiment, where the remote rendering method can be applied to the electronic device 100 shown in fig. 2. The steps included in the remote rendering method are described below.
At step 41, a sequence of graphics API commands initiated by the game client 110 based on the control commands entered by the user is intercepted.
In this embodiment, a user may input a control command to game client 110 in electronic device 100 through a keyboard, a mouse, a joystick, a voice input device, or the like, and game client 110 may process the control command according to a preset game logic, and may determine a game screen corresponding to the control command.
For example, when the user inputs a control instruction c1 for controlling the hand-up of the game character image a, the game picture corresponding to the control instruction c1 is the picture of the hand-up of the game character image a; for another example, when the user inputs a control command c2 for controlling game character avatar a to use game skill t1, the game screen corresponding to control command c2 is a screen for game character avatar a to use game skill t1 and produce a corresponding special effect.
After determining the game screen corresponding to the control instruction input by the user, the game client 110 may further determine a graphics API to be called for rendering the game screen and a call parameter of the determined graphics API, and may generate a call instruction for calling the graphics API for each determined graphics API. The call instruction and the corresponding call parameters are usually sent to a hardware driver of the GPU of the device where the call instruction is located, so that the call instruction is converted into a corresponding GPU instruction for the GPU to execute, thereby rendering a game picture. Each frame of game image usually needs to call a plurality of graphics APIs in a certain order to be rendered, so that the game client 110 generates and initiates a graphics API instruction sequence, which usually includes a plurality of graphics API call instructions, call parameters, and a plurality of graphics API call orders.
In this embodiment, the electronic device 100 generally does not have a GPU supporting 3D hardware acceleration, i.e., cannot execute the graphics API instruction sequence described above. Therefore, in the present embodiment, all the graphics API instruction sequences initiated by the host program (i.e., the game client 110) are intercepted by the agent program 120, and specifically, the interception may be performed through a hook (hook) interface, so that all the graphics API instruction sequences initiated by the game client 110 will not be sent to the hardware driver of the electronic device 100, but will be processed according to the processing flow defined in the hook interface.
In detail, considering that the game client 110 generally sequentially initiates the graphics API instruction sequences according to the actual execution order, the interception order of the graphics API instruction sequences intercepted by the hook interface is the actual execution order. Then, the processing flow shown in step 42 may be defined in the hook interface, so that each graphics API instruction sequence intercepted is sequentially sent to the game server 200 for execution according to the order of interception.
Considering that the game client 110 makes a graphics API call very frequently, accordingly, the agent program 120 needs to send the intercepted graphics API instruction sequence very frequently through the network, that is, needs to perform a large number of network IO (input output) operations, while the game client 110 belongs to a compute intensive program, that is, the call thread occupies a high load in the processor 102 of the electronic device 100, and in order to avoid affecting other threads of the electronic device 100, this embodiment may implement sending the intercepted graphics API instruction sequence through an independent network IO (input output) thread.
In detail, agent 120 may encapsulate each sequence of intercepted graphics API instructions with tasks (tasks) and add the encapsulated tasks to a work queue in the order of the interception, e.g., assuming that sequences of graphics API instructions s1, s2, and s3 are intercepted in turn, s1 may be encapsulated as task1, s2 may be encapsulated as task2, s3 may be encapsulated as task3, and then task1, task2, and task3 may be added to the work queue in turn. In this manner, the order of the tasks in the work queue is consistent with the actual required execution order of the graphics API instruction sequences in the various tasks.
On this basis, the tasks in the work queue are sequentially executed through a network IO thread which is independent from other threads in the electronic device 100, and the graphics API instruction sequence in each task is sent to the game server 200, so that the sequence in which the game server 200 receives and executes the graphics API instruction sequence is consistent with the actually required execution sequence, and the influence on other threads in the electronic device 100 can be avoided.
In order to avoid frequent network IO operations, the embodiment may merge and send the intercepted graphics API instruction sequence. In detail, when the independent network IO thread executes each task, the graphics API instruction sequence in the task may be processed into a to-be-sent data packet and cached, and after caching for a certain interval duration, the to-be-sent data packet cached in the interval duration is sent to the game server 200 together.
In practical applications, a game client usually refreshes its command queue occasionally (for example, a refresh operation is implemented by calling a gflux instruction of OpenGL), so as to send all currently initiated and cached graphics API instruction sequences to a hardware driver of a GPU of a device where the game client is located. Based on this, the present embodiment may use the refresh operation on the command queue of the game client 110 as a trigger signal for defining the interval duration, for example, the electronic device 100 may send the currently buffered data packet to be sent to the game server 200 through the independent network IO thread when detecting that the command queue of the game client 110 is refreshed.
Some graphics APIs that the game client would normally call during rendering require call processing results to previously called graphics APIs, and the call instructions for these graphics APIs are usually synchronous call instructions. Therefore, the present embodiment may use the synchronous call command initiated by the game client 110 as the trigger signal for defining the interval duration. For example, in this embodiment, when the intercepted graphics API instruction sequence includes a synchronous call instruction, all currently cached data packets to be sent are sent to the game server 200, so that the game server 200 can obtain and execute other call instructions before the synchronous call instruction before executing the synchronous call instruction.
Furthermore, the electronic device 100 may send the currently cached data packets to be sent to the game server 200 when any one of the two situations occurs, for example, when a graphics API instruction sequence containing a synchronous call instruction is intercepted, or when it is detected that the command queue of the game client 110 is refreshed.
In practical applications, since the overhead of the synchronous call instruction is too large, most of the call instructions of the graphics API are asynchronous call instructions, and there are only a few graphics APIs that need to use the synchronous call instruction, for example, an API for generating a resource Identifier (ID).
After sending the synchronous call instruction and the call parameter thereof to the corresponding hardware driver, the game client 110 waits for the response parameter to return and then sends the subsequent call instruction, but sometimes there is a delay in returning the response parameter of the synchronous call instruction due to the problems of network slowness or failure and the like. Through research, network problems usually cause at least one network Round-trip delay (RTT) in a response parameter of a synchronization call command returned by a game server, which seriously affects FPS (Frames Per Second) of a rendered game screen, thereby affecting the visual effect of the game.
Based on this, the remote rendering method provided in this embodiment may further include the steps shown in fig. 5.
Wherein the pseudo response parameter may be referred to as a stub. The steps of generating the generated pseudo response parameter to the game client 110 and sending the pseudo response parameter to the game server 200 are not limited in order of execution.
In this embodiment, when the graphics API instruction sequence including the synchronization call instruction is intercepted, the agent program 120 immediately returns the designation information to the game client 110 as the response parameter for the synchronization call instruction if the synchronization call instruction is not executed. The specified information returned by the agent program 120 is the pseudo response information, which may be a preset parameter or a random parameter generated according to a preconfigured instruction, which is not limited in this embodiment.
When the synchronous call instruction is sent to the game server 200, and the response parameter generated after the synchronous call instruction is executed by the game server 200 is the actual response parameter.
Taking the resource ID generation API described above as an example, after sending a graphics API instruction sequence including the call instruction in1 of the resource ID generation API to the game server 200, the agent program 120 may immediately generate the first resource ID to return to the game client 110, and send the first resource ID to the game server 200. The graphic API instruction sequence including the call instruction in1 of the resource ID generation API is transmitted to the game server 200 according to the flow shown in fig. 4, and when the call instruction in1 is executed, a second resource ID is generated. The game server 200 establishes a correspondence relationship between the first resource ID and the second resource ID. In the subsequent process, when the game server 200 receives a call instruction that the first resource ID needs to be used, the first resource ID is found to be used according to the established corresponding relationship.
Wherein the first resource ID may serve as the pseudo response parameter and the second resource ID may serve as the actual response parameter.
Therefore, the waiting time required by the synchronous calling instruction can be reduced, the time required by rendering is reduced, the FPS of the game picture is improved, and a better game display effect is achieved.
Alternatively, the game client 110 in this embodiment has a larger traffic to the game server 200, which usually takes a longer time. In order to reduce the traffic transmitted through the network, the electronic device 100 may compress and encode the data packet to be transmitted, and then transmit the data packet to the game server 200 through the independent network IO thread. The compression encoding method may be, but is not limited to, intra-frame compression, inter-frame compression, and the like.
In this embodiment, in order to further reduce the traffic size transmitted through the network, some calling parameters with small changes or static states, such as texture parameters, texture parameter shaders (shaders), etc., may be cached in the game server 200 in advance. Of course, these parameters may also be sent to the game server 200 by the game client 110 in real time through the network, which is not limited in this embodiment.
In this embodiment, the game client 110 may include a plurality of game threads, and the above-mentioned graphics API instruction sequence generally carries a call sequence of different call instructions initiated by the same game thread, so the game server 200 may ensure an execution sequence of each call instruction initiated by the same game thread according to the call sequence. However, when there is a restriction on the execution order among different game threads, since the intercepted graphics API instruction sequence does not include timing information of different game threads, the game server 200 cannot ensure that the execution order of the call instructions of different game threads is correct.
In order to solve the above problem, the remote rendering method provided in this embodiment may further include the steps shown in fig. 6, which are described in detail as follows.
Wherein, for two game threads Thread1 and Thread2 with execution order restriction, assuming that the call instruction in Thread1 needs to be executed before Thread2, the game client 110 will first run Thread1, Thread1 will initiate the corresponding graphics API instruction sequence { X, Y }, then switch to Thread2, and Thread2 will initiate the corresponding graphics API instruction sequence { Z, M }.
Based on this, when the electronic device 100 detects that the game Thread of the game client 110 is switched from Thread1 to Thread2, a synchronization instruction for creating a synchronization object is immediately generated and packaged as one task (i.e., the synchronization task) to be added to the work queue. The generated synchronization task is to be added to the work queue between a task containing the graphics API instruction sequence { X, Y } and a task containing the graphics API instruction sequence { Z, M }.
In this way, the GPU of the game server 200 immediately executes the synchronization instruction after starting to execute the call instruction X and the call instruction Y, and creates the synchronization object.
The synchronization instruction may be, for example, an eglCreateSyncKHR instruction provided by OpenGL, and the synchronization object created by the synchronization instruction corresponds to a barrier (Fence) set between the call instruction initiated by Thread1 and the call instruction initiated by Thread 2. After setting the synchronization object, the GPU blocks the call instruction of Thread2 through a waiting interface (e.g., eglClientWaitSyncKHR), when the GPU executes to the synchronization object, it indicates that all call instructions (e.g., the above-mentioned X and Y) before the fence are executed, and thus a signal is sent to the waiting interface, the waiting interface stops blocking, and the call instruction (e.g., the above-mentioned Z and M) initiated by Thread1 can be executed by the GPU.
Through the steps shown in fig. 6, it can be ensured that the execution order of the call instructions of the different game threads by the game server 200 is consistent with the game client 110.
According to the above flow, the game server 200 may render a game screen corresponding to the control instruction. In practice, the game server 200 may capture the rendered game screen and encode the captured game screen into a video stream, which is sent to the game client 110 through the stub program 220. The game client 110 receives the video stream through the agent program 120, and decodes and plays the video stream, so that the remote rendering of the picture of the game client 110 can be realized.
By the remote rendering method provided by the embodiment, decoupling of GPU resources and CPU resources of the game server in the prior art is realized, so that hardware requirements on the CPU resources of the game server are reduced.
In addition, in the prior art, the complete game client is operated on the game server, the game server is taken as a huge whole, the requirement on hardware is very high, the software program is huge, and the subsequent maintenance and upgrading are very difficult. Compared with the prior art, the processing logic part of the game client is deployed on the user side equipment, so that the program complexity of one side of the game server is reduced, the subsequent program upgrading, compatibility, environment isolation and the like in the game server are easier, and the cost for developing the program in the game server is reduced.
In order to make those skilled in the art better understand the present solution, a specific example is given below in conjunction with the game client 110 shown in fig. 2 to further illustrate the remote rendering method provided in the present embodiment.
Assume that game client 110 includes the game threads Thread1 and Thread2 described above. The game client 110 firstly runs a game Thread1, and the game Thread1 initiates the above-mentioned graphics API instruction sequence { X, Y } based on a1 input by a user, wherein the graphics API instruction sequence includes call instructions X and Y arranged in sequence; then switching from the game Thread1 to the game Thread2, the game Thread2 initiates the above-mentioned graphics API instruction sequence { Z, M } based on the control instruction a2 input by the user, wherein the calling instructions Z and M are arranged in sequence; the game Thread2 initiates a graphics API instruction sequence { V, W } based on the user-entered control instruction a3, which includes call instructions V and W arranged in sequence, and the call instruction V is a synchronous call instruction.
Then, the remote rendering method provided in this embodiment may include the following processes:
first, an agent 120 provided in the game client 110 intercepts the graphics API instruction sequence { X, Y }, encapsulates it into a task t1, and adds it to the work queue.
Secondly, the agent program 120 detects that the game client 110 is switched from the game Thread1 to the game Thread2, generates a synchronization instruction K for creating a synchronization object, packages the synchronization instruction K into a synchronization task t2, and adds the synchronization task t2 to the work queue. It will be appreciated that the order of sync task t2 follows task t 1.
Third, agent 120 intercepts the graphics API instruction sequence { Z, M }, packages it into task t3, and adds it to the work queue. Currently, tasks t1, t2, and t3 are arranged in the work queue in this order.
Fourthly, sequentially executing tasks in the work queue through independent network IO threads, firstly packaging the graphic API instruction sequence in the task t1 into a data packet data1 to be sent and caching, then packaging the synchronization instruction K packaged in the task t2 into a data packet data2 to be sent and caching, and then packaging the graphic API instruction sequence packaged in the task t3 into a data packet data3 to be sent and caching.
It is worth noting that the operation of the agent 120 intercepting and encapsulating the graphics API instruction sequence into task addition to the work queue is performed in parallel with the operation of executing the tasks in the work queue through independent network IO threads.
Fifthly, the agent program 120 intercepts the graphics API command sequence { V, W }, detects that the command sequence contains a synchronous call command V, and thus sends out the currently cached data packets to be sent data1, data2 and data3, wherein the sending sequence of the data packets to be sent data1, data2 and data3 is consistent with the intercepted sequence of the graphics API command sequence contained therein.
Sixth, agent 120 encapsulates graphics API instruction sequence V, W into task t4 and adds to the work queue. The agent program 120 executes the task t4 according to the fourth step, and sends the data packet data4 to be sent corresponding to the task t4 to the game server 200 when a synchronous call instruction is intercepted next time or the game client 110 refreshes the command queue.
Seventh, the game server 200 receives the data packets data1, data2, data3 and data4 to be transmitted in sequence through the stub program 220, analyzes the graphic API command sequence { X, Y } from the data1, analyzes the synchronization command K from the data2, analyzes the graphic API command sequence { Z, M } from the data3, and analyzes the graphic API command sequence { V, W } from the data 4.
Eighth, GPU240 of game server 200 sequentially starts execution of instructions X, Y, K in the order in which the parsed instructions or instruction sequences are received. When the instruction K is executed, because the instruction K is a synchronous instruction, the GPU creates a synchronous object, which may play a blocking role, and when all instructions before the synchronous object are executed, the GPU240 starts to execute the subsequent instructions Z, M, V and W.
And ninthly, continuously rendering to obtain a corresponding game picture by executing the instruction. The game server 200 captures the rendered game screen by a video encoding program, encodes the captured game screen into a video stream, returns the video stream to the game client 110, and displays the video stream on the electronic device 100 on which the game client 110 is located.
Referring to fig. 7, fig. 7 shows a remote rendering apparatus 140 provided in this embodiment, where the remote rendering apparatus 140 includes at least one functional module that can be stored in the memory 101 in the form of machine executable instructions. Functionally divided, the remote rendering device 140 may include an interception module 141 and a rendering module 142.
The intercepting module 141 is configured to intercept a graphics API instruction sequence initiated by the game client based on a control instruction input by a user, where the graphics API instruction sequence includes a call instruction, a call parameter, and a call sequence of the game client to a graphics API.
The rendering module 142 is configured to send the intercepted graphics API instruction sequence to the game server according to the interception order, so that the GPU of the game server 200 executes the graphics API instruction sequence to obtain a rendered game picture.
Optionally, the intercepting module 141 is further configured to, when it is detected that the game client 110 performs game thread switching, add a synchronization task including a synchronization instruction in the work queue, send the synchronization instruction in the synchronization task to the game server 200 through the independent network IO thread, enable the GPU240 of the game server 200 to create a synchronization object based on the synchronization instruction, and execute a graphics API instruction sequence after the synchronization object when the execution of the graphics API instruction sequence before the synchronization object is completed.
The rendering module 142 is further configured to generate pseudo response information of the synchronous call instruction and return the pseudo response information to the game client 110 when the intercepted graphics API instruction sequence includes the synchronous call instruction, and send the pseudo response information to the game server 200, so that the game server 200 establishes and stores a corresponding relationship between the pseudo response information and actual response information of the synchronous call instruction.
The above description of the functional modules may refer to the foregoing detailed description of the relevant steps.
The present embodiments also provide a readable storage medium having stored thereon machine executable instructions that, when executed, implement the remote rendering method provided by the present embodiments.
In summary, the embodiments provide a remote rendering method, apparatus, electronic device and readable storage medium, where a game client intercepts a graphics API instruction sequence initiated based on a control instruction input by a user, and sends the intercepted graphics API instruction to a game server according to the interception order, so that a GPU of the game server executes the graphics API instruction sequence to obtain a rendered game screen, and logic that can be processed by a CPU is separated from the game server and is put into an electronic device on a user side for operation, thereby reducing hardware cost of the game server, and facilitating subsequent program upgrading, compatibility, environment isolation, and the like, thereby reducing development cost.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (9)
1. A remote rendering method is applied to an electronic device provided with a game client, wherein the electronic device is in communication connection with a game server, and the method comprises the following steps:
intercepting a graphic API instruction sequence initiated by the game client based on a control instruction input by a user, wherein the graphic API instruction sequence comprises a calling instruction, a calling parameter and a calling sequence of the game client to a graphic API;
sending the intercepted graphic API instruction sequence to the game server according to the interception sequence, and enabling a GPU of the game server to execute the graphic API instruction sequence to obtain a rendered game picture;
wherein the step of sending the intercepted graphic API command sequence to the game server in the interception order includes:
processing each intercepted graphic API command sequence into a data packet to be sent in sequence and caching the data packet;
and when a graphic API instruction sequence containing a synchronous calling instruction is intercepted or the command queue of the game client is detected to be refreshed, transmitting the currently cached data packet to be transmitted to the game server through a network IO thread.
2. The method of claim 1, wherein the network IO thread is an independent thread, and wherein the step of sending the sequence of intercepted graphics API instructions to the game server in the order of interception further comprises:
packaging each intercepted graphic API instruction sequence into a task, and adding the packaged tasks into a work queue according to an interception sequence;
the step of processing each intercepted graphic API instruction sequence into a data packet to be sent and caching the data packet to be sent in sequence comprises the following steps: and processing the graphic API instruction sequence in each task into a data packet to be sent and caching the data packet.
3. The method of claim 2, wherein the step of concurrently transmitting the currently cached data packets to be transmitted to the game server comprises:
and compressing and coding the currently cached data packet to be sent and sending the data packet to the game server.
4. A method according to any of claims 2-3, characterized in that the method further comprises:
when detecting that the game client-side carries out game thread switching, adding a synchronization task containing a synchronization instruction in the work queue;
and sending the synchronization instruction in the synchronization task to the game server through the independent network IO thread, so that the GPU of the game server creates a synchronization object based on the synchronization instruction, and executes the graphics API instruction sequence after the synchronization object when the graphics API instruction sequence before the synchronization object is executed.
5. The method according to any one of claims 1-3, further comprising:
when the intercepted graphics API instruction sequence contains a synchronous calling instruction, generating pseudo response information of the synchronous calling instruction and returning the pseudo response information to the game client; and
and sending the pseudo response information to the game server, so that the game server establishes a corresponding relation between the pseudo response information and the actual response information of the synchronous calling instruction and stores the corresponding relation.
6. The method of any of claims 1-3, wherein the electronic device intercepts the sequence of graphics API instructions generated by the game client by invoking a hook interface.
7. A remote rendering apparatus applied to an electronic device installed with a game client, the electronic device being in communication connection with a game server, the apparatus comprising:
the intercepting module is used for intercepting a graphic API instruction sequence initiated by the game client based on a control instruction input by a user, wherein the graphic API instruction sequence comprises a calling instruction, a calling parameter and a calling sequence of the game client to a graphic API;
the rendering module is used for sending the intercepted graphic API instruction sequence to the game server according to the interception sequence, so that the GPU of the game server executes the graphic API instruction sequence to obtain a rendered game picture;
wherein the rendering module is specifically configured to: processing each intercepted graphic API command sequence into a data packet to be sent in sequence and caching the data packet; and when a graphic API instruction sequence containing a synchronous calling instruction is intercepted or the command queue of the game client is detected to be refreshed, transmitting the currently cached data packet to be transmitted to the game server through a network IO thread.
8. An electronic device comprising a memory, a processor, and machine executable instructions stored in the memory and executed in the processor that, when executed by the processor, implement the remote rendering method of any of claims 1-6.
9. A readable storage medium having stored thereon machine executable instructions, wherein the machine executable instructions when executed implement the remote rendering method of any one of claims 1-6.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910599248.7A CN112169322B (en) | 2019-07-04 | 2019-07-04 | Remote rendering method and device, electronic equipment and readable storage medium |
| PCT/CN2020/099053 WO2021000843A1 (en) | 2019-07-04 | 2020-06-29 | Method for processing live broadcast data, system, electronic device, and storage medium |
| US17/566,790 US20220210484A1 (en) | 2019-07-04 | 2021-12-31 | Method for processing live broadcast data, system, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910599248.7A CN112169322B (en) | 2019-07-04 | 2019-07-04 | Remote rendering method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112169322A CN112169322A (en) | 2021-01-05 |
| CN112169322B true CN112169322B (en) | 2022-03-25 |
Family
ID=73915118
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910599248.7A Active CN112169322B (en) | 2019-07-04 | 2019-07-04 | Remote rendering method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112169322B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113018845B (en) * | 2021-03-25 | 2023-07-28 | 广州虎牙科技有限公司 | Operation instruction execution method, client, server and storage medium |
| CN114070875B (en) * | 2022-01-17 | 2022-04-22 | 北京蔚领时代科技有限公司 | Cloud program control method and device |
| CN114077508B (en) * | 2022-01-19 | 2022-10-11 | 维塔科技(北京)有限公司 | Remote image rendering method and device, electronic equipment and medium |
| CN114489459B (en) * | 2022-02-16 | 2023-11-03 | 北京蔚领时代科技有限公司 | Cloud input method and device |
| CN115474074B (en) * | 2022-08-29 | 2024-05-07 | 咪咕文化科技有限公司 | Video background replacement method, device, computing equipment and storage medium |
| CN116991600B (en) * | 2023-06-15 | 2024-05-10 | 上海一谈网络科技有限公司 | Method, device, equipment and storage medium for processing graphic call instruction |
| CN117472371A (en) * | 2023-10-09 | 2024-01-30 | 北京趋动智能科技有限公司 | Remote rendering method, device and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102254292A (en) * | 2010-05-20 | 2011-11-23 | 盛乐信息技术（上海）有限公司 | Remote 3D instruction rendering system and method |
| US20140285519A1 (en) * | 2013-03-22 | 2014-09-25 | Nokia Corporation | Method and apparatus for providing local synchronization of information for augmented reality objects |
| US9270647B2 (en) * | 2013-12-06 | 2016-02-23 | Shape Security, Inc. | Client/server security by an intermediary rendering modified in-memory objects |
| CN105096373B (en) * | 2015-06-30 | 2020-04-28 | 华为技术有限公司 | Media content rendering method, user equipment and system |
| CN108076003B (en) * | 2016-11-07 | 2020-10-30 | 阿里巴巴集团控股有限公司 | Session hijacking detection method and device |
| CN109194642B (en) * | 2018-08-28 | 2022-05-03 | 深圳创维数字技术有限公司 | Whiteboard remote synchronous processing method, system and storage medium |
-
2019
- 2019-07-04 CN CN201910599248.7A patent/CN112169322B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112169322A (en) | 2021-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112169322B (en) | Remote rendering method and device, electronic equipment and readable storage medium | |
| EP4122568A1 (en) | Data processing method and device and storage medium | |
| CN109582425B (en) | GPU service redirection system and method based on cloud and terminal GPU fusion | |
| US7830388B1 (en) | Methods and apparatus of sharing graphics data of multiple instances of interactive application | |
| US8924985B2 (en) | Network based real-time virtual reality input/output system and method for heterogeneous environment | |
| US20230215076A1 (en) | Image frame display method, apparatus, device, storage medium, and program product | |
| US20220210484A1 (en) | Method for processing live broadcast data, system, electronic device, and storage medium | |
| US20240096007A1 (en) | Rendering Method, Device, and System | |
| WO2018112855A1 (en) | Virtualisation method and device, electronic device, and computer program product | |
| CN108762934B (en) | Remote graphic transmission system and method and cloud server | |
| EP3975126A1 (en) | Method and system for cloud-native 3d-scene game | |
| CN113457160A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| WO2024037110A1 (en) | Data processing method and apparatus, device, and medium | |
| CN108389241A (en) | The methods, devices and systems of textures are generated in scene of game | |
| CN112023402B (en) | Game data processing method, device, equipment and medium | |
| CN112843676A (en) | Data processing method, device, terminal, server and storage medium | |
| US10237563B2 (en) | System and method for controlling video encoding using content information | |
| Bulman et al. | A cloud gaming framework for dynamic graphical rendering towards achieving distributed game engines | |
| Yadav et al. | Adaptive GPU resource scheduling on virtualized servers in cloud gaming | |
| CN112423111A (en) | Graphic engine and graphic processing method suitable for player | |
| CN115248714A (en) | Method for improving VDI remote desktop performance by using VGPU | |
| CN113273219A (en) | Distributed video and graphics rendering system | |
| CN115409681A (en) | Rendering method and related device | |
| CN113778593A (en) | Cloud desktop control method and device, electronic equipment, storage medium and program product | |
| CN115955590A (en) | Video processing method, video processing device, computer equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |