WO2023045392A1 - Cloud mobile phone implementation method and apparatus - Google Patents

Abstract

The present application relates to the technical field of computers, and provides a cloud mobile phone implementation method and apparatus, which solve the problem that existing cloud mobile phone implementation modes have relatively large restrictions on the scene, and are not suitable for scenes with high picture quality or high follow-up delay requirements. The method is used for storing and running a cloud service of a cloud mobile phone; an instruction stream processing module and a video stream processing module are mounted on the cloud server; and the method comprises: sending a first data stream to a terminal device by means of a first processing module, the first processing module being the instruction stream processing module or the video stream processing module; if it is determined that a current running state satisfies a switching condition, starting a second processing module and sending a switching instruction to the terminal device, wherein the switching instruction being used for instructing the terminal device to be switched from the first processing module to the second processing module; and sending a second data stream to the terminal device by means of the second processing module and closing the first processing module.

Description

Implementation method and device of a cloud mobile phone

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on September 26, 2021, with the application number 202111128778.7, and the application name "A method and device for implementing a cloud mobile phone", the entire content of which is incorporated by reference In this application.

technical field

The present application relates to the field of computer technology, and in particular to a method and device for implementing a cloud mobile phone.

Background technique

Cloud mobile phone refers to a virtual mobile phone built on a cloud server. The cloud mobile phone platform allows users to remotely operate the virtual cloud mobile phone on the cloud server through physical devices such as computers or mobile phones. The functions of these virtual cloud mobile phones are the same as those of physical mobile phones. Basically the same, based on the processing power of the cloud server, various applications can be efficiently built, such as cloud games, mobile office, video playback or live broadcast applications, etc., to meet various user needs.

For the current remote access methods of physical mobile phones or physical devices to cloud mobile phones, the mainstream method includes video stream access, that is, the cloud mobile phone deployed on the cloud server installs a video streaming engine, and the video streaming engine continuously captures every Display the picture (image) frame by frame, and then encode the acquired frame-by-frame image into a video code stream, such as H.264 code stream or H.265 code stream, etc., and transmit the video code stream to the physical mobile phone in real time through a network connection. The video stream engine on the physical mobile phone decodes the received video code stream, and then displays it in real time on the screen of the physical mobile phone, thus realizing the access control of the physical mobile phone to the cloud mobile phone.

However, the above-mentioned application method has relatively large restrictions on the scene, and is suitable for scenes with low picture quality (that is, low resolution) and low requirements for follow-up delay. For example, if the real-time display picture resolution is 1920*1080 or higher, the transmitted video stream data packets will be larger, which will result in higher transmission costs and a larger occupation of network bandwidth. Poor, there may be packet loss or freeze phenomenon.

Contents of the invention

This application provides a method and device for implementing a cloud mobile phone, which solves the problem that the existing cloud mobile phone implementation method has relatively large restrictions on scenarios, and is not suitable for scenarios with high picture quality or high follow-up delay requirements. restrictions are also relatively large.

In order to achieve the above object, the application adopts the following technical solutions:

In the first aspect, a method for implementing a cloud mobile phone is provided, which is applied to a cloud server. The cloud server is used for storing and running the cloud mobile phone. The cloud server is equipped with an instruction stream processing module and a video stream processing module. The method includes: using the first processing module Sending the first data stream to the terminal device, if the first processing module is a video stream processing module, the first data stream is a video stream, and if the first processing module is an instruction stream processing module, then the first data stream is a rendering instruction stream; If it is determined that the current operating state meets the switching condition, then start the second processing module and send a switching instruction to the terminal device, wherein the switching instruction is used to instruct the terminal device to switch from the first processing module to the second processing module, if the first processing module is Video stream processing module, then the second processing module is an instruction stream processing module, if the first processing module is an instruction stream processing module, then the second processing module is a video stream processing module; send the second data to the terminal device through the second processing module stream, and close the first processing module, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.

In the above technical solution, the cloud mobile phone deployed on the cloud server supports the command stream processing mode and the video stream processing mode, and can switch between the command stream processing mode and the video stream processing mode according to the current actual running state, so that the The terminal device supports image display based on the video stream, or image display based on the rendering instruction stream, which improves the flexibility of the cloud mobile phone implementation and the adaptability to different scenarios, improves the implementation efficiency of the cloud mobile phone, and improves the user experience.

In a possible design manner, determining that the current operating state meets the switching condition specifically includes at least one of the following: determining that the current network state parameter meets the preset condition, or determining that the resolution of the currently displayed image meets the preset condition, or, Determine that the type of the currently running application meets the preset condition, or determine that the bandwidth requirement of the currently running application meets the preset condition, or determine that the data processing capability of the terminal device meets the preset condition, wherein the network status parameters include loss At least one of packet rate, available bandwidth, or delay.

In the above possible implementation methods, the cloud server can judge whether it is necessary to switch between the instruction stream processing mode and the video stream processing mode, which can be realized according to different switching decision strategies, and different strategies can also be used in combination, which improves the cloud mobile phone implementation. The flexibility further improves the accuracy and efficiency of cloud phone implementation in different application scenarios, and improves user experience.

In a possible design mode, if the first processing module is a video stream processing module, start the second processing module and send the switching instruction and the second data stream to the terminal device, specifically including: starting the instruction stream processing module to complete graphics rendering State machine data preparation; sending to the terminal device a switching instruction indicating to switch to the instruction stream processing module of the terminal device; sending a rendering instruction stream to the terminal device.

In the above possible implementation, when the cloud mobile phone determines to switch from the video stream processing module to the command stream processing module, the cloud mobile phone can complete the data preparation of the graphics rendering state machine by starting the command stream processing module, and send the switching command to the terminal device and The rendering instruction stream obtained by the instruction stream processing module can automatically switch from the video stream processing module to the instruction stream processing module, and notify the terminal device to also realize the data preparation for switching from the video stream processing module to the instruction stream processing module, improving the cloud mobile phone The flexibility of the implementation method and the smooth transition of the switching, the screen display is not affected, and the user does not perceive the switching of the access mode, which improves the user experience.

In a possible design manner, shutting down the first processing module specifically includes: receiving a switching completion instruction from the terminal device, and shutting down the video stream processing module.

In the above possible implementation methods, the cloud mobile phone can complete the command according to the switching of the terminal device, and then close the video stream processing module, so as to improve the accuracy and security of the cloud mobile phone access mode switching, the screen display is smooth and excessive, and the user does not perceive the connection mode. Switching of input methods improves user experience.

In a possible design manner, if it is determined that the current operating state meets the switching condition, then start the second processing module and send a switching instruction to the terminal device, which specifically includes: if it is determined that the current packet loss rate is greater than or equal to the first threshold, then determining Satisfying the conditions for switching from the instruction stream processing module to the video stream processing module; starting the video stream processing module and sending an instruction to switch to the video stream processing module to the terminal device.

In the above possible implementation, the cloud server can determine that the network status is not good according to the packet loss rate being greater than or equal to the preset threshold. The stream processing module is switched to the video stream processing module, thereby improving the implementation efficiency of cloud mobile phones and improving user experience.

In a possible design manner, if it is determined that the current operating state satisfies the switching condition, then start the second processing module and send a switching instruction to the terminal device, which specifically includes: if it is determined that the current available bandwidth is greater than or equal to the second threshold, then determining that the condition is satisfied Conditions for switching from the video stream processing module to the instruction stream processing module; starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.

In the above possible implementations, the decision-making strategy for the cloud service to determine the access method can also be based on whether the current available bandwidth is greater than or equal to the preset threshold, that is, if the network status is determined to be good, the cloud server can determine to switch from the video stream processing method to Instruction stream processing mode, so as to realize automatic switching and improve flexibility.

In a possible design manner, if it is determined that the current running state satisfies the switching condition, the second processing module is started and a switching instruction is sent to the terminal device, which specifically includes: if it is determined that the resolution of the currently displayed image is lower than or equal to 1080*720, Then it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met; the video stream processing module is started and an instruction to switch to the video stream processing module is sent to the terminal device; or, if it is determined that the resolution of the currently displayed image is higher than or equal to 1920*1080, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is satisfied; the instruction stream processing module is started and an instruction to switch to the instruction stream processing module is sent to the terminal device.

In the above-mentioned possible implementation, the cloud server determines that the resolution of the currently displayed image is lower than or equal to 720P (ie 1080*720), and the data volume of the video stream is small, then the video stream processing module can be used; the resolution of the currently displayed image If it is higher than or equal to 1080P (that is, 1920*1080), and the data volume of the video stream is large, the instruction stream processing module can be switched to realize automatic switching to meet different application scenarios.

In a possible design mode, if it is determined that the current running state satisfies the switching condition, the second processing module is started and a switching instruction is sent to the terminal device, which specifically includes: determining that the type of the currently running application program is a strongly interactive game or other For applications of strong interaction type, or if it is determined that the currently running application has a small bandwidth requirement for the instruction stream, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met; the instruction stream processing module is started and sent to the terminal device Send an instruction to switch to the instruction stream processing module.

In the above possible implementations, the decision-making strategy for the cloud service to determine the access method can also be based on the type of the currently running application, for example, a game with strong interaction or other applications with strong interaction. If the bandwidth requirement is smaller than that of the video stream, it is determined to use the instruction stream processing module, so as to reduce the bandwidth occupation of the cloud mobile phone implementation and improve the efficiency of the cloud mobile phone implementation.

In a possible design mode, if it is determined that the current running state satisfies the switching condition, the second processing module is started and a switching instruction is sent to the terminal device, which specifically includes: if it is determined that the currently running application program is a weakly interactive game or other weak Interactive type of application, or, if it is determined that the currently running application has a small bandwidth requirement for video streams, or, when the processing capability of the graphics processor GPU on the terminal device is lower than the preset condition, then it is determined that the instruction stream processing module meets the requirements. A condition for switching to the video stream processing module; starting the video stream processing module and sending an instruction to switch to the video stream processing module to the terminal device.

In the above possible implementation manners, when the application is a weak-interaction game or other weak-interaction application, or a mobile office APP or a social APP, it is determined that the bandwidth requirements of the currently running application for the video stream are relatively high. Since the video stream is small, it can be determined to use the instruction stream processing module, thereby reducing the bandwidth occupation of cloud mobile phone implementation and improving the efficiency of cloud mobile phone implementation.

In addition, further, the cloud service's decision-making strategy for determining the access method can also combine the processing capability of the graphics processor GPU of the terminal device. When the GPU processing capability of the terminal device is weak, the cloud server can determine to use the video stream processing module; when When the GPU processing capability of the terminal device is sufficient, switch to using the instruction stream processing module. In this way, the occupation of the processing capacity of the terminal device by the implementation of the cloud mobile phone is reduced, and the efficiency of the implementation of the cloud mobile phone is improved.

In the second aspect, a method for implementing a cloud mobile phone is provided, which is applied to a terminal device, and the terminal device is used to remotely control the cloud mobile phone. The method includes: receiving a first data stream from a cloud server, where the first data stream is a video stream or a rendering instruction Streaming; performing image display according to the first data stream; receiving a switching instruction from the cloud server, the switching instruction is used to instruct the terminal device to switch from the first processing module to the second processing module; wherein, the first processing module is an instruction stream processing module and The second processing module is a video stream processing module, or the first processing module is a video stream processing module and the second processing module is an instruction stream processing module; receiving the second data stream from the cloud server, and performing image processing according to the second data stream display, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.

In the above technical solution, when the terminal device remotely controls the cloud mobile phone, it can flexibly switch between the instruction stream processing method and the video stream processing method according to the switching instruction of the cloud server, so that the terminal device can support image display according to the received video stream , or perform image display according to the received rendering instruction stream, which improves the flexibility of the implementation of the cloud mobile phone and the adaptability of different scenarios, can improve the implementation efficiency of the cloud mobile phone, and improve user experience.

In a possible design manner, if the switching instruction is used to instruct switching from the video stream processing module to the instruction stream processing module, receive the second data stream from the cloud server, and perform image display according to the second data stream, specifically including: Start the instruction stream processing module to rebuild the graphics rendering state machine; receive the second data stream from the cloud server and decode it to obtain rendering instructions; render according to the rendering instructions to obtain the original display image; switch from the video stream drawing surface to the instruction stream drawing Surface; close the video stream processing module.

In the above possible implementation, when the terminal device determines that it needs to switch from the video stream processing module to the instruction stream processing module according to the switching instruction, the graphics rendering state machine can be rebuilt by starting the instruction stream processing module, so that the graphics rendering state machine can be rendered according to the received rendering instruction. The display image is obtained, and the video stream drawing surface is switched to the instruction stream drawing surface, so as to realize the smooth switching of the two access methods, the screen display is smooth and transitional, and the user experience is improved.

In a possible design manner, after switching from the video stream drawing surface to the instruction stream drawing surface, the method further includes: sending a switching completion instruction to the cloud server, which is used to instruct the cloud server to shut down the video stream processing module.

In the above-mentioned possible implementation, when the upper screen display on the terminal device side completes switching from the video stream processing module to the command stream processing module, the cloud server can be notified to close the video stream processing module by sending a switching completion command to the cloud server, so that Improve the accuracy and security of cloud mobile phone access mode switching, the screen display is smooth and transitional, users do not perceive the switching of access modes, and improve user experience.

In the third aspect, there is provided a device for implementing a cloud mobile phone. The device is used for storing and running the cloud mobile phone. The device includes a video stream processing module and an instruction stream processing module. The device also includes a sending module and a processing module. The sending module is used to send the first data stream to the terminal device. If the first processing module is a video stream processing module, the first data stream is a video stream. If the first processing module is an instruction stream processing module, the first data stream is The stream is a stream of rendering instructions; the processing module is used to start the second processing module and send a switching instruction to the terminal device if it is determined that the current running state satisfies the switching condition, wherein the switching instruction is used to instruct the terminal device to switch from the first processing module to The second processing module, if the first processing module is a video stream processing module, then the second processing module is an instruction stream processing module, and if the first processing module is an instruction stream processing module, then the second processing module is a video stream processing module; sending The module is also used to send a second data stream to the terminal device, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream; the processing module is also used to close the first processing module.

In a possible design manner, determining that the current operating state meets the switching condition specifically includes at least one of the following: determining that the current network state parameter meets the preset condition, or determining that the resolution of the currently displayed image meets the preset condition, or, Determine that the type of the currently running application meets the preset condition, or determine that the bandwidth requirement of the currently running application meets the preset condition, or determine that the data processing capability of the terminal device meets the preset condition, wherein the network status parameters include loss At least one of packet rate, available bandwidth, or delay.

In a possible design mode, if the first processing module is a video stream processing module, the processing module is specifically used to start the instruction stream processing module and complete the data preparation of the graphics rendering state machine; the sending module is specifically used to send the video stream to the terminal device. Instruct to switch to the switching instruction of the instruction stream processing module of the terminal device; send the rendering instruction stream to the terminal device.

In a possible design manner, the device further includes a receiving module, configured to receive a handover completion instruction from the terminal device, and shut down the video stream processing module.

In a possible design manner, the processing module is specifically configured to: if it is determined that the current packet loss rate is greater than or equal to the first threshold, then determine that the condition for switching from the instruction stream processing module to the video stream processing module is met; start the video stream processing module and send an instruction to the terminal device to switch to the video stream processing module.

In a possible design manner, the processing module is specifically configured to: if it is determined that the current available bandwidth is greater than or equal to the second threshold, determine that the condition for switching from the video stream processing module to the instruction stream processing module is met; start the instruction stream processing module And send an instruction to switch to the instruction stream processing module to the terminal device.

In a possible design mode, the processing module is specifically used to: if it is determined that the resolution of the currently displayed image is lower than or equal to 1080*720, then determine that the condition for switching from the instruction stream processing module to the video stream processing module is met; start the video The stream processing module sends an instruction to switch to the video stream processing module to the terminal device; or, if it is determined that the resolution of the currently displayed image is higher than or equal to 1920*1080, it is determined that the video stream processing module is switched to the instruction stream processing module. Condition: start the instruction stream processing module and send an instruction to switch to the instruction stream processing module to the terminal device.

In a possible design manner, the processing module is specifically used to: determine that the type of the currently running application program is a game of strong interaction type or other strong interaction type of application program, or determine that the currently running application program has an If the bandwidth requirement is small, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met; the instruction stream processing module is started and an instruction to switch to the instruction stream processing module is sent to the terminal device.

In a possible design mode, the processing module is specifically used for: if it is determined that the currently running application program is a weak-interaction game or other weak-interaction application program, or to determine the bandwidth of the currently running application program for the video stream The demand is small, or when the processing capability of the graphics processor GPU on the terminal device is lower than the preset condition, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met; the video stream processing module is started and sent to the terminal device An instruction to switch to the video stream processing module.

In the fourth aspect, a device for implementing a cloud mobile phone is provided. The device is used to remotely control the cloud mobile phone. The device includes a receiving module and a processing module. The receiving module is used to receive the first data stream from the cloud server, and the first data stream is a video stream Or a rendering instruction stream; the processing module is used to display images according to the first data stream; the receiving module is also used to receive a switching instruction from the cloud server, and the switching instruction is used to instruct the device to switch from the first processing module to the second processing module; wherein , the first processing module is an instruction stream processing module and the second processing module is a video stream processing module, or the first processing module is a video stream processing module and the second processing module is an instruction stream processing module; the receiving module is also used to receive data from The second data stream of the cloud server; the processing module is further configured to display images according to the second data stream, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.

In a possible design mode, if the switching instruction is used to indicate switching from the video stream processing module to the instruction stream processing module, the processing module is specifically used to: start the instruction stream processing module, and rebuild the graphics rendering state machine; the receiving module is specifically used to: Receive the second data stream from the cloud server and decode it to obtain a rendering instruction; the processing module is specifically used to: render according to the rendering instruction to obtain the original display image; switch from the video stream drawing surface to the instruction stream drawing surface; close the video stream processing module.

In a possible design manner, the device further includes a sending module, configured to send a handover completion instruction to the cloud server, and used to instruct the cloud server to shut down the video stream processing module.

According to a fifth aspect, an electronic device is provided, and the electronic device includes: a memory and one or more processors; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code It includes computer instructions, and when the processor executes the computer instructions, the electronic device is configured to execute the method described in any one of the above first aspects.

According to a sixth aspect, an electronic device is provided, and the electronic device includes: a memory and one or more processors; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code It includes computer instructions, and when the processor executes the computer instructions, the electronic device is configured to execute the method as described in any one of the above second aspects.

In a seventh aspect, there is provided a computer-readable storage medium, and instructions are stored in the readable storage medium, and when the readable storage medium is run on an electronic device, the electronic device executes the above-mentioned first aspect. any one of the methods described.

In an eighth aspect, there is provided a computer-readable storage medium, where instructions are stored in the readable storage medium, and when the readable storage medium is run on an electronic device, the electronic device executes the above-mentioned second aspect. any one of the methods described.

In a ninth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is made to execute the method described in any one of the above first aspects.

In a tenth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is made to execute the method described in any one of the above first aspects.

In an eleventh aspect, a communication system is provided, and the communication system includes the device described in any one of the above third aspects and the device described in any one of the above fourth aspects.

It can be understood that any implementation device, electronic device, computer-readable storage medium, computer program product, or communication system of any cloud phone provided above can be realized by the corresponding method provided above. Therefore, the above-mentioned For the beneficial effects that can be achieved, reference can be made to the beneficial effects in the corresponding methods provided above, which will not be repeated here.

Description of drawings

FIG. 1 is an architecture diagram of a communication system provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of an implementation of a video streaming cloud mobile phone provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an implementation of an instruction flow cloud mobile phone provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

FIG. 6 is a schematic flowchart of a method for implementing a cloud mobile phone provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of an execution module and an execution flow of an implementation method of a cloud mobile phone provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a switching strategy included in a cloud mobile phone provided by an embodiment of the present application;

Fig. 9 and Fig. 10 are schematic flow diagrams 1 and 2 of a cloud mobile phone implementation method provided by the embodiment of the present application;

FIG. 11 is a schematic structural diagram of a device for realizing a cloud mobile phone provided by an embodiment of the present application;

FIG. 12 is a schematic structural diagram of another device for implementing a cloud mobile phone provided by an embodiment of the present application.

Detailed ways

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

First, the implementation environment and application scenarios of the embodiments of the present application are briefly introduced.

This application is applied to the communication system shown in Figure 1. The communication system 100 may include a server 101 and a terminal device 102, wherein the server 101 may be a data processing center of a cloud mobile phone, that is, a cloud server, and at least one Cloud phone.

The cloud mobile phone is deployed on the server 101, and the operating system running on the cloud mobile phone can be a native Android (Android) operating system or a customized Android compatible system, etc., and an Android operating environment is provided. In addition, multiple applications (Applications, APPs) can also be installed on the cloud mobile phone, for example, games, video playing APPs, office APPs or other applications that can be installed on the Android operating system.

The terminal device 102 can remotely control the cloud mobile phone deployed on the server 101 through a network connection, and display the image of the cloud mobile phone in real time. For example, the user starts the cloud mobile phone through the terminal device 102, opens a certain application program installed on the cloud mobile phone, such as a video playback APP, selects one of the videos to play, and the terminal device 102 displays in real time the content of the video playback APP running on the cloud mobile phone. screen.

Wherein, the terminal device specifically may be a smart phone, a smart camera, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, and a cellular phone. Phones, personal digital assistants (PDAs), augmented reality (augmented reality, AR), virtual reality (virtual reality, VR) and wearable devices.

It should be noted that, in the embodiments of the present application, the solution of the present application is described in detail by taking the terminal device as an example of a smart phone. In order to distinguish the cloud phone from the physical device that remotely controls the cloud phone, the terminal device that realizes remote control is referred to as a physical phone in this application, which does not limit the scope of protection of this application.

At present, the remote access methods of physical mobile phones to cloud mobile phones include video stream access methods. As shown in Figure 2, cloud mobile phones are deployed on cloud services, and cloud mobile phones include at least one APP, graphics card driver, graphics card, and video stream processing module. Wherein, the video stream processing module can be installed on the Android operating system of the cloud mobile phone, or can be installed on the operating system on the cloud server.

Wherein, the video stream processing module may specifically be a module implemented by software, and may also be called a video stream engine. Furthermore, the video stream processing module installed on the cloud mobile phone may be called a video stream engine server, and the video stream processing module installed on a terminal device such as a physical mobile phone may be called a video stream engine client.

An APP runs on the cloud phone, and the APP completes the development of the interface display based on the Android graphics library. Among them, the graphics libraries supported by Android include OpenGL ES, Vukan and other graphics libraries. Exemplarily, the Android graphics library calls the graphics card driver interface OpenGL ES application programming interface (Application Programming Interface, API) to complete the rendering of graphics.

The video stream processing module continuously captures each frame of image of the APP through the graphics card driver interface or the Android interface, for example, it can be an image stream in RGB, YUV or other formats, and sends each frame of captured image (that is, the image stream) to the encoding Service, the encoding service encodes and compresses each frame of image into a video stream and transmits the video stream to the physical mobile phone through the network. Exemplarily, the video stream may be an H.264 code stream, an H.265 code stream, or other public or custom code streams.

Among them, the encoding service may be a software encoder, such as Open264 or x264, etc.; it may also be a hardware encoder, such as various video encoding cards; or it may be a cloud encoding service, such as providing an encoding service through an API interface of a public cloud.

The video stream processing module on the physical mobile phone decodes the received video code stream to obtain a continuous image stream, such as RGB, YUV or other format image streams, which can display each frame of image on the display screen of the physical mobile phone.

The above-mentioned video stream rendering method is suitable for display scenarios with low image quality (that is, low resolution), such as 720P video, that is, the image resolution is 1080*720, and the frame rate of the video is 30fps, that is, 30 frames of images are displayed per second. . If the resolution of the real-time display screen is 1920*1080 or higher, the transmitted video stream data packets will be larger, which will result in higher transmission costs and a larger occupation of network bandwidth. The delay will also be longer, and if the network status is poor, there may be packet loss or freezing.

In addition, the remote access method of the physical mobile phone to the cloud mobile phone can also be an instruction stream access method. As shown in Figure 3, the cloud mobile phone is deployed on the cloud service, and the cloud mobile phone includes at least one APP and an instruction stream processing module. Wherein, the instruction stream processing module can be installed on the Android operating system of the cloud mobile phone, or can be installed on the operating system on the cloud server.

Wherein, the instruction stream processing module may specifically be a module implemented by software, and may also be called an instruction stream engine. Furthermore, the instruction stream processing module installed on the cloud mobile phone may be called an instruction stream engine server, and the instruction stream processing module installed on a terminal device such as a physical mobile phone may be called an instruction stream engine client.

An APP runs on the cloud phone, and the APP completes the development of the interface display based on the Android graphics library.

The instruction stream processing module takes over the interface calls of the graphics library. The interface calls may include function calls and parameters therein. The instruction stream processing module encodes the interface calls into rendering instruction streams (ie, binary data streams), and sends the rendering instruction streams to physical phone.

The command stream processing module on the physical mobile phone can decode the received rendering command stream, restore it to an interface call to the graphics library, and call the graphics card driver of the physical mobile phone, and the graphics card can complete the rendering according to the rendering command. For example, the graphics card of a physical mobile phone is integrated with a Graphics Processing Unit (GPU), which invokes the GPU driver, and the GPU of the physical mobile phone completes graphics rendering according to rendering instructions, and projects them on the display screen of the physical mobile phone.

In the above instruction stream rendering method, the encoded rendering instruction stream is transmitted between the cloud mobile phone and the physical mobile phone, which can be applied to display scenes with high picture quality (that is, high resolution), that is, ultra-high-definition image quality is supported, and Compared with the video streaming rendering method, the follow-up delay is smaller, and it is suitable for application scenarios with frequent user interactions, and consumes less traffic for small and medium-sized applications. However, for application scenarios with frequent interactions with users, the real-time rendering instruction data generated by the cloud mobile phone is large, which will result in high transmission costs and a large occupation of network bandwidth. In addition, because the command stream rendering method has a small tolerance for packet loss, when the network status is poor, frequent packet loss will cause frequent retransmission of rendering command stream packets, resulting in serious screen freezes and affecting user experience.

To sum up, the command stream rendering method and the video stream rendering method are only applicable to some scenarios. No matter which rendering method is applied on the cloud phone, users cannot get a better experience in all scenarios.

The embodiment of the present application provides a method and device for implementing a cloud mobile phone. By deploying a video stream processing module and an instruction stream processing module on both the cloud mobile phone and the physical mobile phone, when the physical mobile phone remotely starts the cloud mobile phone, the cloud mobile phone can According to the policy, the video stream rendering method or the instruction stream rendering method is selected for image rendering, and different rendering methods can be smoothly switched according to the real-time status, and the user has no perception, which improves the experience of using the cloud mobile phone.

The embodiment of the present application can be applied to the communication system shown in FIG. 1 . In the embodiment of the present application, there are no special restrictions on the specific forms of the cloud server and the terminal device. The following embodiments only use electronic devices as examples for introduction.

It should be noted that the cloud server or the terminal device in the embodiment of the present application may adopt the composition structure shown in FIG. 4 or include the components shown in FIG. 4 . FIG. 4 is a schematic structural diagram of an electronic device 400 provided by the embodiment of the present application. For example, when the electronic device 400 has the function of the network device described in the embodiment of the present application, the electronic device 400 can be a network device or a network device chip or system-on-a-chip. When the electronic device 400 has the function of the terminal device described in the embodiment of the present application, the electronic device 400 may be a terminal device or a chip or a chip system in the terminal device.

As shown in FIG. 4 , the electronic device 400 may include a processor 401 , a communication line 402 and a communication interface 403 . Optionally, the electronic device 400 may further include a memory 404 . Wherein, the processor 401 , the memory 404 and the communication interface 403 may be connected through a communication line 402 .

Wherein, the processor 401 may be a central processing unit (central processing unit, CPU), a general-purpose processor, a network processor (network processor, NP), a digital signal processor (digital signal processing, DSP), a microprocessor, a microcontroller , programmable logic device (programmable logic device, PLD) or any combination thereof. The processor 401 may also be other devices with processing functions, such as circuits, devices, or software modules.

The communication line 402 is used for transmitting information between the components included in the electronic device 400 .

The communication interface 403 is used for communicating with other devices or other communication networks. The other communication network may be an Ethernet, a radio access network (radio access network, RAN), a wireless local area network (wireless local area networks, WLAN), and the like. The communication interface 403 may be a radio frequency module or any device capable of realizing communication. This embodiment of the present application is described only by taking the communication interface 403 as an example of a radio frequency module, where the radio frequency module may include an antenna, a radio frequency circuit, etc., and the radio frequency circuit may include a radio frequency integrated chip, a power amplifier, and the like.

The memory 404 is used for storing instructions. Wherein, the instruction may be a computer program.

Wherein, the memory 404 may be a read-only memory (read-only memory, ROM) or other types of static storage devices capable of storing static information and/or instructions, or may be a random access memory (random access memory, RAM) or may Other types of dynamic storage devices that store information and/or instructions can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD- ROM) or other optical disc storage, optical disc storage, magnetic disk storage media, or other magnetic storage devices, including compact discs, laser discs, compact discs, digital versatile discs, Blu-ray discs, etc.

It should be noted that the memory 404 may exist independently of the processor 401 or may be integrated with the processor 401 . The memory 404 can be used to store instructions or program codes or some data, etc. The memory 404 may be located inside the electronic device 400 or outside the electronic device 400, without limitation. The processor 401 is configured to execute instructions stored in the memory 404, so as to implement the communication method provided by the following embodiments of the present application.

Or, optionally, in the embodiment of the present application, the processor 401 may also perform processing-related functions in the communication method provided in the following embodiments of the present application, and the communication interface 403 is responsible for communicating with other devices or communication networks. The embodiment does not specifically limit this.

Optionally, the computer-executed instructions in the embodiments of the present application may also be referred to as application program codes, which is not specifically limited in the embodiments of the present application.

In an example, the processor 401 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 4 .

As an optional implementation manner, the electronic device 400 includes multiple processors, for example, in addition to the processor 401 in FIG. 4 , it may further include a processor 406 .

As an optional implementation manner, the electronic device 400 further includes an output device 405 and an input device 407 . Exemplarily, the input device 407 is a device such as a keyboard, a mouse, a microphone, or a joystick, and the output device 406 is a device such as a display screen and a speaker (speaker).

It should be noted that the electronic device 400 may be a device or a chip system having a structure similar to that shown in FIG. 4 . In addition, the composition structure shown in FIG. 4 does not constitute a limitation to the electronic device. In addition to the components shown in FIG. 4, the electronic device may include more or less components than those shown in the illustration, or combine certain components , or different component arrangements.

In the embodiment of the present application, the system-on-a-chip may be composed of chips, or may include chips and other discrete devices.

The method provided by the embodiment of the present application will be described below with reference to the electronic device shown in FIG. 4 . Wherein, the electronic equipment in the following embodiments may have the components shown in FIG. 4 . The actions and terms involved in the various embodiments of the present application may refer to each other without limitation. In the embodiment of the present application, the names of messages exchanged between various devices or the names of parameters in messages are just examples, and other names may also be used in specific implementations, which are not limited.

The technical solutions involved in the following embodiments can all be implemented in electronic devices with the above hardware architecture.

In addition, the software system of the cloud mobile phone deployed on the above cloud server can be a native Android system or a customized Android compatible system. The software system of the terminal device may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. For example, the operating system of the physical mobile phone can be Android, ios, Symbian, Hongmeng Harmony or other terminal operating systems, etc. The embodiment of the present application only uses the Android system with a layered architecture as an example to illustrate the software structure of the electronic device. The software architecture shown below does not constitute a specific limitation on the electronic device.

FIG. 5 is a software structure diagram of an electronic device according to an embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into three layers, which are application program layer, application program framework layer (framework), and kernel layer from top to bottom.

The software system of the electronic device 400 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 400 is illustrated by taking the Android system with a layered architecture as an example.

FIG. 5 is a block diagram of a software structure of an electronic device 400 according to an embodiment of the present invention. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, Android runtime (Android runtime) and system library, and the kernel layer from top to bottom.

The application layer can consist of a series of application packages. As shown in FIG. 5, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 5, the application framework layer can include window managers, content providers, view systems, phone managers, resource managers, notification managers, and so on.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 400 . For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manage), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG or PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc. 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

The methods in the following embodiments can all be implemented in the electronic device 400 having the hardware structure shown in FIG. 4 and the software structure shown in FIG. 5 . Next, the method provided by the embodiment of the present application will be described in detail with reference to the foregoing FIG. 4 and FIG. 5 . The following embodiments of the present application are described in detail by taking the electronic device as a physical mobile phone as an example.

The embodiment of the present application provides a method for implementing a cloud mobile phone, which can be applied to the interaction between a cloud server and a terminal device. For example, the following embodiments only use a physical mobile phone as an example for detailed description. As shown in Fig. 6, the method includes the following steps.

601: The cloud server sends the first data stream to the terminal device.

Among them, a cloud mobile phone is deployed on the cloud server, and the cloud mobile phone runs a native Android system or a customized Android compatible system. In addition, the cloud server is equipped with an instruction stream processing module and a video stream processing module, or the cloud mobile phone is equipped with an instruction stream processing module and a video stream processing module, which are used to realize the connection between the instruction stream access and the video stream access. means of access and control.

In one embodiment, the cloud server sends the first data stream to the terminal device through the first processing module in response to an instruction of the terminal device to access or control the cloud mobile phone. Wherein, the initial access mode of the cloud mobile phone can be set as an instruction stream processing module by default, or as a video stream processing module. Alternatively, according to the user's instruction, it may be determined that the initial access mode of the cloud mobile phone is an instruction stream processing module and a video stream processing module.

Further, if the first processing module is a video stream processing module, then the first data stream is a video stream; if the first processing module is an instruction stream processing module, then the first data stream is a rendering instruction stream. Among them, the first processing module corresponds to the first processing method, that is, the aforementioned implementation of cloud mobile phone image display based on video stream; the second processing module corresponds to the second processing method, that is, the aforementioned implementation of image rendering according to the rendering instruction stream and The image shows the cloud phone implementation.

602: The terminal device displays an image according to the first data stream.

After receiving the first data stream, the terminal device may display images according to the first data stream. Wherein, the first data stream is a video stream or the first data stream is a rendering instruction stream, and the specific processing manners of image display are different.

In one embodiment, if the first data stream is a video stream, the terminal device performs image display according to the first data stream, as shown in FIG. 7 , specifically may include:

1. The terminal device decodes the first data stream into an original video stream.

2. The terminal device sequentially displays each frame of image in the original video stream on the screen.

In another implementation manner, if the first data stream is a rendering instruction stream, the terminal device performs image display according to the first data stream, as shown in FIG. 7 , specifically may include:

1. The terminal device decodes the first data stream into a rendering instruction.

2. The terminal device sequentially performs image rendering according to the rendering instructions to obtain a frame of images.

3. The terminal device sequentially displays each frame of image on the screen.

It should be noted that during the execution of the embodiment of the present application, for example, during the execution of the above-mentioned step 601 or 602, the user may click on the terminal device (such as a physical mobile phone) or initiate a control operation on the cloud mobile phone at any time, such as , click to open the video playback app, click the fast forward icon, click to close the shopping app, etc. Then the cloud mobile phone will respond to the user's operation instruction in real time, and update or generate the display image of the cloud mobile phone in real time, and transmit the encoded video stream to the terminal device. Therefore, the display image of the cloud mobile phone can be displayed on the terminal device in real time, and remote control can be realized. In addition, before or during the above step 601 or 602, the user may not perform other control operations after clicking to open a certain video to play. At this time, the cloud mobile phone may also encode the real-time display image of the video. transmitted to the terminal device. Therefore, the display image of the cloud mobile phone can be displayed on the terminal device in real time, realizing remote playback. The present application has no limitation on specific control operations or instructions.

603: The cloud server determines that the current running state satisfies the switching condition, starts the second processing module, and sends a switching instruction to the terminal device.

Wherein, the first processing module is an instruction stream processing module and the second processing module is a video stream processing module, or, the first processing module is a video stream processing module and the second processing module is an instruction stream processing module.

The switching instruction is used to instruct switching between the video stream processing module and the instruction stream processing module. For example, when the first processing module is an instruction stream processing module, the switching instruction is used to instruct switching to a video stream processing module. When the first processing module is a video stream processing module, the switching instruction is used to instruct switching to the instruction stream processing module.

In one embodiment, the cloud server determines that the current running state meets the switching condition, which may specifically include at least one of the following:

Determine that the current network status parameters meet the preset conditions; or, determine that the resolution of the currently displayed image meets the preset conditions; or, determine that the type of the currently running application meets the preset conditions; or, determine the bandwidth requirements of the currently running application Satisfying the preset condition; or, determining that the data processing capability of the terminal device satisfies the preset condition.

Specifically, the network status parameter includes at least one of packet loss rate, available bandwidth, or delay.

Exemplarily, the cloud server determines that the current network state parameter meets the preset condition may specifically include: the cloud server determines that the current packet loss rate is greater than or equal to a preset threshold (such as the first threshold), then the cloud server determines that the network state is not good, and satisfies The conditions for switching from the command stream processing module to the video stream processing module. In order to avoid problems such as display freeze caused by frequent packet loss in the command stream processing mode, the cloud server can determine to switch from the command stream processing mode to the video stream processing mode.

Or, if the cloud server determines that the current available bandwidth is greater than or equal to a preset threshold (such as the second threshold), then the cloud server determines that the network status is better and satisfies the condition for switching from the video stream processing module to the instruction stream processing module, then the cloud server can Determine to switch from the video stream processing mode to the instruction stream processing mode.

Or, if the cloud server determines that the current transmission delay is greater than or equal to a preset threshold (such as the third threshold), then the cloud server determines that the network status is poor and satisfies the condition for switching from the instruction stream processing module to the video stream processing module. For problems such as display freeze caused by excessive delay in the stream processing mode, the cloud server can determine to switch from the command stream processing mode to the video stream processing mode.

In one embodiment, the cloud server may determine whether the resolution of the currently displayed image satisfies a preset condition as a decision to automatically switch between the video stream processing module and the instruction stream processing module. Exemplarily, the cloud server determines that the resolution of the currently displayed image is lower than or equal to 720P (ie 1080*720), then determines to use the video stream processing module; the resolution of the currently displayed image is higher than or equal to 1080P (ie 1920*1080) , it is determined to use the instruction stream processing module.

In one embodiment, the cloud server can make a decision to automatically switch between the video stream processing module and the instruction stream processing module according to the type of the currently running application program or whether the bandwidth requirement of the application program meets a preset condition.

For example, when the cloud server determines that the type of the currently running application program is a game with strong interaction, or other application programs with strong interaction, it determines to use the instruction stream processing module, and may determine to switch from the video stream processing module to the instruction stream processing module.

For example, the cloud server determines that the currently running application has relatively small bandwidth requirements for instruction streams, and this type of APP has relatively small bandwidth requirements for instruction streams compared to video streams. The instruction stream processing module may determine to switch from the video stream processing module to the instruction stream processing module.

For example, the cloud server determines that the currently running application has relatively small bandwidth requirements for video streams, and this type of APP has relatively small bandwidth requirements for video streams compared to instruction streams. For example, if the APP type is a large-scale 3D game, it is determined to use The video stream processing module may determine to switch from the instruction stream processing module to the video stream processing module.

In one embodiment, the cloud server can also automatically switch between the video stream processing module and the instruction stream processing module according to the detection results such as the data processing capability, GPU processing capability, or device compatibility of the terminal device currently connected to the cloud mobile phone. decision-making judgment. The detection results such as data processing capability, GPU processing capability, or device compatibility of the terminal device may be actively reported by the terminal device to the cloud server, or may be obtained by the cloud service requesting the terminal device.

Exemplarily, when the GPU processing capability of the terminal device is weak, the cloud server may determine to use the video stream processing module; when the GPU processing capability of the terminal device is sufficient, switch to using the instruction stream processing module.

Exemplarily, for heterogeneous non-Android terminal devices, the cloud server may determine to use the video stream processing module. Wherein, the heterogeneous non-Android terminal device means that a non-Android operating system, such as an iOS operating system, is installed on the terminal device. Alternatively, a heterogeneous non-Android terminal device refers to a terminal device that is not a mobile phone type. For example, the terminal device is a notebook computer, that is, the user accesses or controls the cloud mobile phone through the notebook computer. For non-Android terminal devices, the cloud server can determine to use the video stream processing module to complete the realization of the cloud mobile phone.

In one embodiment, the decision strategy for the cloud server to automatically switch between the video stream processing module and the instruction stream processing module may be one of the above-mentioned ones, or may be a combination of multiple decision strategies, or may be based on business Other decision-making strategies for feature customization, etc.

As shown in FIG. 8 , the cloud server may also include a switching strategy control module, which is used to make decisions according to various pre-configured decision-making strategies based on the above-mentioned various detection results reflecting the current running state, and determine the operation The video stream processing module or the operation instruction stream processing module realizes automatic switching. Exemplarily, as shown in FIG. 8 , the switching policy control module of the cloud server may include policies 1 to 8.

Among them, different priorities can be set among multiple decision-making strategies, and the cloud server can judge from high to low according to the priorities of the decision-making strategies until it is determined that any of the switching conditions is met to perform switching. Further, the switching policy control module of the cloud server may include policies 1 to 8 sorted by priority.

In addition, the cloud server's decision-making strategy for automatically switching between the video stream processing module and the instruction stream processing module can also support user-defined. That is, the decision-making policy for automatic switching in the cloud server supports users to add, delete, modify or adjust the priority settings of multiple policies by themselves. Thereby, the flexibility and accuracy of switching the implementation mode of the cloud mobile phone in this application are improved, good interaction with the user is increased, and user experience is improved.

604: The terminal device receives the switching instruction, and starts the second processing module.

After receiving the switching instruction from the cloud server, the terminal device determines that it needs to switch from the first processing module to the second processing module to realize the switching of the cloud mobile phone access mode. Therefore, the terminal device starts the second processing module to prepare data for image display.

In the embodiment of the present application, in order to realize the automatic switching between the first processing method and the second processing method of the cloud mobile phone to realize the switching without the user's perception, that is, the image displayed on the terminal device (that is, the physical mobile phone) is smooth Transition, therefore, the terminal device needs to start the second processing module before the first processing module is closed, and prepare the data for image display, so that after the first processing module on the cloud service is closed, the terminal device can real-time according to the cloud service The second processing module started on above performs image processing and display on the image processing data stream. The specific implementation manner will be described in detail below, and will not be repeated here.

605: The cloud server sends the second data stream to the terminal device.

In one implementation, in order to realize the automatic switching between the first processing method and the second processing method of the cloud mobile phone to realize user-unaware switching, after step 603, that is, after the cloud server starts the second processing module, The cloud server sends the second data stream to the terminal device. Wherein, the second data stream is a video stream or a rendering instruction stream. When the second processing module is a video stream processing module, the second data stream is a video stream; when the second processing module is an instruction stream processing module, the second data stream is a rendering instruction stream.

606: The terminal device performs image display according to the second data stream.

In one implementation, if the first processing module is a video stream processing module and the second processing module is an instruction stream processing module, steps 603-606 may specifically include: the cloud server determines to switch to the instruction stream according to the currently running application program. stream processing module, the cloud server starts the command stream processing module and sends the switching command and the second data stream to the terminal device, as shown in Figure 9, the specific switching process may include:

1. The cloud server starts the instruction stream processing module to complete the data preparation of the graphics rendering state machine.

After the cloud server starts the instruction flow processing module, query the real-time snapshot data of the graphics rendering state machine on the cloud server, construct the graphics rendering state machine recovery message packet based on the above real-time snapshot data, and send the graphics to the instruction flow processing module on the terminal device The rendering state machine restoration message packet enables the instruction stream processing module of the terminal device to complete the reconstruction of the graphics rendering state machine according to the graphics rendering state machine restoration message packet.

2. The cloud server sends a switch command to the terminal device to switch to the command stream processing module.

The terminal device starts the instruction flow processing module according to the switching instruction, the instruction flow processing module completes the initialization, and rebuilds the local graphics rendering state machine according to the graphics rendering state machine restoration message package sent by the cloud service.

3. The command stream processing module on the cloud server takes over the call of the graphics library interface, and continuously sends the second data stream, that is, the rendering command stream, to the terminal device.

After the command stream processing module on the cloud server (that is, the command stream engine server) takes over the graphics library interface call of the Android system, the command stream processing module starts to send the rendering to the command stream processing module on the terminal device (that is, the command stream engine client). Instruction stream, so that the terminal device can implement the instruction stream engine client to render the image according to the rendering instruction stream.

In one embodiment, the cloud server sends the terminal device to switch to the instruction stream processing module may be a first switching instruction, and the first switching instruction may specifically instruct the terminal device to start the instruction stream processing module (that is, the instruction stream engine client).

After the cloud server sends the first switching instruction to the terminal device, the instruction stream engine server sends several frame rendering instructions to the instruction stream engine client, for example, it may be pre-configured to send 10 frame rendering instructions.

At this time, the command stream engine client can render the image according to these frame rendering commands, and perform frame synchronization with the video stream engine client, that is, the command stream engine client and the video stream engine client can generate the same image at the same time. frame image.

After the command stream engine server sends several frame rendering commands to the command stream engine client, the cloud server sends a second switching instruction to the terminal device, and the second switching instruction completes the switching from the video stream processing module to the command stream processing module. Or, it can also be that, after the command stream engine client can render images normally and synchronize with the frame of the video stream engine client, the terminal device sends a switching completion command to the cloud service, which has instructed the cloud service to shut down the video stream engine server, Complete the process of switching from video stream processing to instruction stream processing.

In the above embodiments of the present application, the first switching instruction and the second switching instruction sent by the cloud server to the terminal device may be referred to as a switching instruction sequence. This application does not specifically limit the timing of sending the switching instruction sequence. In addition, the switching instruction sequence does not It is not limited to the first switching instruction and the second switching instruction, and may also include the third switching instruction, etc. The above-mentioned embodiment is only an exemplary description, and is used to realize smooth switching of the cloud mobile phone implementation mode, which is not specifically limited in this application.

In one implementation, if the first processing module is a video stream processing module, and the switching instruction is used to indicate switching from the video stream processing module to the instruction stream processing module, then the terminal device receives the second data stream from the cloud server, And perform image display according to the second data stream, as shown in FIG. 9 and FIG. 10 , specifically may include:

1. The terminal device starts the instruction stream processing module and rebuilds the graphics rendering state machine.

At this time, the video stream processing module of the terminal device is running, and the video stream processing module is in the stage of continuous video decoding and playback. For example, the display module decodes the video stream to obtain the original image, image 1 is displayed on the screen, image 2, image 3, etc. are displayed on the screen sequentially.

The terminal device starts the command stream processing module and initializes it according to the received switching command; further, the command stream engine client completes the reconstruction of the graphics rendering state machine.

2. The terminal device receives the second data stream from the cloud server, and decodes it to obtain a rendering instruction.

3. The terminal device renders according to the rendering instruction to obtain the original display image.

Specifically, the GPU on the terminal device renders the image according to the rendering instruction.

4. The video stream drawing surface on the terminal device is switched to the instruction stream drawing surface.

After receiving the switching instruction, the terminal device firstly completes the frame synchronization, that is, keeps the instruction stream processing module and the video stream processing module to generate the same frame of image at the same time.

After frame synchronization is implemented on the terminal device side, the video stream drawing surface on the terminal device is switched to the instruction stream drawing surface.

Specifically, when the display screen module on the terminal device receives the original image generated by the instruction stream processing module (that is, the image a is displayed on the upper screen), the display content of the display screen module can be switched to that of the instruction stream processing module when the next frame is displayed. Render output. For example, the display module renders the original image according to the instruction stream, the image b is displayed on the screen, the image c is displayed on the screen, etc. sequentially.

5. The terminal device closes the video stream processing module.

At this time, the video stream processing module on the terminal device is normally closed, and the instruction stream processing module continues to perform image rendering, and generates a real-time display image until the next switching.

In an implementation manner, after the video stream drawing surface on the terminal device is switched to the instruction stream drawing surface, the terminal device may also send a switching completion instruction to the cloud server, which is used to instruct the cloud server to shut down the video stream processing module.

Therefore, after the cloud server sends the second data stream to the terminal device, it can directly close the first processing module. Alternatively, the cloud server may also shut down the video stream processing module after receiving the switching completion instruction from the terminal device.

Through the above implementation, when the cloud server interacts with the terminal device to realize the remote control of the cloud mobile phone, it is not limited to a specific scenario and is not limited to one implementation method, and the cloud server can determine the execution according to the current operating status. The video stream processing method is still the command stream processing method, and can realize automatic switching without user perception, without affecting the smoothness and real-time performance of the image displayed by the cloud mobile phone, and improving the user experience of the cloud mobile phone.

Based on the aforementioned embodiments, the present application also provides a device for implementing a cloud mobile phone, which is used for storing and running the cloud mobile phone, and the device includes a video stream processing module and an instruction stream processing module. As shown in FIG. 11 , the apparatus 1100 includes a sending module 1101 and a processing module 1102 .

Wherein, the sending module 1101 is used to send the first data stream to the terminal device. If the first processing module is a video stream processing module, the first data stream is a video stream; if the first processing module is an instruction stream processing module, the first The data stream is the stream of rendering instructions.

The processing module 1102 is configured to start a second processing module and send a switching instruction to the terminal device if it is determined that the current running state satisfies the switching condition. Wherein, the switching instruction is used to instruct the terminal device to switch from the first processing module to the second processing module, if the first processing module is a video stream processing module, then the second processing module is an instruction stream processing module, and if the first processing module is an instruction stream processing module, the second processing module is a video stream processing module.

The sending module 1101 is further configured to send a second data stream to the terminal device, where the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.

The processing module 1102 is also used to shut down the first processing module.

In a possible design manner, determining that the current operating state meets the switching condition specifically includes at least one of the following: determining that the current network state parameter meets the preset condition, or determining that the resolution of the currently displayed image meets the preset condition, or, Determine that the type of the currently running application meets the preset condition, or determine that the bandwidth requirement of the currently running application meets the preset condition, or determine that the data processing capability of the terminal device meets the preset condition, wherein the network status parameters include loss At least one of packet rate, available bandwidth, or delay.

In a possible design manner, if the first processing module is a video stream processing module, the processing module 1102 is specifically configured to start the instruction stream processing module and complete the data preparation of the graphics rendering state machine.

The sending module 1101 is specifically configured to send a switching instruction indicating to switch to an instruction stream processing module of the terminal device to the terminal device; and send a rendering instruction stream to the terminal device.

In a possible design manner, the apparatus 1100 further includes a receiving module, configured to receive a handover completion instruction from the terminal device, and shut down the video stream processing module.

In a possible design, the processing module 1102 is specifically configured to: if it is determined that the current packet loss rate is greater than or equal to the first threshold, determine that the condition for switching from the instruction stream processing module to the video stream processing module is met; start the video stream The processing module sends an instruction to switch to the video stream processing module to the terminal device.

In a possible design manner, the processing module 1102 is specifically configured to: if it is determined that the current available bandwidth is greater than or equal to the second threshold, determine that the condition for switching from the video stream processing module to the instruction stream processing module is met; start instruction stream processing module and send an instruction to the terminal device to switch to the instruction stream processing module.

In a possible design manner, the processing module 1102 is specifically configured to: if it is determined that the resolution of the currently displayed image is lower than or equal to 1080*720, determine that the condition for switching from the instruction stream processing module to the video stream processing module is met; start The video stream processing module sends an instruction to switch to the video stream processing module to the terminal device; or, if it is determined that the resolution of the currently displayed image is higher than or equal to 1920*1080, it is determined that the video stream processing module is switched to the instruction stream processing module conditions; start the instruction stream processing module and send an instruction to switch to the instruction stream processing module to the terminal device.

In a possible design manner, the processing module 1102 is specifically configured to: determine that the type of the currently running application program is a game of strong interaction type or other strong interaction type of application program, or determine that the type of the currently running application program is If the bandwidth demand is small, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met; the instruction stream processing module is started and an instruction to switch to the instruction stream processing module is sent to the terminal device.

In a possible design manner, the processing module 1102 is specifically configured to: if it is determined that the currently running application program is a game of weak interaction type or other weak interaction type application program, or determine that the currently running application program has an The bandwidth requirement is small, or when the processing capability of the graphics processor GPU on the terminal device is lower than the preset condition, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met; the video stream processing module is started and sent to the terminal device Send an instruction to switch to the video stream processing module.

Based on the foregoing embodiments, the present application also provides an implementation device for a cloud mobile phone, which is used for remote control of the cloud mobile phone, and can be used to realize the actions performed by the terminal device or the physical mobile phone in the foregoing embodiments. As shown in FIG. 12 , the apparatus 1200 includes a receiving module 1201 and a processing module 1202 .

The receiving module 1201 is configured to receive a first data stream from a cloud server, where the first data stream is a video stream or a rendering instruction stream.

The processing module 1202 is used for performing image display according to the first data stream.

The receiving module 1201 is also used to receive a switching instruction from the cloud server, and the switching instruction is used to instruct the device to switch from the first processing module to the second processing module; wherein, the first processing module is an instruction stream processing module and the second processing module is a video processing module. A stream processing module, or, the first processing module is a video stream processing module and the second processing module is an instruction stream processing module.

The receiving module 1201 is also configured to receive the second data stream from the cloud server.

The processing module 1202 is further configured to display images according to the second data stream, where the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.

In a possible design manner, if the switching instruction is used to indicate switching from the video stream processing module to the instruction stream processing module, the processing module 1202 is specifically configured to: start the instruction stream processing module, and rebuild the graphics rendering state machine.

The receiving module 1201 is specifically configured to receive the second data stream from the cloud server, and decode it to obtain a rendering instruction.

The processing module 1202 is specifically configured to: render according to the rendering instruction to obtain the original display image; switch from the video stream drawing surface to the instruction stream drawing surface; close the video stream processing module.

In a possible design manner, the device 1200 further includes a sending module 1203, and the sending module 1203 is configured to send a switching completion instruction to the cloud server, and is used to instruct the cloud server to shut down the video stream processing module.

It can be understood that in the embodiments of the present application, the same step or steps or messages having the same function can be used for reference in different embodiments.

It should be noted that, for the specific execution process and embodiments of the above-mentioned device or electronic device, refer to the steps and related descriptions executed by the cloud service or terminal device in the above-mentioned method embodiments, and the technical problems solved and the technical effects brought about are also Reference may be made to the content described in the foregoing embodiments, and details are not repeated here.

In this embodiment, the electronic device may be presented in the form of dividing various functional modules in an integrated manner. A "module" here may refer to a specific circuit, a processor and memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above functions. In a simple embodiment, those skilled in the art can imagine that the electronic device can take the form shown in FIG. 4 or FIG. 7 in the foregoing.

In some implementations, the processor 401 in FIG. 4 can call the computer-executed instructions stored in the memory 404, so that the electronic device 400 can execute the cloud service or the operation performed by the terminal device in the above-mentioned various method embodiments, and implement the present application. Each of the above possible implementation methods.

The electronic device in each of the above embodiments can completely correspond to the cloud service or terminal device in the method embodiment, and the corresponding steps are performed by the corresponding modules or units. For example, when the electronic device is implemented in the form of a chip, the above is used to The sending module or receiving module is an interface circuit of the chip, which is used to send signals to other devices.

In an exemplary embodiment, there is also provided a computer-readable storage medium including instructions, or a computer program product. The above-mentioned instructions can be executed by the processor 401 of the electronic device 400 to implement the methods of the above-mentioned embodiments. Therefore, the technical effects that can be obtained can refer to the above-mentioned method embodiments, and will not be repeated here.

The present application also provides a computer program product, the computer program product includes instructions, and when the instructions are executed, the computer can respectively perform operations corresponding to the cloud service or the terminal device in the above method.

The embodiment of the present application also provides a system chip. The system chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin, or a circuit. The processing unit may execute computer instructions, so that the communication device used by the chip executes the operations performed by the cloud service or the terminal device in the method provided by the above embodiments of the present application.

Optionally, any electronic device provided in the foregoing embodiments of the present application may include the system chip.

Optionally, the computer instructions are stored in a storage unit.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. A computer can be a general purpose computer, special purpose computer, computer network, or other programmable device.

Other embodiments of the present application will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the application, these modifications, uses or adaptations follow the general principles of the application and include common knowledge or conventional technical means in the technical field not disclosed in the application .

Finally, it should be noted that: the above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered by this application. within the scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (31)

1. A method for implementing a cloud mobile phone, characterized in that it is applied to a cloud server, the cloud server is used to store and run the cloud mobile phone, and the cloud server is equipped with an instruction stream processing module and a video stream processing module, the method include:

   Send the first data stream to the terminal device through the first processing module, if the first processing module is the video stream processing module, then the first data stream is a video stream, if the first processing module is In the instruction stream processing module, the first data stream is a rendering instruction stream;

   If it is determined that the current operating state satisfies the switching condition, start the second processing module and send a switching instruction to the terminal device, wherein the switching instruction is used to instruct the terminal device to switch from the first processing module to the second processing module Two processing modules, if the first processing module is the video stream processing module, then the second processing module is an instruction stream processing module, if the first processing module is an instruction stream processing module, then the second The processing module is the video stream processing module;

   Sending a second data stream to the terminal device through the second processing module, and closing the first processing module, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is the same as The first data streams are different.
2. The method according to claim 1, wherein the determining that the current operating state satisfies the switching condition specifically includes at least one of the following:

   Determine that the current network status parameters meet the preset conditions, or determine that the resolution of the currently displayed image meets the preset conditions, or determine that the type of the currently running application meets the preset conditions, or determine the bandwidth requirements of the currently running application A preset condition is met, or it is determined that the data processing capability of the terminal device satisfies a preset condition, wherein the network status parameter includes at least one of packet loss rate, available bandwidth, or delay.
3. The method according to claim 1 or 2, wherein if the first processing module is a video stream processing module, the second processing module is started and the switching instruction and the second data are sent to the terminal device flow, including:

   Start the instruction stream processing module to complete the data preparation of the graphics rendering state machine;

   sending to the terminal device a switching instruction indicating to switch to the instruction stream processing module of the terminal device;

   Send a rendering instruction stream to the terminal device.
4. The method according to claim 3, wherein the closing the first processing module specifically comprises:

   receiving a switching completion instruction from the terminal device, and closing the video stream processing module.
5. The method according to any one of claims 1-4, wherein the determining that the current operating state satisfies the switching condition, then starting the second processing module and sending a switching instruction to the terminal device, specifically includes:

   If it is determined that the current packet loss rate is greater than or equal to the first threshold, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

   Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device.
6. The method according to any one of claims 1-5, wherein the determining that the current operating state satisfies the switching condition, then starting the second processing module and sending a switching instruction to the terminal device, specifically includes:

   If it is determined that the current available bandwidth is greater than or equal to the second threshold, then it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met;

   Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
7. The method according to any one of claims 1-6, wherein the determining that the current operating state satisfies the switching condition, then starting the second processing module and sending a switching instruction to the terminal device, specifically includes:

   If it is determined that the resolution of the currently displayed image is lower than or equal to 1080*720, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

   Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device;

   Or, if it is determined that the resolution of the currently displayed image is higher than or equal to 1920*1080, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met;

   Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
8. The method according to any one of claims 1-7, wherein the determining that the current operating state satisfies the switching condition, then starting the second processing module and sending a switching instruction to the terminal device, specifically includes:

   Determine that the type of the currently running application is a game with strong interaction or other strong interaction type of application, or determine that the currently running application has a small bandwidth requirement for the instruction stream, then determine that the video stream processing module meets the requirements A condition for switching to the instruction stream processing module;

   Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
9. The method according to any one of claims 1-8, wherein the determining that the current operating state satisfies the switching condition, then starting the second processing module and sending a switching instruction to the terminal device, specifically includes:

   If it is determined that the currently running application program is a weak interaction game or other weak interaction type application program, or it is determined that the currently running application program has a small bandwidth requirement for video streams, or the graphics processing unit GPU on the terminal device When the capability is lower than the preset condition, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

   Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device.
10. A method for implementing a cloud mobile phone, characterized in that it is applied to a terminal device, and the terminal device is used to remotely control the cloud mobile phone, and the method includes:

    receiving a first data stream from the cloud server, where the first data stream is a video stream or a rendering instruction stream;

    performing image display according to the first data stream;

    receiving a switching instruction from the cloud server, the switching instruction is used to instruct the terminal device to switch from the first processing module to the second processing module; wherein, the first processing module is an instruction stream processing module and the second processing module The second processing module is a video stream processing module, or, the first processing module is a video stream processing module and the second processing module is an instruction stream processing module;

    receiving a second data stream from the cloud server, and performing image display according to the second data stream, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is identical to the first A data flow is different.
11. The method according to claim 10, wherein if the switching instruction is used to indicate switching from the video stream processing module to the instruction stream processing module, the receiving the second data stream from the cloud server , and perform image display according to the second data stream, specifically including:

    Start the instruction stream processing module to rebuild the graphics rendering state machine;

    receiving the second data stream from the cloud server, and decoding to obtain rendering instructions;

    performing rendering according to the rendering instruction to obtain an original display image;

    switch from the video stream drawing surface to the instruction stream drawing surface;

    Close the video stream processing module.
12. The method according to claim 11, wherein after switching from the video stream drawing surface to the instruction stream drawing surface, the method further comprises:

    Sending a switching completion instruction to the cloud server, for instructing the cloud server to shut down the video stream processing module.
13. A device for implementing a cloud mobile phone, wherein the device is used to store and run the cloud mobile phone, the device includes a video stream processing module and an instruction stream processing module, and the device includes a sending module and a processing module:

    The sending module is used to send a first data stream to the terminal device. If the first processing module is a video stream processing module, the first data stream is a video stream. If the first processing module is an instruction A stream processing module, the first data stream is a rendering instruction stream;

    The processing module is configured to start the second processing module and send a switching instruction to the terminal device if it is determined that the current operating state satisfies the switching condition, wherein the switching instruction is used to instruct the terminal device to switch from the first The processing module switches to the second processing module, if the first processing module is the video stream processing module, the second processing module is an instruction stream processing module, and if the first processing module is an instruction stream processing module, Then the second processing module is the video stream processing module;

    The sending module is further configured to send a second data stream to the terminal device, where the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream;

    The processing module is also used to shut down the first processing module.
14. The device according to claim 13, wherein the determining that the current operating state satisfies the switching condition specifically includes at least one of the following:

    Determine that the current network status parameters meet the preset conditions, or determine that the resolution of the currently displayed image meets the preset conditions, or determine that the type of the currently running application meets the preset conditions, or determine the bandwidth requirements of the currently running application A preset condition is met, or it is determined that the data processing capability of the terminal device satisfies a preset condition, wherein the network status parameter includes at least one of packet loss rate, available bandwidth, or delay.
15. The device according to claim 13 or 14, wherein if the first processing module is a video stream processing module, the processing module is specifically used to start the instruction stream processing module to complete the data preparation of the graphics rendering state machine ;

    The sending module is specifically configured to send to the terminal device a switching instruction indicating to switch to the instruction stream processing module of the terminal device; and send a rendering instruction stream to the terminal device.
16. The method according to claim 15, wherein the device further comprises a receiving module, the receiving module is configured to receive a handover completion instruction from the terminal device;

    Then the processing module is used to close the video stream processing module.
17. The device according to any one of claims 13-16, wherein the processing module is specifically configured to:

    If it is determined that the current packet loss rate is greater than or equal to the first threshold, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

    Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device.
18. The device according to any one of claims 13-17, wherein the processing module is specifically configured to:

    If it is determined that the current available bandwidth is greater than or equal to the second threshold, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met;

    Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
19. The device according to any one of claims 13-18, wherein the processing module is specifically configured to:

    If it is determined that the resolution of the currently displayed image is lower than or equal to 1080*720, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

    Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device;

    Or, if it is determined that the resolution of the currently displayed image is higher than or equal to 1920*1080, it is determined that the condition for switching from the video stream processing module to the instruction stream processing module is met;

    Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
20. The device according to any one of claims 13-19, wherein the processing module is specifically configured to:

    Determine that the type of the currently running application is a game with strong interaction or other strong interaction type of application, or determine that the currently running application has a small bandwidth requirement for the instruction stream, then determine that the video stream processing module meets the requirements A condition for switching to the instruction stream processing module;

    Starting the instruction stream processing module and sending an instruction to switch to the instruction stream processing module to the terminal device.
21. The device according to any one of claims 13-20, wherein the processing module is specifically configured to:

    If it is determined that the currently running application program is a weak interaction game or other weak interaction type application program, or it is determined that the currently running application program has a small bandwidth requirement for video streams, or the graphics processing unit GPU on the terminal device When the capability is lower than the preset condition, it is determined that the condition for switching from the instruction stream processing module to the video stream processing module is met;

    Start the video stream processing module and send an instruction to switch to the video stream processing module to the terminal device.
22. A device for implementing a cloud mobile phone, characterized in that the device is used to remotely control the cloud mobile phone, and the device includes a receiving module and a processing module,

    The receiving module is configured to receive a first data stream from the cloud server, where the first data stream is a video stream or a rendering instruction stream;

    The processing module is configured to perform image display according to the first data stream;

    The receiving module is also used to receive a switching instruction from the cloud server, the switching instruction is used to instruct the device to switch from the first processing module to the second processing module; wherein, the first processing module is an instruction stream A processing module and the second processing module is a video stream processing module, or, the first processing module is a video stream processing module and the second processing module is an instruction stream processing module;

    The receiving module is also used to receive the second data stream from the cloud server;

    The processing module is further configured to perform image display according to the second data stream, wherein the second data stream is a rendering instruction stream or a video stream, and the second data stream is different from the first data stream.
23. The device according to claim 22, wherein if the switching instruction is used to instruct switching from the video stream processing module to the instruction stream processing module, the processing module is specifically configured to:

    Start the instruction stream processing module to rebuild the graphics rendering state machine;

    The receiving module is specifically configured to receive the second data stream from the cloud server, and decode it to obtain a rendering instruction;

    The processing module is specifically used for: performing rendering according to the rendering instruction to obtain an original display image;

    switch from the video stream drawing surface to the instruction stream drawing surface;

    Close the video stream processing module.
24. The device according to claim 23, wherein the device further comprises a sending module, the sending module is configured to send a switching completion instruction to the cloud server, and is used to instruct the cloud server to close the video stream processing module.
25. An electronic device, characterized in that the electronic device includes: a memory and one or more processors; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes A computer instruction, when the processor executes the computer instruction, the electronic device is configured to execute the method according to any one of claims 1-9.
26. An electronic device, characterized in that the electronic device includes: a memory and one or more processors; the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes Computer instructions, when the processor executes the computer instructions, the electronic device is configured to execute the method according to any one of claims 10-12.
27. A computer-readable storage medium, characterized in that instructions are stored in the readable storage medium, and when the readable storage medium runs on an electronic device, the electronic device executes any of claims 1-9. one of the methods described.
28. A computer-readable storage medium, characterized in that instructions are stored in the readable storage medium, and when the readable storage medium runs on an electronic device, the electronic device executes any one of claims 10-12. one of the methods described.
29. A computer program product, characterized in that, when the computer program product is run on a computer, the computer is made to execute the method according to any one of claims 1-9.
30. A computer program product, characterized in that, when the computer program product is run on a computer, the computer is made to execute the method according to any one of claims 10-12.
31. A communication system, characterized in that the communication system comprises the device according to any one of claims 13-21 and the device according to any one of claims 22-24.

Images