WO2022048677A1

WO2022048677A1 - Vr application design method and system based on cloud mobile phone

Info

Publication number: WO2022048677A1
Application number: PCT/CN2021/116834
Authority: WO
Inventors: 聂颂; 李东升
Original assignee: 华为云计算技术有限公司
Priority date: 2020-09-07
Filing date: 2021-09-07
Publication date: 2022-03-10
Also published as: CN114157907A

Abstract

Provided are a design method and design system based on a cloud mobile phone. The design system comprises: a cloud mobile phone and at least one client, wherein the at least one client respectively runs on a terminal and is used for receiving an interface data stream sent by the cloud mobile phone, generating a design interface according to the interface data stream, and displaying the design interface on the terminal where each client is located; the cloud mobile phone is arranged on a network side and is used for receiving at least one piece of interaction information sent by one or any combination of the at least one client, running a design application according to the at least one piece of interaction information, and rendering a video stream according to the running process of the design application; and the at least one client is further used for receiving the video stream, and displaying a design picture on the design interface according to the video stream. A video stream is rendered by a cloud mobile phone on a network side, such that collaborative online design by a large number of users is supported, thereby meeting service requirements.

Description

VR application design method and system based on cloud mobile phone

technical field

The present application relates to the field of cloud computing technologies, and in particular, to a design method, system, device, and computer-readable storage medium based on a cloud mobile phone.

Background technique

When carrying out industrial design, such as architectural design and decoration design, designers usually complete the design through computers. Among them, the human-computer interaction in the design process is mainly based on the click and drag of the mouse and the keyboard. With the continuous development of virtual reality (VR) technology, new human-computer interaction methods have emerged.

VR is also called virtual environment. VR generates a three-dimensional virtual world through computational simulation, and provides users with a simulation of vision and other senses. When users perform certain actions, they can adjust the virtual world through calculation, so that users feel as if they are in the real world. When VR is applied to industrial design, VR equipment can generate a virtual design environment, and designers can interact with both hands in this virtual design environment. In addition, designers can also perform human-computer interaction with other personnel in the virtual design environment to complete the design collaboratively, which can greatly improve design efficiency and design quality.

However, the above-mentioned virtual design environment supports a limited number of people participating in the design, usually 8 to 10 people, which is difficult to meet business needs.

SUMMARY OF THE INVENTION

The present application provides a design method and design system based on a cloud mobile phone. By using a cloud mobile phone set on the network side to render video streams in the cloud, the advantages of the cloud mobile phone's computing power that can be elastically scaled can be fully utilized. On the one hand, support for A large number of users collaborate on the design online. On the other hand, offloading work such as rendering that consumes a lot of computing power to the cloud phone can reduce the hardware specification requirements of the terminal, save costs, and have high availability.

In a first aspect, the present application provides a design system. The design system includes: a cloud mobile phone and at least one client. At least one client runs on a terminal respectively, and is used to receive the interface data stream sent by the cloud mobile phone, generate a design interface according to the interface data stream, and display it on the respective terminal. The design interface, cloud mobile phone, is used to receive at least one interaction information sent by one or any combination of at least one client, and the at least one interaction information is specifically from one of the at least one client or any combination of users through their respective The design operation triggered by the design interface displayed on the operating terminal, and then the cloud mobile phone runs the design application according to the interactive information, and renders the video stream according to the design application running process. The design screen is displayed.

Among them, the cloud mobile phone is set on the network side, and the cloud mobile phone on the network side renders the video stream, which can give full play to the advantages of the computing power of the cloud mobile phone and can achieve on-demand rendering. On the other hand, the rendering work that consumes a lot of computing power is offloaded to the cloud phone, which can reduce the hardware specification requirements of the terminal, save costs, and have high availability.

In some possible implementations, the rendering resource refers to a resource for rendering video, for example, the rendering resource may be a graphics processing unit (graphics processing unit, GPU). Cloud phones can be implemented through containers in physical servers. The physical server includes software resources and hardware resources. The hardware resources of different physical servers can be virtualized to form a physical resource pool. For example, rendering resources such as GPUs in different physical servers can be virtualized to obtain a rendering resource pool that supports dynamic allocation, flexible scheduling, and cross-domain sharing.

The cloud phone can schedule the rendering resources in the rendering resource pool for video stream rendering. Specifically, the cloud phone can obtain a target number of rendering resources from the rendering resource pool according to the running process of the design application, and then the cloud phone can use the rendering resources to render the video stream according to the running process of the design application. In this way, cloud mobile phones can accurately provide rendering resources for rendering based on the consumption of rendering capabilities, and meet various performance requirements.

In some possible implementations, the rendering capabilities of different types of rendering resources may be different. For example, for the same rendering object, the rendering capabilities of the GPU of the x86 architecture and the GPU of the advanced RISC machine (ARM) architecture are different. To this end, the cloud phone can also determine the type and quantity of rendering resources required for rendering according to the running process of the design application, and then obtain the rendering resources of the target type and the target quantity from the rendering resource pool. In this way, the rendering capabilities of different types of rendering resources can be fully utilized to meet the rendering requirements of different business scenarios.

In some possible implementations, the number of rendering resources scheduled by the cloud phone is related to the number of terminals participating in the collaborative design. To this end, the cloud phone can obtain a target number of rendering resources from the rendering resource pool according to the design application running process and the number of terminals. In this way, it is possible to support different numbers of users for collaborative design through the elastic scaling of rendering resources.

Further, the terminals participating in the collaborative design may include different types of terminals. The cloud phone can render video streams separately for different types of terminals. Therefore, the cloud phone can also obtain a target number of rendering resources from the rendering resource pool according to the design application running process and the type and quantity of terminals, so as to perform rendering based on the target number of rendering resources. In this way, rendering resources are accurately provided to meet the rendering requirements of different business scenarios.

In some possible implementations, considering that the cloud VR service consumes huge bandwidth resources under the X86 architecture, a single user usually needs 50Mbps to 100Mbps of bandwidth resources. Through device-cloud collaborative rendering, bandwidth is reduced, thereby reducing costs and facilitating large-scale commercial use.

Specifically, the cloud phone can generate a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process, wherein the cloud rendering instruction stream includes at least one cloud rendering instruction, the terminal rendering instruction stream includes at least one terminal rendering instruction, and the cloud rendering instruction indicates Rendering is performed in the cloud, and client-side rendering instructions are rendered on the client-side. The cloud mobile phone can render the video stream according to the cloud rendering instruction stream to obtain the first rendered video stream. Then, the cloud phone sends the first rendering video stream and the end rendering instruction stream to at least one client. The end-rendering instruction stream is used to instruct at least one client to render a video stream according to the end-rendering instruction stream to obtain a second rendered video stream. Correspondingly, at least one client can be specifically configured to combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream, and then display the design image on the design interface according to the third rendered video stream. This reduces bandwidth, which in turn reduces costs.

In some possible implementations, considering that end-side rendering capabilities are usually limited. Based on this, the cloud phone can generate a cloud rendering instruction stream according to the information that the rendering calculation amount is greater than the calculation amount threshold during the design application running process, and generate the end rendering instruction stream according to the information that the rendering calculation amount is less than or equal to the calculation amount threshold value during the design application running process. In this way, while reducing the bandwidth, it also ensures that the end-side can render the video stream normally, providing users with a better experience.

In some possible implementations, the proportion of background information is usually low, and generally requires more rendering resources. Therefore, the cloud mobile phone can generate a cloud rendering instruction stream according to the background information during the running process of the design application, and according to the design application The action information in the running process generates the end rendering instruction stream. This can effectively reduce bandwidth, thereby reducing costs.

In some possible implementations, the terminal includes a virtual reality terminal or a flat terminal. Among them, virtual reality terminals include PC VR terminals and mobile VR terminals, such as VR helmets, VR glasses and other equipment. Flat terminals include devices such as smartphones and tablet computers. The system supports multi-type devices, especially common plane terminals, to participate in collaborative design, meeting the needs of different business scenarios. Moreover, users do not need to configure special equipment, which reduces the cost.

In some possible implementations, when the terminal includes the virtual reality terminal, the interaction information includes at least one of motion information and location information. Specifically, the user can trigger the collaborative design through gestures or other physical actions. Accordingly, the virtual reality terminal can obtain the motion information and position information, and provide the above motion information and position information to the cloud phone for rendering to obtain a video stream, thereby realizing Support virtual reality terminals to participate in collaborative design.

In some possible implementations, when the terminal includes a flat terminal, the interaction information includes an operation instruction. Specifically, the user can perform design operations through the flat terminal, for example, by touching or stylus touch, and the flat terminal receives the design operation and provides corresponding operation instructions to the cloud mobile phone, so that the cloud mobile phone can perform the operation according to the operation instructions. The video stream is obtained by rendering, so as to support the participation of flat terminals in collaborative design.

In a second aspect, the present application provides a design method based on a cloud phone. The method is applied to a design system, the design system includes a cloud phone and at least one client, wherein at least one client runs on a terminal respectively, and the cloud phone is set on the network side, and the method includes:

The at least one client receives the interface data stream sent by the cloud phone, generates a design interface according to the interface data stream, and displays the design interface on the at least one terminal;

The cloud phone receives at least one interaction information sent by one or any combination of the at least one client, runs a design application according to the at least one interaction information, and renders a video stream according to the design application running process; wherein the interaction The information comes from a design operation triggered by one of the at least one client or any combination of users through the design interface displayed on the respective terminal being operated;

The at least one client receives the video stream, and displays a design picture on the design interface according to the video stream.

In some possible implementations, the cloud phone renders the video stream according to the design application running process, including:

The cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process;

The cloud phone uses the rendering resource to render the video stream according to the design application running process.

In some possible implementations, the cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process, including:

The cloud phone determines the type and quantity of rendering resources required for rendering according to the design application running process;

The cloud phone acquires the rendering resources of the target type and the target quantity from the rendering resource pool.

The cloud phone acquires a target number of rendering resources from the rendering resource pool according to the running process of the design application and the number of the terminals.

The cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process;

The cloud mobile phone renders a video stream according to the cloud rendering instruction stream to obtain a first rendered video stream;

The at least one client receives a video stream, and displays a design screen on the design interface according to the video stream, including:

The at least one client receives the first rendered video stream and the end rendering instruction stream;

The at least one client renders a video stream according to the end rendering instruction stream to obtain a second rendered video stream;

The at least one client combines the first rendered video stream and the second rendered video stream to obtain a third rendered video stream;

A design picture is displayed on the design interface according to the third rendered video stream.

In some possible implementations, the cloud phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process, including:

The cloud rendering instruction stream is generated according to the information that the rendering calculation amount is greater than the calculation amount threshold during the running process of the design application, and the end rendering instruction stream is generated according to the information that the rendering calculation amount is less than or equal to the calculation amount threshold value during the running process of the design application.

The cloud rendering instruction stream is generated according to the background information during the running process of the design application, and the end rendering instruction stream is generated according to the action information during the running process of the design application.

In some possible implementations, the terminal includes a virtual reality terminal or a flat terminal.

In some possible implementations, when the terminal includes the virtual reality terminal, the interaction information includes at least one of motion information and location information.

In some possible implementations, when the terminal includes a flat terminal, the interaction information includes an operation instruction.

In a third aspect, the present application provides a design method based on a cloud phone. The method is applied to the first terminal. Specifically, the first terminal sends a design application start request, and the design application start request is used to instruct to start the design application on the cloud mobile phone, the first terminal receives the interface data stream sent by the cloud mobile phone, and generates the above-mentioned design reference according to the interface data stream. Design the interface, display the design interface, then the first terminal sends the first interaction information to the cloud mobile phone, receives the video stream obtained by the cloud mobile phone running the design application according to the first interaction information, and rendering the design application according to the running process of the design application. The design interface displays the design screen.

In some possible implementation manners, the first terminal receives the first rendered video stream rendered by the cloud phone according to the cloud rendering instruction stream, and receives the end rendering instruction stream, wherein the cloud rendering instruction stream and the end rendering instruction stream are based on the cloud phone rendering instruction stream. The first interactive information is obtained by running the design application and rendered according to the running process of the design application. Then, the first terminal renders the video stream according to the end rendering instruction stream, obtains the second rendered video stream, and combines the first rendered video stream and the second rendered video stream to obtain the third rendered video stream. Correspondingly, the first terminal displays the design picture on the design interface according to the third rendered video stream.

In a fourth aspect, the present application provides a design method based on a cloud phone. The method is applied to the first terminal. Specifically, the second terminal receives the interface data stream sent by the cloud mobile phone, generates a design interface referenced by the design according to the interface data stream, and displays the design interface, and then the second terminal sends the second interaction information to the cloud mobile phone, and receives the cloud mobile phone according to the second interaction information. The interactive information runs the design application, and renders the obtained video stream according to the running process of the design application, and displays the design screen on the design interface according to the video stream.

In some possible implementations, the second terminal receives the first rendered video stream rendered by the cloud mobile phone according to the cloud rendering instruction stream, and receives the end rendering instruction stream, wherein the cloud rendering instruction stream and the end rendering instruction stream are based on the cloud phone rendering instruction stream. The second interactive information runs the design application, and is rendered according to the design application running process. Next, the second terminal renders the video stream according to the end rendering instruction stream, obtains the second rendered video stream, and combines the first rendered video stream and the second rendered video stream to obtain the third rendered video stream. Correspondingly, the second terminal displays the design picture on the design interface according to the third rendered video stream.

In a fifth aspect, the present application provides a design method based on a cloud phone. The method is applied to cloud mobile phones. Specifically, the cloud phone sends an interface data stream to the first terminal and the second terminal, and the interface data stream is used to generate a design interface, and then the cloud phone receives the first interaction information sent by the first terminal and the second interaction information sent by the second terminal. , wherein the first interaction information comes from the design operation triggered by the user of the first terminal through the design interface, the second interaction information comes from the design operation triggered by the user of the second terminal through the design interface, and then the cloud mobile phone according to the first interaction information and The second interactive information runs the design application, renders the video stream according to the running process of the design application, and then the cloud mobile phone sends the video stream to the first terminal and the second terminal.

The cloud phone acquires a target number of rendering resources from the rendering resource pool according to the design application running process and the number of the terminals.

The cloud mobile phone sends the video stream to the first terminal and the second terminal, including:

sending, by the cloud phone, the first rendered video stream to the first terminal and the second terminal;

The method also includes:

The cloud mobile phone sends the end rendering instruction stream to the first terminal and the second terminal, and the end rendering instruction stream is used to instruct the first terminal and the second terminal to render the video stream, and obtain the second Render the video stream.

In a sixth aspect, the present application further provides a first client. The first client includes various modules for executing the method in the third aspect or any possible implementation manner of the third aspect.

In a seventh aspect, the present application further provides a second client. The second client includes various modules for executing the method in the fourth aspect or any possible implementation manner of the fourth aspect.

In an eighth aspect, the present application further provides a cloud mobile phone. The cloud phone includes various modules for executing the method in the fifth aspect or any possible implementation manner of the fifth aspect.

The sixth aspect or any implementation manner of the sixth aspect is implemented by a device corresponding to the third aspect or any implementation manner of the third aspect, and the descriptions in the third aspect or any implementation manner of the third aspect are applicable to the third aspect or any implementation manner of the third aspect. The sixth aspect or any implementation manner of the sixth aspect will not be repeated here.

The seventh aspect or any implementation manner of the seventh aspect is implemented by a device corresponding to the fourth aspect or any implementation manner of the fourth aspect, and the description in the fourth aspect or any implementation manner of the fourth aspect is applicable to the seventh aspect Or any implementation manner of the seventh aspect, which is not repeated here.

The eighth aspect or any implementation manner of the eighth aspect is implemented by a device corresponding to the fifth aspect or any implementation manner of the fifth aspect, and the description in the fifth aspect or any implementation manner of the fifth aspect is applicable to the eighth aspect Or any implementation manner of the eighth aspect, which is not repeated here.

In a ninth aspect, the present application provides a first terminal, where the first terminal includes a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute the instructions stored in the memory, so that the first terminal executes the method in the third aspect or any one of the implementation manners of the third aspect.

In a tenth aspect, the present application provides a second terminal, where the second terminal includes a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute the instructions stored in the memory, so that the second terminal executes the method in the fourth aspect or any one of the implementation manners of the fourth aspect.

In an eleventh aspect, the present application provides a cloud phone, where the cloud phone includes a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute the instructions stored in the memory, so that the cloud mobile phone executes the method in the fifth aspect or any one of the implementation manners of the fifth aspect.

In a twelfth aspect, the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs a first terminal to execute the third aspect or any implementation manner of the third aspect the method described.

In a thirteenth aspect, the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs a second terminal to execute the fourth aspect or any implementation manner of the fourth aspect the method described.

In a fourteenth aspect, the present application provides a computer-readable storage medium, where an instruction is stored in the computer-readable storage medium, and the instruction instructs the cloud mobile phone to execute the above fifth aspect or any implementation manner of the fifth aspect. method described.

In a fifteenth aspect, the present application provides a computer program product including instructions, which, when running on the first terminal, enables the first terminal to execute the third aspect or any one of the implementation manners of the third aspect. method.

In a sixteenth aspect, the present application provides a computer program product including instructions, which, when running on a second terminal, enables the second terminal to execute the fourth aspect or any one of the implementation manners of the fourth aspect. method.

In a seventeenth aspect, the present application provides a computer program product containing instructions, which when run on a cloud mobile phone, enables the cloud mobile phone to execute the method described in the fifth aspect or any one of the implementation manners of the fifth aspect.

On the basis of the implementation manners provided by the above aspects, the present application may further combine to provide more implementation manners.

Description of drawings

In order to illustrate the technical methods of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the embodiments.

1 is a schematic diagram of the system architecture of a design system based on a cloud phone provided by an embodiment of the present application;

2 is a schematic diagram of a system architecture of a data center according to an embodiment of the present application;

3 is a schematic structural diagram of a design application provided by an embodiment of the present application;

FIG. 4 is an interactive flowchart of a design method based on a cloud phone provided by an embodiment of the present application;

5a to 5e are schematic interface diagrams of a first terminal according to an embodiment of the present application;

6 is a schematic diagram of a rendering resource pool provided by an embodiment of the present application;

7a to 7e are schematic interface diagrams of a third terminal according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a first terminal according to an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a second terminal according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a server according to an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a first client according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a second client according to an embodiment of the present application;

FIG. 13 is a schematic structural diagram of a cloud mobile phone according to an embodiment of the present application.

detailed description

The terms "first" and "second" in the embodiments of the present application are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature.

First, some technical terms involved in the embodiments of this application are introduced.

In order to facilitate the understanding of the embodiments of the present application, first, some terms involved in the present application are explained.

Cloud phone: A cloud service with a mobile phone operating system running on a physical server and a virtual phone function. The cloud service is usually implemented by using a container. The essence is to transfer the application on the mobile phone to the container running in the physical server of the data center. The container isolates different cloud mobile phones from each other without interfering with each other. The cloud mobile phone can install mobile phone applications, these mobile phone applications run in the cloud mobile phone, and the audio and video streams generated during the running process can be sent to the local mobile phone for display and playback. The control commands generated by the local mobile phone according to the audio and video streams displayed and played locally can also be sent to the cloud mobile phone. The cloud mobile phone controls the running state of the application according to the control command. Therefore, the mobile phone application of the local mobile phone can be transferred to the cloud mobile phone in the data center. To run, the local mobile phone does not need to install a large number of mobile phone applications that consume a lot of hardware resources, such as 2D or 3D games, industry design software, etc. Lightweight. Moreover, since the mobile phone application runs on the cloud mobile phone, the local mobile phone does not require high hardware configuration to allow users to have the opportunity to use mobile phone applications with high hardware configuration requirements, so there is no need to replace the local mobile phone due to higher hardware requirements.

It should be noted that the cloud mobile phone can also be implemented by using a virtual machine, which is not limited in this embodiment of the present application.

Virtual reality (VR): also known as virtual environment, is a virtual world that simulates a three-dimensional space through computing equipment, provides users with a simulation of vision and other senses, and makes users feel as if they are in the real world. Observe things in three-dimensional space. When the user moves, the computing device can immediately perform complex calculations, sending back an accurate 3D image of the world to create a sense of presence.

VR terminal: A terminal with VR function. VR terminals can be divided into two types: integrated and split. The so-called integrated type means that the calculation and display functions are integrated in the same device, and the so-called split type means that the calculation and display functions are implemented by different devices respectively. In some implementations, the split machine includes a main box and a display device. The host box is used to perform calculations to provide images to the display device. The host box may be a special device such as a game console, or a general device such as a personal computer (PC), a smart phone, and the like. Display devices are used to display images provided by the host box. The display device may be a wearable device such as a VR-supported helmet, head-mounted display, glasses, and watch. In some cases, the display device in the split machine can also be called a VR terminal, and the VR terminal can also be divided into PC VR and mobile VR based on the main box type.

VR terminals can be applied to the field of industrial design to provide designers with a new way of interaction. Specifically, the VR terminal can generate a virtual design environment, and in the virtual design environment, designers can perform human-computer interaction with both hands. In addition, designers can also perform human-computer interaction with other personnel in the virtual design environment to complete the design collaboratively, which can greatly improve design efficiency and design quality. However, the above-mentioned virtual design environment supports a limited number of people participating in the design, usually 8 to 10 people, which is difficult to meet business needs.

In order to solve the above technical problems, embodiments of the present application provide a design method and system based on a cloud phone, the system includes a cloud phone and at least one client, and the at least one client runs on a terminal respectively. The method includes the following steps:

The cloud phone receives at least one interaction information sent by one or any combination of at least one client, runs a design application according to the at least one interaction information, and renders a video stream according to the design application running process; wherein the at least one interaction information comes from A design operation triggered by one or any combination of users in at least one client through the design interface;

The cloud phone sends a video stream to at least one client, so that the at least one client displays a design screen on the design interface according to the video stream.

Based on the advantage that the computing capability of the cloud mobile phone can be elastically scaled, the embodiments of the present application utilize the cloud mobile phone to render video streams in the cloud, so that at least one client can display a design screen in the design interface according to the video stream. On the one hand, it can support the online collaborative design of a large number of users; on the other hand, offloading work such as rendering that consumes a lot of computing power to the cloud phone can reduce the hardware specification requirements of the terminal, save costs, and have high availability.

The embodiments of the present application will be described in detail below based on the general idea introduced above.

First, please refer to FIG. 1 . FIG. 1 is a schematic diagram of a system architecture of a design system according to an embodiment of the present application.

As shown in FIG. 1 , the design system of the embodiment of the present application includes a first terminal 21 , a second terminal 22 , a third terminal 23 , and a data center 200 that access the network 150 .

The first terminal 21, the second terminal 22, and the third terminal 23 are installed with clients for designing applications, which are the first client, the second client, and the third client, respectively, the first client, the second client, and the The third client establishes a connection with the data center 200 and performs data interaction.

It should be noted that the number of clients supported by this embodiment of the present application may be any one or more, and the corresponding number of terminals may also be any one or more. FIG. 1 only takes the number of terminals as 3 for illustration.

In addition, the client runs on the terminal, and the actions performed by the client in this embodiment of the present application may be considered to be performed by the terminal running the client.

The data center 200 is located on the network side, and a cloud mobile phone resource pool 100 and a cloud mobile phone management node 101 are set in the data center 200 . The cloud mobile phone resource pool 100 includes at least one cloud mobile phone, for example, may include a cloud mobile phone 102, a cloud mobile phone 103, . . . Design applications are installed in at least one cloud phone. For example, as shown in FIG. 1 , design application 1, design application 2, design application 3 and design application 4 are installed in the cloud phone 102, and design application 5 is installed in the cloud phone 103. Design applications 1 to 5 may be architectural design applications, decoration design applications, integrated circuit design applications, clothing design applications, advertising design applications, and the like.

In an optional embodiment, the terminal may be, for example, a VR terminal, such as a PC VR terminal, or a mobile VR terminal such as VR glasses and VR helmets. In another embodiment, the terminal may also be, for example, a flat terminal such as a tablet computer, a personal digital assistant, and a notebook computer. The above terminal is provided with a voice collection device and a camera. In FIG. 1 , the first terminal 21 is a VR terminal, such as a VR helmet, the second terminal 22 is a flat terminal, such as a large-screen display system used in conference rooms and classrooms, and the third terminal 23 is a portable flat terminal, such as a smart phone. Example description.

In this embodiment of the present application, at least one user who designs a project forms a user group, for example, user 11 , user 12 , and user 13 in FIG. 1 form a user group. Any user in the user group can use the client running on his terminal. For example, user 11 can start the collaborative design through the first client running on the terminal 21. In response to this operation, the cloud mobile phone such as cloud mobile phone 103 generates a interface data stream, and send the interface data stream to the client corresponding to each user of the user group, so that each client of the user group, such as the first client, the second client, and the third client, generates the design interface according to the interface data stream .

At least one of User 11, User 12, and User 13 can trigger the design operation through the design interface, for example, the design operation can be triggered through gestures, or the design operation can be triggered through touch, voice control, etc. At least one of the second client and the third client generates interaction information according to the design operation, and sends the interaction information to the cloud phone 103 .

The cloud phone 103 runs the design application according to the interaction information, such as running the design application 5, and renders the video stream according to the running process of the design application, and then sends the video stream to the client corresponding to each user of the user group, the first client, the second client The terminal and the third client display the design picture in the design interface according to the video stream.

For full explanation, please refer to FIG. 2 below. FIG. 2 is a schematic diagram of a system architecture of a data center 200 according to an embodiment of the present application. As shown in FIG. 2 , the data center 200 includes a cloud mobile phone management node 101 and a cloud mobile phone resource pool 100 , wherein , the cloud mobile phone resource pool 100 includes a plurality of physical servers 110, 120, .

Among them, each server in the cloud mobile phone resource pool 100 includes hardware resources and software resources, and the hardware resources include hardware devices common to servers such as network cards, processors, memory, etc., wherein the processors may include a central processing unit (central processing unit). , CPU) and graphics processing unit (graphics processing unit, GPU). A server may include at least one CPU and at least one GPU. FIG. 2 illustrates that a server includes multiple CPUs and multiple GPUs. The software resources include a server operating system, at least one container running on the server operating system, and a cloud mobile phone management node agent module. Specifically, the cloud mobile phone 102 in FIG. 1 can be implemented by the container 103 in FIG. 2 . The mobile phone operating system is installed in the container 103, and the design applications 1-4 run on the mobile phone operating system; the cloud mobile phone 103 in FIG. The container 102 in FIG. 2 is implemented, and a mobile phone operating system is also installed in the container 102, and the design application 5 runs on the mobile phone operating system. Design applications 1 to 4 and design application 5 are isolated by the container.

The cloud mobile phone management agent node agent module implements the full life cycle management of the container, and can be used to monitor and collect the container information of the server, and report the server information to the cloud mobile phone management node 101 in real time. For example, the cloud phone management agent node agent module 1011 on the server 110 can monitor and collect the container information of the server 110, the cloud phone management agent node agent module 1011 receives the cloud phone control command sent by the cloud phone management node 101, and the cloud phone management The proxy node proxy module 1011 can control the working state of the container 102 and the container 103 on the server 110 according to the cloud mobile phone control command. The cloud mobile phone management proxy node proxy module 1012 is also used to receive the cloud mobile phone control command sent by the cloud mobile phone management node 101. The mobile phone management proxy node proxy module 1011 can control the working status of the container 104 and the container 105 on the server 120 according to the cloud mobile phone control command.

The cloud phone shares hardware resources and operating systems with other cloud phones on the server in the form of containers. Each container can include the applications required by the cloud phone and the dependent resources required by each application, and the dependent resources required by the application. It can be the core library and system library in the previous content. Among them, each cloud phone and other cloud phones share the server's hardware resources and operating system. Specifically, hardware resources of different servers can be virtualized to form a resource pool. For example, rendering resources such as GPUs of different servers can be virtualized to form a rendering resource pool that supports dynamic allocation, flexible scheduling, and cross-domain sharing.

In other words, each container does not have its own kernel, the application process in the container runs directly on the server's kernel, and each cloud phone can run directly on the server hardware independently of each other, creating an independent sandbox running environment for applications.

In the specific implementation, the cloud phone management node and the cloud phone management agent node agent module deployed on each server can be implemented based on container management systems such as Kubernetes and Docker.

In other embodiments, the cloud phone may also be implemented by a virtual machine, which is not limited in this application.

In order to make the technical solution of the present application clearer and easier to understand, please refer to FIG. 3 below, which is a schematic structural diagram of a design application according to an embodiment of the present application. For ease of description, FIG. 3 uses design application 1 as an example. As shown in FIG. 3 , the design application 1 includes a design module and a rendering module. Among them, the design module is used to construct a model. Taking the construction of a three-dimensional (3D) model as an example, the design module can collect model parameters, process the model parameters, and then map them to construct a 3D model. The rendering module is used to render the video stream according to the 3D model. The video stream includes at least a picture stream (also referred to as an image stream), and the picture stream can be used to generate a design picture, such as a 3D picture of a design object.

In some possible implementations, the design application 1 may also include a speech processing module. The voice processing module is used for receiving the voice of at least one client, and synchronizing the voice of the client to other clients, so as to realize user communication. The audio stream corresponding to the voice may be independent of the above-mentioned picture stream. Specifically, the cloud phone may send the audio stream and the picture stream to the client respectively. Of course, the audio stream corresponding to the voice can also be mixed with the picture stream to render a video stream including the audio stream.

In some possible implementations, the design application 1 may further include a rights management module. The authority management module is used to manage the design authority of each user. For example, the rights management module can provide read and write rights for several users in the user group, or provide read-only rights for users in the user group. Among them, the read-write permission means that you can participate in the design, and the read-only permission means that you can watch the design process, but do not perform design operations.

In addition, the rights management module is also used to manage and schedule the control rights of users with read and write rights. For example, the user may send a request for obtaining control authority through the client, and the authority management module assigns the control authority to the user in response to the request. In this way, the user can perform a "write" operation, specifically a design operation, through the design interface, so as to perform an online collaborative design. It should be noted that the authority management module may provide control authority for one user at the same time, and may also provide control authority for multiple users at the same time, which is not limited in this embodiment of the present application.

In some possible implementation manners, the above-mentioned rendering module, language processing module, and rights management module may also be independent from the design application, and corresponding functions can be implemented through independent applications. For example, the management and scheduling of control rights can be realized through an independent rights management application.

The following specifically describes the design method involved in the embodiments of the present application based on the system architecture introduced in FIG. 1 and FIG. 2 .

Please refer to FIG. 4. FIG. 4 is an interaction flow chart of a design method based on a cloud phone according to an embodiment of the present application. The design method in an embodiment of the present application includes the following steps:

S101: The first client sends a design application startup request to the cloud mobile phone management service node 101.

The design application start request includes an identifier of the design application, and is used to start the corresponding design application. In some possible implementations, the design application launch request may also include a user identification. The user identification includes at least the identification of the user who triggered the start operation of the design application. Further, the user identification may also include identifications of other users participating in the design. In some embodiments, the first client may also send the user identification after the design application is launched, or after the design document is opened or created.

The identifiers and user identifiers of the above-mentioned design applications are unique identifiers, such as names, codes, and the like. For example, the identity of the design application may be the name of the design application, and the user identity may be the user's email address.

For ease of understanding, reference may be made below in conjunction with FIGS. 5 a to 5 b , wherein FIGS. 5 a to 5 b are schematic diagrams of screen display images of the first terminal 21 according to an embodiment of the present application.

Referring to FIG. 5a, as shown in FIG. 5a, the user 11 wears the first terminal 21, the first terminal 21 has a head-mounted display device (referred to as a head-mounted display), and the main display interface of the head-mounted display is provided with the first client's icon, the user 11 can trigger the icon of the first client through physical actions, such as gestures, and the first terminal 21 starts the first client in response to the operation, and establishes a connection with the cloud phone management service node 101 . The cloud phone management service node 101 may return a list of applications installed on the cloud phone managed by the cloud phone management service node 101 to the first client, such as an application list including design applications 1 to 7.

Next, referring to FIG. 5 b , as shown in FIG. 5 b , the first client displays icons of each application in the application list on the cloud mobile phone application management interface according to the application list returned by the cloud mobile phone management service node 101 . The user 11 can trigger the icon of the application through a gesture, thereby triggering the operation of starting the application. For example, the user 11 can trigger the icon of the design application 1 through a gesture, thereby triggering the operation of starting the design application 1 . In response to the above operation of the user 11 , the first client generates a design application start request, and sends the design application start request to the cloud mobile phone management service node 101 .

Wherein, the design application startup request includes the identifier of the design application, for example, the name of the design application 1 , thereby instructing to start the design application 1 . The design application start request also includes user identifications, for example including identifications of user 11 , user 12 and user 13 .

S102: The cloud phone management service node 101 determines the cloud phone where the design application requested to be started is located.

The cloud mobile phone management service node 101 is used to manage at least one cloud mobile phone. The cloud phone management service node 101 stores the correspondence between at least one cloud phone managed by the cloud phone and an application installed on the cloud phone. For example, the corresponding relationship between the cloud phone 102 and the design applications 1 to 4, and the corresponding relationship between the cloud phone 105 and the design application 5.

Based on this, the cloud phone management service node 101 can determine the cloud phone where the design application requested to be started is located according to the design application identifier in the design application startup request and the corresponding relationship between the cloud phone and the application installed on the cloud phone. For example, when the design application requested to be started is the design application 1, the cloud mobile phone management service node 101 may determine that the cloud mobile phone where the design application 1 is located is the cloud mobile phone 102 according to the corresponding relationship.

S103: The cloud mobile phone management service node 101 routes the design application start request to the corresponding cloud mobile phone (for example, the cloud mobile phone 102).

Specifically, the cloud mobile phone management service node 101 sends the received design application start request to the cloud mobile phone where the design application requested to be started is located, such as the cloud mobile phone 102 .

The embodiment of the present application only uses the cloud mobile phone 102 as an example. When the design application requested to be started is another design application, the cloud mobile phone management service node 101 may also send the design application start request to other cloud mobile phones. For example, when the design application requested to be activated is the design application 5, the cloud phone management service node 101 may send the design application activation request to the cloud phone 103.

S104: The cloud phone 102 starts a design application (for example, design application 1), and generates an instance of the design application.

Design applications are applications or software used to implement design functions. When the design application is not started, the program code of the design application is statically stored in an external memory such as a disk, and when the design application is started, the program code of the design application is loaded into the memory to generate a corresponding process. This process can be viewed as an instance of a design application.

Wherein, each time the design application is started, an instance is generated. When the same design application is launched multiple times, multiple instances are generated. For example, if different users respectively trigger operations to start the same design application, the cloud phone 102 respectively generates instances of the design application for different users. It should be noted that the instances of the design application generated by the cloud phone 102 for different users are isolated from each other.

In some possible ways, the cloud phone 102 allows an instance of the design application to be generated. For this reason, when multiple users need to start the design application for collaborative design, the clients of multiple users can connect to different cloud mobile phones respectively, and obtain the control authority of the corresponding cloud mobile phone, and then start the cloud mobile phone according to the control authority. Design applications, resulting in corresponding instances. In this way, the isolation of instances of design applications is achieved through different cloud phones.

S105: The cloud phone 102 returns the address and port of the instance to the first client, the second client and the third client.

S106: The first client, the second client and the third client respectively establish connections with the instance of the design application according to the address and port of the instance.

For each instance of the designed application, there is an address and port. The address may specifically be a network address (Internet Protocol address, IP address), and the port may specifically be a communication port or a protocol port. The address and port are relevant information used to establish a connection with the instance.

When the design application startup request carries a user ID, such as the IDs of user 11, user 12, and user 13, the cloud phone 102 can return the address and port of the instance to the corresponding user, specifically to the client of the corresponding user. address and port. For example, the cloud phone 102 may return the address and port of the instance to the first client, the second client, and the third client.

In this way, the first client, the second client and the third client respectively establish connections with the instance of the design application according to the address and port of the above instance.

In some possible implementation manners, the instance of the design application may also authenticate the user who requests to access the instance, and after the authentication is passed, establish a connection with the corresponding client. Wherein, the instance of the design application can perform legality verification according to the user ID of the user who requests to access the instance, so as to realize the authentication. Certainly, in other possible implementation manners, the instance of the design application may also be authenticated through a password, a verification code, or other information that can be used to verify an identity, which is not limited in this embodiment of the present application.

S107: The first client sends a design document creation request to the instance of the design application on the cloud phone 102.

A design document creation request is used to request the creation of a new design document. It should be noted that the design document may be a blank design document or a design document loaded with a template. In some cases, designing with a template-loaded design document can improve design efficiency.

In some possible implementations, the first client may also request to open an existing design document. For example, for a certain design project, the first client may request to open the most recently saved design document for the design project, so as to continuously design based on the design document.

For ease of understanding, the following description is given with reference to a specific example.

Referring to Figure 5c, after the instance of the design application is authenticated, the instance of the design application can return the interface data stream corresponding to the main interface of the design application to the corresponding client, so that the client can generate the design application according to the data stream. Main interface. As shown in 1104 in FIG. 5c, the main interface of the design application displays a list of recently used design documents, specifically including document 1 to document N. In addition, the main interface of the design application also displays blank documents and templates.

The user can click on any document in the list of design documents to request that the document be opened in order to continue designing based on that document. The user can also click on the blank document or click on any one of the template documents such as Template 1, Template 2, etc., to request the creation of a new design document.

In the example of FIG. 5c , the main interface of the design application may specifically be presented in the interface 1100 of the first client in the form of a window. The interface 1100 of the first client may further include the avatar of the user accessing the instance, as shown in 1101, 1102, and 1103 in Fig. 5c. When the number of users accessed by the instance of the design application is greater than the preset threshold, the interface 1100 of the first client can display the avatars of the preset number of users, and the avatars of other users are represented by "...".

S108: The instance of the design application on the cloud phone 102 creates a design document and generates an interface data stream.

S109: The cloud phone 102 returns the interface data stream to the first client, the second client and the third client.

S110: The first client, the second client and the third client generate a design interface according to the interface data stream.

The instance of the design application on the cloud phone 102 responds to the design document creation request, creates a new design document, generates the interface data stream of the design interface of the design document, and then returns the interface data stream to the client connected to the instance, so that the client can The terminal generates the design interface according to the interface data flow.

For ease of understanding, the following description is given by taking the design application 1 as home improvement design as an example.

Referring to FIG. 5d, the user 11 can trigger the icon of the blank document through a gesture to request the creation of a new blank document. The instance of the design application 1 on the cloud phone 102 creates a new blank document according to the request, and generates an interface data flow of the design interface of the design document. The clients of the user 11, the user 12, and the user 13 generate the design interface shown as 1105 in FIG. 5d according to the interface data flow.

The design interface includes toolbars, such as menu tools and shape tools. The menu tools include menu keys such as start, insert, and design, and the shape tools include various shortcut shapes. The design interface also includes a blank design area, and users can drag and drop shortcut shapes in the shape tool, such as "room", "window" and other shapes, to the design area for home improvement design.

S111: The first client sends interaction information 1 to the instance of the design application on the cloud phone 102.

S112: The second client sends interaction information 2 to the instance of the design application on the cloud phone 102.

S113: The third client sends interaction information 3 to the instance of the design application on the cloud phone 102.

Specifically, the user can trigger the design operation through the design interface of the respective client. Depending on the type of terminal where the client is located, the user can trigger the design operation in different ways. For example, when the terminal is a flat terminal such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc., the user can trigger the design operation by means of touch, a keyboard and a mouse, or the like. For another example, when the terminal is a VR terminal such as a VR helmet, VR glasses, etc., the user can trigger the design operation through body movements (such as gestures, etc.).

The client can obtain interaction information according to the design operation triggered by the user. Specifically, when the terminal is a flat terminal, the interaction information may include operation instructions generated based on a user's touch operation or keyboard and mouse operation, such as an operation instruction for adding a wall and an operation instruction for adding a window. When the terminal is a VR terminal, the interaction information may include at least one of motion information and position information determined based on the user's body movements. The motion information may include at least one of motion type, motion range, and the like. The position information may include the positions of the feature points, such as the starting position of the finger movement and the ending position of the finger movement.

Wherein, based on different design operations triggered by different users, the design information obtained by different clients may be different. For example, the first client obtains the interaction information 1, sends the interaction information 1 to the instance of the design application on the cloud phone 102, the second client obtains the interaction information 2, sends the interaction information 2 to the instance of the design application on the cloud phone 102, and the third client obtains the interaction information 2. The client obtains the interaction information 3, and sends the interaction information 3 to the instance of the designed application on the cloud phone.

In some possible implementations, the client may send interaction information to the instance of the design application after obtaining the control authority assigned by the instance of the design application. In some embodiments, the client can assign a control permission to a user at the same time, when the user stops the operation, the control permission is released, and the user or other users can request the instance of the design application to assign the control permission again. This can effectively prevent conflicting operations or repeated operations. In other embodiments, the client can also assign control permissions to multiple users at the same time, so that multiple users can trigger design operations at the same time, thereby improving design efficiency.

It should be noted that, in this embodiment of the present application, the first client, the second client, and the third client send interaction information to an instance of designing an application on the cloud mobile phone 102 for illustration, and other possible implementation manners in the embodiment of the present application , it may also be that some clients send interaction information to the instance of the application designed on the cloud phone 102 . For example, other users such as user 12 and user 13 explain how to design through voice, and user 11 triggers a corresponding design operation according to the voice content.

S114: The cloud phone 102 runs the instance of the design application according to the interaction information, and renders the video stream according to the running process of the design application.

Specifically, the instance of the design application on the cloud phone 102 may run the design application according to the interaction information 1, the interaction information 2, and the interaction information 3, for example, an instance of running the design application. Wherein, different interaction information corresponds to different design application running processes. The cloud phone 102 can perform modeling according to the running process of the design application, and then perform image rendering based on the model information obtained from the modeling, specifically, performing image rendering on consecutive multi-frame images, thereby obtaining a video stream.

Among them, rendering can be understood as the processing of images to conform to the 3D scene. Specifically, the camera in the 3D scene is first positioned. In general, a rendering module or a rendering application provides a default camera, which is a camera corresponding to a top view, a front view, a side view, and a perspective view, respectively. The rendering module or rendering application obtains the range to be rendered through the camera, calculates the influence of the light source on the object, and then the rendering module or rendering application calculates the color of the surface of the object according to the material of the object. Among them, different types of materials, different properties, and different textures can produce different visual effects.

Considering that the screen rendering process usually requires a lot of complex calculations, the cloud phone can obtain a target number of rendering resources from the rendering resource pool according to the running process of the design application, and then use the rendering resources to render the video stream according to the running process of the design application. The rendering resource refers to a resource used for image rendering, and may specifically be a processor, such as a graphics processor GPU. The rendering resources (such as GPU) provided by the hardware device (such as a server) where the cloud phone is located can be pooled to form a rendering resource pool, thereby realizing unified scheduling management of rendering resources.

Specifically, the server can virtualize rendering resources such as GPUs to redefine and divide rendering resources, and obtain a rendering resource pool that can support dynamic allocation, flexible scheduling, and cross-domain sharing. Referring to FIG. 6 , the rendering resource pool includes a redefined virtual graphics processing unit (virtual graphics processing unit, vGPU). When rendering, the instance of the design application on the cloud phone (for example, the rendering module in the instance) can select different numbers of vGPUs from the rendering resource pool for rendering according to different rendering objects. For example, for object A, 2*3 vGPUs may be selected from the rendering resource pool for rendering, and for object B, 4*4 vGPUs may be selected from the rendering resource pool for rendering.

The running process of the design application can indicate the objects to be rendered, and the cloud phone can determine the number of rendering resources (for example, the target number) required for rendering according to the running process of the design application, obtain the rendering resources of the target number from the rendering resource pool, and then According to the design application running process, use the target number of rendering resources to render the video stream.

Considering that the rendering capabilities of different types of GPUs may be different, for example, the rendering capabilities of ARM-based GPUs or x86-based GPUs may be different. The instance of the design application can also determine the rendering resources required for rendering according to the running process of the design application. The type and quantity of the target, and then get the target type and target number of rendering resources from the rendering resource pool. For example, vGPUs of different colors in Figure 6 represent different types of vGPUs, and an instance of a design application on a cloud phone can obtain a first number of vGPUs of the first type and a second number of vGPUs of the second type according to the running process of the design application vGPU for rendering video streams.

In some possible implementation manners, the cloud phone (specifically, an instance of the design application on the cloud phone) may also obtain a target number of rendering resources from the rendering resource pool in combination with the number of terminals according to the design application running process. When the number of terminals is large, the cloud phone can obtain more rendering resources for screen rendering to support the online collaborative design of a large number of users.

Further, the cloud phone can also obtain the target type and target number of rendering resources from the rendering resource pool in combination with the terminal type, so as to be used for screen rendering. For example, when the terminals include mobile phones, tablet computers, desktop computers, VR helmets, VR glasses, etc., the cloud phone can also obtain a set of target types and target number of rendering resources for different types of terminals.

Considering that cloud VR services consume huge bandwidth resources under the X86 architecture, a single user usually needs 50Mbps to 100Mbps bandwidth resources. Cloud mobile phones can also use the compatibility of ARM instructions on the device side and the cloud side to reduce bandwidth through device-cloud collaborative rendering. This reduces costs and facilitates large-scale commercial use.

Specifically, the cloud phone splits the rendering process according to the running process of the design application, specifically generating a cloud rendering instruction stream and a terminal rendering instruction stream, and the cloud phone (specifically, an instance of the design application on the cloud phone) according to the cloud rendering instruction stream , call the corresponding rendering resource to render the video stream, and obtain the first rendered video stream. In addition, the cloud mobile phone (specifically, the instance of the design application on the cloud mobile phone) sends the first rendering video stream and the terminal rendering instruction stream to at least one client, and the client side of the terminal can call the corresponding rendering resource rendering according to the terminal rendering instruction stream. video stream, thereby obtaining the second rendered video stream.

At least one client can combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream, where the third rendered video stream is a fully rendered video stream. At least one client can display a design picture on the design interface according to the third rendered video stream.

Further, when splitting, the cloud phone (specifically, the instance of the design application on the cloud phone) can generate a cloud rendering instruction stream according to the information that the rendering calculation amount is greater than the calculation amount threshold during the running process of the design application, and according to the design application running process. The information for which the rendering calculation amount is less than or equal to the calculation amount threshold generates a side rendering instruction stream.

For example, background information usually consumes a large amount of computation, and the cloud phone (specifically, the instance of the design application on the cloud phone) can generate a cloud rendering instruction stream according to the background information during the running process of the design application, and according to the running process of the design application The action information in generates a stream of side rendering instructions.

S115: The cloud phone 102 sends a video stream to the first client, the second client and the third client.

S116: The first client, the second client, and the third client display a design picture on the design interface according to the video stream.

The cloud phone 102 can use a set of rendering resources to render the video stream, and distribute the video stream to different clients, such as the first client, the second client and the third client, each client receives the same video stream, so It can be displayed on the same screen on different clients.

The first client, the second client, and the third client receive the video stream, and can perform de-IP, decoding, and pre-display processing, and then display the design screen on the design interface. The design screen is specifically the screen presented by the design object. , users can conduct real-time communication on the design screen through the voice call function to achieve collaborative design.

Referring to Fig. 5e, as shown in Fig. 5e, the first client, the second client, and the third client receive a video stream, and display a design screen in the design interface shown in 1105 in Fig. 5e according to the video stream. The first client can display the 3D design screen shown in 1106 in FIG. 5e in the design interface shown in 1105 in FIG. 5e according to the video stream.

In some possible implementations, the cloud phone 102 may use multiple sets of rendering resources to render video streams for different types of terminals, respectively, and distribute the video streams to the clients of the corresponding terminals, for example, send video stream 1 to the first client , and send the video stream 2 to the second client and the third client, so that each client can display a corresponding design screen on the design interface according to the video stream received by each client, and perform collaborative design based on the design screen.

Based on the foregoing description, the embodiments of the present application provide a design method based on a cloud mobile phone. The method is based on the advantage that the computing power of the cloud mobile phone can be elastically scaled, and the cloud mobile phone is used to render the video stream in the cloud, so that at least one client can display the design screen in the design interface according to the video stream. On the one hand, it can support online collaborative design for a large number of users; on the other hand, offloading work such as rendering that consumes a lot of computing power to the cloud phone can reduce the hardware specification requirements of the terminal, save costs, and have high availability.

Further, the method can support multiple types of terminals, such as VR terminals such as VR glasses and VR helmets, as well as flat terminals such as mobile phones and tablets, without the need to configure special equipment, which meets the needs of different users and provides users with a better experience. Moreover, the user can complete the collaborative design by using the mobile terminal, and under the permission of the network conditions, the collaborative design work can be carried out anytime and anywhere.

The embodiment shown in FIG. 4 mainly shows the process of multi-user online collaborative design from the perspective of the first terminal 21 , and then shows the multi-user online coordination from the perspective of other types of terminals such as the second terminal 22 or the second terminal 23 in conjunction with the accompanying drawings. design process. Considering that both the second terminal 22 and the third terminal 23 are flat terminals, the embodiment of the present application uses the third terminal 23 as an example for description.

For ease of understanding, reference may be made below in conjunction with FIGS. 7a to 7e , wherein FIGS. 7a to 5e are schematic diagrams of screen display images of the third terminal 23 according to an embodiment of the present application.

Please refer to FIG. 7a. As shown in FIG. 7a, the icon of the third client is set in the main display interface of the third terminal 23. After the user 13 clicks the icon of the third client, the third client starts and communicates with the cloud mobile phone management service node. 101 A connection is established. The cloud mobile phone management service node 101 may return a list of applications installed on the cloud mobile phone managed by the cloud mobile phone management service node 101 to the third client, such as an application list including design applications 1 to 7.

Next, referring to FIG. 7b, as shown in FIG. 7b, the third client displays icons of each application in the application list on the cloud mobile phone application management interface according to the application list returned by the cloud mobile phone management service node 101. The user 13 clicks on the icon of the application, which can trigger the operation of starting the application. For example, user 13 clicks on design application 1 , thereby triggering an operation to start design application 1 . In response to the above operation of the user 13 , the third client generates a design application start request, and sends the design application start request to the cloud mobile phone management service node 101 .

Wherein, the design application startup request includes the identifier of the design application, for example, the name of the design application 1 , thereby instructing to start the design application 1 . The design application startup request also includes user identifications, for example including identifications of user 13 , user 12 and user 11 .

The cloud mobile phone management service node 101 receives the design application start request, and determines the cloud mobile phone where the design application requested to be started is located according to the corresponding relationship between at least one cloud mobile phone managed by the cloud mobile phone and the application installed on the cloud mobile phone. For example, when the design application requested to be started is the design application 1, the cloud mobile phone management service node 101 may determine that the cloud mobile phone where the design application 1 is located is the cloud mobile phone 102 according to the above corresponding relationship.

Next, the cloud mobile phone management service node 101 may route the design application start request to the cloud mobile phone 102 . The cloud phone 102 starts the design application according to the design application start request, for example, starts the design application 1 to generate an instance of the design application 1 . When the design application startup request includes the identifiers of user 13, user 12, and user 11, the cloud phone 102 can send messages to the first client, the second client, and the third client according to the identifiers of user 13, user 12, and user 11, respectively. Returns the address and port of the instance.

The first client, the second client, and the third client establish connections with the instance of the design application 1 according to the address and port of the instance, respectively. Wherein, the instance of design application 1 can also authenticate the user who requests to access the instance, and after the authentication is passed, establish a connection with the corresponding client.

Next, referring to FIG. 7c, after the instance of design application 1 is authenticated, the instance of design application 1 can return the interface data stream corresponding to the main interface of the design application to the corresponding client, so that the client can generate the data stream according to the Design the main interface of the application. As shown in 1104 in FIG. 7c, the main interface of the design application displays a list of recently used design documents, specifically including document 1 to document N. In addition, the main interface of the design application also displays blank documents and templates.

User 13 may click on any document in the list of design documents, thereby requesting that the document be opened in order to proceed with the design based on that document. The user can also click on the blank document or click on any one of the template documents such as Template 1, Template 2, etc., to request the creation of a new design document.

In the example of FIG. 7c, the main interface of the design application is specifically presented in the interface 1100 of the third client in the form of a window. The interface 1100 of the third client may further include the avatar of the user accessing the instance, as shown in 1101, 1102, and 1103 in Fig. 7c. When the number of users accessed by the instance of the design application is greater than the preset threshold, the interface 1100 of the third client can present the avatars of the preset number of users, and the avatars of other users are represented by "...".

The instance of the design application 1 on the cloud phone 102 responds to the design document creation request, creates a new design document, generates the interface data stream of the design interface of the design document, and then returns the interface data stream to the client connected to the instance, so that The client generates the design interface according to the interface data flow.

Referring to FIG. 7d, the user 13 can trigger the icon of the blank document through a gesture to request the creation of a new blank document. The instance of the design application 1 on the cloud phone 102 creates a new blank document according to the request, and generates an interface data flow of the design interface of the design document. The clients of the user 11, the user 12, and the user 13 generate the design interface shown as 1105 in FIG. 5d according to the interface data flow.

Users can trigger design operations through the design interface of their respective clients. For example, the user 13 can perform a touch operation through the design interface, and the third client can respond to the touch operation of the user 13 to generate an operation instruction, and then send the operation instruction to the instance of the design application 1 on the cloud phone 102 .

The instance of the design application on the cloud phone 102 can run the design application according to the interactive information including the operation instruction, perform modeling according to the operation process of the design application, and then perform screen rendering based on the model information obtained from the modeling, specifically, for continuous multi-frame The image is rendered on the screen to obtain a video stream, and the video stream is sent to the third client. The instance of the design application on the cloud phone 102 can also use another set of rendering resources to model according to the interaction information including the operation instructions, and then perform screen rendering based on the model information obtained from the modeling to obtain another video stream. The cloud phone 102 may send the first video stream to the second client and the third client, and send another video stream to the first client.

Referring to FIG. 7e, the third client can display the design screen shown in 1106 in FIG. 7e in the design interface shown in 1105 in FIG. 7e according to the first video stream. Similarly, the second client can display the design screen shown in 1106 in FIG. 7e in the design interface shown in 1105 in FIG. 7e according to the first video stream. For the design screen displayed by the first client, reference may be made to FIG. 5e, which will not be repeated here.

In this way, a large number of users including the first user, the second user, and the third user can carry out online collaborative design, and support multiple types of terminals, such as VR glasses, VR helmets and other VR terminals, as well as mobile phones, tablets and other flat terminals, without the need for Configure special equipment to meet the needs of different users and provide users with a better experience. Moreover, the user can complete the collaborative design by using the mobile terminal, and under the permission of the network conditions, the collaborative design work can be carried out anytime and anywhere.

8 to 10 , FIG. 8 is a schematic diagram of a hardware structure of a first terminal according to an embodiment of the present application. As shown in FIG. 8 , the first terminal 21 includes a processor 211 , a memory 212 , a communication interface 213 and a bus 214 , the processor 211 , the memory 212 , and the communication interface 213 are respectively connected to the bus 214 , the memory 212 stores program instructions, and the processor 211 executes the program instructions to realize the functions of the first terminal 21 .

FIG. 9 is a schematic diagram of the hardware structure of a second terminal according to an embodiment of the present application. As shown in FIG. 9 , the second terminal 22 includes a processor 221, a memory 222, a communication interface 223, and a bus 224. The processor 221, the memory 222, the communication The interfaces 223 are respectively connected to the bus 224 , the memory 222 stores program instructions, and the processor 221 executes the program instructions to realize the functions of the second terminal 22 .

The first terminal 21 is a VR terminal, and the second terminal 22 and the third terminal 23 are plane terminals. For the hardware structure of the third terminal 23, reference may be made to the hardware structure of the second terminal, which will not be repeated here.

FIG. 10 is a schematic diagram of the hardware structure of a server for deploying a cloud mobile phone, such as a server 110 according to an embodiment of the present application. As shown in FIG. 10 , the server 110 includes a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. The processor 1001, The memory 1002 and the communication interface 1003 are respectively connected to the bus 1004 . The memory 1002 stores program instructions, and the processor 1001 executes the program instructions to implement the functions of the server 110 .

The embodiment of the present application also discloses a design method based on a cloud mobile phone. The design method is applied to the first client and specifically includes the following steps:

Step 1: the first client sends a design application startup request to the cloud mobile phone management service node 101;

Step 2: the first client receives the address and port of the instance of the design application returned by the cloud phone 102;

Step 3: The first client establishes a connection with the instance of the design application according to the address and port of the instance;

Step 4: the first client sends a design document creation request to the instance of the design application on the cloud phone 102;

Wherein, steps 1 to 4 are an optional implementation manner in which the first client on the first terminal starts a design application on the cloud phone 102 to create a corresponding design document. In some possible implementations, when the first client establishes a connection with the cloud phone in advance, the first client can directly send a design application start request to the cloud phone 102 and directly send the design to the instance of the design application on the cloud phone 102 Document creation request.

Step 5: the first client receives the interface data stream returned by the cloud phone 102;

Step 6: The first client generates a design interface according to the interface data flow;

Step 7: The first client sends interaction information 1 to the cloud phone 102;

Step 8: the first client receives the video stream sent by the cloud phone 102;

Step 9: The first client displays the design screen on the design interface according to the video stream.

Optionally, the video stream sent by the cloud phone 102 received by the first client in step 8 may also be the first rendered video stream obtained by the cloud phone 102 rendering according to the cloud rendering instruction stream, and the first client also receives the video stream sent by the cloud phone 102. The end rendering instruction stream, the first client can also perform rendering according to the end rendering instruction stream to obtain the second rendering video stream. The first client may further combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream. Correspondingly, the first client can display the design picture on the design interface according to the third rendered video stream.

Please refer to FIG. 11 below. FIG. 11 is a schematic diagram of a device structure of a first client according to an embodiment of the present application. As shown in FIG. 11 , the first client includes:

The communication module 1102 is used for sending a design application startup request to the cloud mobile phone management service node 101;

The communication module 1102 is further configured to receive the address and port of the design application instance returned by the cloud phone 102;

A connection module 1104, configured to establish a connection according to the address and port of the instance and the instance of the design application;

The communication module 1102 is further configured to send a design document creation request to the instance of the design application on the cloud phone 102;

The communication module 1102 is further configured to receive the interface data stream returned by the cloud phone 102;

an interface generation module 1106, configured to generate a design interface according to the interface data flow;

The communication module 1102 is further configured to send the interaction information 1 to the cloud mobile phone 102;

The communication module 1102 is further configured to receive the video stream sent by the cloud mobile phone 102;

The display module 1108 is configured to display the design picture on the design interface according to the video stream.

The connection module 1104 in the first client is an optional module. When a connection is established between the first client and the cloud phone in advance, the first client does not need to perform the step of establishing the connection. The communication module 1102 may directly send a design application start request to the cloud phone to start the design application and generate an instance of the design application. Similarly, the communication module 1102 can also directly send a design document creation request to the instance of the design application to create a design document and generate an interface data stream, so that the first client generates a design interface according to the interface data stream.

Optionally, the first client may further include a rendering module and a confluence module. The communication module 1102 is specifically configured to receive the first rendered video stream rendered by the cloud mobile phone 102 according to the cloud rendering instruction stream, the communication module 1102 is further configured to receive the terminal rendering instruction stream sent by the cloud mobile phone 102, and the rendering module is used to render according to the terminal rendering instruction stream. The instruction stream is rendered to obtain a second rendered video stream, and the confluence module is configured to combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream. Correspondingly, the display module 1108 may display the design screen on the design interface according to the third rendered video stream.

The embodiment of the present application also discloses a design method based on a cloud phone, and the design method is applied to the second client or the third client. For the convenience of description, the second client is used as an example for description below, and the method specifically includes the following step:

Step 1: the second client receives the address and port of the instance of the design application sent by the cloud phone 102;

Step 2: The second client establishes a connection with the instance of the design application according to the address and port of the instance;

Wherein, when the second client establishes a connection with the instance of the design application in advance, the second client may not perform the above steps 1 to 2.

Step 3: the second client receives the interface data stream sent by the cloud phone 102;

Step 4: The second client generates a design interface according to the interface data flow;

Step 5: The second client sends interaction information 2 to the cloud phone 102;

Step 6: the second client receives the video stream sent by the cloud phone 102;

Step 7: The second client displays the design screen on the design interface according to the video stream.

Optionally, the video stream sent by the cloud phone 102 received by the second client in step 6 may also be the first rendered video stream obtained by the cloud phone 102 rendering according to the cloud rendering instruction stream, and the second client also receives the video stream sent by the cloud phone 102. The second client can also perform rendering according to the end rendering instruction stream, and obtain the second rendering video stream. The second client may further combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream. Correspondingly, the second client can display the design picture on the design interface according to the third rendered video stream.

Please refer to FIG. 12 below. FIG. 12 is a schematic diagram of a device structure of a second client terminal according to an embodiment of the present application. As shown in FIG. 12 , the second client terminal includes:

The communication module 1202 is configured to receive the address and port of the instance of the design application sent by the cloud mobile phone 102;

A connection module 1204, configured to establish a connection with the instance of the design application according to the address and port of the instance;

The communication module 1202 is further configured to receive the interface data stream sent by the cloud phone 102;

an interface generation module 1206, configured to generate a design interface according to the interface data flow;

The communication module 1202 is further configured to send interaction information 2 to the cloud mobile phone 102;

The communication module 1202 is further configured to receive the video stream sent by the cloud mobile phone 102;

The display module 1208 is configured to display the design picture on the design interface according to the video stream.

The connection module 1204 in the second client is an optional module. When a connection is established between the second client and the cloud phone in advance, the second client does not need to perform the step of establishing the connection.

Optionally, the second client may further include a rendering module and a confluence module. The communication module 1202 is specifically used to receive the first rendered video stream rendered by the cloud mobile phone 102 according to the cloud rendering instruction stream, the communication module 1202 is also used to receive the terminal rendering instruction stream sent by the cloud mobile phone 102, and the rendering module is used to render according to the terminal rendering. The instruction stream is rendered to obtain a second rendered video stream, and the confluence module is configured to combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream. Correspondingly, the display module 1208 may display the design picture on the design interface according to the third rendered video stream.

The example of the present application also discloses a design method based on a cloud mobile phone. The design method is applied to a cloud mobile phone, and specifically includes the following steps:

Step 1: the cloud phone 102 receives the design application startup request routed by the cloud phone management service node 102;

Step 2: The cloud phone 102 starts the design application (for example, design application 1), and generates an instance of the design application;

Step 3: The cloud phone 102 returns the address and port of the instance to the first client, the second client and the third client;

Step 4: The cloud phone 102 establishes connections with the first client, the second client and the third client respectively;

Step 5: the cloud phone 102 receives the design document creation request sent by the first client;

Step 6: The instance of the design application on the cloud phone 102 creates a design document, and generates an interface data stream;

Wherein, steps 1 to 6 are an optional implementation manner in which the cloud mobile phone 102 generates an interface data stream. In some possible implementations, for example, when the cloud phone 102 establishes a connection with the cloud phone in advance, the cloud phone 102 can directly send a design application startup request, start the design application, and generate an instance of the design application, and the instance of the design application can automatically create a design document , and generate the interface data flow of the corresponding design interface.

Step 7: The cloud phone 102 returns the interface data stream to the first client, the second client and the third client;

Step 8: The cloud phone 102 receives the interaction information 1 sent by the first client, the interaction information 2 sent by the second client, and the interaction information 3 sent by the third client;

Step 9: The cloud phone 102 runs an instance of the design application according to the interaction information, and renders the video stream according to the design application running process;

Step 10: The cloud phone 102 returns the video stream to the first client, the second client and the third client.

Optionally, the cloud phone 102 generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the interaction information. In step 9, the cloud phone 102 performs rendering according to the above cloud rendering instruction stream to obtain a first rendered video stream. The cloud phone 102 also returns the end rendering instruction stream to the first client, the second client and the third client, so as to instruct the first client, the second client and the third client to perform rendering according to the end rendering instruction stream, and obtain The second renders the video stream. Correspondingly, in step 10, the video stream returned by the cloud mobile phone 102 to the first client, the second client and the third client is specifically the above-mentioned first rendered video stream, the above-mentioned first client, second client and third The three clients may combine the first rendered video stream and the second rendered video stream to obtain a third rendered video stream.

Please refer to FIG. 13 below. FIG. 13 is a schematic diagram of a device structure of a cloud mobile phone according to an embodiment of the present application. As shown in FIG. 13 , the cloud mobile phone includes:

A communication module 1302, configured to receive a design application start request routed by the cloud mobile phone management service node 102;

an instance generation module 1304, configured to start a design application (for example, design application 1), and generate an instance of the design application;

The communication module 1302 is further configured to return the address and port of the instance to the first client, the second client and the third client;

a connection module 1306, configured to establish connections with the first client, the second client and the third client respectively;

The communication module 1302 is further configured to receive a design document creation request sent by the first client;

The interface generation module 1308 is used to create a design document and generate an interface data flow;

The communication module 1302 is further configured to return the interface data stream to the first client, the second client and the third client;

The communication module 1302 is also used for the interaction information 1 sent by the first client, the interaction information 2 sent by the second client, and the interaction information 3 sent by the third client;

a rendering module 1310, configured to run an instance of the design application according to the interaction information, and render a video stream according to the design application running process;

The communication module 1302 is further configured to return video streams to the first client, the second client and the third client.

The instance generation module 1304, the connection module 1306, and the interface generation module 1308 are optional modules. In some embodiments, the cloud phone may not execute the steps corresponding to the above modules.

Optionally, the cloud phone further includes an instruction generation module, configured to generate a cloud rendering instruction stream and a terminal rendering instruction stream according to the interaction information. The rendering module is specifically configured to perform rendering according to the cloud rendering instruction stream to obtain the first rendered video stream. The communication module 1302 is further configured to return the end rendering instruction stream to the first client, the second client and the third client, so as to instruct the first client, the second client and the third client to perform rendering according to the end rendering instruction stream , to get the second rendered video stream. Correspondingly, the video stream returned by the communication module 1302 to the first client, the second client and the third client is specifically the above-mentioned first rendered video stream, and the above-mentioned first client, second client and third client may The first rendered video stream and the second rendered video stream are combined to obtain a third rendered video stream.

FIG. 13 illustrates an example of the first terminal, the second terminal, and the third terminal participating in the collaborative design. In some possible implementations, the terminals participating in the collaborative design may be the first terminal and the second terminal, or more terminal, which is not limited in this embodiment of the present application.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium includes instructions, when the instructions are executed on the computing device, the computing device causes the computing device to execute the first client, the second client, the third client, or the cloud mobile phone method.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware. Special components, etc. to achieve. Under normal circumstances, all functions completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuit, etc. However, a software program implementation is a better implementation in many cases for this application. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that make contributions to the prior art. The computer software products are stored in a readable storage medium, such as a floppy disk of a computer. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to enable a computer device (which may be a personal computer, training device, or network device, etc.) to execute the various embodiments of the application. method.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be retrieved from a website, computer, training device, or data Transmission from the center to another website site, computer, training facility or data center via wired (eg coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

Claims

A design system, comprising: a cloud mobile phone and at least one client, wherein the cloud mobile phone is arranged on the network side;

The at least one client runs on a terminal, respectively, for receiving the interface data stream sent by the cloud mobile phone, generating a design interface according to the interface data stream, and displaying the design interface on the respective terminals;

The cloud phone is configured to receive at least one interaction information sent by one or any combination of the at least one client, run a design application according to the at least one interaction information, and render a video stream according to the design application running process; wherein, The at least one interaction information comes from a design operation triggered by one or any combination of users in the at least one client through a design interface displayed on the respective terminal being operated;

The at least one client is further configured to receive the video stream, and display a design picture on the design interface according to the video stream.
The design system according to claim 1, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process;

The cloud phone uses the rendering resource to render the video stream according to the design application running process.
The design system according to claim 2, wherein the cloud phone obtains a target number of rendering resources from a rendering resource pool according to a design application running process, comprising:

The cloud phone determines the type and quantity of rendering resources required for rendering according to the design application running process;

The cloud phone acquires the rendering resources of the target type and the target quantity from the rendering resource pool.
The design system according to claim 2 or 3, wherein the cloud mobile phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process, including:

The cloud phone acquires a target number of rendering resources from the rendering resource pool according to the design application running process and the number of the terminals.
The design system according to any one of claims 1 to 4, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process;

The cloud mobile phone renders a video stream according to the cloud rendering instruction stream to obtain a first rendered video stream;

The cloud phone sends the first rendering video stream and the end rendering instruction stream to the at least one client, where the end rendering instruction stream is used to instruct the at least one client to render a video according to the end rendering instruction stream stream to obtain the second rendered video stream;

The at least one client is specifically used for:

combining the first rendered video stream and the second rendered video stream to obtain a third rendered video stream;

A design picture is displayed on the design interface according to the third rendered video stream.
The design system according to claim 5, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, comprising:

The cloud rendering instruction stream is generated according to the information that the rendering calculation amount is greater than the calculation amount threshold during the running process of the design application, and the end rendering instruction stream is generated according to the information that the rendering calculation amount is less than or equal to the calculation amount threshold value during the running process of the design application.
The design system according to claim 5, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, including:

The cloud rendering instruction stream is generated according to the background information during the running process of the design application, and the end rendering instruction stream is generated according to the action information during the running process of the design application.
The design system according to any one of claims 1 to 7, wherein the terminal comprises a virtual reality terminal or a plane terminal.
The design system according to claim 8, wherein when the terminal includes the virtual reality terminal, the interaction information includes at least one of action information and location information.
The design system according to claim 8 or 9, wherein when the terminal includes a flat terminal, the interaction information includes an operation instruction.
A design method based on a cloud mobile phone, characterized in that it is applied to a design system, the design system includes a cloud mobile phone and at least one client, the at least one client runs on a terminal respectively, and the cloud mobile phone is set on a network side, the method includes:

The at least one client receives the interface data stream sent by the cloud phone, generates a design interface according to the interface data stream, and displays the design interface on the respective terminals;

The cloud phone receives at least one interaction information sent by one or any combination of the at least one client, runs a design application according to the at least one interaction information, and renders a video stream according to the design application running process; wherein the at least one A piece of interaction information comes from a design operation triggered by one or any combination of users of the at least one client through a design interface displayed on the terminal operated by them;

The at least one client receives the video stream, and displays a design picture on the design interface according to the video stream.
The method according to claim 11, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process;

The cloud phone uses the rendering resource to render the video stream according to the design application running process.
The method according to claim 12, wherein the cloud phone acquires a target number of rendering resources from a rendering resource pool according to a design application running process, comprising:

The cloud phone determines the type and quantity of rendering resources required for rendering according to the design application running process;

The cloud phone acquires the rendering resources of the target type and the target quantity from the rendering resource pool.
The method according to claim 12 or 13, wherein the cloud phone obtains a target number of rendering resources from a rendering resource pool according to a design application running process, comprising:

The cloud phone acquires a target number of rendering resources from the rendering resource pool according to the design application running process and the number of the terminals.
The method according to any one of claims 11 to 14, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process;

The cloud mobile phone renders a video stream according to the cloud rendering instruction stream to obtain a first rendered video stream;

The at least one client receives a video stream, and displays a design screen on the design interface according to the video stream, including:

The at least one client receives the first rendered video stream and the end rendering instruction stream;

The at least one client renders a video stream according to the end rendering instruction stream to obtain a second rendered video stream;

The at least one client combines the first rendered video stream and the second rendered video stream to obtain a third rendered video stream;

A design picture is displayed on the design interface according to the third rendered video stream.
The method according to claim 15, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, comprising:

The cloud rendering instruction stream is generated according to the information that the rendering calculation amount is greater than the calculation amount threshold during the running process of the design application, and the end rendering instruction stream is generated according to the information that the rendering calculation amount is less than or equal to the calculation amount threshold value during the running process of the design application.
The method according to claim 15, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, comprising:

The cloud rendering instruction stream is generated according to the background information during the running process of the design application, and the end rendering instruction stream is generated according to the action information during the running process of the design application.
The method according to any one of claims 11 to 17, wherein the terminal comprises a virtual reality terminal or a plane terminal.
The method according to claim 18, wherein when the terminal includes the virtual reality terminal, the interaction information includes at least one of motion information and location information.
The method according to claim 18 or 19, wherein when the terminal includes a flat terminal, the interaction information includes an operation instruction.
A design method based on a cloud mobile phone, characterized in that, applied to a first terminal, the method comprising:

The first terminal sends a design application start request, and the design application start request is used to instruct to start the design application on the cloud phone; the first terminal receives the interface data stream sent by the cloud phone, and according to the interface data stream generating a design interface referenced by the design, and displaying the design interface;

The first terminal sends the first interaction information to the cloud mobile phone, receives the video stream obtained by the cloud mobile phone running the design application according to the first interaction information, and rendering the design application according to the running process of the design application. The flow displays the design screen on the design interface.
A design method based on a cloud mobile phone, characterized in that, applied to a second terminal, the method comprising:

The second terminal receives the interface data stream sent by the cloud phone, generates the design interface referenced by the design according to the interface data stream, and displays the design interface;

The second terminal sends the second interaction information to the cloud mobile phone, and receives the video stream obtained by the cloud mobile phone running the design application according to the second interaction information, and rendering the design application according to the running process of the design application. The flow displays the design screen on the design interface.
A design method based on a cloud mobile phone, characterized in that, applied to a cloud mobile phone, the method comprising:

The cloud phone sends an interface data stream to the first terminal and the second terminal, where the interface data stream is used to generate a design interface;

The cloud phone receives the first interaction information sent by the first terminal and the second interaction information sent by the second terminal, and the first interaction information is triggered by the user of the first terminal through the design interface The design operation, the second interaction information comes from the design operation triggered by the user of the second terminal through the design interface;

The cloud phone runs a design application according to the first interaction information and the second interaction information, and renders a video stream according to the design application running process;

The cloud phone sends the video stream to the first terminal and the second terminal.
The method according to claim 23, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process;

The cloud phone renders the video stream by using the rendering resource according to the running process of the design application.
The method according to claim 24, wherein the cloud phone obtains a target number of rendering resources from a rendering resource pool according to a design application running process, comprising:

The cloud phone determines the type and quantity of rendering resources required for rendering according to the design application running process;

The cloud phone acquires the rendering resources of the target type and the target quantity from the rendering resource pool.
The method according to claim 24 or 25, wherein the cloud phone obtains a target number of rendering resources from the rendering resource pool according to the design application running process, comprising:

The cloud phone acquires a target number of rendering resources from the rendering resource pool according to the design application running process and the number of the terminals.
The method according to any one of claims 23 to 26, wherein the cloud phone renders the video stream according to the design application running process, comprising:

The cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to the design application running process;

The cloud mobile phone renders a video stream according to the cloud rendering instruction stream to obtain a first rendered video stream;

The cloud mobile phone sends the video stream to the first terminal and the second terminal, including:

sending, by the cloud phone, the first rendered video stream to the first terminal and the second terminal;

The method also includes:

The cloud phone sends the end-rendering instruction stream to the first terminal and the second terminal, and the end-rendering instruction stream is used to instruct the first terminal and the second terminal to render the video stream to obtain the second terminal. Render the video stream.
The method according to claim 27, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, comprising:

The cloud rendering instruction stream is generated according to the information that the rendering calculation amount is greater than the calculation amount threshold during the running process of the design application, and the end rendering instruction stream is generated according to the information that the rendering calculation amount is less than or equal to the calculation amount threshold value during the running process of the design application.
The method according to claim 27, wherein the cloud mobile phone generates a cloud rendering instruction stream and a terminal rendering instruction stream according to a design application running process, comprising:

The cloud rendering instruction stream is generated according to the background information during the running process of the design application, and the end rendering instruction stream is generated according to the action information during the running process of the design application.
A first client, characterized in that the first client comprises:

a communication module, configured to send a design application start request, where the design application start request is used to instruct to start the design application on the cloud phone;

The communication module is further configured to receive the interface data stream sent by the cloud mobile phone;

an interface generation module, configured to generate the design interface referenced by the design according to the interface data flow;

a display module for displaying the design interface;

The communication module is further configured to send first interaction information to the cloud mobile phone, and receive a video stream obtained by the cloud mobile phone running the design application according to the first interaction information, and rendering the design application according to the running process of the design application;

The display module is further configured to display the design picture on the design interface according to the video stream.
A second client, characterized in that the second client comprises:

The communication module is used to receive the interface data stream sent by the cloud mobile phone;

an interface generation module for generating a design interface referenced by the design according to the interface data flow;

a display module for displaying the design interface;

a sending module, configured to send the second interaction information to the cloud mobile phone, and receive the video stream obtained by the cloud mobile phone running the design application according to the second interaction information, and rendering the design application according to the running process of the design application;

A display module, configured to display the design picture on the design interface according to the video stream.
A cloud mobile phone, characterized in that the cloud mobile phone comprises:

a communication module, configured to send an interface data stream to the first terminal and the second terminal, where the interface data stream is used to generate a design interface;

The communication module is further configured to receive first interaction information sent by the first terminal and second interaction information sent by the second terminal, where the first interaction information comes from the user of the first terminal through the the design operation triggered by the design interface, and the second interaction information comes from the design operation triggered by the user of the second terminal through the design interface;

a rendering module, configured to run a design application according to the first interaction information and the second interaction information, and render a video stream according to the design application running process;

The communication module is further configured to send the video stream to the first terminal and the second terminal.
A first terminal, characterized in that the first terminal includes a processor and a memory;

The processor is configured to execute the instructions stored in the memory to cause the first terminal to perform the method of claim 21 .
A second terminal, characterized in that the second terminal includes a processor and a memory;

The processor is configured to execute instructions stored in the memory to cause the second terminal to perform the method of claim 22.
A cloud phone, characterized in that the cloud phone includes a processor and a memory;

The processor is configured to execute the instructions stored in the memory, so that the cloud phone executes the method according to any one of claims 23 to 29.
A computer-readable storage medium, characterized by comprising instructions, the instructions instructing the first terminal to execute the method of claim 21 .
A computer-readable storage medium, characterized by comprising instructions, the instructions instructing the first terminal to execute the method of claim 22 .
A computer-readable storage medium, characterized by comprising an instruction, the instruction instructing the first terminal to execute the method according to any one of claims 23 to 29.