WO2017088484A1

WO2017088484A1 - Cloud computing based real-time off-screen rendering method, apparatus and system

Info

Publication number: WO2017088484A1
Application number: PCT/CN2016/088767
Authority: WO
Inventors: 罗涛; 杨磊; 张微; 陈乐吉; 杨学亮; 周益; 曾锦平; 邱泳天
Original assignee: 成都赫尔墨斯科技有限公司
Priority date: 2015-11-24
Filing date: 2016-07-06
Publication date: 2017-06-01
Also published as: CN105488837B; CN105488837A

Abstract

Disclosed are a cloud computing based real-time off-screen rendering method, apparatus and system. The method comprises: after receiving a cloud rendering service request sent by a client, enabling a rendering resource and creating a direct connection link between a client terminal and the rendering resource, so that the rendering resource directly processes the cloud rendering service request; and the rendering resource performing real-time rendering and directly sending a rendering result generated in real time to the client, so as to enable the client to receive the rendering result in real time and display a picture according to the rendering result. Numerous client devices that are online at the same time can be supported, massive rendering service requests of the client devices can be born, and all online rendering tasks are independent mutually, so that the communication load of a cloud end is reduced and the time delay of a cloud rendering service is reduced.

Description

Cloud-based real-time off-screen rendering method, device and system

Technical field

The present invention relates to the field of cloud rendering, and in particular, to a cloud computing-based real-time off-screen rendering method, apparatus and system.

Background technique

With the development of the customer terminal operating system, mobile phones and tablets have now become more comprehensive and intelligent. Mobile phones and tablets based on IOS, Android and Windows operating systems are becoming more and more popular in people's lives. These smartphones and tablets Powerful features also bring a lot of convenience to people. But limited by the size and power consumption of smartphones and tablets, and the lightness of their smartphones, smartphones and tablets (such as computing and rendering processors) are not powerful enough to be mobile and portable on mobile phones or tablets. Rendering of large 3D high dynamic scenes on the terminal device.

At the same time, with the rapid development of cloud computing and the low cost requirements of IT operation and maintenance, various cloud computing related services such as Iaas (infrastructure as a service), Paas (platform as a service), and Saas (software as a service) have emerged. The general principle is to use cloud computing's powerful virtualization technology to pool computing, storage and network resources, to achieve arbitrary segmentation of resources, to achieve high availability of computing, storage and network resources, and overall improvement The utilization of resources. In the traditional IT industry, 3D large-scale high-dynamic scene rendering services have been the demand of many industries. The popularity of cloud computing services and the demand for large-scale 3D scene rendering have naturally created cloud rendering services.

The existing cloud rendering is roughly divided into two categories: one is online rendering, and the other is offline rendering. Both of these types use the way of cloud rendering, and the implementation of the prior art can only provide a delay rendering knot. If it is mainly used for movie special effects and animation rendering, it can not meet the real-time interoperability requirements of 3D large-scale high-dynamic scene rendering pictures on the mobile device side. The displayed 3D special effects are poor in quality and cannot provide users with high dynamics in the client terminal. A good user experience for 3D real-time scene services.

A Chinese invention patent (Application Publication No.: CN103313120A) discloses a method for displaying a picture, the method comprising the steps of: acquiring picture information of a rendering entity; transmitting the picture information to a cloud, and using the cloud according to the picture information The rendering entity is rendered to obtain a rendering result; the rendering result returned by the cloud is obtained, and the screen is displayed according to the rendering result.

The invention reduces the need for the processing capability of the portable client terminal by performing the rendering operation in the cloud, and can be applied to various types of terminals, even the low-end client terminal is equally applicable, and with the strong processing capability of the cloud, Cloud rendering display of 3D scenes can be performed on the client terminal device.

However, for a large 3D scene, in the process of transmitting the image information of the rendering entity to the cloud and receiving the rendering result returned by the cloud, due to the increased amount of data transmitted, under the same network bandwidth, the cost is The time will increase, ie the delay will increase. Moreover, the number of clients supporting simultaneous online access will also be reduced. Once the communication load of the cloud reaches a critical value, the rendering task of the excess client will be suspended, and the delay is more serious.

Summary of the invention

The object of the present invention is to overcome the above-mentioned deficiencies in the prior art, and provide a real-time off-screen rendering method based on cloud computing, which can support a plurality of simultaneous client devices and can carry a rendering service request of a large number of client devices. Moreover, all online rendering tasks are independent of each other, which not only reduces the communication load of the cloud, but also reduces the latency of the cloud rendering service.

According to an aspect of the present invention, a cloud computing-based real-time off-screen rendering method is provided, including:

After receiving the cloud rendering service request sent by the client, enabling the rendering resource and creating the rendering resource a direct link with the client;

After the direct link is successfully created, the rendering resource is rendered in real time according to the cloud rendering service request, and the rendering result generated in real time is sent to the client through the direct link;

After receiving the rendering result in real time, the client displays the screen in real time according to the rendering result.

According to a specific implementation manner, after receiving the cloud rendering service request sent by the client, first accessing the rendering resource list to query whether the rendering resource is available;

If so, the optimal rendering resource is enabled; otherwise, no rendering resource information is available to the client.

According to a specific implementation manner, when the rendering resource is enabled, an address of the rendering resource is obtained, and a communication port is allocated to the client, and an address of the rendering resource is bound to a communication port of the client. And creating the rendering resource and the direct link of the client.

According to a specific implementation manner, after the direct link is successfully created, according to the rendering entity described in the cloud rendering service request, the rendering resource first downloads resource data of the rendering entity;

After the resource data of the rendering entity is downloaded, the rendering resource renders the resource data of the rendering entity in real time according to the rendering content described in the cloud rendering request, and is in a frame buffer of the rendering resource. Generate rendering results in real time;

The rendering result in the frame buffer is obtained in real time, and is compression-encoded, and then sent to the client through the direct link.

According to a specific implementation manner, the transmission frequency of the rendering result is adjusted in real time according to the communication status on the direct link and the operation of the client.

According to a specific implementation manner, before receiving the cloud rendering service request sent by the client, the client is first sent to send a user verification request;

Parsing user identity information included in the user authentication request, and verifying that the user identity information is consistent with the pre-bound user identity information;

If they are consistent, the identification information is returned to the client, and a heartbeat connection with the client is established. Otherwise, the identification information is not returned to the client.

According to a specific implementation manner, after returning the identification information to the client, if the heartbeat packet sent by the client is received, determining that the heartbeat connection is successfully established, and adding the user to the online list of the user; If the heartbeat packet sent by the client is not received, it is determined that the heartbeat connection establishment fails, and the user is offline;

And after the heartbeat connection is successfully established, if the offline request sent by the client is received, the heartbeat connection is disconnected, the user is removed from the online list of the user, and the created Direct link and enabled rendering resources.

According to a specific implementation manner, after establishing a heartbeat connection with the client, receiving a heartbeat packet sent by the client according to a certain frequency to monitor a login status of the user, and each time after receiving one of the heartbeat packets, immediately Start timing at zero;

If the timing time is greater than or equal to the drop threshold time, the user is removed from the online list of users, and the created direct link and the enabled rendering resource are released.

According to a specific implementation manner, according to the online peak period of the client, a certain number of rendering resources are opened in advance, and in the online trough period of the client, a certain number of rendering resources are closed.

According to an aspect of the present invention, a cloud computing-based real-time off-screen rendering apparatus is further provided, including:

a communication relay unit, configured to receive a cloud rendering service request sent by the client;

a rendering resource unit for providing a rendering resource for rendering services;

a resource management unit, configured to manage and enable the rendering resource;

a service scheduling unit, configured to create a direct link between the rendering resource and the client.

According to a specific implementation manner, the resource management unit includes:

a rendering resource list sub-unit, configured to save an address of each rendering resource in the rendering resource unit, and perform statistics on the working state of each rendering resource according to a certain frequency, and establish a rendering resource list according to the statistical result;

a rendering resource enabling subunit, configured to query whether there is a rendering resource available in the rendering resource list, and enable an optimal rendering resource when the rendering resource is available; otherwise, return no available resource information to the client end;

And a rendering resource address sending subunit, configured to send an address of the rendering resource enabled sub-unit enabled rendering resource to the service scheduling unit, to be used by the service scheduling unit to create the direct link.

According to a specific implementation manner, the service scheduling unit includes:

a communication link management subunit, configured to allocate a communication port to the client, and after receiving the address of the rendering resource sent by the rendering resource address sending subunit, communicate the address of the rendering resource with the client Port binding to create the direct link of the rendering resource to the client.

According to a specific implementation manner, the rendering resource includes:

a resource data downloading subunit, configured to download resource data of the rendering entity according to the rendering entity described in the cloud rendering service request, and perform downloading of the resource data;

And a rendering execution subunit, configured to perform real-time rendering on the resource data according to the rendering content described in the cloud rendering request;

a frame buffer subunit, configured to cache a rendering result generated by the rendering execution subunit in real time;

a compression coding subunit, configured to acquire a rendering result in the frame buffer subunit in real time, and compress and encode the rendering result;

a rendering result sending subunit, configured to send, in real time, a rendering result processed by the compression encoding subunit to the client through the direct link, and according to the communication status and location of the direct link The operation of the client adjusts the transmission frequency of the rendering result in real time.

According to a specific embodiment, the device further includes:

a storage unit, configured to pre-store the rendering entity resource data of the user;

a database management unit, configured to manage rendering entity resource data of the user, and when the rendering resource requests to download resource data, send a storage address of the resource data in the storage unit to the rendering resource, where The rendering resource downloads the resource data.

According to a specific embodiment, the device comprises a user management unit, and

The communication relay unit is further configured to receive a user verification request and an offline request sent by the client;

The database management unit is further configured to manage identity information of the user, and store user identity information pre-bound by the user;

The service scheduling unit is further configured to forward the user verification request and the offline request to the user management unit.

According to a specific implementation manner, the user management unit includes:

An identity information verification subunit, configured to parse user identity information included in the user authentication request, and send the user identity information to the database management unit for comparison, and verify whether the user identity information is pre-bound User identity information is consistent;

a heartbeat connection subunit, configured to return identification information to the client according to whether the user identity information is consistent with pre-bound user identity information, and establish a heartbeat connection with the client; and return the identification information After receiving the heartbeat packet sent by the client, if the heartbeat packet is received, determining that the heartbeat connection is successfully established, and if the heartbeat packet is not received, determining that the heartbeat connection fails The user is offline; and after the heartbeat connection is successfully established, the user is received Deactivating the heartbeat connection when the offline request sent by the client is performed;

a user online list management sub-unit, configured to establish a user online list, and when the heartbeat connection is successfully established, add the user to the online list of the user; and when receiving the offline request sent by the client, Removing the user from the online list of users releases the created direct link and enabled rendering resources.

According to a specific implementation, the heartbeat connection subunit is further configured to start timing from zero after receiving one of the heartbeat packets, and drop the user when the timing time is greater than the offline threshold time Sending line information to the user online list management sub-unit, causing the user online list management sub-unit to remove the user from the online list of users, releasing the created direct link and enabled Render resources.

According to a specific embodiment, the device further includes:

The intelligent management unit is configured to open a certain amount of rendering resources in advance during the online peak period of the user, and close a certain amount of rendering resources during the online trough period of the user.

According to an aspect of the present invention, a cloud computing-based real-time off-screen rendering system is further provided, including a client, a public network, and a plurality of user resource pools, wherein the user resource pools are respectively bound with a unique public network. IP, the client accesses the user resource pool by linking to the public network IP;

The user resource pool includes a router, a rendering resource server, a resource management server, and a service scheduling server;

The router is configured to receive, by the client, a cloud rendering service request, and forward the cloud rendering service request to the service scheduling server;

The service scheduling server is configured to create a direct link between the rendering resource and the client, and forward the cloud rendering service request to the rendering resource server and the resource management server;

The rendering resource server is configured to provide a rendering resource for rendering services;

The resource management server is configured to manage and enable the rendering resource.

According to a specific implementation manner, the resource management server saves an address of each rendering resource in the rendering resource server, and performs statistics on the working state of each rendering resource according to a certain frequency, and establishes a rendering resource list according to the statistical result;

Receiving, by the resource management server, the cloud rendering service request forwarded by the service scheduling server, querying whether the rendering resource is available in the rendering resource list, and if yes, enabling the optimal rendering resource, otherwise returning no available Resource information to the client;

The resource management server enables a rendering resource and sends an address of the rendering resource to the service scheduling server for use by the service scheduling server to create the direct link.

According to a specific implementation manner, the service scheduling server receives an address sent by the resource management server, and allocates to the client a communication port, and binds the address of the rendering resource to the communication port of the client. And creating the rendering resource and the direct link of the client.

According to a specific implementation manner, the rendering resource downloads resource data of the rendering entity according to a rendering entity described in the cloud rendering service request; and downloads according to the rendering content described in the cloud rendering request. The resource data is rendered in real time, and the rendering result generated in real time in the frame buffer;

The rendering resource performs compression encoding on the rendering result in the frame buffer, and sends the compression-encoded rendering result to the client through the direct link, and according to the communication status of the direct link And the operation of the client, adjusting the sending frequency of the rendering result in real time.

According to a specific embodiment, the system includes a database server and a storage server; the user resource pool shares the database server and the storage server;

The storage server is configured to pre-store the rendering entity resource data of the user;

The database management server is configured to manage the rendering entity resource data of the user, and send the storage address of the resource data in the storage server to the rendering resource when the rendering resource requests to download the resource data, The resource data is downloaded for the rendering resource.

According to a specific embodiment, the system includes: a user management server, and,

The router is further configured to receive a user verification request sent by the client;

The database management server is further configured to manage identity information of the user, and store user identity information pre-bound by the user;

The service scheduling server is further configured to forward the user verification request to the user management server.

According to a specific implementation manner, the user management server receives a user authentication request forwarded by the service scheduling server, and parses user identity information included in the user verification request, and sends the user identity information to the database. The management unit performs an comparison to verify whether the user identity information is consistent with the pre-bound user identity information;

The user management server returns the identification information to the client according to whether the user identity information is consistent with the pre-bound user identity information, so that the client establishes a heartbeat connection with the user management server;

After returning the identification information to the client, the user management server receives the heartbeat packet sent by the client, determines that the heartbeat connection is successfully established, and adds the user to the online list of users; If the user management server does not receive the heartbeat connection, it is determined that the heartbeat connection fails, and the user is offline;

After the heartbeat connection is successfully established, the user management server receives the offline request sent by the client, disconnects the heartbeat connection, removes the user from the online list of the user, and releases the The direct link is created and the enabled rendering resources are deactivated.

According to a specific implementation manner, after receiving the heartbeat packet, the user management server starts timing from zero;

If the timing time is greater than the drop threshold time, the user management server removes the user from the online list of users, and releases the created direct link and disables the enabled rendering Dye resources.

According to a specific embodiment, the system comprises:

The intelligent management server is configured to open a certain number of rendering resources in advance during a user online peak period, and close a certain number of rendering resources during a user online trough period.

Compared with the prior art, the beneficial effects of the present invention:

After accepting the cloud rendering service request sent by the client, the invention enables the rendering resource and creates a direct link of the client terminal and the rendering resource, so that the rendering resource directly processes the cloud rendering service request, and the rendering resource is real-time rendered and generated in real time. The rendering result is directly sent to the client, so that the client terminal receives the rendering result in real time and displays the image according to the rendering result, which can not only support multiple online client devices, but also can carry a large number of rendering service requests of the client device, and all online The rendering tasks are independent of each other, reducing the communication load of the cloud and reducing the latency of the cloud rendering service.

At the same time, according to the communication condition on the direct link and the operation of the client, the present invention dynamically adjusts the transmission frequency of the rendering result in real time and compresses the rendering result to different degrees, and further reduces the delay of the rendering service. Real-time interoperability between users and large 3D scenes is achieved, and the client terminal device provides users with a good user experience of highly dynamic 3D real-time scene services.

Moreover, the present invention is also capable of recording the time rule of the user requesting the cloud rendering service, and according to the time rule, reasonably controlling the standby quantity of the rendering resource, reducing energy consumption, saving energy and environmental protection while ensuring sufficient user resources.

BRIEF DESCRIPTION OF THE DRAWINGS:

1 is a schematic diagram showing the basic flow of a cloud rendering service of the method of the present invention;

2 is a schematic flowchart of a method for enabling rendering resources according to the method of the present invention;

3 is a schematic flowchart of real-time rendering of a rendering resource of the method of the present invention;

4 is a schematic diagram of a basic flow of verifying user identity information by the method of the present invention;

5 is a schematic flow chart of managing a heartbeat connection by the method of the present invention;

6 is a schematic flow chart of determining a user login status of the method of the present invention;

Figure 7 is a schematic structural view of a first embodiment of the apparatus of the present invention;

Figure 8 is a schematic structural view of a second embodiment of the apparatus of the present invention;

Figure 9 is a schematic structural view of a third embodiment of the apparatus of the present invention;

Figure 10 is a schematic structural view of a fourth embodiment of the apparatus of the present invention;

Figure 11 is a schematic view showing the structure of a fifth embodiment of the apparatus of the present invention;

Figure 12 is a schematic structural view of a sixth embodiment of the apparatus of the present invention;

Figure 13 is a schematic structural view of a first embodiment of the system of the present invention;

Figure 14 is a schematic structural view of a second embodiment of the system of the present invention;

Figure 15 is a schematic view showing the structure of a third embodiment of the system of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the specific embodiments. However, the scope of the above-mentioned subject matter of the present invention should not be construed as being limited to the following embodiments, and the technology implemented based on the present invention is within the scope of the present invention.

A basic flow diagram of the cloud rendering service of the method of the present invention shown in FIG. 1; on a client with an application having a 3D scene, the user performs certain operations on the application of the 3D scene, such as rotating, moving, and scaling the rendering entity. Operation, etc., when these related operations are recognized by the 3D scene application, a cloud rendering service request including related rendering content and rendering entity information is generated. Therefore, after the user completes the related operations on the client, the client sends a relevant cloud rendering service request to request the cloud rendering service.

The real-time off-screen rendering method based on cloud computing of the invention comprises:

After receiving the cloud rendering service request sent by the client, enable the rendering resource and create a direct link between the rendering resource and the client.

After the direct link is successfully created, the rendering resource is rendered in real time according to the cloud rendering service request, and the real-time generated rendering result is sent to the client through the direct link.

After the rendering result is received in real time, the client displays the picture in real time according to the rendering result.

By creating a direct link between the rendering resource and the client, the present invention makes all online rendering tasks independent of each other, reduces the communication load of the cloud, and reduces the delay of the cloud rendering service.

A schematic diagram of a process for enabling a rendering resource according to the method of the present invention shown in FIG. 2; after receiving a cloud rendering service request sent by a client, first accessing a rendering resource list to query whether a rendering resource is available.

If so, enable the optimal rendering resource; otherwise, no rendering resource information is available to the client.

Specifically, when the rendering resource list has available rendering resources, since the rendering resources are distributed on different physical machines, and some rendering resources are relatively slightly used, the optimal rendering resource is enabled, that is, the running load is determined to be low. The physical machine, and the use of relatively low-frequency rendering resources on the physical machine, which is beneficial to balance the use of rendering resources, rationally plan the enabling of rendering resources, and avoid the idleness of rendering resources.

Moreover, when the rendering resource is enabled, the address of the rendering resource is also obtained, and a communication port is allocated to the client, and the address of the rendering resource is bound to the communication port of the client to create a direct link between the rendering resource and the client.

When no rendering resources are available in the rendering resource list, the cloud rendering service request is suspended, and the rendering service is provided when there are available resources.

The schematic diagram of the real-time rendering of the resource rendering is performed in conjunction with the method of the present invention shown in FIG. 3; after the direct link is successfully created, the rendering resource first downloads the resource data of the rendering entity according to the rendering entity described in the cloud rendering service request.

After the resource data of the rendering entity is downloaded, according to the rendering content described in the cloud rendering request, the rendering resource renders the resource data of the rendering entity in real time, and generates the rendering result in real time in the frame buffer of the rendering resource.

The rendering result in the frame buffer is obtained in real time, and is compressed and encoded, and then sent to the client through the direct link.

Specifically, the rendering resource is a resource with GPU hardware, and the rendering of the data is mainly performed by the GPU. The process of data rendering can be either a traditional rendering pipeline process or a programmable rendering pipeline process, and the final rendering results are placed in the frame buffer.

Moreover, when the rendering result is sent to the client, the transmission frequency of the rendering result is adjusted in real time according to the communication status on the direct link and the operation of the client. When the communication status on the direct link is not good, the transmission frequency of the rendering result is reduced, so as to avoid data congestion and communication overload when the client receives, affecting the user experience, and when the user performs related operations on the client, according to The related operation adjusts the sending frequency in real time, and after the client completes the related operation, sends the rendering result to the client at the adjusted sending frequency, so that the user obviously feels the rendering effect. Through such a processing manner as described above, the present invention reduces the delay of the rendering service, realizes real-time interoperability between the user and the large 3D scene, and provides the user with a good user experience of the highly dynamic 3D real-time scene service in the client terminal device.

A basic flow diagram for verifying user identity information in conjunction with the method of the present invention shown in FIG. 4; before receiving a cloud rendering service request sent by a client, the receiving client sends a user authentication request.

By parsing the user identity information contained in the user authentication request, and comparing the user identity information with the pre-bound user identity information to verify whether the user identity information is pre-bound The identity information of the household is consistent.

A schematic diagram of a process for managing a heartbeat connection according to the method of the present invention shown in FIG. 5; after receiving the identification information to the client, if the heartbeat packet sent by the client is received, it is determined that the heartbeat connection is successfully established, and the user is added to the online list of the user; If the heartbeat packet sent by the client is not received, it is determined that the heartbeat connection establishment fails, and the user is offline;

And, after the heartbeat connection is successfully established, if the offline request sent by the client is received, the heartbeat connection is disconnected, the user is removed from the online list of the user, and the created direct link and the enabled rendering resource are released. .

A schematic diagram of a process for determining a user login status in conjunction with the method of the present invention shown in FIG. 6; after establishing a heartbeat connection with a client, receiving a heartbeat packet sent by the client at a certain frequency to monitor the login status of the user, and each time a heartbeat packet is received , immediately start timing from zero.

If the timing is greater than or equal to the drop threshold time, the user is considered to be offline, the user is removed from the user online list, and the created direct link and the enabled rendering resource are released.

The real-time off-screen rendering method based on cloud computing according to the invention also opens a certain number of rendering resources in advance according to the rule that the client joins the online list of the user, and turns off a certain number of renderings in the online trough period of the client. Resources.

Based on the same inventive concept, the present invention also provides a real-time off-screen rendering device based on cloud computing. A schematic diagram of a first embodiment of the apparatus of the present invention shown in FIG. 7; the real-time off-screen rendering apparatus based on cloud computing of the present invention comprises: a communication relay unit, a rendering resource unit, a resource management unit, and a service scheduling unit.

The communication relay unit is configured to receive a cloud rendering service request sent by the client, and receive the received The cloud rendering service request is forwarded to the business scheduling unit. The service scheduling unit forwards the cloud rendering service request to the resource management unit, the resource management unit manages and enables the rendering resource, the rendering resource is provided by the rendering resource unit, and the service scheduling unit creates a direct link between the rendering resource and the client, and renders by rendering the resource. service.

A schematic structural diagram of a second embodiment of the apparatus of the present invention shown in FIG. 8; wherein the resource management unit comprises: a rendering resource list subunit, a rendering resource enabling subunit, and a rendering resource address transmitting subunit.

The rendering resource list sub-unit saves the address of each rendering resource in the rendering resource unit, and statistics the working state of each rendering resource according to a certain frequency, and establishes a rendering resource list according to the statistical result.

The rendering resource enable subunit is used to query whether there are available rendering resources in the rendering resource list, and enable the optimal rendering resource when there are available rendering resources, otherwise, no available resource information is returned to the client.

The rendering resource address sending subunit is configured to send the address of the rendering resource enabled by the rendering resource enabling subunit to the service scheduling unit for use by the service scheduling unit to create the direct link.

Moreover, the service scheduling unit includes a communication link management sub-unit. When the direct link is created, the communication link management sub-unit allocates a communication port to the client, and after receiving the address of the rendering resource sent by the rendering resource address sending sub-unit Bind the address of the rendering resource to the communication port of the client to create a direct link between the rendering resource and the client.

FIG. 9 is a schematic structural diagram of a third embodiment of the apparatus of the present invention; the cloud-based real-time off-screen rendering apparatus of the present invention further includes a storage unit and a database management unit. The storage unit is configured to pre-store the rendering entity resource data of the user, and the database management unit is configured to manage the rendering entity resource data of the user, and send the resource data in the storage unit when the rendering resource requests to download the resource data. The storage address in the storage resource to the rendering resource for downloading the resource data.

In this embodiment, the rendering resource includes a resource data downloading subunit, a rendering execution subunit, a frame buffer subunit, a compression encoding subunit, and a rendering result sending subunit.

The resource data downloading subunit is configured to request to download resource data of the rendering entity according to the rendering entity described in the cloud rendering service request, and perform downloading of the resource data.

A rendering execution sub-unit for rendering the resource data in real time according to the rendered content described in the cloud rendering request.

The frame buffer sub-unit is used to cache the rendering result generated by the rendering execution sub-unit in real time.

The compression coding sub-unit is configured to acquire the rendering result in the frame buffer sub-unit in real time, and compress and encode the rendering result.

The rendering result sending subunit is configured to send the rendering result processed by the compression encoding subunit in real time to the client through the direct link, and adjust the rendering result in real time according to the communication status of the direct link and the operation of the client. Send frequency.

FIG. 10 is a schematic structural diagram of a fourth embodiment of the apparatus of the present invention; the cloud-based real-time off-screen rendering apparatus of the present invention further includes a user management unit, and the communication relay unit is further configured to receive user authentication sent by the client. The request and the offline request; the database management unit is further configured to manage the identity information of the user, and store the user identity information pre-bound by the user; the service scheduling unit is further configured to forward the user verification request and the offline request to the user management unit.

A schematic structural diagram of a fifth embodiment of the apparatus of the present invention shown in FIG. 11; the user management unit includes:

The identity information verification sub-unit is configured to parse the user identity information included in the user verification request, and send the user identity information to the database management unit for comparison, and verify whether the user identity information is consistent with the pre-bound user identity information.

The heartbeat connection sub-unit, according to whether the user identity information is consistent with the pre-bound user identity information, returns the identification information to the client, and establishes a heartbeat connection with the client; and after returning the identification information to the client, the receiving client sends the If the heartbeat packet is received, the heartbeat connection is successfully determined. If the heartbeat packet is not received, the heartbeat connection is determined to be failed, and the user is offline; and after the heartbeat connection is successfully established, when the offline request sent by the client is received, Disconnect the heartbeat connection.

The user online list management sub-unit is configured to establish a user online list, and when the heartbeat connection is successfully established, add the user to the online list of the user; and when receiving the offline request sent by the client, move the user from the online list of the user In addition, releases the created direct link and enabled render resources.

In this embodiment, the heartbeat connection subunit is further configured to start timing from zero after receiving a heartbeat packet, and send the user drop information to the user online list management subunit when the timing time is greater than the drop threshold time. To remove the user from the user's online list, freeing the created direct link and enabled rendering resources.

A schematic diagram of a structure of a sixth embodiment of the apparatus of the present invention shown in FIG. 12; the real-time off-screen rendering apparatus based on cloud computing of the present invention further includes an intelligent management unit.

The intelligent management unit is configured to open a certain number of rendering resources in advance according to the rule that the user joins the online list of the user, and close a certain number of rendering resources during the online trough period of the user.

Based on the same inventive concept, the present invention further provides a real-time off-screen rendering system based on cloud computing, combined with the structural schematic diagram of the first embodiment of the system of the present invention shown in FIG. 13; the real-time off-screen rendering system based on cloud computing of the present invention The client, the public network, and the user resource pool are bound to each other. The user resource pool is bound to a public IP address. The client accesses the user resource pool by linking to the public network IP address.

And, the user resource pool includes a router, a rendering resource server, a resource management server, and a service scheduling server.

The router is configured to receive a request for sending a cloud rendering service by the client, and forward the cloud rendering service request to the service scheduling server.

A service scheduling server is configured to create a direct link between the rendering resource and the client, and forward the cloud rendering service request to the rendering resource server and the resource management server.

A rendering resource server that provides rendering resources for rendering services.

A resource management server for managing and enabling rendering resources.

Specifically, the resource management server saves the address of each rendering resource in the rendering resource server, and statistics the working state of each rendering resource according to a certain frequency, and establishes a rendering resource list according to the statistical result.

The resource management server receives the cloud rendering service request forwarded by the service scheduling server, and queries whether the rendering resource is available in the rendering resource list. If yes, the optimal rendering resource is enabled, otherwise the available resource information is returned to the client.

The resource management server enables the rendering resource and sends the address of the rendering resource to the service scheduling server for use by the service scheduling server to create a direct link.

The service scheduling server receives the address sent by the resource management server, and allocates a communication port to the client, and binds the address of the rendering resource to the communication port of the client to create a direct link between the rendering resource and the client.

A schematic diagram of the structure of the second embodiment of the system of the present invention shown in FIG. 14; the real-time off-screen rendering system based on cloud computing of the present invention further includes a database server and a storage server.

The user resource pool shares the database server and the storage server, and the storage server is configured to pre-store the rendering entity resource data of the user.

a database management server, configured to manage the rendering entity resource data of the user, and send the storage address of the resource data in the storage server to the rendering resource when the rendering resource requests to download the resource data, For rendering resources to download resource data.

In conjunction with the structural schematic diagram of the third embodiment of the system of the present invention shown in FIG. 15, the real-time off-screen rendering system based on cloud computing of the present invention comprises: a user management server and an intelligent management server.

Specifically, the router is configured to receive a user authentication request sent by the client.

A database management server for managing user identity information and storing user identity information pre-bound by the user.

A service scheduling server for forwarding user authentication requests to the user management server.

The user management server receives the user authentication request forwarded by the service scheduling server, and parses the user identity information included in the user authentication request, sends the user identity information to the database management unit for comparison, and verifies whether the user identity information is associated with the pre-bound user identity. The information is consistent;

After returning the identification information to the client, the user management server receives the heartbeat packet sent by the client, determines that the heartbeat connection is successfully established, and adds the user to the online list of the user; if the user management server does not receive the heartbeat connection, the heartbeat is determined. The connection failed and the user is offline;

After the heartbeat connection is successfully established, the user management server receives the offline request sent by the client, disconnects the heartbeat connection, removes the user from the online list of the user, and releases the created direct link and disables the enabled one. Render resources.

Moreover, after the heartbeat connection between the user management server and the client is successfully established, the client sends the heartbeat packet to the user management server according to a certain frequency. After receiving the heartbeat packet, the user management server starts timing from zero.

If the timing is greater than the drop threshold time, the User Management Server removes the user from the user's online list and releases the created direct link and deactivates the enabled rendering resources.

The intelligent management server is configured to open a certain number of rendering resources in advance according to the rule that the user joins the online list of the user, and close a certain amount of rendering resources in the online trough period of the user.

In order to further explain the cloud computing-based real-time off-screen rendering method, apparatus and system of the present invention, the present invention will be described in more detail below with an example of real-time rendering of a rendering entity teapot.

The user uses the client to send a user authentication request to the user resource pool corresponding to the user through the TCP/IP protocol. In the user resource pool, after receiving the user authentication request, the router forwards the user authentication request to the service scheduling server, and then the service scheduling server forwards it to the user management server.

After receiving the user authentication request, the user management server parses the user identity information in the user authentication request, and compares the user identity information with the user information pre-bound in the database management server, and verifies whether the user identity information is pre-bound User identity information is consistent.

If consistent, the user management server returns unique identification information to the client through the service scheduling server and router. The client establishes a heartbeat connection with the user management server according to the identification information, and the user management server receives the heartbeat packet sent by the client, indicating that the heartbeat connection is successfully established, and the user is added to the user online list.

Then, after the user joins the online list of the user, the client can send the corresponding cloud rendering service request to the user resource pool through a real-time operation. After receiving the cloud rendering service request, the router immediately forwards the user authentication request to the service scheduling server. It is then forwarded by the service scheduling server to the resource management server.

After receiving the cloud rendering service request, the resource management server queries whether there is a rendering resource available in the resource list, and selects a rendering resource with a relatively low activation and use frequency on a physical machine with a lower running load to enable, in response to the cloud. Render a service request.

And, when the rendering resource is enabled, the resource management server will forward the address of the enabled rendering resource. To the service scheduling server, at the same time, the service scheduling server allocates a communication port to the client, and binds the address of the rendering resource to the communication port, thereby establishing a direct link between the rendering resource and the client.

After the direct link is successfully created, the rendering resource requests the database management server to download the resource data of the rendering entity teapot according to the rendering entity teapot described in the cloud rendering service request, and the database management server renders the resource data of the entity teapot in the storage server. The storage address is sent to the rendering resource, and the rendering resource downloads the resource data of the rendering entity teapot according to the storage address of the resource data of the rendering entity teapot in the storage server.

After the resource data of the rendering entity teapot is downloaded, the rendering content requested by the cloud rendering service, such as the rotation, movement and scaling of the rendering entity teapot, is rendered in real time, and the rendering entity teapot is rendered in real time and in the frame buffer of the rendering resource. Produce rendering results. The rendering result in the frame buffer is extracted in real time, and the rendering result is compression-encoded, and after being compression-encoded, it is directly sent to the client through the direct link.

Specifically, in the process of rendering the rendering entity teapot, if the time counted by the user management server after receiving one heartbeat packet is greater than or equal to the drop threshold time, the user is removed from the online list of the user, and the release is created. Direct link and enabled rendering resources. If the client sends an offline request, the user management server disconnects the heartbeat and removes the user from the online list of users, releasing the created direct link and the enabled rendering resources.

In the step of transmitting the compression-encoded rendering result to the client, the rendering resource also adjusts the transmission frequency of the rendering result in real time according to the communication status on the direct link and the operation of the client. The real-time operation of the client is correspondingly described in the rendered content of the cloud rendering service request.

By reasonably managing and enabling the rendering resources, the invention can not only support multiple online client devices, but also can carry a large number of rendering service requests of the client device, and all the online rendering tasks are independent of each other, reducing the cloud's Communication load reduces the latency of cloud rendering services.

At the same time, according to the communication condition on the direct link and the operation of the user on the client, the present invention dynamically adjusts the transmission frequency of the rendering result in real time and compresses the rendering result to different degrees, further reducing the delay of the rendering service. Real-time interoperability between users and large 3D scenes can be realized, and the user terminal equipment provides users with a good user experience of highly dynamic 3D real-time scene services.

One aspect of the invention relates to a non-transitory computer readable medium comprising instruction set encoding adapted for use in the above method, which is operated and/or controlled by a dedicated and integrated software package. Accordingly, the present invention also includes algorithms, computer programs, computer readable media and/or software, implantable and/or executable files on a general purpose computer or workstation equipped with a conventional processor, and for performing one or more Method and/or one or more of the hardware disclosed herein. For example, a computer program or computer readable medium typically includes a set of instructions configured to perform the above when executed by a suitable processing device (eg, a signal processing device such as a microcontroller, microprocessor or DSP device) Method, run, and/or algorithm.

The computer readable medium can include any medium that can be read by a signal processing device to execute code stored thereon, such as a floppy disk, optical disk, magnetic tape, or hard disk drive. Such code can contain object code, source code, and/or binary code. The code is generally numeric and is typically processed by a central processing unit.

The embodiments of the present invention have been described in detail above with reference to the drawings, but the invention is not limited to the embodiments described above, and those skilled in the art can make various kinds without departing from the spirit and scope of the claims of the present application. Modified or modified.

Claims

A real-time off-screen rendering method based on cloud computing, comprising:

After receiving the cloud rendering service request sent by the client, enabling the rendering resource and creating a direct link of the rendering resource to the client;

After the direct link is successfully created, the rendering resource is rendered in real time according to the cloud rendering service request, and the rendering result generated in real time is sent to the client through the direct link;

After receiving the rendering result in real time, the client displays the screen in real time according to the rendering result.
The cloud computing-based real-time off-screen rendering method of claim 1 , after receiving the cloud rendering service request sent by the client, first accessing the rendering resource list to query whether the rendering resource is available;

If so, the optimal rendering resource is enabled; otherwise, no rendering resource information is available to the client.
The cloud computing-based real-time off-screen rendering method according to claim 2, wherein when the rendering resource is enabled, an address of the rendering resource is acquired and allocated to a communication port of the client, The address of the rendering resource is bound to the communication port of the client to create the direct link of the rendering resource and the client.
The cloud computing-based real-time off-screen rendering method according to claim 1, wherein after the direct link is successfully created, the rendering resource is first downloaded according to the rendering entity described in the cloud rendering service request. The resource data of the rendering entity;

After the resource data of the rendering entity is downloaded, the rendering resource renders the resource data of the rendering entity in real time according to the rendering content described in the cloud rendering request, and is in a frame buffer of the rendering resource. Generate rendering results in real time;

The rendering result in the frame buffer is obtained in real time, and is compression-encoded, and then sent to the client through the direct link.
The cloud computing-based real-time off-screen rendering method according to claim 4, wherein the transmission frequency of the rendering result is adjusted in real time according to the communication status on the direct link and the operation of the client.
The cloud computing-based real-time off-screen rendering method of claim 1 , before receiving the cloud rendering service request sent by the client, receiving the client to send a user verification request;

Parsing user identity information included in the user authentication request, and verifying that the user identity information is consistent with the pre-bound user identity information;

If they are consistent, the identification information is returned to the client, and a heartbeat connection with the client is established. Otherwise, the identification information is not returned to the client.
The cloud computing-based real-time off-screen rendering method according to claim 6, wherein after returning the identification information to the client, if the heartbeat packet sent by the client is received, determining the heartbeat If the connection is successfully established, the user is added to the user online list; if the heartbeat packet sent by the client is not received, it is determined that the heartbeat connection establishment fails, and the user is offline;

And after the heartbeat connection is successfully established, if the offline request sent by the client is received, the heartbeat connection is disconnected, the user is removed from the online list of the user, and the created Direct link and enabled rendering resources.
The cloud computing-based real-time off-screen rendering method according to claim 7, wherein after the heartbeat connection with the client is established, the heartbeat packet sent by the client according to a certain frequency is received to monitor the login status of the user. And every time after receiving one of the heartbeat packets, immediately start timing from zero;

If the timing time is greater than or equal to the drop threshold time, the user is removed from the online list of users, and the created direct link and the enabled rendering resource are released.
The cloud computing-based real-time off-screen rendering method according to claim 7, wherein during the online peak period of the client, a certain number of rendering resources are opened in advance, and the client is online. At the low end, close a certain amount of rendering resources.
A real-time off-screen rendering device based on cloud computing, comprising:

a communication relay unit, configured to receive a cloud rendering service request sent by the client;

a rendering resource unit for providing a rendering resource for rendering services;

a resource management unit, configured to manage and enable the rendering resource;

a service scheduling unit, configured to create a direct link between the rendering resource and the client.
The cloud computing-based real-time off-screen rendering apparatus according to claim 10, wherein the resource management unit comprises:

a rendering resource list sub-unit, configured to save an address of each rendering resource in the rendering resource unit, and perform statistics on the working state of each rendering resource according to a certain frequency, and establish a rendering resource list according to the statistical result;

a rendering resource enabling subunit, configured to query whether there is a rendering resource available in the rendering resource list, and enable an optimal rendering resource when the rendering resource is available; otherwise, return no available resource information to the client end;

And a rendering resource address sending subunit, configured to send an address of the rendering resource enabled sub-unit enabled rendering resource to the service scheduling unit, to be used by the service scheduling unit to create the direct link.
The cloud computing-based real-time off-screen rendering apparatus according to claim 11, wherein the service scheduling unit comprises:

a communication link management subunit, configured to allocate a communication port to the client, and after receiving the address of the rendering resource sent by the rendering resource address sending subunit, communicate the address of the rendering resource with the client Port binding to create the direct link of the rendering resource to the client.
The cloud computing-based real-time off-screen rendering apparatus according to claim 12, wherein The rendering resources include:

a resource data downloading subunit, configured to download resource data of the rendering entity according to the rendering entity described in the cloud rendering service request, and perform downloading of the resource data;

And a rendering execution subunit, configured to perform real-time rendering on the resource data according to the rendering content described in the cloud rendering request;

a frame buffer subunit, configured to cache a rendering result generated by the rendering execution subunit in real time;

a compression coding subunit, configured to acquire a rendering result in the frame buffer subunit in real time, and compress and encode the rendering result;

a rendering result sending subunit, configured to send, in real time, a rendering result processed by the compression encoding subunit to the client through the direct link, and according to the communication status and location of the direct link The operation of the client adjusts the transmission frequency of the rendering result in real time.
The device of claim 13 further comprising:

a storage unit, configured to pre-store the rendering entity resource data of the user;

a database management unit, configured to manage rendering entity resource data of the user, and when the rendering resource requests to download resource data, send a storage address of the resource data in the storage unit to the rendering resource, where The rendering resource downloads the resource data.
The cloud computing-based real-time off-screen rendering apparatus according to claim 14, wherein the apparatus comprises a user management unit, and

The communication relay unit is further configured to receive a user verification request and an offline request sent by the client;

The database management unit is further configured to manage identity information of the user, and store user identity information pre-bound by the user;

The service scheduling unit is further configured to forward the user verification request and the offline request to the user management unit.
The cloud computing-based real-time off-screen rendering apparatus according to claim 15, wherein the user management unit comprises:

An identity information verification subunit, configured to parse user identity information included in the user authentication request, and send the user identity information to the database management unit for comparison, and verify whether the user identity information is pre-bound User identity information is consistent;

a heartbeat connection subunit, configured to return identification information to the client according to whether the user identity information is consistent with pre-bound user identity information, and establish a heartbeat connection with the client; and return the identification information After receiving the heartbeat packet sent by the client, if the heartbeat packet is received, determining that the heartbeat connection is successfully established, and if the heartbeat packet is not received, determining that the heartbeat connection fails The user is offline; and after the heartbeat connection is successfully established, the heartbeat connection is disconnected when the offline request sent by the client is received;

a user online list management sub-unit, configured to establish a user online list, and when the heartbeat connection is successfully established, add the user to the online list of the user; and when receiving the offline request sent by the client, Removing the user from the online list of users releases the created direct link and enabled rendering resources.
The cloud computing-based real-time off-screen rendering apparatus according to claim 16, wherein the heartbeat connection sub-unit is further configured to start timing from zero after receiving one of the heartbeat packets, and the timing is greater than Sending the user drop information to the online list management subunit of the user when the line is dropped, so that the user online list management subunit removes the user from the online list of the user, and the release has been created. The direct link and the enabled rendering resources.
The device of claim 15 further comprising:

An intelligent management unit for opening a certain amount of rendering resources in advance during a user's online peak period. In the user's online trough, a certain amount of rendering resources are turned off.
A real-time off-screen rendering system based on a cloud computing system, comprising: a client, a public network, and a plurality of user resource pools, wherein the user resource pools are respectively bound with a unique public network IP, and the client is linked to The public network IP accesses the user resource pool;

The user resource pool includes a router, a rendering resource server, a resource management server, and a service scheduling server;

The router is configured to receive, by the client, a cloud rendering service request, and forward the cloud rendering service request to the service scheduling server;

The service scheduling server is configured to create a direct link between the rendering resource and the client, and forward the cloud rendering service request to the rendering resource server and the resource management server;

The rendering resource server is configured to provide a rendering resource for rendering services;

The resource management server is configured to manage and enable the rendering resource.
The cloud computing-based real-time off-screen rendering system according to claim 19, wherein the resource management server saves an address of each rendering resource in the rendering resource server, and works on each rendering resource according to a certain frequency. Perform statistics, and based on the statistical results, create a rendering resource list;

Receiving, by the resource management server, the cloud rendering service request forwarded by the service scheduling server, querying whether the rendering resource is available in the rendering resource list, and if yes, enabling the optimal rendering resource, otherwise returning no available Resource information to the client;

The resource management server enables a rendering resource and sends an address of the rendering resource to the service scheduling server for use by the service scheduling server to create the direct link.
The cloud computing-based real-time off-screen rendering system according to claim 20, wherein the service scheduling server receives an address sent by the resource management server and allocates to the client a communication port, The address of the rendering resource is bound to the communication port of the client to create And constructing the rendering resource with the direct link of the client.
The cloud computing-based real-time off-screen rendering system according to claim 21, wherein the rendering resource downloads resource data of the rendering entity according to a rendering entity described in the cloud rendering service request; Deriving the rendering content described in the cloud rendering request, real-time rendering the downloaded resource data, and generating a rendering result in a frame buffer in real time;

The rendering resource performs compression encoding on the rendering result in the frame buffer, and sends the compression-encoded rendering result to the client through the direct link, and according to the communication status of the direct link And the operation of the client, adjusting the sending frequency of the rendering result in real time.
The cloud computing-based real-time off-screen rendering system of claim 22, wherein the system comprises a database server and a storage server; wherein the user resource pool shares the database server and the storage server;

The storage server is configured to pre-store the rendering entity resource data of the user;

The database management server is configured to manage the rendering entity resource data of the user, and send the storage address of the resource data in the storage server to the rendering resource when the rendering resource requests to download the resource data, The resource data is downloaded for the rendering resource.
The cloud computing-based real-time off-screen rendering system according to claim 22, wherein the system comprises: a user management server, and

The router is further configured to receive a user verification request sent by the client;

The database management server is further configured to manage identity information of the user, and store user identity information pre-bound by the user;

The service scheduling server is further configured to forward the user verification request to the user management server.
The cloud computing-based real-time off-screen rendering system according to claim 24, wherein the user management server receives a user authentication request forwarded by the service scheduling server, and parses the The user identity information included in the user verification request is sent to the database management unit for comparison, and the user identity information is verified to be consistent with the pre-bound user identity information;

The user management server returns the identification information to the client according to whether the user identity information is consistent with the pre-bound user identity information, so that the client establishes a heartbeat connection with the user management server;

After returning the identification information to the client, the user management server receives the heartbeat packet sent by the client, determines that the heartbeat connection is successfully established, and adds the user to the online list of users; If the user management server does not receive the heartbeat connection, it is determined that the heartbeat connection fails, and the user is offline;

After the heartbeat connection is successfully established, the user management server receives the offline request sent by the client, disconnects the heartbeat connection, removes the user from the online list of the user, and releases the The direct link is created and the enabled rendering resources are deactivated.
The cloud computing-based real-time off-screen rendering system according to claim 25, wherein the user management server receives a heartbeat packet and starts timing from zero;

If the timing time is greater than the drop threshold time, the user management server removes the user from the online list of users, and releases the created direct link and disables enabled rendering. Resources.
The cloud computing-based real-time off-screen rendering system of claim 24, wherein the system comprises:

The intelligent management server is used to open a certain number of rendering resources in advance during the user's online peak period, and close a certain amount of rendering resources during the user's online trough.