CN114827692A - System for operating cloud desktop based on smart television - Google Patents

Abstract

The invention discloses a system for operating a cloud desktop based on a smart TV. By transferring the rendering work to the smart TV terminal, and using the powerful hardware decoding capability of the smart TV terminal itself, the audio and video streaming media sent by the back-end server can be locally rendered in real time. It can effectively save back-end server resources, ensure the encoding efficiency of desktop data on the back-end server, reduce the delay in the transmission of audio and video streaming media between the back-end server and the smart TV terminal, and control the delay of the picture to the range of 80~100ms to ensure The fluency of the cloud desktop screen reduces screen freezes, dropped frames, and operation delays to meet the application of cloud office, cloud TV, and cloud gaming scenarios.

Description

A system for operating cloud desktop based on smart TV

technical field

The invention relates to the technical fields of cloud desktops, cloud TVs and cloud games, in particular to a system for operating cloud desktops based on smart TVs.

Background technique

Cloud desktop, also known as desktop virtualization and cloud computer, is a new model that replaces traditional computers; after adopting cloud desktop, users do not need to purchase a computer host, and the CPU, memory, hard disk and other components contained in the host are all in the back-end server. Virtually out, a single high-performance server can virtualize 1-50 virtual hosts. The front-end device is mainly a display device for displaying virtual desktops. After installing the client on the front-end device, the virtual machine host on the back-end server can be accessed through a unique communication protocol to receive data and display it on the front-end device.

The data processing process of the cloud desktop system is centrally processed by the back-end server, which is only used for screen display and reception of interactive operations. Since the actual interactive operations and screen rendering are only processed by the back-end server, the back-end server The calculation pressure is very high, and the rendered pictures have high requirements on the bandwidth and delay of the transmission network. Under different network environments, the pictures are prone to freeze and frame drops, and there is a large delay in interactive operations.

Therefore, the prior art has shortcomings and needs to be improved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a system for operating a cloud desktop based on a smart TV in order to solve the above problems, in the process of audio and video streaming media data transmission, according to the current network conditions, the network delay and packet loss rate can be automatically switched to the optimum The transmission protocol, its technical scheme is as follows:

A system for operating a cloud desktop based on a smart TV, including a back-end server and a smart TV terminal,

The backend server includes:

Desktop data acquisition module for collecting desktop data from multiple data sources;

The audio and video coding module is used to encode the collected desktop data in H.264/HEVC, and package the encoded desktop data into audio and video streaming media data;

The streaming media transmission module is used to transmit audio and video streaming media data to the smart TV terminal;

And a keyboard and mouse function simulation module, which is used to map the peripheral events of the keyboard and mouse peripheral devices and simulate the operation of the keyboard and mouse.

The smart TV terminal includes:

The streaming media receiving module is used to receive the audio and video streaming media data sent by the back-end server;

Audio and video decoding module, used for hardware decoding of audio and video streaming media data;

The screen rendering module is used for real-time rendering of audio and video streaming media;

and a peripheral access module, which is used to connect the keyboard and mouse peripheral devices and receive peripheral events of the keyboard and mouse peripheral devices.

Optionally, the back-end server further includes an identity verification module for user identity registration and verification;

The smart TV terminal further includes a login module for receiving login information, and sending a registration request or a login request to the identity verification module according to the login information.

Optionally, the streaming media transmission module performs network transmission of audio and video streaming media data through the TCP/UPD protocol;

The streaming media receiving module is used for requesting and receiving audio and video streaming media data through TCP/UDP protocol.

Optionally, the streaming media transmission module and the streaming media receiving module automatically switch the TCP/UPD protocol according to network conditions.

Optionally, the streaming media transmission module and the streaming media receiving module automatically switch the TCP/UPD protocol according to network conditions, including:

Configure the network parameters of the current network;

The network parameters are monitored in real time during the transmission of audio and video streaming media data, and the transmission protocol is switched back and forth between the TCP protocol and the UDP protocol according to changes in the network parameters.

Optionally, the network parameters include network delay and packet loss rate.

Optionally, the configuring the network parameters of the current network includes:

Configure the maximum value of the network delay LAG-max;

Configure the maximum value PLR-max of the packet loss rate.

Optionally, the maximum value LAG-max of the network delay is configured to be 100 ms, and the maximum value PLR-max of the packet loss rate is configured to be 10%.

Optionally, the transfer protocol is switched back and forth between the TCP protocol and the UDP protocol according to changes in network parameters, including:

When the monitored network delay is greater than the maximum value LAG-max of the network delay or the monitored packet loss rate is greater than the maximum value PLR-max of the packet loss rate, the transmission protocol will be switched to the UDP protocol; otherwise, the transmission protocol will be switched to the UDP protocol. The protocol is switched to the TCP protocol.

Optionally, the keyboard and mouse function simulation module simulates keyboard and mouse operations through the SendInput function.

The present invention has the following beneficial effects:

By transferring the rendering work to the smart TV terminal, and using the powerful hardware decoding capability of the smart TV terminal to perform local real-time rendering of the audio and video streaming media sent by the back-end server, it can effectively save the resources of the back-end server and ensure the desktop data of the back-end server. The encoding efficiency is improved, the delay of audio and video streaming media transmission between the back-end server and the smart TV terminal is reduced, and the delay of the picture is controlled to the range of 80~100ms, which ensures the smoothness of the cloud desktop picture and reduces the picture freezing and frame drop. , operation delay and other phenomena to meet the application of various scenarios of cloud office, cloud TV, and cloud gaming.

Description of drawings

FIG. 1 is a block diagram of a system for operating a cloud desktop based on a smart TV provided by the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in FIG. 1 , the present invention provides a system for operating a cloud desktop based on a smart TV, including a back-end server and a smart TV terminal. The back-end server includes a desktop data acquisition module, an audio and video encoding module, a streaming media transmission module, and a keyboard and mouse function simulation module. The desktop data collection module is used for the CPU, memory, hard disk and other components of the back-end server to collect desktop data-DXGI data; the audio and video encoding module hard-codes the collected desktop data through the NVIDIA/AMD graphics card configured in the back-end server, and then The encoded desktop data is packaged into audio and video streaming media data; the streaming media transmission module is a network module, which supports TCP/UPD protocol for network transmission, and is used to transmit audio and video streaming media data to the smart TV terminal. The keyboard and mouse function simulation module is used to map the peripheral events of the keyboard and mouse peripheral devices, and simulate the operation of the keyboard and mouse.

The smart TV terminal includes a streaming media receiving module, an audio and video decoding module, a picture rendering module and a peripheral access module. Among them, the streaming media receiving module is a network module, which also supports TCP/UPD protocol for network transmission, establishes network communication with the streaming media transmission module, and receives the audio and video streaming media data transmitted by the streaming media transmission module; the audio and video decoding module is a smart TV. The encoding and decoding hardware that comes with the terminal is used for hardware decoding of audio and video streaming media data; the picture rendering module is the GPU that comes with the smart TV terminal, which is used to render the audio and video streaming media in real time to ensure the smoothness of the picture. ; Peripheral access module, used to access USB keyboard and mouse peripheral devices, and transmit the peripheral events of buttons and mouse to the keyboard and mouse function simulation module.

In this embodiment, the back-end server further includes an identity verification module for user identity registration and verification; the smart TV terminal further includes a login module for receiving login information and sending the login information to the identity verification module. A registration request or a login request. For unregistered users, account registration can be performed through the login module on the smart TV terminal, and the back-end service receives the registration information and establishes the account. The registration information can be the user's mobile phone number, WeChat account, email, QQ and other electronic accounts. After receiving the information, the back-end server can associate the user information of the relevant electronic account to complete the establishment of the cloud desktop account, eliminating the need for the user to fill in personal information tedious.

For users with electronic accounts, they can log in directly through the login module. After the login information is sent to the back-end server through the identity verification module, audio and video streaming media data can be received through the streaming media receiving module.

In this embodiment, the mouse and keyboard function simulation module simulates the operation of the keyboard and mouse through the SendInput function.

The working principle of the system provided by the present invention is:

1. Screen transmission

The DXGI data collected by the desktop data acquisition module is encoded by H.264/HEVC using the NVIDIA/AMD graphics card configured by the back-end server, and the encoded audio and video streaming media data are packaged; after the smart TV is connected to WIFI, The TCP/UDP protocol requests audio and video data, uses the smart TV terminal's own hardware to decode the audio and video, renders it to the screen of the smart TV terminal, and displays the cloud desktop image in real time.

2. Synchronization of keyboard and mouse peripheral operations

The smart TV terminal is connected to the USB keyboard and mouse, and the peripheral events of the buttons and the mouse are transmitted to the keyboard and mouse function simulation module. The keyboard and mouse function simulation module maps mouse and keyboard events, and calls the SendInput function to simulate the keyboard and mouse functions.

Compared with the prior art, the present invention has the following beneficial effects:

By transferring the rendering work to the smart TV terminal, and using the powerful hardware decoding capability of the smart TV terminal to perform local real-time rendering of the audio and video streaming media sent by the back-end server, it can effectively save the resources of the back-end server and ensure the desktop data of the back-end server. The encoding efficiency is improved, the delay of audio and video streaming media transmission between the back-end server and the smart TV terminal is reduced, and the delay of the picture is controlled to the range of 80~100ms, which ensures the smoothness of the cloud desktop picture and reduces the picture freezing and frame drop. , operation delay and other phenomena to meet the application of various scenarios of cloud office, cloud TV, and cloud games.

In some preferred embodiments of the present invention, the streaming media transmission module and the streaming media receiving module automatically switch the TCP/UPD protocol according to network conditions.

Optionally, the streaming media transmission module and the streaming media receiving module automatically switch the TCP/UPD protocol according to network conditions, so as to realize real-time transmission under different network conditions, and improve the interactive experience of the cloud desktop system. Specifically, the streaming media transmission module and the streaming media receiving module automatically switch the TCP/UPD protocol according to network conditions, including the following steps:

Step 1, configure the network parameters of the current network.

In the process of audio and video streaming media data transmission, the network status is mainly reflected by the network delay and packet loss rate. Therefore, in step 1, the two network parameters of network delay and packet loss rate are used as the analysis objects. In the process of video streaming data transmission, the network delay and packet loss rate are monitored in real time on the smart TV terminal.

Generally, the better the network condition, the smaller the network delay and the smaller the packet loss rate. According to the transmission characteristics of the TCP protocol and UDP protocol, configure the network delay and packet loss rate, and set the maximum network delay LAG-max and the maximum packet loss rate PLR-max allowed during the transmission of the TCP protocol, so as to be based on the detection value in the follow-up. The comparison result with the set value automatically switches the transmission protocol.

Specifically, in this embodiment, the maximum network delay LAG-max is set to 100ms, and the maximum packet loss rate PLR-max is set to 10%. It should be noted that the maximum network delay LAG-max and the maximum packet loss rate PLR-max are set to other values according to the requirements of audio and video streaming media data transmission, and are not limited thereto.

Step 2: Monitor the network parameters in real time during the audio and video streaming media data transmission process, and switch the transmission protocol back and forth between the TCP protocol and the UDP protocol according to changes in the network parameters.

In step 2, the TCP protocol is selected by default for audio and video streaming data transmission. During the audio and video streaming data transmission process, the two network parameters of network delay and packet loss rate are monitored in real time at the smart TV terminal, which are recorded as LAG- Real-time and PLR-real-time.

When either LAG-real-time or PLR-real-time is greater than its set threshold, that is, LAG-real-time > 100ms and/or PLR-real-time > 10%, it indicates that the current network condition is not good, and the transmission protocol is changed from the TCP protocol Switch to UDP protocol, UDP protocol has faster transmission speed than TCP in the case of poor network.

On the contrary, if LAG-real-time≤100ms and PLR-real-time≤10%, it indicates that the current network condition is better, and the TCP protocol is maintained to transmit audio and video streaming media data.

By configuring the network delay and packet loss rate, the smart TV terminal monitors the network delay and packet loss rate of the current network situation in real time, and then automatically switches to the best transmission protocol according to the real-time detection results, so as to ensure the audio and video streaming. The transmission efficiency of media between the smart TV terminal and the back-end server ensures the real-time and fluency of the cloud desktop screen.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention. Inside.

Claims (10)

1. A system for operating a cloud desktop based on an intelligent television is characterized by comprising a back-end server and an intelligent television terminal,

the back-end server includes:

the desktop data acquisition module is used for acquiring desktop data from a plurality of data sources;

the audio and video coding module is used for carrying out H.264/HEVC coding on the acquired desktop data and packaging the coded desktop data into audio and video streaming media data;

the streaming media transmission module is used for transmitting the audio and video streaming media data to the intelligent television terminal;

the keyboard and mouse function simulation module is used for mapping peripheral events of keyboard and mouse peripheral equipment and simulating keyboard and mouse operation;

the intelligent television terminal comprises:

the streaming media receiving module is used for receiving audio and video streaming media data sent by a back-end server;

the audio and video decoding module is used for carrying out hardware decoding on the audio and video streaming media data;

the picture rendering module is used for rendering the played audio and video streaming media in real time;

and the peripheral access module is used for connecting the keyboard and mouse peripheral equipment and receiving peripheral events of the keyboard and mouse peripheral equipment.

2. The system for operating the cloud desktop based on the smart television as claimed in claim 1, wherein the backend server further comprises an identity verification module for registering and verifying the identity of the user;

the intelligent television terminal also comprises a login module used for receiving login information and sending a registration request or a login request to the identity verification module according to the login information.

3. The system for operating the cloud desktop based on the smart television as claimed in claim 1, wherein the streaming media transmission module performs network transmission of audio/video streaming media data through a TCP/UPD protocol;

the streaming media receiving module is used for requesting and receiving audio and video streaming media data through a TCP/UDP protocol.

4. The system for operating the cloud desktop based on the smart television as claimed in claim 3, wherein the streaming media transmission module and the streaming media receiving module automatically switch TCP/UPD protocols according to network conditions.

5. The system for operating the cloud desktop based on the smart television as claimed in claim 2, wherein the streaming media transmission module and the streaming media receiving module automatically switch TCP/UPD protocol according to network conditions, and the system comprises:

configuring network parameters of a current network;

and monitoring the network parameters in real time in the audio and video streaming media data transmission process, and switching the transmission protocol back and forth between a TCP protocol and a UDP protocol according to the change of the network parameters.

6. The system for operating the cloud desktop based on the smart television as recited in claim 5, wherein the network parameters comprise network delay and packet loss rate.

7. The system for operating the cloud desktop based on the smart television as recited in claim 6, wherein the configuring the network parameters of the current network comprises:

configuring a maximum value LAG-max of network delay;

and configuring the maximum value PLR-max of the packet loss rate.

8. The system according to claim 7, wherein the maximum LAG-max of the network delay is configured to be 100ms, and the maximum PLR-max of the packet loss rate is configured to be 10%.

9. The system for operating cloud desktop based on smart television of claim 8, wherein the switching transmission protocol back and forth between TCP protocol and UDP protocol according to the change of network parameters comprises:

when the monitored network delay is larger than the maximum LAG-max of the network delay or the monitored packet loss rate is larger than the maximum PLR-max of the packet loss rate, switching the transmission protocol to a UDP protocol; otherwise, the transmission protocol is switched to the TCP protocol.

10. The system for operating the cloud desktop based on the smart television as claimed in claim 1, wherein the keyboard and mouse function simulation module simulates keyboard and mouse operations through a SendInput function.

Images