CN115865909A

CN115865909A - Data transmission method, device and readable storage medium based on SPICE protocol

Info

Publication number: CN115865909A
Application number: CN202310039291.4A
Authority: CN
Inventors: 请求不公布姓名
Original assignee: Nfs China Software Co ltd
Current assignee: Nfs China Software Co ltd
Priority date: 2023-01-13
Filing date: 2023-01-13
Publication date: 2023-03-28
Anticipated expiration: 2043-01-13
Also published as: CN115865909B

Abstract

The embodiment of the application provides a data transmission method, a data transmission device and a readable storage medium based on a SPICE protocol. The method comprises the following steps: acquiring video data output by QEMU, wherein the video data comprises a complete desktop image; initializing an encoder according to configuration information of the encoder, and encoding the video data by using the encoder to obtain encoded data, wherein the configuration information comprises an encoding type, and the encoding type comprises H264 or H265; according to the modified SPICE protocol, a first transmission channel is created with a SPICE client, and the first transmission channel is used for transmitting data from the SPICE server to the SPICE client; transmitting the encoded data to the SPICE client over the first transmission channel. The method and the device can improve the processing efficiency of the video data and reduce the network pause phenomenon.

Description

Data transmission method, device and readable storage medium based on SPICE protocol

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data transmission method and apparatus based on an SPICE protocol, and a readable storage medium.

Background

SPICE (Simple Protocol for independent Computing Environment Simple Protocol) is an open-source remote desktop transmission Protocol, is a high-performance and dynamic self-adaptive remote desktop delivery technology, and can provide the same end user experience as a physical desktop.

The SPICE client can access the remote desktop and use local devices (e.g., keyboard, USB, etc.) via SPICE protocol. The SPICE server is responsible for completing the interaction between the server virtual machine and the SPICE client.

Currently, the SPICE protocol is to transmit a changed part (called dirty block) of a remote desktop image to a user terminal, and the transmission mode includes a picture mode and a stream mode. And the picture mode refers to that when the remote desktop image changes, the SPICE server side compresses dirty blocks output by the QEMU in a picture mode in a lossy or lossless mode, and then sends the compressed dirty blocks to the SPICE client side. QEMU is a common open source computer emulator and virtualizer. And the stream mode is that when the remote desktop image changes, the SPICE server compresses dirty blocks output by the QEMU in a stream mode and sends the compressed dirty blocks to the SPICE client. After receiving dirty blocks output by QEMU, the SPICE server side can judge the frame rate of each dirty block, and when the frame rate of a certain dirty block is lower than a preset critical value, the dirty block is transmitted in a picture mode; and when the frame rate of a certain dirty block exceeds a preset critical value, transmitting the dirty block in a streaming mode.

However, when the number of dirty blocks is large, the frame rate is determined for each dirty block and the transmission mode is determined, which is inefficient. In addition, if the network condition is poor and the frame rate of the dirty block is always between the picture mode and the streaming mode, the streaming mode cannot be triggered, so that a vicious circle is caused, and the network stuck phenomenon is difficult to solve.

Disclosure of Invention

The embodiment of the application provides a data transmission method, a data transmission device and a readable storage medium based on a SPICE protocol, which can improve the processing efficiency of video data and reduce the network pause phenomenon.

In a first aspect, an embodiment of the present application discloses a data transmission method based on an SPICE protocol, which is applied to a SPICE server, and the method includes:

acquiring video data output by QEMU, wherein the video data comprises a complete desktop image;

initializing an encoder according to configuration information of the encoder, and encoding the video data by using the encoder to obtain encoded data, wherein the configuration information comprises an encoding type, and the encoding type comprises H264 or H265;

according to the modified SPICE protocol, a first transmission channel is created with a SPICE client, and the first transmission channel is used for transmitting data from the SPICE server to the SPICE client;

transmitting the encoded data to the SPICE client over the first transmission channel.

In a second aspect, an embodiment of the present application discloses a data transmission method based on a SPICE protocol, which is applied to a SPICE client, and the method includes:

according to the modified SPICE protocol, a first transmission channel is created with a SPICE server, and the first transmission channel is used for transmitting data from the SPICE server to the SPICE client;

receiving the encoded data sent by the SPICE server through the first transmission channel, wherein the encoded data is obtained by encoding the video data output by the QEMU by the SPICE server through an encoder, the video data comprises a complete desktop image, and the encoding type of the encoder comprises H264 or H265.

In a third aspect, an embodiment of the present application discloses a data transmission device based on SPICE protocol, which is applied to SPICE service terminals, and the device includes:

the video data acquisition module is used for acquiring the video data output by the QEMU, and the video data comprises a complete desktop image;

the video data encoding module is used for initializing an encoder according to configuration information of the encoder, and encoding the video data by using the encoder to obtain encoded data, wherein the configuration information comprises an encoding type, and the encoding type comprises H264 or H265;

a first establishing module, configured to create a first transmission channel with a SPICE client according to the modified SPICE protocol, where the first transmission channel is used to transmit data from the SPICE server to the SPICE client;

and the video data sending module is used for transmitting the encoded data to the SPICE client through the first transmission channel.

In a fourth aspect, an embodiment of the present application discloses a data transmission apparatus based on SPICE protocol, which is applied to SPICE clients, and the apparatus includes:

a third establishing module, configured to create a first transmission channel with a SPICE server according to the modified SPICE protocol, where the first transmission channel is used to transmit data from the SPICE server to the SPICE client;

and the video data receiving module is used for receiving the coded data sent by the SPICE server through the first transmission channel, the coded data is obtained by coding the video data output by the QEMU by the SPICE server through a coder, the video data comprises a complete desktop image, and the coding type of the coder comprises H264 or H265.

In a fifth aspect, the present application discloses an apparatus for data transmission based on SPICE protocol, which includes a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing the SPICE protocol-based data transmission method according to one or more of the foregoing.

In a sixth aspect, an embodiment of the present application discloses a readable storage medium, on which instructions are stored, which when executed by one or more processors of an apparatus, cause the apparatus to perform a data transmission method based on the SPICE protocol as described in one or more of the foregoing.

The embodiment of the application has the following advantages:

according to the data transmission method based on the SPICE protocol, the original SPICE protocol is modified, so that the QEMU always outputs a complete desktop image instead of dirty blocks, therefore, when the picture of the desktop image changes greatly, the processing efficiency cannot be influenced by the fact that the number of the dirty blocks is large, and the processing efficiency is high. In addition, the SPICE server sends the video data which is encoded and compressed by H264 or H265 to the SPICE client, so that the occupied bandwidth is small and stable, even if the bandwidth has a bottleneck, the smooth operation can be ensured, and the network blockage phenomenon is reduced. Moreover, because the complete desktop image is output by the QEMU every time, the resolution ratio is fixed, and the size difference is small after the encoding and compression, the generated bandwidth fluctuation is small, and the stability of the network can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive exercise.

FIG. 1 is a flowchart illustrating the steps of an embodiment of a data transmission method based on the SPICE protocol;

FIG. 2 is a schematic diagram of a video data transmission flow of the present application;

FIG. 3 is a schematic flow diagram of a custom configuration data transmission flow of the present application;

FIG. 4 is a flowchart illustrating steps of an embodiment of a data transmission method based on SPICE protocol according to the present application;

FIG. 5 is a block diagram of an embodiment of a data transmission apparatus based on SPICE protocol according to the present application;

FIG. 6 is a block diagram of an embodiment of a data transmission apparatus based on SPICE protocol according to the present application;

FIG. 7 is a block diagram of an apparatus 800 for SPICE protocol based data transfer according to the present application;

fig. 8 is a schematic diagram of a server in some embodiments of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application are capable of operation in sequences other than those illustrated or described herein, and that the terms "first," "second," etc. are generally used in a generic sense and do not limit the number of terms, e.g., a first term can be one or more than one. Furthermore, the term "and/or" in the specification and claims is used to describe an association relationship of associated objects, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. In the embodiments of the present application, the term "plurality" means two or more, and other terms are similar thereto.

Referring to fig. 1, a flowchart illustrating steps of an embodiment of a data transmission method based on SPICE protocol according to the present application is shown, where the method is applied to SPICE service, and the method may include the following steps:

101, acquiring video data output by QEMU, wherein the video data comprises a complete desktop image;

step 102, initializing an encoder according to configuration information of the encoder, and encoding the video data by using the encoder to obtain encoded data, wherein the configuration information includes an encoding type, and the encoding type includes H264 or H265;

step 103, creating a first transmission channel with the SPICE client according to the modified SPICE protocol, wherein the first transmission channel is used for transmitting data from the SPICE server to the SPICE client;

and 104, transmitting the encoded data to the SPICE client through the first transmission channel.

In the embodiment of the application, the SPICE server refers to SPICE-server and runs on a server. The SPICE client refers to SPICE-gtk and is used for interacting with SPICE-server and displaying the virtual machine.

According to the data transmission method based on the SPICE protocol, provided by the embodiment of the application, the display drive of the QEMU is modified, so that the QEMU always outputs a complete desktop image instead of a dirty block, and the desktop in the embodiment of the application refers to a remote desktop of an SPICE server. The remote desktop image data of the SPICE server is transmitted in a video mode and displayed at the SPICE client.

In addition, the data transmission method based on the SPICE protocol provided by the embodiment of the application modifies the original SPICE-protocol (SPICE-protocol). According to the modified SPICE protocol, the SPICE service end and the SPICE client end can create a first transmission channel. The first transmission channel is a communication channel between the SPICE server and the SPICE client, and the first transmission channel is used for transmitting data from the SPICE server to the SPICE client. For example, the SPICE server can transmit the video data output by the QEMU to the SPICE client through the first transmission channel, and the video data transmitted through the first transmission channel is a complete desktop image and is encoded by H264 or H265.

H264, also known as h.264, is a new generation of digital video compression format following MPEG4, commonly proposed by the international organization for standardization (ISO) and the International Telecommunications Union (ITU).

H265, also known as h.265, is a new video coding standard established by ITU-t vceg following h.264. The h.265 standard surrounds the existing video coding standard h.264, preserving some of the original techniques, while improving some of the related techniques.

According to the modified SPICE protocol, the SPICE service end and the SPICE client end can also create a second transmission channel. The second transmission channel is a communication channel between the SPICE server and the SPICE client, and the second transmission channel is used for transmitting data from the SPICE client to the SPICE server. For example, the SPICE client can transmit the custom configuration data to the SPICE server through the second transmission channel so as to dynamically modify coder configuration parameters of the SPICE server, such as coding type, code rate and the like.

Further, the data transmission method based on the SPICE protocol provided by the embodiment of the application performs encoding compression on the video data output by the QEMU. Specifically, after the video data output by the QEMU is subjected to H264 coding or H265 coding by the SPICE client, the SPICE client sends the video data to the SPICE client through a first transmission channel.

According to the data transmission method based on the SPICE protocol, because the complete desktop image is output by the QEMU all the time, and dirty blocks are not output, when the picture of the desktop image changes greatly, the processing efficiency cannot be influenced due to the fact that the number of the dirty blocks is large, and the processing efficiency is high. In addition, after the complete desktop image output by the QEMU is coded by H264 or H265, the occupied bandwidth is small and stable, even if the bandwidth has a bottleneck, the smooth operation can be ensured, and the network blockage phenomenon is reduced. Moreover, because the complete desktop image is output by the QEMU every time, the resolution ratio is fixed, and the size difference is small after the encoding and compression, the generated bandwidth fluctuation is small, and the stability of the network can be improved.

According to the method and the device, the display driving code of the QEMU is modified, so that the QEMU outputs the complete desktop image. Display driver of QEMU refers to the program that QEMU is responsible for displaying. Specifically, modifying the display driver code of the QEMU, by obtaining the width and height of the remote desktop, display content corresponding to the resolution size of the remote desktop, i.e., a complete desktop image, may be generated.

The embodiment of the application modifies the original SPICE protocol, and the SPICE server and the SPICE client use the modified SPICE protocol to carry out data transmission. In the embodiment of the present application, data transmitted by the first transmission channel is referred to as first transmission channel data, and data transmitted by the second transmission channel is referred to as second transmission channel data. The first transmission channel data comprises video coding data sent by the SPICE server to the SPICE client. The second transmission channel data contains the self-defined configuration data sent by the SPICE client to the SPICE server.

In an optional embodiment of the present application, a data transmission format of the first transmission channel is defined in the modified SPICE protocol, and the transmitting the encoded data to the SPICE client through the first transmission channel may include: and after the coded data is packaged according to the data transmission format of the first transmission channel, the coded data is transmitted to the SPICE client through the first transmission channel.

The embodiment of the application modifies the original SPICE protocol, and specifically, a data transmission format definition of a first transmission channel is added in a SPICE.

The SPICE server side obtains the video data output by the QEMU, and codes the video data according to the coding type in the configuration information to obtain coded data; and encapsulating the coded data according to the data transmission format of the first transmission channel to obtain first transmission channel data, and sending the first transmission channel data to the SPICE client.

Optionally, the format defining the first transmission channel data may include the following: resolution, type of encoding, length of data, and data content (i.e., encoded data). Wherein the resolution includes a width and a height of the remote desktop. After receiving the first transmission channel data of the SPICE server, the SPICE client can analyze the first transmission channel data to obtain the coded data therein, determine a decoder according to the coding type of the coded data, and then decode and display the coded data.

Similarly, a data transmission format of the second transmission channel is also defined in the modified SPICE protocol, and the SPICE client side encapsulates the custom configuration data set by the user according to the data transmission format of the second transmission channel and then transmits the custom configuration data to the SPICE server side through the second transmission channel.

The embodiment of the application modifies the original SPICE protocol, and specifically, a data transmission format definition of a second transmission channel is added in a SPICE.

After receiving the user-defined configuration data set by the user, the SPICE client encapsulates the user-defined configuration data according to the data transmission format of the second transmission channel to obtain second transmission channel data, and sends the second transmission channel data to the SPICE server. The custom configuration data may include, but is not limited to, coding type and code rate. After receiving the second transmission channel data of the SPICE client, the SPICE server can analyze the second transmission channel data to obtain the custom configuration data, and the encoder can be reinitialized according to the custom configuration data to enable the custom configuration data to take effect.

Before the encoder is used for encoding, the SPICE service end carries out initialization operation on the encoder according to the configuration information. The embodiment of the present application does not limit the specific form of the configuration information, and for example, a structure body may be defined to record the configuration information. The configuration information includes configuration parameters of the encoder, such as encoding type, resolution, and code rate. The encoding type may include H264 or H265. The resolution includes the width and height of the remote desktop. The code rate refers to the video code rate, and the higher the code rate is, the more data is transmitted per second, and the clearer the image quality is. The coding type and the code rate can adopt default coding type and code rate, and can be customized and modified through the SPICE client. For example, assuming the default encoding type is H264, the user can customize the encoding type to H265 through SPICE client. In another example, the default bitrate is 5Mbps, and the user can customize the bitrate to 7Mbps through the SPICE client. The resolution may be automatically changed according to a user operation, for example, the resolution may be automatically updated when the user changes the size of the remote desktop display window. Of course, the configuration parameters are not limited to the above listed encoding types, resolutions, and code rates, and may include other parameters such as a frame rate. The frame rate is the frequency (rate) at which bitmap images appear continuously on the display in units of frames.

In the embodiment of the present application, the SPICE server may automatically reinitialize the encoder when receiving the custom configuration data sent by the SPICE client or detecting that the resolution of the video data changes, so as to validate the changed configuration parameters.

In an optional embodiment of the present application, the method may further comprise:

step S11, according to the modified SPICE protocol, a second transmission channel is established with the SPICE client, and the second transmission channel is used for transmitting data from the SPICE client to the SPICE server;

step S12, receiving the self-defined configuration data sent by the SPICE client through the second transmission channel, wherein the self-defined configuration data comprises a self-defined coding type and/or a self-defined code rate;

s13, updating the configuration information according to the custom configuration data;

and step S14, re-initializing the encoder according to the updated configuration information.

In the embodiment of the present application, an original SPICE-protocol (SPICE-protocol) is modified, and a macro definition of video data of an H264 coding type and a macro definition of video data of an H265 coding type are added.

In the embodiment of the application, interface function definitions of a first transmission channel and a second transmission channel are respectively added in a header file enum.h of an SPICE protocol. The first transmission channel is used for transmitting data from the SPICE server to the SPICE client, such as transmitting H264/H265 encoded video data from the SPICE-server to the SPICE-gtk. The second transmission channel is used for transmitting data from the SPICE client to the SPICE server, such as transmitting custom configuration data from SPICE-gtk to SPICE-server.

In a specific implementation, when the SPICE server initializes the encoder for the first time, the configuration information of the encoder may be a default configuration parameter, and the SPICE server initializes the encoder according to the default configuration parameter.

The SPICE client side can set custom configuration data, such as setting custom coding types and/or custom code rates, and the SPICE client side packages the custom configuration data set by a user according to a data transmission format of a second transmission channel and transmits the custom configuration data to the SPICE server side through the second transmission channel. After receiving the second transmission channel data sent by the SPICE client, the SPICE server can obtain the self-defined configuration data of the SPICE client by analyzing the second transmission channel data, writes the self-defined configuration data (such as self-defined coding type and/or self-defined code rate) into a structural body of the configuration information, and re-initializes the encoder by utilizing the updated configuration information so as to enable the self-defined configuration data to take effect. Therefore, the embodiment of the application can realize the dynamic switching of the configuration parameters of the encoder, such as the dynamic switching of the encoding type and/or the code rate.

In the embodiment of the application, a user can customize a required coding type through the SPICE client, and in addition, the user can manually adjust configuration parameters such as code rate and the like which are related to strong bandwidth through the SPICE client, so that the problem of network congestion can be effectively solved when the network condition is not good.

In an optional embodiment of the present application, before the encoding the video data with the encoder, the method may further include: converting the video data from a color space format to a luminance space format; the encoding the video data with the encoder may include: encoding the video data in the luma spatial format with the encoder.

Because the codes of H264 and H265 do not support the color space format (RGB) but support the luminance space format (YUV), after the SPICE server acquires the video data output by QEMU, the video data is converted from the color space format (RGB) to the luminance space format (YUV); and then reading configuration parameters such as the coding type from the structural body of the configuration information, initializing an encoder according to the read configuration parameters, and coding the video data in the YUV format by using the initialized encoder to obtain coded data. And filling the coded data into a data structure of the first transmission channel, and sending the coded data to the SPICE client through the first transmission channel.

Further, after the data of the first transmission channel is sent, the memory space occupied by the data of the first transmission channel can be released, so that the utilization rate of memory resources is improved. It should be noted that, since the process of sending the first transmission channel data is an asynchronous call process, the corresponding memory space should not be released immediately after the command of sending the first transmission channel data is called, otherwise, the program crash may be caused. The embodiment of the application releases the memory space occupied by the first transmission channel data through dereferencing so as to solve the problem. Dereferencing refers to returning the corresponding object in the memory address.

s21, acquiring data transmitted by the first transmission channel;

step S22, if the data transmitted by the first transmission channel is detected to be not in accordance with the preset coding requirement, coding the data transmitted by the first transmission channel again; wherein the preset encoding requirements include: when the data transmitted by the first transmission channel is video data, the coding type is H264 or H265.

In the embodiment of the application, an exception handling mechanism is added in the process of transmitting the video data from the SPICE server to the SPICE client. In the SPICE protocol, the direction of the video data flow is: and the QEMU generates video data, and sends the video data to the SPICE client through the network by calling a marshall _ qxl _ drawable function. The marshall _ qxl _ drawable function is an interface for processing the video data transmitted by the Qemu by the SPICE server, and the embodiment of the application modifies the interface for processing the video data transmitted by the QEMU by the SPICE server and adds an exception processing step. Specifically, data transmitted by the first transmission channel is acquired, the data type and the encoding type of the data transmitted by the first transmission channel are detected, when it is detected that the data transmitted by the first transmission channel is video data, whether the encoding type is H264 or H265 is judged, if the encoding type is not H264 or H265, it is determined that the data transmitted by the first transmission channel does not meet the preset encoding requirement, and at this time, the data transmitted by the first transmission channel is taken over and is re-encoded according to H264 or H264 and then transmitted.

s31, analyzing the video data to obtain the current resolution of the desktop image;

step S32, when the current resolution is determined to be inconsistent with the stored resolution, the encoder is reinitialized according to the current resolution;

and step S33, updating the stored resolution to the current resolution.

When the resolution of the remote desktop changes, a SPICE-server (SPICE-server) calls a display _ channel _ create _ surface function in a display-channel.c file, so that the steps S31 to S33 of adding codes in the function are added for reinitializing an encoder according to the changed current resolution and storing the current resolution for judging whether the resolution changes next time when the current resolution is detected to change.

In an optional embodiment of the present application, the method may further comprise: and receiving a self-defined configuration data synchronization message sent by the SPICE client. After the SPICE client is started each time, the self-defined configuration data synchronization message can be sent to the SPICE server, namely, the self-defined configuration data is sent once through the second transmission channel to synchronize configuration parameters such as the coding type, the code rate and the like of the two ends, so that the configuration parameters of the coder of the SPICE server are matched with the configuration parameters of the decoder of the SPICE client.

Referring to fig. 2, a schematic diagram of a video data transmission flow direction according to an embodiment of the present application is shown, as shown in fig. 2, QEMU outputs video data to SPICE-server, and after H264 or H265 coding is performed on the video data output by QEMU, SPICE-server sends the video data to SPICE-client (SPICE-gtk) through a first transmission channel.

Referring to fig. 3, a schematic diagram illustrating a flow of transmission of custom configuration data according to an embodiment of the present application is shown. As shown in fig. 3, the SPICE client (SPICE-gtk) receives custom configuration data (such as a custom encoding type and/or a custom code rate) set by a user through a preset tab in SPICE, the SPICE client (SPICE-gtk) sends the custom configuration data to the SPICE server (SPICE-server) through a second transmission channel, the SPICE server (SPICE-server) re-initializes an encoder according to the received custom configuration data to enable the custom configuration data to take effect, encodes video data output by the QEMU by using the re-initialized encoder, and sends the encoded video data to the SPICE client (SPICE-gtk) through a first transmission channel.

The Spicy is client software based on the spice-gtk and is used for connecting a virtual machine running on the spice-server. According to the embodiment of the application, the preset option can be added in the Spicy, and the user-defined configuration data set by the user can be received through the preset option. Illustratively, the preset tabs include a tab for setting an encoding type and a tab for setting a code rate.

In summary, the data transmission method based on the SPICE protocol provided by the embodiment of the application modifies the original SPICE protocol, so that the QEMU always outputs a complete desktop image instead of dirty blocks, and therefore, when the picture of the desktop image changes greatly, the processing efficiency is not affected by a large number of dirty blocks, and the processing efficiency is high. In addition, the SPICE server sends the video data which is encoded and compressed by H264 or H265 to the SPICE client, so that the occupied bandwidth is small and stable, and even if the bandwidth has a bottleneck, the smooth operation can be ensured. Moreover, because the complete desktop image is output by the QEMU every time, the resolution is fixed, and the size difference is small after the encoding and compression, the generated bandwidth fluctuation is small.

Referring to fig. 4, a flowchart illustrating steps of an embodiment of the data transmission method based on SPICE protocol according to the present application, the method is applied to SPICE clients, and the method may include the following steps:

step 401, according to the modified SPICE protocol, creating a first transmission channel with a SPICE server, wherein the first transmission channel is used for transmitting data from the SPICE server to the SPICE client;

step 402, receiving the encoded data sent by the SPICE server through the first transmission channel, where the encoded data is obtained by encoding the video data output by the SPICE server through an encoder, where the video data includes a complete desktop image, and the encoding type of the encoder includes H264 or H265.

According to the method and the device, the display driving code of the QEMU is modified, so that the QEMU outputs the complete desktop image. And the SPICE server acquires the video data output by the QEMU, wherein the video data comprises a complete desktop image. The SPICE server initializes an encoder according to configuration information of the encoder, and encodes the video data by using the encoder to obtain encoded data, wherein the configuration information comprises an encoding type, and the encoding type comprises H264 or H265. The SPICE client uses a modified SPICE protocol (SPICE-protocol) to establish a first transmission channel with the SPICE server, and the coded data sent by the SPICE server is received through the first transmission channel.

The operation steps of SPICE client side have been described in the foregoing embodiments, and are not described herein again.

In addition, the modification of the SPICE protocol in the embodiment of the application comprises the modification of the SPICE protocol on both sides of the SPICE server and the SPICE client. The modification modes of the two sides are the same, and the modification modes of the SPICE. Proto file and the organs. H file of the SPICE client (SPICE-gtk) are the same as those of the SPICE-server, which are not described herein again and referred to each other.

s41, analyzing the coded data to obtain the coding type of the coded data;

and S42, initializing a decoder according to the coding type, and decoding the coded data by using the decoder to obtain decoded data.

Data sent by the SPICE server through the first transmission channel is coded, and the SPICE client can display the data after decoding the data according to the coding type of the corresponding coder when the data is required to be displayed.

In this embodiment, the format of the first transmission channel data may include the following: resolution, type of encoding, length of data, and data content (i.e., encoded data).

In the SPICE protocol, a channel _ set _ handles function in a channel-display.c file is a function for processing data transmitted from a SPICE-server to a SPICE-gtk, and a processing code is added in the function for analyzing the first transmission channel data.

Specifically, in the embodiment of the present application, a structure body is created in the channel-display.c, and is used for storing various data transmitted by the spice-server, for example, configuration parameters required by spice-gtk to display ffmpeg video data sent by the spice-server include: resolution-width, resolution-height, encoding type, data length, encoded data, and the like.

The SPICE client side can obtain the coded data and the coding type, the data length, the resolution-width and the resolution-height of the coded data by analyzing the first transmission channel data sent by the SPICE server side. The SPICE client uses the modified SPICE protocol to establish a first transmission channel with the SPICE server, and the coded data sent by the SPICE server is received through the first transmission channel. The SPICE client analyzes the coded data to obtain the coding type of the coded data, so that a decoder can be initialized according to the coding type, and the decoder is used for decoding the coded data to obtain decoded data, so that the decoded data can be rendered and displayed.

In one example of the embodiment of the present application, the process of initializing the decoder may include: initializing decoder-related components; determining a decoder according to the coding type; with the incoming resolution, initialization parameters are set.

Further, when decoding is performed by using a decoder, the embodiment of the present application further increases a judgment condition, and when the judgment condition is satisfied, the decoder is reinitialized. The judgment condition comprises the following steps: the decoder is not initialized or the decoding type is changed or the resolution is changed. For example, when the SPICE service end uses the H264 coding type to code, the SPICE client uses the H264 decoding type to decode the coded data sent by the SPICE service end through the first transmission channel. After decoding the encoded data, the SPICE client can display the decoded video data.

In spice-gtk, drawing is usually performed by using the canvas _ get _ bits function in the canvas _ base.c file, but other external files cannot access the canvas _ get _ bits function, so the canvas _ get _ bits function is encapsulated into a target structure for external function call in the embodiment of the present application, and the target structure is denoted as a voicecanvas ops. Specifically, a special drawing function is firstly created for drawing the video data coded by H264 or H265; then, a function pointer member is newly added in the SpiceCanvasOps structure body and is used for pointing to the special drawing function during initialization. At initialization, the dedicated drawing function pointer is written into a new member in the SpiceCanvasOps structure, and then H264 or H265 encoded video data can be displayed in channel-display.c.

In an optional embodiment of the present application, a data transmission format of the second transmission channel is defined in the modified SPICE protocol, and the sending the custom configuration data to the SPICE server through the second transmission channel may include: and after the custom configuration data is encapsulated according to the data transmission format of the second transmission channel, transmitting the custom configuration data to the SPICE server through the second transmission channel.

s51, receiving self-defined configuration data through a preset tab, wherein the self-defined configuration data comprises a self-defined coding type and/or a self-defined code rate;

step S52, according to the modified SPICE protocol, a second transmission channel is established with the SPICE client, and the second transmission channel is used for transmitting data from the SPICE client to the SPICE server;

and S53, sending the custom configuration data to the SPICE server through the second transmission channel, so that the SPICE server updates configuration information of the encoder according to the custom configuration data and re-initializes the encoder.

And the Spicy is client software based on the spice-gtk and is used for connecting the virtual machine running on the spice-server. According to the embodiment of the application, the preset option can be added in the Spicy, and the user-defined configuration data set by the user can be received through the preset option. Illustratively, the preset tabs include a tab for setting an encoding type and a tab for setting a code rate.

It should be noted that, in the embodiment of the present application, the type of the preset option is not limited, and in the specific implementation, a preset option interface for customizing other configuration parameters may also be reserved.

Optionally, the method may further include: and sending a self-defined configuration data synchronization message to the SPICE server. After the SPICE client is started each time, the self-defined configuration data synchronization message can be sent to the SPICE server, namely, the self-defined configuration data is sent once through the second transmission channel to synchronize configuration parameters such as the coding type, the code rate and the like of the two ends, so that the configuration parameters of the coder of the SPICE server are matched with the configuration parameters of the decoder of the SPICE client.

In summary, the data transmission method based on the SPICE protocol provided by the embodiment of the application modifies the original SPICE protocol, so that the QEMU always outputs a complete desktop image instead of dirty blocks, and therefore, when the picture of the desktop image changes greatly, the processing efficiency is not affected by a large number of dirty blocks, and the processing efficiency is high. In addition, the SPICE server sends the video data which is encoded and compressed by H264 or H265 to the SPICE client, so that the occupied bandwidth is small and stable, even if the bandwidth has a bottleneck, the smooth operation can be ensured, and the network blockage phenomenon is reduced. Moreover, because the complete desktop image is output by the QEMU every time, the resolution ratio is fixed, and the size difference is small after the encoding and compression, the generated bandwidth fluctuation is small, and the stability of the network can be improved.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will also appreciate that the embodiments described in the specification are presently preferred and that no particular act is required of the embodiments of the application.

Referring to fig. 5, a block diagram of an embodiment of a data transmission apparatus based on SPICE protocol according to the present application is shown, the apparatus is applied to SPICE service, and the apparatus may include:

a video data obtaining module 501, configured to obtain video data output by the QEMU, where the video data includes a complete desktop image;

a video data encoding module 502, configured to initialize an encoder according to configuration information of the encoder, and encode the video data by using the encoder to obtain encoded data, where the configuration information includes an encoding type, and the encoding type includes H264 or H265;

a first establishing module 503, configured to create a first transmission channel with a SPICE client according to the modified SPICE protocol, where the first transmission channel is used to transmit data from the SPICE server to the SPICE client;

a video data sending module 504, configured to transmit the encoded data to the SPICE client through the first transmission channel.

Optionally, the apparatus further comprises:

a second establishing module, configured to create a second transmission channel with the SPICE client according to the modified SPICE protocol, where the second transmission channel is used to transmit data from the SPICE client to the SPICE server;

a configuration data receiving module, configured to receive, through the second transmission channel, custom configuration data sent by the SPICE client, where the custom configuration data includes a custom coding type and/or a custom code rate;

the configuration information updating module is used for updating the configuration information according to the user-defined configuration data;

and the first initialization module is used for re-initializing the encoder according to the updated configuration information.

Optionally, the apparatus further comprises:

the resolution acquisition module is used for analyzing the video data to obtain the current resolution of the desktop image;

a second initialization module, configured to reinitialize the encoder according to the current resolution when it is determined that the current resolution is inconsistent with the stored resolution;

and the resolution saving module is used for updating the stored resolution to the current resolution.

Optionally, the apparatus further comprises:

a format conversion module for converting the video data from a color space format to a luminance space format;

the video data encoding module is specifically configured to encode the video data in the luminance space format by using the encoder.

Optionally, a data transmission format of the first transmission channel is defined in the modified SPICE protocol, and the video data sending module is specifically configured to package the encoded data according to the data transmission format of the first transmission channel, and then transmit the encoded data to the SPICE client through the first transmission channel.

Optionally, the apparatus further comprises:

the data interception module is used for acquiring the data transmitted by the first transmission channel;

the data take-over module is used for recoding the data transmitted by the first transmission channel if the data transmitted by the first transmission channel is detected to be not in accordance with the preset coding requirement; wherein the preset encoding requirements include: when the data transmitted by the first transmission channel is video data, the coding type is H264 or H265.

Referring to fig. 6, a block diagram of an embodiment of a data transmission apparatus based on SPICE protocol according to the present application is shown, where the apparatus is applied to SPICE clients, and the apparatus may include:

a third establishing module 601, configured to create a first transmission channel with a SPICE server according to the modified SPICE protocol, where the first transmission channel is used to transmit data from the SPICE server to the SPICE client;

a video data receiving module 602, configured to receive, through the first transmission channel, encoded data sent by the SPICE server, where the encoded data is obtained by the SPICE server encoding video data output by using an encoder to QEMU, the video data includes a complete desktop image, and an encoding type of the encoder includes H264 or H265.

Optionally, the apparatus further comprises:

the video data analysis module is used for analyzing the coded data to obtain the coding type of the coded data;

and the video data decoding module is used for initializing a decoder according to the coding type and decoding the coded data by using the decoder to obtain decoded data.

Optionally, the apparatus further comprises:

the configuration data acquisition module is used for receiving user-defined configuration data through a preset option card, wherein the user-defined configuration data comprises a user-defined coding type and/or a user-defined code rate;

a fourth establishing module, configured to create a second transmission channel with the SPICE client according to the modified SPICE protocol, where the second transmission channel is used to transmit data from the SPICE client to the SPICE server;

and the configuration data sending module is used for sending the self-defined configuration data to the SPICE server through the second transmission channel so that the SPICE server updates the configuration information of the encoder according to the self-defined configuration data and reinitializes the encoder.

Optionally, a data transmission format of the second transmission channel is defined in the modified SPICE protocol, and the configuration data sending module is specifically configured to package the custom configuration data according to the data transmission format of the second transmission channel, and transmit the custom configuration data to the SPICE server through the second transmission channel.

In summary, the data transmission device based on the SPICE protocol provided by the embodiment of the application modifies the original SPICE protocol, so that the QEMU always outputs a complete desktop image instead of dirty blocks, and therefore, when the picture of the desktop image changes greatly, the processing efficiency is not affected by a large number of dirty blocks, and the processing efficiency is high. In addition, the SPICE server sends the video data which is encoded and compressed by H264 or H265 to the SPICE client, so that the occupied bandwidth is small and stable, even if the bandwidth has a bottleneck, the smooth operation can be ensured, and the network blockage phenomenon is reduced. Moreover, because the complete desktop image is output by the QEMU every time, the resolution ratio is fixed, and the size difference is small after the encoding and compression, the generated bandwidth fluctuation is small, and the stability of the network can be improved.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

The present application provides an apparatus for data transmission based on SPICE protocol, which comprises a memory and more than one program, wherein the more than one program is stored in the memory, and the more than one program is configured to be executed by more than one processor, and the apparatus is used for carrying out the SPICE protocol-based data transmission method described in one or more of the above embodiments.

FIG. 7 is a block diagram illustrating an apparatus 800 for SPICE protocol-based data transfer in accordance with an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 7, the apparatus 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing elements 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operation at the device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

Power components 806 provide power to the various components of device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 800.

The multimedia component 808 includes a screen that provides an output interface between the device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 800 is in an operational mode, such as a call mode, a recording mode, and a voice information processing mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect the open/closed state of the device 800, the relative positioning of components, such as a display and keypad of the apparatus 800, the sensor assembly 814 may also search for a change in the position of the apparatus 800 or a component of the apparatus 800, the presence or absence of user contact with the apparatus 800, orientation or acceleration/deceleration of the apparatus 800, and a change in the temperature of the apparatus 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communications between the apparatus 800 and other devices in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

Fig. 8 is a schematic diagram of a server in some embodiments of the present application. The server 1900 may vary widely by configuration or performance and may include one or more Central Processing Units (CPUs) 1922 (e.g., one or more processors) and memory 1932, one or more storage media 1930 (e.g., one or more mass storage devices) storing applications 1942 or data 1944. Memory 1932 and storage medium 1930 can be, among other things, transient or persistent storage. The program stored in the storage medium 1930 may include one or more modules (not shown), each of which may include a series of instructions operating on a server. Still further, a central processor 1922 may be provided in communication with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the server 1900.

The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input-output interfaces 1958, one or more keyboards 1956, and/or one or more operating systems 1941, such as WindowsServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, etc.

A non-transitory computer readable storage medium in which instructions, when executed by a processor of an apparatus (server or terminal), enable the apparatus to perform the SPICE protocol-based data transmission method shown in fig. 1.

A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an apparatus (server or terminal), enable the apparatus to perform the description of the data transmission method based on the SPICE protocol in the embodiment corresponding to fig. 1, and therefore, the detailed description thereof will not be repeated here. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in the computer program product or computer program embodiments referred to in the present application, reference is made to the description of the method embodiments of the present application.

Further, it should be noted that: embodiments of the present application also provide a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor can execute the computer instruction, so that the computer device performs the description of the data transmission method based on the SPICE protocol in the embodiment corresponding to fig. 1, and therefore, the description thereof will not be repeated here. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in the embodiments of the computer program product or the computer program referred to in the present application, reference is made to the description of the embodiments of the method of the present application.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

The data transmission method based on the SPICE protocol, the data transmission device based on the SPICE protocol and the readable storage medium provided by the application are introduced in detail, specific examples are applied in the text to explain the principle and the implementation of the application, and the description of the above embodiments is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A data transmission method based on SPICE protocol is applied to a SPICE server, and the method comprises the following steps:

transmitting the encoded data to the SPICE client through the first transmission channel.

2. The method of claim 1, further comprising:

according to the modified SPICE protocol, a second transmission channel is established with the SPICE client, and the second transmission channel is used for transmitting data from the SPICE client to the SPICE server;

receiving custom configuration data sent by the SPICE client through the second transmission channel, wherein the custom configuration data comprises a custom coding type and/or a custom code rate;

updating the configuration information according to the custom configuration data;

and re-initializing the encoder according to the updated configuration information.

3. The method of claim 1, further comprising:

analyzing the video data to obtain the current resolution of the desktop image;

upon determining that the current resolution is inconsistent with a stored resolution, reinitializing the encoder according to the current resolution;

updating the stored resolution to the current resolution.

4. The method of claim 1, wherein prior to encoding the video data with the encoder, the method further comprises:

converting the video data from a color space format to a luminance space format;

the encoding the video data with the encoder comprises:

encoding the video data in the luma spatial format with the encoder.

5. The method according to claim 1, wherein a data transmission format of the first transmission channel is defined in the modified SPICE protocol, and the transmitting the encoded data to the SPICE client through the first transmission channel comprises:

and after the coded data is packaged according to the data transmission format of the first transmission channel, the coded data is transmitted to the SPICE client through the first transmission channel.

6. The method of claim 1, further comprising:

acquiring data transmitted by the first transmission channel;

if the data transmitted by the first transmission channel is detected to be not in accordance with the preset coding requirement, coding the data transmitted by the first transmission channel again; wherein the preset encoding requirements include: when the data transmitted by the first transmission channel is video data, the coding type is H264 or H265.

7. A data transmission method based on SPICE protocol is applied to SPICE client, and the method comprises the following steps:

8. The method of claim 7, further comprising:

analyzing the coded data to obtain the coding type of the coded data;

and initializing a decoder according to the coding type, and decoding the coded data by using the decoder to obtain decoded data.

9. The method of claim 7, further comprising:

receiving user-defined configuration data through a preset option card, wherein the user-defined configuration data comprises a user-defined coding type and/or a user-defined code rate;

according to the modified SPICE protocol, a second transmission channel is created with the SPICE client, and the second transmission channel is used for transmitting data from the SPICE client to the SPICE server;

and sending the custom configuration data to the SPICE server through the second transmission channel, so that the SPICE server updates the configuration information of the encoder according to the custom configuration data and re-initializes the encoder.

10. The method according to claim 9, wherein a data transmission format of the second transmission channel is defined in the modified SPICE protocol, and the sending the custom configuration data to the SPICE server through the second transmission channel comprises:

and after the custom configuration data is encapsulated according to the data transmission format of the second transmission channel, transmitting the custom configuration data to the SPICE server through the second transmission channel.

11. A data transmission device based on SPICE protocol is characterized in that the device is applied to a SPICE server side and comprises the following components:

12. A data transmission device based on SPICE protocol, which is applied to SPICE client, the device includes:

13. An apparatus for SPICE protocol-based data transfer comprising a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs comprising instructions for performing the SPICE protocol-based data transfer method of any one of claims 1-6 or 7-10.

14. A readable storage medium having stored thereon instructions which, when executed by one or more processors of an apparatus, cause the apparatus to perform the SPICE protocol-based data transfer method of any one of claims 1 to 6 or 7 to 10.