CN111010582A - Cloud desktop image processing method, device and equipment and readable storage medium - Google Patents
Cloud desktop image processing method, device and equipment and readable storage medium Download PDFInfo
- Publication number
- CN111010582A CN111010582A CN201911310533.9A CN201911310533A CN111010582A CN 111010582 A CN111010582 A CN 111010582A CN 201911310533 A CN201911310533 A CN 201911310533A CN 111010582 A CN111010582 A CN 111010582A
- Authority
- CN
- China
- Prior art keywords
- video
- area
- image frame
- coding
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention discloses a cloud desktop image processing method, a device, equipment and a readable storage medium, wherein the method comprises the following steps: performing video region identification on the target desktop, and determining a video region and a non-video region; acquiring an image frame corresponding to a target desktop, coding a video area of the image frame according to a video coding technology, and coding a non-video area of the image frame according to a picture coding technology to obtain coded data; and sending the encoded data to the client so that the client displays the image frame. Therefore, when the client displays the target desktop, the video playing requirement and the display requirement of other contents in the non-video area can be met simultaneously, and the user experience can be improved.
Description
Technical Field
The invention relates to the technical field of cloud computing, in particular to a cloud desktop image processing method, a cloud desktop image processing device, cloud desktop image processing equipment and a readable storage medium.
Background
The cloud desktop application is one of the earliest applications in the industrial layout of the cloud computing technology, combines the cloud computing technology and the desktop virtualization technology, and provides a brand-new PC experience mode.
Compared with the traditional PC, the cloud desktop application provides desktop application services for users based on the cloud computing data center. Specifically, the data center allocates a desktop virtual machine to the user through a server virtualization technology, and maps and binds the desktop virtual machine and the user terminal through a virtual desktop remote transmission protocol, so as to provide real-time desktop experience for the user.
However, in the current desktop cloud scene, the desktop performance (such as desktop definition, video smoothness and time delay) cannot be effectively guaranteed for playing the video. That is, how to effectively solve the technical problem of guaranteeing the desktop performance when playing video is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a cloud desktop image processing method, a cloud desktop image processing device, cloud desktop image processing equipment and a readable storage medium, which can guarantee desktop performance when a video is played.
In order to solve the technical problems, the invention provides the following technical scheme:
a cloud desktop image processing method comprises the following steps:
performing video region identification on the target desktop, and determining a video region and a non-video region;
acquiring an image frame corresponding to the target desktop, encoding the video area of the image frame according to a video encoding technology, and encoding the non-video area of the image frame according to a picture encoding technology to obtain encoded data;
and sending the encoded data to a client so that the client can display the image frame.
Preferably, the encoding the video region of the image frame according to a video encoding technology and the encoding the non-video region of the image frame according to a picture encoding technology to obtain encoded data includes:
judging whether the current network is blocked;
if so, discarding the image frame;
if not, the step of coding the video area of the image frame according to the video coding technology and coding the non-video area of the image frame according to the picture coding technology is executed to obtain coded data.
Preferably, the determining whether the current network is congested includes:
receiving a data receiving response message fed back by the client, and calculating round-trip delay by using the data receiving response message;
judging whether the round-trip delay is greater than a basic delay or not;
if yes, determining that the current network is blocked; if not, the current network is determined to be smooth.
Preferably, the encoding technologies respectively corresponding to the video area and the non-video area are used to encode the image frame corresponding to the target desktop, so as to obtain encoded data, where the encoding technologies include:
counting frame loss information and frame sending information, and calculating video compression quality by using the frame loss information and the frame sending information;
and performing compression coding on the video area of the image frame according to the video compression quality, and coding the non-video area of the image frame according to a coding technology corresponding to the non-video area to obtain the coded data.
Preferably, the sending the encoded data to the client includes:
slicing the coded data to obtain a plurality of sliced data;
and sending the fragment data to the client.
Preferably, the performing video area identification on the target desktop to determine a video area and a non-video area includes:
counting whole image coding statistical information corresponding to the coded historical image frames respectively;
determining a regional video score corresponding to each drawing command in the historical image frame by using the whole image coding statistical information;
combining the drawing commands corresponding to the same updating area into a video stream tracking queue, and determining a queue video score of the video stream tracking queue by using the area video score;
synchronizing a frame rate and a continuous update time of the video stream tracking queue;
and determining the video area by using the queue video score, the frame rate and the continuous updating time which respectively correspond to each video stream tracking queue.
Preferably, the determining the video area by using the queue video score, the frame rate, and the continuous update time respectively corresponding to each video stream tracking queue includes:
respectively judging whether the queue video score, the frame rate and the continuous updating time which respectively correspond to each video stream tracking queue meet video conditions;
if so, determining the updating area corresponding to the video stream tracking queue as a composition area of the video area.
A cloud desktop image processing apparatus, comprising:
the video area identification module is used for identifying a video area of the target desktop and determining the video area and a non-video area;
the image coding module is used for obtaining an image frame corresponding to the target desktop, coding the video area of the image frame according to a video coding technology, and coding the non-video area of the image frame according to a picture coding technology to obtain coded data;
and the data sending module is used for sending the coded data to a client so that the client can display the image frame.
A cloud desktop image processing apparatus comprising:
a memory for storing a computer program;
and the processor is used for realizing the steps of the cloud desktop image processing method when the computer program is executed.
A readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the cloud desktop image processing method described above.
By applying the method provided by the embodiment of the invention, the video area of the target desktop is identified, and the video area and the non-video area are determined; acquiring an image frame corresponding to a target desktop, coding a video area of the image frame according to a video coding technology, and coding a non-video area of the image frame according to a picture coding technology to obtain coded data; and sending the encoded data to the client so that the client displays the image frame.
In the method, a video area of a target desktop is identified, and after an image frame of the target desktop is obtained, the image frame is encoded according to different encoding technologies in the video area and a non-video area to obtain encoded data. And then, the encoded data is sent to the client, and the client can display the image frame corresponding to the target desktop. For the video area, the video coding technology is adopted for coding, and for the non-video area, the image coding technology different from the video coding technology is adopted, so that when the client displays the target desktop, the video playing requirement and the other content display requirement of the non-video area can be met simultaneously, and the user experience can be improved.
Accordingly, embodiments of the present invention further provide a cloud desktop image processing apparatus, a device, and a readable storage medium corresponding to the cloud desktop image processing method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating an implementation of a cloud desktop image processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of video region identification according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating transmission of fragmented encoded data and statistics of network congestion information by using a TCP protocol according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating a specific application of a cloud desktop image processing method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a cloud desktop image processing apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a cloud desktop image processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a cloud desktop image processing apparatus according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart of a cloud desktop image processing method in an embodiment of the present invention, where the method is applicable to a server of a cloud desktop application, and the method includes the following steps:
s101, performing video area identification on the target desktop, and determining a video area and a non-video area.
The target desktop can be a desktop processed by a desktop virtual machine corresponding to any client of the cloud desktop application.
In practical application, in a desktop cloud scene, when a video is played, a target desktop can be divided into a video area and a non-video area. The video area continuously changes the display content according to the video playing frame rate (for example, greater than or equal to 23), while the display content of the non-video area remains unchanged (for example, the navigation bar), or the change frame rate is lower than the video playing frame rate (for example, the playing PPT).
Ways to determine the video region include, but are not limited to, the following enumeration:
mode 1, in a cloud scene, an image frame corresponding to a target desktop is composed of a plurality of drawing commands (drawable). Each drawing command corresponds to a drawing area. Since the content of the video is continuously changed during playing, the drawing command corresponding to the image area has a certain update frame rate and update duration on the time axis. Therefore, the video area can be determined by tracking the drawing command.
In the mode 2, it is considered that the content played by the video is mainly the content directly shot by the camera, so that the content composing the video itself appears as a pixel gradient characteristic (can be regarded as a natural image) on the picture, and therefore, the video area can be determined by counting the pixel change rule of the image frame.
In the mode 3, in an actual application environment, in some special display scenes, for example, when PPT with more picture contents is played, if the modes 1 and 2 are directly adopted, a situation of misjudgment may occur. To further improve the video region identification accuracy, a video region may be determined in conjunction with the encoded historical image frames. The specific implementation process comprises the following steps:
step one, counting whole image coding statistical information corresponding to coded historical image frames respectively;
determining a regional video score corresponding to each drawing command in the historical image frame by using the whole image coding statistical information;
combining drawing commands corresponding to the same updating area into a video stream tracking queue, and determining a queue video score of the video stream tracking queue by using the area video score;
step four, synchronizing the frame rate and the continuous updating time of the video stream tracking queue;
and fifthly, determining a video area by using the video score, the frame rate and the continuous updating time of the queue corresponding to each video stream tracking queue.
For convenience of description, the above five steps will be described in detail with reference to fig. 2.
Fig. 2 is a schematic view illustrating video area identification according to an embodiment of the present invention, wherein a frame: a frame representing a frame image; drawable: drawing commands, namely drawing commands generated inside the virtual machine, wherein one frame of image consists of a plurality of drawable units; drawable _ trace: a drawing command trace item storing a series of drawing instructions satisfying a condition; video _ stream: and the video stream is a series of drawing instructions according with the drawable instruction characteristic of video playing.
Wherein, the whole image coding statistical information at least comprises: the area proportion of h264 compression (a video coding and decoding technology) is taken, namely the proportion of natural images; the larger this fraction is, the higher the probability of belonging to a video; the proportion of the area that hits the cache, i.e. the proportion of the unnatural image, the lower this proportion, the higher the probability of belonging to the video.
The area video score refers to the probability that the area belongs to the video area from the viewpoint of the statistical information of the proportion, and the larger the value is, the more likely the area belongs to the video area; the queue video score is a probability of whether or not a region corresponding to a plurality of drawing commands for a same update region on the time axis belongs to the video region, and the larger the value is, the more likely the update region belongs to the video region.
The frame rate is the change frequency of the drawing command, and the video is more likely to be the change frequency is larger; the update duration is the duration of the continuous update, and the larger the value, the more likely it is to be a video.
Wherein, the fifth step can specifically comprise:
step 1, respectively judging whether queue video score, frame rate and continuous updating time respectively corresponding to each video stream tracking queue meet video conditions;
and 2, if so, determining the updating area corresponding to the video stream tracking queue as a component area of the video area.
Specifically, the video conditions can meet respective corresponding requirements for frame rate, continuous update time and queue video scoring according to specific video quality (such as high-definition, super-definition, standard definition, smoothness and other conventional video quality grades), and when all three indexes meet the requirements, the task meets the video conditions.
For example, the following steps are carried out: the picture coding statistical information is obtained after a complete image (namely a historical image frame) is coded, wherein the picture coding statistical information at least comprises information related to video stream identification: the proportion of natural images and the proportion of non-natural images. Then according to the coding statistical information, sequentially counting the natural image proportion of the image in each drawable to obtain the video score of the drawable. The corresponding relation between the natural image ratio and the video score is that the larger the natural image ratio is, the higher the video score of the drawable is. Forming a video stream tracking queue by using the drawables in the same updating area, and calculating the queue video score of the queue by using the area video score of each drawable; meanwhile, each time a drawable is processed, the frame rate and the update duration of the video stream tracking queue are updated synchronously and continuously. When the video score, the frame rate and the updating duration of a certain video stream tracking queue meet the video condition, the video stream is identified. Specifically, for video scoring, frame rate, and update duration in video conditions, the mutual constraints may include: 1. the higher the queue video score, the lower the demand duration; 2. the closer the frame rate of the queue is to the frame rate of the ordinary video playing (more than or equal to 23), the lower the required duration is; 3. the queue video scoring and the frame rate respectively have minimum limiting conditions, and if the requirements are not met, the queue video scoring and the frame rate cannot be identified as a video stream. After the video stream is determined, the update area corresponding to the video stream can be used as a composition area of the video area, so that the video area and the non-video area can be determined.
S102, obtaining an image frame corresponding to the target desktop, coding a video area of the image frame according to a video coding technology, and coding a non-video area of the image frame according to a picture coding technology to obtain coded data.
In a desktop cloud scene, an image frame of a target desktop is composed of a plurality of drawing commands, so that after the drawing commands corresponding to each region of the image frame are obtained, the image frame can be obtained by combining the drawing commands.
Specifically, the HEDC server can be used for periodically taking canvas images, namely image frames, of the server according to the frame control information. Wherein, HEDC: high Efficiency Desktop Coding scheme is an efficient transmission protocol adopted by Desktop cloud in wide area network.
In the present embodiment, an image frame is encoded by different encoding techniques according to a video region and a non-video region to obtain encoded data. The video coding technology may adopt a video coding technology commonly used in cloud computing, and the picture coding technology may also adopt a conventional picture coding technology, which are not listed and described herein.
S103, the coded data are sent to the client so that the client can display the image frame.
When the encoded data is transmitted, the encoded data can be transmitted to the client by using a common data transmission protocol such as a UDP protocol or a TCP protocol.
After receiving the encoded data, the client decrypts the encoded data to restore an image frame, and then displays the image frame on a display of the client, so that the desktop cloud effect can be achieved. That is, the client need not have a host device or chip that handles the desktop.
Preferably, in this embodiment, when the encoded data is sent to the client, in order to avoid that the client displays an obvious delay due to too long transmission time caused by too large encoded data, in order to enable the client to display an image frame as soon as possible to ensure real-time performance of the target desktop, the encoded data may be sent after being fragmented. That is, sending the encoded data to the client may specifically include: the coded data is sliced to obtain a plurality of sliced data; and sending the fragment data to the client. When transmitting the fragmented data, the fragmented data may be transmitted in parallel or in series.
By applying the method provided by the embodiment of the invention, the video area of the target desktop is identified, and the video area and the non-video area are determined; acquiring an image frame corresponding to a target desktop, coding a video area of the image frame according to a video coding technology, and coding a non-video area of the image frame according to a picture coding technology to obtain coded data; and sending the encoded data to the client so that the client displays the image frame.
In the method, a video area of a target desktop is identified, and after an image frame of the target desktop is obtained, the image frame is encoded according to different encoding technologies in the video area and a non-video area to obtain encoded data. And then, the encoded data is sent to the client, and the client can display the image frame corresponding to the target desktop. For the video area, the video coding technology is adopted for coding, and for the non-video area, the image coding technology different from the video coding technology is adopted, so that when the client displays the target desktop, the video playing requirement and the other content display requirement of the non-video area can be met simultaneously, and the user experience can be improved.
It should be noted that, based on the above embodiments, the embodiments of the present invention also provide corresponding improvements. In the preferred/improved embodiment, the same steps as those in the above embodiment or corresponding steps may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the preferred/improved embodiment herein.
Preferably, on the basis of the first embodiment, an improved scheme for adapting to the network transmission performance change is provided. In this modified solution, for encoding the video region of the image frame according to the video encoding technology and encoding the non-video region of the image frame according to the picture encoding technology in step S102, obtaining the encoded data may specifically include:
step one, judging whether a current network is blocked;
step two, if yes, discarding the image frame;
and step three, if not, the step of coding the video area of the image frame according to the video coding technology and coding the non-video area of the image frame according to the picture coding technology to obtain coded data is executed.
The specific implementation process for judging whether the current network is blocked includes:
step 1, receiving a data receiving response message fed back by a client, and calculating round-trip delay by using the data receiving response message;
step 2, judging whether the round-trip delay is greater than the basic delay or not;
step 3, if yes, determining that the current network is blocked;
and 4, if not, determining that the current network is smooth.
The data receiving response message may specifically be an ACK response message fed back by the client when the client receives data when the TCP protocol is used for data transmission; the data receiving response message may also be a packet loss rate fed back by the client or a frame loss rate calculated based on the packet loss rate when the UDP protocol is used for data transmission.
The basic delay, i.e. the time consumed by the network transmission in general, can be estimated according to the round-trip delay of the historical data transmission. For example, the average value or mode of the round trip delay of the historical data transmission is used as the basic delay.
The discarding of the image frame means that the image frame is discarded, that is, the server is not intended to send the encoded data corresponding to the image frame.
This preferred embodiment is described in detail below with reference to fig. 3. Fig. 3 is a schematic diagram of transmitting the fragmented encoded data by using the TCP protocol and counting network congestion information.
Network probing and congestion estimation may be performed using application level network probing and VEGAS congestion estimation techniques. The VEGAS is a network congestion algorithm in the TCP protocol. The server can be provided with a message slicer and a slicer transceiver.
After receiving a message (carrying encoded data) to be transmitted submitted by a server, the message slicer divides the message into small pieces of sending queues. The fragment transceiver acquires the fragments from the fragment sending queue, sends the fragments to the client and continuously receives ACK (acknowledgement character) messages fed back by the client
After receiving the fragments, the client firstly feeds back a fragment ACK message immediately, and then asynchronously delivers the fragments to the fragment assembly area; after the fragments are assembled by the client, the data are delivered to the message distributor for various message processing, so that the image frames are displayed in the display.
After receiving the fragment response, the server calculates Round Trip Time (RTT) of the round trip time (time spent when the server sends the fragment request to the server and receives the client response), estimates basic delay according to the historical information, and estimates network congestion information
And the message fragmenter determines whether to accept the message according to the feedback delay and the network congestion information. The time delay and network congestion information fed back by each level of the fragmentation controller determine whether to continue to send the fragmentation or not so as to prevent network congestion.
Further, when the network is congested, the image frames are discarded, and the desktop displayed by the client may have a reduced video quality due to the missing of the image frames. In order to ensure the video quality and reduce frame loss, the encoding precision of the image frame can be automatically reduced when the image frame is encoded, namely the video quality is reduced, so that the transmitted data volume is reduced, and the data transmission success rate is improved. That is to say, in the third step, the encoding technology corresponding to the video region and the non-video region respectively is used to encode the image frame corresponding to the target desktop to obtain the encoded data, which may specifically include the following steps:
step 1, counting frame loss information and frame sending information, and calculating video compression quality by using the frame loss information and the frame sending information;
and 2, carrying out compression coding on the video area of the image frame according to the video compression quality, and coding the non-video area of the image frame according to the coding technology corresponding to the non-video area to obtain coded data.
The frame loss information may specifically include the number of frame losses, the frame loss time, and the number of consecutive frame losses; the framing information may specifically include the number of successful frames, the total number of frames, and the number of consecutive transmissions.
When the frame loss quantity is more and the continuous frame loss quantity is more, the current network is blocked, and the video compression quality can be reduced. The more the number of successful frames is sent, the more the number of continuous sending is sent, which indicates that the current network is smooth, and the video compression quality can be improved.
It should be noted that, in practical applications, the first embodiment and the preferred solution provided based on the first embodiment can be reasonably combined to meet different requirements of specific application scenarios. In the following, a specific application scenario is taken as an example, and how to combine the cloud desktop image processing method and the priority scheme provided by the embodiment is described as an example.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating a specific application of a cloud desktop image processing method according to an embodiment of the present invention.
The specific implementation process of the cloud desktop image processing method can include:
1. and the HEDC server periodically acquires the canvas image of the server according to the frame control information.
2. A video region in the canvas image is identified.
3. When the message fragment and transmission control module reports that one frame of data is not transmitted, the frame is discarded. That is, at this time, the network has already congested one frame, and it is undoubtedly a burden on the network to continue to transmit the encoded data corresponding to the current frame, so that the current frame may be discarded.
4. When the message fragmentation and transmission control module is idle (if the message queue is empty or the queue length is smaller than a preset threshold), determining to continuously transmit the image frame, and calculating the change information of the video quality according to the statistical frame loss and frame transmission information to obtain the video compression quality of the current frame. The compression information, i.e. the encoding parameters in the video encoding technique, such as parameters controlling the compression accuracy, is calculated based on the video compression quality.
5. The HEDC encodes the video stream according to the identified video stream region and the incoming compressed information image frame, and then transmits the encoded data to the client after message fragmentation and transmission decision.
Therefore, the cloud desktop image processing method provided by the embodiment of the invention achieves network self-adaptation of video playing by identifying the video playing area and dynamically detecting the network condition. The specific technical effects comprise: 1. most common video playing scenes including webpage playing videos can be accurately identified; the PPT playing scenes which are easy to misjudge can be effectively distinguished;
2. the frame rate and compression quality of video playing can be self-adaptive according to the network quality: 2.1, when the network quality is reduced, the compression quality of the video stream is also reduced, and the video flow is reduced; 2.2, when the network quality is improved, the quality of the video stream can be dynamically improved, the image quality is clear, and the flow is high; and 2.3, the quality of the video stream is stable when the network quality is stable.
Example two:
corresponding to the above method embodiments, the embodiments of the present invention further provide a cloud desktop image processing apparatus, and the cloud desktop image processing apparatus described below and the cloud desktop image processing method described above may be referred to in a corresponding manner.
Referring to fig. 5, the apparatus includes the following modules:
the video area identification module 101 is used for identifying a video area of a target desktop and determining the video area and a non-video area;
the image coding module 102 is configured to obtain an image frame corresponding to a target desktop, code a video region of the image frame according to a video coding technology, and code a non-video region of the image frame according to a picture coding technology to obtain coded data;
and the data sending module 103 is configured to send the encoded data to the client, so that the client displays the image frame.
By applying the device provided by the embodiment of the invention, the video area of the target desktop is identified, and the video area and the non-video area are determined; acquiring an image frame corresponding to a target desktop, coding a video area of the image frame according to a video coding technology, and coding a non-video area of the image frame according to a picture coding technology to obtain coded data; and sending the encoded data to the client so that the client displays the image frame.
In the device, a video area of a target desktop is identified, and after an image frame of the target desktop is obtained, the image frame is encoded according to different encoding technologies in the video area and a non-video area to obtain encoded data. And then, the encoded data is sent to the client, and the client can display the image frame corresponding to the target desktop. For the video area, the video coding technology is adopted for coding, and for the non-video area, the image coding technology different from the video coding technology is adopted, so that when the client displays the target desktop, the video playing requirement and the other content display requirement of the non-video area can be met simultaneously, and the user experience can be improved.
In an embodiment of the present invention, the image encoding module 102 is specifically configured to determine whether a current network is congested; if yes, discarding the image frame; if not, the step of coding the video area of the image frame according to the video coding technology and coding the non-video area of the image frame according to the picture coding technology is executed to obtain the coded data.
In a specific embodiment of the present invention, the image encoding module 102 is specifically configured to receive a data reception response message fed back by the client, and calculate a round-trip delay by using the data reception response message;
judging whether the round-trip delay is greater than the basic delay or not; if yes, determining that the current network is blocked; if not, the current network is determined to be smooth.
In a specific embodiment of the present invention, the image encoding module 102 is specifically configured to count frame loss information and frame sending information, and calculate video compression quality by using the frame loss information and the frame sending information; and carrying out compression coding on a video area of the image frame according to the video compression quality, and coding a non-video area of the image frame according to a coding technology corresponding to the non-video area to obtain coded data.
In a specific embodiment of the present invention, the data sending module 103 is specifically configured to fragment encoded data to obtain a plurality of fragmented data; and sending the fragment data to the client.
In an embodiment of the present invention, the video area identification module 101 is configured to count whole image coding statistical information corresponding to coded historical image frames respectively; determining a regional video score corresponding to each drawing command in the historical image frame by using the whole image coding statistical information; combining drawing commands corresponding to the same updating area into a video stream tracking queue, and determining a queue video score of the video stream tracking queue by using the regional video score; synchronizing the frame rate and the continuous updating time of the video stream tracking queue; and determining a video area by using the video score, the frame rate and the continuous updating time of the queue corresponding to each video stream tracking queue.
In a specific embodiment of the present invention, the video area identification module 101 is specifically configured to respectively determine whether a queue video score, a frame rate, and a continuous update time respectively corresponding to each video stream tracking queue meet video conditions; if so, determining the updating area corresponding to the video stream tracking queue as the composition area of the video area.
Example three:
corresponding to the above method embodiment, an embodiment of the present invention further provides a cloud desktop image processing apparatus, and a cloud desktop image processing apparatus described below and a cloud desktop image processing method described above may be referred to in a corresponding manner.
Referring to fig. 6, the cloud desktop image processing apparatus includes:
a memory D1 for storing computer programs;
and a processor D2, configured to implement the steps of the cloud desktop image processing method according to the foregoing method embodiment when executing the computer program.
Specifically, referring to fig. 7, a specific structural diagram of a cloud desktop image processing apparatus provided in this embodiment is shown, where the cloud desktop image processing apparatus may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) storing an application 342 or data 344. Memory 332 and storage media 330 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 330 may include one or more modules (not shown), each of which may include a series of instructions operating on a data processing device. Still further, the central processor 322 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the cloud desktop image processing apparatus 301.
The cloud desktop image processing apparatus 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input-output interfaces 358, and/or one or more operating systems 341. Such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
The steps in the cloud desktop image processing method described above may be implemented by the structure of the cloud desktop image processing apparatus. The cloud desktop image processing device may be specifically a server of a cloud desktop application, or a device in a cloud server in a cloud scene.
Example four:
corresponding to the above method embodiment, an embodiment of the present invention further provides a readable storage medium, and a readable storage medium described below and a cloud desktop image processing method described above may be referred to in a corresponding manner.
A readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the cloud desktop image processing method of the above-described method embodiment.
The readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Claims (10)
1. A cloud desktop image processing method is characterized by comprising the following steps:
performing video region identification on the target desktop, and determining a video region and a non-video region;
acquiring an image frame corresponding to the target desktop, encoding the video area of the image frame according to a video encoding technology, and encoding the non-video area of the image frame according to a picture encoding technology to obtain encoded data;
and sending the encoded data to a client so that the client can display the image frame.
2. The cloud desktop image processing method of claim 1, wherein the encoding the video region of the image frame according to a video encoding technique and the encoding the non-video region of the image frame according to a picture encoding technique to obtain encoded data comprises:
judging whether the current network is blocked;
if so, discarding the image frame;
if not, the step of coding the video area of the image frame according to the video coding technology and coding the non-video area of the image frame according to the picture coding technology is executed to obtain coded data.
3. The cloud desktop image processing method of claim 2, wherein the determining whether the current network is congested comprises:
receiving a data receiving response message fed back by the client, and calculating round-trip delay by using the data receiving response message;
judging whether the round-trip delay is greater than a basic delay or not;
if yes, determining that the current network is blocked; if not, the current network is determined to be smooth.
4. The cloud desktop image processing method according to claim 2, wherein the encoding technologies corresponding to the video region and the non-video region respectively are used to encode the image frame corresponding to the target desktop to obtain encoded data, and the encoding technologies include:
counting frame loss information and frame sending information, and calculating video compression quality by using the frame loss information and the frame sending information;
and performing compression coding on the video area of the image frame according to the video compression quality, and coding the non-video area of the image frame according to a coding technology corresponding to the non-video area to obtain the coded data.
5. The cloud desktop image processing method of claim 1, wherein sending the encoded data to a client comprises:
slicing the coded data to obtain a plurality of sliced data;
and sending the fragment data to the client.
6. The cloud desktop image processing method according to any one of claims 1 to 5, wherein the performing video region recognition on the target desktop to determine a video region and a non-video region comprises:
counting whole image coding statistical information corresponding to the coded historical image frames respectively;
determining a regional video score corresponding to each drawing command in the historical image frame by using the whole image coding statistical information;
combining the drawing commands corresponding to the same updating area into a video stream tracking queue, and determining a queue video score of the video stream tracking queue by using the area video score;
synchronizing a frame rate and a continuous update time of the video stream tracking queue;
and determining the video area by using the queue video score, the frame rate and the continuous updating time which respectively correspond to each video stream tracking queue.
7. The cloud desktop image processing method of claim 6, wherein determining the video area by using the queue video score, the frame rate, and the continuous update time respectively corresponding to each video stream tracking queue comprises:
respectively judging whether the queue video score, the frame rate and the continuous updating time which respectively correspond to each video stream tracking queue meet video conditions;
if so, determining the updating area corresponding to the video stream tracking queue as a composition area of the video area.
8. A cloud desktop image processing apparatus, comprising:
the video area identification module is used for identifying a video area of the target desktop and determining the video area and a non-video area;
the image coding module is used for obtaining an image frame corresponding to the target desktop, coding the video area of the image frame according to a video coding technology, and coding the non-video area of the image frame according to a picture coding technology to obtain coded data;
and the data sending module is used for sending the coded data to a client so that the client can display the image frame.
9. A cloud desktop image processing apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the cloud desktop image processing method according to any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, realizes the steps of the cloud desktop image processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911310533.9A CN111010582B (en) | 2019-12-18 | 2019-12-18 | Cloud desktop image processing method, device and equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911310533.9A CN111010582B (en) | 2019-12-18 | 2019-12-18 | Cloud desktop image processing method, device and equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111010582A true CN111010582A (en) | 2020-04-14 |
| CN111010582B CN111010582B (en) | 2022-01-18 |
Family
ID=70116484
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911310533.9A Active CN111010582B (en) | 2019-12-18 | 2019-12-18 | Cloud desktop image processing method, device and equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111010582B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953980A (en) * | 2020-08-21 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Video processing method and device |
| CN112584152A (en) * | 2020-12-14 | 2021-03-30 | 上海贝锐信息科技股份有限公司 | Difference detection and regional dynamic encoding method in remote desktop, electronic device and computer-readable storage medium |
| CN113973224A (en) * | 2021-09-18 | 2022-01-25 | 阿里巴巴（中国）有限公司 | Method for transmitting media information, computing device and storage medium |
| CN114095762A (en) * | 2021-11-23 | 2022-02-25 | 湖南麒麟信安科技股份有限公司 | Data transmission method and device, computer equipment and computer readable storage medium |
| CN114265648A (en) * | 2020-09-16 | 2022-04-01 | 阿里巴巴集团控股有限公司 | Code scheduling method, server, client and system for acquiring remote desktop |
| WO2022068276A1 (en) * | 2020-09-29 | 2022-04-07 | 华为云计算技术有限公司 | Desktop cloud system and related method, apparatus, device, and medium |
| CN114827113A (en) * | 2022-04-18 | 2022-07-29 | 阿里巴巴（中国）有限公司 | Webpage access method and device |
| CN115022204A (en) * | 2022-05-26 | 2022-09-06 | 阿里巴巴（中国）有限公司 | RTC transmission delay detection method, device and equipment |
| CN117236566A (en) * | 2023-11-10 | 2023-12-15 | 山东顺发重工有限公司 | Whole-process visual flange plate package management system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932324A (en) * | 2011-09-02 | 2013-02-13 | 微软公司 | Cross-frame progressive spoiling support for reduced network bandwidth usage |
| CN105025335A (en) * | 2015-08-04 | 2015-11-04 | 合肥云中信息科技有限公司 | Method for video synchronization rendering in environment of cloud desktop |
| CN105245914A (en) * | 2015-09-30 | 2016-01-13 | 上海有孚计算机网络有限公司 | Cloud desktop high-definition video transmission protocol and cloud desktop high-definition video transmission architecture |
| CN105874807A (en) * | 2013-12-31 | 2016-08-17 | 谷歌公司 | Methods, systems, and media for remote rendering of web content on a television device |
| CN108174200A (en) * | 2017-11-28 | 2018-06-15 | 西安万像电子科技有限公司 | Image transfer method and device |
| CN108322781A (en) * | 2018-02-08 | 2018-07-24 | 武汉噢易云计算股份有限公司 | Promote the method and system that HD video effect is played in virtual desktop |
| US20180373546A1 (en) * | 2017-06-22 | 2018-12-27 | Vmware, Inc. | Hybrid software and gpu encoding for ui remoting |
| US20190320172A1 (en) * | 2018-04-12 | 2019-10-17 | Qualcomm Incorporated | Hardware-friendly sample adaptive offset (sao) and adaptive loop filter (alf) for video coding |
| CN110446041A (en) * | 2018-05-02 | 2019-11-12 | 中兴通讯股份有限公司 | A kind of video coding-decoding method, device, system and storage medium |
-
2019
- 2019-12-18 CN CN201911310533.9A patent/CN111010582B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932324A (en) * | 2011-09-02 | 2013-02-13 | 微软公司 | Cross-frame progressive spoiling support for reduced network bandwidth usage |
| US20130060886A1 (en) * | 2011-09-02 | 2013-03-07 | Microsoft Corporation | Cross-Frame Progressive Spoiling Support for Reduced Network Bandwidth Usage |
| CN105874807A (en) * | 2013-12-31 | 2016-08-17 | 谷歌公司 | Methods, systems, and media for remote rendering of web content on a television device |
| CN105025335A (en) * | 2015-08-04 | 2015-11-04 | 合肥云中信息科技有限公司 | Method for video synchronization rendering in environment of cloud desktop |
| CN105245914A (en) * | 2015-09-30 | 2016-01-13 | 上海有孚计算机网络有限公司 | Cloud desktop high-definition video transmission protocol and cloud desktop high-definition video transmission architecture |
| US20180373546A1 (en) * | 2017-06-22 | 2018-12-27 | Vmware, Inc. | Hybrid software and gpu encoding for ui remoting |
| CN108174200A (en) * | 2017-11-28 | 2018-06-15 | 西安万像电子科技有限公司 | Image transfer method and device |
| CN108322781A (en) * | 2018-02-08 | 2018-07-24 | 武汉噢易云计算股份有限公司 | Promote the method and system that HD video effect is played in virtual desktop |
| US20190320172A1 (en) * | 2018-04-12 | 2019-10-17 | Qualcomm Incorporated | Hardware-friendly sample adaptive offset (sao) and adaptive loop filter (alf) for video coding |
| CN110446041A (en) * | 2018-05-02 | 2019-11-12 | 中兴通讯股份有限公司 | A kind of video coding-decoding method, device, system and storage medium |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953980B (en) * | 2020-08-21 | 2023-11-21 | 北京达佳互联信息技术有限公司 | Video processing method and device |
| CN111953980A (en) * | 2020-08-21 | 2020-11-17 | 北京达佳互联信息技术有限公司 | Video processing method and device |
| CN114265648B (en) * | 2020-09-16 | 2023-12-26 | 阿里巴巴集团控股有限公司 | Code scheduling method, server, client and system for acquiring remote desktop |
| CN114265648A (en) * | 2020-09-16 | 2022-04-01 | 阿里巴巴集团控股有限公司 | Code scheduling method, server, client and system for acquiring remote desktop |
| WO2022068276A1 (en) * | 2020-09-29 | 2022-04-07 | 华为云计算技术有限公司 | Desktop cloud system and related method, apparatus, device, and medium |
| CN112584152A (en) * | 2020-12-14 | 2021-03-30 | 上海贝锐信息科技股份有限公司 | Difference detection and regional dynamic encoding method in remote desktop, electronic device and computer-readable storage medium |
| CN113973224A (en) * | 2021-09-18 | 2022-01-25 | 阿里巴巴（中国）有限公司 | Method for transmitting media information, computing device and storage medium |
| WO2023040825A1 (en) * | 2021-09-18 | 2023-03-23 | 阿里巴巴（中国）有限公司 | Media information transmission method, computing device and storage medium |
| CN113973224B (en) * | 2021-09-18 | 2024-05-28 | 阿里巴巴（中国）有限公司 | Media information transmission method, computing device and storage medium |
| CN114095762A (en) * | 2021-11-23 | 2022-02-25 | 湖南麒麟信安科技股份有限公司 | Data transmission method and device, computer equipment and computer readable storage medium |
| CN114827113A (en) * | 2022-04-18 | 2022-07-29 | 阿里巴巴（中国）有限公司 | Webpage access method and device |
| CN114827113B (en) * | 2022-04-18 | 2024-04-16 | 阿里巴巴（中国）有限公司 | Webpage access method and device |
| CN115022204A (en) * | 2022-05-26 | 2022-09-06 | 阿里巴巴（中国）有限公司 | RTC transmission delay detection method, device and equipment |
| CN115022204B (en) * | 2022-05-26 | 2023-12-05 | 阿里巴巴（中国）有限公司 | RTC transmission delay detection method, device and equipment |
| CN117236566A (en) * | 2023-11-10 | 2023-12-15 | 山东顺发重工有限公司 | Whole-process visual flange plate package management system |
| CN117236566B (en) * | 2023-11-10 | 2024-02-06 | 山东顺发重工有限公司 | Whole-process visual flange plate package management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111010582B (en) | 2022-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111010582B (en) | Cloud desktop image processing method, device and equipment and readable storage medium | |
| US11706272B2 (en) | Adaptive streaming with early client indication | |
| US11524230B2 (en) | Encoder tuning to improve tradeoffs between latency and video quality in cloud gaming applications | |
| CN111882626A (en) | Image processing method, apparatus, server and medium | |
| EP2974207B1 (en) | Playback stall avoidance in adaptive media streaming | |
| US20140286438A1 (en) | Quality of service management server and method of managing streaming bit rate | |
| US11159834B2 (en) | Managing congestion response during content delivery | |
| CN109819322A (en) | Video transmission method, device, computer readable storage medium and electronic equipment | |
| US11138715B2 (en) | Method and apparatus for determining experience quality of VR multimedia | |
| CN103747077A (en) | Transmission mechanism adjusting method, server side and client | |
| CN111886875B (en) | Method and server for transmitting media content through network | |
| US20230217034A1 (en) | Split Rendering To Improve Tolerance To Delay Variation In Extended Reality Applications With Remote Rendering | |
| CN112866746A (en) | Multi-path streaming cloud game control method, device, equipment and storage medium | |
| CN114389959B (en) | Network congestion control method, device, electronic equipment and storage medium | |
| KR20050091054A (en) | Device and process for adjusting the bit rate of a stream of contents and associated products | |
| US9009344B2 (en) | Method of sending data and associated device | |
| KR20230136128A (en) | Methods, devices and systems for adapting user input in cloud gaming | |
| CN116962179A (en) | Network transmission optimization method and device, computer readable medium and electronic equipment | |
| CN112737971B (en) | Data processing method, device, storage medium and network equipment | |
| WO2020194957A1 (en) | Image transmission/reception system, image transmission device, image reception device, image transmission/reception method and program | |
| CN117914750B (en) | Data processing method, apparatus, computer, storage medium, and program product | |
| JP7350866B2 (en) | Display control device, transmission device, display control method and program | |
| CN114928757B (en) | 360-degree VR video data transmission method and device based on millimeter waves | |
| EP4131952A1 (en) | Transmission device, transmission method, and program | |
| EP4024811A1 (en) | Method and system for enhancing quality of experience (qoe) of video reception at receiver |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |