CN116668741A - Cloud desktop display method, image display method, device and storage medium - Google Patents

Cloud desktop display method, image display method, device and storage medium Download PDF

Info

Publication number
CN116668741A
CN116668741A CN202310599820.6A CN202310599820A CN116668741A CN 116668741 A CN116668741 A CN 116668741A CN 202310599820 A CN202310599820 A CN 202310599820A CN 116668741 A CN116668741 A CN 116668741A
Authority
CN
China
Prior art keywords
area
time
data
update
encoded data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310599820.6A
Other languages
Chinese (zh)
Inventor
方杰
任晋奎
张献涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202310599820.6A priority Critical patent/CN116668741A/en
Publication of CN116668741A publication Critical patent/CN116668741A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/167—Position within a video image, e.g. region of interest [ROI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

The invention provides a cloud desktop display method, an image display method, equipment and a storage medium, wherein the method comprises the following steps: an update area in screen content of the cloud desktop at a first time is determined. And if the area of the updated area is larger than the set area threshold, encoding the screen content by adopting a lossy encoding mode to obtain first encoded data. And transmitting the first encoded data and the position information of the update area to the client so that the client determines first decoding data corresponding to the position information according to the first encoded data. If the updating area does not update the content within the set time after the first time, encoding the content in the updating area at the first time by adopting a lossless encoding mode to obtain second encoded data; and sending the second encoded data to the client so that the client determines second decoded data corresponding to the second encoded data. The invention can simultaneously consider the picture display quality, the image compression efficiency and the bandwidth when the cloud desktop screen content is encoded.

Description

Cloud desktop display method, image display method, device and storage medium
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a cloud desktop display method, an image display device, and a storage medium.
Background
The virtual desktop, which may also be referred to as a cloud desktop, usually a user may install a cloud desktop transport protocol client (abbreviated as a cloud desktop client) on a local terminal to perform communication connection with a cloud desktop server of a cloud. The cloud desktop client and the cloud desktop server perform streaming data transmission through a remote desktop protocol, the cloud desktop server encodes screen content of the cloud desktop and then transmits the encoded screen content to the cloud desktop client, and the cloud desktop client decodes the screen content and then presents the decoded screen content to a user through a local display system.
In the process of encoding screen contents of a cloud desktop, a lossy encoding algorithm or a lossless encoding algorithm is generally used. The lossy coding algorithm has a relatively high compression ratio, but is finished at the cost of losing image quality, so that the picture display quality is poor, and some information in the screen content such as text parts are easy to blur. While the lossless coding algorithm can ensure the image quality, the compression efficiency is low, the occupied bandwidth is large, and the real-time performance of data transmission is poor.
Disclosure of Invention
The embodiment of the invention provides a display method, an image display method, equipment and a storage medium for a cloud desktop, which can simultaneously consider the picture display quality and the bandwidth when carrying out the content coding of a cloud desktop screen.
In a first aspect, an embodiment of the present invention provides a method for displaying a cloud desktop, which is applied to a server running the cloud desktop, where the method includes:
determining an update area in screen content of a cloud desktop at a first moment;
if the area of the updated area is determined to be larger than the set area threshold, the following coding process is executed:
coding the screen content by adopting a lossy coding mode to obtain first coded data;
transmitting the first encoded data and the position information of the update area to a client so that the client determines first decoding data corresponding to the position information according to the first encoded data;
if the updating area does not update the content within the set time after the first time, encoding the content in the updating area at the first time by adopting a lossless encoding mode to obtain second encoded data;
and sending the second encoded data to the client so that the client determines second decoded data corresponding to the second encoded data.
In a second aspect, an embodiment of the present invention provides a display device of a cloud desktop, which is applied to a server running the cloud desktop, where the device includes:
the determining module is used for determining an updating area in screen content of the cloud desktop at the first moment;
the execution module is used for executing the following coding process if the area of the update area is determined to be larger than the set area threshold value:
coding the screen content by adopting a lossy coding mode to obtain first coded data;
transmitting the first encoded data and the position information of the update area to a client so that the client determines first decoding data corresponding to the position information according to the first encoded data;
if the updating area does not update the content within the set time after the first time, encoding the content in the updating area at the first time by adopting a lossless encoding mode to obtain second encoded data;
and sending the second encoded data to the client so that the client determines second decoded data corresponding to the second encoded data.
In a third aspect, an embodiment of the present invention provides a method for displaying a cloud desktop, which is applied to a client corresponding to the cloud desktop, where the method includes:
Receiving first coded data and position information of an update area sent by a server; the first coding data are obtained by coding the screen content in a lossy coding mode when the server determines that the updated area in the screen content of the cloud desktop at the first moment is larger than a set area threshold;
decoding the first encoded data to obtain first decoded data;
extracting decoding data corresponding to the position information from the first decoding data and displaying the extracted decoding data;
receiving second coded data sent by the server; the second coding data are obtained by coding the content in the update area at the first moment in a lossless coding mode when the server determines that the update area does not update the content within a set time period after the first moment;
and decoding the second coded data to obtain second decoded data, and displaying the second decoded data.
In a fourth aspect, an embodiment of the present invention provides a display device of a cloud desktop, which is applied to a client corresponding to the cloud desktop, where the device includes:
the receiving module is used for receiving the first coded data sent by the server and the position information of the update area; the first coding data are obtained by coding the screen content in a lossy coding mode when the server determines that the updated area in the screen content of the cloud desktop at the first moment is larger than a set area threshold;
The decoding module is used for decoding the first coded data to obtain first decoded data;
the display module is used for extracting the decoding data corresponding to the position information from the first decoding data and displaying the extracted decoding data;
the receiving module is further used for receiving second coded data sent by the server; the second coding data are obtained by coding the content in the update area at the first moment in a lossless coding mode when the server determines that the update area does not update the content within a set time period after the first moment;
the display module is further configured to decode the second decoded data obtained by the second encoded data, and display the second decoded data.
In a fifth aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor, a communication interface; wherein the memory has executable code stored thereon, which when executed by the processor causes the processor to perform the method of displaying a cloud desktop according to the first or third aspect.
In a sixth aspect, embodiments of the present invention provide a non-transitory machine-readable storage medium having executable code stored thereon, which when executed by a processor of an electronic device, causes the processor to at least implement a method for displaying a cloud desktop according to the first or third aspect.
In a seventh aspect, an embodiment of the present invention provides an image display method, including:
determining an image update area in a video picture at a target moment;
if the area of the image update area is determined to be larger than the set area threshold value, the following coding process is executed:
coding the video picture by adopting a lossy coding mode to obtain first coded data;
transmitting the first encoded data and the position information of the image update area to a display device, so that the display device determines first decoding data corresponding to the position information according to the first encoded data and displays the first decoding data;
if the image updating area does not generate content updating any more within a set time after the target time, encoding the content in the image updating area by adopting a lossless encoding mode to obtain second encoded data;
and sending the second encoded data to the display device so that the display device determines and displays second decoded data corresponding to the second encoded data.
According to the cloud desktop display method, the updated area in the screen content of the cloud desktop at the first moment is determined by being applied to the service end running the cloud desktop, if the area of the updated area is determined to be larger than the set area threshold, the changed part in the screen content is more, at the moment, the screen content is encoded in a lossy encoding mode to obtain first encoded data, the image compression efficiency of the screen content can be improved, and the occupation of bandwidth is reduced. And then, the server side sends the first encoded data and the position information of the update area to the client side, so that the client side can extract the decoded data corresponding to the position information from the first decoded data obtained by decoding the first encoded data, display the extracted decoded data and present the decoded data to a user. If it is determined that the update area does not update the content within the set period of time after the first time, it is indicated that the update area does not change for a long period of time and may be in a relatively static state, at this time, the content in the update area at the first time is encoded by using a lossless encoding manner to obtain second encoded data, so that the image quality of the update area is improved, and then the second encoded data is sent to the client, so that the client displays second decoded data with better image quality obtained by decoding the second encoded data. In conclusion, the method and the device can simultaneously consider the picture display quality and the bandwidth when the cloud desktop screen content is encoded.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic structural diagram of a cloud desktop system according to an embodiment of the present invention;
fig. 2 is a flow chart of a method for displaying a cloud desktop according to an embodiment of the present invention;
fig. 3 is a flow chart of a method for displaying a cloud desktop according to an embodiment of the present invention;
fig. 4 is a flowchart of an update determination method for updating area content according to an embodiment of the present invention;
fig. 5 is a specific example diagram of a display method of a cloud desktop according to an embodiment of the present invention;
fig. 6 is a flow chart of a method for displaying a cloud desktop according to an embodiment of the present invention;
fig. 7a is a schematic structural diagram of a first layer according to an embodiment of the present invention;
fig. 7b is a schematic structural diagram of a second layer according to an embodiment of the present invention;
fig. 8 is a specific example diagram of a display method of a cloud desktop according to an embodiment of the present invention;
Fig. 9 is a schematic structural diagram of a display device of a cloud desktop according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an electronic device according to the present embodiment;
fig. 11 is a schematic structural diagram of a display device of a cloud desktop according to an embodiment of the present invention;
fig. 12 is a schematic structural diagram of an electronic device according to the present embodiment;
fig. 13 is a flowchart of an image display method according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention. In addition, the sequence of steps in the method embodiments described below is only an example and is not strictly limited.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the embodiments of the present invention are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.
The cloud desktop is cloud desktop service based on cloud computing, and an easy-to-use, safe and efficient cloud desktop office system is provided for users by using desktop as a service (Desktop as a Service, daaS for short). As shown in the schematic structural diagram of the cloud desktop system in fig. 1, the cloud desktop runs on the server 10, for example, a plurality of virtual machines are built on the server 10, and different cloud desktops are built in different virtual machines, so the cloud desktops can also be called as cloud computers for being distributed to different users for use. The user connects the cloud desktop through the public network or the private line by using the client 20 installed with the client program corresponding to the cloud desktop, thereby realizing remote access to the cloud desktop.
The server 10 is a server device that provides cloud desktop service, and may perform cloud desktop management, and may respond to a cloud desktop service request of the client 20. The server 10 not only can be deployed with a cloud desktop, but also can be deployed with related communication protocols, management and control services and the like.
The client 20 refers to a device or a software function module that can access the server 10 through a wireless network, such as a mobile phone, a notebook, a tablet computer, a POS machine, a thin host, etc., where a client program corresponding to a cloud desktop is deployed.
It should be understood that, in order to realize remote access of the client 20 to the cloud desktop, in practical application, the server 10 generally obtains and encodes the screen content of the cloud desktop, and sends the encoded data to the client 20, and after the client 20 receives and decodes the encoded data, the decoded content can be presented to the user through the local display system.
However, when encoding screen content of a cloud desktop, a lossy encoding algorithm or a lossless encoding algorithm is generally used at present. Among them, lossy coding algorithms such as JPEG, H264, etc., although having a relatively high compression ratio, do so at the expense of image quality, which is poor in picture display quality. While lossless coding algorithms such as RLE, LZ, etc., while guaranteeing image quality, compression efficiency is low and occupied bandwidth is large. Therefore, the current screen content coding strategy of the cloud desktop cannot consider both the picture display quality and the bandwidth.
In view of this, the embodiment of the invention provides a display method of a cloud desktop, which realizes the consideration of image quality and network bandwidth.
The flow schematic diagram of the method can be seen in fig. 2, and the method is applied to a server side running a cloud desktop, as shown in fig. 2, and comprises the following steps:
201. An update area in screen content of the cloud desktop at a first time is determined.
202. If the area of the update area is larger than the set area threshold, encoding the screen content by adopting a lossy encoding mode to obtain first encoded data, and transmitting the first encoded data and the position information of the update area to the client so that the client determines first decoding data corresponding to the position information according to the first encoded data.
203. If the updating area does not generate the content updating any more within the set time after the first time, the content in the updating area at the first time is coded by adopting a lossless coding mode to obtain second coded data, and the second coded data is sent to the client so that the client determines second decoding data corresponding to the second coded data.
In practical application, the server side invokes an application program interface (Application Program Interface, abbreviated as API) provided by the operating system through the CPU, and through the API, an update area in the screen content of the cloud desktop at the first moment can be determined, specifically, the API will send a notification when the screen content of the cloud desktop is found to be updated, and at this moment, the server side can determine the update area in the screen content of the cloud desktop at the first moment based on the notification. The first time is the time when it is determined that an update area exists, and actually, after the cloud desktop is started to run, whether the update area exists at different times can be determined in real time. It will be understood simply that if the screen content of the previous frame is changed from the screen content of the previous frame, the changed area is the updated area determined at the current time.
It should be noted that the update area in the screen content may be one or more. In the implementation, if the current screen content only includes one updated region, the area of the updated region is directly compared with the set area threshold, if the area of the updated region is greater than the set area threshold, the updated region is marked as a dirty region (lossy region), and the current screen content is encoded by adopting a lossy encoding mode (such as video encoding). Otherwise, if the area of the updated area is smaller than the set area threshold, the updated area is encoded by adopting a lossless encoding mode, wherein the lossless encoding mode comprises a run-length compression algorithm (Run Length Encoding, RLE for short); lempel Zip code, LZ for short; zstandard, abbreviated ZSTD, etc.
If the current screen content includes a plurality of update areas, for example, two update areas, the total area of the two update areas is calculated and compared with a set area threshold, if the total area is greater than the set area threshold, both update areas are marked as lossy areas, and the current screen content is encoded in a lossy encoding manner, where it should be noted that in the case that the total area of the plurality of update areas is greater than the set area threshold, the encoding process of steps 203-206 needs to be performed for each update area. Otherwise, if the total area is smaller than the set area threshold, the lossless coding mode is adopted for coding for each updated area.
Optionally, the area threshold may be determined according to a scene feature corresponding to the screen content, for example, in a scene with a higher requirement on image quality (for example, in a scene of a written document, a client user needs a particularly clear image quality), the area threshold may be set to be larger, and in a scene with a higher requirement on real-time (for example, in a video scene of watching a game, a movie, etc., the client user is more sensitive to real-time and bandwidth), the area threshold may be set to be smaller. As an example, the area threshold may be set to a size of 256×256, where two 256 are the number of pixels of the image that are long and wide, respectively.
In view of the foregoing, the following describes, by taking the area of the update area being greater than the set area threshold, a lossy coding method as an example of video coding, where the video coding refers to: and performing video compression coding on the complete screen content by using a video coder to obtain a coding result, namely obtaining first coding data. And then, the first encoded data and the position information of the update area are transmitted to the client. After receiving the first encoded data, the client determines the encoding type of the first encoded data, that is, determines whether the encoding mode of the first encoded data is a lossy encoding mode or a lossless encoding mode. After determining that the coding mode of the first coding data is a lossy coding mode (i.e. coding by adopting a video encoder), the client decodes the first coding data by adopting a video decoder corresponding to the video encoder to obtain complete screen content (i.e. first decoding data), and extracts decoding data corresponding to the updating area from the first decoding data by combining the received position information of the updating area so as to render and display the decoding data on the client. Among other things, video encoders include, but are not limited to: h.264, h.265, etc. The data sent by the server to the client includes the first encoded data and the position information of the update area, and also includes information indicating the encoding type corresponding to the first encoded data, so that the client determines what decoding mode should be adopted to decode the first encoded data.
It should be noted that, for the first decoded data, the client only extracts the partially decoded data corresponding to the update area from the first decoded data to perform rendering display, i.e. "incremental rendering": for example, the client displays the picture X before decoding the first decoded data, locates the update area in the picture X, keeps the content of the non-update area in the picture X unchanged, and only renders and displays the first decoded data corresponding to the update area in the update area.
In addition, in the embodiment of the present invention, regarding the update area determined at the first time and having an area greater than the set area threshold, on the one hand, the current complete screen content including the update area is lossy-encoded, the obtained first encoded data and the position information of the update area are sent to the client for decoding and displaying, on the other hand, whether the update area continues to update the content in a period of time or not is observed, if the update is performed, other processing is not performed on the content of the update area, and if the update is not performed, lossless encoding is performed on the content in the update area.
Specifically, it is determined that an update area in the screen content of the cloud desktop occurs at a first moment, if the update area is not updated any more within a set period of time (for example, 50ms, which can be determined according to practical situations) after the first moment begins, it is indicated that the update area may be in a relatively static state at this moment, that is, the content in the update area is not changed in this period of time, at this moment, the content in the update area at the first moment may be encoded in a lossless encoding manner to obtain second encoded data, so as to ensure the definition of the content in the update area, improve the picture quality, and enable the picture to achieve an effect from blurring to definition, or may be understood as enabling the picture to gradually sharpen from the lossy picture quality to the lossless picture quality.
And then, the server side can send the second coded data to the client side. After receiving the second encoded data, the client judges the encoding type of the second encoded data, and after determining that the encoding mode of the second encoded data is a lossless encoding mode, the client decodes the second encoded data by adopting a lossless decoding mode corresponding to the lossless encoding mode to obtain a clearer updating area and displays the clearer updating area on the client.
Based on the cloud desktop display method provided by the embodiment of the invention, the server side aims at the updated area in the screen content of the cloud desktop at the first moment, if the area of the updated area is determined to be larger than the set area threshold value, the changed part in the screen content is more, at the moment, the screen content is encoded in a lossy encoding mode to obtain first encoded data, so that the image compression efficiency of the screen content can be improved, and the occupation of bandwidth is reduced. If it is determined that the updated area is not updated again in the set period after the first time, it is indicated that the updated area is unchanged for a long period of time, at this time, the lossless coding mode is adopted to code the content in the updated area at the first time to obtain second coded data, the image quality of the updated area is improved, and then the second coded data is sent to the client, so that the client can display second decoded data with better image quality obtained by decoding the second coded data. This allows for both bandwidth and image quality requirements.
Fig. 3 is a flow chart of a method for displaying a cloud desktop according to an embodiment of the present invention, as shown in fig. 3, where the method includes:
301. an update area in screen content of the cloud desktop at a first time is determined.
302. It is determined whether the area of the update area is greater than a set area threshold, if so, steps 303-305 are performed, otherwise step 306 is performed.
303. And encoding the screen content by adopting a lossy encoding mode to obtain first encoded data, and transmitting the first encoded data and the position information of the update area to the client so that the client determines first decoding data corresponding to the position information according to the first encoded data.
304. Determining whether the content update occurs again in the update area between the first time and the second time after the interval setting time, if so, executing step 305, otherwise, ending.
305. And encoding the content in the first time update area by adopting a lossless encoding mode to obtain second encoded data, and transmitting the second encoded data to the client so that the client displays second decoded data obtained by decoding the second encoded data.
306. And encoding the updated region by adopting a lossless encoding mode to obtain second encoded data, so that the client displays second decoded data obtained by decoding the second encoded data.
In practical application, after determining the update area in the screen content of the cloud desktop at the first moment, if it is determined that the area of the update area is greater than the set area threshold, the encoding process performed may refer to the foregoing embodiment, which is not described herein in detail. If the area of the updated area is not larger than the set area threshold, the changed part in the screen content is not more, and at the moment, the updated area is coded in a lossless coding mode, so that the picture display quality can be ensured, the whole screen content is not required to be subjected to lossless coding, only the updated area with a small coding area is required, and the transmitted data quantity and occupied bandwidth are small. In summary, the invention can adopt a flexible coding mode according to the area size of the update area, and can simultaneously consider the picture display quality, the image compression efficiency and the bandwidth.
Fig. 4 is a flow chart of a method for determining update of content in an update area according to an embodiment of the present invention, as shown in fig. 4, the method includes:
401. the first encoded data and the location information of the update area are transmitted to the client.
402. Record information corresponding to the update area is generated, wherein the record information comprises the first moment, the position information of the update area and the set lossy coding identifier.
The lossy coding identifier is an identifier indicating the lossy coding method adopted.
403. And when the current timing of the timer reaches the second moment, acquiring the record information.
404. And determining whether the time interval between the first time and the second time in the recorded information is longer than a set duration.
405. If the content update time is longer than the set time length, determining that the content update does not occur in the update area between the first time and the second time.
In practical application, when the cloud desktop is started, a set timer (hereinafter referred to as a timer) is started, the timer has a preset timing duration for judging whether the content in the update area is changed,in specific implementation, assuming that the timing duration is 50ms, a determination is made at intervals of 50ms to determine whether an update area marked as lossy exists in the screen content, where the marking lossy means that the update area satisfies the aforementioned condition of being greater than the set area threshold. If at the first time t 1 If it is determined that the screen content X of the cloud desktop has an update area a, and the area of the update area a is greater than a set area threshold, record information corresponding to the update area, such as "t", is generated 1 And at the moment of sending, updating the position information of the area A, performing video compression coding on the screen content X, and sending the first coded data obtained after the video compression coding and the position information of the area A to the client.
Thereafter, at a second time t when the current timing of the timer arrives 2 Acquiring record information, judging whether the time interval between the first time and the second time in the record information is longer than a set time length (such as 100 ms), if so, determining that the updating area A between the first time and the second time is not updated, encoding the updating area A in a lossless encoding mode, and transmitting the obtained second encoded data to the client. After the second encoded data is sent to the client, the record information is deleted. It follows that for a screen content X containing an update area a, the client is at a first time t 1 The position information of the complete screen content and the updated area A which are coded by adopting the lossy coding mode is received, and at the second time t 2 The encoding result of the content of the update area a encoded by the lossless encoding method is received. The set duration may be determined according to the network quality between the server and the client, and in particular, if the network quality is better (for example, the available bandwidth is more), the set duration may be set to be shorter, so that lossless encoding is facilitated. Otherwise, if the network quality is poor, the set duration may be set to be slightly longer.
Further, if at the first time t 1 And a second time t 2 Third time t therebetween 3 Determining that the content in the update area A is updated, and updating the content in the record informationFirst time t 1 For the third time t 3 The method comprises the steps of carrying out a first treatment on the surface of the At the second time t when the current timing of the timer arrives 2 After the recorded information is acquired, if the third time t in the recorded information is determined 3 And a second time t 2 The timer is reset if the time interval is not greater than the set duration.
Specifically, assume that a first time t 1 And determining an updating area A in the screen content X of the cloud desktop, and encoding the screen content X by adopting a lossy encoding mode if the area of the updating area A is larger than a set area threshold value, so as to generate a piece of record information. Here, it is assumed that the first time is a time at which the timer starts starting the timer for the set period of time. Thereafter, at a first time t 1 Third time t thereafter 3 If it is determined that the content in the update area a has been updated (assuming that the present time has not been completed), the screen content X is continuously encoded by the lossy encoding method, and at this time, the first time t in the record information is updated 1 For the third time t 3 To obtain updated recorded information. Thereafter, at a second time t when the current timing of the timer arrives 2 (assuming that the present time is over), a third time t in the recorded information is determined 3 And a second time t 2 If the time interval of the time is longer than the set time, determining a third time t 3 And a second time t 2 And the updating area A does not update the content any more, the updating area A is coded in a lossless coding mode, and the obtained second coded data is sent to the client. And if the third time t in the recorded information is determined 3 And a second time t 2 The timer is reset if the time interval is not greater than the set duration.
In order to facilitate understanding, in the simple way, if the content of the update area a is static or is not changed frequently, two moments separated by a timing time period are separated from each other in the screen content X, the content in the update area a is not changed, and at this moment, the lossless coding mode is adopted to code the content in the update area a, so that the picture quality can be ensured, and the picture quality enhancement can be realized. Because only the updating area A is coded at the moment and the complete screen content is not coded, the transmitted data volume is small and the occupied bandwidth is small. If the change frequency of the update area A is high, the image quality enhancement processing is not performed at this time, and the corresponding complete screen content is encoded by adopting a lossy encoding mode, so that the bandwidth is not excessively occupied, and the instantaneity is good.
For easy understanding, a complete example is provided below with reference to fig. 5, and the method of the present invention is specifically described when applied to a server running a cloud desktop:
after the screen content of the cloud desktop is updated, the server determines an updating area in the screen content of the cloud desktop at the first moment, and judges whether the area of the updating area is larger than a set area threshold. If the area of the update area is larger than the set area threshold, encoding the screen content by adopting a lossy encoding mode to obtain first encoded data, filling the position information of the update area on the basis of the first encoded data, and transmitting the first encoded data and the position information of the update area to the client.
And if the area of the updated area is not greater than the set area threshold, encoding the updated area by adopting a lossless encoding mode to obtain second encoded data, and transmitting the second encoded data to the client.
In the process, the timer carries out judging operation based on the set time length, judges whether an updated area with the area larger than the set area threshold value exists or not, namely, a loss area exists, and if the updated area does not exist, the work is ended; if so, continuing to judge whether the time interval between the first time and the second time is longer than the set time length, if so, determining that the updating area between the first time and the second time is not updated, encoding the updating area by adopting a lossless encoding mode, and transmitting the obtained second encoded data to the client. If not, the operation is ended.
Fig. 6 is a flow chart of a method for displaying a cloud desktop according to an embodiment of the present invention, where the method is applied to a client corresponding to the cloud desktop, as shown in fig. 6, and the method includes:
601. receiving first coded data and position information of an update area sent by a server; the first coding data are obtained by coding the screen content in a lossy coding mode when the server determines that the updating area in the screen content of the cloud desktop at the first moment is larger than the set area threshold.
602. The first encoded data is decoded to obtain first decoded data.
603. And extracting the decoding data corresponding to the position information from the first decoding data and displaying the extracted decoding data.
604. Receiving second coded data sent by a server; the second coding data are obtained by coding the content in the updated area at the first moment by adopting a lossless coding mode when the server determines that the updated area does not update the content within a set time length after the first moment.
605. And decoding the second encoded data to obtain second decoded data, and displaying the second decoded data.
The above execution may refer to the related descriptions in the other embodiments, which are not described in detail herein.
In an embodiment of the present invention, extracting decoded data corresponding to position information from first decoded data and displaying the extracted decoded data includes: rendering the extracted decoded data at the first layer to display the first layer. Based on this, the second decoded data is displayed, including: the second decoded data is rendered at the second layer to display the second layer on the first layer.
In practical applications, the client determines whether to use a lossy encoding mode or a lossless encoding mode according to the encoding types of the first encoded data and the second decoded data received from the server. That is, the first encoded data is transmitted to the client together with the position information of the update area, and the encoding type information corresponding to the first encoded data is also included, and the second encoded data is transmitted to the client together with the second encoded data.
For first coded data obtained by coding in a lossy coding mode (such as video coding): and decoding by adopting a video decoder corresponding to a video encoder used for video encoding to obtain complete screen content, positioning the content of the update region from the complete screen content according to the position information of the update region, and performing incremental rendering on the content, namely, keeping the content of a non-update region in the displayed content unchanged, and only performing incremental rendering on the content in the update region.
For second coded data obtained by coding in a lossless coding mode: and decoding the updated region by adopting a lossless decoding mode corresponding to the lossless coding mode to obtain the content of the updated region, and rendering.
Through the above processing procedure, for the updated area, a display effect of "sharpening progression" can be obtained, that is: the first decoded data is encoded by a lossy encoding mode at the beginning and then decoded, and the display content of the obtained first decoded data is blurred. Then, the lossless coding mode is adopted for coding and then decoding is carried out, and the obtained second decoded data shows clear content, so that the effect from blurring to definition is achieved.
Note that, the two-time rendering display for this updated region: the decoded data extracted from the first decoded data is rendered and displayed on the first layer, and when the second decoded data is displayed, a new layer, i.e., a second layer, is reproduced, the second layer covers the first layer, and the second decoded data is rendered and displayed on the second layer. Referring to fig. 7a and fig. 7b, fig. 7a is a schematic structural diagram of a first layer, and in fig. 7a, the update area a is the decoded data extracted from the first decoded data, and is rendered on the first layer. FIG. 7b is a schematic diagram of a second layer, in FIG. 7b, the update area A * Is the second decoded data, which is rendered at the second layer, which overlays the first layer.
For easy understanding, a complete example is provided below in conjunction with fig. 8, and the method of the present invention is specifically described when applied to a client corresponding to a cloud desktop:
and receiving the first coded data, the second coded data and the position information of the updating area sent by the server, and judging whether the first coded data and the second coded data are lossy coded data or not according to the coding type.
If the first encoded data is the lossy encoded data, decoding is performed by adopting a decoding mode corresponding to the lossy encoding mode, and the decoded data is extracted according to the position information of the updated region, namely, if the first encoded data is judged to be the lossy encoded data, decoding is performed by adopting the decoding mode corresponding to the lossy encoding mode, the decoded data is extracted according to the position information of the updated region, and then the updated region is copied to the RGB layer buffer area of the first layer, and is rendered and displayed in the first layer.
If the decoded data is not the lossy encoded data, decoding the lossy encoded data by adopting a decoding mode corresponding to the lossless encoding mode, copying the second decoded data obtained by decoding to an RGB layer buffer area of a second layer, and rendering and displaying the second decoded data in the second layer. It should be noted that the second layer is overlaid on the first layer.
A display device of a cloud desktop of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these means may be configured by the steps taught by the present solution using commercially available hardware components.
Fig. 9 is a schematic structural diagram of a display device of a cloud desktop, where the device is applied to a server running the cloud desktop, as shown in fig. 9, and the device includes: a determination module 11 and an execution module 12.
The determining module 11 is configured to determine an update area in screen content of the cloud desktop at a first moment.
An execution module 12, configured to execute, when it is determined that the area of the update area is greater than the set area threshold, the following encoding process: coding the screen content by adopting a lossy coding mode to obtain first coded data; transmitting the first encoded data and the position information of the update area to the client so that the client determines first decoding data corresponding to the position information according to the first encoded data; if the updating area does not update the content within the set time after the first time, encoding the content in the updating area at the first time by adopting a lossless encoding mode to obtain second encoded data; and sending the second encoded data to the client so that the client determines second decoded data corresponding to the second encoded data.
Optionally, the apparatus further comprises: and the encoding module is used for encoding the updated region by adopting a lossless encoding mode to obtain second encoded data if the area of the updated region is determined to be smaller than or equal to the set area threshold value.
Optionally, the screen content includes a plurality of update areas; the execution module 12 is specifically configured to: if it is determined that the total area of the plurality of update areas is greater than the set area threshold, an encoding process is performed for each update area.
Optionally, the apparatus further comprises: and updating the processing module.
The updating processing module is used for generating record information corresponding to the updating area, wherein the record information comprises the first moment, the position information of the updating area and the set lossy coding identifier; when the current timing of the timer reaches the second moment, acquiring record information; determining whether the time interval between the first time and the second time in the recorded information is longer than a set duration; if the content update time is longer than the set time length, determining that the content update does not occur in the update area between the first time and the second time.
Optionally, the update processing module is further configured to: after the second encoded data is transmitted to the client, the recorded information is deleted.
Optionally, the update processing module is further configured to: if the content in the update area is determined to be updated at a third time between the first time and the second time, the first time in the update record information is the third time; and if the time interval between the third time and the second time in the recorded information is not greater than the set duration, resetting the timer.
Optionally, the apparatus further comprises: and the duration determining module is used for determining the set duration according to the network quality between the server and the client.
Optionally, the apparatus further comprises: and the threshold determining module is used for determining an area threshold according to scene characteristics corresponding to the screen content.
The apparatus shown in fig. 9 may perform the steps performed by the service end in the foregoing embodiments, and the detailed performing process and technical effects are referred to the descriptions in the foregoing embodiments, which are not repeated herein.
In one possible design, the structure of the display device of the cloud desktop shown in fig. 9 may be implemented as an electronic device. As shown in fig. 10, the electronic device may include: a processor 21, a memory 22, a communication interface 23. The memory 22 stores executable code, which when executed by the processor 21, causes the processor 21 to at least implement the method for displaying a cloud desktop executed by the server as provided in the foregoing embodiment. The electronic device may be a virtual machine of the cloud.
Fig. 11 is a schematic structural diagram of a display device of a cloud desktop according to an embodiment of the present invention, where the device is applied to a client corresponding to the cloud desktop, as shown in fig. 11, and the device includes: a receiving module 31, a decoding module 32 and a display module 33.
A receiving module 31, configured to receive the first encoded data and the location information of the update area sent by the server; the first coding data are obtained by coding the screen content in a lossy coding mode when the server determines that the updating area in the screen content of the cloud desktop at the first moment is larger than the set area threshold.
The decoding module 32 is configured to decode the first encoded data to obtain first decoded data.
And a display module 33 for extracting the decoded data corresponding to the position information from the first decoded data and displaying the extracted decoded data.
The receiving module 31 is further configured to receive second encoded data sent by the server; the second coding data are obtained by coding the content in the update area at the first moment by adopting a lossless coding mode when the server determines that the update area does not update the content within a set time period after the first moment.
The display module 33 is further configured to decode the second decoded data obtained by the second encoded data, and display the second decoded data.
Optionally, the display module 33 is specifically configured to: rendering the extracted decoded data on a first layer to display the first layer; the second decoded data is rendered at the second layer to display the second layer on the first layer.
The apparatus shown in fig. 11 may perform the steps performed by the client in the foregoing embodiments, and the detailed execution process and technical effects are referred to the description in the foregoing embodiments, which are not repeated herein.
In one possible design, the structure of the display device of the cloud desktop shown in fig. 11 may be implemented as an electronic device. As shown in fig. 12, the electronic device may include: a processor 41, a memory 42, a communication interface 43. Wherein the memory 42 has stored thereon executable code which, when executed by the processor 41, causes the processor 41 to at least enable the display method of a cloud desktop executed by a client as provided in the foregoing embodiments. The electronic device may be a user terminal.
In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium having executable code stored thereon, which when executed by a processor of an electronic device, causes the processor to at least implement a method of displaying a cloud desktop as provided in the foregoing embodiments.
Fig. 13 is a flowchart of an image display method according to an embodiment of the present invention, as shown in fig. 13, where the method includes:
1301. An image update area in the video frame at the target time is determined.
1302. If the area of the image updating area is larger than the set area threshold, encoding the video picture by adopting a lossy encoding mode to obtain first encoded data, and sending the first encoded data and the position information of the image updating area to display equipment so that the display equipment determines and displays first decoding data corresponding to the position information according to the first encoded data.
1303. If the content update does not occur in the image update area within a set time after the target time, the content in the image update area is encoded by adopting a lossless encoding mode to obtain second encoded data, and the second encoded data is sent to the display device, so that the display device determines and displays second decoding data corresponding to the second encoded data.
It should be understood that in practical applications, the image display method may be applied to video playing scenes, such as live scenes, video on demand scenes, and the like, taking live scenes as an example, firstly, it is to determine an image update area in a live frame corresponding to a target moment, and then perform subsequent encoding, decoding and displaying operations based on the image update area, and for a specific execution process of a subsequent step, reference may be made to the related descriptions in the foregoing other embodiments, which are not repeated herein.
The apparatus embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by adding necessary general purpose hardware platforms, or may be implemented by a combination of hardware and software. Based on such understanding, the foregoing aspects, in essence and portions contributing to the art, may be embodied in the form of a computer program product, which may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (13)

1. The display method of the cloud desktop is characterized by being applied to a service end running the cloud desktop, and comprises the following steps:
determining an update area in screen content of a cloud desktop at a first moment;
if the area of the updated area is determined to be larger than the set area threshold, the following coding process is executed:
coding the screen content by adopting a lossy coding mode to obtain first coded data;
transmitting the first encoded data and the position information of the update area to a client so that the client determines first decoding data corresponding to the position information according to the first encoded data;
if the updating area does not update the content within the set time after the first time, encoding the content in the updating area at the first time by adopting a lossless encoding mode to obtain second encoded data;
and sending the second encoded data to the client so that the client determines second decoded data corresponding to the second encoded data.
2. The method according to claim 1, wherein the method further comprises:
and if the area of the updated area is smaller than or equal to the set area threshold, encoding the updated area by adopting a lossless encoding mode to obtain the second encoded data.
3. The method of claim 1, wherein the screen content includes a plurality of update areas therein;
and if the area of the updated area is determined to be larger than the set area threshold, executing the following coding process, wherein the coding process comprises the following steps:
and if the total area of the plurality of updated areas is determined to be larger than the set area threshold, executing the coding process for each updated area.
4. The method of claim 1, wherein after the transmitting the first encoded data and the location information of the update area to the client, further comprising:
generating record information corresponding to the update area, wherein the record information comprises the first moment, the position information of the update area and a set lossy coding identifier;
when the current timing of the timer reaches a second moment, acquiring the recording information;
determining whether the time interval between the first time and the second time in the recorded information is longer than a set duration;
if the content update time is longer than the set time length, determining that the content update does not occur in the update area between the first time and the second time.
5. The method according to claim 4, wherein the method further comprises:
And deleting the record information after the second encoded data is sent to the client.
6. The method according to claim 4, wherein the method further comprises:
if the content in the update area is determined to be updated at a third time between the first time and the second time, the first time in the recorded information is updated to be the third time;
the step of obtaining the record information at the second time when the current timing of the timer arrives, further includes:
and if the time interval between the third time and the second time in the recorded information is not greater than the set duration, resetting the timer.
7. The method according to any one of claims 1 to 6, further comprising:
and determining the set duration according to the network quality between the server and the client.
8. The method according to any one of claims 1 to 6, further comprising:
and determining the area threshold according to scene characteristics corresponding to the screen content.
9. The display method of the cloud desktop is characterized by being applied to a client corresponding to the cloud desktop, and comprises the following steps:
Receiving first coded data and position information of an update area sent by a server; the first coding data are obtained by coding the screen content in a lossy coding mode when the server determines that the updated area in the screen content of the cloud desktop at the first moment is larger than a set area threshold;
decoding the first encoded data to obtain first decoded data;
extracting decoding data corresponding to the position information from the first decoding data and displaying the extracted decoding data;
receiving second coded data sent by the server; the second coding data are obtained by coding the content in the update area at the first moment in a lossless coding mode when the server determines that the update area does not update the content within a set time period after the first moment;
and decoding the second coded data to obtain second decoded data, and displaying the second decoded data.
10. The method of claim 9, wherein displaying the extracted decoded data comprises:
rendering the extracted decoded data on a first layer to display the first layer;
The displaying the second decoded data includes:
rendering the second decoded data at a second layer to display the second layer on the first layer.
11. An electronic device, comprising: a memory, a processor, a communication interface; wherein the memory has stored thereon executable code which, when executed by the processor, causes the processor to perform the method of displaying a cloud desktop as claimed in any one of claims 1 to 8 or to perform the method of displaying a cloud desktop as claimed in claim 9 or 10.
12. A non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the method of displaying a cloud desktop as claimed in any one of claims 1 to 8, or to perform the method of displaying a cloud desktop as claimed in claim 9 or 10.
13. An image display method, comprising:
determining an image update area in a video picture at a target moment;
if the area of the image update area is determined to be larger than the set area threshold value, the following coding process is executed:
Coding the video picture by adopting a lossy coding mode to obtain first coded data;
transmitting the first encoded data and the position information of the image update area to a display device, so that the display device determines first decoding data corresponding to the position information according to the first encoded data and displays the first decoding data;
if the image updating area does not generate content updating any more within a set time after the target time, encoding the content in the image updating area by adopting a lossless encoding mode to obtain second encoded data;
and sending the second encoded data to the display device so that the display device determines and displays second decoded data corresponding to the second encoded data.
CN202310599820.6A 2023-05-23 2023-05-23 Cloud desktop display method, image display method, device and storage medium Pending CN116668741A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310599820.6A CN116668741A (en) 2023-05-23 2023-05-23 Cloud desktop display method, image display method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310599820.6A CN116668741A (en) 2023-05-23 2023-05-23 Cloud desktop display method, image display method, device and storage medium

Publications (1)

Publication Number Publication Date
CN116668741A true CN116668741A (en) 2023-08-29

Family

ID=87716458

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310599820.6A Pending CN116668741A (en) 2023-05-23 2023-05-23 Cloud desktop display method, image display method, device and storage medium

Country Status (1)

Country Link
CN (1) CN116668741A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117237545A (en) * 2023-11-13 2023-12-15 武汉泽塔云科技股份有限公司 Method, system and storage medium for carrying out 3D stereograph based on cloud desktop environment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117237545A (en) * 2023-11-13 2023-12-15 武汉泽塔云科技股份有限公司 Method, system and storage medium for carrying out 3D stereograph based on cloud desktop environment
CN117237545B (en) * 2023-11-13 2024-02-09 武汉泽塔云科技股份有限公司 Method, system and storage medium for carrying out 3D stereograph based on cloud desktop environment

Similar Documents

Publication Publication Date Title
JP2020526994A (en) Chroma prediction method and device
WO2018103568A1 (en) Methods of encoding and decoding cloud desktop content, device, and system
CN110784740A (en) Video processing method, device, server and readable storage medium
CN112954398B (en) Encoding method, decoding method, device, storage medium and electronic equipment
WO2020107972A1 (en) Video decoding control method and apparatus, electronic device, and storage medium
WO2021057697A1 (en) Video encoding and decoding methods and apparatuses, storage medium, and electronic device
KR101805550B1 (en) Image data encoding method for presentation virtualization and server therefor
CN116668741A (en) Cloud desktop display method, image display method, device and storage medium
CN108055595B (en) Video image redirection method and computer-readable storage medium
CN112422868A (en) Data processing method, terminal device and server
CN113794903A (en) Video image processing method and device and server
CN111031032A (en) Cloud video transcoding method and device, decoding method and device, and electronic device
CN111432213A (en) Adaptive tile data size coding for video and image compression
WO2021057686A1 (en) Video decoding method and apparatus, video encoding method and apparatus, storage medium and electronic device
US10536726B2 (en) Pixel patch collection for prediction in video coding system
CN109120979B (en) Video enhancement control method and device and electronic equipment
CN114938408B (en) Data transmission method, system, equipment and medium of cloud mobile phone
CN110572677A (en) video encoding and decoding method and device, storage medium and electronic device
WO2023024832A1 (en) Data processing method and apparatus, computer device and storage medium
CN110572672A (en) Video encoding and decoding method and device, storage medium and electronic device
CN115866297A (en) Video processing method, device, equipment and storage medium
KR20160082521A (en) Chroma down-conversion and up-conversion processing
CN110401835B (en) Image processing method and device
CN114079823A (en) Video rendering method, device, equipment and medium based on Flutter
CN115225615B (en) Illusion engine pixel streaming method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination