CN114237787A - Cloud desktop image transmission method and device - Google Patents
Cloud desktop image transmission method and device Download PDFInfo
- Publication number
- CN114237787A CN114237787A CN202111372689.7A CN202111372689A CN114237787A CN 114237787 A CN114237787 A CN 114237787A CN 202111372689 A CN202111372689 A CN 202111372689A CN 114237787 A CN114237787 A CN 114237787A
- Authority
- CN
- China
- Prior art keywords
- block
- level
- cache
- level block
- primary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a cloud desktop image transmission method and device. The method and the device are used for solving the technical problem that the transmission efficiency of the character cloud desktop image is low. The segmentation of the secondary blocks adopts horizontal block segmentation to distinguish character blocks and non-character blocks, so that the character interface characteristics are fully utilized, the bandwidth occupation of a character desktop scene in a cloud desktop transmission protocol is reduced, and the smoothness of the cloud desktop in a weak network environment is increased.
Description
Technical Field
The invention relates to the technical field of communication and cloud computing, in particular to a cloud desktop image transmission method and device.
Background
In actual deployment of a current Virtual Desktop Infrastructure (VDI) protocol, a cloud Desktop transmission protocol client (cloud Desktop client for short) is generally installed on a thin terminal, Desktop data interaction is performed between the cloud Desktop client and a cloud Desktop server through a remote Desktop protocol, a cloud Desktop server compresses and transmits a cloud Desktop image to the cloud Desktop client, and the cloud Desktop client is decompressed and then is presented to a user through a local display system.
The cloud desktop data transmission efficiency is always a key point of attention of developers and users, on one hand, the reason is that the bandwidth is a resource of the users, cost investment is involved, the occupied bandwidth is high when a single cloud desktop is connected, the total bandwidth is too high when all people use the cloud desktop at the same time, and the cost of network equipment investment and operator bandwidth lease when the users build a cloud desktop environment is influenced. Another reason is that the bandwidth occupation is high, which may affect the user experience in situations with poor link conditions (such as wan remote connections).
At present, in a VDI infrastructure, when a desktop of a virtual machine at a server is refreshed, an interface cache deduplication mode may be adopted to reduce the amount of transmitted data, and only an image of a desktop change area is transmitted to a cloud desktop client. However, in a scenario where the cloud desktop mainly displays a text interface, the server generally needs to adopt a lossless compression mode, and desktop data transmission efficiency of the cloud desktop is not high.
Disclosure of Invention
In view of this, the invention provides a method and an apparatus for transmitting a cloud desktop text image, and a storage medium, which are used for solving the technical problem of low transmission efficiency of text cloud desktop image data.
On the basis of one aspect of the embodiment of the invention, the method for transmitting the cloud desktop image is provided, and is applied to a cloud desktop server, and comprises the following steps:
judging whether the current desktop image to be transmitted is a character interface or not;
when the interface is judged to be a character interface, all primary blocks which intersect with the desktop refreshing area are obtained to form a primary refreshing block set; the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size;
respectively carrying out scanning segmentation of the secondary blocks on each primary block in the primary refreshing block set, and calculating the hash value of each secondary block;
calculating hash values of all primary blocks of the desktop image to be transmitted currently;
performing block cache identification replacement of the hit block of the first-level block on the first-level block in the first-level unrefreshed block set; performing block cache identification replacement of the block of the hit second-level block for each first-level block in the first-level refreshing block set;
and packaging and compressing the current desktop image data to be transmitted after the block cache identifier replacement processing is completed, and then sending the compressed desktop image data to the cloud desktop client.
Further, the method for judging whether the current desktop image to be transmitted is a text interface includes:
judging whether the desktop image is a character interface or not by comparing the change rate of the color change of the desktop image, judging the desktop image to be the character interface when the change rate is greater than a preset change rate threshold value, and otherwise judging the desktop image to be a non-character interface; or the like, or, alternatively,
and judging whether the input desktop image is a character interface or not by adopting the trained deep learning neural network model.
Further, the method for respectively performing scanning segmentation of the secondary block to each primary block in the primary refresh block set includes:
the image data of the first-level block is subjected to pixel-level line scanning, and an area containing characters and a blank area are divided into a plurality of second-level blocks.
Further, the method for calculating the hash value of the primary block comprises the following steps:
and combining the hash values of all the secondary blocks of the primary block, calculating the hash value of the combined data, and taking the hash value of the combined data as the hash value of the image data of the primary block.
Further, the method for performing block cache tag replacement on the block of the hit second-level block for each first-level block in the first-level refresh block set includes:
matching the hash value of the second-level block in a second-level block cache list in the local cache pool, and judging whether the second-level block is hit;
if the second-level block cache data structure in the second-level block cache list is hit, replacing the image data of the second-level block with the block cache identification of the second-level block;
if the second-level block cache data structure in the second-level block cache list is not hit, recording the cache data structure of the second-level block in a local cache pool;
the second-level block cache data structure includes, but is not limited to, a block start coordinate, a block length and width, a block cache id, and a hash value of the block.
Further, the method also comprises an aging cleaning step of the cache data structure of the first-level block and the second-level block, and the aging cleaning step comprises the following steps:
adding an aging time field for each first-level block and each second-level block for recording the cache duration;
refreshing the cache time field of the block when the block is hit each time;
and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
On the other hand, the invention also provides a cloud desktop image transmission device, which is applied to a cloud desktop server and comprises:
the character interface judging module is used for judging whether the current desktop image to be transmitted is a character interface or not;
the refreshing block acquisition module is used for acquiring all primary blocks which intersect with the desktop refreshing area to form a primary refreshing block set when the character interface is judged; the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size;
the secondary block segmentation module is used for respectively carrying out scanning segmentation on the secondary blocks for each primary block in the primary refresh block set and calculating the hash value of each secondary block;
the primary block hash module is used for calculating hash values of all primary blocks of the desktop image to be transmitted currently;
the block cache identifier replacing module is used for replacing the block cache identifier of the hit block of the first-level block in the first-level unrefreshed block set; performing block cache identification replacement of the block of the hit second-level block for each first-level block in the first-level refreshing block set;
and the compression transmission module is used for packaging and compressing the current desktop image data to be transmitted after the block cache identifier replacement processing is completed and then sending the compressed desktop image data to the cloud desktop client.
Further, the character interface judging module judges whether the desktop image is a character interface by comparing the change rate of the color change of the desktop image, and judges the desktop image to be a character interface when the change rate is greater than a preset change rate threshold value, otherwise judges the desktop image to be a non-character interface; or judging whether the input desktop image is a character interface or not by adopting a trained deep learning neural network model.
Further, the secondary block segmentation module performs pixel-level line scanning on the image data of the primary block, and segments an area containing characters and a blank area into a plurality of secondary blocks.
Further, the apparatus further comprises:
the aging processing module is used for adding an aging time field for each first-level block and each second-level block for recording the cache duration; refreshing the cache time field of the block when the block is hit each time; and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
On the other hand of the implementation of the present invention, there is also provided a cloud desktop image transmission apparatus, which is applied to a cloud desktop client, and includes:
the decompression module is used for receiving a compressed data packet of the character interface sent by the cloud desktop server and analyzing the compressed data packet;
the first-level block processing module is used for directly reading the image data of the corresponding first-level block from the local cache through the block cache identification carried in the data packet when the hit type identification of the block in the data packet indicates that the first-level block is hit; when the hit type identification of the block in the data packet indicates that the first-level block and the second-level block are not hit, reading the image data of the first-level block from the data packet, and caching the image data of the corresponding first-level block in a local cache pool;
the second-level block processing module is used for analyzing the data structure of the second-level block included in the data packet and sequentially reading the data structure of the second-level block when the hit type identifier of the block in the data packet indicates that the second-level block is hit; if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches and reads the image data of the corresponding second-level block in the local cache pool according to the block cache identification in the data structure of the second-level block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches the image data in the local cache pool;
and the block combination module is used for splicing the image data of all the primary blocks read from the compressed data packet into complete desktop image data and outputting the complete desktop image data to the display driver of the client.
The invention divides the text desktop image into two stages of blocks, judges the hit of the two stages of blocks of the desktop image to be transmitted, and replaces the image data with the block cache identification of the hit block. The segmentation of the secondary blocks adopts horizontal block segmentation to distinguish character blocks and non-character blocks, so that the character interface characteristics are fully utilized, the bandwidth occupation of a character desktop scene in a cloud desktop transmission protocol is reduced, and the smoothness of the cloud desktop in a weak network environment is increased.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings may be obtained according to the drawings of the embodiments of the present invention.
FIG. 1 is a schematic diagram illustrating a text interface and a desktop image block division of a cloud desktop display according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an exemplary two-level block partitioning based on a first-level block of a text interface according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating steps of a cloud desktop image transmission method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a server cloud desktop image transmission apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a client cloud desktop image transmission apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. As used in this embodiment of the invention, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
In an office scene, a cloud desktop may display a text interface most of the time, for example, when a user views an edited document or mail using software such as word, excel, notepad, outlook, PDF, or reads an article using a web page or reading software, most of the areas displayed in the desktop may be a row of text, a table, or the like.
In order to improve the cloud desktop data transmission efficiency, a cloud desktop block caching mechanism can be used, that is, a cloud desktop image is partitioned according to a preset size, for example, a desktop is partitioned according to the area size of pixel blocks such as 16 × 16, 64 × 64, 128 × 128 or 256 × 256, a cloud desktop server packs the partitioned cloud desktop image data and sends the cloud desktop image data to a cloud desktop client running on a terminal, the cloud desktop server and the client Cache the sent desktop block image data, if a certain cached block image data is hit subsequently, the server only sends a block caching identifier (Cache ID) for the block image data to the cloud desktop client, and the client directly obtains data from a local Cache through the block caching identifier to fill and display the data.
The inventor discovers through analysis that the cloud desktop image block data caching transmission mode improves the transmission efficiency of cloud desktop data to a certain extent, but for a text interface, the block data caching mode does not fully utilize the characteristics of the text interface, so that the hit probability of block image data is not very high, and the transmission efficiency is not high.
Fig. 1 is a schematic diagram of a text interface and a block division of a desktop image displayed on a cloud desktop according to an embodiment of the present invention, where a plurality of lines of text are displayed in the desktop image of the cloud desktop, a margin is left between lines between text lines, the entire desktop image is divided into N blocks, each block has a size L × W, L is a length, W is a width, and a unit may be a pixel. For example, a desktop image with a resolution of 256 × 256 pixels may be divided into blocks of N-16 pixels, L-64 pixels, and W-64 pixels. As can be seen from the figure, there are many margins between lines of text and around the document, and these margin areas are usually unchanged. If a large block in the graph is used for caching, the block data can be changed as long as any character symbol or line position in the block is changed, the server needs to compress and transmit the whole data to the cloud desktop client, in fact, many blank areas are unchanged, the percentage of the area with changed characters in the whole block area is possibly small, and bandwidth waste can be caused if the whole block data is compressed and transmitted in the past.
The invention provides a cloud desktop image transmission method, which relates to processing steps at two ends of a cloud desktop server and a cloud desktop client, and can realize the purpose of the invention by coordinating and coordinating the two ends mutually, wherein the processing action of one end determines the execution time and the execution action of the corresponding action of the other end, for example, the server transmits desktop image data to the client in a secondary interface image block cache mode, and the processing mode of the server determines that the client also receives and processes the cloud desktop image in a corresponding secondary interface image block cache mode.
The basic idea of the invention is to identify the interface type of the cloud desktop image and divide the text interface into two stages of blocks. Firstly, performing first-level block division on the whole desktop image, performing second-level block division inside the first-level block, wherein the second-level block division adopts a non-fixed transverse or longitudinal (determined according to a character typesetting format) block division mode which accords with character typesetting characteristics, the cloud desktop server and the client establish a two-level block cache list locally respectively, when the first-level block does not change and needs repeated transmission, directly transmitting a block cache identifier of the first-level block to the cloud desktop client, when the first-level block cache is not hit, continuing judgment of the second-level block, when the second-level block cache is hit, transmitting the block cache identifier of the second-level block to the cloud desktop client, and when the cloud desktop client receives the block cache identifier of the first-level block or the second-level block, directly reading a cache block corresponding to the block cache identifier from the local for display.
Fig. 2 is an exemplary diagram of performing two-level block division on the basis of a first-level block of a text interface according to an embodiment of the present invention. The size of the secondary blocks may be not fixed, for example, in fig. 2, the ith primary block is divided into 4 secondary blocks according to the positions of the text and blank areas.
Fig. 3 is a schematic flow chart illustrating steps of a cloud desktop image transmission method according to an embodiment of the present invention, where the schematic chart illustrates a specific implementation process of a server side for transmitting a cloud desktop image, and the implementation process includes:
in a cloud desktop infrastructure, a server virtual graphics card driver captures cloud desktop images, the cloud desktop images are processed and sent frame by frame, and the cloud desktop server may select different transmission modes according to scene configuration, for example, streaming mode transmission is adopted in an extreme speed mode, and a micro-frame algorithm is used for processing. In a high-definition mode, a series of processing processes such as duplicate removal, text interface separation and compression are adopted and then the processing processes are sent to the cloud desktop client. Under the high-definition mode, the desktop images can be divided into text interface images and non-text interface images for processing respectively, and different algorithms are adopted for processing so as to improve the transmission efficiency. And finally, after the processed desktop image is packaged, compressing the desktop image by adopting a preset compression algorithm and then sending the desktop image to the client. The process of removing the duplication refers to removing the same part of the previous and next frame images, the refreshing area of the desktop can be known through the duplication removing step, the duplicated part is not sent, and therefore the sending data volume can be reduced.
In an embodiment of the present invention, whether a desktop image is a text interface is determined by comparing the change rate of the color change of the desktop image, the color change rate of a natural image is usually mild, and the change rate of the image of the text interface is usually steep, so that whether the desktop image is the text interface is determined by setting a threshold of the change rate, and when the change rate is greater than a preset threshold of the change rate, the desktop image is determined as the text interface, otherwise, the desktop image is determined as a non-text interface. For example, the rate of change is determined by calculating the hash value of all pixel sums for each pixel row and then comparing the difference in hash values between rows.
In another embodiment of the invention, a deep learning neural network model is adopted to more accurately judge whether the current desktop image is a text interface, for example, Vgg, googlenet and inclusion v4 neural network algorithm models, the neural network model is trained through a sample to obtain a finally applicable neural network model, the desktop image is input into the trained neural network model, and the neural network model can output a judgment result.
the purpose of the step is to determine which first-level blocks are involved in the desktop refresh area, and for the unrefreshed first-level blocks, the first-level blocks can hit the cache data structure of the first-level block image data in the local cache pool, and the image data can be directly replaced by the block cache identification of the first-level blocks; and the hit judgment and processing of the secondary block need to be continued for the refreshed primary block.
When a text interface is transmitted for the first time, the whole desktop is a refresh area, and the server and the client can establish all first-level block cache data structures locally. After the character interface is refreshed, the cloud desktop server can obtain the position and the size of a desktop refreshing area through the virtual display card driving interface, and can obtain all primary blocks which have intersection with the desktop refreshing area based on the distribution position of each primary block on a screen.
In the present invention, the primary blocks may be divided into fixed-size blocks, that is, the size of each primary block is consistent and the position is fixed, as shown in fig. 1. Of course, according to the typesetting characteristics of the actual text type interfaces of different languages, a division mode of a first-level block with a non-fixed size can be adopted as long as the cloud desktop server and the client are synchronous and consistent, and the invention is not particularly limited.
And 303, respectively carrying out scanning segmentation on the secondary blocks of each primary block in the primary refresh block set, and calculating the hash value of each secondary block.
In an embodiment of the present invention, the method for performing scan segmentation on the first-level block includes:
step 3031, performing pixel-level line scanning on the image data of the primary block, and dividing a region containing characters and a blank region into a plurality of secondary blocks;
referring to the example of fig. 2, the ith primary block may be divided into 4 secondary blocks by pixel line scanning, and the specific division may be: if the current line comprises non-white pixel points (assuming the color of a white area between character lines is white), the line is positioned in the character area, and the downward scanning is continued until the whole line is scanned to the white line or the last line of the primary block, and a plurality of continuous lines positioned in the character area in front are divided into a group to form a secondary block. In the same way, if the line (blank area) with white whole line is scanned, which indicates that the line is located in the blank area, the downward scanning is continued until the line including non-white pixel points in the whole line or the last line of the primary block is scanned, and the previous multiple continuous lines located in the blank area are divided into a group to form a secondary block.
In this way, the primary block can be divided into a plurality of secondary blocks. Considering that the background of some text interface documents is not necessarily pure white, the desktop image can be processed before the secondary block is divided, the desktop image is converted into a black-white image (with emphasis on text and white), and then the secondary block is divided.
Step 3032, calculating the hash value of each secondary block, generating a secondary block data structure of each secondary block, and forming a secondary block list of the current primary block.
The second-level block data structure can comprise the initial coordinate of the second-level block, the length and the width of the block, the hash value of the block, the block cache identification of the block and the like, a second-level block list of the current first-level block is formed by the second-level block data structure, and the second-level block list can be temporarily stored in the memory; after the second-level block data structure and the second-level block list are cached in the local cache pool, the second-level block cache data structure and the second-level block cache list are called.
the step calculates all the first-level blocks of the desktop image to be transmitted currently, including the hash values of the first-level refreshing block and the first-level non-refreshing block. In an embodiment of the present invention, the method for calculating the hash value of the first-level block in the first-level refresh block set may have the following two ways:
in the first mode, the hash value of the image data of the primary block is directly calculated to obtain the hash value of the primary block;
and secondly, combining the hash values of all the secondary blocks of the primary block, calculating the hash value of the combined data, and taking the hash value of the combined data as the hash value of the image data of the primary block.
And 305, performing block cache identification replacement of the block of the hit secondary block for each primary block in the primary refresh block set.
The first-level blocks in the first-level refresh block set are all the first-level blocks which are changed relative to the desktop image transmitted before, so that the first-level blocks in the first-level refresh block set do not hit a first-level block cache data structure in a local cache pool, and block cache compression transmission of the second-level blocks is required to be performed on the first-level blocks. The basic idea of block cache compression transmission of the second-level block is as follows: and replacing the image data of the block by the block cache identification of the second-level block when the second-level block of the second-level block cache data structure in the local cache pool is hit.
In this embodiment, the local cache pool is used for caching the first-level block cache data structure and the second-level block cache data structure which have been successfully transmitted before, and both the first-level block cache data structure and the second-level block cache data structure can be managed by a list. For example, all the first-level block cache data structures form a first-level block cache list, and each first-level block cache data structure is connected to a second-level block cache list formed by all the second-level block cache data structures belonging to the first-level block through a pointer. A secondary chunk cache data structure may include, but is not limited to, chunk start coordinates, chunk length, chunk cache identification (or called chunk number), and hash value of the chunk. The block cache identifier of the secondary block may be a hash value of the block, or a unique identifier generated by the server and the client according to a preset rule, and is used to uniquely identify a block at both ends. The present invention does not specifically limit the organization and management of the first-level and second-level block cache data structures.
In this step, the processing procedures 3051-3053 described below need to be called circularly, and all secondary blocks in the primary block are processed one by one until all secondary blocks in the current primary block are processed.
Step 3051, matching the hash value of the second-level block in a second-level block cache list in the local cache pool, and judging whether the second-level block is hit;
step 3052, if the second-level block cache data structure in the second-level block cache list is hit, replacing the image data of the block with the block cache identification of the second-level block.
The image data corresponding to the first-level block and the cache data structure corresponding to the second-level block are cached in the first-level block and the second-level block in the local cache pool at the cloud desktop client, and when the cloud desktop client receives a block cache identifier of the first-level block and the second-level block sent by the server, the corresponding block image data can be read from the local cache for display.
Step 3053, if the second-level block cache data structure in the second-level block cache list is not hit, recording the cache data structure of the second-level block in the local cache pool;
in the case of a miss of the second-level block, the cache data structure of the second-level block needs to be recorded in the second-level block cache list of the current first-level block in the local cache pool.
the block cache compression transmission process of the first-level block is similar to the block cache compression transmission process of the second-level block in principle. And for the first-level block hitting the first-level block cache data structure in the local cache pool, identifying the block image data by the block cache of the first-level block.
And 307, packaging and compressing the image data of the current desktop image with transmission and then sending the image data to the cloud desktop client.
In the embodiment of the invention, the cloud desktop server can perform packing compression on image data to be transmitted by taking a desktop image frame as a unit, the packed data of the desktop image to be transmitted comprises a block cache identifier of a hit block, image data of a miss block and the like, and the packed data packet organizes the image data to be transmitted in a certain data structure. In the packed packet data structure, a hit type identifier indicating whether a secondary block is hit or not may be carried for each block.
In the embodiment of the present invention, although the description is made in a sequential step manner, the order of performing the block cache compression transmission on the first-level refreshed block and the first-level non-refreshed block is not limited, and the refreshed block and the non-refreshed block may be sequentially arranged or may be arranged at intervals, for example, the first-level block in the dashed box in fig. 1 is a refreshed block, the non-dashed box is a non-refreshed block, the service end may process the image data transmission of one first-level non-refreshed block after processing the image data transmission of one first-level refreshed block, and the service end is determined according to the arrangement relationship of the blocks.
In an embodiment of the present invention, to avoid excessive consumption of the local cache and timely release of the data of the cache block that has failed after being refreshed, the method further includes an aging cleaning step of the cache data structure of the first-level block and the second-level block, including:
(1) adding an aging time field for each first-level block and each second-level block for recording the cache duration;
(2) refreshing the cache time field of the block when the block is hit each time;
(3) and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
In an embodiment of the present invention, a hit count field may also be added to the cache data structure of the first-level block and the second-level block, the hit count is added after each hit, and when aging is performed, the cache data of the block with a low hit rate is preferentially cleaned.
In the embodiment of the invention, a cloud desktop client divides a desktop image into blocks by adopting a primary block division method which is the same as that of a server, and when the client receives a compressed data packet of a character interface sent by the cloud desktop server, the data packet is decompressed and the following processing is carried out:
if the hit type identification of the block in the compressed data packet indicates that the first-level block is hit, the client directly reads the image data of the corresponding first-level block from the local cache through the block cache identification carried in the compressed data packet;
if the hit type identification of the block in the compressed data packet indicates that the first-level block and the second-level block are not hit, the client reads the image data of the first-level block from the compressed packet and caches the image data of the corresponding first-level block in a local cache pool;
if the hit type identification of the block in the compressed data packet indicates that the second-level block is hit, the client further analyzes the data structure of the second-level block included in the data packet and sequentially reads the data structure of the second-level block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches the cache data structure of the second-level block in the local cache pool according to the block cache identification in the data structure of the second-level block, and then reads the image data of the corresponding second-level block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches the image data in the local cache pool (or covers the image data of the corresponding area in the current first-level block).
And after reading the image data of all the primary blocks from the compressed data packet, the client side assembles complete desktop image data and outputs the desktop image data to a display driver of the client side, so that the desktop image is displayed.
Fig. 4 is a schematic structural diagram of a server cloud desktop image transmission apparatus according to an embodiment of the present invention, and each functional module in the apparatus 400 may be implemented by software, hardware, or a combination of software and hardware. The apparatus 400 comprises:
the text interface judging module 401 is configured to judge whether a current desktop image to be transmitted is a text interface;
a refresh block acquisition module 402, configured to acquire all primary blocks that intersect with the desktop refresh area to form a primary refresh block set when the interface is determined as a text interface; the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size;
a secondary block segmentation module 403, configured to perform scanning segmentation on a secondary block for each primary block in the primary refresh block set, and calculate a hash value of each secondary block;
a primary block hash module 404, configured to calculate hash values of all primary blocks of the current desktop image to be transmitted;
a block cache identifier replacement module 405, configured to perform block cache identifier replacement on a block of a hit first-level block of a first-level unrefreshed block set; performing block cache identification replacement of the block of the hit second-level block for each first-level block in the first-level refreshing block set;
a compression transmission module 406, configured to pack and compress the current desktop image data to be transmitted after the block cache identifier replacement processing is completed, and send the compressed desktop image data to the cloud desktop client
Further, the text interface determining module 401 determines whether the desktop image is a text interface by comparing the picture change rate, determines that the desktop image is a text interface when the change rate is greater than a preset change rate threshold value, and determines that the desktop image is a non-text interface otherwise; or judging whether the input desktop image is a character interface or not by adopting a trained deep learning neural network model.
Further, the secondary partition module 403 performs pixel-level line scanning on the image data of the primary partition, and partitions an area containing characters and a blank area into a plurality of secondary partitions.
Further, the apparatus 400 further includes an aging processing module for adding an aging time field for each of the first-level block and the second-level block for recording the duration of the cache; refreshing the cache time field of the block when the block is hit each time; and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
Fig. 5 is a schematic structural diagram of a client cloud desktop image transmission apparatus according to an embodiment of the present invention, and each functional module in the apparatus 500 may be implemented by software, hardware, or a combination of software and hardware. The apparatus 500 comprises:
the decompression module 501 is configured to receive a compressed data packet of a text interface sent by a cloud desktop server, and parse the compressed data packet;
a first-level block processing module 502, configured to, when the hit type identifier of the block in the data packet indicates that the first-level block is hit, directly read image data of the corresponding first-level block from the local cache through the block cache identifier carried in the data packet; when the hit type identification of the block in the data packet indicates that the first-level block and the second-level block are not hit, reading the image data of the first-level block from the data packet, and caching the image data of the corresponding first-level block in a local cache pool;
a secondary block processing module 503, configured to, when the hit type identifier of the block in the data packet indicates that the secondary block is hit, analyze the data structure of the secondary block included in the data packet, and sequentially read the data structure of the secondary block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches and reads the image data of the corresponding second-level block in the local cache pool according to the block cache identification in the data structure of the second-level block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches the image data in the local cache pool.
And the block combination module 504 is configured to assemble the image data of all the primary blocks read from the compressed data packet into complete desktop image data, and output the complete desktop image data to a display driver of a client.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, where the electronic device 600 includes: a processor 610 such as a Central Processing Unit (CPU), a communication bus 620, a communication interface 640, and a storage medium 630. Wherein the processor 610 and the storage medium 630 may communicate with each other through a communication bus 620. The storage medium 630 stores a computer program for implementing the cloud desktop image transmission method provided by the present invention, or the storage medium 630 includes a software function module capable of implementing the functions of the apparatus 400 or the apparatus 500, and when the computer program is executed by the processor 610 or the software function module is loaded and run, the functions of the steps of the method provided by the present invention can be implemented.
The storage medium may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. In addition, the storage medium may be at least one memory device located remotely from the processor. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Further, operations of processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A cloud desktop image transmission method is applied to a cloud desktop server and comprises the following steps:
judging whether the current desktop image to be transmitted is a character interface or not;
when the interface is judged to be a character interface, all primary blocks which intersect with the desktop refreshing area are obtained to form a primary refreshing block set; the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size;
respectively carrying out scanning segmentation of the secondary blocks on each primary block in the primary refreshing block set, and calculating the hash value of each secondary block;
calculating hash values of all primary blocks of the desktop image to be transmitted currently;
performing block cache identification replacement of the hit block of the first-level block on the first-level block in the first-level unrefreshed block set; performing block cache identification replacement of the block of the hit second-level block for each first-level block in the first-level refreshing block set;
and packaging and compressing the current desktop image data to be transmitted after the block cache identifier replacement processing is completed, and then sending the compressed desktop image data to the cloud desktop client.
2. The method according to claim 1, wherein the method for determining whether the desktop image to be transmitted currently is a text interface comprises:
judging whether the desktop image is a character interface or not by comparing the change rate of the color change of the desktop image, judging the desktop image to be the character interface when the change rate is greater than a preset change rate threshold value, and otherwise judging the desktop image to be a non-character interface; or the like, or, alternatively,
and judging whether the input desktop image is a character interface or not by adopting the trained deep learning neural network model.
3. The method of claim 1, wherein the scan-slicing of the secondary blocks for each primary block in the set of primary refresh blocks is performed by:
the image data of the first-level block is subjected to pixel-level line scanning, and an area containing characters and a blank area are divided into a plurality of second-level blocks.
4. The method of claim 1, wherein the step of computing the hash value of the primary block comprises:
and combining the hash values of all the secondary blocks of the primary block, calculating the hash value of the combined data, and taking the hash value of the combined data as the hash value of the image data of the primary block.
5. The method of claim 1, wherein for each primary refresh block in the set of primary refresh blocks, the block cache identification replacement of the block of the hit secondary block is performed by:
matching the hash value of the second-level block in a second-level block cache list in the local cache pool, and judging whether the second-level block is hit;
if the second-level block cache data structure in the second-level block cache list is hit, replacing the image data of the second-level block with the block cache identification of the second-level block;
if the second-level block cache data structure in the second-level block cache list is not hit, recording the cache data structure of the second-level block in a local cache pool;
the second-level block cache data structure includes, but is not limited to, a block start coordinate, a block length and width, a block cache id, and a hash value of the block.
6. The method of claim 1, further comprising the step of performing an aging clean of the cache data structure including the level one block and the level two block, comprising:
adding an aging time field for each first-level block and each second-level block for recording the cache duration;
refreshing the cache time field of the block when the block is hit each time;
and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
7. The utility model provides a cloud desktop image transmission device, its characterized in that, the device is applied to cloud desktop server, includes:
the character interface judging module is used for judging whether the current desktop image to be transmitted is a character interface or not;
the refreshing block acquisition module is used for acquiring all primary blocks which intersect with the desktop refreshing area to form a primary refreshing block set when the character interface is judged; the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size;
the secondary block segmentation module is used for respectively carrying out scanning segmentation on the secondary blocks for each primary block in the primary refresh block set and calculating the hash value of each secondary block;
the primary block hash module is used for calculating hash values of all primary blocks of the desktop image to be transmitted currently;
the block cache identifier replacing module is used for replacing the block cache identifier of the hit block of the first-level block in the first-level unrefreshed block set; performing block cache identification replacement of the block of the hit second-level block for each first-level block in the first-level refreshing block set;
and the compression transmission module is used for packaging and compressing the current desktop image data to be transmitted after the block cache identifier replacement processing is completed and then sending the compressed desktop image data to the cloud desktop client.
8. The apparatus of claim 7,
the character interface judging module judges whether the desktop image is a character interface or not by comparing the change rate of the color change of the desktop image, and judges the desktop image to be a character interface when the change rate is greater than a preset change rate threshold value, otherwise, judges the desktop image to be a non-character interface; or judging whether the input desktop image is a character interface or not by adopting a trained deep learning neural network model.
9. The apparatus of claim 7,
the secondary block segmentation module performs pixel-level line scanning on the image data of the primary block, and segments a region containing characters and a blank region into a plurality of secondary blocks.
10. The apparatus of claim 7, further comprising:
the aging processing module is used for adding an aging time field for each first-level block and each second-level block for recording the cache duration; refreshing the cache time field of the block when the block is hit each time; and periodically judging whether the cache duration of each block exceeds a preset threshold through the aging cleaning timer, and if so, releasing the cache space of the corresponding block.
11. The cloud desktop image transmission device is applied to a cloud desktop client and comprises the following components:
the decompression module is used for receiving a compressed data packet of the character interface sent by the cloud desktop server and analyzing the compressed data packet;
the first-level block processing module is used for directly reading the image data of the corresponding first-level block from the local cache through the block cache identification carried in the data packet when the hit type identification of the block in the data packet indicates that the first-level block is hit; when the hit type identification of the block in the data packet indicates that the first-level block and the second-level block are not hit, reading the image data of the first-level block from the data packet, and caching the image data of the corresponding first-level block in a local cache pool;
the second-level block processing module is used for analyzing the data structure of the second-level block included in the data packet and sequentially reading the data structure of the second-level block when the hit type identifier of the block in the data packet indicates that the second-level block is hit; if the hit type field in the data structure of the second-level block indicates that the current second-level block is hit, the client directly searches and reads the image data of the corresponding second-level block in the local cache pool according to the block cache identification in the data structure of the second-level block; if the hit type field in the data structure of the second-level block indicates that the current second-level block is not hit, the client reads the image data of the second-level block from the data structure of the current second-level block and caches the image data in the local cache pool;
and the block combination module is used for splicing the image data of all the primary blocks read from the compressed data packet into complete desktop image data and outputting the complete desktop image data to the display driver of the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111372689.7A CN114237787A (en) | 2021-11-18 | 2021-11-18 | Cloud desktop image transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111372689.7A CN114237787A (en) | 2021-11-18 | 2021-11-18 | Cloud desktop image transmission method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114237787A true CN114237787A (en) | 2022-03-25 |
Family
ID=80750040
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111372689.7A Pending CN114237787A (en) | 2021-11-18 | 2021-11-18 | Cloud desktop image transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114237787A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114422807A (en) * | 2022-03-28 | 2022-04-29 | 麒麟软件有限公司 | Transmission optimization method based on Spice protocol |
| CN115278376A (en) * | 2022-05-25 | 2022-11-01 | 西安万像电子科技有限公司 | Audio and video data transmission method and device |
| CN117640627A (en) * | 2024-01-25 | 2024-03-01 | 微网优联科技(成都)有限公司 | Collaborative work method and system realized through cloud computer |
-
2021
- 2021-11-18 CN CN202111372689.7A patent/CN114237787A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114422807A (en) * | 2022-03-28 | 2022-04-29 | 麒麟软件有限公司 | Transmission optimization method based on Spice protocol |
| CN114422807B (en) * | 2022-03-28 | 2022-10-21 | 麒麟软件有限公司 | Transmission optimization method based on Spice protocol |
| CN115278376A (en) * | 2022-05-25 | 2022-11-01 | 西安万像电子科技有限公司 | Audio and video data transmission method and device |
| CN115278376B (en) * | 2022-05-25 | 2024-03-22 | 西安万像电子科技有限公司 | Audio and video data transmission method and device |
| CN117640627A (en) * | 2024-01-25 | 2024-03-01 | 微网优联科技(成都)有限公司 | Collaborative work method and system realized through cloud computer |
| CN117640627B (en) * | 2024-01-25 | 2024-04-09 | 微网优联科技(成都)有限公司 | Collaborative work method and system realized through cloud computer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114237787A (en) | Cloud desktop image transmission method and device | |
| CN101438319B (en) | Advanced anti-aliasing with multiple graphics processing units | |
| CN102970456B (en) | Image forming apparatus, image forming apparatus control method, and program | |
| CN108579094B (en) | User interface detection method, related device, system and storage medium | |
| CN110235176B (en) | Image processing method and device, data transmission method and device and storage medium | |
| EP3709665A1 (en) | Method for generating video mask information, bullet screen anti-occlusion method, server and client | |
| CN112016545A (en) | Image generation method and device containing text | |
| CN102750689A (en) | Image processing apparatus and control method thereof | |
| CN114429637B (en) | Document classification method, device, equipment and storage medium | |
| EP0949579A2 (en) | Multiple size reductions for image segmentation | |
| CN111061896B (en) | Loading method, device, equipment and medium for 3D (three-dimensional) graph based on glTF (generalized likelihood TF) | |
| CN115588192A (en) | Cell image classification method, image classification system, electronic device, and storage medium | |
| CN114612414A (en) | Image processing method, model training method, device, equipment and storage medium | |
| CN114429640A (en) | Drawing segmentation method and device and electronic equipment | |
| CN106920270B (en) | Animation processing method and device | |
| US8947717B2 (en) | Print data processing apparatus | |
| CN112487871B (en) | Handwriting data processing method and device and electronic equipment | |
| CN112486337B (en) | Handwriting graph analysis method and device and electronic equipment | |
| US20010015820A1 (en) | Method and apparatus for conservation of memory in a printer through selective compression of a display list | |
| CN112565766A (en) | Video transmission method, device and storage medium | |
| CN117014618A (en) | Image compression-based blocking method and system and electronic equipment | |
| CN101901341B (en) | Method and equipment for extracting raster image from transportable electronic document | |
| CN104182396B (en) | Terminal, format document content description optimization apparatus and method | |
| CN113645484B (en) | Data visualization accelerated rendering method based on graphic processor | |
| CN112487897B (en) | Handwriting content evaluation method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |