CN110868599B

CN110868599B - Video compression method of remote desktop

Info

Publication number: CN110868599B
Application number: CN201911242285.9A
Authority: CN
Inventors: 周小波
Original assignee: HANGZHOU SHUNWANG TECHNOLOGY CO LTD
Current assignee: HANGZHOU SHUNWANG TECHNOLOGY CO LTD
Priority date: 2019-12-06
Filing date: 2019-12-06
Publication date: 2021-11-19
Anticipated expiration: 2039-12-06
Also published as: CN110868599A

Abstract

The invention discloses a video compression method of a remote desktop, which comprises the following steps: s1, a coding end firstly blocks current desktop image data, then selects an interframe coding reference frame object, performs maximum region matching search and adjusts background frame content by using the reference frame object, then performs lossless data compression on region matching information and unmatched block image information, and finally transmits a compressed content packet; s2, decompressing by a decoding end according to the received packets, respectively recovering corresponding image information and adjusting the content of a background frame according to the maximum region matching information or unmatched block image information, combining the image blocks and displaying the desktop image. The invention can improve the prior remote desktop image coding technology, can effectively compress image data in various scenes such as software office or video playing, reduces the bandwidth occupation and improves the fluency of remote control.

Description

Video compression method of remote desktop

Technical Field

The invention belongs to the technical field of video compression, and particularly relates to a video compression method of a remote desktop.

Background

The remote desktop technology, which makes it possible to connect and control another device having a remote desktop connection function enabled through a network, includes not only display of a remote desktop image but also transmission of remote sound and mapping of pluggable hardware, etc., and thus may provide a desktop or application experience with rich fidelity. For client devices such as computers or mobile devices in enterprise offices, business halls, hospitals and the like, the client devices can be connected to a data center with a large number of servers through a local area network or a wide area network, the servers of the data center start a remote desktop connection function, and after a remote desktop system is built with the client devices, the client devices can operate the server devices in real time through the remote desktop technology, so that resources of the data center can be used more conveniently and flexibly.

After a traditional remote desktop system is built, the server side equipment responds to the operation of a user on the client side equipment, unified compression and cache processing are carried out on desktop display images, a graphic instruction is formed through packaging according to a remote desktop protocol and then sent to the client side equipment, the client side equipment decodes the received graphic instruction and renders image data into a complete desktop display image, and therefore a complete desktop image is displayed on the client side equipment.

Since the types of desktop display images are various, in the prior art, various types of image data are uniformly processed and compressed by using the same compression algorithm, which inevitably results in that the overhead cannot be minimized or the efficiency cannot be optimized for some type or types of desktop display images. In addition, in daily use of the remote desktop, the maximum bandwidth occupied by image data transmission is increased, and with the popularization of high-definition displays, image encoding data is also increased geometrically, so that the image encoding technology for the remote desktop becomes a core technology in the whole remote system.

The optimization techniques employed for image transmission in existing remote systems are generally classified into two categories:

(1) by using the characteristics of the operating system, the sent image data is the drawing instruction or control vector description information of the system. The technology depends on the realization of a bottom operating system, so the universality is not strong, for example, the controlled end is only supported by windows in the prior art, such as the RDP of the windows; meanwhile, for image information which cannot be described by simple drawing instructions or vector information, such as pictures and videos in a webpage, the data volume transmitted by the method is very large without a video compression algorithm, so that the application is usually only suitable for broadband application scenes such as an intranet.

(2) The images are compressed using common video compression algorithms such as h.264, h.265, etc. These algorithms usually divide the image data into blocks of a specific size, and then find out the interframe motion vectors of the frames to be compressed and the key frame I frame or the forward frame P frame, but the desktop definition is usually higher, so the data recording the block motion vectors is also larger; meanwhile, the I frame is usually determined by a fixed time sequence, or the frame is set as the I frame after the fixed frame is separated, and the background change of the remote desktop cannot be determined by time, which all result in poor image compression rate effect when the browsed web page is scrolled up and down and the window is minimized to the maximum.

Disclosure of Invention

In view of the above, the present invention provides a video compression method for a remote desktop, which can improve the existing remote desktop image coding technology, can effectively compress image data in various scenes such as software office or video playing, reduce bandwidth occupation, and improve remote control fluency.

A video compression method of a remote desktop comprises the following steps:

(1) partitioning a video image in the current desktop by a coding end;

(2) matching the current frame with a forward frame and a background frame, and taking the frame with the maximum pixel matching number with the current frame as a reference frame;

(3) matching search is carried out on the current frame and the reference frame by taking a block as a unit so as to determine a matching area and a matching block in the current frame, and further position information, motion vector and residual error information of the matching area and the matching block, type information of the reference frame and unmatched block information in the current frame are sequentially stored in a lossless compression queue;

(4) performing data compression on the lossless compression queue, and sending packets to a remote decoding end;

(5) and the decoding end decompresses the data packet after receiving the data packet and restores the image information of the current frame according to the decompressed data.

Further, in the step (1), the video image is divided into a plurality of n × n tiles, where n is a natural number greater than 1, and if the size of the entire image cannot be divided by n, the image edge needs to be extended to a default color to adapt the image size to the tiles.

Further, the background frame is a background image formed by combining images with high probability in a certain time period of blocking at a certain position at the beginning of remote control.

Further, in the step (3), the current frame is matched and searched with the reference frame by taking the block as a unit, and if any image block in the current frame is consistent with the corresponding block of the reference frame in pixel values exceeding a certain proportion, the image block is judged to be a matching block; if the current image block is a matching block, whether the image block on the right side is the matching block is continuously judged according to the above steps until the row direction is finished or a non-matching block is searched; and simultaneously judging whether the image block below the current image block is a matching block or not until the column direction is finished or a non-matching block is searched, so as to determine a matching area in the current frame.

Further, in the step (3), matching search is performed in units of blocks, the motion vectors are in units of pixels, the image blocks subjected to matching search need to be marked, and all residual error information should be stored in sequence after the motion vectors of all the pixels are stored in sequence in the storage process.

Further, in the step (3), for the remaining unmatched blocks in the current frame, DCT transformation is performed on the unmatched blocks first, and then the transformed data are sequentially stored in a lossless compression queue.

Further, after the step (3) is completed, updating the background frame of the encoding end, specifically including the following steps:

comparing each image block in the current frame with image blocks at corresponding positions in the forward frame and the background frame in sequence, and if any image block in the current frame is consistent with the image blocks at the corresponding positions in the forward frame or the background frame in pixel values exceeding a certain proportion, judging the two image blocks to be similar blocks;

for any image block in the background frame, if the image block and the image block at the corresponding position in the current frame are similar blocks, the weight of the image block +1 is increased, and the increased weight does not exceed a specified upper limit;

for any image block in the background frame, if the image block at the corresponding position in the current frame is not similar to the image block at the corresponding position in the forward frame, the weight of the image block is made to be-1, and if the weight is already 0, the weight is not reduced;

for any image block in the background frame, if the current weight of the image block is 0 and the image block at the corresponding position in the forward frame and the image block at the corresponding position in the current frame are similar blocks, the weight of the image block is not changed;

for any image block in the background frame, if the current weight of the image block is 0 and the image block at the corresponding position in the forward frame is not similar to the image block at the corresponding position in the current frame, replacing the image block in the background frame by the image block at the corresponding position in the forward frame, and resetting the weight of the image block as a default initial value.

Further, in the step (4), an LZ77 dictionary compression algorithm and an entropy coder (such as a markov prediction chain, a huffman tree, etc.) are used in combination to perform data compression on the lossless compression queue, and the packet is sent to a remote decoding end.

Further, the specific implementation process of the step (5) is as follows:

5.1 the decoding end decompresses the data packet after receiving the data packet to obtain the position information, the motion vector and the residual information of the matching area and the matching block in the current frame, the type information of the reference frame and the unmatched block information in the current frame;

5.2 for the matching area and the matching block, restoring the image information of the matching block in the current frame by reading the motion vector and the residual error information of the matching area and the matching block;

5.3 for the unmatched block, performing IDCT transformation by reading the data information of the unmatched block to recover the image information of the unmatched block in the current frame;

and 5.4, integrating all the image blocks, namely restoring to obtain the current frame image, and further updating the background frame of the decoding end.

The video compression method improves the prior remote desktop image coding technology, can effectively compress image data in various scenes such as software office or video playing, reduces the bandwidth occupation and improves the fluency of remote control.

Drawings

Fig. 1 is a schematic diagram illustrating steps of a video compression method of a remote desktop according to the present invention.

Fig. 2 is a schematic diagram of a frame of an encoding end.

Fig. 3 is a block diagram of a decoding end.

Fig. 4 is a schematic diagram of a process of matching a current frame with a reference frame at an encoding end.

Fig. 5 is a flowchart illustrating a process of searching for a matching area at the encoding end.

Fig. 6 is a schematic diagram of a background frame block weight updating process.

Fig. 7 is a schematic diagram of a decoding flow at the decoding end.

Detailed Description

In order to more specifically describe the present invention, the following detailed description is provided for the technical solution of the present invention with reference to the accompanying drawings and the specific embodiments.

The invention relates to a video compression method of a remote desktop, as shown in figure 1, a coding end firstly blocks current desktop image data, then selects an interframe coding reference frame object, utilizes the reference frame object to carry out maximum region matching search and adjust background frame content, then carries out lossless data compression on region matching information and unmatched block image information, and finally packs and sends compressed content; and the decoding end decompresses the received packets, respectively restores corresponding image information and adjusts the content of a background frame according to the maximum region matching information and the unmatched block data information, combines all image blocks and displays the desktop image.

As shown in fig. 2, the specific implementation process of the encoding end includes:

A1. dividing the original image data into blocks of a suitable fixed size, such as 8 x 8 sized data blocks; if the resolution of the screen image cannot be divided by 8, default color expansion image data needs to be filled to adapt to the block size, and the decoding end also needs to perform corresponding processing.

A2. As shown in fig. 4, the current image data is used to match the image information in the forward frame and the background frame, and the frame with the largest pixel matching number is selected as the reference frame. The forward frame (p frame) is the image of the previous code, the background frame is not the image of the desktop at a certain moment, but the background image formed by combining the images with larger probability in a certain time period of the remote control starting at a certain position, and meanwhile, the background frame can be a plurality of frames, but in actual use, because of factors such as processor performance memory resources, only one background frame is usually provided; pixel matching is not consistent with the value of the same position as matching, but pixel values of different positions are also considered to be matching, for example, 100 a values exist in a reference frame, 80 a values exist in a current image, and then 80 pixels are considered to be matching; and places the selected reference frame type information (whether it be a forward frame or a background frame) into a lossless compression queue.

A3. After the reference frame is selected, the maximum matching area is searched in the reference frame, and the specific process is as shown in fig. 5:

a3-1 searches for a pixel value corresponding to a reference frame by using the first pixel value of a block through a linked list structure in fig. 4 or an index structure such as a suffix tree, and then determines the matching degree of the block by using the size of a macroblock, where all pixel values of the macroblock are not necessarily completely consistent and are considered as a matching block, and only the pixels of the macroblock exceed a certain threshold (e.g., 60%) can be considered as the matching block.

If the corresponding matching block is not searched in A3-2, the non-matching block is considered and the process proceeds to step A4.

A3-3 if finding whether the block on the right side of the adjacent block is a matching block first, if so, continuing the search until the line is finished or not, thereby determining the x coordinate matching range; and then searching whether the next block adjacent to the matching block is the matching block, and if so, searching whether the range of the column is completely matched, and determining the y coordinate matching range in a circulating manner.

A3-4 records information of a search range and a motion vector if it is range matching, and is generally described by { starting x-axis number of a range block, starting y-axis number of a range block, x-axis size of a range block, y-axis size of a range block, x-axis positive-negative pixel shift offset, y-axis positive-negative pixel shift offset }, and it is necessary to record a residual error.

A3-5, if it is a single matching block, is described as { starting x-axis number of range block, starting y-axis number of range block, x-axis positive and negative pixel shift offset, y-axis positive and negative pixel shift offset }, while paying attention to the following points:

the motion vector in the search is in pixel unit, and the original image is searched in matching length unit on the basis of block, but the moving distance of the macro block is usually in pixel unit, not in block unit, so the moving distance in pixel unit is recorded by the motion vector.

Secondly, the area range searched is in units of blocks rather than pixels, and because the unmatched macro blocks need dct conversion to be beneficial to subsequent lossless compression, the blocks are used as basic units for matching.

The matched block should mark the macroblock as a searched block, and the subsequent blocks should not be searched repeatedly, and it can be considered that the row or the column is finished.

Fourthly, because the matching of the blocks is not complete matching, the residual error corresponding to the matching block needs to be recorded.

The single motion vector and the residual error are not stored adjacently, but all the motion vector information is stored sequentially first, and then all the residual error information is stored sequentially, so that the space redundancy caused by the storage can be reduced, and the compression ratio can be improved by subsequent lossless compression; meanwhile, the residual information should be consistent with the sequence of recording the motion vector, so that the residual information can be recorded without recording the coding information of the block, and the macro block to which the residual information belongs can be known.

A4. Performing dct conversion on the rest unmatched blocks to remove high-frequency signals, and then putting the dct-converted data into a lossless compression queue; the dct data should be stored in the specified order of unmatched partitions, so that the coordinates of the partitions are not recorded, reducing redundancy.

A5. The data in the lossless compression queue is compressed by using a lossless compression algorithm, the lossless compression of the data can be performed by combining an lz77 dictionary compression algorithm and various entropy encoders (algorithms such as Markov prediction chain, huffman tree and the like), and the packets can be transmitted through a network after the data compression is completed.

A6. The weight update is performed on the blocks in the background frame, as shown in fig. 6:

a6-1 is compared with the corresponding position macro-blocks of the background frame and the forward frame one by one according to the macro-block order, and if the same pixel value exceeds a certain threshold value, such as 60%, the similar block is considered.

A6-2, if it is a similar block of the background frame, should increase the weight by no more than a specified upper limit for weight + 1.

A6-3 weights-1 if it is not a similar block of the background frame until the weight is reduced to 0 and is not reduced if it is already 0.

A6-4 needs to judge whether it is a similar block of the forward frame if the weight of a certain macro block decays to 0, and does not make any change if it is a similar block of the forward frame.

A6-5, if the weight of a certain macroblock decays to 0 and is not a similar block of the previous forward frame, replacing the macroblock of the background frame with the macroblock corresponding to the forward frame, and setting the weight value of the macroblock to a certain default initial value.

As shown in fig. 3 and 7, the specific implementation process of the decoding end includes:

B1. and decompressing the received compressed data by using a corresponding lossless decompression algorithm, acquiring information of which type the reference frame is after decompression is finished, reading corresponding data, and distinguishing whether subsequent data is information of a matched macro block or information of a non-matched block.

B2. If the area range matches the macro block information, the following steps are executed:

b2-1 reads the corresponding required range-related information or macroblock number information and the corresponding motion vector depending on whether it is range matching or single macroblock matching.

B2-2 finds the corresponding macro block of the reference frame according to the macro block information and the motion vector.

B2-3 reads residual information of a corresponding macroblock and restores image information of the macroblock one by one through the residual.

B2-4 marks the macroblock that has been restored as the restored flag.

B3. After the matched macro block is recovered, recovering the non-matched block, specifically executing the following steps:

b3-1 the first unrecovered macroblock is found because all macroblocks in the matching range have been marked as restored in the above flow.

B3-2 searches dct information of corresponding non-matching blocks in the decompressed data queue, because the non-matching blocks do not record information such as macroblock numbers of the non-matching blocks in order to reduce redundancy during encoding, but the size of each macroblock is fixed, the size of data after filtering the dct high-frequency signal is also fixed, and the dct is stored one by one according to the sequence of the non-matching blocks during storage, so that the corresponding dct information can be searched.

B3-3 restores the image information at the macroblock position by idct (inverse dct transform) based on dct information, and since dct is performed with high frequency filtering, the image information at this time is damaged.

B4. Updating the weight information of the background frame at the decoding end, and specifically comprising the following steps:

b4-1 compares with the corresponding position macro-blocks of the background frame and the forward frame one by one in the macro-block order, and if the same pixel value exceeds a certain threshold value, such as 60%, it is considered as a similar block.

B4-2 if it is a similar block of the background frame, the weight should be increased by no more than a specified upper limit for weight + 1.

B4-3 weights-1 if it is not a similar block of the background frame until the weight is reduced to 0 and is not reduced if it is already 0.

B4-4 needs to judge whether it is a similar block of the forward frame if the weight of a certain macroblock decays to 0, and does not make any change if it is a similar block of the forward frame.

B4-5 if the weight of a certain macroblock decays to 0 and is not a similar block of the previous forward frame, replacing the macroblock of the background frame with the macroblock corresponding to the forward frame and setting the weight value of the macroblock to a certain default initial value.

The embodiments described above are presented to enable a person having ordinary skill in the art to make and use the invention. It will be readily apparent to those skilled in the art that various modifications to the above-described embodiments may be made, and the generic principles defined herein may be applied to other embodiments without the use of inventive faculty. Therefore, the present invention is not limited to the above embodiments, and those skilled in the art should make improvements and modifications to the present invention based on the disclosure of the present invention within the protection scope of the present invention.

Claims

1. A video compression method of a remote desktop comprises the following steps:

(1) partitioning a video image in the current desktop by a coding end;

(2) matching the current frame with a forward frame and a background frame, and taking the frame with the maximum pixel matching number with the current frame as a reference frame; the background frame is a background image formed by combining images with higher probability in a certain time period in a certain position block at the beginning of remote control;

after the above operations are completed, the background frame of the encoding end needs to be updated, and the specific process is as follows:

for any image block in the background frame, if the current weight of the image block is 0 and the image block at the corresponding position in the forward frame is not similar to the image block at the corresponding position in the current frame, replacing the image block in the background frame by the image block at the corresponding position in the forward frame and resetting the weight of the image block as a default initial value;

2. The video compression method of claim 1, wherein: in the step (1), the video image is divided into a plurality of n × n image blocks, where n is a natural number greater than 1, and if the size of the entire image cannot be divided by n, the image edge needs to be extended to a default color to adapt the image size to the image block.

3. The video compression method of claim 1, wherein: in the step (3), the current frame is matched and searched with the reference frame by taking the block as a unit, and if any image block in the current frame is consistent with the corresponding block of the reference frame in pixel values exceeding a certain proportion, the image block is judged to be a matching block; if the current image block is a matching block, whether the image block on the right side is the matching block is continuously judged according to the above steps until the row direction is finished or a non-matching block is searched; and simultaneously judging whether the image block below the current image block is a matching block or not until the column direction is finished or a non-matching block is searched, so as to determine a matching area in the current frame.

4. The video compression method of claim 1, wherein: in the step (3), matching search is performed by taking a block as a unit, the motion vector is taken as a unit, the image blocks which are matched and searched need to be marked, and all residual error information should be stored in sequence after the motion vectors of all pixels are stored in sequence in the storage process.

5. The video compression method of claim 1, wherein: and (4) performing DCT (discrete cosine transformation) on the residual unmatched blocks in the current frame in the step (3), and then sequentially storing the converted data into a lossless compression queue.

6. The video compression method of claim 1, wherein: and (4) performing data compression on the lossless compression queue by adopting an LZ77 dictionary compression algorithm and an entropy encoder in combination, and packaging and sending the data to a remote decoding end.

7. The video compression method of claim 1, wherein: the specific implementation process of the step (5) is as follows: