CN115190303A - A cloud desktop image processing method, system and related equipment - Google Patents

Abstract

The invention discloses a cloud desktop image processing method, system and related equipment. The method includes: dividing the current image into a plurality of image blocks; based on the hash value of each image block in the current image and the image of the previous frame, determining the invariant area and the change area of the current image relative to the image of the previous frame; Hash value of the image blocks in the changed area, perform motion search for the image blocks in the changed area in the previous frame image, determine the moving area and new area of the current image relative to the previous frame image, and determine the image blocks in the moving area The motion vector relative to the previous frame of image; the data to be transmitted in the current image is determined based on the unchanged area, the moving area and the newly added area. It can accurately analyze the invariant area, change area and type of change area in the image, and determine the transmitted data based on the analysis results to reduce the amount of data transmission, improve the data transmission speed, and meet the real-time and low-latency requirements of cloud desktop image transmission. .

Description

A cloud desktop image processing method, system and related equipment

technical field

The invention relates to the technical field of image processing, in particular to a cloud desktop image processing method, system and related equipment.

Background technique

Cloud desktop is a new model that uses cloud computing technology to replace traditional computers. The computer host is virtualized in the cloud server. The mainstream front-end equipment is to use a lightweight client to connect the display and input devices. After installing the client on the front-end device Access the virtual computer host on the cloud server through a unique communication protocol to achieve interactive operation, and achieve the same experience effect as the traditional computer host. A cloud application is an application in which the front-end device interacts with the cloud server. The application runs on the cloud server and is synchronized to the front-end device for display.

To use cloud desktop and cloud applications, the cloud desktop image on the virtual host of the cloud server needs to be compressed and encoded to the local client of the front-end device, so that it can be displayed on the front-end device. In order to improve the efficiency of image compression, the correlation between image frames can be used to analyze the image content of each frame, and then different compression processing methods can be used for different types of image areas, so as to reduce the amount of data transmission and bandwidth. need.

There are two ways to perform image analysis in the prior art, one is to analyze image changes and motion information output by the screen capture of the operating system, and the other is to use an open source computer vision framework (such as opencv) to perform image analysis.

SUMMARY OF THE INVENTION

The inventor of the present application found that the image analysis methods in the prior art cannot meet the requirements of low latency and real-time performance of cloud desktops and cloud applications. Accurate, the movement and changes of the image frame cannot be accurately known, resulting in a large amount of redundancy, which increases the amount of data to be transmitted, cannot meet the requirements of fast data transmission, and the data transmission time is still relatively long; using opencv for image analysis, Although the analysis accuracy has been improved to a certain extent, the analysis process is time-consuming and cannot meet the low-latency and real-time requirements of cloud desktops. In a word, there is no good image analysis method in the prior art, which cannot accurately determine the invariant area, change area, motion situation, image type and other information between image frames, that is, it cannot accurately determine the information such as high precision and high accuracy. Therefore, it is impossible to quickly and accurately divide different types of image areas for targeted compression processing, and it cannot meet the transmission delay requirements of cloud desktops and cloud applications.

In view of the above problems, the present invention is proposed to provide a cloud desktop image processing method, system and related equipment that overcome the above problems or at least partially solve the above problems.

An embodiment of the present invention provides a cloud desktop image processing method, including:

Divide the current image into multiple image blocks;

Based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image, determine the unchanged area and the changed area of the current image relative to the previous frame of image;

Based on the hash value of the image blocks in the changing area, perform a motion search for the image blocks in the changing area in the previous frame of image, determine the moving area of the current image relative to the previous frame image and the newly added area in the current image, and Determine the motion vector of the image block in the moving area relative to the previous frame image;

Based on the unchanged area, the moving area and the newly added area of the current image, the data to be transmitted in the current image is determined.

In some optional embodiments, the dividing the current image into multiple image blocks includes:

According to the preset image block size, the current image is divided into multiple image blocks of equal size; the image block size is determined according to the resolution and configuration information of the current image.

In some optional embodiments, based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image, determine the invariance of the current image relative to the previous frame of image Areas and changing areas, including:

Calculate the hash value of each image block in the current image;

Compare the hash value of each image block in the current image with the image block in the same position in the previous frame image;

Determine the invariant area of the current image relative to the previous frame image based on the image blocks with the same hash value;

Based on the image blocks with different hash values, determine the change area of the current image relative to the previous frame image.

In some optional embodiments, based on the hash value of the image blocks in the changed area, a motion search is performed for the image blocks in the changed area in the previous frame of image, and the moving area and the moving area of the current image relative to the previous frame of image are determined. The newly added area in the current image, and the motion vector that determines the image block in the moving area relative to the previous frame of image, including:

The image blocks in the changed area are used as the image blocks to be matched, and for each image block to be matched, a motion search is performed in the previous frame of image. If a matching image block with the same hash value is found in the previous frame of image , the motion vector of the image block to be matched relative to the previous frame image is determined according to the position of the image block to be matched in the current image and the position of the image block to be matched in the previous frame image;

Based on the to-be-matched image blocks with matching image blocks in the previous frame image, the moving area of the current image relative to the previous frame image is obtained; other areas in the changed area except the moving area are used as new areas.

In some optional embodiments, based on the to-be-matched image blocks that have matching image blocks in the previous frame of image, the moving area of the current image relative to the previous frame of image is obtained, including:

For a to-be-matched image block with a matching image block in the previous frame of image, if its adjacent multiple image blocks have the same motion vector, the image block to be matched is centered on the out-diffusion to find the image block with the same motion vector. image blocks, forming groups of image blocks with the same motion vector;

If the obtained image block group is unique, the obtained image block group is used as the moving area;

If the obtained image block group is not unique, the image block group with the largest area is used as the moving area.

In some optional embodiments, the method further includes: using the following fast hash algorithm to calculate the hash value of the image block:

The pixel coordinate (x, x+N-1) corresponding to each pixel coordinate y in the image is a pixel element meta, where N indicates that the size of the image block is N*N, and N is a positive integer;

According to the pixel values of N pixels included in each meta, calculate the hash value of each meta meta_hash(x, x+N-1; y);

Perform hash calculation on the hash values of consecutive N metas starting from pixel coordinate y, and obtain the hash value of the image block starting from pixel coordinate (x, y), where the hash values of consecutive N metas include meta_hash(x, x+N-1; y), meta_hash(x, x+N-1; y+1), ..., meta_hash(x, x+N-1; y, y+N-1).

In some optional embodiments, the determining the data that the current image needs to transmit to the front-end device includes:

The motion vector of the image block in the moving area relative to the image of the previous frame and the content data of the image block in the non-moving area are taken as the data to be transmitted to the front-end device of the current image.

In some optional embodiments, after determining the moving area of the current image relative to the previous frame of image and the newly added area in the current image, the method further includes:

Color analysis is performed on the data blocks in the newly added area, and the content data type of each data block is determined according to the color complexity, and the content data type includes at least one of text and pictures; correspondingly, the determination of the current image requires After the transmitted data, the method further includes: selecting a compression algorithm according to the content data type of the image block, and using the selected compression algorithm to compress the content data; and/or

Correct the moving area and adjust the moving area to a square or rectangular area.

The embodiment of the present invention also provides another cloud desktop image processing method, including:

Receive the data of the current image transmitted by the cloud server;

The content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame image are parsed from the received data;

Obtain the content data of the image block in the unchanged area of the current image from the image of the previous frame, and obtain the image block in the moving area from the image of the previous frame according to the motion vector of the image block in the moving area relative to the image of the previous frame. content data;

Based on the parsed content data of the data blocks in the newly added area, and the acquired content data of the image blocks in the moving area and the unchanged area, the current image is generated.

In some optional embodiments, obtaining the content data of the image block in the moving area from the image of the previous frame according to the motion vector of the image block in the moving area relative to the image of the previous frame, including:

According to the position of the image block in the moving area in the current image and the motion vector relative to the previous frame image, determine the position of the image block in the moving area in the previous frame image, and obtain the content of the corresponding image block according to the determined position data.

An embodiment of the present invention also provides a cloud server, which includes: a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the above-mentioned cloud desktop when executing the program image processing method.

An embodiment of the present invention also provides a front-end device, including: a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implements the above-mentioned another cloud desktop when executing the program image processing method.

An embodiment of the present invention further provides a cloud desktop image transmission system, including: the above-mentioned cloud server and the above-mentioned front-end device;

the cloud server, configured to transmit the determined transmission data to the front-end device;

The front-end device is configured to receive the data provided by the cloud server, and generate the current image according to the received data and the data of the previous frame of image.

Embodiments of the present invention further provide a computer storage medium, where computer-executable instructions are stored in the computer storage medium, and when the computer-executable instructions are executed by a processor, the above-mentioned one cloud desktop image processing method or the above-mentioned other method is implemented. A cloud desktop image processing method.

The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

The above method provided by the embodiment of the present invention analyzes the image based on the hash value of each image block in the current image of the data to be transmitted and the hash value of each image block in the previous frame of image, For one frame of image, the unchanged area that has not changed and the changed area that has changed in the current image, and the changed area is further analyzed to determine the moving area that has moved relative to the previous frame of image and the new area in the current image. This method can quickly and accurately analyze the unchanged area, moving area and new area in the image, and adopt different transmission processing methods for the content data of the unchanged area, moving area and new area. The area can directly use the content data of the previous frame image without data transmission. For the moving area, the motion vector relative to the previous frame image needs to be transmitted to determine which image block or content data in the previous frame image is moved. Yes, for new areas, content data needs to be transmitted. This method can compress and transmit data in a targeted manner for different areas, reduce the data to be transmitted as much as possible, reduce transmission delay, and meet the requirements of cloud desktop and cloud applications. Delay and high real-time transmission requirements.

Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description, claims, and drawings.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the attached image:

1 is a flowchart of a method for processing a cloud desktop image in Embodiment 1 of the present invention;

FIG. 2 is a schematic diagram of the principle of a cloud desktop image processing method in Embodiment 1 of the present invention;

3 is one of the schematic diagrams of the principle of hash value calculation in Embodiment 1 of the present invention;

4 is the second example diagram of the principle of hash value calculation in Embodiment 1 of the present invention;

5 is a flowchart of a method for processing a cloud desktop image in Embodiment 2 of the present invention;

6 is a schematic structural diagram of a cloud desktop image processing apparatus according to an embodiment of the present invention;

7 is a schematic structural diagram of another cloud desktop image processing apparatus according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a cloud desktop image processing system according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the problem in the prior art that different types of image areas cannot be quickly and accurately divided in the process of cloud desktop image transmission for targeted transmission processing, the embodiment of the present invention provides a cloud desktop image processing method, which can quickly and accurately 、Accurately analyze the unchanged area, moving area and new area in the image, and adopt different transmission processing methods for the content data of the unchanged area, moving area and new area to meet the low latency of cloud desktop and cloud applications , High real-time transmission requirements. The following is a detailed description through specific embodiments.

Example 1

Embodiment 1 of the present invention provides a cloud desktop image processing method, the process of which is shown in FIG. 1 and includes the following steps:

Step S101: Divide the current image into multiple image blocks.

For each frame of image to be transmitted data, according to the preset image block size, the current image is divided into multiple image blocks of equal size; the image block size is determined according to the resolution and configuration information of the current image.

The implementation principle of the cloud desktop image processing method provided in this embodiment is shown in FIG. 2 . This step belongs to the preprocessing step for the current image, and corresponds to the block pro-processing step in FIG. 2 . It refers to the image block division of the image, which can be divided into image blocks for each frame of image, and the entire frame image is divided into several image blocks (blocks) of equal size. The block size can be adjusted according to the resolution and configuration. It can be set according to the actual needs of cloud desktops and cloud applications. For example, the size of the block can be set to 16*16 pixels, 32*32 pixels, etc., and 16*16 pixels will be used as an example in the following examples.

Step S102: Based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image, determine the unchanged area and the changed area of the current image relative to the previous frame of image.

In this step, the hash value of each image block in the current image is calculated; the hash value of each image block in the current image is compared with the image block in the same position in the previous frame of image respectively; based on the same hash value The image block determines the unchanged area of the current image relative to the previous frame of image; the image block with different hash values determines the changed area of the current image relative to the previous frame of image. That is to determine the part of the image block in the current image that has the same hash value as the image block in the same position in the previous frame image, and obtain the unchanged area; determine the image block hash value in the current image and the same position in the previous frame image. Part of the image block that is not in phase, get the change area.

The fast hash algorithm can be used to calculate the hash value of each image block. For each frame of image, the hash value of each image block can be calculated after the image block is divided for subsequent use, or it can be done during Calculate the hash value of each image block in the current image to be compared with the previous frame image before the comparison. Usually, the hash value of the image block in the previous image has been processed before the data of the previous frame image is transmitted. Therefore, when comparing, it is sufficient to calculate the hash value of each image block in the current image and obtain the previously calculated hash value of the image block in the previous frame of image for comparison.

When comparing, compare the hash value of the image block in the current image with the hash value of the image block in the same position in the image block of the previous frame, and determine whether the image block in the current image has changed, if the hash value is the same, then It is considered that no change has occurred, and if the hash values are different, it is considered that a change has occurred, thereby determining the unchanged area and the changed area of the current image relative to the previous frame of image. The unchanged area includes image blocks in which the current image has not changed relative to the previous frame of image, and the changed area includes image blocks in which the current image has changed relative to the previous frame of image.

Referring to the hash calculation part and the changed area detection (dirty detection) part in Figure 2, compare the hash value of each block in the current image with the block in the same position of the previous frame image to determine whether there is any change or not. The block is recorded as a changed (dirty) image block, which forms a changed area, and the unchanged image block is recorded as a constant (skip) image block, which forms an unchanged area. Motion estimation is performed for the dirty image block, and no processing is performed for the skip image block. .

Step S103: Based on the hash value of the image block in the changed area, perform a motion search for the image block in the changed area in the previous frame of image, and determine the moving area of the current image relative to the previous frame of image and the newly added image in the current image. area, and determine the motion vector of the image block in the moving area relative to the previous frame image.

The image blocks in the changed area are used as the image blocks to be matched, and for each image block to be matched, a motion search is performed in the previous frame of image. If a matching image block with the same hash value is found in the previous frame of image , the motion vector of the image block to be matched relative to the previous frame image is determined according to the position of the image block to be matched in the current image and the position of the matching image block in the previous frame image; The image block to be matched is obtained, and the moving area of the current image relative to the image of the previous frame is obtained; other areas in the changed area except the moving area are regarded as the newly added area.

When the motion search is performed in the previous frame of image, the search can be performed row by column, that is, starting from the first row and first column, until the position of the last row and last column that can form an image block is searched. Taking an image block of N*N pixels as an example, N is a positive integer. For each image block in the changing area, search for image blocks with the same hash value in the previous frame of image, starting from the first row and first column , take each position point (x, y) as the starting point to form an image block with the same size as the image block to be matched. For example, the starting point position is (x, y), then the pixels included in each image block are (x, x +N-1; y, y+N-1), that is, the N*N image blocks starting from (x, y), the image blocks to be matched in the change area are matched with each image block formed one by one , the matched image blocks with the same hash value are the matched image blocks of the image blocks to be matched.

For a to-be-matched image block with a matching image block in the previous frame of image, if its adjacent multiple image blocks have the same motion vector, the image block to be matched is centered on the out-diffusion to find the image block with the same motion vector. image block, forming an image block group with the same motion vector; if the obtained image block group is unique, the obtained image block group is used as the moving area; if the obtained image block group is not unique, the image block group with the largest area is used as a mobile area.

Referring to the motion estimation in Figure 2, do a motion search for the dirty area, and for each dirty block of the current image frame, match the block with the same hash value in the previous image, and the positions of these two blocks The difference is recorded as a motion vector (MV). If the four adjacent blocks of the block have unique matching blocks and the motion vectors are the same, the block and MV are used as seed blocks for diffusion search to find more Blocks with the same MV and blocks with the same MV form an image block group.

Referring to the motion vector filter (MV filter) in Figure 2, for a current image, it is possible to obtain multiple image block groups with the same MV, but due to the image changes of the cloud desktop and cloud applications, there can only be one motion at a time Therefore, MV filtering needs to be performed. When filtering, the area area of the image block group corresponding to each MV can be calculated, and the image block group with the largest area can be found, and its MV is used as the MV of the moving area in the current image, and all the dirty MVs The block is denoted as the image block of the moving area.

Optionally, correcting the moving area is performed to adjust the moving area into a square or rectangular area. In order to facilitate the processing on the application layer, try to correct the moving area composed of all dirty blocks into a regular shape, such as a rectangle or a square. The moving area only needs to transmit the motion vector, and the client on the front-end device can be based on the previous frame image and motion vector. Generates the current image without transferring specific image content data.

Step S104: Determine the data to be transmitted in the current image based on the unchanged area, the moving area and the newly added area of the current image.

According to the determined unchanged area, moving area and new area of the current image, different transmission processing methods are used for processing. For the content data of the image blocks in the unchanged area, data may not be transmitted, and for the image blocks in the moving area, the data may not be transmitted. Only the motion vector data is transmitted, and the content data needs to be transmitted for the image blocks in the newly added area. Therefore, based on the unchanged area, the moving area and the newly added area of the current image, the image blocks in the moving area are compared with those of the previous frame image. The motion vector and the content data of the image blocks in the non-moving area are the data that the current image needs to transmit to the front-end device. See the output section in Figure 2.

In some optional embodiments, after determining the moving area of the current image relative to the image of the previous frame and the newly added area in the current image, the method further includes: performing color analysis on the data blocks in the newly added area, according to the color complexity Judging the content data type of each data block, the content data type includes at least one of words and pictures; refer to the color analysis (color analysis) part in Figure 2; Correspondingly, after determining the data that the current image needs to transmit, it also includes : Select the compression algorithm according to the content data type of the image block, and use the selected compression algorithm to compress the content data.

After the cloud server determines the data to be transmitted, it encodes the data when transmitting the data. Encoding refers to the process of converting the original image into a bit video stream through compression technology. Different areas such as text areas and picture areas are compressed differently. The algorithm can further improve the compression efficiency.

In the above method, a fast hash algorithm can be used to calculate the hash value of each image block. Since each frame of image needs to be hashed, the amount of calculation is very large, especially in the motion search step, which needs to be performed row by row and column by column. to search for a match, that is, for each pixel block (x, x+N-1; y, y+N-1) in the previous frame of image, the hash value calculation will be performed, and the calculation amount will be even greater, so , the present invention provides a fast hash algorithm to reduce the overall computing time to meet the real-time requirements.

During motion search, for an image block to be matched in the current image, it needs to be matched row by row and column by column in the previous frame of image. Therefore, for each image block that can be divided into an image, it is necessary to calculate multiple Hash value, or in other words, each pixel (x, y) in the previous frame of image is used as the starting position, and the hash value is calculated for the image blocks that can be formed.

Referring to Figure 3, taking the image block size of 16*16 pixels and the vertical direction of the image block as an example, during motion search, a hash value (MB) is calculated for each row in each image block. hash), that is, if the pixel range of the image block is (x, x+15; y, y+15), starting from the starting position (x, y) of the image block, 16 MB hashes need to be calculated.

MB_hash(x, x+15; y, y+15) = HASH{rgb(x, y), rgb(x+1, y), ..., rgb(x+15, y); rgb(x, y+ 1), rgb(x+1, y+1), …, rgb(x+15, y+1); …; rgb(x, y+15), rgb(x+1, y+15), ...,rgb(x+15,y+15)}

MB_hash(x, x+15, y+1, y+16) = HASH{rgb(x, y), rgb(x+1, y), ..., rgb(x+15, y); rgb(x, y+1), rgb(x+1, y+1), …, rgb(x+15, y+1); …; rgb(x, y+15), rgb(x+1, y+15 ), ..., rgb(x+15, y+15)}

MB_hash(x, x+15, y+2, y+17) = HASH{rgb(x, y), rgb(x+1, y), ..., rgb(x+15, y); rgb(x, y+1), rgb(x+1, y+1), …, rgb(x+15, y+1); …; rgb(x, y+15), rgb(x+1, y+15 ), ..., rgb(x+15, y+15)}

...

MB_hash(x, x+15, y+15, y+30) = HASH{rgb(x, y+15), rgb(x+1, y+15), ..., rgb(x+15, y+15) ), rgb(x, y+16), rgb(x+1, y+16), …, rgb(x+15, y+16); …; rgb(x, y+30), rgb(x +1, y+30), ..., rgb(x+15, y+30)}

The horizontal direction is also similar, and each image block in the image needs to be calculated. Therefore, this method requires a huge amount of calculation and is relatively time-consuming. Using a fast hash algorithm can reduce the amount of calculation and greatly improve the calculation speed.

In the above method, the following fast hash algorithm can be used to calculate the hash value of the image block for the image block pre-divided in each frame of image and each image block that may be used for matching in the motion search process:

The pixel coordinate (x, x+N-1) corresponding to each pixel coordinate y in the image is a pixel meta, where N indicates that the size of the image block is N*N; N is a positive integer, according to each meta including The pixel value of the N pixels, calculate the hash value of each meta meta_hash (x, x+N-1; y); hash the hash value of the consecutive N metas starting from the pixel coordinate y, Get the hash value of the image block starting with pixel coordinates (x, y), where the hash values of consecutive N metas include meta_hash(x, x+N-1; y), meta_hash(x, x+N- 1; y+1), ..., meta_hash(x, x+N-1; y, y+N-1). That is, for each image block, a hash value can be calculated by taking each row of pixels as a pixel element, and then the hash value of the image block can be obtained by hashing the hash values of each row.

Referring to the schematic diagram of the fast hash algorithm shown in Figure 4, taking the image block size of 16*16 pixels as an example, first define the (x, x+15) pixel corresponding to each y coordinate as a meta, and for each meta Calculate meta_hash, and then perform hash calculation on 16 consecutive meta_hash in the vertical direction, that is, MB_hash can be obtained.

meta_hash(x, x+15; y)=HASH{rgb(x, y), rgb(x+1, y), ..., rgb(x+15, y)}

meta_hash(x, x+15; y+1) = HASH{rgb(x, y+1), rgb(x+1, y+1), ..., rgb(x+15, y+1)}

meta_hash(x, x+15, y+2)=HASH{rgb(x, y+2), rgb(x+1, y+2), ..., rgb(x+15, y+2)}

...

meta_hash(x, x+15; y+30) = HASH{rgb(x, y+30), rgb(x+1, y+30), ..., rgb(x+15, y+30)}

Then calculate MB_hash:

MB_hash(x, x+15, y, y+15) = HASH{meta_hash(x, x+15, y), meta_hash(x, x+15, y+1)..., meta_hash(x, x+15 , y+15)}

MB_hash(x, x+15, y+1, y+16) = HASH{meta_hash(x, x+15; y+1), meta_hash(x, x+15; y+2)..., meta_hash(x , x+15; y+16)}

...

MB_hash(x, x+15, y+15, y+30) = HASH{meta_hash(x, x+15; y+15), meta_hash(x, x+15; y+16)..., meta_hash(x , x+15; y+30)}

For the above fast hash algorithm, the hash algorithm accelerated by SSE/AVX instructions and multi-threaded parallel computing can be used for the calculation of each meta hash, which further greatly improves the computing performance. SSE/AVX are hardware accelerated instructions provided by computer systems.

The above method in this embodiment may be implemented in a cloud server, and after analyzing the image of the data to be transmitted, the image data is transmitted according to the analysis result. The method analyzes the image based on the hash value of the image block in the current image of the data to be transmitted, and determines the unchanged area that has not changed and the changed area that has changed in the current image relative to the previous frame of image. , and further analyze the changed area to determine the moving area that has moved relative to the previous frame of image and the newly added area in the current image. This method can quickly and accurately analyze the unchanged area, moving area, new area in the image. In addition, different transmission processing methods are adopted for the content data of the unchanged area, the moving area and the newly added area. The content data of the previous frame image can be directly used for the unchanged area without data transmission, and the moving area needs to be transmitted relatively. It is based on the motion vector of the previous frame of image to determine which image block or content data in the previous frame of image moved, and for the newly added area, the content data needs to be transmitted. This method can be targeted for different areas. Data compression and transmission reduce the data to be transmitted as much as possible, reduce the transmission delay, and meet the low-latency and high-real-time transmission requirements of cloud desktops and cloud applications. The speed and efficiency of data processing can be further improved and the delay can be reduced by means of fast hashing algorithm, color analysis to use different compression algorithms, etc.

Embodiment 2

The second embodiment of the present invention provides a cloud desktop image processing method, which is implemented on a front-end device, and cooperates with the method provided in the first embodiment to receive and process the transmitted image data, so that the cloud desktop, cloud The application, etc. can quickly and correctly display the cloud desktop image. The method flow is shown in Figure 2, including the following steps:

Step S201: Receive data of the current image transmitted by the cloud server.

The data transmitted by the cloud server mainly includes the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame image. Of course, the image block of the invariant area may not be specified. By default, the image block that does not transmit any data is the image block of the invariant area.

Step S202 : parse out the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame of image from the received data.

The cloud server encodes the data when transmitting the data, and the front-end device decodes the received data after receiving the data. Encoding refers to the process of converting an original image into a bit video stream through compression techniques, and decoding is the opposite process.

Step S203: Acquire content data of the image block in the moving area from the image of the previous frame according to the motion vector of the image block in the moving area relative to the image of the previous frame.

In this step, according to the position of the image block in the moving area in the current image and the motion vector relative to the previous frame of image, the position of the image block in the moving area in the previous frame image is determined, and the corresponding image is obtained according to the determined position. The content data of the image block. That is to say, for the moving area, according to the motion vector and the position of the image block in the moving area in the current image, the position of the image block in the previous frame image can be obtained, and the content data of the corresponding position in the previous frame image can be obtained, namely Can.

Step S204: Acquire the content data of the image block in the unchanged area of the current image from the image of the previous frame.

For the unchanged area, the position of the image block in the current image is its position in the previous frame of image, and the content data of the corresponding position in the previous frame of image can be obtained.

The above steps S203 and S204 are executed in no particular order, and may be executed simultaneously.

Step S205: Generate a current image based on the parsed content data of the data blocks in the newly added area, and the acquired content data of the image blocks in the moving area and the unchanged area.

Based on the position of the image block, the current image can be generated by rendering the content data of the image block. Rendering refers to the process of graphics processing to generate and draw an image.

Based on the same inventive concept, an embodiment of the present invention also provides a cloud desktop image processing device, which can be set in a cloud server. The structure of the device is shown in FIG. 6 , including:

The dividing module 11 is used for dividing the current image into multiple image blocks;

The calculation and comparison module 12 is used to determine the invariant area and the change area of the current image relative to the previous frame image based on the hash value of the image block in the current image;

The motion estimation module 13 is configured to perform a motion search for the image blocks in the changed area in the previous frame image based on the hash value of each image block in the changed area and the hash value of each image block in the previous frame image , determine the moving area of the current image relative to the previous frame image and the newly added area in the current image, and determine the motion vector of the image block in the moving area relative to the previous frame image;

The data processing module 14 is configured to determine the data to be transmitted in the current image based on the unchanged area, the moving area and the newly added area of the current image.

Based on the same inventive concept, the embodiment of the present invention also provides another cloud desktop image processing device, which can be set in the front-end equipment. The structure of the device is shown in FIG. 7 , including:

The data receiving module 21 is used for receiving the data of the current image transmitted by the cloud server;

The data parsing module 22 is used to parse out the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame image from the received data;

The data acquisition module 23 is used to acquire the content data of the image block in the unchanged area of the current image from the previous frame image, and acquire from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image Content data of image blocks in the moving area;

The image generation module 24 is configured to generate a current image based on the content data of the data blocks in the newly added area and the acquired content data of the image blocks in the moving area and the unchanged area.

Based on the same inventive concept, an embodiment of the present invention further provides a cloud desktop image transmission system, including a cloud server 1 and a front-end device 2 . There can be multiple cloud servers 1 and front-end devices 2, and the cloud server can adopt a server cluster and allow a large number of front-end devices to access.

In some optional embodiments, the above-mentioned cloud server 1 may include: a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the program described in FIG. 1 when the processor executes the program. The cloud desktop image processing method.

In some optional embodiments, the above-mentioned front-end device 2 may include: a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the program described in FIG. 5 when the processor executes the program The cloud desktop image processing method.

Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, where computer-executable instructions are stored in the computer storage medium, and when the computer-executable instructions are executed by a processor, the cloud desktop image described in FIG. 1 is realized The processing method or the cloud desktop image processing method described in FIG. 5 .

Regarding the apparatus and system in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

In the above-mentioned method, system and device of the embodiments of the present invention, each frame of image is divided into blocks, the image is analyzed based on the blocks, and the hash value is calculated for each block by using a fast hash algorithm. Then, by comparing the hash to find the changed area and the area without change, and further analyze the changed area, try to search the motion vector of the moving area. Content analysis is performed on the non-moving changed area (ie, the newly added area), and the image area and the text area are distinguished. Based on block analysis, high-accuracy and high-precision image change information can be obtained. The size of the block can be flexibly configured, and image changes, motion, and content information can be obtained in one operation, saving computing resources. This process takes each frame of image captured by the cloud desktop/cloud application as input, and takes the analysis result of the current image frame as output. The analysis results include: the unchanged area and the changed area relative to the previous frame of image, and the moving area in the changed area. and motion vector, as well as the content analysis result of the newly added area and the newly added area in the changed area, and the image area and the text area are distinguished by the content analysis result.

The above method is a fast algorithm for image analysis, which can be used to accurately analyze the changes of image frames, obtain good time-consuming performance, and reduce the delay in the data transmission process. The total time taken from operating or inputting commands on the front-end device to receiving the corresponding desktop image.

Unless specifically stated otherwise, terms such as processing, computing, operating, determining, displaying, etc. may refer to the acts and/or procedures of one or more processing or computing systems, or similar devices, which acts and/or procedures would be expressed as Data of physical (eg, electronic) quantities within a processing system's registers or memory is manipulated and converted into other data similarly represented as physical quantities within a processing system's memory, registers, or other such information storage, transmission, or display device. Information and signals may be represented using any of a variety of different technologies and methods. For example, data, instructions, commands, information, signals, bits, symbols and chips referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

It is understood that the specific order or hierarchy of steps in the processes disclosed are examples of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

In the foregoing Detailed Description, various features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This method of disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, present invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the Detailed Description, with each claim standing on its own as a separate preferred embodiment of this invention.

Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments herein may be implemented as electronic hardware, computer software, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, however, such implementation decisions should not be interpreted as a departure from the scope of the present disclosure.

The steps of a method or algorithm described in connection with the embodiments herein may be directly embodied in hardware, a software module executed by a processor, or a combination thereof. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium may reside in an ASIC. The ASIC may be located in the user terminal. Of course, the processor and the storage medium may also exist in the user terminal as discrete components.

For a software implementation, the techniques described in this application may be implemented in modules (eg, procedures, functions, etc.) that perform the functions described in this application. These software codes may be stored in a memory unit and executed by a processor. The memory unit may be implemented within the processor or external to the processor, in which case it is communicatively coupled to the processor via various means, as is known in the art.

The above description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods in order to describe the above embodiments, but one of ordinary skill in the art will recognize that further combinations and permutations of the various embodiments are possible. Accordingly, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, with respect to the term "comprising," as used in the specification or claims, the word is encompassed in a manner similar to the term "comprising," as if "comprising," were construed as a conjunction in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

Claims (14)

1. A cloud desktop image processing method is characterized by comprising the following steps:

dividing a current image into a plurality of image blocks;

determining an invariant area and a variant area of the current image relative to the previous frame of image based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image;

based on the hash value of the image block in the change area, performing motion search on the image block in the change area in the previous frame of image, determining a moving area of the current image relative to the previous frame of image and a newly increased area in the current image, and determining a motion vector of the image block in the moving area relative to the previous frame of image;

and determining the data required to be transmitted by the current image based on the unchanged area, the moving area and the newly added area of the current image.

2. The method of claim 1, wherein the dividing the current image into a plurality of image blocks comprises:

dividing a current image into a plurality of image blocks with equal size according to the preset size of the image blocks; and the size of the image block is determined according to the resolution and the configuration information of the current image.

3. The method of claim 1, wherein determining the invariant region and the variant region of the current image relative to the previous frame image based on the hash value of each tile in the current image and the hash value of each tile in the previous frame image comprises:

calculating the hash value of each image block in the current image;

performing hash value comparison on each image block in the current image and the image block at the same position in the previous frame of image;

determining an invariant region of the current image relative to the previous frame of image based on the image blocks with the same hash value;

and determining the change area of the current image relative to the previous frame image based on the image blocks with different hash values.

4. The method of claim 1, wherein performing a motion search for the image block in the changed region in the previous frame image based on the hash value of the image block in the changed region, determining a moving region of the current image relative to the previous frame image and a newly added region in the current image, and determining a motion vector of the image block in the moving region relative to the previous frame image comprises:

taking the image blocks in the change area as image blocks to be matched, searching in a previous frame of image for motion search aiming at each image block to be matched, and if a matched image block with the same hash value is searched in the previous frame of image, determining a motion vector of the image block to be matched relative to the previous frame of image according to the position of the image block to be matched in the current image and the position of the matched image block in the previous frame of image;

obtaining a moving area of the current image relative to the previous frame image based on the image block to be matched with the matching image block in the previous frame image; and the other areas except the moving area in the change area are used as newly added areas.

5. The method of claim 4, wherein obtaining a moving area of a current image relative to a previous frame image based on an image block to be matched having a matching image block in the previous frame image comprises:

for an image block to be matched with a matching image block in a previous frame image, if a plurality of adjacent image blocks have the same motion vector, the image block to be matched is used as the center to search for the image block with the same motion vector in an outward diffusion mode, and an image block group with the same motion vector is formed;

if the obtained image block group is unique, taking the obtained image block group as a moving area;

if the obtained image blocks are not unique, the image block with the largest area is used as the moving area.

6. The method of claim 1, further comprising: calculating the hash value of the image block by adopting the following fast hash algorithm:

a pixel coordinate (x, x + N-1) corresponding to each pixel coordinate y in the image is a pixel meta, wherein N represents the size of the image block N × N, and N is a positive integer;

calculating a hash value meta _ hash (x, x + N-1 y) of each meta according to pixel values of N pixels included in each meta;

hash calculation is carried out on Hash values of N successive meta with a pixel coordinate y as a start to obtain the Hash value of an image block with the pixel coordinate (x, y) as the start, wherein the Hash values of the N successive meta comprise meta _ Hash (x, x + N-1), 8230, meta _ Hash (x, x + N-1, y + N-1.

7. The method of any of claims 1-6, wherein determining the data that the current image requires to be transmitted to the front-end device comprises:

and taking the motion vector of the image block in the moving area relative to the image of the last frame and the content data of the image block in the non-moving area as data which needs to be transmitted to the front-end equipment by the current image.

8. The method of any of claims 1-6, wherein after determining the moved region of the current image relative to the previous image and the newly added region in the current image, further comprising:

performing color analysis on the data blocks in the newly added region, and judging the content data type of each data block according to the color complexity, wherein the content data type comprises at least one of characters and pictures; correspondingly, after determining the data that needs to be transmitted in the current image, the method further includes: selecting a compression algorithm according to the content data type of the image block, and compressing the content data by adopting the selected compression algorithm; and/or

And correcting the moving area, and adjusting the moving area to be a square or rectangular area.

9. A cloud desktop image processing method is characterized by comprising the following steps:

receiving data of a current image transmitted by a cloud server;

analyzing the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame of image from the received data;

acquiring content data of an image block in a current image unchanged area from a previous frame of image, and acquiring content data of an image block in a moving area from the previous frame of image according to a motion vector of the image block in the moving area relative to the previous frame of image;

and generating the current image based on the analyzed content data of the data blocks in the newly added area, the acquired content data of the image blocks in the moving area and the unchanged area.

10. The method as claimed in claim 9, wherein the obtaining content data of the image block in the moving area from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image comprises:

and determining the position of the image block in the moving area in the previous frame of image according to the position of the image block in the moving area in the current image and the motion vector relative to the previous frame of image, and acquiring content data of the corresponding image block according to the determined position.

11. A cloud server, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the cloud desktop image processing method of any of claims 1-8 when executing the program.

12. A front-end device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the cloud desktop image processing method of any of claims 9-10 when executing the program.

13. A cloud desktop image transmission system, comprising: the cloud server of claim 11 and the head-end apparatus of claim 12;

the cloud server is used for transmitting the determined transmission data to the front-end equipment;

the front-end equipment is used for receiving the data provided by the cloud server and generating a current image according to the received data and the data of the previous frame of image.

14. A computer storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by a processor, implement the cloud desktop image processing method of any one of claims 1-8 or the cloud desktop image processing method of any one of claims 9-10.

Images