CN115190303A

CN115190303A - Cloud desktop image processing method and system and related equipment

Info

Publication number: CN115190303A
Application number: CN202210556395.8A
Authority: CN
Inventors: 田巍; 闵洪波; 黄超
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-05-19
Filing date: 2022-05-19
Publication date: 2022-10-14

Abstract

The invention discloses a cloud desktop image processing method, a cloud desktop image processing system and related equipment. The method comprises the following steps: dividing a current image into a plurality of image blocks; determining an invariant region and a variant region of the current image relative to the previous frame of image based on the hash value of each image block in the current image and the previous frame of image; based on the hash value of the image block in the change area, performing motion search on the image block in the change area in the previous frame of image, determining a moving area and a newly added area of the current image relative to the previous frame of image, and determining a motion vector of the image block in the moving area relative to the previous frame of image; and determining the data required to be transmitted by the current image based on the unchanged area, the moving area and the newly added area. The types of the unchanged area, the changed area and the changed area in the image can be accurately analyzed, the transmitted data is determined based on the analysis result so as to reduce the data transmission amount, the data transmission speed is increased, and the requirements of real-time property and low time delay of cloud desktop image transmission are met.

Description

Cloud desktop image processing method and system and related equipment

Technical Field

The invention relates to the technical field of image processing, in particular to a cloud desktop image processing method, a cloud desktop image processing system and related equipment.

Background

The cloud desktop is a new mode that a cloud computing technology replaces a traditional computer, a computer host is virtualized in a cloud server, a lightweight client is mainly adopted for connecting a display and an input device in front-end equipment, the client is installed on the front-end equipment, and then the virtual computer host on the cloud server is accessed through a specific communication protocol to realize interactive operation, so that the experience effect consistent with that of the traditional computer host is achieved. The cloud application is an application in which the front-end equipment interacts with the cloud server, and an application program runs on the cloud server and is synchronized to the front-end equipment for display.

With the cloud desktop and the cloud application, a cloud desktop image on the cloud server virtual host needs to be compressed and encoded to be transmitted to a local client of the front-end device, so as to be displayed on the front-end device. In order to improve the image compression efficiency, the image content of each frame can be analyzed by utilizing the correlation between image frames, and then different compression processing modes are adopted for different types of image areas, so that the data transmission amount and the bandwidth requirement are reduced.

In the prior art, there are two methods for image analysis, one is to analyze image change and motion information output by an operating system screen, and the other is to analyze images by using an open-source computer vision framework (such as opencv).

Disclosure of Invention

The inventor of the application finds that the image analysis method in the prior art cannot meet the requirements of low time delay and real-time performance of cloud desktop and cloud application, wherein the analysis of image motion and change is not accurate by using an operating system screen-grabbing mode, the motion and change conditions of image frames cannot be accurately known, so that a large amount of redundancy is generated, the data volume to be transmitted is increased, the requirement of rapidly transmitting data cannot be met, and the data transmission time is still long; although the analysis accuracy is improved to a certain extent by using opencv for image analysis, the analysis process consumes too much time, and the requirements of low time delay and instantaneity of a cloud desktop cannot be met. In short, in the prior art, there is no good image analysis method, and information such as an invariant region, a change region, a motion condition, an image type, and the like between image frames cannot be accurately determined, that is, image change information cannot be analyzed with high precision and high accuracy, so that image regions of different types cannot be quickly and accurately divided for targeted compression processing, and requirements of a cloud desktop and cloud application in the aspect of transmission delay cannot be met.

In view of the above problems, the present invention is proposed to provide a cloud desktop image processing method, system and related device that overcome or at least partially solve the above problems.

The embodiment of the invention provides a cloud desktop image processing method, which comprises the following steps:

dividing a current image into a plurality of image blocks;

determining an invariant area and a variant area of the current image relative to the previous frame of image based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image;

based on the hash value of the image block in the change area, performing motion search on the image block in the change area in the previous frame of image, determining a moving area of the current image relative to the previous frame of image and a newly added area in the current image, and determining a motion vector of the image block in the moving area relative to the previous frame of image;

and determining the data required to be transmitted by the current image based on the unchanged area, the moving area and the newly added area of the current image.

In some optional embodiments, the dividing the current image into a plurality of image blocks includes:

dividing a current image into a plurality of image blocks with equal size according to the size of a preset image block; and the size of the image block is determined according to the resolution and configuration information of the current image.

In some optional embodiments, the determining the invariant region and the variant region of the current image relative to the previous frame image based on the hash value of each image block in the current image and the hash value of each image block in the previous frame image includes:

calculating the hash value of each image block in the current image;

performing hash value comparison on each image block in the current image and the image block at the same position in the previous frame of image respectively;

determining an invariant region of the current image relative to the previous frame of image based on the image blocks with the same hash value;

and determining the change area of the current image relative to the image of the previous frame based on the image blocks with different hash values.

In some optional embodiments, performing a motion search on the image block in the change area in the previous frame of image based on the hash value of the image block in the change area, determining a moving area of the current image relative to the previous frame of image and a newly added area in the current image, and determining a motion vector of the image block in the moving area relative to the previous frame of image includes:

taking the image blocks in the change area as image blocks to be matched, searching in a previous frame of image for motion search aiming at each image block to be matched, and if a matched image block with the same hash value is searched in the previous frame of image, determining a motion vector of the image block to be matched relative to the previous frame of image according to the position of the image block to be matched in the current image and the position of the matched image block in the previous frame of image;

obtaining a moving area of the current image relative to the previous frame image based on the image blocks to be matched with the matching image blocks in the previous frame image; and the other areas except the moving area in the change area are used as newly added areas.

In some optional embodiments, obtaining a moving area of the current image relative to the previous frame image based on the to-be-matched image block having the matching image block in the previous frame image includes:

for an image block to be matched with a matching image block in a previous frame image, if a plurality of adjacent image blocks have the same motion vector, the image block to be matched is used as the center to search for the image block with the same motion vector in an outward diffusion mode, and an image block group with the same motion vector is formed;

if the obtained image block group is unique, taking the obtained image block group as a moving area;

if the obtained image block group is not unique, the image block group with the largest area is taken as the moving area.

In some optional embodiments, the method further comprises: calculating the hash value of the image block by adopting the following fast hash algorithm:

a pixel coordinate (x, x + N-1) corresponding to each pixel coordinate y in the image is a pixel meta, where N denotes that the size of the image block is N × N, and N is a positive integer;

calculating a hash value meta _ hash (x, x + N-1 y) of each meta according to pixel values of N pixels included in each meta;

hash calculation is carried out on Hash values of N successive meta with a pixel coordinate y as a start to obtain the Hash value of an image block with the pixel coordinate (x, y) as the start, wherein the Hash values of the N successive meta comprise meta _ Hash (x, x + N-1), 8230, meta _ Hash (x, x + N-1, y + N-1.

In some optional embodiments, the determining that the current image needs to be transmitted to the front-end device includes:

and taking the motion vector of the image block in the moving area relative to the image of the last frame and the content data of the image block in the non-moving area as data which needs to be transmitted to the front-end equipment by the current image.

In some optional embodiments, after determining the moving region of the current image relative to the previous frame image and the newly added region in the current image, the method further includes:

performing color analysis on the data blocks in the newly added region, and judging the content data type of each data block according to the color complexity, wherein the content data type comprises at least one of characters and pictures; correspondingly, after determining the data that needs to be transmitted in the current image, the method further includes: selecting a compression algorithm according to the content data type of the image block, and compressing the content data by adopting the selected compression algorithm; and/or

And correcting the moving area, and adjusting the moving area to be a square or rectangular area.

The embodiment of the invention also provides another cloud desktop image processing method, which comprises the following steps:

receiving data of a current image transmitted by a cloud server;

analyzing the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the image of the previous frame from the received data;

acquiring content data of an image block in a current image unchanged area from a previous frame of image, and acquiring content data of an image block in a moving area from the previous frame of image according to a motion vector of the image block in the moving area relative to the previous frame of image;

and generating the current image based on the analyzed content data of the data blocks in the newly added area, the acquired content data of the image blocks in the moving area and the unchanged area.

In some optional embodiments, the obtaining content data of the image block in the moving region from the previous frame image according to the motion vector of the image block in the moving region relative to the previous frame image includes:

and determining the position of the image block in the moving area in the previous frame of image according to the position of the image block in the moving area in the current image and the motion vector relative to the previous frame of image, and acquiring content data of the corresponding image block according to the determined position.

An embodiment of the present invention further provides a cloud server, which includes: the cloud desktop image processing method comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the program to realize the cloud desktop image processing method.

An embodiment of the present invention further provides a front-end device, including: the cloud desktop image processing method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize the other cloud desktop image processing method.

The embodiment of the present invention further provides a cloud desktop image transmission system, including: the cloud server and the front-end device;

the cloud server is used for transmitting the determined transmission data to the front-end equipment;

the front-end equipment is used for receiving the data provided by the cloud server and generating a current image according to the received data and the data of the previous frame of image.

An embodiment of the present invention further provides a computer storage medium, where computer-executable instructions are stored in the computer storage medium, and when executed by a processor, the computer-executable instructions implement the foregoing cloud desktop image processing method or the foregoing another cloud desktop image processing method.

The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:

the method provided by the embodiment of the invention analyzes the image based on the hash value of each image block in the current image of the data to be transmitted and the hash value of each image block in the previous frame of image, determines an unchanged area and a changed area which are not changed in the current image relative to the previous frame of image, further analyzes the changed area, and determines a moving area which moves relative to the previous frame of image and a newly added area in the current image.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:

fig. 1 is a flowchart of a cloud-side desktop image processing method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating a method for processing a cloud-side desktop image according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an embodiment of a method for computing a Zhonghah value;

FIG. 4 is a second schematic diagram illustrating the calculation of the Hash value according to the first embodiment of the present invention;

fig. 5 is a flowchart of a cloud-side desktop image processing method according to a second embodiment of the present invention;

fig. 6 is a schematic structural diagram of a cloud desktop image processing apparatus according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of another cloud desktop image processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a cloud desktop image processing system according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

In order to solve the problem that in the prior art, in the transmission process of a cloud desktop image, different types of image areas cannot be quickly and accurately divided so as to be conveniently transmitted and processed in a targeted manner, embodiments of the present invention provide a cloud desktop image processing method, which can quickly and accurately analyze an invariant area, a moving area, and a newly added area in an image, and adopt different transmission processing modes for content data of the invariant area, the moving area, and the newly added area so as to meet the transmission requirements of low time delay and high real-time performance of a cloud desktop and a cloud application. The following is a detailed description by way of specific examples.

Example one

An embodiment of the present invention provides a cloud desktop image processing method, a flow of which is shown in fig. 1, and the method includes the following steps:

step S101: the current image is divided into a plurality of image blocks.

For each frame of image of data to be transmitted, dividing a current image into a plurality of image blocks with equal size according to a preset image block size; the size of the image block is determined according to the resolution and configuration information of the current image.

The cloud desktop image processing method provided by this embodiment is implemented according to the principle shown in fig. 2, and belongs to a step of preprocessing a current image, and corresponds to a block preprocessing (block pro-processing) step in fig. 2, where the block preprocessing refers to dividing an image into image blocks, and dividing each frame of image into image blocks, and dividing an entire frame of image into a plurality of image blocks (blocks) with equal size, where the size of a block can be adjusted according to resolution and configuration, and can also be set according to actual needs of a cloud desktop and cloud application. For example, the block size may be set to 16 × 16 pixels, 32 × 32 pixels, and so on, and 16 × 16 pixels are taken as an example in the following description.

Step S102: and determining an invariant area and a variant area of the current image relative to the previous frame of image based on the hash value of each image block in the current image and the hash value of each image block in the previous frame of image.

In the step, the hash value of each image block in the current image is calculated; performing hash value comparison on each image block in the current image and the image block at the same position in the previous frame of image respectively; determining an invariant region of the current image relative to the previous frame of image based on the image blocks with the same hash value; and determining the change area of the current image relative to the image of the previous frame based on the image blocks with different hash values. Namely, determining a part of image blocks with the same hash value as the image blocks at the same position in the previous frame of image in the current image to obtain an invariant region; and determining a part of image blocks of the current image, which have the same positions as the image blocks of the previous frame of image and have different hash values, to obtain a change area.

The hash value of each image block can be calculated by adopting a fast hash algorithm, for each frame of image, the hash value of each image block can be calculated after the image blocks are divided for later use, or the hash values of the image blocks in the current image and the previous frame of image which need to be compared can be calculated before comparison, generally, the hash values of the image blocks in the previous array of images are calculated before data of the previous frame of image is transmitted, so that when comparison is carried out, the hash value of each image block in the current image is calculated, and the hash value of the image block in the previous frame of image which is calculated before is obtained for comparison.

When the hash values of the image blocks in the current image are compared with the hash values of the image blocks at the same positions in the image block of the previous frame, whether the image blocks in the current image are changed or not is determined, if the hash values are the same, no change is determined, and if the hash values are different, the change is determined, so that the unchanged area and the changed area of the current image relative to the image of the previous frame are determined. The unchanged area comprises image blocks of which the current image is unchanged relative to the previous frame of image, and the changed area comprises image blocks of which the current image is changed relative to the previous frame of image.

Referring to a hash calculation (hash calculation) part and a changed area detection (dirty detection) part in fig. 2, hash value comparison is performed on each block in a current image and a block at the same position of a previous frame image, whether the block with the change is changed or not is determined, the changed block is marked as a changed (dirty) image block to form a changed area, the block without the change is marked as an unchanged (skip) image block to form an unchanged area, motion estimation (motion estimation) is performed on the dirty image block, and no processing is performed on the skip image block.

Step S103: and based on the hash value of the image block in the change area, performing motion search on the image block in the change area in the previous frame of image, determining a moving area of the current image relative to the previous frame of image and a newly added area in the current image, and determining a motion vector of the image block in the moving area relative to the previous frame of image.

Taking the image blocks in the change area as image blocks to be matched, searching in a previous frame of image for motion search aiming at each image block to be matched, and if a matched image block with the same hash value is searched in the previous frame of image, determining a motion vector of the image block to be matched relative to the previous frame of image according to the position of the image block to be matched in the current image and the position of the matched image block in the previous frame of image; obtaining a moving area of the current image relative to the previous frame image based on the image block to be matched with the matching image block in the previous frame image; and the other areas except the moving area in the change area are used as newly added areas.

When performing a motion search in the previous image frame, the search may be performed row by row, column by column, starting from the first row and the first column, until the last row and the last column position where the image block may be formed are searched. Taking image blocks of N pixels by N as an example, N is a positive integer, searching image blocks with the same hash value in the previous frame of image for each image block in the change area, starting from the first row and the first column, forming an image block with the same size as the image block to be matched by taking each position point (x, y) as a starting point, for example, the starting point is (x, y), the pixels included in each image block are (x, x + N-1 y, y + N-1), namely the image block of N matched by taking (x, y) as the starting point, matching the image block to be matched in the change area with each formed image block one by one, and the matched image block with the same hash value is the image block of the image block to be matched.

For an image block to be matched with a matching image block in a previous frame image, if a plurality of adjacent image blocks have the same motion vector, the image block to be matched is used as the center to search for the image block with the same motion vector in an outward diffusion mode, and an image block group with the same motion vector is formed; if the obtained image block group is unique, taking the obtained image block group as a moving area; if the obtained image block group is not unique, the image block group with the largest area is taken as the moving area.

Referring to motion estimation (motion estimation) in fig. 2, motion search is performed on a dirty region, for each dirty block of a current image frame, a block with the same hash value is matched in an image of a previous frame, a difference between positions of the two blocks is a Motion Vector (MV), if 4 blocks adjacent to the block have uniquely matched blocks and motion vectors are the same, diffusion search is performed by taking the block and the MV as seed blocks to find more blocks with the same MV, and the blocks with the same MV form an image block group.

Referring to the motion vector filter (MV filter) in fig. 2, for a current image, it is possible to obtain multiple image block groups with the same MV, but because the image changes in the cloud desktop and the cloud application, only one motion direction can exist each time, so MV filtering is required, and during filtering, the area of the image block group corresponding to each MV can be calculated, and the image block group with the largest area can be found, and its MV is used as the MV of the moving area in the current image, and all dirty blocks of the MV are marked as the image blocks of the moving area.

Optionally, the moving area is modified, and the moving area is adjusted to be a square or rectangular area. In order to facilitate processing on an application layer, a moving area formed by all dirty blocks is modified to be a regular shape, such as a rectangle or a square, the moving area only needs to transmit a motion vector, a client on front-end equipment can generate a current image according to a previous frame of image and the motion vector, and specific image content data does not need to be transmitted.

Step S104: and determining the data required to be transmitted by the current image based on the unchanged area, the moving area and the newly added area of the current image.

According to the determined invariant region, the determined mobile region and the determined newly added region of the current image, different transmission processing modes are adopted for processing, the content data of the image blocks in the invariant region can be not transmitted, only the motion vector data can be transmitted for the image blocks in the mobile region, and the content data needs to be transmitted for the image blocks in the newly added region, so that the motion vector of the image blocks in the mobile region relative to the previous frame of image and the content data of the image blocks in the non-mobile region are used as the data which needs to be transmitted to the front-end equipment by the current image based on the invariant region, the mobile region and the newly added region of the current image. See the output (out) section in fig. 2.

In some optional embodiments, after determining the moving region of the current image relative to the previous frame image and the newly added region in the current image, the method further includes: performing color analysis on the data blocks in the newly added region, and judging the content data type of each data block according to the color complexity, wherein the content data type comprises at least one of characters and pictures; see color analysis (color analysis) section in fig. 2; correspondingly, after determining the data that needs to be transmitted in the current image, the method further includes: and selecting a compression algorithm according to the content data type of the image block, and compressing the content data by adopting the selected compression algorithm.

After the cloud server determines the data to be transmitted, the data can be coded when the data are transmitted, the coding refers to a process of converting an original image into a bit video stream through a compression technology, and different compression algorithms are adopted for different areas such as a character area and a picture area, so that the compression efficiency can be further improved.

In the method, the hash value of each image block can be calculated by adopting a fast hash algorithm, the calculation amount is very large because each frame of image needs to be subjected to hash calculation, and particularly in the motion searching step, the searching and matching are required row by row, namely the hash value calculation is carried out on each pixel block (x, x + N-1 y, y + N-1) in the previous frame of image, and the calculation amount is huge, so that the fast hash algorithm provided by the invention can reduce the whole calculation time and meet the real-time requirement.

In the motion search, for an image block to be matched in a current image, the matching needs to be performed row by row and column by column in the previous frame image, so that a plurality of hash values need to be calculated for each image block that an image can be divided into, or each pixel (x, y) in the previous frame image is used as a starting position, and the hash values need to be calculated for all image blocks that can be formed.

Referring to fig. 3, taking an image block size of 16 × 16 pixels and a vertical direction of the image block as an example, during motion search, a hash value (MB hash) is calculated for each row in each image block, that is, if a pixel range of the image block is (x, x +15 y, y + 15), 16 MB hashes need to be calculated from a start position (x, y) of the image block.

MB_hash(x，x+15；y，y+15)＝HASH{rgb(x，y)，rgb(x+1，y)，…，rgb(x+15，y)；rgb(x，y+1)，rgb(x+1，y+1)，…，rgb(x+15，y+1)；……；rgb(x，y+15)，rgb(x+1，y+15)，…，rgb(x+15，y+15)}

MB_hash(x，x+15，y+1，y+16)＝HASH{rgb(x，y)，rgb(x+1，y)，…，rgb(x+15，y)；rgb(x，y+1)，rgb(x+1，y+1)，…，rgb(x+15，y+1)；……；rgb(x，y+15)，rgb(x+1，y+15)，…，rgb(x+15，y+15)}

MB_hash(x，x+15，y+2，y+17)＝HASH{rgb(x，y)，rgb(x+1，y)，…，rgb(x+15，y)；rgb(x，y+1)，rgb(x+1，y+1)，…，rgb(x+15，y+1)；……；rgb(x，y+15)，rgb(x+1，y+15)，…，rgb(x+15，y+15)}

……

MB_hash(x，x+15，y+15，y+30)＝HASH{rgb(x，y+15)，rgb(x+1，y+15)，…，rgb(x+15，y+15)，rgb(x，y+16)，rgb(x+1，y+16)，…，rgb(x+15，y+16)；……；rgb(x，y+30)，rgb(x+1，y+30)，…，rgb(x+15，y+30)}

The horizontal direction is similar, and each image block in the image needs to be calculated, so the method has huge calculation amount and high time consumption. The fast hash algorithm can reduce the calculation amount and greatly improve the calculation speed.

In the above method, for each image block pre-divided in each frame of image and each image block possibly used for matching in the motion search process, the following fast hash algorithm may be used to calculate the hash value of the image block:

a pixel coordinate (x, x + N-1) corresponding to each pixel coordinate y in the image is a pixel element meta, wherein N represents the size of the image block N × N; n is a positive integer, and a hash value meta _ hash (x, x + N-1; hash calculation is carried out on Hash values of N consecutive meta with a pixel coordinate y as a start to obtain a Hash value of an image block with a pixel coordinate (x, y) as a start, wherein the Hash values of the N consecutive meta comprise meta _ Hash (x, x + N-1 y), meta _ Hash (x, x + N-1 y + 1), \ 8230; \8230, meta _ Hash (x, x + N-1 y, y N-1. That is, for each image block, a hash value may be calculated by using each row of pixels as a pixel element, and then the hash values of the rows are subjected to hash calculation to obtain the hash value of the graphic block.

Referring to the schematic principle of the fast hash algorithm shown in fig. 4, taking an image block size of 16 × 16 pixels as an example, first defining (x, x + 15) pixels corresponding to each y coordinate as a meta, calculating a meta _ hash for each meta, and then performing hash calculation on 16 vertically consecutive meta _ hashes to obtain an MB _ hash.

meta_hash(x，x+15；y)＝HASH{rgb(x，y)，rgb(x+1，y)，……，rgb(x+15，y)}

meta_hash(x，x+15；y+1)＝HASH{rgb(x，y+1)，rgb(x+1，y+1)，……，rgb(x+15，y+1)}

meta_hash(x，x+15，y+2)＝HASH{rgb(x，y+2)，rgb(x+1，y+2)，……，rgb(x+15，y+2)}

……

meta_hash(x，x+15；y+30)＝HASH{rgb(x，y+30)，rgb(x+1，y+30)，……，rgb(x+15，y+30)}

Then calculate MB _ hash:

MB_hash(x，x+15，y，y+15)＝HASH{meta_hash(x，x+15，y)，meta_hash(x，x+15，y+1)……，meta_hash(x，x+15，y+15)}

MB_hash(x，x+15，y+1，y+16)＝HASH{meta_hash(x，x+15；y+1)，meta_hash(x，x+15；y+2)……，meta_hash(x，x+15；y+16)}

……

MB_hash(x，x+15，y+15，y+30)＝HASH{meta_hash(x，x+15；y+15)，meta_hash(x，x+15；y+16)……，meta_hash(x，x+15；y+30)}

the fast hash algorithm can adopt SSE/AVX instruction accelerated hash algorithm and multi-thread parallel calculation for calculating each meta hash, thereby further greatly improving the calculation performance. SSE/AVX is a hardware acceleration instruction provided by a computer system.

The method of the embodiment can be implemented in a cloud server, and the image data is transmitted according to the analysis result after the image of the data to be transmitted is analyzed. The method comprises the steps of analyzing an image based on a hash value of an image block in a current image of data to be transmitted, determining an unchanged area which does not change and a changed area which changes in the current image relative to a previous frame of image, further analyzing the changed area, and determining a moving area which moves relative to the previous frame of image and a newly added area in the current image. The data processing speed and efficiency can be further improved and the time delay can be reduced by means of a fast hash algorithm, color analysis, different compression algorithms and the like.

Example two

The second embodiment of the present invention provides a cloud desktop image processing method, which is implemented on a front-end device, and is matched with the method provided in the first embodiment to implement receiving and processing of transmitted image data, so that a cloud desktop, a cloud application, and the like on a cloud device can quickly and correctly display a cloud desktop image, and a flow of the method is as shown in fig. 2, and the method includes the following steps:

step S201: and receiving the data of the current image transmitted by the cloud server.

The data transmitted by the cloud server mainly includes content data of a data block in a newly added area of a current image and a motion vector of an image block in a mobile area relative to a previous frame of image, optionally, an invariant area may be indicated, that is, which blocks are image blocks of the invariant area, certainly, the invariant area may not be indicated, and an image block which does not transmit any data is defined as an image block of the invariant area.

Step S202: and analyzing the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the image of the previous frame from the received data.

The cloud server can encode data when transmitting the data, and the front-end equipment can decode the received data after receiving the data. Encoding refers to the process of converting an original image into a bit video stream by a compression technique, and decoding is the reverse process.

Step S203: and acquiring content data of the image block in the moving area from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image.

In the step, the position of the image block in the moving area in the previous frame of image is determined according to the position of the image block in the moving area in the current image and the motion vector of the image block relative to the previous frame of image, and content data of the corresponding image block is obtained according to the determined position. That is, for the moving area, the position of the moving area in the previous frame image can be obtained according to the motion vector and the position of the image block in the moving area in the current image, and the content data of the corresponding position in the previous frame image can be obtained.

Step S204: and acquiring content data of the image blocks in the unchanged area of the current image from the previous frame of image.

For the invariant region, the position of the image block in the current image, that is, the position of the image block in the previous frame image, is determined from the content data of the corresponding position in the previous frame image.

The steps of step S203 and step S204 are not sequential, and may be executed simultaneously.

Step S205: and generating the current image based on the analyzed content data of the data blocks in the newly added area, the acquired content data of the image blocks in the moving area and the unchanged area.

And rendering the content data of the image block based on the position of the image block to generate the current image. Rendering refers to the process of graphics processing to generate and draw an image.

Based on the same inventive concept, an embodiment of the present invention further provides a cloud desktop image processing apparatus, which may be disposed in a cloud server, and a structure of the apparatus is as shown in fig. 6, where the apparatus includes:

a dividing module 11, configured to divide a current image into a plurality of image blocks;

the calculation and comparison module 12 is configured to determine an invariant region and a variant region of the current image relative to the previous frame of image based on the hash value of the image block in the current image;

the motion estimation module 13 is configured to perform motion search on the image block in the change region in the previous frame of image based on the hash value of each image block in the change region and the hash value of each image block in the previous frame of image, determine a moving region of the current image relative to the previous frame of image and a newly added region in the current image, and determine a motion vector of the image block in the moving region relative to the previous frame of image;

and the data processing module 14 is used for determining the data required to be transmitted by the current image based on the unchanged area, the moving area and the newly added area of the current image.

Based on the same inventive concept, an embodiment of the present invention further provides another cloud desktop image processing apparatus, which may be disposed in a front-end device, and a structure of the apparatus is as shown in fig. 7, where the apparatus includes:

the data receiving module 21 is configured to receive data of a current image transmitted by the cloud server;

the data analysis module 22 is configured to analyze, from the received data, content data of a data block in a newly added area of the current image and a motion vector of an image block in a moving area relative to an image of a previous frame;

the data acquisition module 23 is configured to acquire content data of an image block in an unchanged area of a current image from a previous frame of image, and acquire content data of an image block in a moving area from the previous frame of image according to a motion vector of the image block in the moving area relative to the previous frame of image;

and the image generating module 24 is configured to generate a current image based on the analyzed content data of the data block in the newly added region, the acquired content data of the image block in the moving region and the unchanged region.

Based on the same inventive concept, the embodiment of the invention also provides a cloud desktop image transmission system, which comprises a cloud server 1 and a front-end device 2. The number of the cloud servers 1 and the number of the front-end devices 2 can be multiple, and the cloud servers can adopt a server cluster and allow a large number of front-end devices to access.

In some optional embodiments, the cloud server 1 may include: the cloud desktop image processing method comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein when the processor executes the program, the cloud desktop image processing method in the figure 1 is realized.

In some optional embodiments, the front-end device 2 may include: the cloud desktop image processing method comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein when the processor executes the program, the cloud desktop image processing method in the figure 5 is realized.

Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, where computer-executable instructions are stored in the computer storage medium, and when executed by a processor, the computer-executable instructions implement the cloud desktop image processing method shown in fig. 1 or the cloud desktop image processing method shown in fig. 5.

With respect to the apparatus and system in the above embodiments, the specific manner in which each module performs the operation has been described in detail in the embodiments related to the method, and will not be described in detail here.

According to the method, the system and the device, each frame of image is divided into blocks, image analysis is carried out on the basis of the blocks, and a hash value is calculated for each block by using a fast hash algorithm. Then, by comparing the hash to find the changed area and the unchanged area, and further analyzing the changed area, the motion vector of the motion area is searched. And analyzing the content of the non-moving change area (namely, the newly added area) to distinguish the picture area from the character area. The image change information with high accuracy and high precision can be obtained by analyzing based on the blocks, the size of the blocks can be flexibly configured, the image change condition, the motion condition, the content information and the like can be obtained by one-time operation, and the operation resources are saved. In the process, each frame of image captured by the cloud desktop/cloud application is used as input, an analysis result of the current image frame is used as output, and the analysis result comprises the following steps: and distinguishing the picture area and the character area according to the content analysis result relative to the unchanged area and the changed area of the previous frame of image, the moving area and the motion vector in the changed area, and the content analysis result of the newly added area and the newly added area in the changed area.

The method is a rapid algorithm for image analysis, can accurately analyze the change of the image frame by utilizing the algorithm, obtains good time-consuming performance, and reduces delay in the data transmission process, wherein the delay refers to the time when a user operates or inputs a command on front-end equipment in a cloud desktop or cloud application system and receives the total consumption of the corresponding desktop image.

Unless specifically stated otherwise, terms such as processing, computing, calculating, determining, displaying, or the like, may refer to an action and/or process of one or more processing or computing systems or similar devices that manipulates and transforms data represented as physical (e.g., electronic) quantities within the processing system's registers and memories into other data similarly represented as physical quantities within the processing system's memories, registers or other such information storage, transmission or display devices. Information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

It should be understood that the specific order or hierarchy of steps in the processes disclosed is an example of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

In the foregoing detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments of the subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Of course, the processor and the storage medium may reside as discrete components in a user terminal.

For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in memory units and executed by processors. The memory unit may be implemented within the processor or external to the processor, in which case it can be communicatively coupled to the processor via various means as is known in the art.

What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the aforementioned embodiments, but one of ordinary skill in the art may recognize that many further combinations and permutations of various embodiments are possible. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word in a claim. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or".

Claims

1. A cloud desktop image processing method is characterized by comprising the following steps:

dividing a current image into a plurality of image blocks;

based on the hash value of the image block in the change area, performing motion search on the image block in the change area in the previous frame of image, determining a moving area of the current image relative to the previous frame of image and a newly increased area in the current image, and determining a motion vector of the image block in the moving area relative to the previous frame of image;

2. The method of claim 1, wherein the dividing the current image into a plurality of image blocks comprises:

dividing a current image into a plurality of image blocks with equal size according to the preset size of the image blocks; and the size of the image block is determined according to the resolution and the configuration information of the current image.

3. The method of claim 1, wherein determining the invariant region and the variant region of the current image relative to the previous frame image based on the hash value of each tile in the current image and the hash value of each tile in the previous frame image comprises:

calculating the hash value of each image block in the current image;

performing hash value comparison on each image block in the current image and the image block at the same position in the previous frame of image;

and determining the change area of the current image relative to the previous frame image based on the image blocks with different hash values.

4. The method of claim 1, wherein performing a motion search for the image block in the changed region in the previous frame image based on the hash value of the image block in the changed region, determining a moving region of the current image relative to the previous frame image and a newly added region in the current image, and determining a motion vector of the image block in the moving region relative to the previous frame image comprises:

obtaining a moving area of the current image relative to the previous frame image based on the image block to be matched with the matching image block in the previous frame image; and the other areas except the moving area in the change area are used as newly added areas.

5. The method of claim 4, wherein obtaining a moving area of a current image relative to a previous frame image based on an image block to be matched having a matching image block in the previous frame image comprises:

if the obtained image blocks are not unique, the image block with the largest area is used as the moving area.

6. The method of claim 1, further comprising: calculating the hash value of the image block by adopting the following fast hash algorithm:

a pixel coordinate (x, x + N-1) corresponding to each pixel coordinate y in the image is a pixel meta, wherein N represents the size of the image block N × N, and N is a positive integer;

7. The method of any of claims 1-6, wherein determining the data that the current image requires to be transmitted to the front-end device comprises:

8. The method of any of claims 1-6, wherein after determining the moved region of the current image relative to the previous image and the newly added region in the current image, further comprising:

9. A cloud desktop image processing method is characterized by comprising the following steps:

receiving data of a current image transmitted by a cloud server;

analyzing the content data of the data block in the newly added area of the current image and the motion vector of the image block in the moving area relative to the previous frame of image from the received data;

10. The method as claimed in claim 9, wherein the obtaining content data of the image block in the moving area from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image comprises:

11. A cloud server, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the cloud desktop image processing method of any of claims 1-8 when executing the program.

12. A front-end device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the cloud desktop image processing method of any of claims 9-10 when executing the program.

13. A cloud desktop image transmission system, comprising: the cloud server of claim 11 and the head-end apparatus of claim 12;

14. A computer storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by a processor, implement the cloud desktop image processing method of any one of claims 1-8 or the cloud desktop image processing method of any one of claims 9-10.