CN104735449A - Image transmission method and system based on rectangular segmentation and interlaced scanning - Google Patents

Abstract

The present invention discloses an image transmission method and system based on rectangular division and row-to-column scanning. Firstly, two adjacent frames of images are compared to find out all areas where the images change, and then a non-overlapping rectangular area with the smallest area is obtained according to the coordinates of the changed pixels. set, only the image data included in the rectangular area set is sent each time, so as to reduce the transmission data of each frame, and achieve the purpose of effectively reducing the amount of transmission data. The present invention dynamically divides the screen into different numbers of rectangular blocks according to the number of image change areas in each frame, which overcomes the problems of difficulty in determining the number of screen blocks and poor adaptability of the fixed block image transmission algorithm; divides the screen according to each Each change range of the frame image is divided into rectangular blocks of different sizes, which overcomes the problem that the fixed block image transmission algorithm cannot effectively reduce the amount of transmitted data when the screen image change area is just at the critical point of multiple rectangular blocks.

Description

An image transmission method and system based on rectangular segmentation and scanning

technical field

The invention belongs to the technical field of computer network communication, and in particular relates to an image transmission method and system based on rectangular division and row scanning.

Background technique

With the development of network technology, remote computer teaching, remote network monitoring, video conferencing and other projects have been widely used, and in these applications, real-time transmission of computer screen bitmaps is required. Therefore, in order to provide users with good screen image transmission services without affecting the services provided by other applications, the real-time screen image transmission system should meet the following conditions: low CPU usage, low bandwidth occupancy, and try to avoid delay, jitter and The occurrence of Huaping phenomenon, etc. In order to meet this condition, two methods are usually adopted: one is to increase the bandwidth of the network; the other is to reduce the amount of transmitted data. Since the network bandwidth is largely limited by the network hardware, the key to solving the problem is to reduce the amount of data transmission by optimizing the compression and transmission algorithms.

Computer screen images do not change all the time, most of the time only part of the images are changing, if only the changed part of the images are transmitted, the amount of screen data transmission will be greatly reduced. Patent 201210264229.7 divides the current frame of the desktop image into blocks, and constructs a hierarchical index, detects the changed image block and the unchanged image block of the current frame relative to the image block cache according to the layered index, and compresses the changed image block and the unchanged image block The tags in the image block cache are packaged together into the final transmission data; patent 201310318030.2 first divides the image into multiple areas of X columns and Y rows, and compares the content of each area with the same area of the previous frame image, if this If the content of an area changes, fill in the header information identifying the area information for the data in the area, and then compress and transmit; patent 201210544289.4 recognizes the format type of the source data to be transmitted before the encoding process, and targets different format types. Source data chooses a different compression algorithm.

At present, the image transmission algorithm commonly used in the screen image transmission system is the fixed block image transmission algorithm. This algorithm divides the screen into blocks based on rectangles, and numbers each block. The size and number of blocks of each block are Fixed, then save the data of the two adjacent bitmaps, and compare them according to the corresponding numbered blocks, if the image changes, compress and send the image in the current block. The methods for judging whether the screens of the two frames before and after have changed include the direct comparison method and the CRC comparison method. The direct comparison method compares the memory data of the two frame screens one by one byte by byte; the CRC comparison method calculates the CRC value of the image data, and judges whether the image has changed by comparing the CRC values of the two frames of images. Since the fixed-block image transmission algorithm only sends the image data in the changing block each time, it can reduce the amount of data transmission.

Fixed block image transmission algorithm The number of screen blocks is difficult to determine. If the number is too large, the sum of the block processing time will exceed the transmission time of the entire screen data. Although the network bandwidth is small, the real-time performance may decrease; If the number is too small, the network bandwidth occupied by the full-screen processing will not drop much, and the effect of speed improvement will not be obvious. There is a defect that reducing the amount of transmitted data depends on the number of image blocks, and the number of blocks is difficult to set; Block image transmission algorithm The number of screen blocks is closely related to the resolution of the screen. If the resolution changes, the algorithm must adjust the number of screen blocks to ensure that the network bandwidth is small, and its adaptability is poor; fixed block image transmission algorithm The size and number of screen blocks are fixed. When the screen image change area is just at the critical point of multiple rectangular blocks, there will be too many blocks of image changes, which cannot effectively reduce the amount of transmitted data.

Contents of the invention

The object of the present invention is to provide an image transmission method and system based on rectangular partitioning and scanning, aiming to solve the problem of difficulty in determining the number of screen blocks and poor adaptability in the fixed block image transmission algorithm. A rectangular block critical point cannot effectively reduce the problem of the amount of transmitted data.

The present invention is achieved in this way, an image transmission method based on rectangular division and inter-column scanning, the image transmission method based on rectangular division and inter-column scanning first compares two adjacent frames of images, finds out the areas where all images change, and then according to the change The coordinates of the pixel points get the set of non-overlapping rectangular areas with the smallest area; only the image data and corresponding coordinate information contained in the set of rectangular areas are sent each time;

According to the coordinates of the pixel points, the variable rectangular area is obtained, and formula (1) and formula (2) are formulas for judging the range of the rectangular R according to the variable pixel points;

R _l ≤ P _x AND R _t = P _yi (1)

R _r ≥ P _x AND R _b ≥ P _y (2)

Among them, R _l and R _t represent the abscissa and ordinate of the upper left corner of the rectangle, R _r and R _b represent the abscissa and ordinate of the lower right corner of the rectangle, P _x and P _y represent the abscissa and ordinate of the changed pixel, P _y0 represents the ordinate of the first changed pixel point; calculate the range of the changed rectangular area according to formula (1) and formula (2); first save the data of the two adjacent bitmaps, and judge the two frame screens before and after Whether the value of the corresponding pixel changes; when the changed sampling point is detected for the first time, the coordinates (P _X0 , P _Y0 ) of the changed sampling point will be recorded as the coordinates of the upper left corner of the changed rectangular area (R _l , R _t ), and mark no change in the line as false; continue to compare, when a different sampling point is detected again, first mark the line without change as false, and then the abscissa P _x of the sampling point is the same as the abscissa R of the upper left corner of the rectangle _l compare and take the minimum value, and the coordinates (R _r , R _b ) of the lower right corner of the rectangle will be compared with the point coordinates (P _x , P _y ) and take the maximum value; that is:

R _l =min(P _xi ,R _l )(i>1) R _t =P _yi (i=1)

R _r ＝max(R _xi ,R _r )(i>1) R _b ＝max(R _yi ,R _b )(i>1)

When it is detected that the values of the sampling points in a certain row are all the same, a changed rectangular area block is obtained.

Further, in a scanning area, the rectangular segmentation algorithm for the changed area adopts the direct comparison method every other column to judge whether the pixels corresponding to the two frames of screen images in the front and rear image buffers have changed, so as to find out the changed rectangular area; from top to bottom, from Based on the principle of left to right, the image transmission method based on the rectangular segmentation alternate scanning finds out all the changed areas of the subsequent frame image relative to the previous frame image and obtains the smallest non-overlapping rectangular area set based on the rectangular segmentation algorithm.

Further, use the method of direct column-by-column comparison to determine whether the pixels corresponding to the two frames of screen images have changed. First, extract the pixels corresponding to the two frames of images from left to right at intervals of N columns in row units as sampling points, and compare the values of the corresponding pixels. Whether they are the same; according to different application scenarios and bandwidth requirements, the number of interval columns N can be adjusted. The smaller the value of N, the longer the detection time required by the direct comparison method between intervals.

Further, the specific method of performing a rectangular segmentation algorithm on the changing area in a scanning area is as follows:

Step 1, the image sending end first obtains the resolution of the screen, and obtains the range 0-C of the column scan and the range 0-R of the row scan;

Step 2, the sending end saves the data in the current frame image storage area to the previous frame image buffer; intercepts the current screen bitmap data and saves it in the current frame image buffer;

Step 3, the sender first initializes the coordinates of the upper left corner and the lower right corner of the changing rectangular area to (0, 0), the coordinates of the starting point of the next scan are (0, 0), the flag of no change in the row is true, and the range and row of the column scan are updated scan range;

Step 4, judge whether it is within the line scanning range, if not, jump to step 10;

Step 5, judge whether it is within the column scanning range, if not, jump to step 8; within the column scanning range, use the method of direct comparison between columns to detect the current sampling point; if the values are different, first set the row no change flag to false, Then judge whether it is the first change sampling point detected. The coordinates of the sampling point are used as the coordinates of the upper left corner of the change rectangular area, not the first change sampling point. The coordinates of the lower right corner of the rectangle are compared with the coordinates of this point and taken The maximum value is used as the coordinate of the lower right corner of the new rectangle, and then it is judged whether the sampling point is the first changed sampling point of the line, and the vertical coordinate of the sampling point is compared with the vertical coordinate of the upper left corner of the rectangle and the minimum value is used to update the change Coordinates of the upper left corner of the rectangular area; if the values are the same, it is necessary to judge whether the row no change flag is false, if it is false, record the coordinates as the starting point of the next scan, detect the last column of sampling points, and use the coordinates of the last column of sampling points as the next time The starting point of scanning, skip to step 7;

Step 6, move the column coordinates to the right by N columns, and jump to step 5 to detect the next sampling point;

Step 7: After the detection of this line is completed, compare the coordinates of the next scanning start point of this line with the coordinates of the next scanning start point recorded in the previous line, and take the maximum value as the new starting point coordinate of the next scan, add 1 to the line number, and jump to Step 4: Detect from the next line from the beginning to the right;

Step 8, judge whether the row no change flag is true and the coordinates of the upper left corner of the changed rectangular area are not (0, 0), if it is not true, add 1 to the row number, and jump to step 4; if it is true, it means that the entire row has no different pixels point, a changed rectangular area block is obtained; the ordinate of the upper left corner of the obtained changed rectangular area block is moved to the left by N columns, and the ordinate of the lower right corner is moved to the right by N columns to contain image boundary information;

Step 9: Record the coordinates of the detected change rectangular area and the corresponding starting point coordinates of the next scan, determine whether the range of the current column scan is 0-C and whether the range of the row scan is 0-R, and if yes, set a flag to indicate the currently detected change The rectangular area mark is detected for the first time, then add 1 to the line number and jump to step 4 to detect the next changed rectangular area block from the next line; until the detection exceeds the range of line scanning;

Step ten, after this detection is completed, process all the starting points of the next scan in this detection, and calculate the set of the next scanning range; first check whether the ordinate of the first starting point of the next scan detected this time is It is smaller than the ordinate of the last column of sampling points. No, the detection of this area is completed, and the ordinate of the next scanning starting point is detected; The ordinate of the starting point coordinates of the next scan related to the current changing rectangular area is the ordinate, and a coordinate of the upper left corner of the next scanning range is generated; the abscissa of the lower right corner of the changing rectangular area detected for the first time is taken as the abscissa, and the screen The maximum number of columns C is the vertical coordinate to generate the coordinates of the lower right corner of the next scanning range; then process the second next scanning starting point until all the next scanning starting points in this detection are processed;

Step 11: Detect all scan areas in the next scan range set, firstly generate the range of row scan and column scan based on the width and height of the first scan area in the next scan range set, repeat steps 3 to 10 to detect The changed rectangular area block in the first scanning area, and then process the second scanning area until all scanning areas in the next scanning area set are detected;

Step 12, repeating step 10 to step 11, to obtain the changing rectangular area block of the next scanning range, until the vertical coordinates of all the starting points of the next scanning are greater than or equal to the vertical coordinates of the last column of sampling points, and the detection of the entire screen is completed;

Step 13, obtain the set of non-overlapping rectangular areas with the smallest area of the frame image relative to the previous frame image change, check the rectangular areas in the set, the vertical coordinates of the upper left corner and the lower right corner of the two rectangles are the same , and the abscissa of the lower right corner of one rectangle is adjacent to the abscissa of the upper left corner of another rectangle, merged into one rectangle, and then compress and send the image data and corresponding coordinates contained in the set of rectangular areas to the client;

Step 14, the image receiving end decompresses the received data and integrates the received data into the previous frame image based on the image data and corresponding coordinates of each rectangular area and displays it;

Step 15: Repeat steps 2 to 14 every T seconds. The interval T can be adjusted according to different application scenarios and bandwidth requirements.

Another object of the present invention is to provide a kind of image transmission system based on the rectangular division every other column scanning, this image transmission system based on the rectangular division every other column scanning comprises a sending end and a receiving end:

The sender includes:

The screen image acquisition module is used for image acquisition of the screen at the sending end;

Rectangular segmentation and row scanning module, based on the rectangular segmentation and row scanning algorithm, compares the current frame image with the cached previous frame image, finds out the areas where all images change, and then obtains the smallest non-overlapping rectangle according to the coordinates of the changed pixels collection of regions;

The compression sending module compresses the image data contained in the collection of rectangular areas, and then sends the compressed image data and corresponding coordinates to the client through the network;

Receiver includes:

The receiving module receives the changed image data sent by the sending end;

The decompression module decompresses the received data to obtain the image data and corresponding coordinates contained in the set of rectangular areas;

The update display module updates and displays the cached corresponding area of the previous frame image based on the image data of each rectangular area and the corresponding coordinates.

In the image transmission method and system based on rectangular division and alternate column scanning provided by the present invention, firstly compare two adjacent frames of images to find out all areas where the images change, and then obtain a set of non-overlapping rectangular areas with the smallest area according to the coordinates of the changed pixels , each time only the image data contained in the set of rectangular areas is sent, so as to reduce the transmission data of each frame, and achieve the purpose of effectively reducing the amount of transmission data. The present invention dynamically divides the screen into different numbers of rectangular blocks according to the number of image change areas in each frame, and overcomes the problems of difficulty in determining the number of screen blocks and poor adaptability of the fixed block image transmission algorithm; Each change range of the frame image is divided into rectangular blocks of different sizes, which overcomes the problem that the fixed block image transmission algorithm cannot effectively reduce the amount of transmitted data when the screen image change area is just at the critical point of multiple rectangular blocks. Compared with the image transmission system adopting the fixed-block interlaced scanning algorithm, the image transmission system adopting this method can more effectively reduce the CPU utilization rate, reduce the bandwidth occupation rate, and improve the performance of image transmission.

Description of drawings

FIG. 1 is a flow chart of an image transmission method based on rectangular division and row-by-column scanning provided by an embodiment of the present invention;

Fig. 2 is a schematic diagram of scanning provided by an embodiment of the present invention;

FIG. 3 is a flow chart for detecting a rectangular area provided by an embodiment of the present invention;

Fig. 4 is a collection flow chart of detecting changing rectangles in a scanning area provided by an embodiment of the present invention;

Fig. 5 is a collection flowchart for calculating the next scan range provided by an embodiment of the present invention;

Fig. 6 is a flow chart of detecting changing rectangles in a set of scanning ranges provided by an embodiment of the present invention;

Fig. 7 is a schematic diagram of image changes provided by an embodiment of the present invention;

Fig. 8 is a schematic diagram of the structure of an image transmission system based on rectangular division and alternate column scanning provided by an embodiment of the present invention;

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The application principle of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

As shown in FIG. 1 , the image transmission method based on rectangular segmentation and scanning every other row according to the embodiment of the present invention includes the following steps:

S101: first compare two adjacent frames of images, and find out all areas where the images change;

S102: Then obtain a set of non-overlapping rectangular areas with the smallest area according to the coordinates of the changed pixels, and only send the image data contained in the set of rectangular areas each time.

Concrete steps of the embodiment of the present invention:

In the alternate column scanning algorithm based on rectangular segmentation in the embodiment of the present invention, at first, two adjacent frames of images are compared to find out the regions where all images change, and then the set of non-overlapping rectangular regions with the smallest area is obtained according to the coordinates of the changed pixels; the key point of the algorithm is How to obtain the changing rectangular area according to the coordinates of the pixel points; as shown; the specific principle is described below based on Figure 2;

R _l ≤ P _x AND R _t = P _yi (1)

R _r ≥ P _x AND R _b ≥ P _y (2)

Formulas (1) and (2) are formulas for judging the range of the rectangle R based on changing pixels; where R _l and R _t represent the abscissa and ordinate of the upper left corner of the rectangle, R _r and R _b represent the abscissa of the lower right corner of the rectangle Coordinates and ordinates, P _x and P _y represent the abscissa and ordinate of the changed pixel point, P _y0 represents the ordinate of the first changed pixel point; according to the habit of 2D graphics, the X axis increases horizontally to the right, and the Y axis Increment horizontally downwards; according to formula (1) and formula (2), the range of the changing rectangular area can be obtained; the algorithm first saves the data of the two adjacent bitmaps, and judges the corresponding two frame screens Whether the value of the pixel changes; when the changed sampling point is detected for the first time, the coordinates (P _X0 , P _Y0 ) of the changed sampling point will be recorded as the coordinates (R _l , R _t ) of the upper left corner of the changed rectangular area , and mark no change in the line as false; continue the comparison, when a different sampling point is detected again, first mark the line without change as false, and then the abscissa P _x of the sampling point is the same as the abscissa R _l of the upper left corner of the rectangle Compare and take the minimum value, and at the same time, the coordinates (R _r , R _b ) of the lower right corner of the rectangle will be compared with the coordinates (P _x , P _y ) of the point and take the maximum value; that is:

R _l =min(P _xi ,R _l )(i>1) R _t =P _yi (i=1)

R _r ＝max(R _xi ,R _r )(i>1) R _b ＝max(R _yi ,R _b )(i>1)

When it is detected that the values of the sampling points in a certain row are all the same, a changed rectangular area block is obtained; based on the above principle, the flow of the rectangular segmentation algorithm for the changed area in a scanning area is shown in Figure 3;

In order to improve the speed of judging the changing area of the screen, the image transmission method based on rectangular segmentation and scanning alternately adopts the direct comparison method of alternating column to judge whether the pixels corresponding to the two frames of screen images in the front and rear image buffers have changed, so as to find out the changed rectangular area; Behavioral units are separated by N columns (the number of detection interval columns N needs to be determined according to the specific situation). From left to right, the pixels corresponding to the two images before and after are extracted as sampling points, and the values of corresponding pixel points are compared whether they are the same; according to from top to bottom, Based on the principle of left to right, the image transmission method based on the rectangular segmentation alternate column scanning finds out all the changed areas of the next frame image compared to the previous frame image, and obtains the smallest set of non-overlapping rectangular areas based on the rectangular segmentation algorithm; its details The workflow is as follows:

Step 1, the image sender first obtains the resolution of the screen, and obtains the range 0~C of the column scan and the range 0~R of the row scan;

Step 2, the sending end saves the data in the current frame image storage area to the previous frame image buffer; intercepts the current screen bitmap data and saves it in the current frame image buffer;

Step 3, the sender first initializes the coordinates of the upper left corner and the lower right corner of the changed rectangular area to (0, 0), the coordinates of the starting point of the next scan are (0, 0), and the row no change flag is true, and the range and row of the column scan are updated scan range;

Step 4, judge whether it is within the line scanning range, if not, jump to step 10;

Step 5, judge whether it is within the column scanning range, if not, jump to step 8; if within the column scanning range, use the method of direct comparison between columns to detect the current sampling point; if the values are different, first set the row no change flag is false, then judge whether it is the first detected change sampling point, if it is the coordinate of the sampling point as the coordinate of the upper left corner of the change rectangular area, if it is not the first change sampling point, the coordinate of the lower right corner of the rectangle and the point Compare the coordinates and take the maximum value as the coordinate of the lower right corner of the new rectangle, and then judge whether the sampling point is the first change sampling point of the line, if so, compare the ordinate of the sampling point with the ordinate of the upper left corner of the rectangle And take the minimum value to update the coordinates of the upper left corner of the changing rectangular area; if the values are the same, it is necessary to judge whether the no-change flag is false, if it is false, record its coordinates as the starting point of the next scan, if it is the last column of sampling points detected, Use the coordinates of the last column of sampling points as the starting point of the next scan, and skip to step 7;

Step 6, move the column coordinates to the right by N columns, and jump to step 5 to detect the next sampling point;

Step 7: After the detection of this line is completed, compare the coordinates of the next scanning start point of this line with the coordinates of the next scanning start point recorded in the previous line, and take the maximum value as the new starting point coordinate of the next scan, add 1 to the line number, and jump to Step 4 detect from the beginning from left to right from the next line;

Step 8, judge whether the no-change flag of the line is true and the coordinates of the upper left corner of the changed rectangular area are not (0, 0), if not true, add 1 to the line number, and jump to step 4; if true, it indicates that the entire line has no Different pixels, thus a changed rectangular area block is obtained; due to the use of the direct comparison method every other column, there will be a phenomenon that the boundary information of the changed image is not detected and the boundary information is lost. In order to solve this problem, the obtained rectangular area of change The ordinate of the upper left corner of the block moves to the left by N columns, and the ordinate of the lower right corner moves to the right by N columns to include image boundary information;

Step 9, record the coordinates of the detected changing rectangular area and the corresponding starting point coordinates of the next scan, judge whether the range of the current column scan is 0-C and whether the range of the row scan is 0-R, if yes, set the flag to indicate the currently detected The variable rectangular area mark is detected for the first time, and then the line number is increased by 1 to jump to step 4 to detect the next changed rectangular area block from the next line; until the detection exceeds the range of line scanning, the process is shown in Figure 4;

Step 10, after this detection is completed, check whether there is any detected part of the entire screen. Therefore, it is necessary to calculate and generate the next scanning range based on the coordinates of the changed rectangular area detected this time and the corresponding starting point coordinates of the next scan. The method is Check whether the ordinate of the starting point of the first next scan detected this time is smaller than the ordinate of the last column of sampling points, if not, the detection of this area is completed, and the ordinate of the starting point of the next next scan is detected; if yes, use The abscissa of the upper left corner of the changing rectangular area detected for the first time is the abscissa, and the ordinate of the starting point coordinates of the next scan related to the current changing rectangular area is used as the ordinate to generate a coordinate of the upper left corner of the next scanning range; The abscissa of the lower right corner of the changed rectangular area detected once is the abscissa, and the maximum column number C of the screen is used as the ordinate to generate the coordinates of the lower right corner of the next scanning range; Scanning starting point processing, calculate the set of the next scanning range, the process is shown in Figure 5;

Step 11, calculate the range of row scan and column scan based on the width and height of the first scan area in the next scan range set, repeat step 3 to step 10 to obtain the changed rectangular area block in the scan area, scan based on the above method All scan area processing in the range set, the flow is shown in Figure 5;

Step 12, repeat steps 10 to 11 to obtain the changing rectangular area block of the next scan range until the ordinates of all the starting points of the next scan are greater than or equal to the ordinates of the last column of sampling points, and the entire screen is detected;

Step 13, obtain the set of non-overlapping rectangular areas with the smallest area of the frame image relative to the previous frame image change, check the rectangular areas in the set, if the vertical coordinates of the upper left corner and the lower right corner of the two rectangles are the same , and the abscissa of the lower right corner of one rectangle is adjacent to the abscissa of the upper left corner of another rectangle, merge them into a rectangle, and then compress and send the image data and corresponding coordinates contained in the set of rectangular areas to the client;

Step 14, the image receiving end decompresses the received data and integrates the received data into the previous frame image based on the image data and corresponding coordinates of each rectangular area and displays it;

Step 15, repeat steps 2 to 14 every T seconds, and the interval T can be adjusted according to different application scenarios and bandwidth requirements.

As shown in Fig. 8, the image transmission system based on the rectangular division and every column scanning of the embodiment of the present invention mainly includes: a sending end and a receiving end

The sender includes:

The screen image acquisition module is used for image acquisition of the screen at the sending end;

Rectangular segmentation and row scanning module, based on the rectangular segmentation and row scanning algorithm, compares the current frame image with the cached previous frame image, finds out the areas where all images change, and then obtains the smallest non-overlapping rectangle according to the coordinates of the changed pixels collection of regions;

The compression sending module compresses the image data contained in the collection of rectangular areas, and then sends the compressed image data and corresponding coordinates to the client through the network;

Receiver includes:

The receiving module receives the changed image data sent by the sending end;

The decompression module decompresses the received data to obtain the image data and corresponding coordinates contained in the set of rectangular areas;

The update display module updates and displays the cached corresponding area of the previous frame image based on the image data of each rectangular area and the corresponding coordinates.

Below in conjunction with accompanying drawing 7 the realization of the present invention is described in detail:

A network connection is established between the sending end and the receiving end. The sending end compares the current frame image with the cached previous frame image, obtains the set of non-overlapping rectangular areas with the smallest area in the two frames of images, and only compresses the set of sending rectangular areas. Include image data to reduce the transmission data of each frame; the receiving end integrates the received image data into the previous frame image and displays it, thereby saving bandwidth usage and CPU usage, and realizing fast image transmission; the following Taking screen image transmission with a screen resolution of 1024×768 as an example, Fig. 7 shows the change of the current frame image relative to the previous frame image, which is described in detail below in conjunction with Fig. 7:

Step 1, the sender first obtains the screen width of 1024 and height of 768, these two variables will be used as the range of column scanning and row scanning;

Step 2, the sending end first obtains the first frame image and saves the current frame image buffer, then compresses the frame image and sends it to the receiving end, and the receiving end displays the image information after receiving the image data;

Step 3, after 2 seconds, the sender saves the data of the current frame image buffer to the previous frame image buffer; intercepts the current screen image data and saves it in the current frame image buffer;

Step 4, the sender first initializes the coordinates of the upper left corner and the lower right corner of the changing rectangular area as (0, 0), the starting point coordinates of the next scan are (0, 0), the row no change flag is true, and the range of the updated column scan is 0 ～1024 and the range of line scanning is 0～768;

Step 5, use the direct comparison method to judge whether the pixels corresponding to the two frames of the screen before and after have changed to find out the changed rectangular area; the pointers pCurData and pBakData point to the pixel points of the current frame image and the previous frame image whose coordinates are (1, 5) respectively Compare whether the corresponding values are the same; if they are the same, space 4 pixels to the right in row units (the number of pixels in the interval can be determined according to the specific situation, take 4 pixels as an example) and take the next corresponding pixel of the two frames of images before and after It is a sampling point, compare the value of the pixel corresponding to the coordinate (1, 10) in the two frames of images to be the same, then detect that the value of the pixel corresponding to the next sampling point coordinate (1, 15) is different, and set the line no change flag to false, This is the first change sampling point detected, and its coordinates are used as the coordinates of the upper left corner of the change rectangular area; the sampling point (1, 15) is the first change detected in this line, and then the sampling point (1, 15) needs to be ) is compared with the ordinate of the upper left corner of the rectangle and takes the minimum value, and the coordinates of the upper left corner of the rectangle are updated to (1, 15);

Step 6, in the same line, then detect that the value of the next sampling point (1, 20) is the same, record its coordinates (1, 20) as the starting point of the next scan, and then start from the beginning from the next line (the second line) detection from left to right;

Step 7, the sender detects that the values of the sampling points (2, 5) and (2, 10) in the second row are the same, and continues to detect that the values of the next sampling point (2, 15) are different, and the lower right corner of the rectangle The coordinates (1, 15) of the point are compared with the coordinates of this point and take the maximum value to update the coordinates of the lower right corner of the rectangle to be (2, 15); the sampling point (2, 15) is the first changed sampling point in the second line, which needs to be The ordinate of the sampling point is compared with the ordinate of the upper left corner of the rectangle and takes the minimum value, and the coordinates of the upper left corner of the rectangle are updated to (1, 15); then it is detected that the value of the next sampling point (2, 20) is the same, update The starting point of the next scan (2, 20), and then start from the next row (third row) to detect from left to right;

Step 8. The sender detects that the values of the sampling points (3, 5) in the third line are different, compare the coordinates (2, 15) of the lower right corner of the rectangle with the coordinates of this point and take the maximum value to update the coordinates of the lower right corner of the rectangle as (3, 15); sampling point (3, 5) is the first changed sampling point in the third row, and the vertical coordinate (5) of the sampling point needs to be compared with the vertical coordinate (15) of the upper left corner of the rectangle and taken The minimum value, the coordinates of the upper left corner of the rectangle are updated to (1, 5); then compare the values of the following sampling points (3, 10) (3, 15) (3, 20) are all different, and the coordinates of the lower right corner of the rectangle are compared with these Compare the coordinates of the points and take the maximum value to update the coordinates of the lower right corner of the rectangle to (3, 20);

Step 9, the sender detects the sampling points of each row in turn; the value of the sampling point (10, 20) in the 10th row is different, compare the coordinates of the lower right corner of the rectangle with the coordinates of this point and take the maximum value to update the lower right corner of the rectangle The coordinates are (10, 20); the sampling point (10, 20) is the first changed sampling point in line 10, and the ordinate (20) of the sampling point needs to be compared with the ordinate (5) of the upper left corner of the rectangle And take the minimum value, update the coordinates of the upper left corner of the rectangle to (1, 5); in the same line, then detect that the value of the next sampling point (10, 25) is the same, record its coordinates (10, 25) as the next scan The starting point, and then from the next line (line 11) to detect from left to right from the beginning;

Step 10, the sender detects that the values of the sampling points in the 11th row are all the same, the flag of no change in the row is true and the coordinates of the upper left corner of the changed rectangular area are not (0, 0), it indicates that there is no different pixel point in the entire row, so The upper left corner of a changed rectangular area block is (1, 5), the lower right corner is (10, 20), and the starting point of the next scan is (10, 25); in order to solve the problem of loss of boundary information of the changed image, the changed rectangle obtained The ordinate of the upper left corner of the area block moves 5 columns to the left, and the ordinate of the lower right corner moves 5 columns to the right to contain the image boundary information; the upper left corner of the updated rectangular area block is (1, 0), and the lower right corner (10, 25 ), the range of the current front row scan is 0～C and the range of the row scan is 0～R, and the setting mark indicates that the currently detected change rectangle area mark is detected for the first time;

Step 11, use the above method to continue to detect downwards to obtain the next changed rectangular area block with (12, 20) in the upper left corner, (20, 40) in the lower right corner, and (18, 40) as the starting point of the next scan, and set the first The logo was detected for the second time;

Step 12, the sending end detects whether the values of the sampling points corresponding to each row are the same until the last row of the screen; based on the completion of the first scan in this example, the upper left corners of two changing rectangular areas are (1, 0), the lower right corner (10, 25), the starting point of the next scan (10, 25) and the upper left corner of the rectangular area block is (12, 20), the lower right corner (20, 40), the starting point of the next scan (18, 40);

Step 13, since the ordinates of the starting points (10, 25) and (18, 40) of the next scan are smaller than the ordinates of the last column of sampling points, the entire screen still has undetected parts, so the sender needs to determine For the next detected area, the second scanning range is calculated based on the coordinates of the changing rectangular area detected from top to bottom and the corresponding starting point coordinates of the next scan; first, the upper left corner of the first rectangular area is (1, 0) , the lower right corner (10, 25), the starting point of the next scan (10, 25) to calculate the first scan area; first, because the rectangular area (1, 0), (10, 25) is the change rectangle detected in the first round area, so the abscissas of the upper left corner and lower right corner of the scanning area are respectively the abscissas of the upper left corner of the first rectangular area (1, 0) and the lower right corner (10, 25), that is, the abscissas of the upper left corner and lower right corner of the scanning area are respectively 1 and 10; then compare the ordinate 25 of the scan starting point coordinates with the ordinate (1020) of the last column of sampling points on the screen; the ordinate of the scan starting point coordinates is smaller than the ordinate of the last column of the screen, and obtain the coordinate ordinate of the upper left corner of the scanning area Be the ordinate 25 of scanning starting point coordinate; Take the width 1024 of screen as the coordinate ordinate of the lower right corner of the scanning area, thus obtained a scanning area (1,25)(10,1024) in the scanning range for the second time; Then based on The upper left corner of the rectangular area block is (12, 20), the lower right corner is (20, 40), and the starting point of the next scan is (18, 40) to calculate the next scan area; the same as the method for calculating the first scan area, the second scan area is obtained A scanning area (12, 40) (20, 1024) in the secondary scanning range;

Step 14, after calculating and generating the area set of the second scan range, the range of the second scan column scan and row scan can be obtained; in this example, the second scan area set is the first area (1, 25) (10, 1024) and the second area (12, 40) (20, 1024); the range of the column scan of the first scan area in the next scan area set is (25≤x≤1024) and the range of row scan ( 1≤y≤10), the column scanning range of the second scanning area is (40≤x≤1024) and the row scanning range (12≤y≤20);

Step 15, the sender first initializes the coordinates of the upper left corner and the lower right corner of the changing rectangular area to (0, 0), and then uses the method of direct comparison every other column to determine whether there is a changing rectangular area block in the first scanning area. Based on this example, In the first area, a changed rectangular area block is obtained (2, 40) in the upper left corner, (9, 55) in the lower right corner, and the starting point of the next scan (7, 55);

Step 16, then the sender judges whether there is a changed rectangular area block in the second scanning area by using the method of direct comparison between columns, based on the value of the sampling point (20, 1020) in the last row and last column of the area scanned by the sender in this example All are the same, there is no changing rectangular area; the sending end sets the starting point coordinates of the next scan to (20, 1020); the ordinate of the changing rectangular area corresponding to the scanning starting point coordinates is the same as the ordinate of the last column of sampling points, The changed rectangular area is scanned; the second scan is completed, and a changed rectangular area block is obtained with the upper left corner (2, 40), the lower right corner (9, 55), and the starting point of the next scan (7, 55);

Step 17, since the ordinate of the starting point (7, 55) of the next scan is smaller than the ordinate of the last column of sampling points, the sending end calculates and generates the third scanning range again, first based on the upper left corner of the rectangular area being (2, 40), the lower right corner (9, 55), the starting point of the next scan (3, 55) is calculated; because the upper left corner (2, 40) of the current rectangular area, the lower right corner (9, 55) is not the change detected in the first round In the rectangular area, the abscissas of the upper left corner and lower right corner of the scanning area in the scanning area are kept unchanged, and the ordinate 55 of the scanning starting point is compared with the ordinate (1020) of the last column of sampling points on the screen; the ordinate of the scanning starting point is smaller than the last The ordinate of a row, obtains the ordinate of the upper left corner of the scanning area and is the ordinate of the scanning starting point coordinate 55; Take the width 1024 of the screen as the ordinate of the lower right corner; thus obtain a scanning area (1, 55)(10,1024);

Step 18, then the sender uses the method of direct comparison between columns to determine whether there is a changed rectangular area block in the third scanning range, based on the value of the sampling point (10, 1020) in the last row and last column of the area scanned by the sender in this example All are the same, there is no changing rectangular area; the sending end sets the starting point coordinates of the next scan to (20, 1020); the ordinate of the changing rectangular area corresponding to the scanning starting point coordinates is the same as the ordinate of the last column of sampling points, The changing rectangular area is scanned;

Step 19, so far, the ordinates of all scanning starting point coordinates are the same as the ordinates of the last column of sampling points, the entire screen is scanned, and the obtained changing rectangular area is (1, 0) in the upper left corner of the rectangular area block, and (1, 0) in the lower right corner (10, 25), the upper left corner of the rectangular area block is (12, 20), the lower right corner is (20, 40), the upper left corner of the rectangular area block is (2, 40), and the lower right corner is (9, 55);

Step 20, the sending end compresses and sends the image data and coordinate information contained in the set of changing rectangular areas to the client; the image receiving end integrates the received image data into the previous frame image and displays it.

Step 21, repeat steps 3 to 20 every 2 seconds, and the interval can be adjusted according to different application scenarios and bandwidth requirements.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (5)

1. An image transmission method based on rectangular segmentation and scanning every other column, it is characterized in that the image transmission method based on rectangular segmentation and scanning every other column first compares two adjacent frames of images, finds out the regions where all images change, and then according to the changed pixel The coordinates of the points get the set of non-overlapping rectangular areas with the smallest area; each time only the image data and corresponding coordinate information contained in the set of rectangular areas are sent; According to the coordinates of the pixel points, the variable rectangular area is obtained, and formula (1) and formula (2) are formulas for judging the range of the rectangular R according to the variable pixel points; R _l ≤ P _x AND R _t = P _yi (1) R _r ≥ P _x AND R _b ≥ P _y (2) Among them, R _l and R _t represent the abscissa and ordinate of the upper left corner of the rectangle, R _r and R _b represent the abscissa and ordinate of the lower right corner of the rectangle, P _x and P _y represent the abscissa and ordinate of the changed pixel, P _y0 represents the ordinate of the first changed pixel point; calculate the range of the changed rectangular area according to formula (1) and formula (2); first save the data of the two adjacent bitmaps, and judge the two frame screens before and after Whether the value of the corresponding pixel changes; when the changed sampling point is detected for the first time, the coordinates (P _X0 , P _Y0 ) of the changed sampling point will be recorded as the coordinates of the upper left corner of the changed rectangular area (R _l , R _t ), and mark no change in the line as false; continue to compare, when a different sampling point is detected again, first mark the line without change as false, and then the abscissa P _x of the sampling point is the same as the abscissa R of the upper left corner of the rectangle _l compare and take the minimum value, and the coordinates (R _r , R _b ) of the lower right corner of the rectangle will be compared with the point coordinates (P _x , P _y ) and take the maximum value; that is: R _l =min(P _xi ,R _l )(i>1) R _t =P _yi (i=1) R _r ＝max(R _xi ,R _r )(i>1) R _b ＝max(R _yi ,R _b )(i>1) When it is detected that the values of the sampling points in a certain row are all the same, a changed rectangular area block is obtained. 2. the image transmission method based on rectangular segmentation and scanning every other row as claimed in claim 1, it is characterized in that, in a scan area, carry out rectangular segmentation algorithm to change area and adopt the direct comparison method every other row to judge two frames in the front and rear image buffers Whether the pixel corresponding to the screen image changes to find out the changed rectangular area; according to the principle of from top to bottom and from left to right, based on the image transmission method of rectangular segmentation and scanning, find out the difference between the next frame image and the previous frame image All the changed regions are obtained based on the rectangle segmentation algorithm to obtain the set of non-overlapping rectangular regions with the smallest area. 3. the image transmission method based on rectangular segmentation and scanning every other column as claimed in claim 2, is characterized in that, adopts every column direct comparison method to judge whether the corresponding pixels of the two frame screen images before and after change, first with row unit every N column from From left to right, start to extract the pixels corresponding to the front and back images as sampling points, and compare whether the corresponding pixel values are the same; according to different application scenarios and bandwidth requirements, adjust the number of interval columns N, the smaller the value of N, the smaller the interval Column direct comparison method detection takes longer. 4. the image transmission method based on rectangular segmentation and scanning every other column as claimed in claim 2, is characterized in that, in a scanning area, the specific method of rectangular segmentation algorithm is carried out to changing area as follows: Step 1, the image sending end first obtains the resolution of the screen, and obtains the range 0-C of the column scan and the range 0-R of the row scan; Step 2, the sending end saves the data in the current frame image storage area to the previous frame image buffer; intercepts the current screen bitmap data and saves it in the current frame image buffer; Step 3, the sender first initializes the coordinates of the upper left corner and the lower right corner of the changing rectangular area to (0, 0), the coordinates of the starting point of the next scan are (0, 0), the flag of no change in the row is true, and the range and row of the column scan are updated scan range; Step 4, judge whether it is within the line scanning range, if not, jump to step 10; Step 5, judge whether it is within the column scanning range, if not, jump to step 8; within the column scanning range, use the method of direct comparison between columns to detect the current sampling point; if the values are different, first set the row no change flag to false, Then judge whether it is the first change sampling point detected. The coordinates of the sampling point are used as the coordinates of the upper left corner of the change rectangular area, not the first change sampling point. The coordinates of the lower right corner of the rectangle are compared with the coordinates of this point and taken The maximum value is used as the coordinate of the lower right corner of the new rectangle, and then it is judged whether the sampling point is the first changed sampling point of the line, and the vertical coordinate of the sampling point is compared with the vertical coordinate of the upper left corner of the rectangle and the minimum value is used to update the change Coordinates of the upper left corner of the rectangular area; if the values are the same, it is necessary to judge whether the row no change flag is false, if it is false, record the coordinates as the starting point of the next scan, detect the last column of sampling points, and use the coordinates of the last column of sampling points as the next time The starting point of scanning, skip to step 7; Step 6, move the column coordinates to the right by N columns, and jump to step 5 to detect the next sampling point; Step 7: After the detection of this line is completed, compare the coordinates of the next scanning start point of this line with the coordinates of the next scanning start point recorded in the previous line, and take the maximum value as the new starting point coordinate of the next scan, add 1 to the line number, and jump to Step 4: Detect from the next line from the beginning to the right; Step 8, judge whether the row no change flag is true and the coordinates of the upper left corner of the changed rectangular area are not (0, 0), if it is not true, add 1 to the row number, and jump to step 4; if it is true, it means that the entire row has no different pixels point, a changed rectangular area block is obtained; the ordinate of the upper left corner of the obtained changed rectangular area block is moved to the left by N columns, and the ordinate of the lower right corner is moved to the right by N columns to contain image boundary information; Step 9: Record the coordinates of the detected change rectangular area and the corresponding starting point coordinates of the next scan, determine whether the range of the current column scan is 0-C and whether the range of the row scan is 0-R, and if yes, set a flag to indicate the currently detected change The rectangular area mark is detected for the first time, then add 1 to the line number and jump to step 4 to detect the next changed rectangular area block from the next line; until the detection exceeds the range of line scanning; Step ten, after this detection is completed, process all the starting points of the next scan in this detection, and calculate the set of the next scanning range; first check whether the ordinate of the first starting point of the next scan detected this time is It is smaller than the ordinate of the last column of sampling points. No, the detection of this area is completed, and the ordinate of the next scanning starting point is detected; The ordinate of the starting point coordinates of the next scan related to the current changing rectangular area is the ordinate, and a coordinate of the upper left corner of the next scanning range is generated; the abscissa of the lower right corner of the changing rectangular area detected for the first time is taken as the abscissa, and the screen The maximum number of columns C is the vertical coordinate to generate the coordinates of the lower right corner of the next scanning range; then process the second next scanning starting point until all the next scanning starting points in this detection are processed; Step 11: Detect all scan areas in the next scan range set, firstly generate the range of row scan and column scan based on the width and height of the first scan area in the next scan range set, repeat steps 3 to 10 to detect The changed rectangular area block in the first scanning area, and then process the second scanning area until all scanning areas in the next scanning area set are detected; Step 12, repeating step 10 to step 11, to obtain the changing rectangular area block of the next scanning range, until the vertical coordinates of all the starting points of the next scanning are greater than or equal to the vertical coordinates of the last column of sampling points, and the detection of the entire screen is completed; Step 13, obtain the set of non-overlapping rectangular areas with the smallest area of the frame image relative to the previous frame image change, check the rectangular areas in the set, the vertical coordinates of the upper left corner and the lower right corner of the two rectangles are the same , and the abscissa of the lower right corner of one rectangle is adjacent to the abscissa of the upper left corner of another rectangle, merged into one rectangle, and then compress and send the image data and corresponding coordinates contained in the set of rectangular areas to the client; Step 14, the image receiving end decompresses the received data and integrates the received data into the previous frame image based on the image data and corresponding coordinates of each rectangular area and displays it; Step 15: Repeat steps 2 to 14 every T seconds, and adjust the interval T according to different application scenarios and bandwidth requirements. 5. A kind of image transmission system based on rectangular division scanning every other row, it is characterized in that, this image transmission system based on rectangular segmentation scanning every other row comprises sending end and receiving end: The sender includes: The screen image acquisition module is used for image acquisition of the screen at the sending end; Rectangular segmentation and row scanning module, based on the rectangular segmentation and row scanning algorithm, compares the current frame image with the cached previous frame image, finds out all areas where the image changes, and then obtains the smallest non-overlapping rectangle according to the coordinates of the changed pixels collection of regions; The compression sending module compresses the image data contained in the collection of rectangular areas, and then sends the compressed image data and corresponding coordinates to the client through the network; Receiver includes: The receiving module receives the changed image data sent by the sending end; The decompression module decompresses the received data to obtain the image data and corresponding coordinates contained in the set of rectangular areas; The update display module updates and displays the cached corresponding area of the previous frame image based on the image data of each rectangular area and the corresponding coordinates.