TWI766434B - Image stitching apparatus, image processing chip and image stitching method - Google Patents

Abstract

An image stitching apparatus, an image processing chip and an image stitching method are provided. The image stitching apparatus includes a motion detection unit, a determining unit and a stitching unit. The motion detection unit performs motion detection on an overlapped area of a first image and a second image to be stitched to obtain a motion area having a moving object in the overlapped area. The determining unit calculates a target stitching line by using constrained criteria of avoiding the motion area. The stitching unit stitches the first image and the second image according to the stitching line to generate a stitched image.

Description

Image splicing device, image processing chip and image splicing method

The present invention relates to image processing technology, in particular to an image splicing device, an image processing chip and an image splicing method.

Image stitching refers to the fusion of two or more images into one image, so that the fused image contains more information and can be more conveniently viewed by users or processed by computers. Image stitching is an important research direction in the field of image processing technology and has a wide range of application values. However, in the related art, when performing image splicing, only the visual difference between the images to be spliced is concerned, resulting in low quality of the spliced images finally obtained.

In view of the problems of the prior art, one object of the present invention is to provide an image splicing device, an image processing chip and an image splicing method to improve the prior art.

The invention includes an image splicing device, which includes a motion detection unit, a determination unit and a splicing unit. The motion detection unit performs motion detection on the overlapping area of a first image and a second image to be spliced to obtain a moving area in which a moving object exists in the overlapping area. The determining unit calculates a target splicing line with avoiding the moving area as a constraint. The splicing unit splices the first image and the second image according to the target splicing line to obtain a spliced image.

The present invention further includes an image processing chip, which includes an interface unit, an area determination unit and an image splicing device. The interface unit acquires a first image and a second image that need to be stitched. The area determination unit determines an overlapping area between the first image and the second image. The image splicing device includes a motion detection unit, a determination unit and a splicing unit. The motion detection unit performs motion detection on the overlapping area of a first image and a second image to be spliced to obtain a moving area in which a moving object exists in the overlapping area. The determining unit calculates a target splicing line with avoiding the moving area as a constraint. The splicing unit splices the first image and the second image according to the target splicing line to obtain a spliced image.

The present invention also includes an image splicing method, comprising: performing motion detection on an overlapping area between a first image and a second image to obtain a moving area in which a moving object exists in the overlapping area; avoiding the moving area as Constraints, calculating a target splicing line; and splicing the first image and the second image according to the target splicing line to obtain a spliced image.

Regarding the features, implementation and effects of this case, a preferred embodiment is described in detail as follows in conjunction with the drawings.

It should be noted that the principles of the present invention are exemplified by implementation in an appropriate application environment. The following description is by way of illustrative embodiments of the invention and should not be construed as limiting other embodiments of the invention not detailed herein.

The technology provided by the embodiments of the present invention relates to the technical field of image processing, specifically related to image splicing, which will be described through the following embodiments. Please refer to Figure 1. FIG. 1 shows a first block schematic diagram of an image splicing apparatus 100 according to an embodiment of the present invention. The image splicing device 100 may include a motion detection unit 110 , a determination unit 120 and a splicing unit 130 which are connected to each other. In practice, the motion detection unit 110 , the determination unit 120 and the splicing unit 130 can be implemented by hardware circuits and software.

Please refer to Figure 2. FIG. 2 shows an example diagram of the motion detection unit 110 in FIG. 1 performing motion detection on the overlapping area of the first image and the second image to obtain a motion area according to an embodiment of the present invention. The motion detection unit 110 is configured to perform motion detection on the overlapping area between the first image and the second image to be spliced, so as to obtain a moving area with moving objects in the overlapping area of the first image and the second image. The moving object at can be any object with movement, including but not limited to people, objects, and the like. The motion detection method of the motion detection unit 110 is not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs.

In addition, in this embodiment of the present invention, there is no specific limitation on the sources of the first image and the second image. For example, the first image and the second image may be part of the field of view captured by the same camera at different angles during the horizontal rotation process. The two overlapping images, or, the first image and the second image may be two images respectively shot by two cameras with partially overlapping visual fields.

Please refer to Figure 3. FIG. 3 shows an example diagram of a target splice line calculated by the determining unit 120 in FIG. 1 according to an embodiment of the present invention. The determining unit 120 is configured to calculate a splicing line for splicing the first image and the second image with avoiding the moving area as a constraint, which is recorded as a target splicing line. As shown in FIG. 3 , the target splicing line calculated by the determining unit 120 avoids the moving area, that is, the target splicing line does not pass through the moving area. The manner in which the determination unit 120 calculates the target splicing line is not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs.

Please refer to Figure 4. FIG. 4 shows an exemplary diagram in which the splicing unit 130 in FIG. 1 splices the first image and the second image according to the target splicing line shown in FIG. 3 to obtain a spliced image according to an embodiment of the present invention. The splicing unit 130 is configured to splicing the first image and the second image according to the target splicing line to obtain a spliced image. As shown in FIG. 4 , the spliced image obtained by splicing the first image and the second image includes the image content of the first image and the second image, wherein the image content on the left side of the splicing line in the spliced image comes from the first image, and the right side of the splicing line in the spliced image comes from the first image. The content of the image comes from the second image.

Please refer to Figure 5. FIG. 5 shows a second block diagram of the image splicing apparatus 100 according to an embodiment of the present invention. In one embodiment, the image stitching apparatus 100 further includes a difference calculation unit 140 configured to calculate at least one difference matrix between the first image and the second image for the overlapping area. After the difference matrix is obtained, the determining unit 120 is further configured to calculate the target splice line according to the difference matrix with the constraints of minimizing the difference on both sides of the splice line and avoiding the moving area.

It should be noted that the difference matrix calculated by the difference calculation unit 140 is used to describe the difference of each position in the overlapping area of the first image and the second image, which may be the difference between pixels, or may be composed of multiple pixels. difference between pixel blocks. The difference calculation unit 140 calculates the difference between the first image and the second image, which can be one or more of color difference, gray level difference, and edge difference. For example, the difference calculation unit 140 is configured to calculate the color difference between the pixels, and the difference calculation unit 140 calculates the color difference matrix between the first image and the second image for the overlapping area of the first image and the second image, The color difference matrix is used to describe the color difference between two pixels at the same position in the overlapping area of the first image and the second image.

As above, after the difference calculation unit 140 calculates and obtains the aforementioned difference matrix, and after the motion detection unit 110 detects the aforementioned moving area, the determining unit 130 uses the aforementioned difference matrix to minimize the difference between the two sides of the splicing line and avoid the aforementioned moving area as Constraints, the above-mentioned target splicing line is obtained by calculation. Therefore, when the splicing unit 130 uses the target splicing line to splices the first image and the second image, the difference between the image contents on the left and right sides of the spliced image obtained is the smallest, and the target splicing line will not pass through the moving object , which can further improve the image stitching quality.

In one embodiment, the difference calculating unit 140 is configured to obtain a plurality of difference matrices of different types between the first image and the second image by calculating for the overlapping area according to a plurality of different difference calculation methods.

Please refer to Figure 6. FIG. 6 shows a more detailed block diagram of the difference calculation unit 140 in FIG. 5 according to an embodiment of the present invention. In one embodiment, the difference calculation unit 140 includes a color difference calculation circuit 1402 , an edge detector 1404 and an edge difference calculation circuit 1406 . The color difference calculation circuit 1402 calculates the color difference between each group of pixel points at the same position in the first image and the second image for the overlapping area, so as to obtain a pixel point at the same position in the overlapping area describing the first image and the second image. Color difference matrix of color differences between. Or, by configuring the color difference calculation circuit 1402, the color difference calculation circuit 1402 divides the overlapping area of the first image and the second image into a plurality of pixel blocks (each pixel block includes multiple pixels), and calculate the pixel block at the same position in the overlapping area of the first image and the second image (for a pixel block, fuse the color values of all pixels in the pixel block to obtain a fusion color value, which is used for color The calculation of the difference, such as for each color channel, calculate the color difference of the average color value of all pixels respectively, so as to obtain a description of the color difference between the pixel blocks at the same position in the overlapping area of the first image and the second image. Color difference matrix.

For example, please refer to Figure 7. FIG. 7 shows an example diagram of a color difference matrix obtained by the color difference calculation circuit 1402 in FIG. 6 according to an embodiment of the present invention. The overlapping areas of the first image and the second image are respectively a right area of the first image and a left area of the second image. When performing color difference calculation, the color difference calculation circuit 1402 divides the right side area of the first image and the left side area of the second image into 4×4 16 pixel blocks in the same block manner. Then, the color difference calculation circuit 1402 calculates the color difference for each group of pixel blocks in the same position in the right area of the first image and the left area of the second image. For example, the pixel block of the upper left vertex of the right area of the first image and the pixel block of the upper left vertex of the left area of the second image are a group of pixel blocks in the same position. Assuming that the pixel values of the red channel, green channel, and blue channel of the upper left vertex pixel block in the right area of the first image are 56, 255, and 251, respectively, and the red channel, green channel, and blue channel of the upper left vertex pixel block in the left area of the second image are the red channel, green channel, and blue channel. The pixel values are 52, 253, and 250, respectively. The color difference calculation circuit 1402 calculates the absolute value of the difference between the red channel, green channel and blue channel of the previous group of pixel blocks respectively, and the obtained results are 4, 2, 1. Take the maximum value of 4 as the difference value of the previous group of pixel blocks.

According to the above calculation method, the color difference calculation circuit 1402 will calculate the color difference between other pixel blocks at the same position in the overlapping area of the first image and the second image, so that the difference values corresponding to different pixel block groups The color difference matrix shown in FIG. 7 is constructed.

The edge detector 1404 performs edge detection on the overlapping regions of the first image and the second image respectively, so as to generate the first edge detection result and the second edge detection result respectively. In practice, the edge detector 1404 can use the The edge of the object in the image is detected by analyzing the pixel value change or relative relationship between each pixel in the image and its surrounding pixels.

The edge difference calculation circuit 1406 can be configured to calculate the edge between each group of pixel points or pixel blocks at the same position in the overlapping area of the first image and the second image according to the first edge detection result and the second edge detection result difference, thereby obtaining an edge difference matrix describing the edge difference between pixels or pixel blocks at the same position in the overlapping area of the first image and the second image. When the calculation is based on a pixel block, a pixel block is a fused edge value obtained by fusing the edge values of all the pixel points in the pixel block, such as the average edge value of all the pixel points in the pixel block.

For example, please refer to Figure 8. FIG. 8 shows an example diagram of edge detection performed by the edge detector 1404 in FIG. 6 and an edge difference matrix calculated by the edge difference calculation circuit 1406 in FIG. 6 in an embodiment of the present invention. The overlapping areas of the first image and the second image are respectively a right area of the first image and a left area of the second image. When performing edge difference calculation, the edge difference calculation circuit 1406 divides the right side area of the first image and the left side area of the second image into 4x4 blocks of 16 pixels in the same block manner. Then, the edge difference calculation circuit 1406 performs edge difference calculation for each group of pixel blocks at the same position in the right area of the first image and the left area of the second image. For example, if the pixel block of the upper left vertex of the right area of the first image and the pixel block of the upper left vertex of the left area of the second image are a group of pixel blocks in the same position, the calculated difference value is 2.

According to the above calculation method, the edge difference calculation circuit 1406 will calculate the edge difference between other pixel blocks at the same position in the overlapping area of the first image and the second image, so that the difference values corresponding to different pixel block groups The edge difference matrix shown in FIG. 8 is formed, which is used to describe the edge difference matrix between each pixel block at the same position in the overlapping area of the first image and the second image.

In the embodiment of the present invention, by configuring the color difference calculation circuit 1402 and the edge difference calculation circuit 1406, the color difference matrix calculated by the color difference calculation circuit 1402 and the edge difference matrix obtained by the edge difference calculation circuit 1406 have the same shape . For example, by configuring the color difference calculation circuit 1402 and the edge difference calculation circuit 1406 to use pixel points as the difference calculation object, or both pixel blocks (pixel blocks are divided according to the same block method) as the difference calculation objects, The shapes of the edge difference matrix and the color difference matrix calculated by the two are the same.

Please refer to Figure 9. FIG. 9 shows a block diagram of the determining unit 120 in FIG. 5 according to an embodiment of the present invention. The determination unit 120 is composed of two parts, which are a data processing circuit 1202 and a splice line calculation circuit 1204 respectively. In order to effectively utilize the differences in different aspects of the overlapped regions between the first image and the second image, the data processing circuit 1202 first fuses the color difference matrix and the edge difference matrix, and fuses the obtained color difference matrix with the color difference and the fused disparity matrix of edge disparities as a cost map for computing splice lines. In addition, in order to enable the calculated target splicing line to avoid the moving area, the data processing circuit 1202 further adds a penalty cost to the element at the corresponding position of the cost map according to the previously detected moving area, so as to prevent the splicing line from moving through the overlapping area area probability. That is to say, the data processing circuit 1202 further corrects the cost map according to the moving area to obtain the corrected cost map. The fusion method of the color difference matrix and the edge difference matrix and the specific value of the penalty cost are not specifically limited here, and can be configured by those of ordinary skill in the art according to actual needs. It can be understood that the larger the value of the penalty cost, the higher the probability that the calculated splicing line avoids the moving area.

For example, the fusion method of the color difference matrix and the edge difference matrix can be configured to directly add the corresponding position elements, and the penalty cost can be configured to be 6. Please refer to Figure 10. FIG. 10 shows an example diagram of a cost map calculated by the data processing circuit 1202 in FIG. 9 according to an embodiment of the present invention. Assuming that the moving area is the two-pixel block area in the lower right corner of the overlapping area, the data processing circuit 1202 will generate a moving matrix according to the moving area, and the element whose value is 0 in the moving matrix indicates that the pixel block at the corresponding position in the overlapping area is different. is the moving area, and the element whose value is the penalty cost 6 indicates that the pixel block at the corresponding position in the overlapping area is the moving area. As shown in FIG. 10 , the data processing circuit 1202 directly adds the elements at the corresponding positions of the color difference matrix and the edge difference matrix to obtain a cost map, and then directly adds the cost map and the elements at the corresponding positions of the movement matrix, which is the cost The graph increases the penalty cost, resulting in a revised cost graph.

In one embodiment, the data processing circuit 1202 is configured to perform weighted addition of elements at corresponding positions of the color difference matrix and the edge difference matrix to obtain a cost map. For example, the sum of the weights of the color difference matrix and the edge difference matrix is 1 as a constraint, and a person of ordinary skill in the art assigns weights to the color difference matrix and the edge difference matrix according to actual needs. It can be understood that if the weight assigned to the color difference matrix is larger, the color difference in the overlapping area between the first image and the second image will affect the calculation of the target splicing line more. Similarly, if the weight of the edge difference matrix is assigned, If it is larger, the edge difference of the overlapping area between the first image and the second image will affect the calculation of the target splicing line more.

After obtaining the cost map, the splicing line calculation circuit 1204 can calculate a minimum cost splicing line according to the cost map using the minimum cost method, and use the minimum cost splicing line as the target splicing line for splicing the first image and the second image.

In one embodiment, the splice line calculation circuit 1204 accumulates a plurality of elements of the cost map and one of the adjacent elements in the adjacent row along an accumulation direction with the constraint condition of minimizing the accumulation value, so as to obtain the target splice line .

For example, please refer to Figure 11. FIG. 11 shows an exemplary diagram of calculating the target splice line by the splice line calculation circuit 1204 in FIG. 9 according to an embodiment of the present invention. It is an example graph of calculating target splice lines. The splicing line calculation circuit 1204 first determines an accumulation direction for cost accumulation according to the overlapping area of the first image and the second image. Assuming that the first image and the second image overlap left and right, the splicing line calculation circuit 1204 can select from top to bottom. The direction is the accumulation direction, and the bottom-to-top direction can also be selected as the accumulation direction. Here, the top-to-bottom direction is selected as the accumulation direction for illustration.

After the accumulation direction is determined, the splice line calculation circuit 1204 determines a start row of elements and an end row of elements in the cost map according to the accumulation direction. Please continue to refer to Fig. 11, according to the aforementioned accumulation direction, the horizontal row "6, 1, 0, 1" of the revised cost map is the first horizontal row, and there is no accumulation object, correspondingly the horizontal row "6, 1, 0, 1" is used as the For the initial horizontal row of the cost accumulation graph, set horizontal row "1, 6, 6, 1" as the starting row element, and set horizontal row "1, 7, 8, 9" as the ending row element. That is, the second row element in the accumulation direction is set as the start row element, and the last row element in the accumulation direction is set as the end row element.

After determining the starting row element and ending row element transition, for each element in the starting row element, the splice line calculation circuit 1204 is in an adjacent row element in the opposite direction to the accumulation direction to minimize the accumulation cost (ie, Minimize Accumulation) Select a target element for the constraint to accumulate cost. Among them, for the accumulated row of elements, except that the two elements at both ends search for the target element in the two adjacent elements in the adjacent row of elements in the opposite direction of the accumulation direction, the other elements in the adjacent row of elements in the opposite direction of the accumulation direction Searches for the target element among the three adjacent elements in it. Please continue to refer to Figure 11, for the "1" at the left end of the initial row element, search for the target element in the two adjacent elements "6, 1" in the adjacent row elements "6, 1, 0, 1" in the opposite direction of accumulation. , at this time, select "1" as the target element for accumulation, and obtain the accumulated cost "2" and the corresponding accumulation path "1"-"2" (as shown by the solid arrow in Figure 11); and for the initial row element In "6" (here is the first "6" from left to right), three adjacent elements "6, 1, 1" in the adjacent row of elements "6, 1, 0, 1" in the opposite direction of accumulation Search for the target element in 0. At this time, select "0" as the target element for accumulation, and obtain the accumulated cost "6" and the corresponding accumulation path "0"-"6"; The cost is accumulated, and the corresponding accumulated cost is "2, 6, 6, 1".

By analogy, the splicing line calculation circuit 1204 continues to perform cost accumulation on the adjacent row elements in the accumulation direction of the initial row elements (that is, the elements in the third row from top to bottom in the cost diagram), to obtain the corresponding accumulated costs as shown in Figure 11 . The third row of elements from top to bottom in the accumulation diagram is "3, 2, 1, 2". In this way, it is accumulated to the end row element (that is, the fourth row element from top to bottom in the cost graph).

Finally, the cost accumulation graph is also obtained with 4 horizontal rows and 4 vertical rows, a total of 16 elements, which are "6, 1, 0, 1", "2, 6, 6, 1", " 3, 2, 1, 2" and "3, 8, 9, 10". Correspondingly, the splice line calculation circuit 1204 generates the minimum cost splice line according to the accumulated path with the smallest accumulated cost ("1"-"2"-"2"-"3" in this case), as shown in FIG. 11 .

In another embodiment, the splice line calculation circuit 1204 accumulates a plurality of elements in the cost map and an element in the adjacent elements in the same row and adjacent rows along an accumulation direction with the constraint condition of minimizing the accumulation value, so as to obtain Get the target splice line. The difference from the previous embodiment is that in this embodiment, the splicing line calculation circuit 1204 selects a target element from the adjacent elements in the same row and adjacent rows, not only from the adjacent row, in the process of selecting a target element for cost accumulation. Select from adjacent elements.

Please refer to Figure 12. FIG. 12 shows another exemplary diagram of calculating the target splice line by the splice line calculation circuit 1204 in FIG. 9 according to an embodiment of the present invention. It calculates an example graph of the target splice line for this embodiment. In this embodiment, the top-to-bottom direction is selected as the accumulation direction for illustration. As in the previous embodiment, set the horizontal row "1, 6, 6, 1" as the starting row element, and set the horizontal row "1, 7, 8, 9" as the ending row element.

Please continue to refer to Figure 12, for the "1" at the left end of the initial row element, search for the target element in the two adjacent elements "6, 1" in the adjacent row elements "6, 1, 0, 1" in the opposite direction of the accumulation direction , at this time, select "1" as the target element for accumulation, and obtain the accumulated cost "2" and the corresponding accumulation path "1"-"2" (as shown by the solid arrow in Figure 12); then, the splicing line calculation circuit 1204 further updates the accumulated cost according to a first vertical direction of the accumulation direction, that is, updates the accumulated cost for adjacent elements in the same row. Herein, the left-to-right direction is taken as the first vertical direction for illustration. At this time, there is no adjacent element on the left of "1", and its accumulated cost is not updated, and the accumulated cost "2" is retained.

For the "6" on the left side of the initial row of elements (here, the first "6" from left to right), three elements in the adjacent row of elements "6, 1, 0, 1" in the opposite direction of accumulation Search for the target element in the adjacent elements "6, 1, 0". At this time, select "0" as the target element for accumulation to obtain the accumulated cost "6" and the corresponding accumulated path "0"-"6"; then, splicing The line calculation circuit 1204 further updates the accumulated cost according to a first vertical direction of the accumulation direction (in this case, the direction from left to right). At this time, the adjacent element on the left of "6" is "1", and the splicing line calculation circuit 1204 determines whether the accumulated cost corresponding to "6" is greater than the sum of the accumulated cost corresponding to "1" and the cost of "6". The cumulative cost corresponding to 6" is replaced by the sum of the cumulative cost corresponding to "1" and the cost of "6". As shown in Figure 12, the accumulated cost corresponding to "6" is 6, which is less than the sum (8) of the accumulated cost (2) corresponding to "1" and the cost (6) of "6", and the accumulated cost corresponding to "6" is not make a replacement.

By analogy, after the update of all elements in the first vertical direction is completed, the accumulated cost corresponding to the first row of elements is "2, 6, 6, 1". At this time, the splicing line calculation circuit 1204 calculates In the opposite second vertical direction (in this case, the direction from right to left), the accumulated cost of each element is updated again according to the opposite adjacent elements of each element in the second vertical direction. For the "1" at the right end of the element in the initial row, at this time, there is no adjacent element on the right side of "1", and the accumulated cost is not updated, and the accumulated cost "1" is retained. And for the "6" on the right in the first row of elements (here, the second "6" from left to right). At this time, the adjacent element on the right side of "6" is "1", and the splicing line calculation circuit 1204 determines whether the accumulated cost corresponding to "6" is greater than the sum of the accumulated cost corresponding to "1" and the cost of "6". The cumulative cost corresponding to 6" is replaced by the sum of the cumulative cost corresponding to "1" and the cost of "6". As shown in Figure 12, the accumulated cost corresponding to "6" is 6, which is less than the sum of the accumulated cost corresponding to "1" and the cost of "6" by 7, and the accumulated cost corresponding to "6" is not replaced. By analogy, the cumulative cost of the initial row element is updated again in the second vertical direction, and finally the cumulative cost corresponding to the initial row element is still "2, 6, 6, 1".

As above, the splice line calculation circuit 1204 continues to perform cost accumulation on the adjacent row elements of the initial row elements in the accumulation direction (that is, the third row element "1, 0, 0, 1" from top to bottom in the cost map), and in the first row A vertical direction and a second vertical direction are updated for adjacent elements in the same row. In this way, it is accumulated to the end row element (that is, the fourth row element from top to bottom in the cost graph). Finally, the cost accumulation graph is also obtained with 4 horizontal rows and 4 vertical rows, a total of 16 elements, which are "6, 1, 0, 1", "2, 6, 6, 1", " 2, 1, 1, 2" and "2, 8, 9, 10". Correspondingly, the splice line calculation circuit 1204 generates the minimum cost splice line according to the accumulated path with the smallest accumulated cost (in this case, "0"-"1"-"1"-"1"-"1"-"2"), such as Figure 12.

Please refer to Figure 13. FIG. 13 shows a schematic diagram of a third block of the image splicing apparatus 100 according to an embodiment of the present invention. In one embodiment, the image splicing apparatus 100 further includes a displacement limiting unit 150 for acquiring a previous splicing line calculated by the splicing line calculating circuit 1204 before, and generating a displacement limiting data according to the previous splicing line. The determining unit 120 then calculates the target splicing line according to the aforementioned at least one difference matrix and the displacement restriction data with avoiding the moving area as a constraint. By considering the displacement restriction data related to the previous splicing line, it is avoided that the difference between the target splicing line calculated in the current picture and the previous splicing line of the previous picture is too large, which will affect the continuity of the picture.

In detail, the displacement limiting unit 150 may determine the moving cost according to the distance between each element in the cost map and the previous splicing line. For example, the larger the distance, the greater the determined displacement cost as a constraint, thereby generating corresponding displacement limiting data. The data processing circuit 1202 in the determining unit 120 then corrects the cost map according to the displacement restriction data.

For example, there are a first camera and a second camera with partially overlapping fields of view, and both the first camera and the second camera are in a shooting mode, and images are continuously captured. At the start time t0 when the first camera and the second camera start capturing, the image captured by the first camera at time t0 is taken as the first image, and the image captured by the second camera at time t0 is taken as the second image, Since the previous first image and the previous second image do not exist at this time, after the data processing circuit 1202 fuses the cost map, the splicing line calculation circuit 1204 calculates a minimum cost splicing line according to the cost map, which is used for splicing at time t0 The target stitching line of the first image and the second image. At the next time t1 at time t0, the image captured by the first camera at time t1 is used as the first image, and the image captured by the second camera at time t1 is used as the second image. The previous first image at this time is The image captured by the first camera at time t0, the previous second image is the image captured by the second camera at time t0, and correspondingly, the previous stitching line is the stitching line calculation circuit 1204 according to the previous first image and the previous second image. Calculated splice line. The displacement limiting unit 150 generates corresponding displacement limiting data according to the previous splice line. The data processing circuit 1202 can comprehensively consider the color difference matrix, edge difference matrix, movement matrix and displacement limit data to generate the final cost map. For example, the data processing circuit 1202 may, after obtaining a cost map according to the color difference matrix, the edge difference matrix and the movement matrix as shown in FIG. 10, correct the cost map according to the displacement limit data. The cost value of the element is added to the movement cost corresponding to each element in the displacement limit data. Correspondingly, the splice line calculation circuit 1204 calculates and obtains the target splice line according to the revised cost map.

In one embodiment, the motion detection unit 110 is configured to perform motion detection on the first image for the overlapping area to obtain a first candidate motion area, and perform motion detection on the second image for the overlapping area to obtain a second candidate motion area. candidate moving regions, and fusing the first candidate moving regions and the second candidate moving regions to obtain the moving regions.

It should be noted that the motion detection method adopted by the motion detection unit 110 can be configured by those of ordinary skill in the art according to actual needs, which is not specifically limited here.

For example, see Figure 14. FIG. 14 shows a schematic diagram of a motion area detected by the motion detection unit 110 in FIG. 1 according to an embodiment of the present invention. The overlapping areas of the first image and the second image are respectively a right area of the first image and a left area of the second image. The right area of the first image and the left area of the second image are both divided into 4×4 16 pixel blocks according to the same division method. For the first image, the first candidate motion area detected by the motion detection unit 110 is composed of two pixel blocks as shown in FIG. 14 , and for the second image, the second candidate motion area detected by the motion detection unit 110 The area is composed of two pixel blocks shown in FIG. 14 . Correspondingly, the motion detection unit 110 fuses the first candidate motion area and the second candidate motion area to obtain a motion area composed of three pixel blocks as shown in FIG. 14 .

In one embodiment, the motion detection unit 110 is configured to acquire a previous first image that is the same as the first image field of view area, and calculate the pixel gray level difference between the first image and the previous first image for the overlapping area, and according to The first candidate moving region is detected by the difference of pixel gray level values.

For example, there are a first camera and a second camera with partially overlapping fields of view, and both the first camera and the second camera are in a shooting mode, and images are continuously captured. At the start time t0 when the first camera and the second camera start capturing, the image captured by the first camera at time t0 is taken as the first image, and the image captured by the second camera at time t0 is taken as the second image, Since the previous first image and the previous second image do not exist at this time, the detection of the moving area is not performed.

At the next time t1 at time t0, the image captured by the first camera at time t1 is used as the first image, and the image captured by the second camera at time t1 is used as the second image. The previous first image at this time is The image captured by the first camera at time t0, the previous second image is the image captured by the second camera at time t0. Correspondingly, the motion detection unit 110 calculates the difference of pixel gray level values of the first image and the previous first image for the overlapping area, and detects the first candidate motion area according to the difference of pixel gray level values.

By analogy, at a time after time t1, the motion detection unit 110 will perform motion detection in the above manner.

In one embodiment, the motion detection unit 110 is configured to divide the overlapping area into a plurality of sub-blocks, fuse the pixel gray-scale value differences in each sub-block to obtain a fused gray-scale value difference, and obtain a fused gray-scale value difference according to the fused gray-scale value. The sub-blocks whose differences are greater than the preset threshold generate a first candidate moving area. The specific value of the preset threshold can be configured by those of ordinary skill in the art according to actual needs, which is not specifically limited here.

As an optional embodiment, the motion detection unit 110 is configured to calculate the average gray-level value difference of the pixel gray-level value difference in each sub-block, and use the average gray-level value difference of each sub-block as each sub-block The fused grayscale value difference of the block.

It should be noted that the motion detection method of the motion detection unit 110 for the second image can be implemented with reference to the motion detection method of the motion detection unit 110 for the first image, which will not be repeated here.

Please refer to Figure 15. FIG. 15 shows a block diagram of the image processing wafer 10 according to an embodiment of the present invention. The present invention also provides an image processing chip 10 , which includes an interface unit 200 , a region determination unit 300 and an image splicing device 100 . The interface unit 200 is used to obtain a first image and a second image that need to be stitched. The area determination unit 300 is used for determining an overlapping area of the first image and the second image. The image splicing device 100 is used for splicing the first image and the second image according to the overlapping area to obtain a spliced image.

For example, the interface unit 200 may be a Mobile Industry Processor Interface (MIPI). The interface unit 200 can receive image data. For example, the interface unit 200 can receive two images with partially overlapping view areas shot by the same camera at different angles during the horizontal rotation, or can receive the respective images of two cameras with partially overlapping view areas. Two images were taken. After the interface unit 200 receives the image data, it can transmit the image data to the area determination unit 300 to identify the overlapping area.

For example, the interface unit 200 receives the first image and the second image partially overlapped by the field of view area captured by the same camera at different angles during the rotation process, and transmits the first image and the second image to the area determination unit 300 . Correspondingly, the area determination unit 300 identifies the overlapping area of the first image and the second image. In addition, the image splicing apparatus 100 may be the image splicing apparatus 100 provided by any of the above embodiments of the present invention.

In one embodiment, the interface unit 200 acquires the first image from a first surveillance camera, and acquires the second image from a second surveillance camera.

The present invention also provides an image splicing method, please refer to FIG. 16 . FIG. 16 shows a schematic flowchart of an image stitching method according to an embodiment of the present invention. The flow of the image stitching method is described as follows.

In step 510, motion detection is performed on the overlapping area of a first image and a second image to be spliced to obtain a moving area in which a moving object exists in the overlapping area.

In step 520, a target splicing line is calculated with avoiding the moving area as a constraint.

In step 530, the first image and the second image are spliced according to the target splicing line to obtain a spliced image.

In one embodiment, the image stitching method provided by the present invention further includes calculating at least one difference matrix between the first image and the second image for the overlapping area. Then, according to the difference matrix, the target splicing line is calculated with the constraints of minimizing the difference on both sides of the splicing line and avoiding the moving area.

In one embodiment, the image splicing method provided by the present invention further includes acquiring a previous splicing line calculated previously. The cost map is then revised based on the previously spliced lines.

It should be noted that, for the detailed description of the image splicing method, please refer to the relevant description of the image splicing device in the above embodiment, and details are not repeated here.

The image splicing device, the image processing chip, and the image splicing method provided by the embodiments of the present invention have been described in detail above. The principles and implementations of the present invention are described herein by using specific examples, and the descriptions of the above embodiments are only used to help understand the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. In conclusion, the content of this specification should not be construed as a limitation to the present invention.

10: Image processing chip 100: Image stitching device 110: Motion detection unit 120: Determine unit 1202: Data Processing Circuits 1204: Splice line calculation circuit 130: Splicing unit 140: Difference calculation unit 1402: Color Difference Calculation Circuit 1404: Edge Detector 1406: Edge Difference Computation Circuit 150: Displacement limit unit 200: interface unit 300: area determination unit 510~530: Steps

[FIG. 1] shows a first block schematic diagram of an image splicing device in an embodiment of the present invention; [FIG. 2] In an embodiment of the present invention, the motion detection unit in FIG. 1 performs motion detection on the overlapping area of the first image and the second image to obtain an example diagram of a motion area; [Fig. 3] shows an example diagram of the target splicing line calculated by the determining unit in Fig. 1 in an embodiment of the present invention; [FIG. 4] shows an example diagram in which the splicing unit in FIG. 1 splices the first image and the second image according to the target splicing line shown in FIG. 3 to obtain a spliced image in an embodiment of the present invention; [ Fig. 5 ] shows a second block diagram of an image splicing device in an embodiment of the present invention; [FIG. 6] shows a more detailed block diagram of the difference calculation unit in FIG. 5 in an embodiment of the present invention; [Fig. 7] shows an example diagram of a color difference matrix calculated by the color difference calculation circuit in Fig. 6 in an embodiment of the present invention; [FIG. 8] shows an example diagram of edge detection performed by the edge detector in FIG. 6 and an edge difference matrix calculated by the edge difference calculation circuit in FIG. 6 in an embodiment of the present invention; [Fig. 9] shows a block diagram of the determination unit in Fig. 5 in an embodiment of the present invention; [FIG. 10] shows an example diagram of a cost map calculated by the data processing circuit in FIG. 9 in an embodiment of the present invention; [FIG. 11] shows an example diagram of calculating the target splice line by the splice line calculation circuit in FIG. 9 in an embodiment of the present invention; [ Fig. 12 ] shows another example diagram of calculating the target splice line by the splice line calculation circuit in Fig. 9 in an embodiment of the present invention; [ Fig. 13 ] shows a schematic diagram of a third block of an image splicing device in an embodiment of the present invention; [FIG. 14] shows a schematic diagram of a motion area detected by the motion detection unit in FIG. 1 in an embodiment of the present invention; [ FIG. 15 ] shows a block diagram of an image processing chip in an embodiment of the present invention; and [ FIG. 16 ] is a schematic flowchart showing an image stitching method according to an embodiment of the present invention.

100: Image stitching device 110: Motion detection unit 120: Determine unit 130: Splicing unit

Claims (11)

An image splicing device, comprising: a motion detection unit for performing motion detection on an overlapping area between a first image and a second image at a current moment, to obtain a moving area with a moving object in the overlapping area ; A determination unit, with avoiding this moving area as a constraint condition, calculates a target splicing line in this overlapping area; A splicing unit, splicing this first image and this second image according to this target splicing line, obtains a splicing image ; A difference calculating unit, for this overlapping area, calculates and obtains at least one difference matrix between the first image and the second image; A displacement limit data is generated from a previous splicing line obtained; wherein, the determining unit calculates the target splicing line according to the at least one difference matrix and the displacement limiting data with avoiding the moving area as a constraint. The image splicing device according to claim 1, wherein the difference calculation unit calculates a plurality of the difference matrices of different types between the first image and the second image for the overlapping area according to a variety of different difference calculation methods. The image splicing device as claimed in claim 1, wherein the difference calculation unit comprises: a color difference calculation circuit, which calculates a color difference matrix between the first image and the second image for the overlapping area; an edge detector for performing edge detection on the first image and the second image respectively with respect to the overlapping area to obtain a first edge detection result corresponding to the first image and a first edge detection result corresponding to the second image two edge detection results; and an edge difference calculation circuit, which performs difference calculation according to the first edge detection result and the second edge detection result to obtain an edge difference matrix. The image splicing device of claim 3, wherein the edge difference matrix and the color difference matrix have the same shape. The image splicing device as claimed in claim 3, wherein the determining unit comprises: a data processing circuit, which obtains a cost map according to the color difference matrix, the edge difference matrix and the moving area; and a splicing line calculation circuit, according to The costmap computes the target splice line. The image splicing device as claimed in claim 5, wherein the splicing line calculation circuit is along an accumulation direction with a minimum accumulation value as a constraint condition for an element of the plurality of elements of the cost map and its adjacent elements in its adjacent rows Accumulate to get the target splice line. The image splicing device as claimed in claim 5, wherein the splicing line calculation circuit extends along an accumulation direction with a minimum accumulation value as a constraint condition among the plurality of elements of the cost map and its adjacent elements in the same row and adjacent rows One element of is accumulated to obtain the target splice line. The image splicing device according to claim 1, wherein the motion detection unit performs motion detection on the overlapping area according to a previous first image corresponding to the first image to obtain a first candidate motion area, and perform motion detection on the second image according to a previous second image corresponding to the second image to obtain a second candidate moving area, and fuse the first candidate moving area and the second moving area The candidate moving area gets the moving area. An image processing chip, comprising: an interface unit for acquiring a first image and a second image at a current moment; an area determination unit for determining an overlapping area between the first image and the second image; and an image A splicing device for splicing the first image and the second image according to the overlapping area to obtain a splicing image, the image splicing device comprising: a motion detection unit for performing motion detection on the overlapping area of the first image and the second image A moving area with a moving object in the overlapping area is obtained; a determining unit, taking avoiding the moving area as a constraint, calculates a target splicing line; a splicing unit, splicing the first image according to the target splicing line and the second image to obtain a spliced image; a difference calculation unit to calculate at least one difference matrix between the first image and the second image for the overlapping area; A previous splice line calculated at a previous time before the current time generates a displacement limit data; wherein, the determining unit calculates the displacement limit data according to the at least one difference matrix and the displacement limit data with avoiding the moving area as a constraint condition Target splice line. The image processing chip of claim 9, wherein the interface unit acquires the first image from a first surveillance camera, and acquires the second image from a second surveillance camera. An image splicing method, comprising: performing motion detection on an overlapping area between a first image and a second image at a current moment to obtain a moving area with a moving object in the overlapping area; calculating the overlapping area to obtain at least one difference matrix between the first image and the second image; a displacement restriction data is generated according to a previous splicing line calculated at a previous time before the current time; the constraint condition is to avoid the moving area , calculating a target splicing line according to the at least one difference matrix and the displacement limit data; and splicing the first image and the second image according to the target splicing line to obtain a spliced image.

Images