KR100526397B1 - Method for producing the global image mosaic - Google Patents

Abstract

The present invention relates to a global image mosaic method, and more particularly, to reduce an accumulation of errors generated when generating homographies between adjacent images. The present invention relates to a sequential structure of an algorithm, a fast execution time, a certain amount of system memory, and a direction. The present invention relates to a global image mosaic method suitable for real time applications by using a grid structure utilizing features such as uncertainty and similarity.

In the global image mosaic method of the present invention, in the method for mosaicing sequentially input images, the initial image among the structurally input images is set as the reference mosaic plane M, and the reference mosaic plane M Obtaining homographies (Q _i ) with images that are sequentially input may be sequentially combined with images input to the reference mosaic plane (M).

Description

Global image mosaic method {Method for producing the global image mosaic}

The present invention relates to a global image mosaic method, and more particularly, to reduce an accumulation of errors generated when generating homographies between adjacent images. The present invention relates to a sequential structure of an algorithm, a fast execution time, a certain amount of system memory, and a direction. The present invention relates to a global image mosaic method suitable for real time applications by using a grid structure utilizing features such as uncertainty and similarity.

As computer hardware technology makes it easier to output video from a computer, video is becoming an important information transmission medium that replaces and supplements existing media. These videos are distributed online and offline, and typically have a large capacity of tens to hundreds of Mb or Gb, which makes it difficult to store, transmit, search, and display the videos. Accordingly, various techniques for efficiently storing, transmitting, searching, and displaying moving images in a summarized and compressed form have been developed.

One such technique is the Image Mosaic method. Image mosaic is a technology that expresses all moving images as a single image by calculating a projected transformation between each image in a specific moving image.

Most image mosaics begin by obtaining homography between adjacent images in time. Homography refers to a 3X3 matrix representing the correspondence (ie warping parameters) of image points between two images when joining two adjacent images. After calculating the homographies between all adjacent images in a series of videos, the entire video sequence can be expressed as a single image by combining them successively. This image expression method is called Fairwise Local Registration. However, since the conventional pairwise local registration calculates and connects only homographies between adjacent images, an error is accumulated and the entire mosaic image does not show a clear result.

To solve this problem, a global mosaic method has been developed. According to a study on the global mosaic method, James Davis writes "Mosaics of Scenes with Moving Objects" IEEE Comp, Soc. Conf. In On Comupter Vision and Pattern-Recognition (CVPR1998), we proposed a method to reduce global error by constructing and calculating a linear system from all possible pairwise homographies. H. W. Kim (and H. S. Hong) also described "Soccer video mosaicking using self-calibration and line tracking", In Proc. In International Conference on Pattern Recognition, 2000, a method using line features and self-calibration was proposed. HY Shum (and R. Szeliski) also described patched based alignment in all overlapping regions of a panoramic mosaic in "Paroramic Image Mosaics", Technical Report MSR-TR-97-23, Microsoft Research, 1997. A method of reducing global errors by minimizing errors between ray directions has been proposed. H. W. Kim (and H.S. Hong) also proposed a global mosaic approach based on graph models in "Graph-based Global Registration of Dynamics Scenes", Journal of Electronic Imaging, 2001. Also Sawhney. H.S sought to minimize errors by directly obtaining homographies as mosaic planes, and Philip F. McLauchlan discloses the application of a bundle adjust.

However, the conventional global image mosaic methods such as the above, basically calculate the Pairwise Homography and then apply each of the proposed methods based on this to adjust the whole in the state including the accumulated error. do. Therefore, there is a fundamental problem that the image mosaic does not produce a clear result when there are many accumulated errors in the actual image sequence.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to reduce the accumulation of errors generated when generating homographies between adjacent images. In addition, the present invention provides a global image mosaic method suitable for real-time applications by using a grid structure that utilizes features such as directional uncertainty and similarity.

It is another object of the present invention to provide a global image mosaic method that can significantly reduce the accumulation of global errors by providing a pairwise global registration in a more advanced manner based on conventional feature point matching.

In the global image mosaic method of the present invention for achieving the above object, in the method for mosaicing the sequentially input image, the first image of the structurally input image is set as a reference mosaic plane, and the reference mosaic plane Obtaining homographies with sequentially input images, respectively, and sequentially combining the images with the reference mosaic plane.

Hereinafter, a global image mosaic method of the present invention will be described with reference to the accompanying drawings.

The present invention can reduce the accumulation of errors generated during homography between adjacent images, and uses a grid structure utilizing features such as sequential structure of algorithm, fast execution time, maintenance of a certain amount of system memory, directional uncertainty and similarity. The present invention relates to a global image mosaic method suitable for real time applications.

The method of the present invention can also drastically reduce the accumulation of global errors by providing pairwise homography based on conventional feature point matching but by an improved manner of pairwise registration. Such pairwise registration uses corner characteristics, similarity of matching, and directional uncertainty to safely use evenly distributed features throughout the image, and calculation is performed in a short time. In addition, by using the feature located on the line line has the effect that can use the line feature.

In addition, a new graph structure is presented, and a sequential shortest path search (SSPS) algorithm available on the graph is provided to find the shortest path and newly determine the matching position.

It is apparent that the method of the present invention is usually operable in a computer system capable of displaying moving images, and it is also possible to configure and operate a dedicated system to increase processing speed and accuracy.

First, a method for calculating homographies between adjacent images used in the method of the present invention will be described, and sequential shortest path search (SSPS) will be described.

Homography Hi is a matrix representing the transformation relationship between adjacent images (I _i-1 , I _i ). With respect to the adjacent i-th image I _i , the homography Hi with the previous image I _i-1 is expressed in a relationship as shown in Equation 1 below. In the method of the present invention, the newly proposed scheme is used based on hierarchical feature matching. Using a hierarchical structure has the advantage of speed improvement and local minimum.

The method for obtaining homography H _i includes the steps of: a) generating a Gaussian pyramid, obtaining approximate translation between the two images through frequency domain transformation at the lowest level, and b) calculating the pyramid level. Extracting feature points from the I _i-1 image while increasing the original image size, c) performing NCC for matching feature points with the Ii image, and d) matching feature points matched through the NCC. Obtaining homographies at the current level using a list square from the pairs of?

The Gaussian pyramid is one of the techniques of creating an image pyramid that scales down a given image to produce multiple images. It is a method of subsampling pixels after Gaussian filtering. to be.

First, a Gaussian pyramid is generated, and a rough translation between two images is obtained by transforming from the lowest level to the frequency domain.

Using the Gaussian pyramid as described above not only improves the speed, but also has the advantage of solving the disadvantage of falling into the local minimum when performed at the original image size.

After increasing the pyramid level to the original image size, the feature points are extracted from the I _i-1 image, and the matching position is found through the normalized cross correlation (NCC) with the I _i image. Obtain homographies (H _{i-1, i} ) of.

For convenience of calculation, if Equation 1 is assumed to be 1, Equation 2 can be derived, and Equation 2 must be satisfied for all P feature matching. Equation 2 derived from Equation 1 is as follows.

Here, H is expressed as in Equation 3 below.

Feature matching (feature matching) is obtained by using a least square method homo homography H _i to minimize the overall error, as shown in equation (4).

Feature point matching at the current pyramid level uses the values from the previous level, so one feature point extraction is performed at the lower level, and a new feature is extracted only at a portion having a small feature point distribution at the upper level.

In addition, a graph refers to a network structure composed of nodes and edges. An edge represents a path with a direction that connects a node to a node, and each edge has a cost from node to node. The shortest path refers to a path having a minimum total cost from one node to another node, and is a representative example of the Dijkstra algorithm searching for the shortest path.

In the present invention, a graph structure suitable for a problem is defined based on graph theory, and a sequential and fast shortest diameter etching method that can be used for real-time application is proposed and applied to a global mosaic.

First, the graph structure applied to the present invention is defined as follows.

1) The edge cost always has a value greater than or equal to zero.

2) The nodes of the graph are sequentially entered in the graph structure.

3) The start node of the graph is always the first incoming node, and all edges always have a direction from the first node to the next node.

4) The edge coast with existing nodes is calculated when the node comes in.

A sequential shortest path search (SSPS) algorithm for searching the shortest path for each node in the graph structure defined above is shown in FIG. 2. The algorithm may be written in a programming language such as C, C ++, assembler, Java, or the like.

In order to apply the above SSPS algorithm, it is basically assumed that N nodes come in sequentially. When the nth node is input, the nodes up to n-1th are arranged such that the total cost W of each node is in small order, and the number of nodes corresponding to the order is stored in the s [] array, and then the total cost W is Start the inner loop from the smallest node.

Next, calculate the edge cost d (s [i], n) between the node n currently selected in the inner loop and the selected node s [i], and add this to the final total cost W up to node s [i]. Store in C If C is less than the total cost W of the node at s [i + 1], set node s [i] as the previous node of the shortest path to the current nth node, and set C to node n. Save as the total cost, W, and wait for the next node to come in.

Before executing the inner loop, if C is less than the total cost W (s [i + 1]), because it is arranged in small order beforehand, if it passes through the node with the total cost after the s [i + 1] th Since you can't always have a value smaller than C, you don't need to check the nodes later to find the shortest path. If you need to check all nodes, select the node with the least C as the previous node.

3 shows an example of the shortest path search to which the SSPS algorithm is applied.

Assume that node A is a reference node and that new nodes come in in the order of B-C-D-E. The thick arrows in the figure represent the shortest paths and represent calculations that are not performed in the inner loop of the dashed line. Each edge is marked with d (s [i], n) in parentheses and the total cost to that node.

If it is assumed that the node E comes in after the node D is calculated, the comparison is performed in the order of total cost A-B-C-D, and the shortest path is A-E. Here, the result compared with node A is smaller than the total cost of node B in the next step, so that there is no need to compare with B, C, and D.

This SSPS algorithm is very efficient to find the shortest path in the graph structure defined above. Whenever a new node comes in based on the first node, the shortest path from the reference node can be found in a short time. In addition, if the edge cost of the graph needs to be calculated in the process, and the calculation time affects the entire search algorithm, it can probably reduce the computation amount by eliminating unnecessary calculations. It also has a sequential structure, which is very useful for real-time applications.

The method of the present invention using the homography and graph theory as described above: sets the first image of the structurally input image as a reference mosaic plane, and obtains the homographies of the reference mosaic plane and the sequentially input images, respectively. As the image is sequentially combined to the reference mosaic plane, it will be described in detail as follows.

1 is a flowchart illustrating the global image mosaic method of the present invention.

First, the first step is an initialization step. The first image is set to the reference mosaic plane M, and a grid having a predetermined interval is generated. Each grid is used to find a new match to correctly obtain homography Qi into the mosaic plane and has a graph structure as shown in FIG. 3.

The second step is the prediction step. The homograph Q of the current image is obtained by obtaining a (pairwise) homograph Hi with the previous image for the _i-th image and connecting the homograph Q _i-1 of the previous image. obtain and _i, and adds a new node in the graph on a grid present in the overlap between the earlier results by Q _i. The graph has the features defined above.

The third step is the observation step, which applies the SSPS algorithm to the overlapping graphs to find the new matching point in each grid of the newly input image and accurately calculate the homography Q _i . Paste the current image onto the mosaic plane through the newly obtained homography Q _i , and create a new graph for the newly created grid position.

Basically, since the first image is directly attached to the mosaic plane without error, by setting the first image as the reference mosaic image and pasting the incoming images sequentially while minimizing the error, a mosaic image that minimizes the overall error can be obtained. That is, in the graph structure of the present invention, minimizing an error with respect to a node in the first image always prevents errors from accumulating.

A method for obtaining homographies into a mosaic plane using the SSPS algorithm on a grid set in the mosaic plane is described.

When n images overlap at the same grid position, a graph having the number of overlapping nodes is generated for each grid. Each node has a path with the minimum cost from the reference node, a matching location from the reference grid, and so forth.

When a node corresponding to a new image is input, a new matching position on the mosaic plane and a path having a minimum cost to the newly input node may be determined by the SSPS algorithm. The factor to be determined according to the global mosaic situation in the SSPS algorithm is d (s [i], n). This may be determined by the NCC value N _cc .

Perform NCC in the patch in the image corresponding to each node in the grid and the corresponding range in the new image, set the position of the largest N _cc value as the new match, and set the value -log (N _cc ^max ). Specifies d (s [i], n). Applying this value to the SSPS algorithm, we can find the path with the lowest cost and the new matching position, and use this value to find the homographies into the mosaic plane.

In calculating d (s [i], n), since the NCC is used for all nodes, the execution time may increase as the number of images continues to increase. However, the SSPS algorithm has the advantage of finding the shortest path in a very fast time in the case of high probability that the inner loop stops at the initial stage for edge cost greater than or equal to zero.

When the new feature point matching (matching) is determined on each grid as described above, homographs into the mosaic plane can be obtained using the coordinates thereof in the same manner as in Equations 1 to 4 above.

However, in the case of using a grid with a constant spacing, the result may be worsened by a grid having poor corner characteristics. Therefore, you should give less weight to grids that adversely affect the whole.

In addition, in the case of a grid on a line feature, the uncertainty in the line direction is greater than the uncertainty in the vertical direction, and thus more weight in the vertical direction should be given. This can be determined by the eigenvalues and eigenvectors of the covariance matrix in each grid. By using this directional uncertainty, the grid on the line line can be used efficiently, so that the effect of using the line feature can be seen, and a very robust homograph can be obtained because the feature is evenly distributed throughout. In order to satisfy this condition, an energy function such as Equation 5 was defined, and homography to the mosaic plane was obtained to minimize the energy function.

U _p is represented by Equation 6 below.

Equation 5 represents the 2X2 covariance matrix in each grid, and Equation 6 represents the degree of similarity of the matching points between the two images in the form of a Cauchy function. Also it used a standard deviation σ of N _cc ^max value and the N _cc ^max values for the entire grid.

In Equation 6, the weight between the two matching points is given as a weight to give a large weight to the matching points (N _cc ^max ˜1). In addition, the use of Q _p has the effect of giving the directional weight according to the direction uncertainty as well as the corner characteristics of the grid itself.

If the order of the matching points for all P matching pairs and homograph H is rearranged using Equation 8, Equation 5 may be expressed as Equation 9. Using Equations 10 and 11, Homograph H can be easily calculated using the existing least squares only with W ^c and S ^c converted from W and S to C space.

Features in applying the method of the present invention in real time are as follows.

Since the proposed method is basically a sequential system, it can be easily applied to real-time applications. However, there are two things to consider in real-time applications. When a new image comes in, the execution time must be constant regardless of the total number of images, and the memory on hardware must be kept constant at all times. The proposed algorithm can satisfy the above two with little effect on performance.

That is, since the SSPS algorithm has an edge cost greater than or equal to zero, the probability that the inner loop stops initially for the aligned nodes increases. With this effect, we can always have only a few nodes where the cost to each node is minimal, and the nodes that are kept have a very small cost, which does not significantly affect the overall performance.

In addition, when a new image is entered and NCC is performed, patch images must be taken from the original images of the previous nodes. Therefore, if each node has the patch as data, there is no need to store all the images in memory.

Therefore, if you maintain k nodes in each grid, you can always keep system memory smaller than the number of grids x k x patch size in addition to the memory required for the overall mosaic result. Execution time can also be faster as the number of k decreases, and setting k to 2 does not affect the overall performance.

In addition, it is possible to shorten the time by not using a grid having bad line characteristics or corner characteristics when generating a grid at the beginning, and parallel processing is easy by using a grid with a predetermined interval.

4 shows four sequential image strings containing many line features. As shown in the figure, a line feature in which the screen is largely divided by vertical and horizontal lines is included. When grids having a predetermined interval are used for the images having the sequential image sequences as shown in FIG. 5, it can be seen that most grids do not have line characteristics or corner characteristics at all. These grids will find inaccurate matching through the NCC, which adversely affects the overall result. The result of global mosaic without writing direction uncertainty and similarity information is shown in FIG. 6. However, using the direction uncertainty and similarity information according to the present invention, that is, as a result of applying Equation 6, it can be seen that a clear image is obtained as shown in FIG.

In addition, the results of comparing the method of the present invention to those using conventional graph theory are shown in FIGS. 8 to 11. The patch size and grid spacing were both set to 15X15, and the maximum number of nodes, k, maintained per grid was set to 3. It is clear that the processing speed can be improved by adjusting the maximum number of nodes to an optimal state.

8 is a result image without global mosaic processing. As shown in the drawing, when the conventional graph theory is used, as shown by an arrow, the movement of the camera moves from left to right and then back to the left, thereby showing a problem that many errors accumulate on the far left in time.

Since the conventional method using the graph theory performs a global algorithm with the accumulated error, an accurate matching image is not generated when the accumulated error exceeds the search range of the global algorithm. Do not.

However, as shown in Fig. 10, when the method of the present invention is applied, the desired result can be obtained in a short time by continuously minimizing the accumulation of errors. 11 is a final image obtained by blending so that the boundary is not visible.

12 and 13 are graphs showing images having a size of 320 × 240 and time required for processing them. 12 shows processing results for 10 sequentially input image strings. On the left side, image results obtained by applying the conventional method are shown, and on the right side, result images of applying the method of the present invention are displayed. In processing 10 images that are sequentially input, the method according to the present invention takes about 10 seconds and the conventional method takes about 80 seconds, so the speed by the method of the present invention is very fast and finally obtained. It can be seen that the image is clearer.

According to the present invention, it is possible to reduce the accumulation of errors generated when generating homographies between adjacent images, and take advantage of features such as sequential structure of algorithm, fast execution time, maintenance of a certain amount of system memory, direction uncertainty and similarity. Using a grid structure can provide a global image mosaic method suitable for real-time applications.

In addition, by providing a pairwise global registration based on conventional feature point matching but in a more improved manner, a global image mosaic method that can dramatically reduce accumulation of global errors can be provided.

1 is a flowchart illustrating a global image mosaic method of the present invention.

2 is a coding example of the SSPS algorithm used in the present invention,

Figure 3 is an embodiment of the shortest path search using the SSPS algorithm,

4 is a view showing an image string having line characteristics;

5 is a diagram illustrating an image and a grid without global mosaic processing;

6 is a diagram of an image using no direction uncertainty, similarity information, and corner characteristics;

7 is a view showing a resultant image using the method of the present invention;

8 is a view illustrating an image without global mosaic processing;

9 is a view showing an image obtained by using a conventional graph theory,

10 is a view showing an image obtained by applying the method of the present invention,

FIG. 11 is a diagram illustrating a final image obtained by blending the image of FIG. 10. FIG.

12 is a view showing a resultant image by the conventional and the method of the present invention,

13 is a graph showing the calculation time according to the number of images.

Claims (10)

In the method for mosaic the sequentially input image, The first image of the structurally input image is set as the reference mosaic plane, and a) a Gaussian pyramid is generated for the i th image (I _i ) of the homography between the reference mosaic plane and the sequentially input image, Obtaining approximate translation between two images through frequency domain transformation at low level, b) extracting feature points from the I _i-1 image while increasing the pyramid level to the original image size; c) performing NCC for feature point matching with the Ii image, and d) previous image (I _i-1 ) at a current level using a list square from pairs of matched feature points matched through the NCC. ) To obtain a pairwise homograph (Hi) with the homograph (Q _i-1 ) of the previous image, and to obtain a homograph (Qi) of the current image by calculating with the homograph (Q _i-1 ) of the previous image, New nodes are added to the graph assigned to the grid of the overlapping part by Qi, and the new matching point in each grid of the newly input image by applying the shortest path search algorithm to the graph of the overlapping part grid. And collecting new homographies (Qi) using the collected matching points to sequentially combine the images with the reference mosaic plane. The method of claim 1, And dividing the reference mosaic plane into a plurality of grids having a predetermined interval, assigning a graph structure to each grid, and obtaining a match for a feature point in each grid. The method of claim 2, In the graph structure, 1) the edge cost always has a value greater than or equal to 0, 2) The nodes of the graph are sequentially entered in the graph structure, 3) The starting node of the graph is always the first incoming node, all edges always have a direction from the first node to the next node, and 4) already An edge coast with existing nodes is calculated when a node is newly entered. delete delete The method of claim 1, The shortest path search algorithm Global video mosaic method, characterized in that. The method of claim 1, Global image mosaic method, characterized in that by limiting the number of nodes of the graph in the grid to improve the execution time and maintain a constant system memory. The method according to any one of claims 1 to 6, And said shortest path search algorithm is executed in a graph satisfying the setting conditions as defined in claim 3. The method of claim 1, A method of global image mosaic, characterized by obtaining the homography (Qi) using the matching points obtained from each grid using the direction uncertainty, similarity, and corner characteristics of each grid. The method of claim 9, Using the direction uncertainty, similarity and corner characteristics of the grid, the equation is: Where U _p is  Global video mosaic method, characterized in that.