JP4287391B2 - Image matching device - Google Patents

Description

  The present invention relates to an image matching method and apparatus for detecting corresponding points from two images and performing image matching.

  In many technical fields such as motion detection, stereo matching, image morphing, image recognition, and moving image coding, image matching technology for obtaining a correspondence relationship from one image to another is a basic problem.

  According to “Non-Patent Document 1”, image matching techniques can be broadly classified into four. These are the optical flow method, block-based method, gradient method, and Bayesian method. The optical flow method derives an optical flow equation that “a change in luminance is constant” and obtains a flow using the optical flow equation as a constraint. The block-based method is a method for obtaining motion by template matching for each block. The gradient method is a method of performing matching in the direction in which the luminance gradient of the image decreases. The Bayesian method is a technique for obtaining probabilistic plausible matching.

“Patent Document 1” discloses an image matching method using a multi-resolution filter as a technique that does not belong to the above classification. This technique is a highly robust matching technique capable of matching a large motion to a small motion by generating a plurality of multi-resolution image pyramids using a plurality of multi-resolution filters and performing a matching process on the image pyramids in order from the top.
Japanese Patent No. 2927350 A. Murat Tekalp, “Digital Video Processing”, Prentice Hall, 1995

  However, such a conventional image matching technique has the following problems. The optical flow method has a problem that it is sensitive to noise and it is inherently difficult to cope with fast movement. In the block-based method, image matching is performed for each block in the image. However, there is a problem that it is difficult to essentially cope with the movement of the object in the image such as deformation or rotation. In the gradient method, since matching is performed in a direction in which the luminance gradient of the image decreases, there is a problem that it is difficult to stably search for the movement of the object. The Bayesian method has a problem that it is difficult to obtain a global optimum.

  On the other hand, the technique disclosed in Patent Document 1 requires a plurality of multi-resolution filters in principle. In addition, since static optimization including a static constraint condition is locally performed for each pixel, the energy of the entire screen is not always optimal.

  In a general conventional method, matching tends to become difficult as the correlation between frames decreases. Therefore, matching is difficult for an image that is greatly deformed or includes an object that moves quickly. Although the technique of Patent Document 1 is more robust, it is essentially a static local optimization, so there is no guarantee that the generated mapping is optimal for the entire screen. When image matching between the target image and the reference image is performed using the equation of motion, the calculation is heavy because the force received by the gradient of the image correlation potential energy is obtained for all grid points.

  An object of the present invention is to provide an image matching method and apparatus that reduce the amount of calculation of a motion equation in image matching for obtaining a correspondence relationship between a target image and a reference image.

In the first aspect of the present invention, in image matching for obtaining a correspondence relationship between a target image and a reference image, a plurality of target grid points are set in the target image, and target grid points on the target image are set in the reference image. Setting reference grid points corresponding to each one-to-one, dividing the target image into a plurality of clusters including at least one target grid point, and determining representative target grid points of each cluster; Pixel information of a representative reference grid point that is a reference grid point corresponding to the representative target grid point on the reference image, pixel information of the representative target grid point, and positions of the representative reference grid point and the representative target grid point Calculating the image correlation force received by the representative reference grid point by the gradient of the image correlation potential energy determined by the relationship, and the target grid belonging to each cluster Setting the image correlation forces acting on the reference grid points corresponding to each of the image correlation forces calculated for the representative reference grid points corresponding to the representative target grid points of the cluster, on the reference image For each reference grid point, the image correlation force acting on the reference grid point, and the elasticity that the reference grid point receives by the elastic energy between the reference grid point and another reference grid point adjacent to the reference grid point A model generation step of constructing an equation of motion related to the reference grid point using a force and a frictional force acting on the reference grid point; and an equilibrium state of each of the reference grid points by numerically analyzing the motion equation And a numerical analysis step for obtaining an image matching method.

According to a second aspect of the present invention, in the image matching device for obtaining a correspondence relationship between a target image and a reference image, a plurality of target grid points are set in the target image, and target grid points on the target image are set in the reference image. Setting means for setting a reference grid point corresponding to each of the first and the second, and a determination means for dividing the target image into a plurality of clusters including at least one target grid point and determining a representative target grid point of each cluster And pixel information of a representative reference grid point that is a reference grid point (y) corresponding to the representative target grid point on the reference image, pixel information of the representative target grid point, and the representative reference grid point and the representative Calculating means for calculating the image correlation force received by the representative reference grid point by the gradient of the image correlation potential energy obtained by the positional relationship with the target grid point; and a pair belonging to each cluster Setting means for setting the image correlation force acting on the reference grid point corresponding to each of the grid points to the image correlation force calculated for the representative reference grid point corresponding to the representative target grid point of the cluster; and the reference For each reference grid point on the image, the reference grid point by the image correlation force acting on the reference grid point, and the elastic energy between the reference grid point and another reference grid point adjacent to the reference grid point Model generating means for constructing an equation of motion related to the reference lattice point using the elastic force applied to the reference lattice point and the frictional force acting on the reference lattice point, and each of the reference lattice points by numerically analyzing the equation of motion There is provided an image matching device characterized by having numerical analysis means for obtaining an equilibrium state of the image.

Representative target grid points to calculate the force F _u by the image correlating potential energy within each cluster to determine, is allocated a force F _u in each representative object grid points as a force other points in the cluster, the amount of calculation becomes heavy The calculation of the force Fu by the image correlation potential energy does not need to be performed for all the grid points, and the calculation time is shortened.

[First Embodiment]
The first embodiment will be described with reference to the block diagram of FIG.
According to FIG. 1, for example, the image matching unit 11 configured by a processor including a memory or the like includes a model generation module 12 and a numerical analysis module 13. When two images of the target image and the reference image are input to the image matching unit 11, the same part of both images in the frame memory, for example, the same part of the 3D image is matched. That is, when two images are received, a dynamic model is generated using a dynamic concept. Since this dynamic model is output from the model generation module 12 in the form of an ordinary differential equation, the numerical analysis module 13 repeatedly solves the output by a general numerical solution. The resulting image obtained at the final iteration is the final matching state and is output as a mapping. In the present embodiment, for example, a representative target lattice point determination module 14 and an image correlation potential energy calculation module 15 configured by a processor are provided, and an image is divided into a plurality of clusters as described later, and pixels of each cluster are provided. The force due to the image correlation potential energy is calculated using at least one of the representative target grid points, and this calculated value is applied to the equation of motion for each pixel in the cluster.

  In the present embodiment, a target image and a reference image are input, and a mapping relationship between these images is obtained. In this embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system. First, the basic image matching step will be described. Consider the following model as the target image.

Continuous image model

  This is a real-based continuous image model. Since the digital image is considered here, the following model obtained by sampling the above model is used.

Sampling image model

(However, in this formulation, it is assumed that the image values of the same object do not change with time). In addition, since the motion vector d is a real number, it should be noted that the right side uses the notation of the continuous image model (Equation (1)). Here, since image matching between the target image, the reference image, and the two images is considered, the following equivalent problem is considered.

Image matching problem

  Since x = Vn, x uniquely corresponds to a point n on the lattice space. Since map g is a unique map, y also uniquely corresponds to x. Therefore, y uniquely corresponds to n. This is illustrated in FIG. That is, the space to be handled here is a deformed lattice space corresponding to one-to-one by the point n on the lattice space.

Since y = g (x) = g (Vn) as described above, this is redefined as the following equation (8) in order to make it easy to understand that one-to-one correspondence with n.

Image matching problem

In order to solve Equation (9), dynamics is introduced for the point y _n here. In other words, the image matching problem is reduced to the problem of solving the dynamic system about the point y _n . This image is shown in FIG. The point y _n moves to a state satisfying Equation (9) while considering the relationship with surrounding points, and converges to an equilibrium state. Assume that the image matching problem is completed with the equilibrium state.

A new time axis τ∈R is introduced for the point y to define the function y _n (τ). Here, the initial value is assumed to be the same as the square lattice x as shown in the following equation (10).

Since a new time axis is introduced, the derivative with respect to time can be defined as in the following equation (12).

Ordinary dynamics are described by an ordinary differential equation such as the following equation (13).

Where F∈R ² is the sum of forces. This is also called the equation of motion.

Next, consider the force applied to y _n (τ). First, consider the potential energy that drives the dynamics. This is a force for moving y _n (τ) to a state satisfying Equation (9). When the mathematical formula is transformed, the mathematical formula (14) is obtained.

However, it is difficult to search for a point that normally satisfies Equation (14) because of noise components included in the image. Therefore, an energy function as shown in the following equation (15) is considered, and a point where the energy function Eu is minimized is searched.

If the principle of steepest descent is used, the local minimum can be reached by descending in the direction of steepest descent of the energy function Eu around y _n (τ). Therefore, the gradient in the direction of the steepest descent is defined as the force against y _n (τ). Since the energy function Eu can also be considered as image correlation, this force is defined as a force Fu due to image correlation potential energy.

There are various methods for calculating the gradient in the direction of steepest descent. Here, the following method is adopted. As shown in FIG. 4B, the gradient in the steepest descent direction is obtained directly by local optimization. Although the image model Sc is a continuous image model, only the sampled image model Sp can actually be used. Therefore, local optimization is also performed based on the sampled image model. Since the sampling point closest to y _n (τ) is to be used as the local space center y _c , the following equation (16) is obtained.

When local optimization is performed to obtain a vector in that direction, and the vector is normalized and multiplied by the magnitude of the gradient, the following equation is obtained.

Force (square error energy) by image correlation potential energy (19)

A force (absolute value difference error energy) (formula (21)) based on image correlation potential energy in which an energy function is defined as the following formula (20) can be used for ease of handling in mounting.

Next, consider the ability to describe the relationship with surrounding points. Assume that the matching target image is a two-dimensional projection of a three-dimensional space. If the object in the three-dimensional space is a rigid body, the surface of the rigid body is observed as an object in the two-dimensional image. If an object in the three-dimensional space is observed in the target image and the reference image, there is a high probability that the phase of the object observed in each image will be maintained. As shown in FIG. 5, the positional relationship of the point x on the target image object should be maintained at the point y _n (τ) on the reference image object. This property can be simulated by connecting the points y _n (τ) with a spring. The relationship with the periphery is described by the spring force F _k . First, a lattice point space N _n around the target point is defined as follows. If there are four surrounding points, the following equation (22) is obtained.

If the spring constant (elastic constant) is k, the restoring force of the spring applied to the point y _n (τ) is the spring force (elastic force) represented by Expression (23). The elastic constant is a balance between the image correlation energy and the elastic energy. If the elastic constant is large, the elastic constant is difficult to deform and the result is stable. However, the compatibility with the image is deteriorated. If the elastic constant is small, the image can be easily deformed, so that the adaptability of the image is improved. However, the results are too flexible. So, at present, this parameter is given empirically. Since the behavior is not so sensitive to the value of this parameter, it is basically given a fixed value.

Spring force

Specifically, when the four surrounding points are connected, the following equation (24) is obtained.

4-point connection spring model

Finally, consider the power to dissipate the stored energy. If the force applied to y _n (τ) is only F _u and F _k , the energy is conserved and the system is in a steady state where it vibrates. Therefore, the power to dissipate the stored energy is introduced. Frictional force can be used for this. When the speed can be approximated to be constant, the frictional force can be described by the following equation (25).

Frictional force

Summarizing the above forces, the equation of motion is as shown in the following equation (26).

Equation of motion

ODE a force F _u by the image correlation potential energy is not solved analytically (26) can not be solved analytically. Therefore, it is difficult to take the limit at τ → ∞ of the system. Therefore, a sufficiently large time T for the system to converge is considered, and the convergence state of the system is estimated by calculating the t = (0, T) interval by numerical analysis.

  If the initial value of the ordinary differential equation is determined, a solution can be uniquely obtained by numerical analysis. In general, it is called the initial value problem of ordinary differential equations. There are many numerical solutions to this problem, but famous ones include the Euler method, the Runge-Kutta method, the Birsch store method, the predictor / corrector method, and the hidden Runge-Kutta method. The Runge-Kutta method is the most famous and frequently used. However, since Equation (26) has dimensions corresponding to the size of the image, a complicated numerical solution method is difficult to adapt. Therefore, here we consider applying the Euler method, which is the simplest implementation.

  Since the Euler method is a numerical solution for a first-order ordinary differential equation, first, Equation (27) is applied to Equation (26) to convert Equation (26) to a first-order ordinary differential equation. Thereby, the converted equation of motion (28) is obtained.

Variable conversion

Transformed equations of motion

The Euler scheme for the ordinary differential equation (29) is represented by equation (30).

This advances the solution from t ⁽ⁿ⁾ to t ^{(n + 1)} ≡t ^{(n + 1)} + h. Here, x ⁽ⁿ⁾ indicates n steps, and h is a step width. When the Euler scheme is applied to the equation (28), an update equation of the following equation (31) is obtained.

Renewal formula by Euler method

Therefore, in order to eliminate waste of these calculations, representative subjects is divided into clusters of arbitrary shape the entire screen as shown in FIG. 6 (divided pieces), to calculate the force F _u by the image correlating potential energy within each cluster The grid point ● is determined, and the force _Fu at each representative target grid point is assigned as the force of another point in the cluster. That is, when you decide the representative target grid point of the first pixel y ₅ as in the following equation (32) obtains the force F _u in the representative object grid points (y _5), the other pixels y1-y4, y6- for yn substitutes F _u of the representative target lattice point (y _5).

If the representative target grid point is shifted for each step, it can be approximated that the force _Fu is calculated for all points in a certain period. A square block can be used for an arbitrarily shaped cluster. For example, a 2 × 2 square block or a 4 × 4 square block can be used. The cluster may be a rectangle instead of a square.

The representative target lattice point is determined in the representative target lattice point determination step. The cluster is set to a 2 × 2 square block, and the index of the lattice point in each cluster is given as in the following equation (33).

  For example, a method of periodically changing the method of determining the representative target grid point according to the time step as shown in FIG. Specifically, the representative target grid point is determined by the following equation (34).

Representative object grid point determination method 1

Here, mod (.) Calculates the remainder.

Moreover, you may determine by following Formula (35) using a uniform random number.

Representative object grid point determination method 2

Here, uni_rand (.,.) Is an operator that generates a uniform integer random number in the argument range.

  As described above, when image matching is summarized as an algorithm, the representative target lattice point determination method 2 is as shown in the flow of FIG.

Image matching algorithm:

  In this embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system.

[Second Embodiment] (use of area division)
This embodiment will be described with reference to the block diagram of FIG. In the present embodiment, similar to the first embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system. . In the first embodiment, clusters are described using blocks, but in this embodiment, a cluster that is more suitable for a picture is used by using a region division technique.

  The basic configuration is the same as that of the first embodiment, but an area dividing module 16 is added to the first embodiment. That is, a region dividing step is added in the image matching step. In the first embodiment, a block such as a 2 × 2 square block is used as a cluster having an arbitrary shape. However, in this embodiment, a cluster suitable for a picture is constructed on the basis of the segmentation pieces.

  There is one problem with block clusters. FIG. 10 shows an example of a block type cluster existing at the boundary between the object and the background. This is an image with the object moving to the left and the background moving to the right. At that time, the lattice energy on the object generates potential energy in the left direction, and the lattice point on the background generates potential energy in the right direction. Further, the method of determining the representative target lattice point uses the method of Equation (34). At this time, the force due to the potential energy generated in the cluster periodically repeats the right direction and the left direction as shown in FIG. Considering the integration in the time direction, the cluster is equivalent to applying a slight force in the left direction. This is not a good boundary behavior.

  This problem arises because there are clusters that span the boundaries of motion. In the case of a block-type cluster that is uniform on the screen, there is a high probability that a cluster across this boundary will occur.

  When considering a normal moving image such as a captured moving image, the boundary of movement often overlaps the edge of the image. It can be considered that the edge region of the image is a necessary condition for becoming a boundary of motion. Therefore, as shown in FIG. 11, dividing the area along the edge of the image is a necessary condition for the area dividing means.

  Fast watersheds method [L. Vincent and P. Soille; “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations”, IEEE Trans. On Pattern Analysis and Machine Intell., Vol.13] , No. 6, 1991]. This is a technique in which a luminance gradient image (edge extracted image) is obtained from a luminance image to be processed by a difference operator, and a closed region surrounded by a ridge of the luminance gradient obtained there is divided as one region. The Fast Watershed method is a method of dividing a closed region surrounded by a ridge of a luminance gradient image, that is, an edge, as a single region, and thus a cluster and an edge are easily matched with each other. The Fast Watersheds method also considers the calculation speed, and the impact of the calculation amount is small compared to the entire image matching step.

  Next, the representative target lattice point determination step will be described. Since the clusters divided by region division vary in size and shape, it is difficult to take the method as in the first embodiment (formula (34)) in which the index is periodically changed. Therefore, a determination method is used in which the index is randomly changed as shown in Equation (35).

As shown in FIG. 12, the index of each lattice point in the cluster is assigned in order from the upper left point to the lower right direction. Assuming that a certain cluster is c _i , the representative target lattice point determination method can be described as the following equation (36).

Representative object grid point determination method 3

Here, N (.) Is an operator that calculates the number of grid points in the cluster.

  Experiments with test patterns yielded successes and failures as shown in FIG. The representative target grid point is generated by a random number, but depending on how this random number is generated, it may or may not go well. This is considered to be because there are portions where the potential at the representative target lattice point appears properly and portions where it does not. Therefore, if the part that can calculate the potential well is selected as the representative target grid point, the result will be successful, otherwise it will fail. This can be improved by increasing the number of representative target grid points in the cluster.

  There are large and small clusters. Therefore, the number of representative target grid points is obtained from the ratio in the cluster. In this way, it is possible to estimate the calculation amount of the entire screen regardless of the number and size of clusters. Assuming that the ratio of lattice points to be representative representative object table points in the cluster is α, the number of representative object lattice points can be described by the following equation (37).

Number of representative target grid points

For example, the force F _u by the image correlating potential energy calculated for each representative object grid points according to the following equation (38), and the average thereof as the representative force F _u cluster c _i with.

Force F _u by representative image correlation potential energy

  FIG. 14 shows the results of four experiments with α = 0.2. It can be seen that the map can be calculated stably.

  As described above, when image matching is summarized as an algorithm, the representative target lattice point determination method 2 is as shown in the flow of FIG.

Image matching algorithm:

  In this embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system.

[Third Embodiment] (Recalculation if distance is long)
This embodiment will be described with reference to the block diagram of FIG. In the present embodiment, similar to the first embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system. . In the first embodiment, the representative target grid point in the block is fixed to one, but in the present embodiment, the representative target grid point is adaptively selected according to the state between the grid points. That is, the representative target grid point determination module 114 has a function of determining an additional representative target grid point.

  As shown in FIG. 17, in the lower right pattern or the like, the lattice points in the cluster may exist at positions separated from each other. Despite this situation, if the calculation is performed with only one representative target grid point in the cluster, the result is distorted. Therefore, in this embodiment, a method is used in which a lattice point in the cluster having a large distance is used as a representative target lattice point as in this pattern.

The representative target grid point determination step will be described. First, the representative representative lattice point determination module 14 determines the initial representative target lattice point according to the method of Equation (34) or Equation (35). The representative target grid point and x _d = Vn _d. At this time, the mapping point is y _nd = g (x _d ). At this time, if the target point to which the potential force is assigned is x = Vn, the mapping point is y _n = g (x). Discontinuous points that should be added to representative target grid points without assigning potentials are points connected by dotted lines in FIG. 17 (recalculation without assigning). Such a point is defined, for example, by the following equation (39).

Determination of additional representative target grid points

  This is a point where the distance ratio between lattice points exceeds the constant α.

  As described above, when image matching is summarized as an algorithm, the representative target lattice point determination method 2 is as shown in the flow of FIG.

Image matching algorithm:

  In this embodiment, the difficulty of the prior art is solved by using the feature that the entire screen converges to the natural optimum state of the system by capturing the entire screen as a dynamic system.

  In the above embodiment, when the ratio between the distance between the representative target grid point and the target grid point and the distance between the representative reference grid point on the reference image mapping the representative target grid point and the reference grid point is greater than a certain value. Calculates the force received by the gradient of the image correlation potential energy determined by the positional relationship between the target lattice point and the reference lattice point, not the force received by the gradient of the image correlation potential energy at the representative target lattice point.

1 is a block diagram of an image matching apparatus according to a first embodiment of the present invention. The figure which shows the connection of each space in the image Diagram showing dynamics for image matching problem yn Image showing force by image correlation potential energy The figure which shows the state by which the phase of the object on a target image and the object of a reference image is maintained Figure showing an image divided into clusters The figure explaining the method of selecting a representation target lattice point The flowchart figure of the image matching algorithm by 1st Embodiment Block diagram of an image matching device according to a second embodiment of the present invention Diagram for explaining the problem of block type cluster Diagram for explaining representative model of potential in cluster Diagram showing indexes in a cluster Diagram showing examples of successful and unsuccessful image matching The figure which shows the image matching result when α is set to 0.2 Flowchart diagram of an image matching algorithm according to the second embodiment Block diagram of an image matching device according to a third embodiment of the present invention The figure which shows 3rd Embodiment notionally Flowchart diagram of an image matching algorithm according to the third embodiment

Explanation of symbols

11 ... Image matching unit, 12 ... Model generation module, 13 ... Numerical analysis module,
DESCRIPTION OF SYMBOLS 14 ... Representative object grid point determination module, 15 ... Image correlation potential energy calculation module, 16 ... Area division module, 114 ... Representative object grid point determination module

Claims (7)

In the image matching device for obtaining the correspondence between the target image and the reference image,
Setting means for setting a plurality of target grid points in the target image, and setting a reference grid point corresponding to each of the target grid points on the target image in a one-to-one manner in the reference image;
Determining means for dividing the target image into a plurality of clusters including at least one target grid point, and determining a representative target grid point of each cluster;
On the reference image, pixel information of a representative reference grid point that is a reference grid point corresponding to the representative target grid point, pixel information of the representative target grid point, and the representative reference grid point and the representative target grid point Calculation means for calculating the image correlation force received by the representative reference grid point by the gradient of the image correlation potential energy determined by the positional relationship;
The image correlation force acting on the reference lattice points corresponding to each of the target lattice points belonging to each cluster is set to the image correlation force calculated for the representative reference lattice points corresponding to the representative target lattice points of the cluster. Setting means;
For each reference grid point on the reference image, the reference by the image correlation force acting on the reference grid point, the elastic energy between the reference grid point and another reference grid point adjacent to the reference grid point Model generation means for constructing an equation of motion related to the reference grid point using the elastic force received by the grid point and the frictional force acting on the reference grid point;
Numerical analysis means for obtaining an equilibrium state of each of the reference lattice points by numerically analyzing the equation of motion;
An image matching apparatus comprising: The image matching apparatus according to claim 1, wherein the determining unit sets the cluster as a uniform block over the entire screen. The image matching apparatus according to claim 1, wherein the determination unit divides a region based on the image information of the target image, and sets a result of the region division as a cluster. The image matching apparatus according to claim 1, wherein the determination unit performs region division based on luminance information of the target image. The image matching apparatus according to claim 1, wherein the determination unit performs region division based on a color similarity of the target image. The calculation means includes a distance between the representative target grid point and the target grid point, and a ratio of a distance between the representative reference grid point and the reference grid point on the reference image mapping the representative target grid point. When the value is larger than a certain value, it is received not by the force received by the gradient of the image correlation potential energy at the representative target grid point but by the gradient of the image correlation potential energy obtained by the positional relationship between the target grid point and the reference grid point. The image matching apparatus according to claim 1, wherein the force is calculated. The determining means determines a plurality of representative target grid points according to the size of the cluster, and the calculating means calculates a force received by the gradient of the image correlation potential energy for each representative target grid point, and a plurality of calculation results The image matching apparatus according to claim 1, wherein an average of the values is a representative power of the cluster.

Images