JPH05290172A - Pattern matching method for image - Google Patents

Abstract

PURPOSE:To eliminate the need to extract a feature quantity and to estimate conversion parameters in a short time by converting a reference image by density gradation conversion and geometric conversion and then estimating an image so that the reference image matching with an input image. CONSTITUTION:The image conversion is represented with an unknown pattern vectors (x) consisting of conversion parameters x1-x6 Then an area wherein variation in the density of the estimated image exceeds a specific value is selected. In the area, intermediate parameters by the partial differentiation of the estimated image as to the respective parameters are calculated and a normal equation is found from residual to calculate the corrected vector DELTAx' of the unknown parameter vector x<(k)>. Then it is decided whether or not the image is converged by using the degree of coincidence between the estimated image based upon the unknown parameter vector and the input image. When not, similar processing is repeated to approximate the reference image to the input image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pattern matching method for matching patterns of two images which are similar to each other when one image is converted.

[0002]

2. Description of the Related Art Conventionally, the following method is known as a pattern matching method for converting one image and matching the other image when two similar images are given. The first method is to detect feature points such as corners of an image by preprocessing, find correspondences between the feature points, and then estimate conversion parameters by the least square method, relaxation method, graph theory, or the like. In this method, the extraction of the feature points may not be guaranteed, and the association itself has been a problem.

As another method, a direct comparison method of directly comparing two images can be considered. In this method, image conversion is actually performed on the reference image, and the correlation coefficient with the input image and the like are calculated to find the conversion parameter that most matches.

[0004]

However, since the direct comparison method does not give a guideline for the change of parameters, in the worst case, the degree of coincidence must be calculated for all combinations of parameters. Therefore, there is a drawback that a huge amount of calculation may be required as the number of parameters increases.

The present invention has been made in view of such conventional problems, and does not require extraction of a feature amount, and statistically obtains a guideline for a change in a parameter, thereby reducing the calculation amount and shortening the calculation. The technical task is to be able to estimate the conversion parameter in time.

[0006]

The present invention provides a first image y _M.
(U ′, v ′) (≡A), and the second image y (u, v)
Hold (≡C) as pixel unit data, the brightness ratio is x ₁ , the rotation angle (rad) about the origin is x ₂ ,
The translation amount (pixel) in the horizontal and vertical directions is x ₃ , x
₄ , the scaling ratios in the horizontal and vertical directions are x ₅ and x _{6, respectively.}
A pattern matching method of an image for performing pattern matching to match the second image by converting the first image based on a conversion parameter that is u ′, v ′) and the partial differential image E (u ′, v ′) of the first image in the y direction are calculated, and x = [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , x ₆ ] ^T
Is an unknown parameter vector having conversion parameters x _{1 to} x ₆ as elements, and the partial differential for each element of the unknown parameter vector of the estimated image y ′ (u, v) is represented by Expression (1) and corresponds to the second image. Intermediate parameters G, H, K, L and F, M, N, O, respectively for each pixel (u, v)
P, Q, and R are calculated by the equations (2) to (8), and the normal equations (the equation (9)) are calculated from these intermediate parameters F to Q.
The coefficient matrix B and the column vector b that form
3), select a region in which the density change of the estimated image that satisfies the equation (14) is large, and solve the normal equation (9), so that the estimated number of times is k and the correction vector Δx ′ ^(k ′ of the unknown parameter vector x is ⁾ Is solved, and the correction vector Δx ′ ^(k) is added to the ^kth unknown parameter vector x ′ ^(k) to obtain k
The + 1st-order unknown parameter vector is calculated, the convergence of the estimated image obtained by converting the first image by the unknown parameter vector and the second image are determined, and if they are converged, this conversion parameter is used as the output parameter. If not converged, the intermediate image parameter is calculated again to calculate the conversion parameter to perform the image pattern matching.

[0007]

According to the present invention having such characteristics, the image conversion is represented by the unknown parameter vector x consisting of the conversion parameters x _{1 to} x ₆ . Then, an area in which the density change of the estimated image exceeds a predetermined value is selected. Then, in that region, an intermediate parameter by partial differentiation of the estimated image for each parameter is calculated, a normal equation is obtained from the residual, and a correction vector Δx ′ of the unknown parameter vector x ^(k) is calculated.
^(k) is calculated to calculate the unknown parameter vector x. Then, it is determined whether or not the image is converged based on the degree of coincidence between the estimated image based on the unknown parameter vector and the input image, and if not converged, the same process is repeated to bring the reference image closer to the input image. ..

[0008]

EXAMPLES The principle of the present invention will be described. First, consider a reference image, that is, a reference image. 2A is an example of the reference image, while FIG. 2B is an input image. Both are images input to the computer after being photographed by a television camera or the like. Here, it is assumed that the image is estimated so as to match the input image by performing some image conversion operation such as rotation or parallel movement on the reference image, and is approximated to the input image as shown in FIG. Here, density gradation conversion and geometric conversion are handled as image conversion. The density gradation conversion considers a linear conversion with x ₁ as an unknown parameter, and the geometrical conversion considers an affine conversion with the following x ₂ to x ₆ as an unknown parameter. Here, x ₁ to x ₆ are defined as follows. x ₁・ ・ ・ Brightness ratio x ₂・ ・ ・ Rotation angle around the origin (rad) x ₃ , x ₄・ ・ ・ Horizontal and vertical translation amount (pixel) x ₅ , x ₆ ... A ratio of scaling in the horizontal and vertical directions When the reference image is converted using these conversion parameters x _{1 to} x ₆ , values that best match the input image, x ₁ ′ to x ₆ ′ The purpose is to estimate.

Next, a nonlinear regression equation is derived using the following coordinate transformation. Here, as shown in FIG. 2B, the coordinates of the actual input image are (u, v), and the density of the input pattern at the point (u, v) is y (u, v). Also (u, v)
Is the image after conversion by the estimated value, that is, the estimated image y ′.
It also represents the coordinates of (u, v). Further, (u ′, v ′) is the coordinates before conversion with respect to the image (u, v), that is, the reference image y.
Let it be the coordinates of _M (u ', v').

First, the reference image is enlarged. FIG. 3A shows the reference image, and FIG. 3B shows the reference image enlarged in the horizontal and vertical directions. When the reference image is enlarged 1 / x ₅ and 1 / x ₆ times in the horizontal direction and the vertical direction, respectively, the following equations (17) and (18) are established.

[Equation 16]

Next, when the reference image is translated in the horizontal and vertical directions by -x ₃ and -x ₄ (pixels) respectively as shown in FIGS. 3B to 3C, the following equation (1)
9) and (20) are established.

[Equation 17]

Next, this reference image is rotated clockwise about the origin by x ₂ (rad). FIG. 3D shows a state in which the reference image is rotated. X around the origin of the reference image
When rotated by ₂ (rad), the following equations (21) and (22) hold.

[Equation 18]

When the brightness of the reference image is multiplied by x ₁ as shown in FIG. 3 (e), the relationship between y _M (u ', v') and y '(u, v) is given by the following equation (23) ).

[Formula 19]

In this way, the reference image is enlarged 1 / x ₅ and 1 / x ₆ times in the horizontal and vertical directions, respectively, and translated in the horizontal and vertical directions by -x ₃ and -x ₄ (pixels), respectively, and the origin is obtained. When rotating around x ₂ (rad) and multiplying the brightness by x ₁ , the following equation (24) is established.

[Equation 20]

Therefore, y _M (u ', v') and y '(u,
The relationship of v) is shown by the following expression (25).

[Equation 21] Y ′ (u, v) given by this equation is the input image y
Parameter x ₁ ′ when it best matches (u, v)
And thus we continue to estimate the ~x ₆ '.

The regression equation of equation (25) is non-linear,
The Gauss-Newton method, which is a linear approximation iterative improvement method, is used as the least squares method for this nonlinear model. Unknown parameter vector x ' ^(k) obtained by the completion of the k-th iteration (estimated value)

[Equation 22] A small amount around x Δ ^(k)

[Equation 23] When the equation (25) is expanded by Taylor, it can be expressed as the equation (28) by a first-order approximation. However, formula (29)
~ (31).

[Equation 24]

[Equation 25]

As a result, assuming that the input image is y (u, v), the linear approximation regression equation is expressed by the following equation (32).

[Equation 26] Then, for all the pixels on the input image to be compared with the reference image, that is, for all combinations of (u, v), equation (3
Generate the observation equation of 2). Then, a normal equation is set up to obtain a correction vector Δx ′ ^(k) , and further, an estimated value of the (k + 1) th order is obtained by the following equation (33).

[Equation 27]

Next, selection of a region for which the estimation process is performed will be described. Here, the density distribution for a specific row of the image is shown in FIG. In this figure, the input image is shown by a solid line and the estimated image is shown by a broken line. Regions W1, W3, and W5 show a state in which neither the input image nor the estimated image changes in the u direction, W2 has the input image, and W4 has the density gradient only in the estimated image. Considering such a one-dimensional image, Δy ^(k) is almost 0 because the input image and the estimated image match for the pixels in the range of W1 and W5. Further, since Δy ′ ^(k) (u, v) is constant, a ₂ ^{(k) to} a ₆ ^(k) represented by partial differentiation thereof are almost 0.
Becomes Therefore, the pixels within the range of W1 and W5 are expressed by the formula (32)
To Δx ′ ^(k) , there is no relation to parameter estimation. For pixels within the range of W2 and W3,
_{^{2 (k) ~a 6 (k}} ) is substantially 0 but, [Delta] y ^(k) has become a large value. However, if a ₂ ^{(k) to} a ₆ ^(k) are minute and vary positively and negatively, and the average is 0, then Δx ^(k)
Can be anything. That is, it is irrelevant to parameter estimation stochastically. However, if the variations are biased, the parameter estimation will be adversely affected. Therefore, it is desirable that the estimation parameter is determined only in the range of W4, that is, in the area where the density change of the estimated image is large.

Although the above area selection has been described for a one-dimensional image, since the input image and the estimated image are actually two-dimensional images, the pixels in the area where the density change of the estimated image satisfying the equation (14) is large. Only adopt as data. Here, t _g is, for example, the expected value of Expression (14) obtained from the variation in the image density, and is set to a predetermined value that is about twice the expected value. Since this processing is to extract an image of an area in which the density change of the estimated image is large, the square sum of the first term and the second term of the equation (14) or the square root of the square sum is not limited to the equation (14). May be compared with a predetermined threshold. Also, the estimated image y '(u, v)
, And u ′ and v ′ of the reference image y _M (u ′, v ′)
You may use the partial differential value of a direction. In this way, the number of data at the time of parameter estimation can be significantly reduced, and the estimation can be speeded up.

Next, the structure of a specific embodiment of the present invention will be described. FIG. 5 is a block diagram showing an electrical configuration for realizing the parameter estimation of the present invention. In the figure, an image pickup device 100 such as a CCD camera is a device for photographing a reference image and a subject, and its output is given to an A / D converter 101. A / D converter 101
Is for converting the density data of each pixel into a digital value, the output of which is output via the switching circuit 102 to the image memory group 1
03-104. The image memory 103 is a reference image memory, the image memory 104 is an input image memory, the image memory 105 is an estimated image memory, and the image memory 106.
Is a memory for calculating intermediate parameters. An arithmetic processing unit (CPU) for arithmetic processing is connected to these image memories 103 to 106. The arithmetic processing unit 107 has a program memory 10 for holding its operation program.
8 is provided, and the conversion parameter which is an output is given to the display 110 through the output interface 109.

Next, the operation of this embodiment will be described with reference to the flow chart. FIG. 1 is a main flowchart showing the operation of this embodiment. When the operation is started in this figure, the preprocessing routine 121 is first executed. As shown in the detailed flowchart of FIG. 6, the preprocessing routine 121 first inputs a reference image in step 131.
This is because the reference image is picked up by the image sensor 100, and A /
The D converter 101 converts the digital signal. Then, it is recorded in the image memory 103 via the switching circuit 102. Here, y _M (u ′, v ′) is the data of the entire area of the image memory 103 and is A. A indicates the array for each point of (u ', v').

Next, at step 132, comparison windows W1 'to W3' of the reference image are selected. The comparison windows are selected by comparing the comparison windows W1 to W3 of the input images and the comparison windows W1 'to W3' of the reference image at the same position.
Set the range to include the area where the common points appear in. For example, as shown in FIGS. 7A and 7B, the selection is made so as to include the corner points in the entire area. It is also possible to perform calculations on all of the reference image and the input image without selecting such a comparison window. However, by selecting the comparison window and executing the following calculation only within the range, the calculation amount of the pattern matching can be reduced and the speed can be greatly improved.

Then, the operation proceeds to step 133, and a partial differential image in the x direction is generated by all of the following equations (34) or (35) for all the pixels (u ', v') in the plurality of comparison windows.

[Equation 28] The partial differential image thus generated is D.

Further proceeding to step 134, a partial differential image in the y direction is generated by the following equation (36) or (37) for each pixel (u ', v') in this in-reference comparison window.

[Equation 29] Let E be the partial differential image generated in this way. These partial differential images D and E are calculated by taking the difference between the images deviated in the x direction and the y direction. Since the partial differentiation is calculated in this way, the comparison windows W1 'to W3' are
Select slightly larger than W3.

When the preprocessing is completed in this way, the process proceeds to step 122 in FIG. 1 to capture the input image. Image sensor 10
A predetermined image corresponding to the reference image is captured by 0, A
It is converted into a digital signal via the / D converter 101. Then, the input image is captured by capturing it in the image memory 104 via the switching circuit 102. This input image y
Let (u, v) be the input image C.

Then, in step 123, this reference image itself is set as the first (that is, k = 0) estimated image. This corresponds to setting the unknown parameter vector x ′ ⁽⁰⁾ of the conversion parameters x _{1 to} x ₆ to “100011” ^T. Here, in the unknown parameter vector x ' ^(k) ,
The dash represents the estimated value and (k) represents the estimated number of times. Then, the process proceeds to step 124 to select the area. In this area selection, as described above, the range in which the density of the estimated image satisfies the expression (14) is selected. Then, in step 125, an observation equation for estimating unknown parameters by the method of least squares is set up. This processing is performed in the input image comparison windows W1 to W3.
This is performed for the pixel (u, v) selected in the area. That is, first, in this observation equation, the partial differentiation of the estimated image y ′ (u, v) with respect to the unknown parameters x _{1 to} x _{6 is given} by the equation (1), specifically, the equations (4), (5), and (6). To do. Then, the residual of the input image C from the estimated image y ′ (u, v) is
It is calculated by the equation (8). The former partial derivative is a so-called Jacobian matrix, and the latter residual is a residual vector.
The processing of this routine will be described in more detail below.

Calculation of Preliminary Values First, the preliminary values G (u, v) and H (u, v) are calculated by the equation (2).
Calculate by Next, the coordinates (u ′, v ′) of the reference image corresponding to the coordinates (u, v) of the input image y (u, v) are calculated by the equation (3).

Next, the partial differential vector is sequentially calculated for each element. First, the partial differential intermediate parameter F (u, v) for the conversion parameter x ₁ is calculated by the equation (4).
Here, A is a reference image obtained in the preprocessing. Further, by substituting K (u, v) and L (u, v) obtained by the equation (3) into the partial differential image D in the x direction and the partial differential image E in the y direction by preprocessing, the following equation (5) is used. The next intermediate parameter I (u,
v), J (u, v).

Further, the conversion parameters x _{2 to} x _{6 of the} estimated image
The intermediate parameters M to Q of partial differentiation by
Obtained by (7).

Next, the residual Δy (u, v) ^(k) ≡R (u,
v) is calculated by the following equation (8). In this way, the routine 125 calculates each element of the partial differential vector and the residual.

Next, the routine 126 is proceeded to, from these elements, a coefficient matrix B and a column vector b forming a normal equation are obtained. Here, the normal equation is expressed by equation (9).

The coefficient matrix forming this normal equation is expressed by the equation (1
0). Here, each element of the matrix B is represented by Expression (11). Further, the column vector b is expressed by Expression (12), and its elements are as shown in Expression (13).

Then, the routine proceeds to step 127 shown in FIG. 1 to solve a six-element linear equation for obtaining each element of the correction vector Δx 'from the normal equation (Equation (9)) thus obtained. Then, in step 128, an unknown parameter vector x ' ^{(k + 1)} having a new conversion parameter as an element is calculated. Then, the process proceeds to step 129, and it is determined whether or not this satisfies a predetermined convergence condition. If it does not converge, the process returns to step 124 and the same processing is repeated.

Here, the convergence condition is, for example, Δy ^(K) (u, v) shown in the equation (8) is used, and n is the number of all pixels in the selected regions in all the comparison windows W1 to W3. Then, the residual variance v _yy is calculated based on the equation (15). Then, when the condition that the ratio of the difference between the residual differences and the current variance is equal to or less than a predetermined value, that is, the expression (16) is satisfied, it is determined that the convergence has occurred. When the convergence is thus determined, the process proceeds to step 130 and the unknown parameter vector x ' ^{(k + 1)}
Each element of is used as an estimation parameter. Then, the estimated image is calculated from x _{1 to} x _{6 based} on the above-mentioned formula (24).
This estimated image is displayed by the display device 110.

Thus, the reference image can be gradually brought closer to the input image by converting the reference image using the parameters obtained for each successive estimation and displaying it sequentially.

FIGS. 8A to 8D are density sectional views of the reference image and the input image, and FIGS. 8B to 8D show the reference image and the input image changing as the estimation progresses. It shows how they match. Here, in order to simplify the description, it is shown in one dimension. In this example, it is shown that the estimated image is considerably close to the input image in four estimations.

Next, a second embodiment of the present invention will be described. In the above-described embodiment, the reference image and the input image are processed as they are. However, a smoothing filter having an appropriate size is applied to the reference image and the input image, and the smoothed images are repeatedly estimated to perform parameter matching. You may do it. In this case, the density gradient of the image becomes gentle,
The optimal estimate can be reached quickly. Such smoothing processing is performed, for example, in the preprocessing routine of FIG. 6 after the reference image is input and after the input image is captured in step 122. Subsequent processing is the same as in the first embodiment described above.

When this smoothing process is not performed (curve S
0), the case where smoothing processing is performed using a smoothing filter having a size of 3 pixels, 5 pixels, and 7 pixels on one side is defined as curves S3, S5, and S7, respectively, and the relationship between the number of iterations and the squared value of the variance is shown. Are shown in FIGS. 9 (a) and 9 (b). FIG. 9A shows the residual sum of squares of the residual in a simple pattern, and FIG. 9B shows the convergence state of the residual sum of squares in a complicated pattern. As is clear from this graph, it is shown that any image can be converged faster by performing the smoothing process.

In the above embodiment, the estimation is repeated so that the reference image approaches the input image. However, the estimation is repeated so that the input image approaches the reference image, and the patterns of the two images may be matched. It goes without saying that it is good. In this case, similar processing can be performed by exchanging the reference image and the input image of all the mathematical expressions described above.

In the above-described embodiment, the comparison windows are set for the reference image and the input image, and the intermediate parameters are calculated and the normal equations are analyzed within the range.
This may be done for all areas. Even if common areas such as corners are not located at the same position in the reference image and the input image, if the approximate value of image conversion is known in advance,
Areas at different positions can be used as comparison windows. In this case, it is necessary to set the initial values of the conversion parameters shown in step 123 of FIG. 1 in correspondence with changes in the comparison window.

[0041]

As described in detail above, according to the inventions of claims 1 to 4 of the present application, it is possible to sequentially bring the change of the parameter closer to the input image for each estimation without the need to extract the feature quantity. In addition, since the area in which the six-dimensional linear equation is established based on the density gradient to obtain the intermediate parameter is limited to the area where the density change of the estimated image exceeds the predetermined value, the conversion parameter can be estimated in a short time. Is obtained.

Further, according to the third aspect of the present invention, since it is not necessary to perform the arithmetic processing on all of the reference image and the input image, the conversion parameter can be calculated at high speed. Further, according to the invention of claim 4 of the present application, since the smoothing process of the input image and the reference image is performed, it is possible to obtain the effect that the number of times of estimating the conversion parameter is reduced and the estimation can be converged quickly.

[Brief description of drawings]

FIG. 1 is a flowchart showing an estimation process according to an embodiment of a parameter estimation method according to the present invention.

2A is a reference image, FIG. 2B is an input image, and FIG.
FIG. 4 is a diagram showing a state in which a reference image is brought closer to an input image by an image conversion operation.

FIG. 3A to FIG. 3E are diagrams showing changes in a reference image during image conversion.

FIG. 4 is a diagram for explaining region selection based on density change according to the present invention.

FIG. 5 is a block diagram showing an outline of an apparatus for realizing the present invention.

FIG. 6 is a flowchart showing a pre-processing routine of the present embodiment.

FIG. 7 is a diagram illustrating an area of a comparison window in which an input image and a reference image match each other.

FIG. 8 is a diagram showing a state in which a one-dimensional reference image is approaching an input image sequentially by estimation in the present embodiment.

FIG. 9A is a graph showing a change in the number of times of estimation when smoothing processing is performed on a simple graphic and a complicated graphic.

[Explanation of symbols]

 100 image sensor 101 A / D converter 102 switching circuit 103 image memory (for reference image) 104 image memory (for input image) 105 image memory (for estimated image) 106 image memory (for intermediate parameter calculation) 107 arithmetic processing unit 108 Program memory 109 Output interface 110 Display

Claims (4)

[Claims] 1. A first image y _M (u ′, v ′) (≡
A) and the second image y (u, v) (≡C) are held as pixel unit data, the brightness ratio is x ₁ , the rotation angle (rad) about the origin is x ₂ , and the horizontal , The first image is converted based on the conversion parameters in which the vertical translation amounts (pixels) are x ₃ and x ₄ , and the horizontal and vertical scaling ratios are x ₅ and x ₆ , respectively. Second by
Pattern matching method for performing pattern matching to match the first image, the partial differential image D (u ′, v ′) in the x direction of the first image, and the partial differential image in the y direction of the first image. E (u ′, v ′) is calculated, and x = [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , x ₆ ] ^T is an unknown parameter vector having conversion parameters x _{1 to} x ₆ as elements. , The partial differential for each element of the unknown parameter vector of the estimated image y ′ (u, v) is expressed by the following equation (1) As intermediate parameters G, H, K, L and F, M, N, for each pixel (u, v) corresponding to the second image, respectively.
O, P, Q and R are calculated by the following equations (2) to (8), and [Equation 3] [Equation 4] [Equation 5] [Equation 6] [Equation 7] [Equation 8] With these intermediate parameters F to Q, the normal equation (Equation (9)) The coefficient matrix B and the column vector b that form
Calculated according to 3), [Equation 11] [Equation 12] An area where the density change of the estimated image that satisfies the following equation (14) is large is selected, and By solving the normal equation (9), the correction vector Δx ′ of the unknown parameter vector x is set with the estimated number of times k.
^(k) solved 6-way linear equations for calculating a calculates the k + 1-order unknown parameter vector by adding the k-th order of the unknown parameter vector x ^'(k) to the correction vector [Delta] x' ^(k), the unknown parameters The convergence of the estimated image obtained by converting the first image and the second image is determined by the vector, and if they are converged, this conversion parameter is used as the output parameter,
An image pattern matching method characterized in that if the image parameters do not converge, the intermediate image parameter is calculated again to calculate the conversion parameter to perform image pattern matching. 2. The convergence judgment is made by the residual variance expressed by the following equation (15): Between the previous estimation and the previous equation (16) 2. The pattern matching method for an image according to claim 1, wherein it is determined that the image has converged when the above condition is satisfied. 3. The comparison windows are areas in which the same points may appear in the corresponding areas in the first and second images, and the intermediate parameters F to Q are calculated only within this range. 3. The image pattern matching method according to claim 1, wherein the normal equation analysis process is performed. 4. The pattern matching method for an image according to claim 1, wherein the first image and the second image are subjected to smoothing processing in a predetermined range at the time of input.