JP2003157408A - Distortion image associating method, device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately and quickly associate a distorted input image of a form with a reference image having no distortion. SOLUTION: Three kinds of feature points (longitudinal and lateral edge points, lateral edge point and longitudinal edge point) are extracted from an image input from an image input portion 101 and cut out of a cut-out processing portion 102 by a feature point extracting portion 104, and the displacement between the corresponding point determined by Hough conversion and the corresponding point determined by an affine conversion efficiency is determined in a corresponding point detector 106, and the corresponding point is specified by using a triangular analogy ratio using as a reference point the corresponding point providing little displacement.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distorted image associating method, device and program for associating each feature point of a reference image without distortion with a corresponding point of an input image with distortion, and more particularly to a form with distortion. The present invention relates to a distortion image associating method, device, and program capable of quickly and accurately associating an input image with a reference image without distortion.

[0002]

2. Description of the Related Art When a document is read by a scanner, the document image often has a distorted ruled line as shown in FIG. 18 due to bulging or twisting of the document. In particular, when an overhead scanner is used, the influence of such a bulge of the form becomes a distortion of the form image and becomes apparent. Further, even when a form is read by a scanner having a high conveyance speed, a slight change in the conveyance speed produces a form image with distortion such as expansion and contraction.

For these reasons, it becomes necessary to associate the input image, which is a distorted form image, with the reference image. Therefore, for example, in Japanese Patent Laid-Open No. 11-3430, the edge image of the reference image is divided into small blocks, the search range and priority of each small block are determined, and the image corresponding to each small block is set as the input image. A matching method for searching within a search range is disclosed. According to this prior art,
The reference image and the input image can be associated with each other without depending on a specific element such as a line segment or a table.

[0004]

However, according to this prior art, the amount of deviation from the estimated position is determined based on the vertical and horizontal distances between the reference edge image and the input edge image for each of the vertical and horizontal edges. Therefore, there is a problem that the shift amount cannot be obtained in the case of an edge image having few features such as one having only vertical edges, one having only horizontal edges, or one having no edges.

The present invention has been made in order to solve the above-mentioned problems caused by the prior art, and distortion in which an input image of a form with distortion and a reference image stored in advance can be quickly and accurately associated with each other. An object of the present invention is to provide an image associating method, device, and program.

[0006]

[Means for Solving the Problems]
In order to achieve the object, the method for associating a distorted image according to the invention of claim 1 is a method for associating a distorted image in which each feature point of a reference image without distortion is associated with a corresponding point of an input image with distortion. Based on the horizontal and vertical amounts of the direction vector at each edge point located in the edge image of the reference image, both vertical and horizontal edge points, horizontal edge points and vertical edge points of the reference image are used as feature points. It is characterized by including a feature point extracting step of extracting and a corresponding point detecting step of detecting a corresponding point on the input image corresponding to each characteristic point of the reference image.

The distortion image associating method according to a second aspect of the present invention is the method according to the first aspect, wherein the feature point extracting step is a predetermined centering on each edge point of the edge image of the reference image. The sum of the horizontal direction amount and the vertical direction amount in the area is defined as the vertical and horizontal direction amounts, and the horizontal direction amount and the vertical direction amount in the predetermined area are each a predetermined value or more, and the vertical and horizontal direction amount is the maximum. Vertical and horizontal both edge point extraction step of extracting the point as the vertical and horizontal both edge points, the horizontal amount in a predetermined region centered on each edge point of the edge image of the reference image is a predetermined value or more, and A horizontal edge point extraction step of extracting a point having the maximum horizontal direction amount as the horizontal edge point, and a vertical direction amount within a predetermined area centered on each edge point of the edge image of the reference image is predetermined. Greater than or equal to the value,
And a vertical edge point extracting step of extracting the point having the maximum amount in the vertical direction as the vertical edge point.

Further, the method of associating a distorted image according to a third aspect of the present invention is the method according to the first or second aspect, wherein the feature point extracting step is performed in both vertical and horizontal directions within a predetermined area centered on an edge point. When extracting an edge point, a horizontal edge point, or a vertical edge point, a pixel value in a predetermined area of the edge image is erased, and a next edge point is extracted based on the feature amount of the erased edge image. Is characterized by repeating.

Further, a method of associating a distorted image according to the invention of claim 4 is the method according to claim 1, 2 or 3,
The corresponding point detecting step calculates a first corresponding point on the input image corresponding to a characteristic point of the reference image by a Hough transform, and an affine transformation coefficient for converting the characteristic point to the corresponding point. And an affine transformation coefficient calculating step for calculating the above, comparing the deviation amount between the first corresponding point obtained in the Hough transformation step and the second corresponding point obtained from the affine transformation coefficient with a predetermined threshold value, And a feature point specifying step of specifying a fourth corresponding point whose shift amount exceeds the predetermined threshold value by a triangle similarity ratio with the third corresponding point which is less than or equal to the threshold value as a reference point. And

Further, the method for associating a distorted image according to a fifth aspect of the present invention is the method according to the fourth aspect, wherein the feature point specifying step is performed by a triangle similarity ratio with the third corresponding point as a reference point. It is characterized in that the fourth corresponding point having the maximum evaluation amount is specified by the Hough transform using the edge point on the edge image of the input image as a center point.

The distortion image associating device according to the invention of claim 6 is a distortion image associating device for associating each feature point of a reference image without distortion with a corresponding point of an input image with distortion. Extracting both vertical and horizontal edge points, horizontal edge points and vertical edge points of the reference image as feature points based on the horizontal and vertical amounts of the direction vector at each edge point located in the edge image of the reference image And a corresponding point detecting means for detecting a corresponding point on the input image corresponding to each characteristic point of the reference image.

According to a seventh aspect of the present invention, there is provided the distortion image associating device according to the sixth aspect, wherein the feature point extracting means has a predetermined center point at each edge point of the edge image of the reference image. The sum of the horizontal direction amount and the vertical direction amount in the area is defined as the vertical and horizontal direction amounts, and the horizontal direction amount and the vertical direction amount in the predetermined area are each a predetermined value or more, and the vertical and horizontal direction amount is the maximum. Vertical and horizontal both edge point extraction means for extracting the point as the vertical and horizontal both edge points, and the horizontal amount within a predetermined area centered on each edge point of the edge image of the reference image is a predetermined value or more, and A horizontal edge point extracting means for extracting a point having the maximum horizontal direction amount as the horizontal edge point, and a vertical direction amount within a predetermined area centered on each edge point of the edge image of the reference image is predetermined. Greater than or equal to the value,
Further, it is characterized by further comprising a vertical edge point extraction means for extracting a point having the maximum vertical direction amount as the vertical edge point.

Further, in the distortion image associating device according to the invention of claim 8, in the invention of claim 6 or 7, the corresponding point detecting means is provided on the input image corresponding to the feature point of the reference image. Hough transform means for calculating a first corresponding point by Hough transform, affine transform coefficient calculating means for calculating an affine transform coefficient for converting the characteristic point into the corresponding point, and first correspondence obtained by the Hough transform means. The deviation amount between the point and the second corresponding point obtained from the affine transformation coefficient is compared with a predetermined threshold value, and the deviation is made by the triangle similarity ratio with the third corresponding point which is less than the threshold value as a reference point. And a feature point specifying means for specifying a fourth corresponding point whose amount exceeds the predetermined threshold value.

Further, the program according to the invention of claim 9 can be machine-readable by causing a computer to execute the method described in any one of claims 1 to 5, whereby the program can be read. The operation of any one of items 1 to 5 can be realized by a computer.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of a distortion image associating method, apparatus and program according to the present invention will be described in detail below with reference to the accompanying drawings. In the present embodiment, the case where the images of the forms are associated with each other will be described.

(Regarding the Configuration of the Image Matching Device) First, the configuration of the image matching device according to the present embodiment will be described. FIG. 1 is a functional block diagram showing the configuration of the image association device according to the present embodiment. As shown in the figure, the image associating device 100 includes an image input unit 10
1, a cutout processing unit 102, and an edge image creation unit 103
And a feature point extraction unit 104, a storage unit 105, and a corresponding point detection unit 106. The feature point extracting means of claim 7 corresponds to the feature point extracting section 104, and the corresponding point detecting means of claim 7 corresponds to the corresponding point detecting section 106.

Here, the image associating device 100 is
Different processing is performed in the association mode in which the input image of the form in which the handwritten characters are written is accepted and the template registration mode in which the template (reference image) to be compared with the input image is accepted. Specifically, in the case of the template registration mode, the processing flow of the image input unit 101, the cutout processing unit 102, the edge image creation unit 103, the feature point extraction unit 104, and the storage unit 105 is performed. Includes the image input unit 101 to the cutout processing unit 1
02-edge image creation unit 103-corresponding point detection unit 106. It should be noted that mode switching between both modes is performed by a changeover switch or the like not shown.

The image input unit 101 is a processing unit for inputting an input image or a template. Specifically, the input image is captured by, for example, an overhead type scanner, and the template is captured by, for example, a pressure contact type scanner. . Here, any input device may be used, but it is assumed that the input image has distortion due to bulge of the form and the like, and the template does not have such distortion as the input image.

The cutout processing unit 102 is a processing unit that cuts out an area to be associated from the input image or the template input from the image input unit 101. When the input image is cut out, the region is cut out in a slightly larger size than in the case of the template in consideration of distortion and displacement.

The edge image creating unit 103 is a cutout processing unit 1.
A differential operator such as a Sobel operator is applied to the area cut out by 02,
It is a processing unit that creates a direction image in which the edge portion is emphasized. The differential image obtained as a result of such processing has the direction vector of the density gradient at each point as a pixel value.

The feature point extraction section 104 determines both vertical and horizontal edge points, horizontal edge points, and vertical edge points based on the horizontal and vertical amounts of the direction vector at each edge point located in the cut-out region of the edge image of the template. It is a processing unit that extracts edge points and surrounding points as feature points. Specifically, the lateral direction amount Nh within a predetermined area centered on each edge point
The sum of the vertical and horizontal directions Nv is the vertical and horizontal directions Nhv, and the points where the horizontal and vertical directions Nh and Nv in this area are equal to or greater than the specified values, and the vertical and horizontal directions Nhv are the maximum. Both edge points. Further, the lateral direction amount Nh in a predetermined area centered on each edge point is a predetermined value or more, and the point where the lateral direction amount Nh is the maximum is the lateral edge point,
A vertical edge point is a point in which a vertical direction amount Nv in a predetermined area centered on each edge point is a predetermined value or more and the vertical direction amount Nv is maximum.

FIG. 2 is an explanatory diagram for explaining the concept of calculating the direction amount. When an edge image as shown in FIG. 7A exists, for example, a density gradient direction vector is stored in each pixel as shown in FIG. The direction vector of a point having a horizontal edge such as the pixel a and the pixel f is vertical, and the direction vector of a point having a vertical edge such as the pixel c and the pixel d is horizontal. From this, the horizontal amount is obtained by the sine function (sin), and the vertical amount is obtained by the cosine function (cos). Therefore, the point (x, y)
The respective directional amounts in the above are as follows, and the directional amounts of the respective pixels a to f in the figure are as shown in FIG.

That is, assuming that the direction of the direction vector is th (the value of the edge image at the point (x, y)), the lateral direction amount (Nh) = | sin (th) | the longitudinal direction amount (Nv) = | cos ( th) | Amount in both vertical and horizontal directions (Nhv) = Nh + Nv.

The feature point extraction unit 104 uses the observation window to sequentially determine the feature points in the order of vertical and horizontal edge points, horizontal edge points, vertical edge points, or vertical and horizontal edge points, vertical edge points, horizontal edge points. To extract. That is, when extracting the feature point of the central pixel of the observation window, the horizontal direction amount, the vertical direction amount, and the vertical and horizontal direction amount of each pixel in the observation window are added, and the addition result is the horizontal direction of the central pixel. Amount, the amount in the vertical direction, and the amount in both the vertical and horizontal directions. Then, when these direction amounts satisfy a predetermined condition, the central pixel is set as both vertical and horizontal edge points, horizontal edge points, or vertical edge points.

When a certain characteristic point is extracted, the extraction of the characteristic point is repeated while deleting the pixel values of the edge image in this area. This is because feature points are extracted at appropriate intervals. Surrounding points are evenly spaced on the boundaries of the template (regardless of edge features).

The storage unit 105 is a storage unit that stores the feature points extracted by the feature point extraction unit 104, that is, both vertical and horizontal edge points, horizontal edge points, vertical edge points, peripheral points, and edge images, and the stored data. Is the corresponding point detection unit 106
Accessed by.

The corresponding point detector 106 roughly aligns the template with the input image by Hough transform. Next, the feature points of the template and the corresponding points on the input image are aligned by the Hough transform, the affine transformation coefficient that transforms these feature points into the corresponding points is calculated, and the corresponding points and the affine transform coefficients obtained by the Hough transform are calculated. The shift amount of the corresponding point obtained from is calculated. Then, the calculated shift amount is compared with a predetermined threshold value, and corresponding points below the threshold value are used as reference points, and a feature point whose shift amount exceeds the predetermined threshold value is determined by a triangle similarity ratio described later. Processing for identifying the corresponding points is performed.

As described above, the image associating device 100
The feature point extraction unit 104 extracts three types of feature points (vertical and horizontal edge points, horizontal edge points, and vertical edge points), and the corresponding point detection unit 106 calculates corresponding points and affine transformation coefficients obtained by Hough transform. The shift amount of the corresponding point obtained from is obtained, and the corresponding point is specified by using the triangle similarity ratio with the corresponding point having the small shift amount as the reference point.

(Regarding Processing in Template Registration Mode) Next, a processing procedure in the template registration mode by the image associating device 100 shown in FIG. 1 will be described. 3 is a flowchart showing a processing procedure in the template registration mode by the image association apparatus 100 shown in FIG. 1, and FIG. 4 is a diagram showing an example of a template in the template registration mode.

As shown in FIG. 3, the image associating device 10
When 0 is the registration mode, when an image with no distortion of the blank form as shown in FIG. 4A is input from the image input unit 101 (step S301), the cutout processing unit 102 causes the cutout processing unit 102 to perform the processing illustrated in FIG. ) A target area, that is, a template is cut out (step S302). Specifically, in the registration mode, the operator cuts out the template by operating the mouse. After that, the edge image creation unit 1
03 applies the differential operator to this template to obtain the edge image shown in FIG. 4C (step S30).
3).

After that, from FIG. 4C, the feature point extraction unit 10
4 extracts feature points as shown in FIG. 4D (step S304). FIG. 5 is an enlarged view of FIG. 4D. As shown in FIG. 5, both vertical and horizontal edge points indicated by black circles in the drawing,
Horizontal edge points indicated by crosses (x), vertical edge points indicated by black triangles, and surrounding points indicated by black squares are obtained as feature points.

When these characteristic points are obtained, the characteristic points are associated with the template and stored in the storage unit 10.
5 (step S305) so that it can be used in the processing of the corresponding point detection unit 106 described later.

(Regarding Processing in Matching Mode) Next, the processing procedure in the matching mode by the image matching apparatus 100 shown in FIG. 1 will be described. 6 is shown in FIG.
9 is a flowchart showing a processing procedure in the association mode by the image association apparatus 100 shown in FIG. 7, and FIG. 7 is a diagram showing an example of an input image in the association mode.

As shown in FIG. 6, the image association device 10
When 0 is the association mode, when a distorted input image is input from the image input unit 101 (step S60).
1), the cutout processing unit 102 cuts out a part of the input image as shown in FIG. 7A (step S602). Specifically, the type of form is separately determined and a predetermined area to be read is cut out. After that, the edge image creating unit 103 applies the differential operator to obtain the edge image shown in FIG. 7B (step S603).

Then, rough alignment is performed by the Hough transform with the edge image of the template shown in FIG. 7C stored in the storage unit 105 to obtain the movement amount (step S604), and the movement amount is taken into consideration. As shown in FIG. 6D, pixels corresponding to each feature point, that is, corresponding points are detected (step S605). 8 is shown in FIG.
It is an enlarged view of (d), as shown in the figure, corresponding points of vertical and horizontal edge points indicated by black circles, corresponding points of horizontal edge points indicated by crosses (x), and vertical points indicated by black triangles. The corresponding points of the edge points and the corresponding points of the surrounding points indicated by black squares are obtained.

In the present embodiment, only the processing until the corresponding points are obtained will be described. However, if necessary, a corrected image in which distortion is corrected based on the amount of displacement (displacement) between corresponding characteristic points can be obtained. It can also be created (FIG. 7 (e)).

(Regarding Feature Point Extraction Process) Next, the feature point extraction process by the feature point extraction unit 104 shown in FIG. 1 will be described more specifically. 9 and 10 are flowcharts showing the procedure of extracting the characteristic points by the characteristic point extracting unit 104 shown in FIG. 1, and FIGS.
FIG. 4 is an explanatory diagram showing an example of image data during a feature point extraction process.

As shown in FIGS. 9 and 10, the feature point extraction unit 104 extracts feature points in the order of vertical and horizontal edge points, horizontal edge points, and vertical edge points. First, in order to extract both vertical and horizontal edge points, as shown in FIG. 11A, an area (observation window) of size S × S centering on the edge point of the edge image of the template is provided, and this observation window The horizontal and vertical directions (Nhv), the horizontal direction (Nh), and the vertical direction (Nv) are calculated (step S901). Specifically, the direction amount for each pixel in this observation window is examined, and this direction amount is added for each direction amount to obtain the direction amount of the central pixel. This process is repeated while moving the observation window, and the direction quantities of all pixels are obtained.

Then, the points in which the horizontal direction amount (Nh) and the vertical direction amount (Nv) are each larger than a predetermined value, for example, S / 4, and the vertical and horizontal direction amount (Nhv) is the maximum are both vertical and horizontal edge points. Yes (step S902).

If there are vertical and horizontal edge points that satisfy step S902 (Yes at step S903, FIG. 11).
(B)), as shown in FIG. 11 (b '), a part of the edge image around both the vertical and horizontal edge points (all the edge images inside the observation window centering on the vertical and horizontal edge points) are deleted ( In step S904), the erased image is used as a new edge image (step S905), the observation window is applied to each edge point again to calculate the direction amount, and the process is repeated (steps S901 to S905).

When there are no vertical or horizontal edge points satisfying step S902 (No at step S903), the edge image is not erased and step S is performed as it is.
Move to 906. The edge image at that time is, for example, as shown in FIG.
It is like 1 (c). When the process of extracting both the vertical and horizontal edge points is completed in this way, the process proceeds to the process of extracting the horizontal edge points. Specifically, after calculating the lateral direction amount (Nh) (step S906), the lateral direction amount (Nh) is larger than a predetermined value, for example, S / 4, and the lateral direction amount (Nh).
The point at which h) is the maximum is the horizontal edge point (step S9).
07).

If there is a horizontal edge point satisfying step S907 (Yes at step S908, FIG. 12A), the edge image around the horizontal edge point is set as in the case of both vertical and horizontal edge points. Erase (step S90
9), the erased image is used as a new edge image (step S910), the observation window is applied to each edge point again to calculate the direction amount, and the process is repeated (step S906).
~ S910). When there are no horizontal edge points that satisfy step S907 (No at step S908), the edge image is not erased and step S91 is directly performed.
Move to 1. The edge image at this time is, for example, as shown in FIG.
It looks like (b).

When the horizontal edge point extraction processing is completed in this way, the process moves to vertical edge point extraction processing. Specifically, after calculating the vertical direction amount (Nv) (step S91)
1) A point where the vertical direction amount (Nv) is larger than a predetermined value, for example, S / 4, and the vertical direction amount (Nv) is maximum is set as a vertical edge point (step S912).

If there is a vertical edge point satisfying step S912 (Yes at step S913, FIG. 12C), the edge image around the vertical edge point is erased (step S914), and the erased image is obtained. As a new edge image (step S915), and the above step S911.
~ The process of S915 is repeated. Note that step S912
If there is no vertical edge point that satisfies the condition (step S
(No at 913), the process ends without erasing the edge image.

Although the description is omitted here for convenience of explanation, when both vertical and horizontal edge points already exist in the observation window, the horizontal edge points and vertical edge points are not extracted, Even if both the vertical and horizontal edge points do not exist and the horizontal edge points do exist, the vertical edge points are not extracted.

(Corresponding Point Detection Processing) Next, referring to FIG.
The processing procedure of the corresponding point detection unit 106 shown in will be described. FIG. 13 is a flowchart showing a processing procedure of the corresponding point detecting unit 106 shown in FIG. 1, and FIG. 14 is a diagram showing an example of an image when detecting corresponding points. As shown in FIG. 13, here, corresponding points are detected in the order of vertical and horizontal edge points, horizontal edge points, and vertical edge points. This is because both vertical and horizontal edge points have more features of association than horizontal edge points and vertical edge points.

As shown in the figure, corresponding points on the input image corresponding to both vertical and horizontal edge points are searched by Hough transform (step S1301). Since the feature point is on the edge, the corresponding point is also considered to be on the edge without fail. Therefore, the point closest to the corresponding point obtained by the Hough transform is set as the corresponding point. As a result, as shown in FIG. 14A, corresponding points (xi ', yi') corresponding to both vertical and horizontal edge points (xi, yi) are obtained.

After that, affine transformation coefficients for transforming both vertical and horizontal edge points (xi, yi) into corresponding points (xi ', yi') are obtained by the least square method shown in the following equation (step S1302).

[Equation 1]

Then, the shift amount ei between the corresponding point and the corresponding point obtained by the affine transformation coefficient is calculated (step S1303),
As shown in FIG. 14B, both vertical and horizontal edge points (ei> Cσ) with a large shift amount and vertical and horizontal edge points with a small shift amount (ei ≦ Cσ).
Cσ) (step S1304). Then, the affine transformation coefficient is obtained again only with respect to both the vertical and horizontal edge points having a small shift amount (step S1305). C is a coefficient that should be determined according to the degree of distortion of the input image to be handled, and may be C = 1, for example.

For both vertical and horizontal edge points having a large displacement amount, corresponding points are obtained using the triangle similarity ratio (step S1306). FIG. 15 is an explanatory diagram for explaining how to obtain corresponding points of both vertical and horizontal edge points having a large shift amount. If both the vertical and horizontal edge points 1501 shown in FIG. 9A are associated with the corresponding points 1502 shown in FIG. 9B, the amount of deviation becomes large.

Therefore, both vertical and horizontal edge points 1502
(Point C shown in FIG. 6C), vertical and horizontal edge points with a small shift amount located at the lower left of this point C (point B shown in FIG. 7C), and located at the lower right of point C. Consider a triangle formed by both vertical and horizontal edge points (point A shown in FIG. 7C) with a small displacement amount. Considering in this way, as shown in FIG. 7D, the corresponding points of points A and B become points A ′ and B ′, respectively, so that the similarity ratio between triangle ABC and triangle A′B′C ′ is From, a temporary corresponding point C ′ of the point C is obtained. There should be a corresponding point of the point C to be obtained near this point C ′.

That is, the coordinates of the point A'is (ax ', ay'), the coordinates of the point C'is (cx ', cy'), and the distances AB and A'B 'are AB and A'B, respectively. ',

[Equation 2] The point C ′ can be obtained by

Then, the corresponding points of the vertical and horizontal edge points C on the template are obtained by a search by Hough transform centering on the temporary point C'on the input image. One side of the image at this time is made shorter than one side of the Hough transform performed in step S1301. In this way, by narrowing the range of Hough transform, more accurate corresponding points can be obtained.

FIG. 16 is an explanatory diagram for explaining reference points when the matching amount of the Hough transform is obtained using the similarity ratio of two triangles. In the same figure (c), point xS
Is a point whose x-coordinate is smaller than both vertical and horizontal edge points C and has the shortest distance to the point C, and the point xL has a larger x-coordinate than both vertical and horizontal edge points C and the distance to the point C. Is the shortest point.

The points yS and yL are similar with respect to the y coordinate.
It is a point that has a relationship. Also, edit the input images.
Let the corresponding points on the image be xS ', xL', yS ', yL'.
Then, in the Hough transform, the pair with the maximum evaluation amount given by
You will be asked for a response point. M is the matching amount,
This is the voting value in the Hough transform. M-α x (| 1- (v1 '/ v0') / (v1 / v0) |²＋ ｜ 1- (v0 '/ v
2 ') / (v0 / v2) ｜² ＋ ｜ 1- (v2 '/ v1') / (v2 / v1) ｜²+ | 1-
(w1 '/ w0') / (w1 / w0) ｜² ＋ ｜ 1- (w0 '/ w2') / (w0 / w2) ｜²+
｜ 1- (w2 '/ w1') / (w2 / w1) ｜²)

When the corresponding points of the vertical and horizontal edge points are detected in this way, the corresponding points of the horizontal edge points, the corresponding points of the vertical edge points, and the corresponding points of the feature points on the boundary are sequentially calculated (steps S1307 to S1307). S1309).

Specifically, when the corresponding point of the horizontal edge point is obtained, the x coordinate is smaller than that of the horizontal edge point among the feature points (reference points) having a small shift amount, and the horizontal edge point The point xS having the shortest distance and the point xL having a larger x coordinate than the horizontal edge point and having the shortest distance to the horizontal edge point are used as reference points, and the rotation frequency and the triangle are the same as in the case of both the vertical and horizontal edge points. Using the similarity ratio, the center point of the Hough transform is obtained on the edge image of the input image, and the obtained center point is used as the center point of the Hough transform, and the corresponding point having the maximum evaluation amount of the following equation is obtained.

M-2 × α × (│1- (v1 '/ v0') / (v1 / v0)
｜ ² ＋ ｜ 1- (v0 '/ v2') / (v0 / v2) ｜ ² ＋ ｜ 1- (v2 '/ v1') /
(v2 / v1) ｜ ² )

Further, when the corresponding points of the vertical edge points are obtained, a point yS having a smaller y coordinate than the vertical edge points among the reference points and having the shortest distance to the vertical edge points, and the vertical edge points. In the edge image of the input image, the y-coordinate is large and the point yL having the shortest distance from the vertical edge point is used as a reference point, and the rotation frequency and the triangle similarity ratio are used as in the case of both the vertical and horizontal edge points. The center point of the Hough transform is obtained, and the obtained center point is used as the center point of the Hough transform to find the corresponding point that maximizes the evaluation amount of the following equation.

M-2 × α × (│1- (w1 '/ w0') / (w1 / w0)
│ ² ＋ ｜ 1- (w0 '/ w2') / (w0 / w2) ｜ ² ＋ ｜ 1- (w2 '/ w1') /
(w2 / w1) ｜ ² )

When the feature points on the boundary are to be obtained,
At the outermost part of the template, feature points are added evenly and surround the image. Corresponding points corresponding to the feature points on the boundary are obtained by substituting into the affine transformation formula.

(Regarding Interpolation between Data) According to the above series of description, both vertical and horizontal edge points, horizontal edge points, and vertical edge points can be obtained as feature points, and corresponding points corresponding to each feature point can be obtained. With, it is also possible to interpolate between data.

FIG. 17 is an explanatory diagram for explaining the interpolation between data. As shown in FIG. 17A, a Delaunay triangulation network connecting the feature points is created, and the inside of which triangle each element is located is searched. Then, the displacement amount of each element is obtained by linear conversion from the obtained coordinates of the corresponding points of the triangular vertices and the displacement amount at each point. For example, the displacement amount of the point A in FIG. 17B is (a1, a2), and the displacement amount of the point B is (b1, b2).
In the case of 2), the displacement amount (c1, c2) of the point P can be obtained by the following equation. Note that α and β are the vector OP
Is a coefficient obtained by linearly decomposing vector OA and vector OB.

[Equation 3]

As described above, according to the present embodiment, the feature point extraction unit 104 extracts three types of feature points (vertical and horizontal edge points, horizontal edge points and vertical edge points) and detects corresponding points. The unit 106 is configured to obtain a shift amount between the corresponding point obtained by the Hough transform and the corresponding point obtained from the affine transformation coefficient, and to specify the corresponding point using a triangle similarity ratio with the corresponding point having a small shift amount as a reference point. Therefore, the input image of the form with distortion and the template without distortion can be associated with each other quickly and accurately.

In the present embodiment, the case where the images of the form are associated with each other has been described, but the present invention is not limited to this, and the invention can be applied to the case where images other than the form are associated with each other.

Further, in the present embodiment, both vertical and horizontal edge points, vertical edge points, and horizontal edge points are defined by FIG. 2 and its description, but the present invention is not limited to this, and, for example, vertical and horizontal edge points. It is also possible to manually point the corners of the frame as both edge points and the internal division points of both horizontal and vertical edge points as horizontal edge points and vertical edge points with a mouse.

[0067]

As described above, according to the first aspect of the invention, the reference image of the reference image is calculated based on the horizontal direction amount and the vertical direction amount of the direction vector at each edge point located in the edge image of the reference image. Both vertical and horizontal edge points, horizontal edge points, and vertical edge points are extracted as feature points, and the corresponding points on the input image corresponding to each feature point of the reference image are detected. An effect is obtained that a distorted image associating method capable of quickly and accurately associating with a non-reference image is obtained.

According to the second aspect of the present invention, the sum of the horizontal and vertical amounts within a predetermined area centered on each edge point of the edge image of the reference image is taken as the vertical and horizontal bidirectional amounts, and the predetermined amount is set. The horizontal and vertical amounts in the area are respectively greater than or equal to a predetermined value, and the points with the maximum vertical and horizontal directions are extracted as both vertical and horizontal edge points, and each edge point of the edge image of the reference image is set as the center. A point in which the lateral amount in the prescribed area is equal to or greater than a prescribed value and the lateral amount is maximized is extracted as a lateral edge point, and a predetermined value is set around each edge point of the edge image of the reference image. Since the point in which the vertical direction amount in the area is equal to or larger than a predetermined value and the vertical direction amount is maximum is extracted as a vertical edge point, an edge point having a lot of information about corresponding points is characterized. It is possible to make a point and to make an accurate correspondence. Mapping methods such distorted image is an effect that can be obtained.

According to the third aspect of the present invention, when the vertical and horizontal both edge points, the horizontal edge points or the vertical edge points within a predetermined area centering on an edge point are extracted, the predetermined area of the edge image is extracted. Since it is configured to repeat the process of erasing the pixel value in the inside and extracting the next edge point based on the feature amount of the erased edge image, a distorted image associating method capable of speeding up the process The effect is obtained.

According to the invention of claim 4, the first corresponding point on the input image corresponding to the characteristic point of the reference image is calculated by the Hough transform, and the affine transformation coefficient for converting the characteristic point into the corresponding point is calculated. The calculated amount of deviation between the first corresponding point and the second corresponding point obtained from the affine transformation coefficient is compared with a predetermined threshold value, and the third corresponding point that is less than the threshold value is used as a reference point. Since the fourth corresponding point having the deviation amount exceeding the predetermined threshold value is specified by the triangle similarity ratio, the correspondence of the distorted image can be efficiently searched again for the corresponding point having the large deviation amount. There is an effect that a method can be obtained.

Further, according to the invention of claim 5, the evaluation amount is obtained by the Hough transform with the edge point on the edge image of the input image obtained by the triangle similarity ratio using the third corresponding point as the reference point as the center point. Since the fourth corresponding point that is the maximum is specified, it is possible to obtain the distorted image matching method capable of performing the matching quickly and efficiently while increasing the accuracy of the Hough transform. .

According to the sixth aspect of the invention, the vertical and horizontal edge points of the reference image and the horizontal direction of the reference image are determined based on the horizontal and vertical amounts of the direction vector at each edge point located in the edge image of the reference image. Since edge points and vertical edge points are extracted as feature points and the corresponding points on the input image corresponding to each feature point of the reference image are detected, the input image with distortion and the reference image without distortion can be quickly processed. In addition, it is possible to obtain an apparatus for associating a distorted image that can be accurately associated.

According to the seventh aspect of the present invention, the sum of the horizontal and vertical amounts within a predetermined area centered on each edge point of the edge image of the reference image is defined as the vertical and horizontal bidirectional amounts, The horizontal and vertical amounts in the area are respectively greater than or equal to a predetermined value, and the points with the maximum vertical and horizontal directions are extracted as both vertical and horizontal edge points, and each edge point of the edge image of the reference image is set as the center. A point in which the lateral amount in the prescribed area is equal to or greater than a prescribed value and the lateral amount is maximized is extracted as a lateral edge point, and a predetermined value is set around each edge point of the edge image of the reference image. Since the point in which the vertical direction amount in the area is equal to or larger than a predetermined value and the vertical direction amount is maximum is extracted as a vertical edge point, an edge point having a lot of information about corresponding points is characterized. It is possible to make a point and to make an accurate correspondence. An effect that is obtained associating device such distorted image is.

Further, according to the invention of claim 8, the first corresponding point on the input image corresponding to the characteristic point of the reference image is calculated by the Hough transform, and the affine transformation coefficient for converting the characteristic point into the corresponding point is calculated. The calculated amount of deviation between the first corresponding point and the second corresponding point obtained from the affine transformation coefficient is compared with a predetermined threshold value, and the third corresponding point that is less than the threshold value is used as a reference point. Since the fourth corresponding point having the deviation amount exceeding the predetermined threshold value is specified by the triangle similarity ratio, the correspondence of the distorted image can be efficiently searched again for the corresponding point having the large deviation amount. There is an effect that the device can be obtained.

According to the invention of claim 9, claim 1
By causing a computer to execute the method described in any one of claims 1 to 5, the program can be machine-readable, and thereby the operation of any one of claims 1 to 5 can be realized by the computer. There is an effect that various programs can be obtained.

[Brief description of drawings]

FIG. 1 is a functional block diagram showing a configuration of an image associating device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram for explaining a concept of calculating a direction amount.

3 is a flowchart showing a processing procedure in a template registration mode by the image associating device shown in FIG.

FIG. 4 is a diagram showing an example of a template in a template registration mode.

5 is an enlarged view of FIG. 4D showing vertical and horizontal edge points, a horizontal edge point and a vertical edge point.

6 is a flowchart showing a processing procedure in an associating mode by the image associating apparatus shown in FIG.

FIG. 7 is a diagram showing an example of an input image and the like in the association mode.

FIG. 8 is an enlarged view of FIG. 7D showing vertical and horizontal edge points, a horizontal edge point, and a vertical edge point.

9 is a flowchart (1) showing a characteristic point extraction processing procedure by the characteristic point extraction unit shown in FIG. 1;

10 is a flowchart (2) showing a procedure of extracting feature points by the feature point extracting section shown in FIG. 1.

FIG. 11 is an explanatory diagram (1) illustrating an example of image data in the middle of feature point extraction processing.

FIG. 12 is an explanatory diagram (2) illustrating an example of image data in the middle of feature point extraction processing.

13 is a flowchart showing a processing procedure of a corresponding point detection unit shown in FIG.

FIG. 14 is a diagram showing an example of an image when detecting corresponding points.

FIG. 15 is an explanatory diagram for explaining how to obtain corresponding points of both vertical and horizontal edge points having a large shift amount.

FIG. 16 is an explanatory diagram for explaining reference points when a matching amount of Hough transform is calculated using a similarity ratio of two triangles.

FIG. 17 is an explanatory diagram for explaining interpolation between data.

FIG. 18 is an explanatory diagram for explaining distortion that occurs in an input image.

[Explanation of symbols]

100 image matching device 101 Image input section 102 Cutting processing unit 103 Edge image creation unit 104 Feature point extraction unit 105 storage 106 corresponding point detector

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 5B029 BB03 CC29 EE12                 5B057 AA11 CA08 CA12 CA16 CB08                       CB12 CB16 CD12 CE15 DA08                       DB02 DB09 DC05 DC13 DC16                       DC34 DC36 DC39

Claims (9)

[Claims] 1. A method of associating a distorted image in which each feature point of a reference image having no distortion is associated with a corresponding point of an input image accompanied by distortion, wherein at each edge point existing in the edge image of the reference image Corresponding to each feature point of the reference image, a feature point extraction step of extracting both vertical and horizontal edge points of the reference image, horizontal edge and vertical edge points as feature points based on the horizontal direction amount and the vertical direction amount of the direction vector. And a corresponding point detecting step of detecting corresponding points on the input image. 2. The feature point extracting step determines a sum of a horizontal direction amount and a vertical direction amount within a predetermined region centered on each edge point of the edge image of the reference image as a vertical and horizontal direction amount, and determines the predetermined amount. A horizontal and vertical edge point extraction step of extracting, as the vertical and horizontal edge points, a point in which the horizontal and vertical amounts in the area are respectively equal to or greater than a predetermined value, and the vertical and horizontal directions are maximum. Horizontal edge point extraction for extracting, as the horizontal edge point, a point in which the horizontal amount within a predetermined area centering on each edge point of the edge image is equal to or more than a predetermined value and the horizontal amount is maximum. Steps, the vertical direction amount within a predetermined area centered on each edge point of the edge image of the reference image is a predetermined value or more, and the point where the vertical direction amount is maximum is the vertical edge point. Vertical edge point extraction step for extracting The method for associating a distorted image according to claim 1. 3. The feature point extracting step extracts a vertical edge and a horizontal edge point, a horizontal edge point, or a vertical edge point within a predetermined area centered on a certain edge point, and extracts the edge area within a predetermined area of the edge image. 3. The method for associating a distorted image according to claim 1 or 2, wherein the process of erasing the pixel value of, and extracting the next edge point based on the characteristic amount of the erased edge image is repeated. 4. The Hough transform step of calculating a first corresponding point on the input image corresponding to the feature point of the reference image by Hough transform, and the feature point being the corresponding point. An affine transform coefficient calculating step of calculating the affine transform coefficient to be converted, and a deviation amount between the first corresponding point obtained in the Hough transform step and the second corresponding point obtained from the affine transform coefficient are compared with a predetermined threshold value. And a feature point specifying step of specifying a fourth corresponding point whose shift amount exceeds the predetermined threshold value by a triangle similarity ratio with a third corresponding point which is equal to or less than the threshold value as a reference point. The distortion image associating method according to claim 1, 2 or 3, wherein the distortion image associating method is included. 5. In the feature point specifying step, an evaluation amount is obtained by Hough transform with an edge point on an edge image of the input image obtained by a triangle similarity ratio using the third corresponding point as a reference point as a center point. The method of associating a distorted image according to claim 4, wherein the maximum corresponding fourth corresponding point is specified. 6. A distortion image associating device for associating each feature point of an undistorted reference image with a corresponding point of an input image accompanied by distortion, in each edge point existing in the edge image of the reference image. Based on the horizontal direction amount and the vertical direction amount of the direction vector, both vertical and horizontal edge points of the reference image, feature point extraction means for extracting horizontal edge points and vertical edge points as feature points, and to each feature point of the reference image And a corresponding point detecting means for detecting corresponding points on the corresponding input image. 7. The feature point extracting means sets an addition value of a horizontal direction amount and a vertical direction amount within a predetermined area centered on each edge point of the edge image of the reference image as a vertical and horizontal direction amount, and the predetermined amount. Vertical and horizontal edge point extraction means for extracting as points the vertical and horizontal edge points, the horizontal and vertical amounts in the region are respectively greater than or equal to a predetermined value, and the vertical and horizontal directions are maximum, and the reference image Horizontal edge point extraction for extracting, as the horizontal edge point, a point in which the horizontal amount within a predetermined area centering on each edge point of the edge image is equal to or more than a predetermined value and the horizontal amount is maximum. And a point in which a vertical direction amount in a predetermined area centered on each edge point of the edge image of the reference image is a predetermined value or more, and the maximum vertical direction amount is the vertical edge point. A vertical edge point extracting means for extracting; The distorted image associating device according to claim 6. 8. The corresponding point detecting means calculates a first corresponding point on the input image corresponding to a characteristic point of the reference image by a Hough transform, and the characteristic point to the corresponding point. An affine transform coefficient calculating means for calculating an affine transform coefficient to be converted, and a deviation amount between the first corresponding point obtained by the Hough transforming means and the second corresponding point obtained from the affine transform coefficient are compared with a predetermined threshold value. And a feature point specifying means for specifying a fourth corresponding point whose shift amount exceeds the predetermined threshold value by a triangle similarity ratio with the third corresponding point which is less than or equal to the threshold value as a reference point. The distortion image associating device according to claim 6 or 7, further comprising: 9. A program that causes a computer to execute the method according to any one of claims 1 to 5.