JP2015114949A - Image processor, image processing method, program for image processor, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor or the like capable of simply correcting distortion of an image picked up obliquely.SOLUTION: An original image obtained by obliquely photographing an object in a plane part of a subject is acquired (S1), information of a restoration image of the front of the object is acquired (S2), each feature point corresponding to each vertex forming a first square in the front restoration image is extracted from the original image (S3), conversion mapping for converting the original image into the front restoration image is generated from an area compound ratio in a plurality of first triangles formed by each vertex of the first square and a first point on a plane including the first square and a plurality of second triangles formed by each extracted feature point and a second point corresponding to the first point on a plane including the second square with each feature point as a vertex (S4), and pixel information of the original image corresponding to each pixel of the front restoration image is calculated on the basis of the conversion mapping (S6, S7).

Description

  The present invention relates to an image processing apparatus, an image processing method, a program for an image processing apparatus, and a recording medium that correct image distortion.

  It can be realized by projective transformation of projective geometry to restore the distorted image to its original shape by photographing the subject from an oblique direction. For example, Patent Document 1 discloses a building image composition apparatus that uses the principle of the cross ratio of projective geometry.

Japanese Patent No. 3235929

  However, since the technique of Patent Document 1 uses the distance ratio, when applying the distance ratio ratio principle to an image, it is necessary to select a straight line so that the distance ratio ratio principle holds. Yes, pre-processing was complicated.

  Therefore, the present invention has been made in view of the above-described problems, and an example of the problem is to provide an image processing apparatus and the like that can easily correct distortion with respect to an image captured obliquely. And

  In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that original image acquisition means for acquiring an original image obtained by photographing an object on a plane portion of an object from an oblique direction, and information on a front-reconstructed image of the object. Front-reconstructed image information obtaining means for obtaining, feature-point extracting means for extracting each feature point corresponding to each vertex forming the first rectangle in the front-reconstructed image from the original image, and each vertex of the first rectangle And a plurality of first triangles formed from the first points on the plane including the first rectangle, the extracted feature points, and a plane including the second rectangle having each feature point as a vertex. Conversion map generating means for generating a conversion map for converting the original image into the front-reconstructed image based on the area ratio in the plurality of second triangles formed from the second points corresponding to the first points; Based on the transformation map, And pixel information calculation means for calculating the pixel information of the original image corresponding to each pixel of the front restored image, characterized by comprising a.

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the conversion map includes a plurality of the first triangles formed from the sides of the first quadrangle and the first points. And a plurality of the second triangles formed from the sides of the second quadrangle and the second points, the mapping as a result of a predetermined combination of the formed first triangles and the second triangles It is characterized by being.

  According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, in the predetermined combination, the first quadrangle is a parallelogram, and the first triangle is a parallel of the parallelogram. And the second triangle is a two-sided triangle corresponding to the first base, and the second triangle is a triangle having the second base corresponding to the first base. To do.

  According to a fourth aspect of the present invention, in the image processing device according to any one of the first to third aspects, the first point includes pixel information for drawing in the front restoration image. A drawing point to be obtained and a first prescribed point defined from each vertex of the first quadrangle, wherein the second point is a drawing corresponding point corresponding to the drawing point in the original image, and the first prescribed point. And a second specified point defined from each vertex of the second quadrangle.

  According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the first predetermined point is an intersection of diagonal lines of the first quadrangle, and the second predetermined point is the second It is an intersection of square diagonal lines.

であることを特徴とする。 According to a sixth aspect of the present invention, in the image processing apparatus according to the fifth aspect, the coordinates of the drawing corresponding points are expressed as coordinates (u ₅ , v ₅ ), and the coordinates of the vertices of the second quadrangle. Is expressed as coordinates (u ₁ , v ₁ ) (u ₂ , v ₂ ) (u ₃ , v ₃ ), (u ₄ , v ₄ ) numbered in a predetermined circumferential direction in the second quadrangle, Of the first triangle formed by the vertices of the first square corresponding to the coordinates (u ₁ , v ₁ ) (u ₂ , v ₂ ) (u ₃ , v ₃ ), (u ₄ , v ₄ ) When each area is expressed as [ijk] (i = 1, 2, 3, j = 1, 2, 3, k = 1, 2, 3), the conversion map is

It is characterized by being.

  According to a seventh aspect of the present invention, in the image processing device according to any one of the first to sixth aspects, a restored image feature point that extracts at least four restored image feature points from the front restored image. The image processing apparatus further includes: extraction means; and parallelogram generation means for generating the first quadrangle that is a parallelogram from the restored image feature points.

  The invention according to claim 8 is an image processing method in which an image processing apparatus processes an image, and an original image acquisition step of acquiring an original image obtained by photographing an object in a plane portion of a subject from an oblique direction; A front restoration image information acquisition step for obtaining information of a target front restoration image, and a feature point extraction step for extracting each feature point corresponding to each vertex forming a first square in the front restoration image from the original image; , A plurality of first triangles formed from the vertices of the first quadrangle and the first point on the plane including the first quadrangle, the extracted feature points, and a second having the feature points as vertices A conversion map for converting the original image into the front-reconstructed image is generated from the area ratio of a plurality of second triangles formed on the plane including the quadrangle and corresponding to the second points corresponding to the first points. Conversion An image generation step, based on the conversion map, characterized in that it comprises a and a pixel information calculation step of calculating the pixel information of the original image corresponding to each pixel of the front restored image.

  According to the ninth aspect of the present invention, there is provided a computer, an original image acquisition unit that acquires an original image obtained by photographing an object on a plane portion of an object from an oblique direction, and a front restored image that obtains information on the front restored image of the object. Information acquisition means, feature point extraction means for extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image, each vertex of the first square and the first square A plurality of first triangles formed from the first points on the plane, the extracted feature points, and a plane including a second quadrangle having the feature points as vertices, corresponding to the first points. Based on an area ratio in a plurality of second triangles formed from the second point, a conversion map generating means for generating a conversion map for converting the original image into the front restoration image, and the conversion map, front The function as pixel information calculation means for calculating the pixel information of the original image corresponding to each pixel of the original image.

  Further, the invention according to claim 10 is an original image acquisition means for acquiring an original image obtained by photographing a target in a plane portion of a subject from an oblique direction, and a front-restored image for acquiring information on a front-restored image of the target. Information acquisition means, feature point extraction means for extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image, each vertex of the first square and the first square A plurality of first triangles formed from the first points on the plane, the extracted feature points, and a plane including a second quadrangle having the feature points as vertices, corresponding to the first points. Based on an area ratio in a plurality of second triangles formed from the second point, a conversion map generating means for generating a conversion map for converting the original image into the front restoration image, and the conversion map, Positive The image processing device program to function as the pixel information calculation means for calculating the pixel information of the original image corresponding to each pixel of the restored image is recorded on a computer-readable.

  According to the present invention, it is possible to easily obtain a front restoration image in which distortion is corrected with respect to an original image taken from an oblique direction.

It is a mimetic diagram showing an example of outline composition of an information processing system concerning an embodiment of the present invention. It is a block diagram which shows the example of a schematic structure of the image processing apparatus of FIG. (A) It is a schematic diagram which shows the object in an original image. (B) It is a schematic diagram which shows the object in a front restoration image. (A) It is a schematic diagram which shows the intersection of the diagonal of the object rectangle in an original image. (B) It is a schematic diagram which shows the intersection of the diagonal of the object rectangle in a front restoration image. It is a schematic diagram which shows the specific method of the area of a triangle. (A) It is a schematic diagram which shows the point corresponding to the intersection of the diagonal of the object rectangle in an original image, and a drawing point. (B) It is a schematic diagram which shows the intersection and drawing point of the diagonal line of the object rectangle in a front restoration image. (A) It is a schematic diagram for demonstrating the principle which concerns on embodiment. (B) It is a schematic diagram for demonstrating the principle which concerns on embodiment. (A) It is a schematic diagram for demonstrating the principle which concerns on embodiment. (B) It is a schematic diagram for demonstrating the principle which concerns on embodiment. It is a schematic diagram for demonstrating the principle which concerns on embodiment. 3 is a flowchart illustrating an operation example of the information processing system in FIG. 1. It is a schematic diagram which shows an example of the object in an original image. FIG. 11 is a flowchart showing an example of a subroutine for generating a conversion map of area cross ratio in FIG. 10. FIG. (A) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (B) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (A) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (B) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (A) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (B) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (A) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (B) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map. (A) It is an example which shows the 1st modification of the object in an original image. (B) It is a schematic diagram which shows the 1st modification of the object in a front restoration image. (A) It is an example which shows the 2nd modification of the object in an original image. (B) It is a schematic diagram which shows the 2nd modification of the object in a front restoration image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an embodiment when the present invention is applied to an image processing system.

[1. Overview of image processing system configuration and functions]
(1.1 Configuration and Function of Image Processing System 1)

  First, the configuration and outline function of an image processing system according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is a block diagram illustrating a schematic configuration example of an image processing system 1 according to the present embodiment.

  As shown in FIG. 1, an image processing system 1 (an example of an image processing apparatus) is a camera 5 that captures an object in a plane portion of a subject such as a vehicle license plate (a target T in the plane portion of the subject) from an oblique direction. And an image processing device 10 that performs image processing on an image photographed by the camera 5.

  The camera 5 is a digital camera having an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  The image processing apparatus 10 restores the original image from the camera 5 to a front restoration image.

(1.2 Configuration and Function of Image Processing Apparatus 10)
Next, the configuration and function of the image processing apparatus 10 will be described with reference to FIG.

  FIG. 2 is a block diagram illustrating a schematic configuration example of the image processing apparatus 10.

  As illustrated in FIG. 2, the image processing apparatus 10 includes a communication unit 11, a storage unit 12, a display unit 13, an operation unit 14, an input / output interface unit 15, and a system control unit 16. . The system control unit 16 and the input / output interface unit 15 are connected via a system bus 17.

  The communication unit 11 is connected to a network and controls a communication state with the network.

  The storage unit 12 (an example of a storage unit) includes, for example, a hard disk drive, a silicon disk drive, and the like, and stores various programs such as an operating system and a server program, map information, and the like. Note that the various programs may be acquired from, for example, another server device via a network, or may be recorded on a recording medium and read via a drive device.

  The storage unit 12 has, as a front-restored image database, information on a front-restored image when the object in the plane portion of the subject is imaged from the front. As information of the front restoration image, the shape of the target, the coordinates of the feature points, the relationship between the target feature points, and the like are stored in the storage unit 12 in advance. For example, when the vehicle license plate is rectangular, the coordinates of each vertex of the rectangle and the positional relationship of each vertex, the length of each side of the rectangle, the ratio of each side of the rectangle, the size of the rectangle, etc. Information is stored in the storage unit 12 in advance.

  Here, when the target is a parallelogram, the coordinates of each vertex of the parallelogram and the positional relationship of each vertex, the length of each side of the parallelogram, the ratio of each side of the parallelogram, Information of the front restoration image, such as the size, can be given. If the target is a general quadrilateral, the coordinates of each vertex of the quadrilateral and the positional relationship of each vertex, the length of each side of the quadrilateral, the ratio of each side of the quadrilateral, the size of the quadrilateral, etc. Information on the front restoration image is given.

  In addition, when the target is not a parallelogram, information on the front restoration image such as the coordinates of the target feature points and the positional relationship between the feature points is included. The number of feature points is preferably 4 or more.

  The plane portion of the subject may be a portion that can be approximated by a plane.

  The display unit 13 is configured by, for example, a liquid crystal display element or an EL (Electro Luminescence) element.

  The operation unit 14 is configured by, for example, a keyboard and a mouse.

  The input / output interface unit 15 performs interface processing between the communication unit 11 and the like and the system control unit 16. The input / output interface unit 15 is connected to the camera 5.

  The system control unit 16 includes a CPU (Central Processing Unit) 16a, a ROM (Read Only Memory) 16b, a RAM (Random Access Memory) 16c, and the like.

[2. Principle of image processing using area cross ratio]
Next, the principle of image processing using the area ratio will be described with reference to the drawings.

(2.1 Projective transformation and distance ratio)
It is well known that a license plate image (an example of an object in a plane portion of an object) that is distorted by photographing from an oblique direction can be realized by a method called projective transformation of projective geometry in order to restore the original shape. . In order to perform projective transformation, it is necessary to obtain a 3 × 3 component projection matrix. Projection matrix components give a corresponding four-point relationship between the distorted image and the restored image, and solve the linear equation, which is computationally intensive and difficult to obtain a numerically stable solution. . Therefore, another concept of projective geometry, the cross ratio theorem (principle) is known to reduce the amount of calculation, and a method using the same has been proposed (for example, Patent Document 1).

  However, in Patent Document 1, since the cross ratio of the distance is used, when the principle of the distance cross ratio is applied to the image in the object in the plane portion of the subject, the cross ratio is calculated on the selected straight line. The principle must be established. That is, there is a complexity of preprocessing for applying the principle of the cross ratio and its drawing control procedure.

(2.2 Overview of the principle of area ratio)
Next, the outline of the principle of area cross ratio will be described with reference to FIGS. 3 and 4 using a rectangle which is an example of a parallelogram.
FIG. 3A is a schematic diagram showing a target in the original image. FIG. 3B is a schematic diagram illustrating a target in the front restoration image. FIG. 4A is a schematic diagram showing the intersection of diagonal lines of a target rectangle in the original image. FIG. 4B is a schematic diagram illustrating the intersection of diagonal lines of a target rectangle in the front restoration image.

  Instead of a straight line distance that is difficult to directly correspond to an image, a cross-ratio relationship of areas that are thought to be directly compatible is used. As shown in FIG. 3 (A), quadrilateral vertices Q1, Q2, Q3, and Q4 in the original image taken from an oblique direction and distorted into a quadrangle, and in the front-reconstructed image as shown in FIG. 3 (B). Consider a combination of triangles with respect to a pair of rectangular vertices P1, P2, P3, and P4 and a pair of points P5 to be restored and drawn in the rectangle and a corresponding point Q5 of the original image.

  Here, for example, select four different triangles (area ratio of 2 on the numerator side and 2 on the denominator side), and combine the area of triangles with the same area ratio of the original image and the front restoration image. It is defined as

  As shown in FIGS. 4 (A) and 4 (B), by selecting a pair of intersections (P0, Q0) of the diagonals of a rectangle and a rectangle as a pair of vertices that satisfy the principle of area cross ratio, the rectangle side becomes a molecule and Can be offset because the area is equal in the denominator. Also, as shown in FIGS. 3A and 3B, the area of the triangle does not change if the height of the triangle is the same as the base, and therefore the area does not change even if the position is moved in the operation on the drawing line Y. The fact that it is only necessary to calculate the areas on the two vertical and horizontal lines passing through the diagonal intersections, not the drawing points of the entire rectangle, makes use of the fact that the amount of calculation can be greatly reduced.

(2.3 Details of the principle of area cross ratio)
Next, details of the principle of area cross ratio will be described with reference to FIGS.
FIG. 5 is a schematic diagram showing how to specify the area of a triangle. FIG. 6A is a schematic diagram showing points corresponding to the intersections of the diagonal lines of the target rectangle and the drawing points in the original image. FIG. 6B is a schematic diagram showing intersections and drawing points of the diagonal lines of the target rectangle in the front restoration image. 7A, 7B, 8A, 8B, and 9 are schematic diagrams for explaining the principle according to the embodiment.

  In order to restore a distorted plate image obtained by photographing from an oblique direction to its original shape, it is generally realized by a method called projective transformation of projective geometry. In order to perform projective transformation, it is necessary to obtain a 3 × 3 component projection matrix H. The projection matrix component is obtained by solving a linear simultaneous equation by giving a corresponding four-point relationship between the distorted image and the restored image.

  On the other hand, the cross ratio used in Patent Document 1 is an invariant equation regarding a distance ratio on a straight line. The cross ratio used in the present application is an invariant equation relating to the area ratio. In order to explain these differences and features, their relationship and properties will be explained using a matrix of linear algebra (Matrix) and a determinant (Determinant).

First, in order to explain the difference between the distance cross ratio with a concept of distance and the method of projective transformation, the real dimension is a distance because it is a distance, but in order to deal with the projective transformation, a two-dimensional Expressed in a homogeneous coordinate system.

である。復元被写体像の座標系がｘ、被写体の斜め撮影像の座標系がｕとする。 here,

It is. The coordinate system of the restored subject image is x, and the coordinate system of the oblique photographed image of the subject is u.

となる。 Then, when the two points x _i and x _j and the corresponding points u _i and u _j of the respective projective transformations are expressed in a matrix using equation (2),

It becomes.

となる。 And when calculating the determinant of Equation (2),

It becomes.

や

であることに注意する。 Here, in the properties of the determinants of the matrices A and B,

And

Note that

とおいて、数式（３）の行列式を計算すると、

である。

When calculating the determinant of Equation (3),

It is.

Here, the determinant || of the 1 × 1 matrix represents the distance. Therefore, consider the following ratio equations for the four points x ₁ , x ₂ , x ₃ , x ₄ on the straight line and the corresponding points u ₁ , u ₂ , u ₃ , u ₄ of the respective projective transformations.

Here, | H | is omitted because it is clearly offset by the denominator and the numerator. Furthermore, the term of h (x _i ) is also canceled by the denominator and the numerator, and as a result, Equation (5) is derived. This clarifies the relationship between the distance cross-ratio and the projective transformation method and the differences. That is, the projection matrix H of the projective transformation method changes its appearance to the projection determinant | H | in the distance cross ratio and disappears from the formula (5).

Next, the area ratio will be described. The formula (1) of the 2 × 2 matrix is expanded to the formula 6 of the 3 × 3 matrix.

となる。 Then, three points (x _i , y _i ) (x _j , y _j ) (x _k , y _k ) and corresponding points (u _i , v _i ) (u _j , v _j ) (u) of the respective projective transformations _k , v _k ), expressed as a matrix using Equation (6),

It becomes.

を得る。 Furthermore, it is expressed as a determinant like Equation (3),

Get.

Following the mathematical formula (5) on the straight line, as shown in FIG. 3B, five points P1 (x ₁ , y ₁ ), P2 on the surface (rectangle of the front restoration image (an example of the first quadrangle)), P2 (X ₂ , y ₂ ), P 3 (x ₃ , y ₃ ), P 4 (x ₄ , y ₄ ), P 5 (x ₅ , y ₅ ), and their projections as shown in FIG. Corresponding points Q1 (u ₁ , v ₁ ), Q2 (u ₂ , v ₂ ), Q3 (u ₃ , v ₃ ), Q4 (u ₄ , For v ₄ ) and Q 5 (u ₅ , v ₅ ), an equation with the following ratio is considered from Equation (8).

  Here, as shown in FIG. 5, the order of the vertices of the triangles is unified in the clockwise direction with respect to the determinant. This is because the sign of the determinant is reversed when it is replaced.

The determinant of a triangle vertex is twice the area of the triangle. Then, a point P0 (x ₀ , y ₀ ) and its corresponding point Q0 (u ₀ , v ₀ ) are added, which means that one of the five points is selected. Formula (9) is an example of an invariant relating to the area, that is, the area cross ratio C _r1 .

  As shown in FIGS. 3A and 3B, if the points P1, P2, P3, and P4 and the corresponding points Q1, Q2, Q3, and Q4 are four vertices of a quadrangle, their coordinate values are You can know by measuring.

Next, a point P5 (x ₅ , y ₅ ) that is a drawing point (an example of a first point) on the scan line Y for drawing a front-reconstructed image of the subject is designated. Then, the coordinate values (u ₅ , v ₅ ) of the point Q5 that is the drawing corresponding point (an example of the second point) on the corresponding scan line Y ′ on the image sensor are obtained by solving the equation (9). be able to.

の形をした２変数（ｕ_５、ｖ_５）に関する１次方程式になる。 That is, when formula (9) is expanded and arranged with respect to the coordinate values (u ₅ , v ₅ ),

A linear equation with respect to two variables (u ₅ , v ₅ ) in the form of

Therefore, another equation is necessary to solve a linear equation of two variables (u ₅ , v ₅ ). As the candidate, another combination different from the multi-ratio C _r1 in the formula (9) can be considered in the multi-ratio regarding (u ₅ , v ₅ ). For example, the next multi-ratio C _r2 is selected and expanded. Equation (11) is obtained.

The coefficients (a _i , b _i , c _i ) (i = 1, 2) are determined from the respective expansions of Equation (9) and Equation (11). Equations (10) and (12) are combined to obtain (u ₅ , v ₅ ).

のかたちである。 The solution (u ₅ , v ₅ ) is

It is a form.

と定義しておくと、数式（９）と数式（１１）は、それぞれ、

と表せる。 Before expanding and assigning to each coefficient,

If defined, Equation (9) and Equation (11) are respectively

It can be expressed.

  Here, as shown in FIG. 5, when the triangle vertices are numbered, the area of the second triangle on the square side of the original image is (i, j, k), and the area of the first triangle on the rectangle side of the front restoration image Is represented as [i, j, k].

As for the area ratio, since 3 points are selected from 5 points in the definition formula of the formula (8), the number of combinations is calculated using the formula: ₅ C ₃ = 5! / {(5-3)! 3! } = (5 × 4) / 2 = 10.

When enumerating with ignoring the sign, it is as follows.
(123), (124), (125), (134), (135), (145), (234), (235), (245), (345)

  As can be seen from Equation (9), the same point is added to the denominator and the numerator.

である。 When the fifth point (an example of a drawing corresponding point) is fixed as follows and the area cross ratio is considered,

It is.

などの組み合わせがある。 First, for the combination (◯◯ 5), 2 points are selected from the remaining 4 points, so there are ₄ C ₂ = 4 × 3/2 = 6. For the remaining (XXX) combinations, i, j, k, and l are different numbers of 4 or less because the same point is constrained by the condition that the denominator and numerator cancel each other.

There are combinations.

  These are the plurality of first triangles formed from the sides of the first quadrangle and the first point, and the plurality of second triangles formed from the side of the second quadrangle and the second point. And it is an example of the predetermined combination of each formed said 1st triangle and each said 2nd triangle. Further, in the predetermined combination, the first quadrangle is a parallelogram, the first triangle is a triangle having two parallel sides of the parallelogram as first bases, and the second This is an example in which a triangle is a triangle that is two opposite sides of the second quadrangle and that is each second base corresponding to the first base.

  The number that appears twice in the denominator and numerator is i. If the fifth point appears twice in the denominator and numerator, it will become a quadratic equation and it will be difficult to solve this problem, so we will not consider it this time. do it.

  Now, once again, formula (15) is reviewed and a combination that makes calculation as simple as possible is considered. Until now, the 0th point has been set to one of the 1st to 4th points, but as shown in FIGS. 6A and 6B, the diagonal point P0 (the first point) is changed to the 0th point. If an example of a point, an example of a first specified point) is selected, since the areas of the first triangles [320] and [140] and [430] and [210] are equal to each other, they can be deleted. The equation becomes as follows:

Thus, C _ry and C _rx are formed from the plurality of first triangles formed from the sides of the first square and the first point, and from the sides of the second square and the second point. It is an example of a predetermined combination of the formed first triangles and the second triangles. In the predetermined combination, C _ry and C _rx are triangles in which the first quadrangle is a parallelogram, and the first triangle is the first base of two parallel sides of the parallelogram. The second triangle is an example in which the second quadrangle is two opposite sides of the second quadrangle and is a second base corresponding to the first base.

Therefore, Equation (16) is modified as follows and developed as simultaneous equations of variables (u ₅ , v ₅ ).

であることから、

である。 Expanding the determinant for variables (u ₅ , v ₅ ) and moving the left and right terms together,

Because

It is.

  In the drawing process, when considering what happens when the triangle is degenerated into a straight line, in Equation (18), the denominator is multiplied to eliminate division, so the problem of division by zero does not occur. Substituting the coefficient equation (19) into the equation (13) and putting together the calculation equations simplifies the calculation.

here,

  As shown in FIGS. 7A and 7B and FIGS. 8A and 8B, in Expression (22), the area does not change even if y5 (vertical direction) and x5 (horizontal direction) move. , Respectively, can be represented by the intersection (x0, y0) (an example of the first point, an example of the first specified point). As a result, as shown in FIG. 9, the area calculation for the triangle can be performed only for the vertical line x0 and the horizontal line y0, and the amount of calculation can be greatly reduced.

From here, formula expansion, factorization, reduction, etc. are performed so that the calculation formula is as simple as possible. First, since the intersection of the diagonals of the rectangle of the first quadrangle is the middle of the four vertices, it is not necessary to calculate it, and what is calculated is the intersection Q0 (u ₀ , v ₀ ) of the two diagonals of the second quadrangle. That is, the intersection (second) from the diagonal line of the four vertices Q1 (u ₁ , v ₁ ), Q2 (u ₂ , v ₂ ), Q3 (u ₃ , v ₃ ), Q4 (u ₄ , v ₄ ) of the second square An example of a point, an example of a second specified point). It is directly substituted into the area formulas of the following second triangles (320), (140), (210), and (430) in the second square. Factoring the results together, the following is a simple formula:

The formula (23) is organized as follows.

となる。 Then

It becomes.

In Equation (20), the common product terms h ₁ h ₂ h ₃ h ₄ appear in the numerator and denominator and can be canceled out.

That is, the mathematical formula (25) is surprisingly simple as follows.

となる。 Then, Equation (20) is rewritten as simplified by Equation (26). The transformation map is

It becomes.

[3. Operation of image processing system]
Next, the operation of the image processing system according to the embodiment of the present invention will be described with reference to the drawings.

(3.1 Operation of image processing apparatus)
The operation of the image processing apparatus will be described with reference to FIGS.
FIG. 10 is a flowchart illustrating an operation example of the information processing system. FIG. 11 is a schematic diagram illustrating an example of an object in an original image.

  First, the camera 5 captures an image of a target T (for example, a license plate) in a plane portion of a subject such as a vehicle from an oblique direction.

  As shown in FIG. 10, the image processing apparatus 10 acquires an original image (step S1). Specifically, as shown in FIG. 11, the system control unit 16 of the image processing apparatus 10 acquires the original image 20 of the target T on the plane portion of the subject from the camera 5.

  As described above, the image processing apparatus 10 functions as an example of an original image acquisition unit that acquires an original image obtained by photographing an object on a plane portion of a subject from an oblique direction.

Next, the image processing apparatus 10 acquires information on the front restoration image (step S2). Specifically, the system control unit 16 of the image processing apparatus 10 acquires, from the storage unit 12, coordinate information indicating the positional relationship of four or more feature points when the target is imaged from the front. Then, the system control unit 16 selects four points to be used for the area cross ratio conversion mapping. When the object is a license plate of a vehicle, as shown in FIG. 6B, coordinate information (P1 (x ₁ , y ₁ ), P2 (x ₂ , y ₂ ), P3 of each vertex of the license plate that is a rectangle (X ₃ , y ₃ ), P4 (x ₄ , y ₄ )) is acquired.

  In this manner, the image processing apparatus 10 functions as an example of a front restored image information acquisition unit that acquires information on the target front restored image.

Next, the image processing apparatus 10 extracts feature points corresponding to the four points of the front restoration image from the original image (step S3). Specifically, the system control unit 16 of the image processing apparatus 10 extracts the edge from the acquired original image 20 by performing differentiation processing or the like, and corresponds to the selected four points of the front restoration image from the edge. Extract feature points. As shown in FIG. 6A, each coordinate Q1 (u ₁ , v ₁ ), Q2 (u ₂ , v ₂ ), Q3 (u ₃ , v ₃ ), Q4 (u ₄ , v ₄ ) of the feature point. Is calculated.

Here, a first quadrangle is formed by the selected four points of the front restoration image. A second quadrangle is obtained by the feature points Q1 (u ₁ , v ₁ ), Q2 (u ₂ , v ₂ ), Q3 (u ₃ , v ₃ ), Q4 (u ₄ , v ₄ ) corresponding to the four points of the front restoration image. Is formed.

  As described above, the image processing apparatus 10 functions as an example of a feature point extracting unit that extracts each feature point corresponding to each vertex forming the first quadrangle in the front restoration image from the original image.

Next, the image processing apparatus 10 generates an area cross ratio conversion map based on the four points of the front restoration image and the extracted feature points (step S4). Specifically, the system control unit 16 determines that the intersection point P0 (an example of the first point, an example of the first specified point) of the rectangular diagonal line, that is, the intersection point of the straight line P1-P3 and the straight line P2-P4, P0 ( x ₀ , y ₀ ) is calculated. Then, the system control unit 16 calculates the intersection Q0 of the square diagonal of the feature points, that is, the intersection of the straight line Q1-Q3 and the straight line Q2-Q4, Q0 (u ₀ , v ₀ ).

  The system control unit 16 then coordinates information of the four vertices and intersections (an example of the first point and an example of the first specified point) of the front restoration image, and the four extracted feature points and intersections (of the second point). The coefficient of Equation (27) is calculated from the coordinate information of an example, an example of the second specified point), and a conversion map of the area cross ratio is generated. The details of the generation of the area cross ratio conversion map will be described in the area cross ratio map conversion subroutine.

  As described above, the image processing apparatus 10 includes a plurality of first triangles formed from the vertices of the first rectangle and the first points on the plane including the first rectangle, the extracted feature points, and the From the original image, the front surface from the area ratio of a plurality of second triangles formed on the plane including the second quadrangle having each feature point as the vertex and corresponding to the first point It functions as an example of a conversion map generation unit that generates a conversion map to be converted into a restored image.

Next, the image processing apparatus 10 specifies a drawing point (step S5). Specifically, the system control unit 16 of the image processing apparatus 10 specifies the target drawing point P5 (x ₅ , y ₅ ) in the front restoration image, as shown in FIG. 6B. For the drawing point P5 (x ₅ , y ₅ ), for example, the drawing point is specified by sequentially scanning the scan line Y of the raster scan. In the original image, the trajectory of the scan line Y ′ is followed.

  When the scan line Y is on the straight line P1-P4 (straight line P2-P3), the triangle [1, 4, 5] (triangle [3, 2, 5]) is collapsed into a straight line. In (16) and the like, if the area cross ratio is not a division format but a multiplication format, division by zero can be prevented.

Next, the image processing apparatus 10 calculates the coordinates of the drawing corresponding point of the drawing point from the converted map (step S6). Specifically, the system control unit 16 of the image processing apparatus 10 uses the generated conversion map (Equation (27)) to use the drawing corresponding point Q5 (u) corresponding to the drawing point P5 (x ₅ , y ₅ ). ₅ , v ₅ ).

Next, the image processing apparatus 10 performs interpolation processing on the original image to calculate pixel information of the drawing point (step S7). Specifically, the system control unit 16 of the image processing apparatus 10 obtains pixels of the original image 20 around the drawing corresponding point Q5 (u ₅ , v ₅ ), interpolates the pixel information of the obtained pixels, Pixel information of the drawing point P5 (x ₅ , y ₅ ) is calculated. Here, since the coordinate value of the drawing corresponding point Q5 (u ₅ , v ₅ ) may not be an integer, the pixel information of the pixels adjacent to the drawing corresponding point Q5 (u ₅ , v ₅ ) is averaged. Perform value interpolation.

Next, the image processing apparatus 10 displays pixel information (step S8). Specifically, the system control unit 16 of the image processing apparatus 10 displays the calculated pixel information at the position of the drawing point P5 (x ₅ , y ₅ ) on the display unit 13.

  Next, the image processing apparatus 10 determines whether or not it is the last drawing point (step S9). Specifically, the system control unit 16 of the image processing apparatus 10 determines whether or not the front restoration image is the last drawing point in the raster scan.

  If it is not the last drawing point (step S9; NO), the process returns to step S5, and the image processing apparatus 10 specifies the next drawing point P5 in the raster scan.

  If it is the last drawing point (step S9; YES), the image processing apparatus 10 ends the process.

(3.2 Subroutine of area conversion ratio mapping)
Next, an area cross ratio mapping conversion subroutine will be described with reference to FIGS.
FIG. 12 is a flowchart illustrating an example of a subroutine for generating an area cross ratio conversion map. FIGS. 13A and 13B to FIGS. 16A and 16B are schematic diagrams showing an example of combinations of triangles in the conversion map. (B) It is a schematic diagram which shows an example of the combination of the triangle in a conversion map.

The image processing apparatus 10 calculates and stores the triangular area in the first direction (for example, the y direction) (step S11). Specifically, the system control unit 16 of the image processing apparatus 10 determines the area of the _Cry triangle of Expression (16) as shown in FIGS. 13 (A), 13 (B), 14 (A), and 14 (B). Calculate and save in RAM 16c or the like. For example, the triangle [3, 2, 0] of the first triangle, the triangle [1, 4, 5], the triangle [1, 4, 0], the triangle [3, 2, 5], and the triangle of the second triangle ( 3, ₂ , ₀ ), the area of the triangle ( ₁ , 4, ₀ ) is P0 (x ₀ , y ₀ ), P1 (x ₁ , y ₁ ), P2 (x ₂ , y ₂ ), P3 (x ₃ , Y ₃ ), P 4 (x ₄ , y ₄ ), P 5 (x ₅ , y ₅ ), Q 0 (u ₀ , v ₀ ), Q 1 (u ₁ , v ₁ ), Q 2 (u ₂ , v ₂ ), It is calculated from Q3 (u ₃ , v ₃ ) and Q4 (u ₄ , v ₄ ).

Here, FIGS. 13A and 13B and FIGS. 14A and 14B show combinations of the first triangle and the second difference _{angle in} the _Cry denominator of Equation (16).

Next, the image processing apparatus 10 calculates and stores a triangular area in the second direction (for example, the x direction) (step S12). Specifically, the system control unit 16 of the image processing apparatus 10 determines the area of the triangle of _{Crx in} Expression (16) as shown in FIGS. 15 (A), 15 (B), 16 (A), and 16 (B). Calculate and save in RAM 16c or the like. For example, the triangle [4, 3, 0] of the first triangle, the triangle [2, 1, 5], the triangle [2, 1, 0], the triangle [4, 3, 5], and the triangle of the second triangle ( 4, ₃ , ₀ ), the area of the triangle ( ₂ , ₁ , ₀ ) is P0 (x ₀ , y ₀ ), P 1 (x ₁ , y ₁ ), P ₂ (x ₂ , y ₂ ), P 3 (x ₃ , Y ₃ ), P 4 (x ₄ , y ₄ ), P 5 (x ₅ , y ₅ ), Q 0 (u ₀ , v ₀ ), Q 1 (u ₁ , v ₁ ), Q 2 (u ₂ , v ₂ ), It is calculated from Q3 (u ₃ , v ₃ ) and Q4 (u ₄ , v ₄ ).

Here, FIGS. 15A and 15B and FIGS. 16A and 16B show combinations of the first triangle and the second triangle in the denominator of C _{rx in} Expression (16).

  When the front restoration image is a parallelogram, the first direction and the second direction are directions parallel to the sides of the parallelogram.

Next, the image processing apparatus 10 generates a conversion map of the parallelogram area (step S13). Specifically, the system control unit 16 of the image processing apparatus 10 calculates the coefficients in Equation (27) (Equation (24) and Equation (26)) as the triangle area in the first direction and the triangle in the second direction. Conversion of the parallelogram region by calculating from the area and Q1 (u ₁ , v ₁ ), Q2 (u ₂ , v ₂ ), Q3 (u ₃ , v ₃ ), Q4 (u ₄ , v ₄ ) Generate a map.

  As described above, according to the present embodiment, the original image 20 corresponds to the plurality of first triangles formed from the vertices (P1, P2, P3, P4) of the first square of the front restoration image and the first point P0. Frontal reconstruction from the original image 20 from the area ratio of each vertex (Q1, Q2, Q3, Q4) of the second square to be formed and a plurality of second triangles formed from the second point Q0 corresponding to the first point Since a conversion map to be converted into an image is generated, and pixel information (Q5) of the original image 20 corresponding to each pixel P5 of the front restoration image is calculated based on the generated conversion map, the original image captured from an oblique direction A front-reconstructed image in which distortion is corrected with respect to 20 can be easily obtained.

  As shown in Equation (20) or Equation (27), it is not necessary to solve the projection matrix component equation, and the amount of calculation is reduced. Further, since the calculation target points are fixed except for the drawing point P5, the drawing control is easy. As shown in the mathematical formula (20) or the mathematical formula (27), since the mathematical formula calculation formula is simple, the present application is easier to calculate and control than the conventional method. Therefore, the present application is suitable for application in an embedded device.

  Moreover, since it becomes a simple conversion map in programming, coding becomes easy and bugs are reduced. Further, when the target is a vehicle license plate, the distortion is corrected and the recognition rate can be increased.

  In addition, the transformation map includes a plurality of first triangles (for example, triangles P1P4P0 and the like) formed by the sides of the first square (for example, sides P1P4 and the like) and the first points (for example, intersection point P0 and drawing point P5). And a plurality of second triangles (for example, triangle Q1Q4Q0, etc.) formed from the sides of the second quadrangle (for example, side Q1Q4, etc.) and the second point (for example, intersection point Q0, drawing corresponding point Q5). In the case of a mapping as a result of a predetermined combination of each formed first triangle and each second triangle, the conversion mapping is simplified by combining each first triangle and each second triangle.

  In addition, in the predetermined combination, the first quadrangle is a parallelogram (for example, a rectangle P1P2P3P4), and the first triangle has two parallel sides of the parallelogram as the first bases (for example, the side P1P4 and the side P2P3). And the second triangle is the two opposite sides of the second quadrangle (eg, quadrangle Q1Q2Q3Q4) and each second base (eg, side Q1Q4, side) corresponding to the first base Q2Q3) is a triangle (for example, triangle Q1Q4Q5) (for example, in the case of a combination of triangles shown in Equation (15), Equation (16), etc.), the first rectangle is a parallelogram. By setting parallel scan lines Y, the area of the first triangle in the scan lines is equal, so the formula in area cross ratio is simplified.And at arbitrary points parallel to each side, the area of the triangle formed from these arbitrary points and the two vertices of the horizontal or vertical sides of the parallelogram is the same as long as the area of the triangle does not change from the base Therefore, the movement on the scan line is unchanged.

  In addition, the first point includes a drawing point P5 for obtaining pixel information for drawing in the front restoration image, and a first specified point (for example, the intersection point P0) defined from each vertex of the first square, When the two points include the drawing corresponding point Q5 corresponding to the drawing point in the original image and the second specified point (for example, the intersection point Q0) corresponding to the first specified point and defined from each vertex of the second quadrangle. A first triangle formed from a given drawing point based on an area ratio of a first triangle formed from the first specified point and a second triangle formed from the second specified point, and a desired drawing An algebraic formula for obtaining the drawing corresponding point is determined from the area ratio with the second triangle formed from the corresponding point, and the pixel information of the drawing point can be easily obtained from the pixel information of the drawing corresponding point.

  Further, when the first specified point is the intersection point P0 of the diagonal lines of the first quadrangle and the second specified point is the intersection point Q0 of the diagonal lines of the second quadrangle, the parallelogram (including the rectangle) to be restored is restored. Since the intersection point is also the middle point, the area of the triangle formed by the top two vertices and the intersection of the parallelogram is equal to the area of the triangle formed by the bottom two vertices and the intersection, and the triangle formed by the left two vertices and the intersection Since the area of the triangle formed by the two vertices of the right side and the intersection is equal, the expression is simplified by canceling out the terms of the numerator and denominator in the area cross ratio.

  Further, when the conversion map is Equation (27), it is not necessary to find the equation of the projection matrix component, and a simple conversion map is obtained.

  In the case where the parallelogram is a rectangle, for example, at any point on any horizontal or vertical line in the rectangle (a line parallel to each side of the parallelogram) Since the area of the triangle formed from the same is the same as long as the base and the height do not change, the area on the line does not change. In other words, the necessary area calculation can be performed only by calculating the area of the triangle formed by the relationship between the four vertices and the intersection.

[4. Modified example]
Next, a case where the subject is a non-parallelogram will be described with reference to FIGS. 17 and 18.

  First, as shown in FIG. 17B, four points (P1, P2, P3, P4) that are features in the front restoration image are considered.

  The diagonal lines P1-P3, P2-P4, and their intersection point P0 are set for the target quadrilateral created by the four feature points.

  As shown in FIG. 17B, the distance P2P0 and P′4P0 to the intersection point P0 are equal by extending one side of the subject diagonal line from the intersection point P0 (P4 → P′4, P3 → P′3). Like that. Similarly, the distances P1P0 and P'3P0 are set equal. The quadrilaterals P1P2P'3P'4 thus produced are parallelograms.

  Then, as shown in FIG. 17A, the diagonal lines Q1-Q3 and Q2-Q4 and their intersections Q0 are set for the quadrangle (Q1Q2Q3Q4) which is taken and distorted from the corresponding oblique direction.

  On the other hand, for the extension of the diagonal lines Q1Q3 and Q2Q4 on the original image side, the principle of distance cross ratio is used. That is, since four points (P1, P0, P4, P′4) on the front restoration image side and three points (Q2, Q0, Q4) on the original image side are determined (Formula (5)), the extension point The coordinates of Q′4 are obtained. Similarly, the coordinates of the extension point Q′3 are obtained.

  Then, by providing the scan line Y in parallel to the side P1P′4 of the parallelogram P1P2P′3P′4 (an example of the first direction), By making it parallel to P1P2 (an example of the second direction), the same method as that of the parallelogram (rectangle) can be used for the triangle P5P′4P1 so that the base × height does not change.

  Next, as shown in FIGS. 18A and 18B, even when the intersection point P0 and the intersection point Q0 do not exist inside the quadrangle, similarly, the expansion points P′3 and P′4 correspond to these. Expansion points Q′3 and Q′4 and parallelograms P1P2P′3P′4 are obtained.

  In this way, when at least four restored image feature points are extracted from the front restored image, and the first quadrilateral that is a parallelogram is generated from the extracted restored image feature points, the target is also applied to a non-parallelogram. it can.

  In addition to the license plate, it can also be applied to machine recognition of contents such as business cards, signs, menus, signboards, licenses, etc. whose aspect ratio is known, and correction of keystone distortion of the projector.

  Furthermore, the present invention is not limited to the above embodiments. Each of the embodiments described above is an exemplification, and any configuration that has substantially the same configuration as the technical idea described in the claims of the present invention and has the same operational effects can be used. It is included in the technical scope of the present invention.

1: Image processing system 5: Camera 10: Image processing device 20: Original image T: Object P0: First point, first specified point P5: First point, drawing point Q0: Second point, second specified point Q5: Second point, drawing correspondence point

Claims (10)

Original image acquisition means for acquiring an original image obtained by photographing an object in a plane portion of a subject from an oblique direction;
Front restoration image information acquisition means for acquiring information of the target front restoration image;
Feature point extraction means for extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image;
A plurality of first triangles formed from the vertices of the first quadrangle and a first point on a plane including the first quadrangle; a second quadrangle having the extracted feature points and the feature points as vertices; A conversion map for converting the original image into the front-reconstructed image is generated from the area ratio of the plurality of second triangles formed from the second points corresponding to the first points. Conversion map generation means;
Pixel information calculation means for calculating pixel information of the original image corresponding to each pixel of the front restoration image based on the converted map;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The transformation map includes a plurality of the first triangles formed from the sides of the first quadrangle and the first point, and a plurality of the first triangles formed from the side of the second quadrangle and the second point. An image processing apparatus comprising two triangles, which is a mapping of a result of a predetermined combination of each of the formed first triangles and each of the second triangles. The image processing apparatus according to claim 2,
In the predetermined combination, the first quadrangle is a parallelogram, the first triangle is a triangle having two parallel sides of the parallelogram as first bases, and the second triangle is the second An image processing apparatus, wherein the image processing apparatus is a triangle having two opposite sides of a quadrangle and each second base corresponding to the first base. The image processing apparatus according to any one of claims 1 to 3,
The first point includes a drawing point for obtaining pixel information for drawing in the front restoration image, and a first prescribed point defined from each vertex of the first quadrangle,
The second point includes a drawing correspondence point corresponding to the drawing point in the original image and a second prescribed point corresponding to the first prescribed point and defined from each vertex of the second rectangle. A featured image processing apparatus. The image processing apparatus according to claim 4.
The first prescribed point is an intersection of diagonal lines of the first quadrangle;
The image processing apparatus, wherein the second specified point is an intersection of diagonal lines of the second quadrangle. The image processing apparatus according to claim 5.
The coordinates of the drawing corresponding point are expressed as coordinates (u ₅ , v ₅ ),
Coordinates (u ₁ , v ₁ ) (u ₂ , v ₂ ) (u ₃ , v ₃ ), (u ₄ ), which are the numbers of the vertices of the second square numbered in a predetermined circumferential direction in the second square. , V ₄ )
The first triangle formed by the vertices of the first rectangle corresponding to the coordinates (u ₁ , v ₁ ) (u ₂ , v ₂ ) (u ₃ , v ₃ ), (u ₄ , v ₄ ) Is expressed as [ijk] (i = 1, 2, 3, j = 1, 2, 3, k = 1, 2, 3),
The transformation map is

An image processing apparatus characterized by the above. The image processing apparatus according to any one of claims 1 to 6,
Restored image feature point extracting means for extracting at least four restored image feature points from the front restored image;
Parallelogram generating means for generating the first quadrangle that is a parallelogram from the restored image feature points;
An image processing apparatus further comprising: An image processing method in which an image processing apparatus processes an image,
An original image acquisition step of acquiring an original image obtained by photographing an object in a plane portion of a subject from an oblique direction;
A front restoration image information acquisition step of acquiring information of the target front restoration image;
A feature point extracting step of extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image;
A plurality of first triangles formed from the vertices of the first quadrangle and a first point on a plane including the first quadrangle; a second quadrangle having the extracted feature points and the feature points as vertices; A conversion map for converting the original image into the front-reconstructed image is generated from the area ratio of the plurality of second triangles formed from the second points corresponding to the first points. A conversion map generation step;
A pixel information calculation step for calculating pixel information of the original image corresponding to each pixel of the front restoration image based on the conversion map;
An image processing method comprising: Computer
Original image acquisition means for acquiring an original image obtained by photographing an object in a plane portion of a subject from an oblique direction;
Front restoration image information acquisition means for acquiring information of the target front restoration image;
Feature point extracting means for extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image;
A plurality of first triangles formed from the vertices of the first quadrangle and a first point on a plane including the first quadrangle; a second quadrangle having the extracted feature points and the feature points as vertices; A conversion map for converting the original image into the front-reconstructed image is generated from the area ratio of the plurality of second triangles formed from the second points corresponding to the first points. Conversion map generation means, and
A program for an image processing apparatus that functions as pixel information calculation means for calculating pixel information of the original image corresponding to each pixel of the front restoration image based on the conversion map. Computer
Original image acquisition means for acquiring an original image obtained by photographing an object in a plane portion of a subject from an oblique direction;
Front restoration image information acquisition means for acquiring information of the target front restoration image;
Feature point extracting means for extracting each feature point corresponding to each vertex forming the first square in the front restoration image from the original image;
A plurality of first triangles formed from the vertices of the first quadrangle and a first point on a plane including the first quadrangle; a second quadrangle having the extracted feature points and the feature points as vertices; A conversion map for converting the original image into the front-reconstructed image is generated from the area ratio of the plurality of second triangles formed from the second points corresponding to the first points. Conversion map generation means, and
A recording medium on which is recorded a computer-readable program for an image processing device that functions as pixel information calculation means for calculating pixel information of the original image corresponding to each pixel of the front restoration image based on the converted map.

Images