JPH06129833A - Three dimensional restoration method - Google Patents

Abstract

PURPOSE:To facilitate solving of an equation by finding out parameters expressing positional relations at first, and finding out the movement parameters having complicated constraint relations later, when the three dimensional structure of a body is restored from plural picture images input from one camera. CONSTITUTION:In a three dimensional restoration method for restoring a three dimensional structure from picture images, a device therefor is provided with a feature point detecting portion 1 of a first time picture image the feature point detecting portion 2 of a second time picture image and a feature point detecting portion 3 of a third time picture image. The device is further provided with a positional relations computing means 4 and a motion parameter computing means 5. At first parameters on a plane are computed by the positional relations computing means 4, and next the movement parameters are computed by the movement parameter computing means 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional reconstruction system for reconstructing a three-dimensional structure from two-dimensional image information, and particularly for reconstructing the three-dimensional structure from a plurality of images input from one camera. Is.

[0002]

2. Description of the Related Art A method of restoring a three-dimensional structure of the outside world by image processing is an essential technique for realizing a vision system for robots and unmanned vehicles. Conventionally, as a method for restoring a three-dimensional structure from two-dimensional information, a method based on binocular stereoscopic vision using the principle of triangulation has been proposed. However, in this case, it is necessary to precisely set the positions and directions of the two cameras, which is difficult to use.

As a method of using only one camera, the movement of the rigid object and the four characteristics are determined by the correspondence of the four feature points on the rigid body included in the images input by Ullman at three different times. We proposed a method to obtain the three-dimensional positional relationship of points. However, this method requires solving a non-linear equation in order to obtain a solution, and is not necessarily guaranteed to obtain a correct solution.

To solve this problem, Huang et al.
We proposed a linear solution under the same conditions as in the case of an. As shown in FIG. 6 (B), Huang et al.'S method uses Q as a two-dimensional image when a three-dimensional object P (a four-sided pyramid in the example of FIG. 6B) is captured by a TV camera. T ₁ , T
_As shown in FIG. 6A, the feature point detecting unit 61 for the image at the first time and the feature point detecting unit 62 for the image at the second time are obtained from these three types of two-dimensional images at three different times T ₂ and T ₃ . , The motion parameter and 4 from the (X, Y) information of each of four points at each time obtained from the feature point detection unit 63 of the image at the third time.
After deleting the unknown number representing the positional relationship by the motion parameter calculation unit 64 from the two unknowns of the positional relationship between the points,
The equation of only the motion parameters was derived, and the positions of the four points were calculated by the position relation calculation unit 65.

[0005]

However, FIG. 6 (A)
In the method shown in (1), the motion parameter calculation unit 64 first derives an equation of only the motion parameter and solves it. However, the motion parameter is complicated because many unknowns are bound to each other and the solution is fast. The calculation was not easy.
Therefore, it is an object of the present invention to provide a three-dimensional reconstruction method in which motion parameters are deleted and then parameters representing a positional relationship are first calculated.

[0006]

In order to achieve the above object, according to the present invention, as shown in FIG. 1, a feature point detecting section 1 for an image at a first time, a feature point detecting section 2 for an image at a second time, From the two-dimensional information of each time obtained from the feature point detection unit 3 of the image at the third time, first, in the positional relationship calculation unit 4, the motion parameter having a complicated hugging relationship is first erased. After that, an equation for a parameter that represents the positional relationship is derived and obtained.

[0007]

As a result, the nature of the equation is simplified, that is, the equation is easy to solve, and the amount of calculation can be reduced. Note that the parameters representing the positional relationship can also be simplified by solving the equations by replacing them with appropriate variables as will be described later, and can be solved easily.

[0008]

EXAMPLES Prior to describing the present invention based on an example
Next, the principle of the present invention will be described. In FIG. 6 (B), P is
It is a three-dimensional object and the three-dimensional position P of four feature points₀,
P ₁, P₂, P₃, But, for example, shooting with a YV camera
, The point Q moved to the two-dimensional image Q on the XY plane
₀, Q₁, Q₂, Q₃Represents the relationship. FIG. 6 (B)
, The XY plane is the image plane, and the point in the three-dimensional space
Was assumed to be imaged in parallel projection. Another object
It is assumed that P is a rigid body. At this time, the movement of the object
Can be completely described by the translation amount and the rotation amount.

Since the image Q is assumed to be a parallel projection, the object P
6 does not change no matter how much is moved in the direction perpendicular to the image Q (Z-axis direction in FIG. 6B). Therefore, the position of the object P in the vertical direction cannot be detected from the image Q in principle. Moreover, the vertical movement of the object cannot be detected from the change of the image.

On the other hand, the amount of movement in the direction parallel to the image plane (horizontal direction) can be considered as the amount of movement of the object as it is, as is easily understood from FIG. 6B. That is, the movement amount in the horizontal direction can be easily obtained if the correspondence between the feature points is known on the image plane.

Therefore, the problem is to find the amount of rotation of the object and the relative positional relationship between the four feature points. With respect to the positions of the four feature points, the component parallel to the image plane is equal to the relationship of the coordinate values on the image, so that the relative Z
If the coordinates are obtained, the positional relationship of the feature points in the three-dimensional space can be obtained.

In order to obtain the relative positional relationship between the rotation amount of the object and the Z coordinate, it can be considered that one of the four feature points of the object is at the origin. That is, as shown in FIG. 2, one feature point is fixed to the origin O, the rotation amount around the origin O and the remaining three feature points P ₁ , P
The Z coordinate of ₂ and P ₃ should be calculated. Note that P ₁ and P in FIG.
₂ and P ₃ do not correspond to P ₁ , P ₂ and P _{3 in} FIG. 6 (B).

Considering _two different times T ₁ and T ₂ ,
The rotation around the origin O is represented by a 3 × 3 matrix R. By this rotation R, at the first time T ₁ , (x, y, z)
The point at is moved to the point (x ', y', z ') determined by the following equation.

[0014]

[Equation 1]

However, Rij is a component of the rotation matrix R.
It Here, three features other than the origin at the first time
Point P₁, P₂, P₃Coordinates of (x₁, Y₁,
z₁), (X₂, Y₂, Z₂), (X₃, Y₃, Z ₃)
And The quantity to be calculated here is the rotation matrix R and three special characteristics.
Z-coordinate z of the point₁, Z₂, Z₃Is.

Now, the equation of the plane defined by the three feature points is set as z = ax + bx + c (2). The point (x, y, ax + bx + c) on the plane is rotated by R

[0017]

[Equation 2]

Move on to. Regarding the coordinates of points on the image plane,
Image Q _{1 at} the first time and Q ₂ at the second time

[0019]

[Equation 3]

The following relationship holds. By rearranging these expressions,

[0021]

[Equation 4]

Is obtained. The expression (3) indicates that the points on the image plane are transformed by the affine transformation. On the other hand, three feature points P ₁ 'at the second time,
The coordinates of P ₂ 'and P ₃ ' are (x ₁ ', y ₁ ', z
_{1 '), (x 2'} , y 2 ', z 2'), (x 3 ',
y ₃ ', z ₃ '). At this time, three points (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ) on the image plane
Are (x ₁ ', y ₁ '), (x ₂ ', y ₂ '),
Move to (x ₃ ', y ₃ '). The affine transformation that connects the image at the first time and the image at the second time is completely determined by the correspondence of these three points.

In fact, the affine transformation to be defined is

[0024]

[Equation 5]

Then,

[0026]

[Equation 6]

## EQU3 ## These six equations are represented by A,
Solving for B, C, D, α, and β, the coefficients of the affine transformation of the above equation (4) are obtained, and by comparing the above equations (3) and (4), the rotation R and the plane parameter to be obtained The following relational expression between a, b, c and the coefficient of the affine transformation is obtained.

[0028]

[Equation 7]

Using the last two equations of the above equation (5), R ₁₃ =
When these equations (5) are transformed by substituting α / c and R ₂₃ = β / c into the first four equations,

[0030]

[Equation 8]

Is obtained. Where R is the rotation matrix

[0032]

[Equation 9]

Is satisfied. By substituting the equation (6) into the equation (7), the following equation having only plane parameters a, b, and c as unknowns is obtained.

[0034]

[Equation 10]

The following equation (8-
Expressions 1) to (8-3) are obtained.

[0036]

[Equation 11]

From (8-1) × β- (8-3) × α, the following equation (9) is obtained.

[0038]

[Equation 12]

From (8-2) × α- (8-3) × β, the following equation (10) is obtained.

[0040]

[Equation 13]

When the above equations (9) and (10) are arranged,
The following equations (11) and (12) are obtained.

[0042]

[Equation 14]

Here, it is considered that the above equations (11) and (12) should be solved for two unknowns a / c and b / c. However, in practice, the left sides of both equations are proportional to each other. The two equations are not independent, and a solution cannot be obtained only from these two equations.

Since the left side of (11) × α- (12) × β is always zero, the following expression (A) must be used.

[0045]

[Equation 15]

When this relationship does not hold between the coefficients of the affine transformation, it can be determined that the correspondence between the two times is incorrect. Since a solution cannot be obtained only from the equations (11) and (12), the expansion after the equation (3) is repeated for the images at the first time and the third time. However, the'symbol is added to each of the affine transformation coefficient and the rotation matrix. At this time, the following expressions (13) and (14) are established corresponding to the expressions (11) and (12).

[0047]

[Equation 16]

In equations (11) to (14), (1
Since equations (1) and (12) are generally independent, a / c and b
It can be solved as a simultaneous linear equation for / c. When a / c and b / c are obtained, (8-
Using the equation 3), a, b, and c can be calculated. However, in the formulas (8-1) to (8-3), when the signs of a, b, and c are simultaneously inverted, the original formulas are restored, and therefore -a, -b,
-C is also a solution. That is, there are two solutions. here,
a, b, and c are parameters indicating the plane on which the bottom surface of the three-dimensional object is located. The intersection of this plane and the Z axis is c, and a and b indicate the inclination of the plane.

Further, by substituting these values into the equation (6), R ₁₁ , R ₁₂ , R ₁₃ , R ₂₁ , R ₂₂ ,
R ₂₃ is required. Since the rotation matrix has the property of the following equation, the remaining components R ₃₁ , R ₃₂ , and R ₃₃ are also determined by the following equation.

[0050]

[Equation 17]

The other rotation R'is similarly obtained.
By the above method, the plane parameter and the motion parameter (rotation matrix) can be calculated. First of the present invention
An embodiment will be described with reference to FIG. 3, the same symbols as in FIG. 1 indicate the same parts. In FIG. 3, reference numerals 11 and 12 are correspondence sections between feature points, 13 and 14 are affine transformation coefficient calculation sections, 15 is a plane parameter calculation section, and 16 and 17 are rotation parameter calculation sections.

First, the feature point detection unit 1 of the image at the first time is
Therefore, for example, the image Q at the first time in FIG.₁Features of
(X₁, Y₁), (X₂, Y₂), (X₃, Y₃)
Of the image Q by the feature point detecting unit 2 of the image at the second time.₂
Feature points (x₁’、 Y₁’, (X₂’、 Y₂’), (X
₃’、 Y₃′) Is calculated, and the feature points of the image at the third time are detected.
Image Q by Part 3₃Feature points (x₁", Y₁)), (X
₂", Y₂)), (X ₃", Y₃)) Is calculated.

The associating unit 11 between feature points determines the correspondence relationship between the feature points of the image Q ₁ at the first time and the feature points of the image Q ₂ at the second time, that is, the correspondence relationship of four sets of feature points. Ask.
Actually, as shown in FIG. 2, one of the feature points is fixed at the origin O, and therefore the correspondence relationship between the three sets of feature points is obtained.

Similarly, the associating unit 12 between the feature points and the feature point of the image Q ₂ at the second time and the image Q at the third time.
The correspondence relationship with the _three feature points, that is, the correspondence relationship between the four feature points is obtained. In reality, three sets of correspondences are also obtained.

Then, the affine transformation coefficient calculation unit 13
It is output from the associating unit 11 between the feature points (x ₁ ,
y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₁ ',
_{y 1 ', (x 2'} , y 2 ')' based on the _{(4), (x 3 '} , y 3)' Solving equation. Similarly, the affine transformation coefficient calculation unit 14 outputs (x ₁ ′, y ₁ ′), (x ₂ ′, y ₂ ′) from the feature point association unit 12 described above,
_{(X 3 ', y 3'} ), (x 1 ", y 1"), (x 2 ",
Based on y ₂ ″) and (x ₃ ″, y ₃ ″), the equation (4) ′ is solved. By these, the affine transformation coefficients A, B, C, D, α and β by the images Q ₁ and Q ₂ are obtained. The affine transformation coefficients A ′, B ′, obtained by the images Q ₂ and Q ₃ ,
C ′, D ′, α ′ and β ′ are obtained.

The plane parameter calculation unit 15 calculates the value of 4 at the first time based on these two sets of affine transformation coefficients.
Of the two feature points, the equation of the plane defined by the three-dimensional positions of the remaining three feature points not located at the origin z = ax
In + bx + c, a / c and b / c are obtained by solving the equations (11) to (13), and then a, b, and c are calculated using the equation (8-3).

The rotation parameter calculation unit 16 has a plane parameter a obtained from the plane parameter calculation unit 15,
b, c and the affine transformation coefficients A, B, C, D, α,
Based on β, the motion parameters R ₁₁ and R are calculated from the equation (6).
₁₂ , R ₁₃ , R ₂₁ , R ₂₂ , and R ₂₃ are calculated.

Similarly, the rotation parameter calculator 17 has the plane parameters a, b and c obtained from the plane parameter calculator 15 and the affine transformation coefficients A ′ and B ′ obtained from the affine transformation coefficient calculator 14. , C ', D', α ',
β ′ is used to calculate the motion parameter R ₁₁ ′ from the equation (6),
_{R 12 ', R 13',} R 21 ', R 22', calculates the R ₂₃ '.

Next, a second embodiment of the present invention will be described with reference to FIG. In image processing, it is not always successful in associating feature points with each other, and in some cases, failure cannot be achieved. In FIG. 4, it is determined whether or not the feature points are associated with each other. In order to determine whether or not this association has been established, in FIG. 4, association correctness determination units 18 and 19 are provided.

In FIG. 4, as in the embodiment of FIG.
The image Q ₁ , Q ₂ , and Q ₂ shown in FIG. 2 are respectively provided by the feature point detection unit 1 of the image at the first time, the feature point detection unit 2 of the image at the second time, and the feature point detection unit 3 of the image at the third time. Each feature point of ₃ is calculated, and the associating unit 11 between feature points calculates the images Q ₁ , Q 2.
Correspondence section 1 between feature points is obtained by obtaining the correspondence relationship between the _two feature points.
In step 2, the correspondence between feature points of the images Q ₁ Q ₃ is obtained.

The affine transformation coefficient calculation unit 13 then associates the feature points obtained from the images Q ₁ and Q ₂ with each other.
The equation (4) ′ is solved based on the data output from 1 to obtain the affine transformation coefficients A, B, C, D, α and β.
Further, the affine transformation coefficient calculation unit 14 uses the images Q ₁ , Q 2.
Based on the data output from the associating unit 12 between the feature points obtained from _{3, the} equation (4) ′ is also solved,
Affine transformation coefficients A ', B', C ', D', α ', β'
To get

The association correctness determination unit 18 calculates the equation (A) using the affine transformation coefficients A, B, C, D, α and β. The association correctness determination unit 19 also performs the calculation of the formula (A) using the affine transformation coefficients A ′, B ′, C ′, D ′, α ′ and β ′.

If the calculation of the equation (A) is established by these, the plane parameter calculation unit 15 calculates the plane parameters a, b, c, and the rotation parameter calculation unit, as in the embodiment of FIG. 16 and 17, the motion parameters R ₁₁ , R ₁₂ , R ₁₃ , R ₂₁ , R ₂₂ , R ₂₃ and R ₁₁ ′, R ₁₂ ′, R ₁₃ ′, R ₂₁ ′, R ₂₂ ′, R ₂₃ ′ are respectively calculated. calculate.

If the operation of the expression (A) does not hold, N
Based on the respective images obtained by the feature point detection unit 1 of the image at the first time, the feature point detection unit 2 of the image at the second time, and the feature point detection unit 3 of the image at the third time by the route of O, The same association process will be redone.

Further, a third embodiment of the present invention will be described with reference to FIG. In the present invention, it cannot be solved when the three-dimensional object is rotating around the Z axis. However, I can see how much it is rotating, so I can output it.

In order to determine whether or not the three-dimensional object is rotating around the Z axis, Z axis rotation determining sections 20 and 21 are provided as shown in FIG. Then, it is determined whether or not both α and β are zero among the affine transformation coefficients obtained by the equation (4) ′. If both are zero, it is determined that the rotation is about the Z axis. In this case, the rotation parameter calculation units 22 and 23 calculate the equation (6) to calculate only the rotation parameter.

[0067]

According to the present invention, a complicated motion parameter such as the above equation (7) is first erased, and an equation for a parameter representing a positional relationship, that is, a plane parameter is derived and calculated. Since the power equation is made easy to solve, the amount of calculation can be reduced.

Moreover, according to the second embodiment of the present invention, it is possible to determine whether or not the feature points have been associated with each other, so that an accurate determination can be made quickly. Furthermore, according to the third embodiment of the present invention, it is possible to determine whether or not the object is rotating in the Z-axis on the way, so that an accurate determination can be made quickly.

[Brief description of drawings]

FIG. 1 is a principle diagram of the present invention.

FIG. 2 is an explanatory diagram of a moving state of an object.

FIG. 3 is a first embodiment of the present invention.

FIG. 4 is a second embodiment of the present invention.

FIG. 5 is a third embodiment of the present invention.

FIG. 6 is an explanatory diagram of a conventional example, feature points, and an image.

[Explanation of symbols]

 1 1st time image feature point detection unit 2 2nd time image feature point detection unit 3 3rd time image feature point detection unit 4 Positional relationship calculation unit 5 Operating parameter calculation unit 11, 12 Between feature points Associating unit 13 and 14 Affine transformation coefficient calculator 15 Planar parameter calculator 16 and 17 Rotation parameter calculator

Claims (5)

[Claims] 1. A feature point detection unit (1) for an image at a first time
And a feature point detection unit (2) for the image at the second time and a feature point detection unit (3) for the image at the third time, and in the three-dimensional restoration method for restoring the three-dimensional structure from the image, The calculation means (4) and the motion parameter calculation means (5) are provided. First, the positional relationship calculation means (4) calculates planar parameters, and then the motion parameter calculation means (5) calculates motion parameters. A three-dimensional reconstruction method characterized by being configured as follows. 2. A feature point detection unit (1) for an image at a first time.
And a feature point detection unit (2) for the image at the second time and a feature point detection unit (3) for the image at the third time, and in the three-dimensional restoration method for restoring the three-dimensional structure from the images, Correspondence detection means (1
1) and (12), affine transformation coefficient calculation means (13) and (14) for calculating affine transformation coefficients from the correspondence of feature points between images, and affine transformation coefficients between the first time and the second time. , A parameter of a plane for calculating from the affine transformation coefficient between the first time and the third time a parameter of the plane determined by the remaining three points when one of the four feature points at the first time is regarded as the origin A three-dimensional reconstruction method comprising a calculating means (15). 3. The matching correctness according to claim 2, wherein it is determined whether or not a predetermined relationship is established between the affine transformation coefficients, and if not, it is determined that the correspondence of the feature points is incorrect. Determination means (18),
A three-dimensional reconstruction method characterized in that (19) is provided. 4. The plane parameter calculated by the plane parameter calculating section (15) according to claim 2, and the affine transformation coefficient calculating means (1).
3), from the affine transformation coefficient calculated from (14),
A three-dimensional reconstruction method comprising rotation parameter calculation means (16), (17) for calculating a rotation parameter between the first time and the second time and a rotation parameter between the first time and the third time. . 5. The method according to claim 3, wherein both two specific affine transformation coefficients between the first time and the second time or between the first time and the third time are zero. , Z-axis rotation determination means (20) for determining that the object is rotating around the Z-axis, (2
A three-dimensional reconstruction method characterized by comprising 1).