JPH11120361A - Three-dimensional shape restoring device and restoring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To restore a three-dimensional shape highly precisely at low calculation cost under an optional pickup condition. SOLUTION: The feature point high in the possibility of being eliminated by an image of a second viewpoint among the feature points of the image detected at a first viewpoint is calculated from posture information of an image input means 2 calculated by a posture detection means 3 and a parallel motion component of the image input means 2 calculated by a parallel component detection means 4 by a feature point loss detection means 5. The plural feature points of the images measured at the first viewpoint is extracted and the feature points between the images at the first and the second viewpoints are made to correspond except the feature point specified by the feature point loss detection means 5 by a correspondence detection means 6. The three-dimensional shape of a subject is calculated and restored by the posture information, the parallel motion component of the image input means 2 and correspondence between the feature point executed by the correspondence detection means 6 by a three-dimensional calculation means 7.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image measuring device, a digital camera, a video camera, a portable information device,
A three-dimensional shape restoring device for restoring the position and orientation of the camera at the time of photographing and the three-dimensional shape of the photographed object from a plurality of continuous images by a visual device of a mobile robot, a self-sustained traveling vehicle, and other image equipment; It relates to a restoration method.

[0002]

2. Description of the Related Art Research for restoring the three-dimensional shape of an object is as follows.
It is being promoted in various fields including the vision of autonomous mobile robots. In particular, in recent years, computers and electronic devices have rapidly spread due to the dramatic progress in electronic technology, and three-dimensional display of three-dimensional information can be easily enjoyed. On the other hand, the development of three-dimensional information input technology for objects and scenes in the real world is expected.

Methods for measuring the distance and shape to an object include an active method of irradiating the object with light waves or ultrasonic waves and a passive method represented by a stereo image method. Active methods irradiate the object with light, radio waves, or sound waves.
A method to determine the distance to the target by measuring the propagation time of the reflected wave from the target, or a light source such as a slit light or spot light with a specific pattern from a light source with a known positional relationship with the camera , And observing the distortion to determine the shape of the object. The active method generally has a problem in that the distance can be measured with high speed and high accuracy, although there is a problem in miniaturization of the device. On the other hand, passive methods are roughly classified into multi-view stereoscopic vision and motion stereoscopic vision. In multi-view stereoscopic vision, feature points or regions between images are associated with each other from images obtained by photographing the object using a plurality of cameras whose positions and orientations are known. This is a method for calculating a dimensional shape. In this case, since the relative position and orientation information between the cameras is known, the epi-curve constraint (ep
A basic constraint condition called ipolar constraint can be used, and the amount of calculation required for association can be reduced. However, there is a problem that a large distance measurement error is likely to occur when there is an association error due to superimposed noise or the like or when parallax cannot be sufficiently obtained. Motion stereoscopic vision is performed by moving a single camera, photographing an object, associating continuous images, and calculating the position and orientation of the camera and the three-dimensional shape of the object. This method has the same problems as multi-view stereoscopic vision,
Unlike multi-view stereoscopic vision, the position and orientation information of the camera between images is unknown, so that epipolar constraint cannot be applied, and the amount of calculation required for association increases. Further, in order to determine the relative position and orientation of the camera, it is generally necessary to solve a complicated nonlinear equation by an iterative operation. As a result, the amount of calculation required for three-dimensional reconstruction is enormous, and the solution tends to be unstable.

In order to solve the problem of the passive method, an apparatus for restoring a three-dimensional shape with a small calculation cost by using a sensor other than an image such as a distance, an acceleration, an angular velocity, and magnetism in addition to an image is disclosed in, for example, Japanese Patent Application Laid-Open No. HEI 5-1993. JP-1996437 and JP-A-7-1810
No. 24, JP-A-9-81790.

In the method disclosed in Japanese Patent Application Laid-Open No. 5-196437, a measurement point on a subject is photographed by a camera by orthogonal projection, and the attitude of the camera at that time is obtained by a three-axis gyro fixed to the camera. To extract the three-dimensional information of the subject. In the method disclosed in Japanese Patent Application Laid-Open No. Hei 7-181024, a moving amount detecting unit such as an acceleration sensor or an angular velocity sensor is installed in an image input unit, and a moving amount of a camera obtained by the moving amount detecting unit is translated. The three-dimensional shape of the object is restored from the corresponding point search result. Further, the method disclosed in Japanese Patent Application Laid-Open No. 9-81790 detects the movement of a photographing device using an angular velocity sensor and an acceleration sensor, and corrects the optical axis direction so that optical axes from different viewpoints intersect at an arbitrary point. Thus, three-dimensional restoration is performed at high speed while preventing the disappearance of feature points.

[0006]

The method disclosed in Japanese Patent Application Laid-Open No. 5-196437 enables three-dimensional reconstruction of an object only by tracing a single feature point, but presupposes orthogonal projection. Therefore, accuracy is insufficient to extract three-dimensional information from an image projected by the camera of the central projection model. Further, the method disclosed in Japanese Patent Application Laid-Open No.
Determining the relative position and orientation of the camera between different viewpoints by installing the movement amount detection unit enables high speed and miniaturization of the device, but when calculating the movement amount, sensor signals such as mechanical angular velocity sensors are used. Need to be integrated, there is a problem that error components of the movement amount are accumulated over time.
Further, in the motion stereoscopic view, a point (hereinafter, referred to as a feature point) used for association in an image at a previous time according to a camera motion disappears in an image at the next time,
Although there is a great possibility that a mapping error will occur,
In one method, this is not considered at all.

On the other hand, the method disclosed in Japanese Patent Application Laid-Open No. Hei 9-81790 discloses a method in which an estimated value of a motion vector calculated based on sensor information and a distance between a set object and a camera, and a motion vector obtained by image processing. Although the object is detected by comparing with the above, since the distance between the object and the camera is set in advance, the three-dimensional shape can be restored only under specific photographing conditions. Further, since a driving mechanism for changing the direction of the optical axis is required, the structure of the device becomes complicated.

An object of the present invention is to solve such a problem and to provide a three-dimensional shape restoring apparatus and a restoring method capable of realizing a high-precision three-dimensional shape restoration under a small calculation cost under arbitrary photographing conditions. It is assumed that.

[0009]

A three-dimensional shape restoring apparatus according to the present invention inputs an image of a subject from a plurality of viewpoints by image input means, and calculates attitude information and translational motion components of the image input means at each viewpoint. Then, from the input image information, the calculated attitude information of the image input means and the translational motion component, the feature points of each image are associated, and the correspondence between the image information, the attitude information of the image input means, the translational motion component, and the feature point is obtained. In a three-dimensional shape restoring apparatus for calculating and restoring a three-dimensional shape of a subject from an attached result, a feature point which detects a feature point disappearing from an image plane taken from a different viewpoint is detected from posture information and a translational motion component of an image input unit. The image processing apparatus is characterized in that the image processing apparatus further includes an erasure detection unit, and removes the erasure feature points detected by the feature point erasure detection unit and associates the feature points of each image.

A second three-dimensional shape restoration apparatus according to the present invention inputs an image of a subject from a plurality of viewpoints by image input means, calculates posture information of the image input means at each viewpoint, and calculates an image plane of each viewpoint. Calculate the distance to a point in space corresponding to the point projected above, calculate the translational motion component of the image input means from the attitude information of the image input means and the distance information to the point in space, and input the image Based on the information and the calculated attitude information of the image input means and the translational motion component, the feature points of each image are associated with each other, and the image information and the attitude information of the image input means are translated from the translational motion component and the feature point. In a three-dimensional shape restoration device that calculates and restores a three-dimensional shape,
A feature point loss detection unit that detects a feature point that disappears from an image plane captured from a different viewpoint based on the attitude information of the image input unit and the distance information to a point in space; The feature points are removed from each other and the feature points of each image are associated with each other.

In the three-dimensional shape restoration method according to the present invention, an image of a subject is input from a plurality of viewpoints by image input means,
Calculate the posture information and the translational motion component of the image input means at each viewpoint, perform correspondence between the feature points of each image from the input image information and the calculated posture information and the translational motion component of the image input means, and In a three-dimensional shape restoring method for calculating and restoring a three-dimensional shape of a subject from the result of associating the attitude information of the image input means with the translational motion component and the feature point, a different viewpoint is obtained from the attitude information of the image input means and the translational motion component. The method is characterized in that disappearing feature points are detected from the image plane photographed in step (1), the disappearing feature points are removed, and the feature points of each image are associated with each other.

In a second three-dimensional shape restoring method according to the present invention, an image of a subject is inputted from a plurality of viewpoints by an image input unit, posture information of the image input unit at each viewpoint is calculated, and an image plane of each viewpoint is calculated. Calculate the distance to a point in space corresponding to the point projected above, calculate the translational motion component of the image input means from the attitude information of the image input means and the distance information to the point in space, and input the image Based on the information and the calculated attitude information of the image input means and the translational motion component, the feature points of each image are associated with each other, and the image information and the attitude information of the image input means are translated from the translational motion component and the feature point. In a three-dimensional shape restoration method for calculating and restoring a three-dimensional shape,
From the attitude information of the image input means and the distance information to points in space, feature points that disappear from the image plane taken from different viewpoints are detected, and the detected disappearing feature points are removed to correspond to the feature points of each image. It is characterized by attaching.

It is preferable that the existence range of the feature point in which the disappearance is detected in the space and its image feature amount be stored, and that the stored feature point is projected again on another screen.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A three-dimensional shape restoring apparatus according to the present invention has an image input means, a posture detecting means, a translation component detecting means, a characteristic point disappearance detecting means, a correspondence detecting means, and a three-dimensional calculating means. This three-dimensional shape restoration device can convert the same object to the first
When the three-dimensional shape is restored by photographing at the first and second viewpoints, the image of the object to be measured is photographed at the first viewpoint by the image input unit, and the posture of the image input unit is calculated by the posture detecting unit. To the feature point disappearance detecting means and the corresponding point detecting means. Next, the image input means is moved to the second viewpoint to capture an image of the measurement object, the attitude of the image input means is calculated by the attitude detection means, and the translational motion component of the image input means is calculated by the translation component detection means. Then, it is sent to the feature point disappearance detecting means and the correspondence detecting means. The feature point disappearance detecting means detects the first one of the feature points of the image detected from the first viewpoint from the attitude information of the image input means calculated by the attitude detecting means and the translational motion component of the image input means calculated by the translation component detecting means. A feature point that is likely to disappear in the image of the second viewpoint is obtained. The correspondence detection unit extracts a plurality of feature points of the image measured at the first viewpoint, and removes the feature points between the images at the first viewpoint and the second viewpoint except for the feature points designated by the feature point disappearance detection unit. Is associated. The three-dimensional calculation means calculates and restores the three-dimensional shape of the subject based on the correspondence between the attitude information of the image input means, the translational motion components, and the feature points performed by the correspondence detection means.

As described above, the disappearance of the feature point is detected from the attitude information of the image input means and the translational motion component. Therefore, the calculation required for the unnecessary association of the feature point and the erroneous correspondence can be prevented, and the low calculation can be performed. A highly accurate three-dimensional shape can be restored at a low cost.

[0016]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention. As shown in FIG.
Has image input means 2, attitude detecting means 3, translation component detecting means 4, feature point disappearance detecting means 5, correspondence detecting means 6, and three-dimensional calculating means 7. For example, as shown in FIG. 2, the image input unit 2 captures an image of the measurement target M at a first viewpoint A1 and a second viewpoint A2, and an image plane I1 of the first viewpoint A1 and a second viewpoint A2.
The image formed on the image plane I2 is input. The attitude detecting means 3 detects a magnetic field or a gravitational field acting on the image input means 2 or detects an angular velocity accompanying the movement of the image input means 2 to perform the first
The posture information (rotation component) R of the image input means 2 at the viewpoint A1 and the second viewpoint A2 is calculated. The translation component detection unit 4 detects an inertial force generated by the movement of the image input unit 2 from the first viewpoint A1 to the second viewpoint A2, and detects, for example, a GPS (Global
A translational motion component D of the image input means 2 is calculated using a navigation system such as a positioning system. The feature point disappearance detecting means 5 detects the image detected at the first viewpoint A1 from the attitude information R of the image input means 2 detected by the attitude detecting means 3 and the translational motion component D of the image input means 2 detected by the translational component detecting means 3. Among the feature points, the feature point having a high possibility of disappearing in the image of the second viewpoint A2 is obtained. Correspondence detecting means 6 is the first viewpoint A
A plurality of feature points of the image measured in step 1 are extracted, and the feature points between the images at the first viewpoint A1 and the second viewpoint A2 are associated except for the feature points designated by the feature point disappearance detecting means.
The three-dimensional calculation means 7 calculates the three-dimensional shape of the subject by associating the attitude information R of the image input means 2 with the translational motion component D and the feature points performed by the corresponding point detection means 6.

In the three-dimensional shape restoring apparatus 1 configured as described above, as shown in FIG.
The operation of restoring the three-dimensional shape by photographing at the first and second viewpoints A2 will be described with reference to the flowchart of FIG.

At the same time as capturing an image of the measuring object M from the first viewpoint A 1 by the image input means 2, the attitude of the image input means 2 is calculated by the attitude detection means 3 to detect the feature point disappearance detection means 5.
Is sent to the correspondence detecting means 6 (step S1). Next, the image input means 2 is moved to the second viewpoint A2 to photograph an image of the measuring object M, the attitude of the image input means 2 is calculated by the attitude detection means 3, and the image input means 2 is calculated by the translation component detection means 4. And the correspondence detection means 6
(Step S2). By photographing the measurement target M at the first viewpoint A1 and the second viewpoint A2, as shown in FIG. 4, a plurality of predetermined target points Pn (n = 1 to n) of the measurement target M are set to the first positions. The feature point P1n is detected on the image plane I1 of the viewpoint A1, and the image plane I of the second viewpoint A2 is detected.
2 is also detected as a feature point P2n. The feature point disappearance detecting means 5 disappears at the second viewpoint A2 among the feature points P1n detected at the first viewpoint A1 based on the relative posture information R and the translational motion component D between the first viewpoint A1 and the second viewpoint A2. A feature point that is highly likely to be obtained is obtained (step S3).

The feature point disappearance detecting means 5 uses the first viewpoint A1
A process for obtaining a feature point having a high possibility of disappearing at the second viewpoint A2 from among the feature points P1n detected in step (1) will be described.
As shown in FIG. 4, the image input means 2 takes the optical center as the origin and places x on the image planes I1 and I2 in a direction orthogonal to each other.
It is assumed that this is a center projection model with a known focal length f using an xyz coordinate system having an axis and a y-axis and the z-axis as an optical axis direction and a camera coordinate system. The posture information between the first viewpoint A1 and the second viewpoint A2 is a rotation matrix of the image input means 2 between the first viewpoint A1 and the second viewpoint A2 based on the first viewpoint A1 expressed by the following equation (1). It is represented by R.

[0020]

(Equation 1)

The translational motion component D can be represented by a column vector D expressed by the following equation (2) with reference to the camera coordinate system at the first viewpoint A1.

[0022]

(Equation 2)

Here, the position of the point P1n (x, y, f) on the image plane I1 of the first viewpoint A1 projected on the image plane I2 of the second viewpoint A2 will be described with reference to FIG. A target point Pn in the space indicated by a point P1n (x, y, f) on the image plane I1 of the first viewpoint A1 is a vector P1 expressed by the following equation (3) with reference to the camera coordinate system at the first viewpoint A1. Can be represented by In equation (3), Ln is the distance between the optical center of the first viewpoint A1 and the target point Pn.

[0024]

(Equation 3)

On the other hand, in the camera coordinate system at the second viewpoint A2, the target point Pn is represented by a vector P2 expressed by the following equation (4).

[0026]

(Equation 4)

Therefore, a point P1n on the image plane I1 of the first viewpoint A1 is projected on a point P2n indicated by a vector P2 of the following equation (5) on the image plane I2 photographed at the second viewpoint A2.

[0028]

(Equation 5)

The point P1 on the image plane I1 of the first viewpoint A1
n and the point P2n of the image plane I2 of the second viewpoint A2
Attitude information R, translational motion component D, focal length f, and distance L ₁
Can be expressed by the following equation (6) as a function using

[0030]

(Equation 6)

Here, the posture information R and the translational motion component D are detected, and the focal length f is known. Therefore, the function F in the equation (6) depends only on the distance Ln. Therefore, a certain feature point P1
In order to know the disappearance of n, the distance Ln needs to be known.
If it is set to restore the shape of the object existing between Lmax, the range of the line passing through the target point Pn and the optical center of the second viewpoint A2 is limited to the range of the straight line PminPmax in FIG. As a result, the range in which the feature point P1n on the image plane I1 of the first viewpoint A1 is projected on the image plane I2 of the second viewpoint A2 is limited. Therefore, the feature point disappearance detecting means 5 sets the image plane I
The feature points that are likely to disappear from above are output to the correspondence detection means 6 so as not to be used for correspondence. For example, in FIG. 5, a straight line PminPmax at the second viewpoint A2
Is the length on the screen of the part projected on the image plane I2 is Li.
n, assuming that the length of the unprojected portion on the screen is Lout, it is determined that the feature points satisfying [Lout / (Lin + Lout)] ≧ 0.8 disappear.

The correspondence detection means 6 removes the feature points between the images of the first viewpoint A1 and the second viewpoint A2 except for the feature points designated by the feature point disappearance detection means 5 in the group of feature points P1n which are scheduled in advance. (Step S4). Since the attitude information R and the translational motion component D are measured when the feature points are associated, epipolar constraint can be used. Then, the association is performed by a general method such as a method using local image features such as a correlation method, a feature matching method, and a sparse / dense method, and a method of calculating a moving area using a spatiotemporal differentiation method.
For example, when the correlation method is used, the image plane I1 of the first viewpoint A1
Point i (x _i0 , y _i0 ) and the point (x _i0 + dx, y _i0 ) of the second viewpoint A2 on the image plane I2
+ Dy) is associated with (2N + 1) as shown in FIG.
When performing block matching using the correlation window 61 of (2P + 1) and (2P + 1), the cross-correlation value S calculated by the following equation (7)
The point that maximizes i is selected as the corresponding point.

[0033]

(Equation 7)

In the above equation (7), I ₁ (x, y) is the density at the point (x, y) on the image plane I1 of the first viewpoint A1.
I ₂ (x, y) is a point (x, y) on the image plane I2 of the second viewpoint A2.
The density, MI ₁ (x, y) in y) is the average density in the (2N + 1), (2P + 1) correlation window 61 centered on the point (x, y) on the image plane I1, and MI ₂ (x, y) is (2N + 1), (2P +) centered on the point (x, y) on the image plane I2
The average density in the correlation window 61 of 1) is shown, and K is a constant.

From the posture information R obtained as described above, the translational motion component D, and the association result, the three-dimensional calculation means 7 calculates the three-dimensional structure of the subject according to the principle of triangulation (step S5). The position and orientation information and 3
The dimension information and each image are stored in a storage unit (not shown) as necessary (steps S6 and S7).

In this manner, the disappearance of a feature point is detected from the attitude information R of the image input means 2 and the translational motion component D. Therefore, it is possible to prevent the calculation required for useless association of feature points and erroneous correspondence. Thus, a highly accurate three-dimensional shape can be restored at a low calculation cost.

In the above embodiment, the translation component detecting means 4 uses the first
The case where the inertial force generated by the movement of the image input unit 2 from the viewpoint A1 to the second viewpoint A2 is detected and the translational motion component of the image input unit 2 is calculated using a navigation system such as GPS has been described. Information to a certain point in the space and the attitude information R of the image input means 2
May be used to calculate the translational motion component of the image input means 2.

The three-dimensional shape restoration apparatus 1a according to the second embodiment for calculating the translational motion component of the image input means 2 from the distance information to a certain point in the space and the attitude information R of the image input means 2
As shown in the block diagram of FIG. 7, a distance detecting means 8 for detecting a distance to a point in space, the distance information detected by the distance detecting means 8 and an image input means 2 calculated by the attitude detecting means 3 Of the image input means 2 from the posture information R of the image input means 2.

With this three-dimensional shape restoring apparatus 1a, as shown in FIG. 2, the same object M is moved from a first viewpoint A1 to a second viewpoint A2.
When restoring the three-dimensional shape by photographing in step S11, as shown in the flowchart of FIG. 8, the user first determines a gazing point for measuring the distance of the measuring object M before photographing the image (step S11). Various methods can be used to determine the gazing point, such as automatically selecting a small region having a characteristic density distribution. Then, the first viewpoint A1 is input by the image input unit 2.
, An image of the measurement object M is photographed, the posture detection means 3 calculates the posture of the image input means 2, the distance detection means 8 measures the distance to the point of gaze of the measurement object M, and obtains image information and posture information. And the measured distance information is used as a translation component detecting means 4a.
And sends the image information and the posture information to the feature point disappearance detecting means 5 (step S12). When the distance to the fixation point of the measuring object M is measured by the distance detecting means 8, the infrared stereo method using the principle of triangulation, the projection of waves such as ultrasonic waves, and the transmission time of reflection from the object are used. The distance to the gazing point of the measurement target M is measured by using a method of measuring a distance, a method of obtaining distance information at the time of focusing from an encoder installed in an optical system, or the like. Next, the image input means 2 is moved to the second viewpoint A2 to capture an image of the measurement target M, the attitude of the image input means 2 is calculated by the attitude detection means 3, and the distance detection The distance to the gazing point is measured, the image information, the posture information, and the measured distance information are sent to the translation component detection means 4a, and the image information and the posture information are sent to the feature point disappearance detection means 5 (step S13). Translation component detecting means 4a
Calculates the translational motion component D of the image input means 2 from the sent image information, attitude information, and distance information at the first viewpoint A1 and the second viewpoint A2 (step S14).

When the translational motion component D of the image input means 2 is calculated by the translational component detection means 4a, as shown in FIG. 9, the distances from the first viewpoint A1 and the second viewpoint A2 to the gazing point Q are respectively determined. L ₁ and L ₂ , a first viewpoint A1 and a second viewpoint A
2, the image coordinates of the gazing point Q on the image planes I1 and I2 are Q ₁ (x ₁ , y ₁ , f), Q ₂ (x ₂ , y ₂ ,
f), the posture information R is detected by the posture detecting means 3 and the distances L ₁ and L ₂ are detected by the distance detecting means 8. Therefore, based on the camera coordinate system at the first viewpoint A 1, the first viewpoint Unit line-of-sight vector p from A1 to point of regard Q
₁ and the unit line-of-sight vector R from the second viewpoint A2 to the gazing point Q
p ₂ can be expressed by the following equation (8).

[0041]

(Equation 8)

Therefore, the first viewpoint A with respect to the gazing point Q
By detecting the distances L ₁ and L ₂ from the first and second viewpoints A2 and L2 and the posture information R, the translational motion component D is calculated by the following equation (9).
Can be calculated.

[0043]

(Equation 9)

The feature point disappearance detecting means 5 uses the calculated attitude information R, the translational motion component D, and the focal length f to obtain an image taken at the second viewpoint A2 for each feature point of the image at the first viewpoint A1. It is determined whether there is a possibility of disappearing in (Step S15). The correspondence detection unit 6 associates the feature points between the images of the first viewpoint A1 and the second viewpoint A2, except for the feature points designated by the feature point disappearance detection unit 5 in the preset feature point group. (Step S16). 3
The dimension calculation means 7 calculates the three-dimensional structure of the subject based on the principle of triangulation from the posture information R, the translational motion component D, and the association result (step S17). The position, orientation information, three-dimensional information and each image obtained in this way are stored in the storage means as needed (steps S18, S19).

As described above, the translational motion component D is obtained from the distance information to a certain point in the space projected on the image and the attitude information R of the image input means 2, so that a highly accurate translational motion component can be obtained. And the three-dimensional restoration accuracy can be further improved.

Further, by storing the line-of-sight direction of the feature point designated by the feature point disappearance detecting means 5 and removed by the correspondence detecting means 6 and its image feature amount, the feature point which may be lost is restored on the image plane. May be quickly detected and reset as a feature point. As shown in the block diagram of FIG. 10, the three-dimensional shape restoring device 1b of the third embodiment that stores the feature points that may be lost includes
The configuration is the same as that of the three-dimensional shape restoring apparatus 1 shown in FIG. 1 except that a feature point storage unit 9 and a feature point resetting unit 10 are provided.

When the three-dimensional shape restoring device 1b is set to restore the shape of an object whose distance Ln is between Lmin and Lmax, as shown in the flowchart of FIG. Image of the measurement target M at the first viewpoint A1 by using the
Calculates the attitude of the image input means 2 and sends it to the feature point disappearance detecting means 5 and the correspondence detecting means 6 (step S21). Next, the image input means 2 is moved to the second viewpoint A2 to photograph an image of the measuring object M, the attitude of the image input means 2 is calculated by the attitude detection means 3, and the image input means 2 is calculated by the translation component detection means 4. Is calculated and sent to the feature point disappearance detecting means 5 and the correspondence detecting means 6 (step S22). Feature point disappearance detecting means 5
Is a feature point P detected at the first viewpoint A1 based on relative posture information between the first viewpoint A1 and the second viewpoint A2 and a translational motion component.
In 1n, a feature point having a high possibility of disappearing at the second viewpoint A2 is obtained (step S23). The feature point storage unit 9 saves the existence range in the space of the feature point determined to have a high possibility of disappearance and the image feature point (step S24). When the possibility of this loss to save high and the determined characteristic point, from the distance Ln of the object is between Lmin and Lmax, the presence range of the point P ₁ on the corresponding spatial linear PminPmax Ask for. Then, as shown in FIG.
A viewpoint A1 using the attitude information R ₁₂ and translational motion component D ₁₂ between the second viewpoint A2, (3) ~ (5 ) second viewpoint from Eq A2
Pmin and Pmax calculated based on the camera coordinate system of
And the image feature values of the feature points determined to be lost, such as light and shade patterns and edge shapes, are stored. Correspondence detecting means 6 has a feature point P1
The feature points between the images of the first viewpoint A1 and the second viewpoint A2 are associated with each other except for the feature points designated by the feature point disappearance detecting means 5 in the n groups (step S25).

On the other hand, when the image input device 2 moves from the second viewpoint A2 to the third viewpoint A3, the feature point resetting means 10 sets the posture information R ₂₃ between the second viewpoint A2 and the third viewpoint A3.
And from translational motion component D _23, we calculate the spatial coordinates of the third viewpoint 3 of the camera coordinate system point relative to the Pmin and the point Pmax. Then, the point P at the third viewpoint A3 is obtained from the equations (3) to (5).
The image coordinates of min and the point Pmax are calculated, and the reappearance of the feature point is determined (steps S26 and S27). For example, in FIG. 12, assuming that the length of the portion where the straight line PminPmax is projected on the screen at the third viewpoint A3 is Lin and the length of the portion where the straight line PminPmax is not projected on the screen is Lout, [Lin / (Lin + Lou).
t)] When ≧ 0.8, it is determined that the feature point once disappeared is projected again at the third viewpoint A3. When the reproduction of the feature point is detected in this manner, it is determined using the image feature amount of the stored feature point and the epipolar constraint to determine where the stored feature point is reprojected in the image of the third viewpoint A3. The characteristic points that have been obtained and once lost are reset as characteristic points again (step S2).
8). This process is repeated for all the lost feature points, and then the three-dimensional shape of the subject is restored from the result of associating the posture information R, the translational motion component D, and the feature points (step S).
29). The position, orientation information, three-dimensional information and each image obtained in this way are stored in the storage means as needed (steps S30, S31).

In this way, by storing the line-of-sight direction of the lost feature point and its image feature amount, it is quickly detected that the lost feature point has reappeared in the image plane.
Since the feature points can be reset, the three-dimensional shape of the same object can be easily restored using a large number of images, and a highly accurate three-dimensional shape can be restored.

The three-dimensional shape restoring device 1b detects the inertial force generated by the motion of the image input means 2 and detects, for example, the GPS
A case has been described in which the translational motion component D of the image input means 2 is calculated using a navigation system such as that described above. However, the distance information detected by the distance detection means 8 and the attitude information of the image input means 2 calculated by the attitude detection means 3 The same applies to the case where the translational motion component D of the image input means 2 is calculated from R.

In each of the above embodiments, a case where an image is photographed from two or three viewpoints has been described. However, the present invention can be similarly applied to a case where an image is photographed from more than two viewpoints. Also, instead of performing the above processing in real time,
The image, the distance information, the posture information, and the like may be collectively stored in the storage unit, and then may be processed offline. Furthermore, various detected information may be transferred to a network or the like and then processed.

[0052]

As described above, according to the present invention, the disappearance of a feature point is detected from the attitude information of the image input means and the translational motion component. Thus, a highly accurate three-dimensional shape can be restored at a low calculation cost.

Further, by calculating the translational motion component of the image input means from the attitude information of the image input means and the distance information to a certain point in the space projected on the image and restoring the three-dimensional shape, more accurate accuracy can be obtained. The three-dimensional shape can be well restored.

Further, by storing the line-of-sight direction of the lost feature point and its image feature amount, it is possible to quickly detect that the lost feature point has reappeared in the image plane and reset it as a feature point. Therefore, the three-dimensional shape of the same object can be easily restored using a large number of images, and a highly accurate three-dimensional shape can be restored.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a state in which the same object is photographed from two viewpoints.

FIG. 3 is a flowchart showing the operation of the embodiment.

FIG. 4 is an arrangement diagram showing an optical system of an image input unit.

FIG. 5 is an explanatory diagram showing detection of a feature point that disappears.

FIG. 6 is an explanatory diagram showing correspondence of feature points.

FIG. 7 is a block diagram showing a configuration of a second embodiment.

FIG. 8 is a flowchart showing the operation of the second embodiment.

FIG. 9 is an explanatory diagram showing an operation of calculating a translational motion component from distance information.

FIG. 10 is a block diagram illustrating a configuration of a third embodiment.

FIG. 11 is a flowchart showing the operation of the third embodiment.

FIG. 12 is an explanatory diagram showing a process of storing and reappearing disappearing feature points.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 3D shape restoration device 2 Image input means 3 Attitude detection means 4 Translation component detection means 5 Feature point disappearance detection means 6 Correspondence detection means 7 3D calculation means 8 Distance detection means 9 Feature point storage means 10 Feature point resetting means

Claims (6)

[Claims] An image of a subject is input from a plurality of viewpoints by an image input unit, posture information and a translational motion component of the image input unit at each viewpoint are calculated, and the input image information and the calculated posture information of the image input unit are calculated. And the translational motion component, the feature points of each image are associated with each other, and the three-dimensional shape of the subject is calculated and restored from the correspondence between the image information, the attitude information of the image input means, the translational motion component, and the feature point. In the three-dimensional shape restoration device, there is provided a feature point disappearance detecting means for detecting a feature point disappearing from an image plane photographed from a different viewpoint from the posture information and the translational motion component of the image input means. A three-dimensional shape restoring apparatus characterized in that disappearing feature points are removed and feature points of each image are associated with each other. 2. An image of a subject is input from a plurality of viewpoints by an image input unit, posture information of the image input unit at each viewpoint is calculated, and a position corresponding to a point projected on an image plane of each viewpoint is calculated. Calculate the distance to the point, calculate the translational motion component of the image input means from the attitude information of the image input means and the distance information to the point in space, and input the image information and the calculated attitude information of the image input means and the translation. The three-dimensional shape for associating the feature points of each image from the motion component, and calculating and restoring the three-dimensional shape of the subject from the correspondence between the image information, the posture information of the image input means, the translational motion component, and the feature point The restoration device has feature point disappearance detecting means for detecting a feature point disappearing from an image plane photographed from a different viewpoint from the posture information and the translational motion component of the image input means, and the feature point disappearance detected by the feature point disappearance detecting means is provided. 3D reconstruction apparatus characterized by removing the feature points to associate feature points in each image. 3. A feature point storage means for storing the existence range in the space of the feature point whose loss has been detected by the feature point loss detection means and its image feature quantity, and the stored feature point is displayed on another screen again. 3. The three-dimensional shape restoring device according to claim 1, further comprising a feature point resetting means for detecting that the image is projected on the three-dimensional shape. 4. An image input unit inputs an image of a subject from a plurality of viewpoints, calculates posture information and a translational motion component of the image input unit at each viewpoint, and calculates the input image information and the calculated posture information of the image input unit. And the translational motion component, the feature points of each image are associated with each other, and the three-dimensional shape of the subject is calculated and restored from the correspondence between the image information, the attitude information of the image input means, the translational motion component, and the feature point. In the three-dimensional shape restoring method, feature points disappearing from an image plane taken from different viewpoints are detected from the posture information and the translational motion component of the image input means, and the disappearing feature points are removed to associate feature points of each image. A three-dimensional shape restoring method. 5. An image of a subject is input from a plurality of viewpoints by an image input unit, posture information of the image input unit at each viewpoint is calculated, and a position corresponding to a point projected on an image plane of each viewpoint is calculated. Calculate the distance to the point, calculate the translational motion component of the image input means from the attitude information of the image input means and the distance information to the point in space, and input the image information and the calculated attitude information of the image input means and the translation. The three-dimensional shape for associating the feature points of each image from the motion component, and calculating and restoring the three-dimensional shape of the subject from the correspondence between the image information, the posture information of the image input means, the translational motion component, and the feature point In the restoration method, feature points disappearing from an image plane photographed from different viewpoints are detected from the posture information of the image input means and distance information to a point in space, and the detected disappearing feature points are removed to remove each feature point of each image. Feature point pairs 3D reconstruction method and performing Paste. 6. The image processing apparatus according to claim 1, wherein an existence range of the feature point in which the disappearance is detected in a space and an image feature amount thereof are stored, and it is detected that the stored feature point is projected on another screen again. 7. The three-dimensional shape restoration method according to 5 or 6.