JPH0981790A - Device and method for three-dimensional shape restoration - Google Patents

Abstract

PROBLEM TO BE SOLVED: To restore a three-dimensional shape with high precision by enabling a viewpoint of an image to be freely set according to the shape of a subject. SOLUTION: This device is provided with an angular velocity sensor 6 and an acceleration sensor 7 which detect the movement of an image pickup device and an analysis part 8 which decides a correction quantity in an optical-axis direction on the basis of the movement of the image pickup device detected by the two sensors 6 and 7 and the movement of a subject image found by an image processing part 5; and a viewpoint at photography time can freely be selected by correcting an optical-axis direction at each viewpoint so that the optical axes from different viewpoints cross each other in the photography, and the respective images are made to correspond to each other so that the coordinate axes of the images from the respective viewpoints become common, thereby lightening the load of processing when the solid shape is restored.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image in which a specific subject is photographed from different viewpoints, corresponding points in a plurality of images obtained thereby are determined, and the three-dimensional shape of the subject is restored on the image. Shape restoration apparatus and method.

[0002]

2. Description of the Related Art Conventionally, a method is known in which a specific subject is photographed from different viewpoints, corresponding points in a plurality of obtained images are determined, and the three-dimensional shape of the subject is restored. Such 3D information acquisition method is, for example, “robot vision”.
(Masahiko Taniuchida: Shokoido). This method obtains three-dimensional information of a subject based on the principle of triangulation by utilizing parallax caused by different viewpoints.

[0003]

However, in the above-mentioned conventional technique, it is difficult to determine the corresponding points in a plurality of images. Therefore, in order to restore the three-dimensional shape with high accuracy, the distance between the images is reduced. It was necessary to understand the relative positional relationship.

However, in the past, the position of the subject with respect to the measuring device was fixed, and the shooting position was also predetermined, so that it was not possible to shoot from a free position. Therefore, it may not be possible to shoot from a viewpoint suitable for the shape of the subject, and it may be difficult to restore the three-dimensional shape depending on the shape of the subject.

The present invention has been made in view of such a situation, and has a function of detecting the movement of the viewpoint at the time of photographing to control the position of the subject in the image, thereby making the viewpoint of the image the subject. It is possible to freely set the shape of the three-dimensional shape and to restore the three-dimensional shape with high accuracy.

[0006]

A three-dimensional shape restoration device of the present invention takes a specific subject from different viewpoints, obtains corresponding points of the subject in a plurality of obtained images, and determines the three-dimensional shape of the subject. The three-dimensional shape restoration device for restoring the image on the image includes optical axis direction correction means for correcting the optical axis direction at each viewpoint so that the optical axes from the different viewpoints intersect at arbitrary points.

Another feature of the present invention is that the optical axis direction correcting means detects the movement of the photographing optical system during photographing, and corrects the optical axis direction based on the detected movement. It is characterized by

Another feature of the present invention is that a specific subject is photographed from different viewpoints, corresponding points of the subject in a plurality of obtained images are obtained, and the three-dimensional shape of the subject is restored on the image. In the three-dimensional shape restoration device, the photographing means for photographing a predetermined pattern together with the subject at the time of photographing the subject, and the pattern and the viewpoint by recognizing the pattern in the image obtained by the photographing by the photographing means. And an optical axis direction correcting means for correcting the deviation in the optical axis direction from an arbitrary point based on the detection result.

Another feature of the present invention is that the optical axis direction correcting means detects a viewpoint at the time of photographing from the pattern deformation state at the time of photographing the pattern, and based on the detection result, the optical axis direction correcting means detects the viewpoint. The feature is that the correction amount is obtained.

Another feature of the present invention is that a specific subject is photographed from different viewpoints, corresponding points of the subject in a plurality of obtained images are obtained, and the three-dimensional shape of the subject is restored on the image. In the three-dimensional shape restoration device, the optical axis direction correction means for correcting the optical axis direction at each viewpoint is provided so that the subject is contained in the image.

Another feature of the present invention is that the optical axis direction correcting means detects the movement of the photographing optical system during photographing and the movement of the photographing optical system detected by the detecting means. The subject area is determined by comparing the movement obtained by the transformation means with the transformation means for transforming the movement at the subject position with the movement obtained from the image obtained by photographing, and the subject area is the image. And a means for correcting the optical axis direction at each viewpoint so as to be contained therein.

According to the three-dimensional shape restoration method of the present invention, a specific subject is photographed from different viewpoints, corresponding points of the subject in a plurality of obtained images are obtained, and the three-dimensional shape of the subject is restored on the image. In the three-dimensional shape restoration method, the optical axis directions at the respective viewpoints are corrected so that the optical axes from the different viewpoints intersect at arbitrary points.

Another feature of the present invention is that the correction in the optical axis direction according to claim 7 is performed by detecting the movement of the photographing optical system during photographing and based on the detected movement. Is characterized in that.

Another feature of the present invention is that a specific subject is photographed from different viewpoints, corresponding points of the subject in a plurality of obtained images are obtained, and the three-dimensional shape of the subject is restored on the image. In the three-dimensional shape restoring method, the predetermined pattern is photographed together with the subject at the time of photographing the subject, and the pattern in the image obtained thereby is recognized to detect the relative positional relationship between the pattern and the viewpoint, Based on the detection result, the deviation from the arbitrary point in the optical axis direction is corrected.

Another feature of the present invention is that the correction in the optical axis direction according to claim 9 detects the viewpoint at the time of photographing from the deformed state of the pattern at the time of photographing the pattern, and the detection result thereof. It is characterized in that the correction amount in the optical axis direction is obtained based on

Another feature of the present invention is that a specific subject is photographed from different viewpoints, corresponding points of the subject in a plurality of obtained images are obtained, and the three-dimensional shape of the subject is restored on the image. In the three-dimensional shape restoration method described above, the optical axis direction at each viewpoint is corrected so that the subject is contained in the image.

Another feature of the present invention is that the correction in the optical axis direction according to claim 11 detects the movement of the photographing optical system during photographing, and detects the movement of the photographing optical system. The subject area is determined by converting it into the above movement and comparing the movement thus obtained with the movement obtained from the image obtained by shooting, and each viewpoint is adjusted so that the above-mentioned subject area fits in the image. It is characterized in that the optical axis direction is corrected.

[0018]

Since the present invention comprises the above technical means, it is possible to freely select the viewpoint at the time of shooting, and even if the viewpoint is freely set in this way, the optical axis from each viewpoint does not have to be an arbitrary point. Since the optical axis directions are adjusted so as to always intersect, the coordinate axes of the images from the respective viewpoints are common, the correspondence between the images is facilitated, and the processing load at the time of restoring the three-dimensional shape is reduced.

Further, according to another feature of the present invention, it becomes possible to freely select the viewpoint at the time of photographing without requiring special sensors for detecting the movement of the photographing optical system. The load of processing when the three-dimensional shape is restored is reduced.

[0020]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 5 shows a state in which a subject is photographed in the present embodiment. The subject is in a specific photographing range, and the photographer views the viewpoints A, B, and
The figure shows the case of shooting while changing to C, ....

By the way, as a method of highly accurately restoring a three-dimensional shape, there is a method of associating images by utilizing the relative positional relationship between different viewpoints or the relative positional relationship between a subject and a viewpoint.

For example, as shown in FIG. 6, considering the coordinates around the viewpoint O, the point a in the photographed image
The relationship between (x, y) and the actual subject position A (X, Y) is as follows, where f is the focal length of the optical system at the time of shooting and Z is the distance between the viewpoint O and the point A on the subject. It can be represented by an expression. x = Xf / Z, y = Yf / Z (1 expression)

When the viewpoint is changed in this state, it is necessary to convert the expressions before and after the change by parallel movement and rotation in the optical axis direction. For example, if the viewpoint moves in the X direction by a distance dX and the coordinate axis rotates by an angle dQ, the coordinates (X ′, Y ′, Z ′) of the point A ′ after the point A is converted.
Is as follows. X ′ = Xcos dQ + Zsin dQ + dX (Equation 2) Y ′ = Y (Equation 3) Z ′ = − Xsin dQ + Zcos dQ (Equation 4)

Further, a point b commonly existing in a plurality of images
If there is, the distance from the viewpoint to the subject is obtained by the following method. Now, assume that the viewpoint is simply translated in the X direction by dX. At this time, if the inclination of the pixel at the point b in the image is Eb and the inclination of the pixel at the point b between the two images before and after the change is Et, then the following equation is established with f being the focal length. −f · Eb · dX / Z + Et = 0 (5 expression)

Therefore, if the relative positional relationship between viewpoints and the focal length f of the optical system are known, it becomes possible to associate the images. However, in reality, the relative positional relationship between the viewpoints has six degrees of freedom in directions and angles, and if there are a plurality of images, the association process becomes complicated. Therefore, in the present embodiment, in order to simplify this processing, the coordinate axes are made common between a plurality of images.

As shown in FIG. 5, the coordinate axes are commonly used by controlling so that the optical axes from the viewpoints A, B, and C always intersect at one point. That is, the control is performed such that the optical axis direction is always directed to the common point corresponding to the point P in FIG. As a result, the processing load is reduced, and the three-dimensional shape can be restored in real time.

An example of an apparatus for realizing this method is shown in FIG. As shown in FIG. 1, the three-dimensional shape restoration apparatus according to the present embodiment has a lens 1, an imaging device 2, and a first A / A unit.
D converter 3a, second A / D converter 3b, third A / D
The converter 3c, the signal processing unit 4, the image processing unit 5, the angular velocity sensor 6, the acceleration sensor 7, the analysis unit 8, the memory 9, the optical axis control unit 10, the prism 11, the drive unit 12, and the restoration unit 13 are included.

The lens 1 is for photographing a subject, and forms a subject image on the image pickup device 2. The image pickup device 2 is for converting the subject image formed by the lens 1 into an electric signal.

The first A / D converter 3a converts the analog signal from the image pickup device 2 into a digital signal, and the second A / D converter 3b converts the analog signal from the angular velocity sensor 6 into a digital signal. Converted to digital signal, the third A
The / D converter 3c converts the analog signal from the acceleration sensor 7 into a digital signal.

The signal processing section 4 includes the image pickup device 2
The electric signal output from the first A / D converter 3a and digitized by the first A / D converter 3a is converted into an image signal. The image processing unit 5 processes the image signal converted by the signal processing unit 4 to detect a motion vector.

The angular velocity sensor 6 detects the movement of the image pickup device based on the angular velocity, and the acceleration sensor 7 detects the movement of the image pickup device based on the acceleration. The analysis unit 8 determines the correction amount in the optical axis direction, and the memory 9 stores the information and the image in the optical axis direction.

The optical axis control unit 10 adjusts the optical axis in accordance with the signal from the analysis unit 8, and the prism 11 is for actually adjusting the optical axis. The drive unit 12 is for driving the prism 11 to adjust the optical axis direction, and the restoration unit 13 restores the three-dimensional shape of the subject from the obtained image and the relative positional relationship between the images. It is a thing.

In the three-dimensional shape restoration apparatus of this embodiment having the above-described structure, a subject image is formed on the image pickup device 2 via the lens 1 and is converted into an electric signal at the time of photographing. The electric signal output from the image pickup device 2 is digitized by the first A / D converter 3a and then converted into an image signal by the signal processing unit 4.

The image signal output from the signal processing unit 4 is sequentially sent to the image processing unit 5 and the analyzing unit 8. The image processing unit 5 calculates a motion vector in a plurality of regions set in the image from the image signal directly sent from the signal processing unit 4 and the image signal stored in the memory 9 for a predetermined period. I do.

Here, as a method for obtaining a motion vector from two image signals in time series, a known method can be used. The detailed description is omitted here. The motion vector calculated by the image processing unit 5 is then sent to the analysis unit 8.

On the other hand, the angular velocity sensor 6 and the acceleration sensor 7 attached to the image pickup apparatus main body (not shown) detect the movement (rotation / parallel motion component) of the image pickup apparatus. The signals detected by these two sensors 6 and 7 are digitized through A / D converters 3b and 3c, respectively. These digitized angular velocity signal and acceleration signal are sent to the analysis unit 8. In the analysis unit 8,
In this way, the image signal, the motion vector, the angular velocity signal and the acceleration signal are input respectively.

FIG. 2 shows a process of processing in the analysis unit 8.
The analysis unit 8 first determines what kind of motion the imaging device has performed. That is, in the first step S1, a signal indicating the motion of the image pickup apparatus is sent from the two sensors 6 and 7 in the form of an angular velocity signal and an acceleration signal, which is input.

Then, the process proceeds to step S2 and step S1.
The angular velocity signal and the acceleration signal input at are synchronized with the image signal and integrated to form an angle, a velocity and a moving distance. These values are recorded in the memory 9 and the next angle
It is added as a constant term when calculating the speed / movement distance.

Next, the process proceeds to step S3, and step S2
A motion vector corresponding to the subject portion in the image is calculated and calculated from the angle and the speed obtained in. That is,
An arbitrary point p (x, y) in the area is selected for each of the plurality of areas for which the motion vector is set in the image. Then, x centered on the optical node on the observation side
An yz coordinate system is set, horizontal motion components in the x, y, and z axis directions are Tx, Ty, and Tz, and rotary motion components around the x, y, and z axes are Wx, Wy, and Wz.

Now, assuming that the focal length of the optical system is f and the distance Z to the object is set to a value corresponding to the distance to the object, the motion vector (u, v) can be obtained using the following approximate expression. . u = (-f ・ Tx + x ・ Tz) / Z + (-f ・ Wy + y ・ Wz + x ・ y ・ Wx / fx ²・ Wy / f) (6 formulas) v = (-f ・ Ty + y ・ Tz) / Z + (f ・ Wz-x ・ Wz-x ・ y ・ Wy / f + y ²・ Wx / f) (7 formulas)

Next, in step S4, the motion vector from the image is input from the image processing unit 5. Then, in the next step S5, the motion vector calculated in step S3 is compared with the motion vector obtained from the image input in step 4, and if these vector elements are substantially equal, a subject exists. Judge as an area. Next, in step S6, the presence of the subject is determined for all the regions, and the subject region in the image is determined.

After the subject area is determined, the process proceeds to step S7 to check the correlation with the past subject area stored in the memory 9. Here, when the obtained correlation value is larger than the preset threshold value, it is determined that the subject is the same as the previous subject, and the process proceeds to the next process.

On the other hand, if the correlation value is smaller than the threshold value at this point, the optical axis is not adjusted and the current state is maintained. At this point, in step S8, the image signal and the data regarding the area determined to be the subject in the image are stored in the memory 9.

Next, in steps S9 to S11, the correction amount in the optical axis direction is determined and the subject distance is calculated. That is, the angular velocity sensor 6 and the acceleration sensor 7
From this viewpoint angle / distance change amount obtained by, the current correction amount for the position where the optical axis intersects up to the previous time is geometrically calculated, and this is set as the angle correction amount.

Further, as described above, the motion vector between the image captured last time and the image captured this time is obtained by the image processing section 5, and it is judged from this motion vector that it is the subject region this time. The vector of the region is extracted, and the displacement amount on the two-dimensional image is obtained based on this.

Since the calculated shift amount is a positional shift in units of pixels, it is converted into angle information. The relationship between the shift amount and the angle can be expressed by the following equation, where f is the focal length, x is the shift amount in pixel units on the image plane, and a is the angle. f = x · tan (a) (Equation 8)

The shift amount is converted into an angle by using the above (Equation 8). Next, the distance to the subject is calculated from the deviation amount (angle) and the respective signals of the angle, speed, and movement distance obtained from the two sensors 6 and 7.

At this point, if the point where the optical axes of the respective viewpoints intersect is not set, steps S9 to S10 are performed.
Then, based on the distance to the subject obtained earlier, the intersection is set to be farther than this distance. Then, in step S11, the correction amount is calculated in the optical axis direction with respect to the set position, and the position of the intersection is stored in the memory 9.

The angle information for correcting the deviation in the optical axis direction obtained by the analysis unit 8 as described above is sent to the optical axis control unit 10 as a correction signal. In addition, information about the position of the subject such as angle, speed, and moving distance is stored in the memory 9.
In the optical axis control unit 10, the amount of change of the prism 11 corresponding to the correction signal given from the analysis unit 8 is obtained and used as the correction signal for the prism.

The correction signal obtained by the optical axis control unit 10 is
It is sent to the drive unit 12. The drive unit 12 changes the optical axis direction by changing the angle of the prism 11 placed in front of the lens 1, and corrects the subject position in the image. With respect to this optical axis correction method, the same effect can be obtained by adjusting the lens barrel direction by a stepping motor in addition to the correction by the prism 11 as described above.

Further, in the restoration section 13, a plurality of images obtained by photographing and information on the relative positional relationship between the images are read out from the memory 9, and the above-mentioned (1 formula) to (4) are read.
(3) is used to restore the three-dimensional shape of the subject.

By the above method, even when the subject is photographed while moving the viewpoint to the subject,
The optical axis from each viewpoint always passes through the set point. Therefore, when an image obtained by using this method is used for processing such as three-dimensional shape recognition, the processing accuracy can be increased and the processing load can be reduced.

The second embodiment will be described below with reference to the drawings. FIG. 3 shows an image pickup apparatus according to the second embodiment, which includes a lens 1, an image pickup device 2, and a first A / D.
Converter 3a, fourth A / D converter 3d, signal processor 4,
The image extraction unit 16, the magnetic sensor 14, the sensor drive unit 15, the analysis unit 8, the memory 9, and the restoration unit 13 are included.

Here, the image extraction unit 16 performs a process of cutting out an image in a required range from the image signal obtained by the signal processing unit 4. The magnetic sensor 14 detects the position and angle of the image pickup device. The sensor drive unit 15 generates a magnetic field for operating the magnetic sensor 14.

The fourth A / D converter 3d is for converting the analog signal output from the magnetic sensor 14 into a digital signal. Memory 9 above
Is for storing subject information. Other parts are the same as those shown in FIG.

As for the situation of photographing in the present embodiment, it is assumed that the sensor drive unit 15 shown in FIG. 3 is near or at the same position as the subject. Note that, in FIG. 3, the processing until the image signal is obtained by the signal processing unit 4 is the first
The description is omitted because it is the same as the embodiment. The image signal obtained by the signal processing unit 4 is sent to the image extracting unit 16.

On the other hand, from the magnetic sensor 14 attached to the main body of the image pickup device, a signal indicating the position and angle of the image pickup device with the point where the sensor drive unit 15 is placed as the coordinate center is obtained. The magnetic sensor 14 detects this position / angle signal in synchronization with the image signal. The position / angle signals detected by the magnetic sensor 14 are digitized through the fourth A / D converter 3d. Then, the digitized position / angle signal is sent to the analysis unit 8.

The analysis unit 8 determines the correction amount in the optical axis direction. That is, the direction of the optical axis at the time of imaging and the ideal direction of the optical axis are calculated from the obtained position / angle signals. The ideal direction of the optical axis can be set in advance as a relative positional relationship between the target point and the image capturing position. Here, a straight line connecting the position where the sensor / drive unit 15 is located and the image capturing position is an ideal optical axis. The axis.

As described above, the position / angle (orientation) of the image pickup device can be known by the magnetic sensor 14. Therefore, the position of the ideal optical axis in the image is determined from the geometrical relation as described above. It is possible to calculate from (Equation 8). The analysis unit 8 sends the position of the optical axis on the image thus calculated to the image extraction unit 16 as an optical axis correction signal.

The image extraction unit 16 cuts out a necessary portion of the image from the image signal obtained by the signal processing unit 4 based on the optical axis correction signal obtained by the analysis unit 8. That is, when the image signal is input from the signal processing unit 4, the image extraction unit 16 performs optical axis adjustment by cutting out the image based on the optical axis correction signal from the analysis unit 8.

In other words, the image extraction unit 16 centers on the position on the image indicated by the optical axis correction signal from the analysis unit 8,
Cuts out the image within the set range. Then, the cut-out image is used as a new image signal, and the image signal and the position / angle signal of the imaging device are sent to the restoration unit 13. The restoration unit 13 restores the three-dimensional shape of the subject based on the image thus obtained and the relative positional relationship between the viewpoint and the subject.

The three-dimensional shape of the subject can be obtained by the above method. Compared to the first embodiment, this embodiment has a merit that control is easy because there is less time delay in processing from detection of the position and angle of the image pickup device to optical axis correction.

The third embodiment will be described below with reference to the drawings. FIG. 4 shows an image pickup apparatus according to the third embodiment, which includes a lens 1, an image pickup device 2, and a first A / D.
Converter 3a, signal processing unit 4, analysis unit 21, memory 9, optical axis control unit 10, restoration unit 13, stepping motor 18
And a drive unit 22.

Here, the analysis unit 21 is for determining the correction amount of the subject position. The stepping motor 18 is for moving an optical system of the image pickup device on a rail 17 described later, and drives the optical system of the image pickup device on the rail 17 in the yaw angle, pitch angle, and roll angle directions.
The drive unit 22 is for driving the stepping motor 18.

As for the situation of photographing in the present embodiment, as shown in FIG. 7, the imaging device is fixed to the rail 1
7 can be freely scanned. Then, a card 20 on which an appropriate plurality of patterns 19 are drawn is placed at an arbitrary position within the rail 17 in the photographing range. This card 20 is used to determine the position within the rail 17.

The signal of the pattern 19 is digitized and stored in the memory 9 in the image pickup apparatus in advance. As shown in FIG. 7, it is assumed that the subject is placed on the card 20 in a state where all the patterns 19 are not hidden. When the subject is larger than the photographing area in the rail 17, the pattern 19 may be present on the subject.

The subject and the image of the card 20 within the photographing range set by the rail 17 are imaged on the image pickup device 2 through the lens 1 of FIG. 4 and converted into an electric signal. The electric signal output from the imaging device 2 is digitized through the first A / D converter 3a and then converted into an image signal by the signal processing unit 4. The image signal output from the signal processing unit 4 is sent to the analysis unit 21.

The analysis unit 21 first matches the image signal input from the signal processing unit 4 with the signal of the pattern 19 on the card 20 stored in the memory 9. If the pattern 19 is not present in the image, the analysis unit 21 determines that there is no subject in the image, and the optical axis adjustment is not performed.

If the pattern 19 is present in the image, the pattern shown in FIG.
As shown in, the appearance of the pattern 19 on the card 20 changes depending on the optical axis direction of the optical system. Therefore, in consideration of this point in the matching process, some patterns 19 ′ obtained by modifying the existing pattern 19 are created, and the matching process between the modified pattern 19 ′ and the pattern in the image is sequentially performed. How to transform this pattern is
A pattern 19 'formed by projecting the card 20 at an appropriate angle with respect to a certain plane is used.

Next, the best matching transformation pattern 1
9'is selected, this modified pattern 19 'is compared with the original pattern 19, and the optical axis direction looking at the original pattern 19 is calculated. This can be calculated by obtaining the angle projected when the pattern 19 is deformed.

Further, a plurality of patterns 1 are formed on the card 20.
9 exists, the relative positional relationship between the card 20 and the optical system can be known from the positional relationship of these patterns 19. With the above processing, the amount of deviation between the arbitrary position on the card 20 and the position where the optical axis intersects the card 20 can be calculated.

Therefore, by setting an arbitrary position on the card 20 in the memory 9 in advance, even if the position of the optical system moves, the shift amount can always be detected. Since this displacement amount is calculated in units of pixels, it is converted into an angle using (Equation 8) to obtain a final position correction value. The position correction value and the image are stored in the memory 9.

Further, the position correction signal obtained as described above is sent to the optical axis controller 10. Optical axis control unit 10
Then, the drive amount of the stepping motor 18 corresponding to the position correction signal from the analysis unit 21, that is, the drive amount for changing the yaw angle, the pitch angle, and the roll angle of the lens barrel is obtained and sent to the drive unit 22. .

Then, the drive unit 22 drives the stepping motor 18 connected to the lens barrel to change the optical axis direction and correct the subject position in the image. Restoration unit 1
In 3, the three-dimensional shape of the subject is restored from the image obtained by photographing and the relative positional relationship between the viewpoint and the subject.

By the above method, the three-dimensional shape of the subject can be obtained. Compared with the first and second embodiments, this embodiment does not require a sensor, and thus has an advantage that the device configuration can be simplified.

[0076]

As described above, according to the present invention, the optical axis direction at each viewpoint is corrected so that the optical axes from different viewpoints intersect at arbitrary points. It becomes possible to make a selection, and the coordinate axes of the images from the respective viewpoints can be made common to facilitate the correspondence between the images, thereby reducing the processing load when the three-dimensional shape is restored. be able to. Therefore, it is possible to perform shooting from a viewpoint suitable for the subject, which can improve the accuracy of restoring the three-dimensional shape of the subject and increase the processing speed at the time of restoration.

According to another feature of the present invention, a predetermined pattern is photographed together with a subject, and the optical axis correction is performed based on the relative positional relationship between viewpoints detected by pattern matching, so that it is special. It is possible to shoot from a viewpoint suitable for the subject without the need for a sensor. With a simple device configuration, it is possible to improve the accuracy of the three-dimensional shape restoration of the subject and increase the processing speed during restoration. be able to.

According to another feature of the present invention, since the optical axis direction at each viewpoint is corrected so that the subject is contained in the image, the viewpoint at the time of photographing can be freely selected. In addition, the correspondence between the images can be facilitated, and the processing load when the three-dimensional shape is restored can be reduced. Therefore, it is possible to perform shooting from a viewpoint suitable for the subject, which can improve the accuracy of restoring the three-dimensional shape of the subject and increase the processing speed at the time of restoration.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a flowchart showing processing of an analysis unit in the first embodiment.

FIG. 3 is a block diagram showing a second embodiment of the present invention.

FIG. 4 is a block diagram showing a third embodiment of the present invention.

FIG. 5 is a diagram showing a state in which a subject is photographed in the first embodiment.

FIG. 6 is a diagram showing a relationship between a subject and an imaging surface when a viewpoint is an origin.

FIG. 7 is a diagram showing a state in which a subject is photographed in the third embodiment.

FIG. 8 is a diagram showing a deformed state of a pattern from a viewpoint in the third embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 lens 2 imaging device 3a, 3b, 3c, 3d A / D converter 4 signal processing unit 5 image processing unit 6 angular velocity sensor 7 acceleration sensor 8, 21 analysis unit 9 memory 10 optical axis control unit 11 prism 12, 22 drive unit 13 restoration unit 14 magnetic sensor 15 sensor drive unit 16 image extraction unit 17 rail 18 stepping motor 19 pattern 20 card

 ─────────────────────────────────────────────────── ─── Continued Front Page (72) Inventor Toshiaki Kondo 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Incorporated (72) Inventor Katsumi Iijima 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (12)

[Claims] 1. A specific subject is photographed from different viewpoints,
Obtaining corresponding points of the subject in the obtained plurality of images, in a three-dimensional shape restoration device that restores the three-dimensional shape of the subject on the image, so that the optical axes from the different viewpoints intersect at any point,
A three-dimensional shape restoration device comprising an optical axis direction correction means for correcting the optical axis direction at each viewpoint. 2. The optical axis direction correcting means detects a movement of a photographing optical system during photographing, and corrects the optical axis direction based on the detected movement. The three-dimensional shape restoration device described. 3. A specific subject is photographed from different viewpoints,
In a three-dimensional shape restoration device that finds corresponding points of the subject in the obtained plurality of images and restores the three-dimensional shape of the subject on the image, a photographing means for photographing a predetermined pattern together with the subject at the time of photographing the subject. Detecting the relative positional relationship between the pattern and the viewpoint by recognizing the pattern in the image obtained by the photographing by the photographing means, and based on the detection result, in the optical axis direction from an arbitrary point. A three-dimensional shape restoring device, comprising: an optical axis direction correcting means for correcting a deviation. 4. The optical axis direction correction means detects a viewpoint at the time of photographing from a pattern deformation state at the time of photographing the pattern, and obtains a correction amount in the optical axis direction based on the detection result. The three-dimensional shape restoration device according to claim 3. 5. A specific subject is photographed from different viewpoints,
In a three-dimensional shape restoration device that finds corresponding points of the subject in the obtained plurality of images and restores the three-dimensional shape of the subject on the image, the optical axis at each viewpoint is set so that the subject fits in the image. A three-dimensional shape restoration device comprising an optical axis direction correction means for correcting the direction. 6. The optical axis direction correcting means detects the movement of the photographing optical system during photographing, and converts the movement of the photographing optical system detected by the detecting means into movement on the subject position. The subject area is determined by comparing the movement obtained by the converting means and the movement obtained by the image obtained by photographing with the movement obtained by the converting means, and at each viewpoint so that the subject area is contained in the image. 6. The three-dimensional shape restoration device according to claim 5, further comprising means for correcting the optical axis direction of the. 7. A specific subject is photographed from different viewpoints,
Obtaining corresponding points of the subject in the obtained plurality of images, in the three-dimensional shape restoration method of restoring the three-dimensional shape of the subject on the image, so that the optical axes from the different viewpoints intersect at any point,
A three-dimensional shape restoring method characterized by correcting the optical axis direction at each viewpoint. 8. The three-dimensional correction according to claim 7, wherein the correction in the optical axis direction is a correction performed by detecting a movement of a photographing optical system during photographing and based on the detected movement. Shape restoration method. 9. A specific subject is photographed from different viewpoints,
In the three-dimensional shape restoration method of obtaining corresponding points of the subject in the obtained plurality of images and restoring the three-dimensional shape of the subject on the image, a predetermined pattern is photographed together with the subject at the time of photographing the subject, and By recognizing the pattern in the image obtained by detecting the relative positional relationship between the pattern and the viewpoint, and correcting the deviation in the optical axis direction from an arbitrary point based on the detection result. A three-dimensional shape restoration method characterized by: 10. The correction in the optical axis direction according to claim 9,
A three-dimensional shape restoration method, comprising: detecting a viewpoint at the time of shooting from the deformed state of the pattern at the time of shooting the pattern, and obtaining a correction amount in the optical axis direction based on the detection result. 11. A three-dimensional shape restoration method for photographing a specific subject from different viewpoints, obtaining corresponding points of the subject in a plurality of obtained images, and restoring the three-dimensional shape of the subject on the image. A three-dimensional shape restoration method, characterized in that the optical axis direction at each viewpoint is corrected so that the subject fits in the image. 12. The optical axis direction correction according to claim 11, wherein a movement of the photographing optical system at the time of photographing is detected, and the detected movement of the photographing optical system is converted into a movement on a subject position. A subject area is determined by comparing the obtained movement with the movement obtained from an image obtained by shooting, and the optical axis direction at each viewpoint is corrected so that the subject area is contained in the image. A three-dimensional shape restoration method characterized by: