JP2001028049A - Method and device for three-dimensional shape acquisition, and computer-readable recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for three-dimensional shape acquisition, and a recording medium which restore the three-dimensional shape of a body or scene with high precision even when the objective three-dimensional body has large depth. SOLUTION: The method has an input step for inputting an image from a camera to a computer, a 1st extraction step (S3) for extracting a three- dimensional shape from the image by a specific method, a generation step (S7) for generating a depth image from the three-dimensional shape, a correction step (S8) for correcting a shake of the image due to the movement of the camera by using the depth image, a 2nd extraction step for extracting a three- dimensional shape by a specific method by using the shake-corrected image, and an output step for outputting the three-dimensional shape to a storage device. The above mentioned methods are based upon epipolar image analysis.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method, an apparatus, and a recording medium for obtaining a three-dimensional shape for estimating a three-dimensional structure required for generating a highly realistic image from a real image.
In recent years, demand for three-dimensional computer graphics has been increasing in various aspects such as games, movies, and commercial films. However, creating a three-dimensional model as the original data of the three-dimensional CG is a labor-intensive operation. Therefore, there is a need for a method of automatically or semi-automatically three-dimensionally modeling an actually existing object or landscape.

[0002]

2. Description of the Related Art Conventionally, various input methods for inputting a three-dimensional shape such as an object and a landscape have been proposed. For example, in the literature "RC Bolles, HH Baker and DH
In Marimont, “IJCV, vol.1, No.1, 1987, pp.7-55” (hereinafter referred to as Reference 1), a spatio-temporal image is constructed by inputting a large number of images of a landscape taken with a video camera. And that xt
A method called epipolar image analysis for obtaining a three-dimensional shape by analyzing a linear pattern appearing in a plane image has been proposed. This method is characterized in that the analysis method is simple, but it is necessary to accurately and linearly move the camera at the time of photographing. If an unexpected movement is accompanied by camera shake or vibration, the pattern appearing in the image on the xt plane will not be a straight line, and the accuracy of extracting a three-dimensional shape will be significantly reduced.

Japanese Patent Application Laid-Open No. 11-339043, entitled "Three-dimensional shape input method and a recording medium recording the three-dimensional shape input method" (hereinafter referred to as Document 2) discloses a method of pre-processing the method of Document 1 in which feature points of an image are used. There has been proposed a method of estimating a motion due to camera shake or the like from a track that has been tracked, and correcting an image from the estimation result as if the camera performed a linear motion at a constant velocity. According to this method, it is not necessary to accurately move the camera in a linear motion at the time of photographing, and the epipolar image analysis can be easily performed. Similarly, the references “Z. Zhu, G. Xu and X. Lin,“ Constructing
3D Natural Scenefrom Video Sequences with Vibrate
d Motions, ”Proc. IEEE VRAIS '98, 1998, pp.105-1
In “12”, there is proposed a method of estimating a motion due to camera shake or the like from an optical flow of an image, and correcting the image from the estimation result as if the camera performed a linear motion at a constant velocity. However, the method as described in Document 2 has a problem in that when the width of the target three-dimensional object in the depth direction is large, the three-dimensional shape calculation accuracy is reduced. In addition, the literature `` C. Tomasi and T. Kan
ade, "Shape and Motion from Image Streams: a Fact
orization Method-Full Report on the Orthographi
c Case, "Computer Science Technical Report, CMU-CS
-104, Carnegie Mellon Univ., 1992 ”(reference 3) and“ CJ Poelman and T. Kanade, “A Parap
erspective factorization method for Shape and Moti
on Recovery, ”IEEE PAMI, vol.19, no. 3, 1997, pp.
206-218 ", a method is proposed in which a plurality of images of a landscape are input, feature points between the images are associated, and three-dimensional coordinates of the feature points are obtained by a factor decomposition method. This method does not require special equipment or special consideration for input, and can be easily implemented.However, since the camera model is not a perspective transformation, the input error is large when the depth of the target object is wide. There are disadvantages.

Japanese Patent Application Laid-Open No. 7-146121, entitled "Visually Recognition Method for 3D Position and Orientation and Apparatus for Recognizing 3D Position and Orientation Based on Vision", photograph a 3D object whose position and size are known by a camera. From one of the images
A camera calibration method for estimating the position and orientation of a camera is disclosed. However, this method cannot obtain a three-dimensional shape of an unknown object or landscape.

In Japanese Patent Application Laid-Open No. Hei 8-181903, an image pickup apparatus is disclosed, in which a plurality of images are input, and a shift in parallel movement and a shift in rotation between the images are estimated and attached. However, this method cannot obtain a three-dimensional shape of an object or a landscape.

Japanese Patent Application Laid-Open No. 11-183139 discloses a device for measuring a three-dimensional shape of a target object from a plurality of images obtained by projecting slit light on the target object and photographing the target object. This apparatus has a drawback in that the apparatus for projecting slit light and the camera need to be operated in conjunction with each other, which is expensive to implement.

[0007]

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and enables highly accurate three-dimensional shapes of objects and scenes even when the width of the target three-dimensional object in the depth direction is large. An object of the present invention is to provide a method, an apparatus and a recording medium for acquiring a three-dimensional shape to be restored.

[0008]

In order to achieve the above object, the present invention can be configured as follows.

According to the present invention, an input step of inputting an image from a camera to a computer, a first extraction step of extracting a three-dimensional shape from the image by a predetermined method, and a generation step of generating a depth image from the three-dimensional shape Step, a correction step of correcting image blur due to camera fluctuation using the depth image, and a second extraction step of extracting a three-dimensional shape by the predetermined method using the image corrected for blur, Outputting the three-dimensional shape to a storage device.

According to the present invention, a three-dimensional shape is extracted by using an image which has been subjected to accurate blur correction using a depth image, so that a highly accurate three-dimensional shape can be obtained. Further, the predetermined method may be a method based on epipolar image analysis. According to the present invention, it is possible to extract a highly accurate three-dimensional shape by epipolar image analysis.

Further, the above configuration further includes a step of generating a reproduced image after extracting the three-dimensional shape, and a step of calculating a difference between the reproduced image and the image subjected to the blur correction, wherein the difference is a predetermined value. The generation step, the correction step, and the second extraction step may be repeatedly performed until the value becomes equal to or less than the value of.

According to the present invention, it is possible to further improve the accuracy of the three-dimensional shape by repeating the generation step, the correction step, and the second extraction step. Further, it is possible to determine the end of the processing by comparing with the reproduced image.

Further, according to the present invention, there is provided a three-dimensional shape obtaining apparatus suitable for carrying out the above three-dimensional shape obtaining method and a recording medium storing a three-dimensional shape obtaining program.

[0014] Other features and advantages of the present invention will become apparent from the following description, taken in conjunction with the accompanying drawings.

[0015]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Before describing the principle of the present invention, a method for estimating camera fluctuation due to camera shake or the like described in Reference 2 will be described. Here, a plurality of feature points on the screen are tracked in the time direction, and the input image is deformed so that the trajectories of these feature points are on the same scan line.

A pinhole camera model is used as a camera model used for image input. No camera rotation,
If the line of sight is on the z axis and the scan direction is on the x axis,
The projected point (X _s , Y _s ) of the object point existing at (x, y, z) is represented by the angular magnification a
Expressed as _{X s = ax / z ··· (} 1) Y s = ay / z ··· (2) used. a is a camera-specific parameter.

At time t, the camera moves around the x-axis at -α,
Assume that the rotation is -β around the y-axis and -γ around the z-axis. Projected point of time _{(X 's, Y' s} ) is obtained as follows.

First, the rotation matrix about each axis is

[0019]

(Equation 1) And the overall rotation matrix R is

[0020]

(Equation 2) Becomes In particular, when the rotation angles -α, -β, and -γ are small,

[0021]

(Equation 3) Can be approximated. The projection point is

[0022]

(Equation 4) Becomes Using the approximate expression (7),

[0023]

(Equation 5) Can be approximated.

Next, the camera position is -δ _y , -δ _z from the x-axis.
Let's consider the case of deviation. In this case,

[0025]

(Equation 6) Becomes When −δ _y and −δ _z are small,

[0026]

(Equation 7) Can be approximated. Here, the second and third terms depend on z, but when the change in z is small, they can be regarded as constants. Equation (12), from (16), the direction of the camera, variation _{D xs = X s -X 's} , D ys = Y s -Y' projection points due to fluctuation of the position _s
Is

[0027]

(Equation 8) Can be written as A (t) to E (t) are constants determined for each frame t, and represent distortion of the input image. Camera constant a
If is known from these, α (t), β (t), γ (t),
δ _y and δ _z can be obtained, and the variation in the x direction D _xs = X _s -X ′
_s can be calculated.

When A (t) to E (t) are estimated from the trajectories (X _i (t), Y _i (t)) of five or more feature points, D _ys is obtained using the estimated _values .

[0029]

(Equation 9) Can deform the input image and reduce distortion. f _new indicates the corrected input image. This estimate is, for example,

[0030]

(Equation 10) Can be obtained by solving A (t) to E (t) that minimizes by the method of least squares. Also,

[0031]

[Equation 11] Can be used for robust estimation (Z. Zhang, et.
al., “A robust technique for matching two uncalib
rated images through the recovery of the unknown e
pipolar geometry, ”Artificial Intelligence, vol.
78, 1995, pp. 87-119).

The above is the method disclosed in Reference 2, but the method for estimating camera fluctuations described in Reference 2 has drawbacks. According to Literature 2, by solving the minimization problem using the above equations (25) and (26) from the trajectory {(X's, Y's)} that tracks the feature points in the image, the camera fluctuation parameters ( The camera rotation angles α (t), β (t), γ (t) and camera translations δ _y (t), δ _z (t)) can be estimated. However, Expressions (15) and (16) cannot be applied unless the depth coordinate value (z) of the subject is known. In Document 2, assuming that the change in the depth (z) of the subject is small, z is approximately regarded as a constant. Thus, δ _y / z and δ _z / z were treated as one variable, and this value was estimated (Equations (22) and (23)). In this method, even if the subject is an object having a spread in the depth direction,
Since the depth is uniformly regarded as a constant, the estimation accuracy is low.

In the present invention, first, a three-dimensional shape is input by the method shown in Reference 2. A depth image is created using the three-dimensional shape (step 7 in FIG. 2 described later), and the camera posture is accurately estimated using depth information obtained from the depth image (step in FIG. 2 described later). 8) Realization of high-precision extraction of a three-dimensional shape is realized.

For this purpose, the video correction method is firstly combined with the above-described image correction method disclosed in Document 2 and the three-dimensional shape extraction method described in Document 1 (three-dimensional shape extraction processing based on epipolar image analysis). A three-dimensional shape is once extracted from a plurality of images taken by the camera (steps 1 to 3 in FIG. 2 described later). As described above, the image correction method disclosed in Document 2 estimates a camera variation parameter from a trajectory obtained by associating feature points in an image,
Based on the fluctuation parameters of the camera, the image is transformed into an image as if the video camera performed a linear motion. The three-dimensional shape extraction method disclosed in Document 1 forms a spatiotemporal image from a plurality of images taken while moving a video camera at a constant linear motion, and analyzes a linear pattern appearing in the image on the xt plane. To obtain a three-dimensional shape. At this time, the three-dimensional shape is obtained as a relative relationship between the movement locus of the camera and the movement speed of the camera. Also, at this time, according to the method of Document 2, since the fluctuation of the camera is estimated and known, how the camera actually moves when the subject is photographed is known.

Next, based on the three-dimensional shape obtained as described above, the camera is virtually placed at the same position and orientation as on the camera at the time of shooting on the camera trajectory at the time of shooting, and obtained when shooting from there. Create a depth image to be displayed (see FIG. 2 described later).
Step 7). This can be performed by a method of drawing a three-dimensional shape using a generally known Z buffer method or the like and taking out a Z buffer created as a result. Each pixel in the Z buffer contains a depth value from the viewpoint to the object point reflected in that pixel. As a result, the depth when the input image is captured can be obtained for each pixel. This depth is used as a first estimated value of the depth.

By the way, when estimating the fluctuation of the camera for the first time, the depth to the object point is unknown, but the estimated value up to the object point is obtained at the present time by the above-described depth image creation processing. So, using this depth information,
The fluctuation of the camera is accurately estimated again (see FIG. 2 described later).
Step 8). In the above equations (15) and (16), since the depth (z) is unknown, z is regarded as a constant. In the present invention, depth information is obtained from a depth image, and a value can be set for z. That is, the camera fluctuation is estimated by using the following equation based on the above equations (17) and (18).

[0037]

(Equation 12) Where D _xs (X ' _s , Y' _s ): coordinates of input image (X ' _s , Y'
_s ), the correction amount in the x-axis direction D _ys (X ' _s , Y' _s ): of the point at the coordinates (X ' _s , Y' _s ) of the input image
Correction amounts in the y-axis direction α, β, γ, δ _y , δ _z : Variables representing camera fluctuations determined for each input image.

[0038] a: an internal parameter X _'s, Y' camera _s: z represents the coordinate in the image of the input image ': the coordinates of the input image _{(X' s, Y 's} ) the depth of the object point that is reflected in ,.

The above equation indicates how much the pixels in the input image should be moved in order to correct the camera shake of the input image. z ′ is obtained from the pixel value at the coordinates (X ′s, Y ′s) of the depth image. The unknowns α, β, γ, δ _y , δ _z representing the camera fluctuations are the trajectories (X
_i (t), Y _i (t)) can be estimated using the least squares method or the like. This least square method is performed as follows. If the camera is moving linearly in the x-axis direction, the y-coordinate of the feature point should be constant. Therefore, the unknowns α, β, γ, δ _y , δ _z are determined by the least square method so as to satisfy the condition that the y coordinate of the corrected feature point is constant as much as possible. That is,

[0040]

(Equation 13) By solving, the unknown can be determined. Or
Reference “Z. Zhang, et. Al.,“ A robust technique for
matching two uncalibrated images through therecove
ry of the unknown epipolar geometry, ”Artificial
Intelligence, vol. 78, 1995, pp. 87-119,

[0041]

[Equation 14] Accordingly, robust estimation can be performed by excluding outliers by the median operation.

Using the unknowns α, β, γ, δ _y , δ _z representing the camera fluctuations obtained in this way, equation (27),
D _xs, the D _ys determined by (28),

[0043]

(Equation 15) However, f is an input image before correction, and f ⁺ _new can correct an image by moving each pixel of the input image from the input image after correction.

In the image correction method described here, the depth is regarded as a constant in the method disclosed in Reference 2, whereas the values of the depth are calculated in equations (27) and (28). High accuracy. Therefore, if the three-dimensional shape is extracted again using the corrected image as an input, the three-dimensional shape can be extracted with higher accuracy even if the width of the target three-dimensional object in the depth direction is large.

Further, if the above procedure is repeated, the estimated value of the depth is increased each time, and the accuracy of the depth is improved.

The number of times the above procedure is repeated is arbitrary. In the following embodiments, a method of repeating until the difference between the reproduced image and the input image becomes equal to or less than a certain value and a method of repeating the method a predetermined number of times will be described.

The algorithm for estimating camera fluctuation according to Reference 2 corrects the amount of parallel movement in the left, right, up, down, and diagonal directions on a mapped two-dimensional pixel regardless of the distance (depth) between the object and the camera. In the present invention, since the estimation is performed using the depth information, considering how far the object is away from the camera, the two-dimensional movement is performed such that the movement of the distant object is small and the movement of the close object is large. Reflect on the pixel. The above expressions 27 and 28 express such a meaning by expressions. That is, the fourth on the right side of equation 27
Term and the fourth and fifth terms on the right-hand side of equation 28 represent depth
The value z 'appears. This makes it possible to more accurately estimate the variation than when the depth is regarded as a constant.

(First Embodiment) FIG. 1 shows an example of the configuration of a three-dimensional shape obtaining apparatus according to an embodiment of the present invention. The three-dimensional shape acquisition device according to the embodiment of the present invention includes a CPU (central processing unit) 1, a memory 3, an input device 5, a display device 7, a CD, and the like.
It has a ROM drive 9 and a hard disk 11; C
The PU 1 controls the entire three-dimensional shape acquisition device. The memory 3 holds data and programs to be processed by the CPU 1. The input device 5 is a camera or the like for inputting an image. The display device 7 is a device such as a display. CD-
The ROM drive 9 drives a CD-ROM or the like to perform reading and writing. The hard disk 11 stores programs and three-dimensional shape data obtained by the processing of the present invention. The three-dimensional shape acquisition device of the present invention operates by a program that executes a three-dimensional shape acquisition process based on the principle of the present invention. The program may be installed in the three-dimensional shape acquisition device in advance, or for example, a CD-RO
M, and may be loaded into the hard disk 11 via the CD-ROM drive 9. When the program is started, a predetermined program portion is expanded in the memory 3 and the processing is executed. In the present invention, an image is input from the input device 5, a three-dimensional shape extraction process is performed, and the result is output to the display device 7 or the hard disk 11.

Next, the procedure of a three-dimensional shape acquisition process executed by the three-dimensional shape acquisition device will be described with reference to FIG. This processing is based on the principle of the present invention.

In step 1, a subject is photographed while the camera is moved in parallel with the direction of the optical axis, and an image is input. At this time, unknown motion may be included in the motion of the camera due to camera shake or other factors.

In step 2, as camera shake correction,
The image is deformed to remove the effect of camera shake. This processing can be performed by equation (24). After this,
Performs processing by epipolar analysis.

The epipolar image analysis is based on the condition that the camera is shooting an object while moving in parallel with the optical axis direction. Hereinafter, the parallel movement in the direction perpendicular to the optical axis is defined as an ideal movement. Call. If the camera moves unknown due to camera shake or other factors, the accuracy of the epipolar image analysis will be significantly reduced. The camera shake correction processing in step 2 estimates how much the camera deviates from ideal movement (hereinafter referred to as camera fluctuation) in order to improve the accuracy of epipolar image analysis, and based on the estimation result, To obtain an image as if the camera were photographed while moving ideally.

In the three-dimensional shape extraction processing based on the epipolar image analysis in step 3, the image after the above-mentioned camera shake correction is input and a three-dimensional shape is extracted. This processing can be performed, for example, by the method described in Document 1.

In the reproduction image creation processing of step 4, the input image is reproduced using the above-described three-dimensional shape. In this process, a camera is virtually placed on each of the actual camera movement trajectories, and an image obtained when an object is photographed from this point is created. To perform this image creation processing, for example, general perspective transformation and the Z-buffer method are used, but the present invention is not limited thereto. The reproduction image creation is used for determining the end of the loop.

In the difference calculation process between the reproduced image and the input image in step 5, the difference between the reproduced image and the input image is calculated to determine the end of the loop. If the extraction result of the three-dimensional shape and the estimation result of the fluctuation of the camera are accurate, the reproduced image reproduced here and the input image should exactly match. However, in reality, they do not match because the estimation of camera fluctuation is not accurate. Therefore, in this step, the difference is evaluated. Specifically, in this embodiment, the expression

[0056]

(Equation 16) Where ε: difference I _t (x, y): luminance value of coordinates (x, y) of t-th input image S _t (x, y): luminance value of coordinates (x, y) of t-th reproduced image The value is calculated, but other formulas may be used.

In step 6, a condition is determined. That is, it is determined whether the difference is equal to or less than a certain value or the number of loops has reached a certain number, and if the result is true, the loop is exited. If the result is false, the loop is counted up once and the loop is continued. The fixed value of the difference used for the determination and the fixed value of the number of loops are set to values deemed appropriate by the practitioner, and given from outside.

In the depth image creation processing in step 7, depth images corresponding to each of the above-described reproduced images are created. A depth image is a grayscale image in which the luminance of a pixel of an image is associated with the distance to an object shown in that pixel. FIG. 3 shows an example of the depth image.

In step 8, camera shake correction using the depth information is performed. That is, camera shake correction is accurately performed using the depth information included in the above-described depth image. This processing can be performed using Equations (27) to (30) as described as the principle of the present invention.

In step 6, when the difference has reached a certain value or the number of loops has reached a certain number, the three-dimensional shape finally obtained in step 9 is output to, for example, a file.

FIG. 4 is a diagram showing the results of the implementation of the present invention. An outdoor scene was used as an input image, and the operation was performed based on the flow of the above embodiment. In order to perform the three-dimensional shape extraction processing based on the epipolar image analysis in Step 3, the method described in Reference 1 and the reference “M. Shiny
a, T. Saito, T. Mori and N. Osumi, “VR Models fro
mEpipolar Images: An Approach to Minimize Errors i
n Synthesized Images, ”LNCS 1352 Computer Vision-
ACCV '98, Vol. II, pp. 471-478 "
-Used in combination with the Strip method. The DP-Strip method is a method of determining the topological structure of a plane between restored object points, and a method of determining a topological structure that minimizes a difference between a reproduced image and an input image by a dynamic programming method. It is. By determining the topological structure of the surface, it becomes possible to reproduce the occlusion phenomenon that the front surface hides a distant object.

FIG. 4A is a diagram showing a spatiotemporal image of an input image. FIG. 4 also shows an epipolar image in order to clearly show the effect of the present invention. That is, FIG. 4B is an epipolar image of the input image on the cutting plane in FIG. It can be said that the closer the reproduced image in which the three-dimensional shape extracted in Step 3 is reproduced to the input image, the more accurately the shape is reproduced.

FIG. 4C is an epipolar image reproduced using a three-dimensional shape obtained as a result of performing a three-dimensional shape extraction without applying the present invention, which is seen in a circled portion. The occlusion is different from that of the input image and 3
It can be seen that the accuracy of dimensional shape extraction is low.

FIG. 4D shows that, for the same input data,
This is an example in which the present invention is implemented with one repetition. The occlusion in the circled area is improved closer to the input image. This is considered to be because the depth estimation of the object point was improved.

FIG. 4E shows that, for the same input data,
This is an example in which the present invention is implemented with two repetitions. FIG.
In addition to the improvement shown in (d), the occlusion in the circled part is improved close to the input image. This means
It is considered that the depth estimation of the object point was further improved.

(Second Embodiment) Next, a second embodiment of the present invention will be described. FIG. 5 is a flowchart of the process in the second embodiment. In the second embodiment, an example will be described in which an image obtained when an image is captured from another camera position and orientation is created from one image captured by the camera.

In step 10, one image is input. Here, equations (27) and (28) according to the present invention are used.
Can be used as an expression indicating how much each pixel of this image moves when the camera is changed with respect to the image taken by the camera. Here, the camera fluctuation is
The rotation angles α, β, and γ around the x, y, and z axes and the minute parallel movement distances δ _y , δ _z in the y and z axis directions are represented by pixel movement amounts D _xs and D _ys. If the depth z 'of the object point reflected in the pixel is not known, it cannot be calculated accurately.
Therefore, in step 11, the depth value of each pixel of the input image is input. As a method for obtaining the depth value, for example, when the input image is one of a large number of images taken by a video camera, the shape of the object is obtained by a factor decomposition method as shown in Reference 3. There is a method of calculating the depth.

In the process of creating an image obtained when the camera is changed in step 12, the depth obtained in step 11 is substituted for z 'in equations (27) and (28), and D
_The parameters α, β, γ, δ _y , δ _z are obtained by calculating _xs and D _ys and moving the pixels of the input image according to equation (30).
An image obtained when photographing from another camera position represented by can be created. As described above, by using the expressions (27) and (28) according to the present invention, it is possible to accurately create an image obtained when photographing from another camera position.

The present invention can be applied to a method other than the three-dimensional shape extraction method based on the epipolar analysis method. In particular, it can be effectively applied to a three-dimensional shape extraction method that is vulnerable to unexpected movement of the camera.

It should be noted that the present invention is not limited to the above-described embodiment, but can be variously modified and applied within the scope of the claims.

[0071]

According to the present invention, a three-dimensional shape is extracted by using an image which has been subjected to accurate blur correction using a depth image. Therefore, even when the width of the target three-dimensional object in the depth direction is large, It is possible to restore the three-dimensional shape of the object or the landscape with high accuracy. Further, according to the method described in the second embodiment, an image obtained when photographing from another camera position can be accurately created.

[Brief description of the drawings]

FIG. 1 is a configuration example of a three-dimensional shape acquisition device according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a procedure of a three-dimensional shape acquisition process according to the first embodiment.

FIG. 3 is an example of a depth image obtained in step 7;

FIG. 4 is a diagram showing the results of the implementation of the present invention.

FIG. 5 is a flowchart of a process in the second embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 CPU 3 Memory 5 Input device 7 Display device 9 CD-ROM drive 11 Hard disk

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Tatsuki Matsuda 2-3-1 Otemachi, Chiyoda-ku, Tokyo Within Nippon Telegraph and Telephone Corporation (72) Mikito Notomi 2-chome, Otemachi, Chiyoda-ku, Tokyo No. 1 Nippon Telegraph and Telephone Corporation F-term (reference) 5B057 BA02 CA12 CA16 CB13 CB16 CD02 CD03 CD14

Claims (15)

[Claims] 1. An input step of inputting an image from a camera to a computer, and a first step of extracting a three-dimensional shape from the image by a predetermined method.
Extraction step; a generation step of generating a depth image from the three-dimensional shape; a correction step of correcting image blur due to camera fluctuation using the depth image; and a predetermined step of using the blur-corrected image. A three-dimensional shape obtaining method, comprising: a second extracting step of extracting a three-dimensional shape by the method of (1); and an output step of outputting the three-dimensional shape to a storage device. 2. The method according to claim 1, wherein the predetermined method is a method based on epipolar image analysis. 3. The method according to claim 1, further comprising: generating a reproduced image after extracting the three-dimensional shape; and calculating a difference between the reproduced image and the blur-corrected image, wherein the difference is equal to or less than a predetermined value. The three-dimensional shape acquisition method according to claim 1, wherein the generation step, the correction step, and the second extraction step are repeatedly performed until the generation step. 4. The correction step comprises: setting a line of sight direction to the z axis, a camera scanning direction to the x axis, a as a camera constant, and (X _s , Y _s ) an object point existing in (x, y, z). Let (X ' _s , Y' _s ) be the camera around the x-axis
-β around the y-axis, -γ around the z-axis, and -δ _y , -δ _z from the x-axis as projection points, (D _xs , D _ys )
Was a variation of the projected point due to variations in the orientation and position of the camera, 'obtained from the depth image _{(X' z s, Y '} s) to the depth _{of, D xs (X' s,} Y 's) _{= αX 's Y' s /} a-β (a + X 's 2 / a) -γY' s +
(δ _z / z ') X' _s , D _ys (X ' _s , Y' _s ) = α (a + Y ' _s ² / a) -βX' _s Y ' _s / a + γX' _s
+ (δ _z / z ') Y' _s -aδ _y / z ', f ⁺ _new (X _s , Y _s ; t) = f (X' _s + D _xs (X ' _s , Y' _s ), Y ' _s + D
_{ys (X 's, Y'} s); t), 3 -dimensional shape acquiring method according to claim 2 including the step of deforming the input image f to f ⁺ _{new new} by. 5. An image inputting step of inputting an image captured from a certain camera position to a computer, a depth inputting step of inputting a depth value of the image, and using the depth value to correct an image blur due to camera fluctuation. Creating an image taken from a different camera position by applying an equation for performing. 6. An input means for inputting an image from a camera, and a first means for extracting a three-dimensional shape from the image by a predetermined method.
Extraction means, generation means for generating a depth image from the three-dimensional shape, correction means for correcting image blur caused by camera fluctuation using the depth image, and the predetermined method using the blur-corrected image. A three-dimensional shape obtaining apparatus, comprising: a second extracting unit that extracts a three-dimensional shape by the method according to (1); and a storage device that stores the extracted three-dimensional shape. 7. The three-dimensional shape acquiring apparatus according to claim 6, wherein the predetermined method is a method based on epipolar image analysis. 8. A means for generating a reproduced image after three-dimensional shape extraction, and means for calculating a difference between the reproduced image and the blur-corrected image, wherein the difference is equal to or less than a predetermined value. The three-dimensional shape acquisition device according to claim 6, wherein the processing by the generation unit, the correction unit, and the second extraction unit is repeated until the processing. 9. The method according to claim 1, wherein the correcting unit sets the line of sight to the z axis, the scan direction of the camera to the x axis, a as a camera constant, and (X _s , Y _s ) an object point existing at (x, y, z). Let (X ' _s , Y' _s ) be the camera around the x-axis
-β around the y-axis, -γ around the z-axis, and -δ _y , -δ _z from the x-axis as projection points, (D _xs , D _ys )
Was a variation of the projected point due to variations in the orientation and position of the camera, 'obtained from the depth image _{(X' z s, Y '} s) to the depth _{of, D xs (X' s,} Y 's) _{= αX 's Y' s /} a-β (a + X 's 2 / a) -γY' s +
(δ _z / z ') X' _s , D _ys (X ' _s , Y' _s ) = α (a + Y ' _s ² / a) -βX' _s Y ' _s / a + γX' _s
+ (δ _z / z ') Y' _s -aδ _y / z ', f ⁺ _new (X _s , Y _s ; t) = f (X' _s + D _xs (X ' _s , Y' _s ), Y ' _s + D
_{ys (X 's, Y'} s); t), claim including means for deforming the input image f to f ⁺ _{new new} by 7
3. The three-dimensional shape acquisition device according to 1. 10. An image input means for inputting an image photographed from a certain camera position, a depth input means for inputting a depth value of the image, and using the depth value to perform image blur correction due to camera fluctuation. Creating means for creating an image taken from another camera position by applying the formula: 11. A computer-readable recording medium in which a program for causing a computer to execute a process of acquiring a three-dimensional shape is recorded, wherein: an input procedure for inputting an image from a camera to the computer; First to extract 3D shape
Extraction procedure; a generation procedure for generating a depth image from the three-dimensional shape; a correction procedure for correcting image blur due to camera fluctuation using the depth image; and a predetermined procedure using the blur-corrected image. A computer-readable recording medium recording a program for causing a computer to execute a second extraction procedure for extracting a three-dimensional shape by the method according to the above, and an output procedure for outputting the three-dimensional shape to a storage device. 12. The computer-readable recording medium according to claim 11, wherein the predetermined method is a method based on epipolar image analysis. 13. A method for generating a reproduced image after extracting a three-dimensional shape, further comprising: calculating a difference between the reproduced image and the blur-corrected image, wherein the difference is equal to or less than a predetermined value. 2. The method according to claim 1, wherein the generation step, the correction step, and the second extraction step are repeatedly performed until the time point.
A computer-readable recording medium recording the program according to claim 1. 14. The correction procedure according to claim 1, wherein the viewing direction is the z-axis, the camera scanning direction is the x-axis, a is a camera constant, and (X _s , Y _s ) is an object point existing in (x, y, z). Let (X ' _s , Y' _s ) be the camera around the x-axis
-β around the y-axis, -γ around the z-axis, and -δ _y , -δ _z from the x-axis as projection points, (D _xs , D _ys )
Was a variation of the projected point due to variations in the orientation and position of the camera, 'obtained from the depth image _{(X' z s, Y '} s) to the depth _{of, D xs (X' s,} Y 's) _{= αX 's Y' s /} a-β (a + X 's 2 / a) -γY' s +
(δ _z / z ') X' _s , D _ys (X ' _s , Y' _s ) = α (a + Y ' _s ² / a) -βX' _s Y ' _s / a + γX' _s
+ (δ _z / z ') Y' _s -aδ _y / z ', f ⁺ _new (X _s , Y _s ; t) = f (X' _s + D _xs (X ' _s , Y' _s ), Y ' _s + D
_{ys (X 's, Y'} s); t), claim comprising the steps of deforming the input image f to f ⁺ _{new new} by 1
3. A computer-readable recording medium recording the program according to 2. 15. A computer-readable recording medium storing a program for causing a computer to execute an image creation process, comprising: an image inputting step of inputting an image captured from a certain camera position to a computer; A depth input procedure for inputting, and a creation procedure for creating an image photographed from another camera position by using the depth value and applying an equation for performing image blur correction due to camera fluctuation, to the computer. A computer-readable recording medium on which a program to be recorded is recorded.